Forem: Alex Anie

Scrape and build a LinkedIn user profile app with Bright Data and React.js

Alex Anie — Sat, 14 Mar 2026 11:30:04 +0000

Imagine a situation where you are given a task to build a user dashboard to help keep track of users’ contacts based on their LinkedIn profiles.

To do this, you can simply go to LinkedIn to copy as many contacts as possible and then type and format the collected data in a JSON format or the exact format you want for the project. But there is a problem with this approach.

You can end up making so many typos or spend a stupendous amount of time doing repetitive work that can be used to build the actual app.

Better still, you can take the web scraping approach to manually scrape the data and then export it as a CSV or JSON file. But this is not a scalable option either, as many users tend to edit, change, or update their profiles regularly, leaving the collected data you have out of date.

So the question remains, “How do I get an up-to-date user profile dataset and then keep track of the changes?” Well, in this case, you can make use of Brightdata, a proxy network for bypassing and scraping ready-to-use, up-to-date, structured large datasets.

These datasets can be exported in a CSV or JSON format for easy implementation in your application.

In this blog post, you’ll learn about how to scrape the LinkedIn user profile dataset and download it as a JSON file locally. Additionally, you’ll also learn how to dynamically render each user profile when a particular user is clicked on.

We are going to achieve this using Brightdata and React.js, and by the end of this blog, you should be able to use Brightdata to scrape blocked datasets and implement the scraped data in a React application.

Prerequisites

For a better understanding, a basic knowledge of the following is required.

React.js
Web Scraping

Overview

Here is a preview of what we’ll be building. It’s a simple but feature-rich app that stores LinkedIn user profile contacts.

Click the names to open specific information about each user. 👉 To get started, download the GitHub file here.

Now that we have that out of the way, let’s get started.

Create a Bright Data Account

Firstly, we have to create an account with Bright Data to begin scraping data. Create an account by clicking the Start free with Google button or log in to Bright Data if you already have an account.

You will be directed to the dashboard. Click the datasets & web scraper IDE icon.

Next, scroll down to the Web Scraper IDE section and click Get Started.

Next, from the dialog box, select the first item “LinkedIn People's Profile”, then hover and click Use Template.

This will open the web scraper IDE interface as indicated below 👇

Web Scraper IDE Interface

The Web Scraper IDE is composed of distinct sections for interacting with and collecting data.

A: See more examples: With this button, you can choose another dataset template from the list of predefined templates
B: Add Another Step (Stage): With this button, you can add another stage. This helps to scrape data from multiple pages.
C: Help: List of predefined functions to help add extra functionality when collecting dataset.
D: Debugging Tools: This tab contains tools for testing, running, and debugging collected datasets before its ready for use.
- Input: used to add parameters and run tests on collected datasets.
- Output: This is where a list of collected datasets and their parameter are listed.
- Children: List of children which will be input sets of the next stage.
- Run Log: Where collected dataset parameters are logged.
- Browser Console: Where collectors are logged. Function similarly to the browser console tab.
- Browser Network: Where network collector requests are displayed. Functions similarly to the browser network tab.
- Last Errors: Where the latest error information is listed
E: input:
- Add input: To set a new input parameter such as its name and type.
- Add another input: To add another parameter
- Preview: To preview the already established parameter.
F: Settings:
- For setting: A set of behaviors for the collector error case
- Take Screenshots: Take screenshots during the preview test.

With the web scraper IDE, we can define how our collected dataset is processed and structured. You can read more on the docs here.

Scrape your first dataset

Now that we have an idea of how the web scraper IDE works and have a collected dataset set up.

Next, go to the preview section and click on the play icon to preview the structure of your collected dataset. This preview section contains HTML elements of how your collected dataset is structured on the webpage.

However, to see the actual browser preview, click on the Pop-out button.

This will open an “about:blank” page with a visual preview of what is listed on the dataset. This is very useful as it gives you a preview of the type of dataset you are retrieving or working with.

From the browser preview, we have a preview of a user profile with different types of data about the user.

Once you are satisfied with your desired data collection, head back to the Web Scraper IDE and click on Finish Editing.

This will open the initiate manually tab. Here you will see all the defined parameters for your dataset. These parameters are the individual URLs of the user’s profile we need to build our application.

You can copy any of the URLs listed below and paste it into your browser address bar to see what type of dataset set it contains.

Next, scroll down and click on the start button.

This will open the Runs tab. Here, our collected dataset will go through a retrieval process. This will take some time, depending on the speed of your network.

Notice the downloading animation indicated by the blue arrow.

Once the dataset has been retrieved (downloaded), hover on the Download file option button and click on the option icon, then click on the quick view option.

From the preview dialog box, you can see all the retrieved datasets collected. Next, click on the Download file button to download the dataset. This will enable us to get access to the raw JSON file for local development.

It’s important to note that there are other options for retrieving this same dataset aside from downloading it directly.

To do this, click on the little option button beside the More tabs and select the delivery preferences.

From the Delivery Preferences tab, we can specify how, when, and the type of format we want the dataset to have.

Depending on the type of application you are building, this might be just what you need.

However, due to the scope of this tutorial, we will not be covering this section. This is just to highlight that this option exists.

React.js set up with Vite

Now that we’ve completed scraping the dataset that we need for building our app. Let’s get our project up and running.

Create a folder on your Desktop and name it user-contact, then open it on VS Code like so.

Still on VS code, click on Ctrl + backtick ( ` ) to open the terminal.
Type the command npm create vite@latest to initialize Vite

Type dot . and press enter to replace the vite-project name with the project folder as the default name.
Select react and press enter.

Select JavaScript and press enter

Type npm install and press enter to scaffold your project. Your terminal should display the following details to show the installation is complete.

Before we run the dev server, we need to install the necessary dependencies needed to build the project. This includes;

react-router-dom 6.4: for client-side routing. To install run npm i react-router-dom on the terminal and press enter.
feather icons: for adding icons. To install run npm i react-feather on the terminal and press enter. This will install feather icon for react.

Once you have react-router-dom and feather-icons installed. Next, install tailwindcss by following the commands below.

Install tailwindcss via npm npm install -D tailwindcss postcss autoprefixer
Next, create your tailwind.config.js file and postcss.config.cjs npx tailwindcss init -p

This will generate two files as indicated above.

Next, copy and paste the code below on your tailwind.config.js file to Configure your template paths

js /** @type {import('tailwindcss').Config} */ export default { content: ["./index.html", "./src/**/*.{js,ts,jsx,tsx}"], theme: { extend: {}, }, plugins: [], }

Next, delete the default styles in your index.css files then copy and paste the @tailwind directives for each of Tailwind’s layers to your i_ndex.css_ file.

js @tailwind base; @tailwind components; @tailwind utilities;

Now that we’ve included @tailwind directives, the next thing to do is to include the font family we need for styling the font on the app.

Copy and paste the URL for the raleway font to the index.css files as indicated below.

`js
@tailwind base;
@tailwind components;
@tailwind utilities;

@import url('https://fonts.googleapis.com/css2?family=Raleway:ital,wght@0,200;0,300;0,400;0,500;0,600;0,700;0,800;0,900;1,100;1,200;1,300;1,400;1,500;1,600;1,700;1,800;1,900&display=swap');
`

Next, go to the tailwind.config.js file and edit it as follows to enable tailwindcss to recognize the font. This will enable us to use it with the tailwindcss utility classes.

js /** @type {import('tailwindcss').Config} */ export default { content: ["./index.html", "./src/**/*.{js,ts,jsx,tsx}"], theme: { extend: { fontFamily: { raleway: ['Raleway', 'sans-serif'] } }, }, plugins: [], }

We have now successfully installed tailwindcss in our project. let’s set up the folder structure for rapid development.

Folder structure setup

To set up the folder structure of our project, delete the App.css, App.jsx and react.svg from the project folder. Here is an overview of what the complete folder structure looks like

js 📁src 📁apis ├── DisplayError.jsx 📁assets ├── alexanie.png ├── colorful.jpg ├── exception.jpg 📁components ├── ContentList.jsx 📁layouts ├── DisplayContent.jsx ├── SideNav.jsx ├── DisplayError.jsx ├── index.css ├── main.jsx index.html
Copy the images from the completed project on GitHub here.

Creating Routes
Now that we’ve sorted out the folder structure, we need to specify client-side routing. We do this with the help of react-router we’ve just installed.

To get started, click the main.jsx file open, delete the import App from './App.jsx’ statement, and import the following functions from react-router-dom as indicated below.
import React from 'react'
import ReactDOM from 'react-dom/client'
import './index.css';

import {
createRoutesFromElements,
createBrowserRouter,
Route,
RouterProvider
}

from "react-router-dom";

Next, create a router function and then parse the createBrowserRouter, createRoutesFromElements as follows. With Route, the URL segments, components, data loaders, mutations, complex layouts, and route nesting are coupled together to achieve client-side routing. This makes it easy to navigate from one route to the other in a single-page application.
const router = createBrowserRouter(
createRoutesFromElements(

)
)

Next, replace the component in the ReactDOM.createRoot function with the RouterProvider and then parse the router function to the router props as specified below.
ReactDOM.createRoot(document.getElementById('root')).render(

)

If everything above seems foreign to you, try learning how react-router-dom handles routing.

Next, go back to router function and add the path and element props. Inside the element props, parse the HTML tag

Hello World!

as indicated below.
Now that we have the routing set up, let’s start the dev server to see a live preview of what we’ve been working on.

Go to the terminal and type npm run dev then press enter.
Next, press control key and click on the dev server path 👉 http://localhost:5173/. You should see the Hello World display as indicated below. 👇

Hurray! 🎉, if you made it this far in this tutorial, congrats. Our app is now live.

Sidebar Components
While printing Hello World! feels great. There is much into building an app than just plain text, in the section, we are going to be building the Sidebar component. This is where the header, names of users, and footer will reside. For the sake of understanding, we’ll break this into three sections.

Header: where the search bar will be placed
ContentList: where names of each user will be placed.
Footer: Noting special just a simple footer that directs you to my X page 🤓 and you can customize it with your user profile.
Header section
Create a folder and component as src/layouts/SideNav.jsx
Next, open the SideNav.jsx components, then type the code below.
import {Search} from "react-feather";

export default function SideNav() {

return (
<>

<aside className="renderOutlet">

</aside>

</>

)
}

The tag contains all content that goes inside the SideNav components while inside the tag, we design the heading of the navbar.

Next, go to main.jsx Components, import the SideNav Component and change the Hello World to component as indicated below.
import SideNav from './layouts/SideNav';

const router = createBrowserRouter(
createRoutesFromElements(
}>
)
)

Your app should look just like this 👇

ContentList section

Create a folder as apis and move the downloaded dataset inside the APIs folder.

For readability purposes rename the file to mylinkedin.json.

Next, create a folder and component as src/component/ContactList.jsx.
Next, open the ContactList.jsx component and type the code below

`js
import { NavLink } from 'react-router-dom';
import linkedinData from '../apis/mylinkedin.json';

export default function ContactList() {
const activeState = ({ isActive }) => {
return {
color: isActive ? "rgb(59 130 246)" : "",
borderLeft: isActive ? '3px solid rgb(59 130 246)' : ""
};
};

const names = linkedinData.map((name)=> (
        <NavLink key={name.id} className="p-2 my-3 block font-raleway hover:bg-blue-300/20 transition-all rounded-xl" style={activeState} to={`profile/${name.id}`}>
          <span>{name.name}</span>
        </NavLink>
    ))

return (
<>

{names}

</>
)
}
`

What we are doing here is simply mapping through the arrays of object data and returning all the names in an anchor tag.

Next, go to the SideNav component and create a div and import the ContactList.jsx as indicated below.

`js
import {Search} from "react-feather";
import ContactList from "../components/ContactList";

export default function SideNav() {

return (
<>

{/* some other code... */}

        <div className="p-4 mt-[8em]">
👉             <ContactList />
        </div>

    </section>

Your app should have all the names listed as shown below.

Footer Section

Download these images and move them into the assets folder. Next, go to the SideNav component and type the code below.

`js
import {Search} from "react-feather";
import ContactList from "../components/ContactList";
import alexImages from "../assets/alexanie.png";
import { Link } from "react-router-dom";

export default function SideNav() {
const alexData = {
firstName: 'Alex',
lastName: 'Anie',
img: alexImages,
x: 'https://twitter.com/alexanie_'
}
return (
<>

{/* some other code... */}

        <div className="absolute bottom-0 bg-white w-[17em] py-2 px-8 hover:bg-blue-200 transition-all ease-linear">
            <Link to={alexData.x} className="flex justify-start" target="_blank">
                <div className="w-[2.5em] h-[2.5em] bg-slate-200 rounded-full py-2 px-2">
                    <img src={alexData.img} alt=""  className="w-full rounded-full"/>
                </div>
                <span className="font-raleway font-semibold my-2 mx-4 text-blue-600">{`${alexData.firstName} ${alexData.lastName}`}</span>
            </Link>
        </div>
    </section>
</aside>
<aside className="renderOutlet">

</aside>

</>

)
}
`

Your app should have the little footer below 🤓

Handling Error in your Application - errorElement

If you click on any of the names on the SideNav, your browser will display an error page. For example, click on the first name khon noma and notice how react-router throws an error page.

React Router refers to this as Not Found Page or simply 404. This means the page we are trying to access does not exist or is not specified on the path. To handle this we use errorElement.

With the errorElement, we can create a page that displays to the users what went wrong when a particular error occurs such as 404, and indicate on the page what action the user should take.

To do this, simply create a new component as src/DisplayError.jsx.
open the DisplayError.jsx component and type the code below.

`js
import { useRouteError, Link } from "react-router-dom"
import imgError from "./assets/exception.jpg";
export default function DisplayError() {
const error = useRouteError();

return (
<>
width: "100vw",
height: "100vh",
position: "relative",
backgroundImage: `url('${imgError}')`,
backgroundPosition: "center center",
backgroundSize: "cover",
backgroundRepeat: "no-repeat",
}}>

            <div className="absolute top-[8em] left-[40%] ">
                <p className="text-[10em] text-red-600 font-mono font-extrabold">{error.status}</p>
            </div>

        <div className="absolute bottom-12 left-[35%]">
            <p className="font-mono text-center text-slate-700 bg-slate-100 px-2 rounded-full">click <Link to="/" className="text-red-600">here</Link> to go back to home page</p>
            <p className="font-mono text-center text-slate-700 bg-slate-100 px-2 rounded-full">{error.error.message}</p
        </div>
    </main>
</>

)
}

Note that we are using a hook from react-router called useRouteError that helps handle the error that occurs during an action.

Next, go to the main.jsx component import the DisplayError component and add it to Route.

`js
import DisplayError from './DisplayError';

const router = createBrowserRouter(
createRoutesFromElements( 👇
} errorElement={}>
)
)
`

Now click any of the names and see what happens. You should see something like this.

The page above displays an error code of 404 letting the user know that the page can not be found with a text redirecting them to the home page.

Click on the text that says click here to go back to home page.

Implementing Dynamic Segment

The main idea of our application is to create an app with a list of users generated from LinkedIn. The app contains a list of user names and upon clicking, reviews the details of each user. To achieve this behavior, we need to implement what is referred to as a dynamic segment using the useParams hook.

In the layouts folder, create a component as DisplayContent.jsx
Next, type the code below inside the DisplayContent component

`js
import { BookOpen, MapPin } from "react-feather";
import linkedinData from "../apis/mylinkedin.json";
import bgImg from "../assets/colorful.jpg";
import { Link, useParams } from "react-router-dom";

export default function DisplayContent() {
const {profileId} = useParams();

const profile = linkedinData.find(profile => profile.id === profileId);

if(!profile){
  return <p>Profile not found</p>
}

return (
<div className="bg-slate-200 ">
  {
    profile && (
      <>
      <section className="grid grid-cols-6 gap-8 w-[95%] mt-0 mx-auto overflow-y-scroll h-[100vh] pr-[1em]" id="scrollStyles">
        {/* Header */}
        <aside className="col-span-4 rounded-2xl bg-white mt-12 relative">
          <div className="w-full h-[10em] bg-blue-700 rounded-t-2xl">
            <img src={bgImg} alt="" className="w-full h-full rounded-t-2xl object-left-bottom" />
          </div>
          <div className="w-[7em] h-[7em] absolute top-[6em] left-12"> 
            <img src={profile.avatar} alt="" className="rounded-full w-full h-full border-4 border-white"/>
          </div>
          <div className="py-[4em] px-[2em] grid grid-cols-6 gap-8">
              <div className="col-span-2">
                <h1 className="text-xl font-bold mb-2">{profile.name}</h1>
                <p className="text-sm">
                  <BookOpen size={12} className="inline-block"/>
                  <em className="not-italic inline-block">{profile.position}</em></p>
                <p className="text-sm mt-4">
                  <MapPin size={12} className="text-slate-500 inline"/>
                  <em className="not-italic text-slate-600 inline-block pl-2">{profile.region}</em>
                </p>
                <Link to={profile.url} target="_blank" className="text-sm mt-2 block text-blue-700 font-bold hover:text-blue-500">Connect on Linkedin</Link>
              </div>

              <div className="col-span-4">
                <div>
                  <h2 className="font-bold text-xl mb-4">
                    About
                  </h2>
                  <p className="text-sm">
                    {profile.about}
                  </p>
                </div>
                <p>
                  <span className="text-sm mt-2 text-blue-700 font-bold  inline-block cursor-default">following <em className="not-italic text-slate-500 font-normal px-2">{profile.following}</em></span>
                  <span className="text-sm mt-2 text-blue-700 font-bold  inline-block cursor-default">followers <em className="not-italic text-slate-500 font-normal px-2">{profile.followers}</em></span>
                  <span className="text-sm mt-2 text-blue-700 font-bold inline-block cursor-default">connections <em className="not-italic text-slate-500 font-normal px-2">{profile.connections}</em></span>
                </p>
              </div>
          </div>
        </aside>

          {/* Experience */}
          <aside className="col-span-2 bg-white rounded-2xl mt-12 py-8 px-4">
          <h2 className="font-bold text-xl pb-3">Experience</h2>
          {
            profile.experience.map((ele, index)=> (
              <>
              <section className="flex gap-x-4 mt-8">
              <div className="w-[50px] h-[50px] bg-red-600 rounded-xl">
                <p className="text-center font-block text-3xl mt-1">{ele.title.charAt(0)}</p>
              </div>
                <div key={index}>
                  <p className="font-bold text-sm">{ele.title}</p>
                  <p><span className="text-slate-600 text-sm">Duration</span> <span className="text-sm">{ele.duration}</span></p>
                </div>
              </section>
              </>
            ))
          }
          </aside>

        {/* Posts */}
        <aside className="col-span-4 bg-white rounded-2xl py-8 px-4  h-[fit-content]">
          <h2 className="font-bold text-xl pb-3">Posts</h2>
          <div className="flex gap-4">
            {
              profile.posts.map((ele)=>(
                <>
                      <Link to={ele.link} className="">
                          <div>
                            <img src={ele.img} alt={ele.title} className="rounded-xl" />
                          </div>
                          <p className="mt-4 font-bold">{ele.title}</p>
                          <p className="text-slate-500 my-2">{ele.attribution}</p>
                          <p className="text-sm">{ele.created_at}</p>
                      </Link>
                </>
              ))
            }
            </div>
        </aside>

        {/* Education */}
        <aside className="col-span-2 bg-white rounded-2xl py-8 px-4 mb-[8em] h-[fit-content]">

        <h2 className="font-bold text-xl pb-3">Education</h2>
            {
              profile.education.map((ele)=>(
                <>
                  <div className="flex">
                    <div className="w-[50px] h-[50px] bg-blue-600 rounded-xl text-white">
                      <p className="text-center font-block text-3xl mt-1">{ele.title.charAt(0)}</p>
                    </div>

                    <Link to={ele.url} target="_blank" className="ml-4 mb-8">
                        <div>
                          <h2 className="font-bold">{ele.title}</h2>
                        </div>
                        <p className="my-2"><span className="text-slate-700 inline-block pr-4 text-sm">Degree</span><span className="text-sm">{ele.degree}</span></p>
                        <p><span className="text-slate-700 inline-block pr-4 text-sm">Field</span><span className="text-sm">{ele.field}</span></p>
                        <p><span className="text-sm mt-4">Start Year {ele.start_year}</span> | <span className="text-sm">End Year {ele.end_year}</span></p>
                    </Link>
                  </div>
                </>
              ))
            }
        </aside>
      </section>
      </>
    )
  }
</div>

)
}

When we click on a particular user name from the SideNav, it appears on the address bar. So the find() method checks if the user. id present in the LinkedIn data matches with the dynamic segment (params) parent in the URL (address bar) and if this is true. The specific user data from the LinkedIn data should rendered on the page.

If the LinkedIn data does not match, <p>profile not found</p> should be printed on the page.

Next, go to main.jsx, import the DisplayContent component, and add it to Route.

`js
import DisplayContent from './layouts/DisplayContent';

const router = createBrowserRouter(
createRoutesFromElements(
} errorElement={}>
} />

)
)

Since the DisplayContent is a nested route, we need to render it as an Outlet.
Go to the SideNav.jsx, import the Outlet component from react-router-dom and render it inside the renderOutlet tag.

`js
👉 import { Link, Outlet } from "react-router-dom";

{/* some other code... */}

👉

`

Now, click on any of the names in the SideNav. Your app should look like this 👇

Styling the Scrollbar of the application

The page scrollbar still keeps the default styling. Let’s add some styling to make it look nice.

Go to the SideNav and scrollStyles to the id attribute on the section tag.

`js

👉
{/* Some other code... /}

{/ Some other code... */}

Next, go to the DisplayContent.jsx and scrollStyles to the id attribute on the section tag.

`js

Next, go to the index.html file and add the style rules.

`js

<!doctype html>

User Content

<br>
body {<br>
overflow: hidden;<br>
}</p>
<div class="highlight"><pre class="highlight plaintext"><code> #scrollStyles::-webkit-scrollbar {
width: 10px;
}

#scrollStyles::-webkit-scrollbar-track {
background: #f1f1f1;
}

#scrollStyles::-webkit-scrollbar-thumb {
background: #888;
border-radius: 5px;
}

#scrollStyles::-webkit-scrollbar-thumb:hover {
background: #555;
}
</style>
</code></pre></div>
<p></head><br>
<body><br>
<div id="root"></div><br>
<script type="module" src="/src/main.jsx"></script><br>
</body><br>
</html></p>

<p>Your scrollbar should look like this on your app.</p>

<p><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/melkla0qd984osetbg3c.webp" alt=" "/></p>
<h2>
<a name="conclusion" href="#conclusion" class="anchor">
</a>
Conclusion
</h2>

<p>We have concluded <em><strong>scrape and build Linkedin user profile app with Bright data and React.js.</strong></em></p>

<p>Using <a href="https://get.brightdata.com/bd7914?sid=c027">Bright data</a> can help reduce the tedious task of sourcing for data by using the already established data from any platform.</p>

Selenium Proxy for Browser Automation

Alex Anie — Fri, 13 Mar 2026 21:26:02 +0000

Proxies are dedicated servers that can be used to bypass restricted content or block IP addresses between a user and a website. When using a Selenium Proxy with Python and Selenium WebDriver, you can route your requests through various IP addresses, helping you avoid bot detection, bypass rate limits, and access geo-restricted content. Using Selenium Proxy enables a scalable, reliable scraping system.

What Is a Selenium Proxy?
Key Features of Selenium Proxy
Setting Up Selenium Proxy
Running Selenium Proxy
Running Selenium Proxy on Cloud
Using Selenium Proxy with Authentication
Implement a Rotating Selenium Proxy
Handling Selenium Proxy Errors
Best Practices of Selenium Proxy
Conclusion

What Is a Selenium Proxy?

A Selenium proxy (or proxy server) acts as an intermediary between your automation script and the target server. When Selenium sends a request, the proxy forwards it to the destination server, receives the response, and passes it back. This setup masks your actual IP address, making the request appear as though it’s coming from a different location.

By rotating IP addresses through a Selenium proxy, you can send multiple simultaneous requests without being easily detected or rate-limited. This is especially helpful for accessing geographically restricted content, bypassing IP bans, and maintaining anonymity while scraping or automating website interactions.

When integrated with Selenium, a well-configured Selenium proxy not only reduces the risk of being blocked but also simulates user activity from various regions and ensures scalable, reliable automation testing.

To fully grasp how a Selenium proxy integrates into browser automation, it’s important to understand what Selenium WebDriver is, since WebDriver is the backbone that manages browser interactions.

Key Features of Selenium Proxy

When working with Selenium for browser automation, proxies open up a wide range of possibilities beyond just sending requests.

Here are some of the most valuable features you can leverage with Selenium Proxy:

Bypass Restrictions: Access geo-blocked or region-specific websites by routing Selenium sessions through different servers. Perfect for testing localized content.
Avoid IP Blocking: Rotate proxies to mimic real traffic and prevent detection during scraping or large-scale automation.
Enhanced Privacy: Mask your real IP and simulate requests from multiple regions for safer testing and experimentation.
Seamless Integration: Easily configure proxies with Selenium WebDriver, locally or on cloud Selenium Grids, for scalable testing.

Setting Up Selenium Proxy

To set up a proxy in Selenium, the first step is to choose the programming language and web driver you want to work with.

Selenium supports multiple languages, including Python, Java, JavaScript (Node.js), C#, Ruby, and Kotlin, as well as different browser drivers such as Chrome, Firefox, and Edge.

Depending on the WebDriver you choose, the configuration and setup vary, and require a different configuration to run.

To get started, you can use Python for scripting and Chrome WebDriver for browser automation to integrate proxy configurations into Selenium.

This combination is widely used due to its simplicity, flexibility, and strong community support, making it ideal for demonstrating how to integrate proxy configurations into your Selenium setup.

Installing Required Packages

Before getting started, ensure that Python is installed on your computer. You can verify this by running the following command in your terminal:

python --version

This should return the version number of Python currently installed. If Python is not installed, you can download and install it from the official Python website.

Once Python is set up, it's time to install the necessary packages for our project. Open your terminal in your preferred text editor if you’re using Visual Studio Code, and press Ctrl + J to open the terminal panel.

Run the following command to install the required dependencies:

pip install selenium webdriver-manager python-dotenv

You can also install colorama to enable colored output in the terminal. This is helpful for debugging and enhanced readability:

pip install colorama

After installation, you can confirm that all packages are properly installed by running:

pip list

You should see output similar to this:

Package                     Version
------------------          -------
colorama                     0.4.6
pip                          25.1.1
python-dotenv                1.1.1
selenium                     4.34.2
webdriver-manager            4.0.2

This confirms that your development environment is correctly set up and ready for Selenium automation with proxy support.

Basic Proxy Configuration

To start configuring with Selenium Proxy, you first need a list of proxies to work with. One reliable source for free public proxies is free-proxy-list.net, which provides proxies from various countries and regions.

Free proxies often have limitations. Many are unreliable because they're heavily used, slow, or incompatible with certain platforms. That’s why it's a good practice to test them first before use.

On the website, click the “Get Raw List” button to copy a list of proxies. These proxies will be in the format IP:PORT.

Note: While free proxies are useful for testing, they are often unreliable. For production workloads, use premium proxy providers such as Bright Data, Oxylabs, or ScraperAPI that offer stable IP pools and compliance guarantees.

Next, create a new Python file named proxy_utils.py. Inside it, define a variable called proxy_list and paste your copied proxies as a list. Then, add the following Python code:

import requests

from colorama import init, Fore, Style
init(autoreset=True)

# List of proxies to check from https://free-proxy-list.net/en/#
proxy_list = [
    "85.215.64.49:80",
    "161.35.70.249:8080",
    "139.59.1.14:80",
    "113.160.132.195:8080",
    "123.30.154.171:7777",
    "8.211.194.78:1081",
    "47.238.134.126:81",
    "35.180.23.174:3128",
    "133.18.234.13:80",
    "219.65.73.81:80",
    "114.6.27.84:8520",
    "4.156.78.45:80",
    "3.101.76.84:18242",
    "205.198.65.77:80",
    "195.158.8.123:3128",
    "5.78.129.53:80",
    "4.245.123.244:80",
    "92.67.186.210:80",
    "23.247.136.248:80",
    "23.247.136.254:80",
    "78.47.127.91:80",
    "45.146.163.31:80",
    "4.195.16.140:80",
    "108.141.130.146:80",
    "124.108.6.20:8085",
    "59.7.246.4:80",
    "95.47.239.65:3128",
    "89.117.145.245:3128",
    "179.60.53.25:999",
    "41.59.90.171:80",
    "185.123.101.160:80",
    "198.49.68.80:80",
    "123.141.181.24:5031",
    "103.75.119.185:80",
    "37.187.74.125:80",
    "41.191.203.161:80"
]

def find_working_proxy(test_url, proxies=proxy_list, timeout=5):
    """Return first working proxy that gives 200 status for test_url."""
    for proxy in proxies:
        proxy_conf = {"http": f"http://{proxy}", "https": f"http://{proxy}"}
        try:
            print(Fore.YELLOW + f"=== Please wait! program is currently routing for the correct `proxy` ===")

            response = requests.get(test_url, proxies=proxy_conf, timeout=timeout)
            if response.status_code == 200:
                from colorama import Fore

print(Fore.GREEN + f"✓ Working proxy found: {proxy}" + Fore.RESET)

                return proxy
            else:
                print(Fore.RED + f"× Bad status from {proxy}: {response.status_code}")
        except Exception as e:
            print(Fore.RED + f"× Proxy failed: {proxy} -- {e}")
    return None

The Python script above tests a list of HTTP proxies to determine which one is usable, i.e., successfully returns a 200 OK status when accessing a given URL.

The find_working_proxy() function accepts three parameters:

test_url: The URL to test the proxy against.
proxies: The list of proxies to loop through.
timeout: The maximum number of seconds to wait for each response.

The function loops through each proxy in the list and sends a request through it. If a request returns a 200 status, the proxy is considered working and is returned. If none of the proxies succeed, the function returns None.

This script is particularly useful when working with public proxies, as it automates the process of selecting a working IP.

You can then integrate the working proxy into your Selenium Python scripts for automation, reducing request errors and improving reliability in your browser automation tasks.

Running Selenium Proxy

In the previous section on configuring Selenium proxy, you built a proxy configuration system using the find_working_proxy() function. This function loops through a list of free proxies and returns the first one that successfully connects (i.e., returns a 200 OK status).

Now, to implement a real-world scenario, you will scrape a list of datasets through a working proxy. This demonstrates how Selenium proxy can help mask your real location while performing automation tasks and shows how to connect a proxy to a Selenium test.

For this example, you will use the TestMu AI Playground site as the target for scraping and testing your Selenium proxy setup.

Code Implementation:

Create a new Python file and name it scrape_blog.py, and type out the code below.

import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

from proxy_utils import find_working_proxy

# Define the target URL (can vary per script)
test_url = "https://ecommerce-playground.lambdatest.io/index.php?route=extension/maza/blog/home"

# Get a healthy proxy
working_proxy = find_working_proxy(test_url)
if not working_proxy:
    print("  No working proxy found.")
    exit()

# Set up Selenium with working proxy
options = Options()
options.add_argument(f'--proxy-server=http://{working_proxy}')
options.add_argument('--headless=new')
options.add_argument('--disable-blink-features=AutomationControlled')

driver = webdriver.Chrome(
    service=Service(ChromeDriverManager().install()),
    options=options
)

# ----- 4. Open Target Page -----
driver.get(test_url)
time.sleep(3)

# ----- 5. Extract First Carousel Slides -----
all_cards = []

try:
    # ----- Wait until the first carousel appears -----
    wait = WebDriverWait(driver, 10)
    first_carousel = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, ".swiper-wrapper")))

    # Now find the child slides inside the first carousel
    cards = first_carousel.find_elements(By.CLASS_NAME, "swiper-slide")

    for card in cards:
        try:
            title = card.find_element(By.CSS_SELECTOR, ".title a").text.strip()
            author = card.find_element(By.CSS_SELECTOR, ".author a").text.strip()
            comments = card.find_element(By.CSS_SELECTOR, ".comment").text.strip()
            views = card.find_element(By.CSS_SELECTOR, ".viewed").text.strip()

            all_cards.append({
                "title": title,
                "author": author,
                "comments": comments,
                "views": views
            })
        except Exception as e:
            print("⚠️ Error parsing card:", e)

except Exception as e:
    print(f  Failed to locate carousel: {e}")

# ----- Output Results -----
print(f"
 Scraped using proxy: {working_proxy}")
for i, card in enumerate(all_cards, 1):
    print(f"
Card {i}:")
    print(f"  Title   : {card['title']}")
    print(f"  Author  : {card['author']}")
    print(f"  Comments: {card['comments']}")
    print(f"  Views   : {card['views']}")

# ----- Cleanup ----------------------------
driver.quit()

Code Walkthrough:

Import Modules: The script starts by importing all necessary modules: Selenium libraries for browser automation, webdriver-manager to manage Selenium ChromeDriver, and proxy_utils to use the find_working_proxy() function created earlier.

from proxy_utils import find_working_proxy

Define the Target URL: A variable test_url is set to the blog home page on TestMu AI’s demo site:

test_url = "https://ecommerce-playground.lambdatest.io/index.php?route=extension/maza/blog/home"

Find a Working Proxy: The script calls find_working_proxy(test_url) to get a live, working proxy. If no working proxy is found, the script exits:

working_proxy = find_working_proxy(test_url)

Configure Selenium with the Proxy: If a working proxy is found, it’s passed to Chrome via the –proxy-server flag using Options(). The script also:
Enables headless mode.
Disables automation detection features.
Scrape Blog Carousel Data: Using the Selenium WebDriver, the script loads the blog homepage. With WebDriverWait, it waits for the blog carousel to load and then extracts data like:
Blog title
Author name
Number of comments
Number of views
These are stored as dictionaries in the all_cards list.
Print the Extracted Data: After collecting the blog post data, the script loops through each blog card and neatly prints the extracted information to the terminal:

print(f"
Card {i}:")
print(f"  Title   : {card['title']}")
print(f"  Author  : {card['author']}")
print(f"  Comments: {card['comments']}")
print(f"  Views   : {card['views']}")

Clean Up: The browser session is closed using driver.quit() to free up system resources. This example demonstrates how you can integrate a working proxy with Selenium to scrape data anonymously.

Test Execution: Now, type the command below to run the script:

python scrape_blog.py

Running Selenium proxy locally can strain your system, limit parallel testing, and make it difficult to manage multiple browsers, OS versions, or unstable proxies. These issues can slow down automation and cause frequent test failures.

Cloud testing platforms provide a scalable and reliable way to run tests. With robust infrastructure, dependable proxies, and pre-configured browser environments, you can execute proxy-enabled tests in parallel across different browsers and operating systems.

This makes it easier to monitor executions in real time, test across multiple geolocations, and avoid the hassle of complex local setups and maintenance. One such platform that offers all these capabilities is TestMu AI.

Running Selenium Proxy on Cloud

To scale proxy-based tests across different browsers and OS, A useful platform for this is TestMu AI, a GenAI-native test execution platform that provides a Selenium Grid online with 3,000+ browser, OS combinations. This setup lets you run proxy-enabled tests in parallel, monitor executions in real-time, and eliminate the need for local infrastructure.

Note: Proxies in Selenium are most commonly used in QA testing scenarios such as geo-testing, performance monitoring, and restricted network simulation. If using proxies for scraping, always comply with site terms and laws.

Here, to implement Selenium proxy on TestMu AI, all you need to do is move your Selenium test to the cloud and configure your code for scalable, cloud-based execution.

To get started with Selenium proxy on TestMu AI, follow the steps given below:

Create a New File: Create a Python file named track_product.py and paste your test code.
Set Up Credentials: Get your Username and Access Key from the TestMu AI dashboard under Account Settings > Password & Security. Save them in a .env file to keep them secure and avoid public exposure.
Update the Selenium Grid URL: Use the credentials in your script when connecting to TestMu AI’s Selenium Grid:
Configure Capabilities: Define key automation testing parameters, such as browser, version, operating system, and additional test settings, to ensure your tests run in the exact environment you need.

Example for Chrome:

You can generate the required Selenium proxy capabilities using the TestMu AI Automation Capabilities Generator.

Code Implementation:

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from proxy_utils import find_working_proxy

import os
from dotenv import load_dotenv

# Load variables from .env
load_dotenv()

username = os.getenv("LT_USERNAME")
access_key = os.getenv("LT_ACCESS_KEY")

# LambdaTest Hub URL
grid_url = f"https://{username}:{access_key}@hub.lambdatest.com/wd/hub"

# Define the target URL
test_url = "https://ecommerce-playground.lambdatest.io/index.php?route=product/category&path=34"

# 1. Get a working proxy
working_proxy = find_working_proxy(test_url)
if not working_proxy:
    print(“ No working proxy found.")
    exit()

# 2. Set desired capabilities for LambdaTest
options = webdriver.ChromeOptions()
options.add_argument(f'--proxy-server=http://{working_proxy}')
options.add_argument('--disable-blink-features=AutomationControlled')
# options.add_argument('--headless=new')

lt_capabilities = {
    "browserName": "Chrome",
    "browserVersion": "latest",
    "platformName": "Windows 11",
    "seCdp": True,
    "LT:Options": {
        "username": username,
        "accessKey": access_key,
        "build": "E-commerce Product Details",
        "project": "Product List",
        "name": "Get the product info",
        "selenium_version": "4.19.0",
        "w3c": True,
        "visual": True,
        "video": True,
    },
}

for key, value in lt_capabilities.items():
    options.set_capability(key, value)

# 3. Create a Remote WebDriver session
try:
    driver = webdriver.Remote(
        command_executor=grid_url,
        options=options
    )

    driver.get(test_url)
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CSS_SELECTOR, "#entry_212408 > div"))
    )

    # Scrape details
    name = driver.find_element(By.CSS_SELECTOR, "#entry_212408 > div > div:nth-child(1) > div > div.caption > h4 > a").text.strip()
    image_url = driver.find_element(By.CSS_SELECTOR, "#mz-product-grid-image-28-212408 > div > div.carousel-item.active > img").get_attribute('src')
    price = driver.find_element(By.CSS_SELECTOR, "#entry_212408 > div > div:nth-child(1) > div > div.caption > div > span").text.strip()

    print(f"
 Scraped using proxy: {working_proxy}
")
    print(f"🔹 Product : {name}")
    print(f"🔹 Price   : {price}")
    print(f"🔹 image Url   : {image_url}")

except Exception as e:
    print(f" Error during scraping: {e}")

finally:
    if 'driver' in locals():
        driver.quit()

Code Walkthrough:

Load Environment Variables: Load TestMu AI credentials from the .env file using load_dotenv().
Get Credentials: Retrieve LT_USERNAME and LT_ACCESS_KEY using os.getenv().
Set Grid URL: Construct the TestMu AI Selenium Grid URL with your credentials.
Define Target URL: Specify the product page to scrape.
Find Working Proxy: Use find_working_proxy(test_url) to select a proxy that successfully connects.
Configure Chrome Options: Set Chrome options, including proxy server, disabling automation detection, and headless mode if needed.
Set TestMu AI Capabilities: Define browser, version, OS, project, build, and additional test settings.
Create Remote WebDriver: Start a Selenium session on TestMu AI with the configured options and capabilities.
Open Target Page: Navigate to the product page using driver.get().
Wait for Elements: Use WebDriverWait to ensure the product elements are loaded before scraping.
Scrape Product Details: Extract product name, price, and image URL using Selenium locators.
Print Results: Display the scraped product information along with the proxy used.
Clean Up: Close the Selenium session using driver.quit() to free cloud resources.
Test Execution: Once your environment is configured, run the test script using:

python track_product.py

TestMu AI Dashboard Output:

Using Selenium Proxy with Authentication

Small-scale websites can often be tested or scraped using free random proxies. However, when dealing with large-scale platforms like Amazon, which frequently update their DOM structure and dynamically render content, free proxies are often insufficient.

For these complex sites, you’ll need a premium proxy service that offers reliable IPs and supports proxy authentication (such as username and password).

Most commercial proxy services provide credentials like:

Proxy host
Proxy port
Username
Password
(Sometimes an API key or session token)

Here, you will learn how to set up an authenticated Selenium proxy. You’ll see how to send a request through a proxy server that requires credentials and verify that the proxy is working by retrieving your IP and location from a target site.

Code Implementation: Create a new Python file as auth_proxy.py, and type the following code below

import zipfile
import os
import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

# -------------------------
# Proxy credentials & info
# -------------------------
proxy_host = "proxy.example.com"   # Replace with actual proxy host
proxy_port = "8080"                # Replace with actual proxy port
proxy_user = "your_username"       # Replace with your proxy username
proxy_pass = "your_password"       # Replace with your proxy password

# Target URL to visit after setting up the proxy
test_url = "https://ipinfo.io/json"

# -------------------------
# Create a Chrome extension to handle proxy auth
# -------------------------

plugin_file = 'proxy_auth_plugin.zip'

manifest_json = f"""
{{
    "version": "1.0.0",
    "manifest_version": 2,
    "name": "Proxy Auth Extension",
    "permissions": [
        "proxy",
        "tabs",
        "unlimitedStorage",
        "storage",
        "<all_urls>",
        "webRequest",
        "webRequestBlocking"
    ],
    "Background": {{
        "scripts": ["background.js"]
    }},
    "minimum_chrome_version":"22.0.0"
}}
"""

background_js = f"""
var config = {{
        mode: "fixed_servers",
        rules: {{
        singleProxy: {{
            scheme: "http",
            host: "{proxy_host}",
            port: parseInt({proxy_port})
        }},
        bypassList: ["localhost"]
        }}
    }};

chrome.proxy.settings.set({{value: config, scope: "regular"}}, function() {{}});

function callbackFn(details) {{
    return {{
        authCredentials: {{
            username: "{proxy_user}",
            password: "{proxy_pass}"
        }}
    }};
}}

chrome.webRequest.onAuthRequired.addListener(
    callbackFn,
    {{urls: ["<all_urls>"]}},
    ['blocking']
);
"""

# Create ZIP file with the extension content
with zipfile.ZipFile(plugin_file, 'w') as zp:
    zp.writestr("manifest.json", manifest_json)
    zp.writestr("background.js", background_js)

# -------------------------
# Set up Chrome options
# -------------------------
options = Options()
options.add_argument('--headless=new')
options.add_extension(plugin_file)

# Optional: Remove "automated" detection
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)

# -------------------------
# Start Chrome WebDriver
# -------------------------
driver = webdriver.Chrome(
    service=Service(ChromeDriverManager().install()),
    options=options
)

# -------------------------
# Visit the target URL
# -------------------------
try:
    print("Opening target page using authenticated proxy...
")
    driver.get(test_url)
    time.sleep(3)

    # Retrieve and display IP address from response
    body = driver.find_element(By.TAG_NAME, "body").text
    print(" Proxy Auth Worked -- Page Output:
")
    print(body)

except Exception as e:
    print("  Failed to load page or extract content:", e)

finally:
    driver.quit()

# -------------------------
# Cleanup plugin file
# -------------------------
if os.path.exists(plugin_file):
    os.remove(plugin_file)

*Code Walkthrough:
*

Implement Proxy Authentication: Since Chrome does not natively support proxy authentication via command-line flags, implement the code using a temporary Chrome extension that injects the proxy credentials.
Embed Credentials: Include the username and password in a background script within the extension.
Launch Chrome with Proxy: Start Chrome in headless mode with the extension and proxy configured.
Test Proxy Connection: Visit a test page like https://ipinfo.io/json to confirm that the proxy is working.
Extract IP and Location: Use Selenium to extract the IP address and location details from the page.
Clean Up: Remove the temporary extension ZIP after the test is complete.

Note: Always store proxy credentials securely in environment variables or secret managers (like AWS Secrets Manager or Vault) instead of hardcoding them into scripts

Text Execution:

Use the following command in your terminal to print the output.

python auth_proxy.py

Output:

Proxy Auth Worked -- Page Output:

{
  "ip": "102.89.40.60",
  "city": "Lagos",
  "region": "Lagos",
  "country": "NG",
  "loc": "6.4541,3.3947",
  "org": "AS29465 MTN NIGERIA Communication limited",
  "timezone": "Africa/Lagos",
  "readme": "https://ipinfo.io/missingauth"
}

When using premium proxy services, Selenium proxy setup instructions may vary slightly depending on the provider. Always refer to the proxy provider’s documentation for details.

Implement a Rotating Selenium Proxy

When scraping large and dynamic platforms like Amazon or Alibaba, it’s common to encounter strict anti-bot mechanisms. These websites can easily detect automated scraping tools like Selenium, especially if multiple requests originate from the same IP address.

If your script repeatedly hits a website with a single proxy IP, it risks being rate-limited, flagged, or completely blocked.

Using a rotating Selenium proxy helps you avoid detection by randomly switching between multiple proxy IPs during the script’s execution. This simulates requests from different users across various locations, making your automation look more natural and harder to detect.

In this example, you’ll use a list of public proxies. Each time your script runs, a random proxy is selected from the list to make a request. You can use https://ipinfo.io/json

Code Implementation:

Create a file named proxy_geo_checker.py and type the following code:

import random
import json
from selenium import webdriver


# some code snippet commented here…

# Initialize colorama
init(autoreset=True)

# Proxy list (public proxies may be unreliable)
proxy_list = [
    "195.158.8.123:3128",
    "85.215.64.49:80",
    "161.35.70.249:8080",
    "139.59.1.14:80",
    "41.59.90.171:80",
    "185.123.101.160:80",
    "198.49.68.80:80",
    "123.141.181.24:5031",
    "103.75.119.185:80",
    "37.187.74.125:80",
    "41.191.203.161:80"
]

# Randomly choose one
proxy = random.choice(proxy_list)

# Set up Selenium with proxy
chrome_options = Options()
chrome_options.add_argument(f'--proxy-server=http://{proxy}')
chrome_options.add_argument('--disable-blink-features=AutomationControlled')
chrome_options.add_argument('--headless=new')

driver = webdriver.Chrome(
    service=Service(ChromeDriverManager().install()),
    options=chrome_options
)

try:
    url = "https://ipinfo.io/json"
    driver.get(url)

    # Wait for the page to load and extract the raw JSON text
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.TAG_NAME, "pre"))
    )
    json_text = driver.find_element(By.TAG_NAME, "pre").text
    data = json.loads(json_text)

    # Extract details
    print(Fore.GREEN + f"
✅ Proxy Used     : {proxy}")
    print(Fore.CYAN + f"🌐 IP Address     : {data.get('ip', 'N/A')}")
    print(Fore.CYAN + f"🏙️  City           : {data.get('city', 'N/A')}")
    print(Fore.CYAN + f"📍 Region         : {data.get('region', 'N/A')}")
    print(Fore.CYAN + f"🌍 Country        : {data.get('country', 'N/A')}")
    print(Fore.CYAN + f"⏰ Timezone       : {data.get('timezone', 'N/A')}")
    print(Fore.CYAN + f"🔌 ISP (Org)      : {data.get('org', 'N/A')}")
    print(Fore.CYAN + f"📡 Location (Lat,Lon): {data.get('loc', 'N/A')}")

except Exception as e:
    print(Fore.RED + f"
❌ Error: {e}")
finally:
    driver.quit()

Code Walkthrough:

Import Modules: Load Selenium, Chrome WebDriver, Colorama, and random for proxy selection.
Select Proxy: Randomly pick one IP from the proxy_list using random.choice().
Configure Browser: Launch Chrome with the selected proxy in headless mode and disable automation detection.
Open Target URL: Navigate to https://ipinfo.io/json to check the proxy.
Extract Data: Read the JSON response to get IP address, city, country, ISP, and geolocation.
Clean Up: Close the browser session with driver.quit(). Test Execution:

Type the command below in the terminal to run the script.

python proxy_geo_checker.py

Output:

ANIE@Anie MINGW64 ~/Desktop/Anie/webdev/blogs/LambdaTest/selenium-proxy/code (main)
$ python proxy_geo_checker.py

DevTools listening on ws://127.0.0.1:54240/devtools/browser/a29e15f0-ea69-4063-9b0e-c28320b20ea5

✅ Proxy Used     : 161.35.70.249:8080
🌐 IP Address     : 161.35.70.249
🏙️  City          : Frankfurt am Main
📍 Region         : Hesse
🌍 Country        : DE
⏰ Timezone       : Europe/Berlin
🔌 ISP (Org)      : AS14061 DigitalOcean, LLC
📡 Location (Lat,Lon): 50.1155,8.6842

The output shows where the Selenium proxy appears to be located, helping verify that the rotating proxy is working as intended.

Handling Selenium Proxy Errors

Working with a Selenium proxy comes with its own set of challenges, such as network instability, poor proxy quality, and configuration mismatches. These issues can quickly lead to failed tests, blocked IPs, or inconsistent results.

To avoid common pitfalls and maximize performance, it’s essential to understand how to properly handle proxy-related issues and implement optimization strategies.

Here are some common Selenium proxy errors you might face and effective ways to solve them:

Problem: When faced with net::ERR_PROXY_CONNECTION_FAILED, it’s due to an invalid proxy or a proxy server being down.
Solution: Validate proxies before starting Selenium; skip or retry invalid ones.
Problem: When faced with net::ERR_TUNNEL_CONNECTION_FAILED, it’s because the proxy cannot tunnel HTTPS traffic. -** Solution**: Ensure the proxy supports HTTPS.
Problem: SSL Handshake Failed occurs due to a broken SSL certificate or an incompatible protocol.
Solution: Switch to a different HTTPS proxy or update Selenium/WebDriver versions.
Problem: Timeouts and long page loads happen when the proxy server is slow or overloaded.
Solution: Use high-quality or paid proxies; implement timeouts and fallback proxies.
Problem: Blocked or CAPTCHA pages occur when the proxy IP is blacklisted or flagged as suspicious.
Solution: Rotate proxies frequently; combine with user-agent spoofing or CAPTCHA solvers.

Best Practices of Selenium Proxy

For effective Selenium testing, a Selenium proxy helps route traffic securely and manage requests without triggering site blocks. By validating proxies, preferring HTTPS support, and rotating them, you can maintain consistent and reliable test execution.

Use HTTPS Proxies: Prefer HTTPS proxies for better security, compatibility, and ease of use.
Test Proxy Before Use: Always validate that the proxy works with your target site to avoid runtime failures. Implement a health-check script to ensure proxies return valid HTTP status codes (e.g., 200 OK).
Rotate Proxies: Use rotating proxies for large-scale scraping or automation to prevent IP bans and rate-limiting. Rotate through a list of proxies periodically or per request batch.
Check Latency and Speed: Select proxies with low latency to ensure your Selenium automation runs efficiently. Avoid Free Proxies for Critical Tasks: Free proxies can be unreliable, slow, or blocked; consider premium proxies for important tasks.
Use Proxy Authentication When Needed: For secure or paid proxies, integrate the username and password correctly in your Selenium proxy setup.
Limit Requests per Proxy: Avoid sending too many requests from a single proxy to reduce the risk of detection.
Integrate with Headless Browsers: Ensure your Selenium proxy setup works with headless mode if your automation requires it. Headless browsers consume fewer resources but may trigger anti-bot measures; combine with proxy and browser fingerprinting tweaks.
Use Exponential Backoff for Retries: If a proxy fails, retry the operation with gradually increasing delays to reduce server stress.
Log Proxy Usage: Keep logs of working/failing proxies, response times, and proxy activity to troubleshoot issues and monitor performance.
Consider Geolocation Needs: Choose proxies from regions relevant to your testing or scraping requirements.

Conclusion

A Selenium proxy is an essential tool for large-scale scraping and testing, helping you bypass restrictions, avoid rate limits, and simulate real-user behavior across different locations. From simple HTTP setups to rotating and authenticated proxies, each method enhances the reliability of your automation.

By combining Selenium with effective proxy strategies and best practices, you can build more resilient, efficient, and scalable automation scripts. Whether you’re testing locally, handling authentication, or deploying to the cloud with platforms like TestMu AI, mastering Selenium proxy integration gives you a strong edge in modern web automation.

Why We Should Stop Saying “MVC” and Start Saying “RCMV” Instead

Alex Anie — Sat, 20 Dec 2025 23:01:33 +0000

MVC (Model, View, Controller) is one of the most popular architectural terms in web development.

But here’s the uncomfortable truth:

MVC doesn’t fully describe how modern web applications actually work.

It explains separation of concerns, but it doesn’t explain execution flow. And that distinction matters.
The Missing Piece in MVC: Routing
In every modern web application, nothing happens without routing.
Before a controller runs.
Before a model is touched.
Before a view is rendered.

A request must first be matched to a route.
Yet when we say MVC, routing is completely invisible.

This creates confusion, especially for beginners, because the real request flow looks more like this:

Request → Router → Controller → Model → View

Not MVC.
Introducing RCMV
RCMV stands for:
Router

Controller

Model

View

In that order.

RCMV doesn’t replace MVC; it extends it to reflect how modern applications actually respond to requests.
What Each Part Does (In Reality)

Router

The true entry point of the application. It decides what code should respond to a request and whether that request is even allowed.

Without routing:

No pages load

No APIs respond

No CRUD operations happen

Controller

Activated only after the router allows it. Controllers coordinate the request, validate intent, and delegate responsibility.

They don’t own data, they orchestrate it.

Model

Responsible for structuring and managing data:
Relationships

Rules

Persistence

Business constraints

Models don’t respond to users; controllers do.

View

The final destination.
This is where structured data becomes something users can see and interact with.
The quality of the view depends entirely on how well the other layers were designed.
Why RCMV Is Useful
RCMV aligns better with:
The actual request lifecycle

How frameworks like Laravel, Django, Rails, and Express work

How developers debug real-world issues

How teams avoid spaghetti code

It also fits naturally with DDD-style thinking and cleaner system design.

Important Clarification
RCMV is not an attack on MVC.
It doesn’t change how applications are built.
It doesn’t “kill” MVC.
It simply provides a more accurate label for how modern web applications function.

Final Thought
MVC explains responsibilities.
RCMV explains reality.
Both matter, but we should be honest about which one we’re teaching.

What do you think?
Is MVC enough, or does modern development need a better mental model?

Scrape Unscrapeable Amazon Dataset with BrightData, React.js and Node.js

Alex Anie — Sat, 28 Dec 2024 13:47:55 +0000

This is a submission for the Bright Data Web Scraping Challenge: Scrape Data from Complex, Interactive Websites

What I Built

This project uses Brightdata to scrape data from Amazon and return the data output on the page. You can search anything you want and expect to see it load on the page as long what you search can be found on amazon.

Demo
The project uses two different GitHub repo. One for the frontend the other for the backend.

You can find the live 🔥 demo here.
Github Repo Frontend
GitHub Repo Backend

How I Used Bright Data

Project is built using bright data.

I used Brightdata Scraping browser to retrieve the data set from amazon.

import 'dotenv/config'
import { Router } from 'express';
import puppeteer from 'puppeteer-core';
import process from 'node:process';

const router = Router();

// Scraping logic using Puppeteer and BrightData
const scrapeData = async (searchTerm) => {
  const BROWSER_WS = process.env.BROWSER_WS; // set your bright data proxy credential here
  const URL = "https://www.amazon.com";

  const browser = await puppeteer.connect({
    browserWSEndpoint: BROWSER_WS,
  });

 // ... some code here

  await browser.close();
  return products;
};

// Define the API route for scraping
router.get('/scrape', async (req, res) => {
     // ... some code here
  }
});

export default router;

The Brightdata scraping browser uses puppeteer-core to scrape amazon data and return the contents as a json respones.

I used express.js to create an api endpoint and server for the frontend appication which is a React and vite.js setup.

import express from 'express';
import scrapeRouter from './index.js'; // Import the logic from index.js
import cors from 'cors';

const app = express();

// allow all origin
app.use(cors());

// Use the scrapeRouter for /api routes
app.use('/api', scrapeRouter);

// Set the port
const PORT = 4040;

// Start the server
app.listen(PORT, () => {
  console.log(`Server running on port ${PORT}`);
});

Tailwindcss is used for the staling and React Icons for the Icons. Other Stacks are listed below.

Deployment

The backend express app is deployed seperately

Backend deployed on Render.com
Frontend deployed on Netlify.com

Stacks Used

React
Vite
Tailwindcss
React Icons
Axios
Cors
Brightdata (for proxy and data fetching)
Render (for api hosting)
Dotenv (load env)
express (to setup server and routes)
nodemon (local dev)
puppeteer-core (Scraping data from Amazon)

Revolutionizing Website Testing: A Deep Dive into LambdaTest's KaneAI

Alex Anie — Mon, 16 Sep 2024 14:30:44 +0000

In the ever-evolving landscape of web development, ensuring website quality and performance across multiple platforms and browsers has become increasingly complex. KaneAI an AI-powered next-gen testing assistant by LambdaTest brings a groundbreaking solution that revolutionizes how we approach website testing.

As the digital world expands, so does the need for efficient, accurate, and scalable testing methodologies. Traditional manual testing methods, while valuable, often struggle to keep pace with the rapid development cycles and the sheer variety of devices and browsers in use today. This is where KaneAI steps in, offering a potential paradigm shift in validating and optimizing testing web applications.

LambdaTest, a well-known player in the cross-browser testing arena, has taken a significant leap forward with the introduction of KaneAI. This innovative AI-powered tool aims to streamline the testing process, enhance accuracy, and provide deeper insights into website performance and user experience.

In this blog post, we'll deeply dive into KaneAI, exploring its features, benefits, and potential impact on the website testing landscape. We'll examine how this tool fits into the broader context of AI in software testing and what it means for developers, QA professionals, and businesses striving to deliver flawless web experiences.

By the end of this blog, you learn what KaneAI is, its key features, useful KaneAI commands, and how to run AI-powered text with prompts.

What is Kane AI

KaneAI is an AI-powered next-gen testing assistant that enables testers to run automated AI tests on web, mobile, and cloud-based applications using prompts.

If you ever used chatGTP, KaneAI allows you to automate your testing workflows and deliver faster, more accurate results with prompts. This saves time spent on manual testing and provides better ways to catch critical edge cases you might otherwise miss.

Key Features of KaneAI

Test Generation & evolution: This lets you effortlessly use natural language inputs, simplifying the testing process with intelligent automation.
Multi-Language Code Export: Testers can convert automated tests in all major languages and frameworks
Intelligent Test Planner: Testers can automatically generate and automate test steps using high-level objectives.
Smart show-me mode: Testers can convert your actions into natural language instructions to generate bullet-proof tests
Sophisticated Testing capabilities: Testers can express sophisticated conditions and assertions in natural language.
Generate your tests: generate your tests with just high-level objectives.
Resilient tests from inception: Utilize built-in smart capabilities and auto-heal.
Web, mobile, or both: Develop tests across your stack on both web and mobile devices for extensive test coverage.
Plan and organize your test cases: Use the Test Management platform to aid in planning your test.
Tag KaneAI: In your conversations start test automation from various sources. Use KaneAI tags that are available for JIRA, Slack, and GitHub issues!
2-way test editing: lets you focus on one and we'll sync the other—instruction to code and Code to instruction translation for effortless maintenance of tests.
Smart versioning support: KaneAI maintains separate versions for every change you make.
Run scheduled tests on HyperExecute: This lets you accelerate test execution up to 70% more than the traditional cloud.
Run generated tests: across 3000+ combinations of browsers, OS, and real devices.
AI-Powered Debugging: Experience assisted Debugging with KaneAI’s Test Agent to help you troubleshoot with the breeze.
360 Test Observability: Detailed test execution reports with Test Intelligence and deep analytics.
Seamless integrate KaneAI into your workflow: Tag KaneAI wherever you go.

Useful Command Types for KaneAI

In other to perform tests with KaneAI, commands called “prompts” are required to be typed into the KaneAI text field. This will enable KaneAI to know what kinds of tests to perform and how to catch bugs if the test fails.

Open / Go to: This command lets you launch or open a website by specifying its URL.

👉 go to https://www.lambdatest.com

👉 open https://www.lambdatest.com

Wait: Use to perform a wait operation.

👉 wait for 5 seconds

Click: Use to tell KaneAI to click on elements.

👉 click on login button

Type or Enter Text: Used to tell KaneAI to enter a text into an input field.

👉 enter "random@gmail.com"

👉 type "random@gmail.com"

👉 send keys "random@gmail.com"

Clear: Used to tell KaneAI to clear a text in an input field.

👉 clear email input field

👉 clear the password field

Select: Use to tell KaneAI to select date from a select tag.

👉 select 21st september date

Hover: Used to trigger a hover state

👉 hover on the shop by category option

Search: used to perform a search query

👉 search for iPhone

Refresh: used to refresh the page

👉 refresh the page

Navigate Back/Forward: To move back or forward to the previous page from the browser history.

👉 go back

👉 navigate back

👉 go forward

👉 navigate forward

Open New Tab: Open a new browser tap

👉 open new lambdatest tab

Switch Tabs: Navigate between opened tabs on the browser.

👉 switch to 2nd tab

👉 switch to lambdatest tab

Close Tab: Close the opened tab.

👉 close lambdatest tab

👉 close 2nd tab

Scroll: Scroll within the current page.

👉 scroll once

👉 scroll by 100px

👉 scroll to the bottom

👉 scroll to the top

👉 scroll 3 times

👉 scroll by 20 percent

Assert Element Text: The assert keyword lets you test if a condition in your code returns True, if so it returns “Assertion True” . If not, the program will raise an AssertionError called “Assertion Failed”.

👉 assert if red button text is "subscribe"

Query Information: Used to perform a query operation.

👉 query the current url

Conditional Actions: Used to tell KaneAI to perform a specified task if the condition statement is true.

👉 if pop up is present in the viewport then click on it.

Using the KaneAI prompt can improve your test experience overall. You can learn more from the documents here.

Run your first KaneAI web Test.

KaneAI lets you perform automated tests using prompts as explained in the previous section. In this section, we will learn how to conduct our first test using real-world examples. These examples involve running a test on Tesla’s official website.

You will see how KaneAI can be used to run tests on the cloud on any website. To get started login to KaneAI, you should see a page display like the one below.

Before we start writing prompts, it’s important to note that KaneAI processes prompts sequentially, as a result, a particular prompt has to be completed before the next prompt is evaluated. However, how we can choose to write prompts is up to us, we can choose to write them sequentially or multiply the prompt and let KaneAI then decide how to process it.

Sequential Prompt

All prompts in this section are going to be processed sequentially. That is each command will be a single line of code and then we’ll press enter for KaneAI to process it.

At the time of writing, KaneAI is in beta, so you might be required to be on a waitlist. But if you’ve already signed in, let’s keep going.

First, click on the Create a Web Test button. This will load the Chrome Virtual Browser on the cloud.

From the textarea panel on the right bottom section, type your first prompt.

Go to https://www.tesla.com/

Next press enter or press the return icon on the textarea box. This will open the Tesla home page on your Chrome Virtual Browser as indicated below.

From the above illustration, we have three important sections marked.

Number 1: This is KaneAI input box, where all commands/prompts are typed to let KaneAI know what test to perform.

Number 2: This box displays the current operation/tests KaneAI has processed and displays the result as soon as the test is completed.

Number 3: This is the Chrome Virtual Browser, where test activities performed on the browser are displayed.

However, from the Chrome Virtual Browser, we have the Tesla home page displayed. But from the view above, it seems the Navbar is covering the home page. To fix this issue type the prompt below to refresh the Tesla home page.

refresh page

Your test result should look like this below.

Notice from the mark section above, that we now have a new test result showing the new prompt we just ran.

Next, type scroll 3 times on the KaneAI input box, this will navigate to the Model S section on the Tesla home page as indicated below.

Next, we want to click on the “Order now” button and then click on the finance tab in the Model S categories, to do this type the first prompt below and press enter, then follow by the second one.

1. click order now button
2. click finance tab in the Model S categories

The first prompt should open the page. as indicated below.

The second prompt will switch the tab from “cash” to “finance”. as indicated below

If you feel like navigating back to the previous page in the history stack, you can use the back command to do so.

1. back to previous page
2. refresh page

You should have the Model S page displayed again.

If you are performing small test cases, typing out the prompts one after the other is not a bad idea. However, you can save valuable time by typing out the prompts in steps.

Multiply Prompts

KaneAI allows multiple prompts to be specified at once. By typing out multiple prompts, KaneAI adds them to a queue and processes them one after the other until everything is completely processed.

This is very useful as it speeds up testing and it allows KaneAI to know what to work on next after a specific prompt is completed.

Let’s take a list of prompts below as a use case. Go ahead and type it on the KaneAI input box and click enter.

1. Scroll to bottom of the page

2. Click shop Now

3. Click on Wall Connector card

4. Click on add to card button

Firstly, KaneAI will add them to the queue, go through each prompt one after another, and then load the page corresponding to each prompt.

From the image above, KaneAI has processed all the prompts and loaded the page accordingly. But one out of all the tests did not respond as expected, as you would have noticed. KaneAI has provided a text description to explain why this is the case.

From the above instruction, KaneAI has reported why the previous test was not run as expected. So let’s run another prompt below.

click on the add to cart buttons 4 times

You should see a model asking you to “view cart” or “checkout” as indicated below.

Next, we are going to run another prompt. Type the prompt below and press enter.

click on view cart button

Your test result should look like this. Here we have the list of items added to the cart.

We have seen how KaneAI can be used to run tests with immediate feedback from the browser by using prompts. However, you can also interact with the UI instead of running prompts. This is called Manual Interaction.

Manual Interaction

Manuel Interaction lets us run a test by interacting with the UI of the Chrome Virtual Browser, and then KaneAI records and generates the interaction as a test.

To activate manual interaction, click on the “Manuel Interaction” button on the top right corner of the Chrome Virtual browser.

Once the Manual Interaction button is activated, we can click on the element we want to interact with and then watch as KaneAI returns it as a test. For example, click on the “checkout” button and watch KaneAI return the interaction.

The red box on the image shows the interaction we made. Notice the written text. “Click on the Checkout button”.

Manuel Interaction can be very useful in many cases, especially in cases where you are unsure what kind of prompt is required to perform or interact with the application UI.

Finish Test

KaneAI provides a “finish Test” button, used to save tests when you finish your testing in your current session.

Click on the “Finish Test” button, this will open the session page with information on the test cases performed.

On the session page, click on the select folder and choose “new test”.

Next, click on the type select box and change the “type” to “functional”.
Next, click on the “save test case”.

This will open up the “Test Manager” panel. Where all information and code from the test are stored.

Test Manager

The Test Manager session includes all test cases performed, it helps in managing our tests. The Test Manager includes the following.

number 1: The name of the project
number 2: Summary of the project, such as;
1. name of the tester.
2. description of the test
3. steps performed during the test.
number 3: Generate python/selenium code from the performed test. you can click on the view code to see the code.
number 4: Hyperexecute code.
number 5: Issues from the performed test cases.
number 6: Display the history regarding when tests are performed.
number 7: The name of the tester.
number 8: Steps taken when performing tests
number 9: Use to edit steps taken or perform additional tests.
number 10: Description of the test taken. ****

KaneAI provides a streamlined process for running automated tests using AI-powered prompts. With Task Manager, you can effectively manage performed tests from every test session you make.

Summary

KaneAI is an AI-powered testing assistant by LambdaTest that revolutionizes website testing through automated, prompt-based testing across web, mobile, and cloud-based applications

Key features include test generation, multi-language code export, intelligent test planning, and sophisticated testing capabilities using natural language inputs.

KaneAI uses prompts for various testing commands, including opening websites, clicking elements, entering text, and performing assertions.

Tests can be run sequentially or with multiple prompts, allowing for efficient and flexible testing processes.

KaneAI aims to streamline the testing process, enhance accuracy, and provide deeper insights into website performance and user experience

Create a Sales Dashboard with Neon, Prisma, Kinde-Auth, Next.js, and Tailwindcss.

Alex Anie — Sat, 31 Aug 2024 15:18:02 +0000

This is a submission for the Neon Open Source Starter Kit Challenge : Ultimate Starter Kit

My Kit

Next.js (React Framework)
Tailwindcss (For Styling)
Shadcn (Reuseable Components)
Kinde (Authentication)
Prisma (Database validating and modeling)
Neon (Database)
Lucide React ( Icons Libery)

Link to Kit

You can find the link to the project here

Your Journey

Here's a short introduction for the Neon Open Source Starter Kit Challenge.

I created this project titled:
"Create a Sales Dashboard with Neon, Prisma, Kinde Auth, Next.js, and TailwindCSS",

focusing on Next.js, Kinde-Auth for authentication, Prisma and Neon

The introduction sets the stage by highlighting Neon's key features and benefits in the context of building a sales dashboard. It emphasizes how Neon's capabilities align perfectly with the needs of a modern, scalable web application.

The introduction also touches on the other technologies such as (Prisma, Kinde Auth, Next.js, and TailwindCSS), positioning Neon as the cornerstone of your data management strategy.

This approach should engage readers by demonstrating the practical benefits they'll gain from, and also showcasing why Neon is an excellent choice for this type of project.

ChatGPT AI chat model with Llama2, Workers AI Model, Hono.js and Wranger.

Alex Anie — Thu, 11 Apr 2024 22:04:24 +0000

This is a submission for the Cloudflare AI Challenge.

ChatGPT AI chat model with Llama2, Workers AI Model, Hono.js and Wranger.

What I Built

A ChatGPT app using llama-2-7b-chat-int8 model from Cloudflare. Implemented with Worker AI, Hono.js

Demo

Features:

History panel
Content Streaming
New Chat functionality

👉 Link to project

Link to the project

GitHub Code

Link to GitHub repo

Implementation

This app is built to function exactly like ChatGPT. You can ask questions and the app will generate a stream response. It has a history panel to track previous prompts, which is useful for tracking what you entered.

I used the following tools to accomplish this task.

**Llama A text generator language model
Hono.js: A web framework for Cloudflare Workers
Worker AI: To run machine learning models, on the Cloudflare network,

Web Scraping with Cheerio, Axios, and Express

Alex Anie — Fri, 10 Nov 2023 16:41:04 +0000

Web Scraping is the process of extracting data from the DOM of a website and then returning it data for external use in another application.

Web Scraping is a very complicated yet simple technique and can be useful in different cases and can reduce the amount of effort needed by a developer to create a valid JSON or API for their application.

For example, you might want to build an app that displays product items and their current price list directly on the app, and sourcing for this data or creating it manually might not be the ideal thing to do due to the time it will take to create the data. Then in this case you might need to look for an online store with the exact data and price list you are looking for and then scrap the data directly on your app.

Another scenario where web scraping can also be very useful might be an app that requires users to be able to search items or places based on a certain location. With web scraping, you can scrap popular map platforms like Google Maps to track changes in different locations and return the result to the users.

In this blog post, we are going to be looking into how to implement web scraping using Node.js. With Node.js we can install essential packages that can be used to scrap data on the web such as Cheerio, Express, Axios and Nodemon.

By the end of this blog post, you should have a basic understanding of how web scraping works with Node.js

Prerequisites

It would help if you had a basic understanding of the following;

JavaScript DOM Manipulation
Selecting element from the DOM using JavaScript
CSS Selectors
Axios

How to install packages in Node.js

Now let’s begin,

Overview

In this blog post, we will be scraping a blog post with 50 book reviews. We will be scraping the following data;

id: random numbers from the blog as our id
title: title of the book
author: the name of the book author
about: a short description of the book
imgUrl: the URL where the cover of the book is saved.

After that, the data will be converted to a valid JSON file and exported.

check the finished preview here 👉 books.json and the full project here 👉 on GitHub.

Dependencies

The following is a list of dependencies we will be working with in this project.

Cheerio: Cheerio is an npm package that provides APIs that make it easy to traverse/manipulate DOM elements. We are going to be using it to grab elements in the DOM.
Axios: Axios is an npm package, a promise-based HTTP Client for Node.js and the browser. We are going to be using it to get the website URL and return the content we want to scrap.
Express: Express is a fast, unopinionated, minimalist web framework for Node.js.
Nodemon: Nodemon is a simple monitor script for use during development of a Node.js App. We are going to use this package to create a local server to track the changes on the index.js file.

Project Setup

To set the project for development, create a folder scrape-app and open it on VS Code like so 👇

Click on Ctrl + ` (backtick) to open up the VS Code terminal. 👇

Next, type the command below to generate a package.json file.

npm init -y

The -y flag which stands for Yes, will accept all the default options and generate a package.json file as shown below 👇.

Next, install all four packages as indicated below 👇

npm install cheerio express axios nodemon

Next, npm will install all four packages (node_modules folder), package-lock.json and update the package.json file. 👇

Now, that we’ve installed the necessary dependencies, let’s create the index.js file as the entry point of our app.

Index.js file

To create the index.js file, type the command below 👇

touch index.js

You should see the new create index.js file as indicated below 👇

Express server setup

Open the index.js file and require the express module as indicated below. 👇

const express = require("express");

This is a commonJS syntax.

Next, create two variables called app and PORT, set the app to listen for the PORT server as specified below.

const PORT = 8000;
const app = express();

app.listen(PORT, () => console.log(`server running on PORT ${PORT}`));

The app will listen for changes in the file and log the result on the terminal.

Scraping data from a web page

Create a variable url and assign the absolute URL of the blog you want to scrape to it. 👇

const axios  = require("axios");
const express = require("express");
const cheerio = require("cheerio");

const PORT = 8000;
const app = express();

👉 const url = "https://www.panmacmillan.com/blogs/general/must-read-books-of-all-time";

Specify the url as an argument on the axios() method like so 👇

axios(url)

Now, let's get the promise response (content) of the url parse in.

axios(url)
    .then(response => {
        const html = response.data;
        const $ = cheerio.load(html);
        const books = [];
    })

The code example above is as follows;

html : we are getting the entire document of the page (The HTML document) and storing it into a constant variable of html
$:here, the return html document which is parsed to cheerio.load(html), this will help load the HTML document to Cheerio and give us access to the Cheerio APIs for grabbing the HTML interface.
book: this is an empty array where all the data entries will be stored as an object.

Next, let's inspect the DOM of the website we want to scrape.

Right-click on the screen and select inspect from the popup box or click F12 to open DevTools.

From the DevTools panel, click on the inspect icon and select the image. This will load the HTML document on the elements panel below the DevTools. Then select the figure tag with an data-component attribute.

The figure tag with a data-component attribute serves as the parent element for the main content we want to scrape, such as the id, title, author, about, imgURL.

Clicking on the figure tag with a data-component attribute will give you a highlight preview of the children elements as shown in the screenshot above. The select preview is the selected area on the webpage we want to scrape.

axios(url)
    .then(response => {
        const html = response.data;
        const $ = cheerio.load(html);
        const books = [];

   👉  $("#primary-content figure[data-component='book-spotlight']", html)
    })

From the code above, we targeted the #primary-content id which is the parent tag of the figure element and then we grabbed figure[data-component='book-spotlight'] attribute to be able to select the children's elements of the figure element where the data we want to return are.

axios(url)
    .then(response => {
        const html = response.data
        const $ = cheerio.load(html)
        const books = [];
        $(
          "#primary-content figure[data-component='book-spotlight']",
          html
        )
    👉             .each(function () {
  👉       const id = $(this).find("div a").attr("data-isbn");
  👉       const title = $(this).find("figcaption h3 a").text(); 
  👉       const author = $(this).find("figcaption h4 a span").text(); 
  👉       const about = $(this).find("div.rte--default div p").text(); 
  👉       const imgUrl = $(this).find("div a img").attr("src");

Next, the each() method is a callback function.

The find() function is used to grab the element we want to return
The attr() function is used to grab the attribute of the return element
The text() function is used to return the text content within the element

The declared variable id, title, author, about, imgUrl stores the values of the return object.

const axios = require("axios");
const cheerio = require("cheerio");
const express = require("express");

const PORT = 8000;

const app = express();

const url =
  "https://www.panmacmillan.com/blogs/general/must-read-books-of-all-time";

axios(url)
  .then((response) => {
    const html = response.data;
    const $ = cheerio.load(html);
    const books = [];
    $("#primary-content figure[data-component='book-spotlight']", html).each(
      function () {
        const id = $(this).find("div a").attr("data-isbn");
        const title = $(this).find("figcaption h3 a").text();
        const author = $(this).find("figcaption h4 a span").text();
        const about = $(this).find("div.rte--default div p").text();
        const imgUrl = $(this).find("div a img").attr("src");

    👉   books.push({
          id,
          title,
          author,
          about,
          imgUrl,
        });
      }
    );

 👉  console.log(books);


  })

app.listen(PORT, () => console.log(`server running on PORT ${PORT}`));

Next, we grab the book array and push all the return object values. This will return an array of objects based on the length (number of elements) of the parent element.

Then log the value to the terminal with console.log(books)

npm run start

Go to the Terminal and type the above command to start the dev server.

If for any reason the dev server does not start, check the package.json file and make sure the script is set as follows

{
  "name": "scrape-app",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
 👉 "start": "nodemon index.js"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "axios": "^1.6.0",
    "cheerio": "^1.0.0-rc.12",
    "express": "^4.18.2",
    "nodemon": "^3.0.1"
  }
}

Save the package.json file and type the command npm run start on the terminal to start the dev server. Once the server is started the value will be logged into the terminal.

Now we have successfully scraped the data we want. The data we scraped is as follows

id: these are random numbers returned from a specific a tag in the blog
title: The title of each book in the blog
author: the names of each author in the blog
about: description of each book in the blog
imgUrl: The absolute URL of where the image is stored in the blog.

Furthermore, the JavaScript object can’t not be used from the console, we have to export it as a valid JSON file on the project folder.

Exporting scraped data

To export the data as a valid JSON file, require the fs module as follows

const fs = require("fs");

Next, specify how you want to export the file as follows

fs.writeFile("books.json", JSON.stringify(books), (err) => {
      if (err) throw err;
      console.log("book.js is now saved!");
    })

The fs.writeFlie(”books.json”) specifies the name of the file
The JSON.strigify(book)specifies the JavaScript object that needs to be exported
The err checks and returns the error if any.

Save the index.js file and check the folder panel, the book.json should be saved. 🥳

Conclusion

Web Scraping is a very complicated yet simple technique. It can be useful in different cases and can reduce the amount of effort needed by a developer to create a valid JSON or API for their application.

Scraping with puppeteer and export in JSON format

Alex Anie — Sat, 04 Nov 2023 08:27:53 +0000

App development is no walk in the park, figuring out the right themes to use, frameworks or libraries when building an app can be a tedious process, also sourcing for the right APIs to build on can be quite challenging especially when the one you need is not available.

Well in this case scraping the data can be all you need to provide the right data for your app.

In this blog post, we are going to be learning how to use Puppeteer to scrape data from the web and then export it in JSON format.

This will enable you to know how to scrape data with Puppeteer and implement it on any project of your choice.

Prerequires

A basic knowledge of Node.js and JavaScript is required to be able to follow along.

Setup

To get started, create a folder and name it as puppeteer, then open it up as a workspace on VS Code as indicated below 👇

Click on Ctrl + ` backtick to open the terminal on VS Code. 👇

You should already have node.js installed on your system, but if you are unsure if you have it installed. run the command below.

node -v

This will return the current version you have installed on your system

👉 // v21.1.0

Am currently running on version v21.1.0 but depending on when you are reading this or what version you have installed, it will be different from mine.

Next step, generate a package.json file. Type the command below and hit enter.

npm init -y

This will create and set up a package.json file where all dependencies and devDependencies will be saved.

Now that we have that figured out, let's install Puppeteer.

npm i puppeteer

The above command will install Puppeteer on your system if your default package manager is npm but if you are making use of yarn or pnpm you can try the following command below.

# using yarn
yarn add puppeteer

# using pnpm
pnpm i puppeteer*

Directly on your terminal, the installed node_modules should indicate as follows. 👇

By default, Puppeteer will install node_modules and package-lock.json and update the package.json file with the latest version of Puppeteer. As at the time of writing, I have 21.5.0 installed, yours should be similar or higher. Use npm i puppeteer@12.5.0 to install the exact version used in this tutorial.

Entry Point Setup

Create index.js file. This will serve as an entry point for our application.

const puppeteer = require('puppeteer');

click to open the index.js file, type the code above. The syntax above is commonJS, this is what we’ll be using in this tutorial.


(async () => {

 👉 // your code should come write here

})();

The code above is immediately-invoked function expression (IIFE), this allows the code specified inside the function to be executed immediately and it is associated with async/await syntax for handling asynchronous operations.

Now that we have set the function boilerplate, let's create a headless browser instance with Puppeteer.

(async () => {

 const browser = await puppeteer.launch({ headless: "new" });

 const page = await browser.newPage();

 await browser.close();

})();

Here, we are creating an instance of a browser and puppeteer is set to launch. This opens up the browser.

The key and value { headless: "new" } object syntax indicates that the browser should operate in a headless mode, that is to run in the background without displaying its UI.

Next the browser.newPage() opens up a new tag and assigns it to a page variable.

Then the await browser.close() method closes the browser when the function stops running.

Scraping web page data

To begin scraping data, you have to decide which platform data you want to scrap. While it is easy to scrap data, some platforms prevent access to scraping of it data and you will have to use a third-party tool like ZenRows or Oxylabs just to mention a few.

In this example, I will be making use of Techmeme.com for scraping.

(async () => {
  const browser = await puppeteer.launch({ headless: "new" });

  const page = await browser.newPage();

 👉 await page.goto("https://www.techmeme.com/");

  await browser.close();
}
  )();

Here, the goto() method is used to specify the web URL we want to scrap.

Once the URL has been specified. visit and inspect the website you want to scrap to have an overview of the HTML structure. In this case, we will visit techmeme.com and inspect the page to know which element to grab and return.

Right-click on the page, this will open a dialog box, then click inspect. for short hit F12 key. See the example below 👇

Your DevTools should open up as indicated below.

From the above screenshot, we have identified the tag we want to scrap which is the div with the class of clus. The parent of clue has a class of topcol1. So we will be grabbing this with Puppeteer.

Try the code sample below. 👇

const browser = await puppeteer.launch({ headless: "new" });

  const page = await browser.newPage();
  await page.goto("https://www.techmeme.com/");

 👉 const techNewsApis = await page.evaluate(() =>
    Array.from(document.querySelectorAll("#topcol1 .clus"), (e) => ({
     // Some code example here.
    }))
  );

Here, the Array.from() method is used to grab all the elements with a class of clus that is a direct parent of topcol1, this will create a shallow-copied Array instance from the returned elements.

Also, notice we are using the evaluate() method, which takes a call-back function. It evaluates the response of a Promise and returns its values when the Promise is resolved. In this case, the resolved value is returned to the techNewsApis variable.

We have seen the basic example of how to grab elements, but this is not all, what we have are elements that Puppeteer does not know what to do with it.

What we want to do is return the headline on the webpage and the link associated with it. We also want to implement a unique id to the data.

From the above screenshot, I inspect the first <a> tag and return the name value as an id.

Similarly, I inspect the <a> tag with a class of ourh which are children of the clue tag and then return both the headline and the link associated with it.

See the code below

const techNewsApis = await page.evaluate(() =>
    Array.from(document.querySelectorAll("#topcol1 .clus"), (e) => ({

     👉   id: e.querySelector(".clus > a").name,
     👉   title: e.querySelector("a.ourh").innerText,
     👉   url: e.querySelector("a.ourh").href,

    }))
  );

Here, querySelector() method is used to access the elements and a property and two attribute is attached to return the respective value associated with it.

name the attribute returns the number string value specified on the <a> tag
innerText returns the text content in between the open and close <a> tag
href returns the link specified on the <a> tag

Now that we have successfully scraped the data, let's return it and see what the data looks like.

const techNewsApis = await page.evaluate(() =>
    Array.from(document.querySelectorAll("#topcol1 .clus"), (e) => ({
      id: e.querySelector(".clus > a").name,
      title: e.querySelector("a.ourh").innerText,
      url: e.querySelector("a.ourh").href,
    }))
  );

 👉 console.log(techNewsApis);

  await browser.close();

Parse the resolved value to the console.log() method.

node index.js

Go to the terminal and run the above command.

The scraped data is now logged to the console as shown above. With the id, title and url specified respectively.

Exporting scraped data in a JSON format

Logging data to the terminal is not a good use case since we can not access or use it directly from the terminal.

What to do next is to save it on the local directory of the project we are working on so that we can make use of it.

    const puppeteer = require('puppeteer');
 👉 const fs = require('fs');

With the fs (file system) module specified, we can write to save the data to something we can use in an application.

const puppeteer = require('puppeteer');
const fs = require('fs');

(async () => {
  const browser = await puppeteer.launch({ headless: "new" });

  const page = await browser.newPage();
  await page.goto("https://www.techmeme.com/");

  const techNewsApis = await page.evaluate(() =>
    Array.from(document.querySelectorAll("#topcol1 .clus"), (e) => ({
      id: e.querySelector(".clus > a").name,
      title: e.querySelector("a.ourh").innerText,
      url: e.querySelector("a.ourh").href,
    }))
  );

  //Save data to JSON file
👉    fs.writeFile("techNewsApis.json", JSON.stringify(techNewsApis), (error) => {
👉    if (error) throw error;
👉       console.log(`techNewsApis is now saved on your project folder`);
          });

  console.log(techNewsApis);
  await browser.close();
}
  )();

Here, the fs module is implemented as follows;

fs.writeFile("techNewsApis.json") set the name of the file to be saved.
JSON.stringify(techNewsApis) is used to receive and parse the value to a valid JSON file
if (error) throw error throws an error if any occur

Next, go to the terminal and run the command as shown below.

node index.js

This will save the exported data as a valid JSON file for external use.

From the above screenshot, the exported data is saved directly in the project folder.

Conclusion

Scraping can be very useful in getting specific data in situations where the data we need is not readily available or can be found on another website, in this case, we can scrape such data and use it in our project.

However, this comes with some caveats as some web owners do not grant access for the scraping of their website.

useLocation Hook in React Router

Alex Anie — Wed, 27 Sep 2023 14:38:58 +0000

Introduction

Single Page Application takes a different approach when navigating from one page (Component) to another.

This is because unlike Multi-page Applications with multiple HTML files that are uploaded to the server and then accessed later based on a new request from the client side.

Single Page Application has one HTML file that is uploaded to the server and once a request is made, the entire app (html file) is sent to the client. This helps users to navigate directly within the client-side (client-side routing) without sending a new request to the server.

With useLocation hook, we can get access to the pathname, hash, state, key or search relative to a specific component and perform side effects in a web app.

In this blog post, I explained how to implement useLocation hook in React Router and how to return the location object and access the object props relative to the components.

Prerequisites

In other to follow along, you should have a basic understanding of the following:

How to setup a React project
How to install React Router

So with that said, let's dive in.

What is useLocation hook

The useLocation hook in React Router is used to return the current location of a React component. The useLocation returns the current location as an object and comes with props such as

pathname
state
search
key
hash

These props can be used with useEffect hook to perform side effects such as clicks onScroll or return state parsed to a Link component.

To use useLocation hook, import it from the react-router-dom as indicated below.

 👉 import { useLocation } from 'react-router-dom';

Then create and store the useLocation hook to a location variable as indicated below.

export default function App(){

    👉 const location = useLocation()

    retun(
        <><p>Hello, useLocation</p></>
    )
}

With the location variable, the useLocation props can be accessed. For example

export default function App(){

    const location = useLocation()
    👉    console.log(location)

    retun(
        <><p>Hello, useLocation</p></>
    )
}

This should return the location variable as an object.

Props

The defined location object returns an object that contains the URL segment of the current location. 👇

pathname

The pathname property (prop) from the location object is used to return the path name of a given component. The path name is the name that comes after the port number.

                    //pathname 
                         👇
http://localhost:5173/products
                  👆
             //port number

From the code sample, the pathname is the forward slash and the products text 👉 /products

Access the pathname as follows ;

export default function App(){
            // Dot Notation
            const location = useLocation()
    👉    console.log(location.pathname)

            // Object Destructuring 
      const {pathname} = useLocation()
    👉    console.log(pathname)

    retun(
        <><p>Hello, useLocation</p></>
    )
}

// layouts/Navbar.js

<div className="main">
     {pathname !== "" ? (
         <aside>
           <Link to="#sectionFour" className="product-btn">
             Click Button
                </Link>
           </aside>
     ) : null}
</div>

Here, the pathname returns a button element when the user clicks.

Try the code example on code Codesandbox below

%[https://codesandbox.io/s/uselocation-pathname-xyq46q?file=/src/Layouts/Navbar.js]

state

The state property (prop) from the location object is used to set data on parent components that can be accessed by the children components.

To set state on the parent component, parse the state property and assign it a value as specified below. 👇

           <li>
                <NavLink
                  to="/"
              👉  state={`Welcome to the ${
                    pathname.substring(1) !== "" ? "home" : ""
                  } page`}
                >
                  Home
                </NavLink>
              </li>
                                    ...
              <li>
               <NavLink
                  to="products"
                  state={`Welcome to the ${pathname.substring(1)} page`}
                >
                  Products
                </NavLink>
           </li>

Here, the pathname of the specific link the user clicks is returned as a string and we applied the following functions to the state props.

substring(1) This method removes the forward slash from the pathname
the ternary operator checks and returns “home” if the path contains an empty string.

Try the code example on Codesandbox below

[https://codesandbox.io/s/uselocation-state-h6wrrj?file=/src/Layouts/Navbar.js]

search

The search property (prop) from the location object is used to get the query string in a URL.

For example, take this URL "http://example.com/search?age=20&name=alex"

The query string refers to a portion in the URL starting from the ? character followed by a key (’age’ and ‘name’ ) and value (’20’ and ‘alex’) pairs. These keys and value pairs are called parameters and are separated by the & character.

Access the search as follows ;

export default function App(){

            // Dot Notation
            const location = useLocation()
    👉    console.log(location.search)

            // Object Destructuring 
      const {search} = useLocation()
    👉    console.log(search)

    retun(
        <><p>Hello, useLocation</p></>
    )
}

key

The key property (prop) from the location object is used to hold a unique identifier (id) from the current URL.

Access the key as follows ;

export default function App(){

            // Dot Notation
            const location = useLocation()
    👉    console.log(location.key)

            // Object Destructuring 
      const {key} = useLocation()
    👉    console.log(key)

    retun(
        <><p>Hello, useLocation</p></>
    )
}

export default function Navbar() {
  const { pathname, key } = useLocation();

  return (
    <>
         <ul>
              <li>
                <NavLink to="/">Home</NavLink>
              </li>
            ...
              <li>
                <NavLink to="products">Products</NavLink>
              </li>
            </ul>

      {/* Outlet */}
        <Outlet />
        <div className="main">
          {pathname !== "" ? (
            <aside>
              <Link className="product-btn">Click Button</Link>
            </aside>
          ) : null}

          <p>
            the return text is a key: <i className="key">{key}</i>
          </p>
        </div>
    </>
  );
}

Here, we are returning the key of each component when the navlink is clicked.

Try code example on Codesandbox

%[https://codesandbox.io/s/uselocation-key-3p5vtg?file=/src/Layouts/Navbar.js]

hash

The hash property (prop) from the location object is used to set the current location in the # hash segment of the current URL.

This is useful in cases where you want to navigate between different sections on the same page or navigate to a specific section on another page.

Access the hash as follows ;

export default function App(){

            // Dot Notation
            const location = useLocation()
    👉    console.log(location.hash)

            // Object Destructuring 
      const {hash} = useLocation()
    👉    console.log(hash)

    retun(
        <><p>Hello, useLocation</p></>
    )
}

%[https://codesandbox.io/s/uselocation-hash-zs94kw?file=/src/App.js]

Here, from the example above, we store the list of countries within the page as a hash fragment. So clicking each link navigates to a specific section of the page.

Conclusion

The useLocation hook is used to return the current location of a react component. The useLocation uses the location object to access the key, pathname, state, hash and search of the component.

This is useful for client-side routing and for navigating users within and around different pages in a single-page application.

References

React Router Official Docs

useLocation Hook in React Router

Alex Anie — Wed, 27 Sep 2023 14:19:38 +0000

Introduction

Single Page Application takes a different approach when navigating from one page (Component) to another.

This is because unlike Multi-page Applications with multiple HTML files that are uploaded to the server and then accessed later based on a new request from the client side.

With useLocation hook, we can get access to the pathname, hash, state, key or search relative to a specific component and perform side effects in a web app.

In this blog post, I explained how to implement useLocation hook in React Router and how to return the location object and access the object props relative to the components.

Prerequisites

In other to follow along, you should have a basic understanding of the following:

How to setup a React project
How to install React Router

So with that said, let's dive in.

What is useLocation hook

The useLocation hook in React Router is used to return the current location of a React component. The useLocation returns the current location as an object and comes with props such as

pathname
state
search
key
hash

These props can be used with useEffect hook to perform side effects such as clicks onScroll or return state parsed to a Link component.

To use useLocation hook, import it from the react-router-dom as indicated below.

 👉 import { useLocation } from 'react-router-dom';

Then create and store the useLocation hook to a location variable as indicated below.

export default function App(){

    👉 const location = useLocation()

    retun(
        <><p>Hello, useLocation</p></>
    )
}

With the location variable, the useLocation props can be accessed. For example

export default function App(){

    const location = useLocation()
    👉    console.log(location)

    retun(
        <><p>Hello, useLocation</p></>
    )
}

This should return the location variable as an object.

Props

The defined location object returns an object that contains the URL segment of the current location. 👇

pathname

The pathname property (prop) from the location object is used to return the path name of a given component. The path name is the name that comes after the port number.

                    //pathname 
                         👇
http://localhost:5173/products
                  👆
             //port number

From the code sample, the pathname is the forward slash and the products text 👉 /products

Access the pathname as follows ;

export default function App(){
            // Dot Notation
            const location = useLocation()
    👉    console.log(location.pathname)

            // Object Destructuring 
      const {pathname} = useLocation()
    👉    console.log(pathname)

    retun(
        <><p>Hello, useLocation</p></>
    )
}

// layouts/Navbar.js

<div className="main">
     {pathname !== "" ? (
         <aside>
           <Link to="#sectionFour" className="product-btn">
             Click Button
                </Link>
           </aside>
     ) : null}
</div>

Here, the pathname returns a button element when the user clicks.

Try the code example on code Codesandbox below

%[https://codesandbox.io/s/uselocation-pathname-xyq46q?file=/src/Layouts/Navbar.js]

state

The state property (prop) from the location object is used to set data on parent components that can be accessed by the children components.

To set state on the parent component, parse the state property and assign it a value as specified below. 👇

           <li>
                <NavLink
                  to="/"
              👉  state={`Welcome to the ${
                    pathname.substring(1) !== "" ? "home" : ""
                  } page`}
                >
                  Home
                </NavLink>
              </li>
                                    ...
              <li>
               <NavLink
                  to="products"
                  state={`Welcome to the ${pathname.substring(1)} page`}
                >
                  Products
                </NavLink>
           </li>

Here, the pathname of the specific link the user clicks is returned as a string and we applied the following functions to the state props.

substring(1) This method removes the forward slash from the pathname
the ternary operator checks and returns “home” if the path contains an empty string.

Try the code example on Codesandbox below

%[https://codesandbox.io/s/uselocation-state-h6wrrj?file=/src/Layouts/Navbar.js]

search

The search property (prop) from the location object is used to get the query string in a URL.

For example, take this URL "http://example.com/search?age=20&name=alex";

Access the search as follows ;

export default function App(){

            // Dot Notation
            const location = useLocation()
    👉    console.log(location.search)

            // Object Destructuring 
      const {search} = useLocation()
    👉    console.log(search)

    retun(
        <><p>Hello, useLocation</p></>
    )
}

key

The key property (prop) from the location object is used to hold a unique identifier (id) from the current URL.

Access the key as follows ;

export default function App(){

            // Dot Notation
            const location = useLocation()
    👉    console.log(location.key)

            // Object Destructuring 
      const {key} = useLocation()
    👉    console.log(key)

    retun(
        <><p>Hello, useLocation</p></>
    )
}

export default function Navbar() {
  const { pathname, key } = useLocation();

  return (
    <>
         <ul>
              <li>
                <NavLink to="/">Home</NavLink>
              </li>
            ...
              <li>
                <NavLink to="products">Products</NavLink>
              </li>
            </ul>

      {/* Outlet */}
        <Outlet />
        <div className="main">
          {pathname !== "" ? (
            <aside>
              <Link className="product-btn">Click Button</Link>
            </aside>
          ) : null}

          <p>
            the return text is a key: <i className="key">{key}</i>
          </p>
        </div>
    </>
  );
}

Here, we are returning the key of each component when the navlink is clicked.

Try code example on Codesandbox

%[https://codesandbox.io/s/uselocation-key-3p5vtg?file=/src/Layouts/Navbar.js]

hash

The hash property (prop) from the location object is used to set the current location in the # hash segment of the current URL.

This is useful in cases where you want to navigate between different sections on the same page or navigate to a specific section on another page.

Access the hash as follows ;

export default function App(){

            // Dot Notation
            const location = useLocation()
    👉    console.log(location.hash)

            // Object Destructuring 
      const {hash} = useLocation()
    👉    console.log(hash)

    retun(
        <><p>Hello, useLocation</p></>
    )
}

%[https://codesandbox.io/s/uselocation-hash-zs94kw?file=/src/App.js]

Here, from the example above, we store the list of countries within the page as a hash fragment. So clicking each link navigates to a specific section of the page.

Conclusion

This is useful for client-side routing and for navigating users within and around different pages in a single-page application.

References

React Router Official Docs

NavLink Component in React Router

Alex Anie — Fri, 22 Sep 2023 12:11:07 +0000

Introduction

The <NavLink> component in React-Router is used for navigating to different components within a single page application. This is very useful when creating a navbar component in your React app. The <NavLink> component does the same thing as the <Link> component but provides additional features like styling props that enable developers to add an active state.

This helps users to visually know which part of the navbar is clicked and what part of the page they are currently in.

In this blog post, I explained what <NavLink> do, how to add active class to navbar components and also how to work with other props that come with the <NavLink> components.

Prerequisite

To follow along, you should have a basic understanding of the following.

How to set up a React App
How to install React Router

NavLink

The <NavLink> component in React Router is used for navigation. The <NavLink> renders as <a> tag when loaded on the DOM.

This is very useful for linking pages that require an active state for example navbar, breadcrumbs, and list items with active class that shows which item is currently selected.

To apply <NavLink> import it form the react-dom-dom like so 👇

import { NavLink } from "react-router-dom";

The <NavLink> is an opening and closing tag, so text content should be applied followed by the closing tag.

<li>
    <NavLink to="/">Home</NavLink>
</li>

Props

The <NavLink> components come with props (properties), these props can be used to style the <NavLink> component which determines how NavLink is rendered on the page. These include

to
active (can be specified but exist by default)
style
className
children
end
caseSensitive
aria-current
reloadDocument

to

The to prop sets the path or route that should display when the link is clicked.

import { NavLink } from "react-router-dom";

    <ul>
            <li>
              <NavLink to="/">Home</NavLink>
            </li>
            <li>
              <NavLink to="service">Services</NavLink>
            </li>
            <li>
              <NavLink to="about">About</NavLink>
            </li>
        </ul>

The to is equivalent to the href in the <a>. This means that when the to prop renders on the DOM it returns a href attribute.

active

The <NavLink> component by default has an active class which can be styled to indicate which component or page is active.

import { NavLink } from "react-router-dom";

    <ul>
            <li>
              <NavLink to="/">Home</NavLink>
            </li>
            <li>
              <NavLink to="service">Services</NavLink>
            </li>
            <li>
              <NavLink to="about">About</NavLink>
            </li>
       </ul>

Unlike most props, the active class is Not specified directly on the <NavLink> component but it exists and can be styled as follows.

.navbar ul a.active {
  color: rgb(34 197 94);
}

See Codesandbox for a code sample.

%[https://codesandbox.io/s/blue-rain-wcvrwd?file=/src/components/css/navbar.css:334-383]

style

The style prop in <NavLink> component works differently from another style prop in other components. It takes in two functions isActive and isPending as a parameter and can be destructured within JSX to specify what should happen to the element when it’s active or pending.

The style prop can be very useful in situations where you are not using CSS for styling but still want to apply styles on active state.

isActive

Example one

<NavLink
   to="/"
   style={({ isActive }) => {
       return isActive ? { color: "plum" } : {};
       }}
       >
     Home
</NavLink>

Here, isActive function is destructed with a callback function that returns an object with a color property of plum if isActive is true and otherwise return an empty object if it false.

See the code sample on Codesandbox.

%[https://codesandbox.io/s/02-navlink-classname-qjyxkk?file=/src/components/Navbar.js]

Example two

You might want to specify your code at the top level of your component and then parse the return value to the <NavLink>. This can be very useful in situations where you want to pass in multiple style properties as objects and you don’t want to mess up your JSX syntax with convoluted code.

<NavLink to="/" style={activeState}> Home</NavLink>

See the code example on Codesandbox

%[https://codesandbox.io/s/02-navlink-style-example-2-pqyt7q?file=/src/components/Navbar.js]

isPending

The isPending function is used to set what style should apply to the link when a user clicks on it and the link is still in the navigating phase and is not yet complete.

It works like a promise, where you have a pending state before the promise is fulfilled.

Example four

<NavLink
   to="/"
   style={({ isPending }) => {
         return {
           color: isPending ? "yellow" : ""
         };
      }}
     >
   Home
</NavLink>

See the code example on Codesandbox

%[https://codesandbox.io/s/02-navlink-style-example-4-hl4tqg?file=/src/components/Navbar.js]

isActive and isPending

The isActive state and isPending state can both be specified in the same function.

const activeState = ({ isActive, isPending }) => {
    return {
      color: isPending ? "rgb(253 230 138)" : "",
      backgroundColor: isActive ? "rgb(69 26 3)" : "",
      fontWeight: isActive ? "bold" : ""
    };
  };

See the code example on Codesandbox

%[https://codesandbox.io/s/02-navlink-style-example-5-97373q?file=/src/components/Navbar.js]

ClassName

The className prop does the same thing as the style prop but with class attributes.

<NavLink
    to="/"
    className={({ isActive }) => {
       return isActive ? "active" : "";
    }}
      >
    Home
</NavLink>

See the code example on Codesandbox

%[https://codesandbox.io/s/02-navlink-classname-example-1-8l5yvg?file=/src/components/Navbar.js]

tailwind classes

The className prop can be very useful in a tailwind application when you need to apply an active state to a link element component.

<NavLink
    to="/"
       className={({ isActive }) => {
          return isActive ? "text-orange-500" : "text-white";
         }}
       >
        Home
   </NavLink>

See the code example on Codesandbox

%[https://codesandbox.io/s/02-navlink-classname-tailwindcss-1-ktxj7q?file=/src/components/Navbar.js]

children

The children prop is not a prop but a way of parsing JSX functions as children that conditionally returns an element as direct children when the <NavLink> is active.

<NavLink to="/">
    {({ isActive }) => {
         return isActive ? (
         <span className="">Home</span>
          ) : (
          <span className="">Page</span>
         );
       }}
  </NavLink>

See the code example on Codesandbox

%[https://codesandbox.io/s/03-navlink-children-538zlq?file=/src/components/Navbar.js:226-509]

Example two

There are cases where you might want to display a text or an emoji alongside the text on the <NavLink> when it's active. This can be done using the method below.

<NavLink
    to="about"
    style={({ isActive }) => {
        return isActive ? { color: "plum" } : {};
      }}
     >
       {({ isActive }) => {
         return isActive ? "💁 About" : "About";
      }}
 </NavLink>

See the code example on Codesandbox

%[https://codesandbox.io/s/03-navlink-children-example-2-w5t8wx]

end

The end prop is used to indicate the route where the active or pending state should stop. It prevents the children from inheriting the active state from the parent <NavLink>.

<NavLink
   to="service"
   end
   style={({ isActive }) => {
       return isActive ? { color: "plum" } : {};
   }}
     >
  Services
</NavLink>

Image Credit React Reouter Docs

caseSensitive

By default react router matches any path or route specified in the to prop regardless of Its case. To change this behaviour and make it match based on the case parsed to the to prop the caseSensitive prop is used.

This means that about and About are not going to match because of differences in casing.

<NavLink
    to="serVicE"
    caseSensitive
    style={({ isActive }) => {
        return isActive ? { color: "plum" } : {};
           }}
    >
     Services
</NavLink>

Image Creadit React Reouter Docs

aria-current

The aria-current prop is used to set the aria-current on the active element.

<NavLink
   to="serVicE"
   aria-current="page"
   style={({ isActive }) => {
         return isActive ? { color: "plum" } : {};
          }}
      >
      Services
  </NavLink>

reloadDocument

The reloadDocument prop is used to set the <NavLink> component back to the default behaviour by preventing client-side routing. This will enable the page to reload anytime the link is clicked.

<NavLink
  to="serVicE"
  reloadDocument
  style={({ isActive }) => {
  return isActive ? { color: "coral" } : {};
      }}
  >
    Services
</NavLink>

See the code example on Codesandbox

%[https://codesandbox.io/s/05-navlink-reloaddocument-r8qjjg?file=/src/components/Navbar.js]

Parsing Attribute on the NavLink Component

Aside the default props, the <NavLink> can accept HTML attribute same as the <a> tag.

<NavLink className="card" key={item.id} to={item.src} target="_blank">text</NavLink>

Conclusion

The <NavLink> component in React Router makes it possible for client-side routing just like the <Link> component, they both share the properties and functions the same way. However, the <NavLink> component comes with some additional properties that make it easy for styling navigation menus and adding active or pending state to link element.

We can determine which part of the navigation menu a user is currently on when the isActive function is set on the <NavLink>

Reference

React Router Official Docs

Forem: Alex Anie

Scrape and build a LinkedIn user profile app with Bright Data and React.js

Prerequisites

Overview

Create a Bright Data Account

Web Scraper IDE Interface

Scrape your first dataset

React.js set up with Vite

Folder structure setup

ContentList section

Footer Section

Handling Error in your Application - errorElement

Implementing Dynamic Segment

Styling the Scrollbar of the application

Selenium Proxy for Browser Automation

Table of Contents

What Is a Selenium Proxy?

Key Features of Selenium Proxy

Setting Up Selenium Proxy

Installing Required Packages

Basic Proxy Configuration

Running Selenium Proxy

Running Selenium Proxy on Cloud

Using Selenium Proxy with Authentication

Implement a Rotating Selenium Proxy

Handling Selenium Proxy Errors

Best Practices of Selenium Proxy

Conclusion

Why We Should Stop Saying “MVC” and Start Saying “RCMV” Instead

Router

Controller

Model

View

Scrape Unscrapeable Amazon Dataset with BrightData, React.js and Node.js

What I Built

How I Used Bright Data

Deployment

Stacks Used

Revolutionizing Website Testing: A Deep Dive into LambdaTest's KaneAI

What is Kane AI

Key Features of KaneAI

Useful Command Types for KaneAI

Run your first KaneAI web Test.

Sequential Prompt

Multiply Prompts

Manual Interaction

Finish Test

Test Manager

Summary

Create a Sales Dashboard with Neon, Prisma, Kinde-Auth, Next.js, and Tailwindcss.

My Kit

Link to Kit

Your Journey

ChatGPT AI chat model with Llama2, Workers AI Model, Hono.js and Wranger.

What I Built

Demo

GitHub Code

Implementation

Web Scraping with Cheerio, Axios, and Express

Prerequisites

Overview

Dependencies

Project Setup

Index.js file

Express server setup

Scraping data from a web page

Exporting scraped data

Conclusion

Scraping with puppeteer and export in JSON format

Prerequires

Setup

Entry Point Setup

Scraping web page data

Exporting scraped data in a JSON format

Conclusion

useLocation Hook in React Router

Introduction

Prerequisites

What is useLocation hook

Props