Forem: Vinayak Sharma

Selenium with Python Tutorial: Creating Automated Web Bot

Vinayak Sharma — Wed, 21 Sep 2022 14:59:40 +0000

Selenium is one of the preferred test automation frameworks, primarily used for web automation testing. Though Selenium supports six popular programming languages, it can be used to weave magic with lesser implementation when used with Python such as creating web bot using Selenium with Python.

Beyond web automation testing, web scraping using Selenium Python is a very popular use case that you can do with fewer LOC with Python. As a QA engineer, I still use other programming languages like Java and JavaScript with Selenium, but Python has been my go-to language for simple/complex web automation tasks like is this tutorial for web bots using selenium with python.

As mentioned in this Selenium Python tutorial, Python is the third-most-popular programming language after HTML/CSS and JavaScript. Web bot using Selenium with Python or Selenium Python bot can be used extensively for automating multiple scenarios (or tasks).

In this Selenium Python tutorial on Selenium Python bot, I deep dive into how to create a web bot in Selenium and Python; the learnings of which will be useful for multiple aspects of Selenium automation testing.

Tools For Creating Web Bot using Selenium with Python

One of the primary objectives of creating a Selenium with Python bot is to leverage the capabilities offered by Selenium for automating interactions with WebElements in the DOM. There is a suite of Selenium testing tools for automating web browsers, which can be used for a range of automation tasks like web scraping, web automation testing, cross browser testing, and much more.

So, let’s look at the setup essentials that are required for creating a web bot using Selenium with Python.

Selenium

The Python library that wraps the Selenium WebDriver provides binding and a simple API to write functional tests for automating web tasks such as clicking buttons, navigating web pages, and filling out forms.

The test script does not interact directly with the web browser. Selenium WebDriver is the core of the Selenium framework. As mentioned in this Selenium WebDriver tutorial, Selenium WebDriver is instrumental in establishing the connection with the browser using its corresponding browser driver. However, you can further deep dive into what is Selenium, if you want to know more about Selenium and its components.

If the tests have to be run on a local Selenium grid, the corresponding browser driver has to be installed on the target machine:

Downloading and installing the browser drivers are required only when the tests have to be performed on the local Selenium Grid. In the case of a Remote Selenium WebDriver, the tests are executed on a cloud Selenium Grid (e.g. LambdaTest).

Python Automation Frameworks

Python for Selenium Automation Testing is a preferable choice since the language is less verbose, supports dynamic typing, supports hassle-free reporting, and much more. The wide range of Selenium with Python frameworks support parallel testing in Selenium; thereby expediting the time to market. Also, creating a Web Bot using Selenium with python is a very Simple and easy-to-implement process.

In this Selenium with Python tutorial, I have touched upon the essentials of Selenium with Python, let’s me, deep dive, into the most popular Selenium with Python frameworks for automation testing:

PyUnit (Unittest)

Unittest (or PyUnit) is the default Python testing framework that comes with the Python package. This testing framework for Python is inspired by the popular JUnit framework. Creating a Web Bot using Selenium with Python Can be achieved with Unittest framework.

Unittest provides simple and flexible test case execution with the quick generation of test reports. Unittest uses camelCase naming conventions instead of Python’s snake_case naming conventions.

Selenium PyUnit tutorial can be a good starting point for someone looking to leverage the framework for web automation testing.

PyTest

PyTest is an open-source testing framework that supports unit testing, functional testing, and API testing. It is available on Python version 3.5 (and higher). Making a Web Bot using Selenium with Python Can be accomplished with PyTest.

Page Object Model (POM) in Selenium Python is instrumental in improving the maintainability and reusability aspects of the source code. I prefer the PyTest framework over PyUnit due to the rich set of plugins that are a boon to the automation engineers.

Some of the most popular PyTest plugins for automation engineers are below:

pytest-randomly
pytest-cov
pytest-django
pytest-bdd

Selenium PyTest tutorial deep dives into all the integral aspects of web automation testing with Selenium and PyTest.

This PyTest Tutorial for beginners and professionals will help you learn how to use the PyTest framework with Selenium and Python for performing Selenium automation testing.

Learn what Selenium is and its benefits for automated cross browser testing. See how Selenium works and get ready to use it for testing. Read more.

Robot Framework

The Robot Framework is an open-source automation framework that allows the creation of human-readable test cases using Gherkin.

Watch this video to learn the basics of the robot framework for automation testing. This video will give you an understanding of the various functionalities of the Robot Framework with live examples. Developers and testers mainly use Robot Framework to make a Web Bot using Selenium with Python as it as various functionalities.

The Robot framework is best suited to leverage the benefits of Behaviour Driven Development (BDD) using Selenium and Gherkin. Robot is also used for acceptance of Test-Driven Development (TDD) and Robotic Process Automation (RPA).

The core implementation of Robot Framework is in Python but can also run on Jython and IronPython. It natively doesn’t support parallel testing but provides a variety of extensions and libraries which are easy to use.

For demonstrating the implementation of the Selenium Python bot, I am going to use the PyTest framework.

Behave

Behave is testing frameworks used for Behavior-Driven Development (BDD). It has a large amount of documentation and support from the community. To top it all, it provides inbuilt integrations of Django and Flask — two of the most popular web frameworks in Python. After Robot framework Behave to make a Web Bot using Selenium with Python.

Behave is also useful for black box testing. BDD with Behave and Selenium comes with its own set of advantages & shortcomings but it is still a nifty Python automation testing framework for Selenium testing.

Apart from these frameworks that I normally use for web automation testing, there are a number of popular Selenium Python frameworks that can be looked into to leverage the capabilities of Selenium and Python.

Task Schedulers in Python

Task Schedulers are programs that enable scheduling of tasks or scripts after certain time intervals. Alternately, scripts can also be scheduled (or triggered) based on certain conditions.

Task Scheduler can be used extensively for running a batch (or .bat) file that triggers the corresponding Python script.

Some of the common programs that provide task scheduling in Python are:

Cron

Cron is a time-based job scheduler in Unix-like computer operating systems such as Linux and macOS. It allows users to schedule tasks such as taking backups, create logs, and run scripts

Windows Task Scheduler

Task Scheduler is a component of Windows OS that lets you create and run essentially any task in an automatic fashion. Usually, the system and certain apps are scheduled for automation and maintenance. Tasks like disk defragmentation, disk cleanup, and updates, etc. can be scheduled automatically using the Windows Task Scheduler.

Windows Task Scheduler works by keeping track of the time and events on the computers and executes the task if the necessary conditions are met.

For this blog, I am going to use the Windows Task Scheduler program, along with the Win32com Library — a thin Python wrapper that helps automate Windows applications. In this case, it’s the Task Scheduler.

Now that we have touched upon the integral aspects of building a web bot using Selenium with Python, let’s get our hands dirty with some implementation.

Execute Playwright testing in parallel and cut down your test execution time by multiple folds.

Building Automated Web Bot using Selenium with Python

Follow the below mentioned steps to create a development environment for the implementation:

Creating a Virtual environment and installing dependencies

Virtual environment (or virtualenv) is a tool for creating isolated Python environments. For this purpose, Python provides a venv module that supports creation of lightweight “virtual environments” with its isolated site directories.

You can check out this Selenium Python tutorial for a quick refresher on web automation testing with Selenium and Python.

python3 -m venv env

Run the below mentioned command to activate this virtual environment:

Windows

.\env\Scripts\activate

Linux

source env/bin/activate

Run the following command to install Selenium for Python:

pip install selenium
Run the following command to install the PyTest framework:

pip install pytest
Install the pywin32 package to for managing the Windows Task Scheduler:

pip install pywin32

Folder structure of Test Scripts

The project majorly consists of two files:

main.py — Contains the main script that enables logging on to the LambdaTest platform and fetching details of the respective tests.
scheduler.py — Contains the implementation to schedule the test scripts.

A comprehensive end to end Testing tutorial that covers what E2E Testing is, its importance, benefits, and how to perform it with real-time examples.

Implementation of Automated Selenium Python bot

For demonstrating the creation of Web Bot using Selenium with Python, we automate the test scenario of logging on to the LambdaTest platform. The URL under test is https://accounts.lambdatest.com/login

Since the test is demonstrated on the LambdaTest Grid, I am using the Remote WebDriver in Selenium instead of the local WebDriver. To get started, we import the necessary packages.

from selenium import webdriver

We define two variables that point to the username and access key, the combination of which is used for accessing the LambdaTest cloud grid. The username and access key can be accessed from LambdaTest profile page.

username = "<YOUR USERNAME>"
access_key = "<YOUR ACCESS KEY>"

Define a class named ‘TestClass’ with the following functions:

Setup_method — It is a function used for setting up the remote driver and its corresponding properties.

Generate capabilities from here: https://www.lambdatest.com/capabilities-generator/

setUp runs before each test case and

def setup_method(self):
desired_caps = {
"build": "your build name",
"name": "your test name",
"platform": "Windows 10",
"browserName": "Chrome",
"version": "92.0",
"selenium_version": "3.13.0",
"geoLocation": "IN",
"chrome.driver": "91.0",
}
"""
Setup remote driver
-------
username and access_key can be found on the lambdatest platform

"""
self.driver = webdriver.Remote(
command_executor="@hub.lambdatest.com/wd/hub">https://{}:{}@hub.lambdatest.com/wd/hub".format(
username, access_key
),
desired_capabilities=desired_caps )
Teardown_method — It is a function used for tearing down runs after each test case.

def teardown_method(self):
self.driver.quit()
Test_login — Implementation under this function is used for logging into the LambdaTest dashboard. Here, I make use of SendKeys in Selenium to fill out the login form so that I can login to the LambdaTest website. Here is the syntax of the SendKeys method in Selenium Python:

element.send_keys("text to enter")

For locating the respective WebElements, I have used the XPath locator in Selenium. However, you can use other popular Selenium locators like ID, Name, linktext, etc. for locating the WebElements.

Shown below is the syntax of find_element_by_xpath() method in Selenium Python. Alternatively, you can also use the driver.find_element(By.XPATH) method for locating the element using XPath locator.

element = driver.find_element_by_xpath("xpath_here")

driver.find_element(By.XPATH, "xpath")

Here is how the entire login method (i.e. test_login) looks like:

def test_login(self):
       """
       this function logins you in lambdatest dashboard
       """
       LOGIN_URL = "[https://accounts.lambdatest.com/login](https://accounts.lambdatest.com/login)"
       DASHBOARD_URL = "[https://accounts.lambdatest.com/dashboard](https://accounts.lambdatest.com/dashboard)"
       self.driver.get(LOGIN_URL)
       login_email = self.driver.find_element_by_xpath(
           "//input[[@id](http://twitter.com/id)='email']"
       )
       login_password = self.driver.find_element_by_xpath(
           "//input[[@id](http://twitter.com/id)='password']"
       )
       login_button = self.driver.find_element_by_xpath(
           "//button[[@id](http://twitter.com/id)='login-button']"
       )

       login_email.send_keys(email)
       login_password.send_keys(password)
       login_button.click()

       # if we are on right page
       if self.driver.current_url == DASHBOARD_URL :
           assert True
           # time to get recent tests
           self.get_recent_tests()
       else:
           print(" something went wrong !!")
           assert False

*Get_recent_tests *— Once I have logged into the LambdaTest Dashboard, I fetch all recent tests from the lambdatest dashboard.

As seen above, .cut-text is a common class for all tags that we are interested to read. Next, I will perform web scraping using Selenium and Python for fetching the required information.

titles = self.driver.find_elements_by_class_name("cut-text")

Here, I clean and scrape text from the respective tags:

def get_recent_tests(self):
       titles = self.driver.find_elements_by_class_name("cut-text")
       titles_list = []
       for i in titles:
           titles_list.append(i.text)
       message = ' '.join(titles_list[0::2])
       print(message)
       send_email(message)

As seen in the implementation, we are using Selenium 3 Grid for testing. With the release of stable Selenium 4, many QA engineers are migrating Selenium automation tests to Selenium 4 grid. Before doing so, I recommend reading Selenium 3 vs Selenium 4 comparison so that tests can be migrated in a seamless manner.

Sending emails using Selenium for Python

Creating web bot using selenium with is not the only use. Python provides built-in smtplib library to send basic emails with secure connection using SMTP_SSL() and .starttls(). The smtplib library uses the RFC 821 protocol for Simple Mail Transfer Protocol (STMP).

For sending an email using the smtplib module, I have created a function that accepts a message as a string and subsequently sends an email.

def send_email(message):
   context = ssl.create_default_context()
   try:
       server = smtplib.SMTP(smtp_server,port)
       server.ehlo() # Can be omitted
       server.starttls(context=context) # Secure the connection
       server.ehlo() # Can be omitted
       server.login(sender_email, sender_password)
       server.sendmail(sender_email, receiver_email, message)
       server.close()
   except:
       print(" failed to send email ")

Putting all together, this is how the overall implementation looks like in main.py.

from selenium import webdriver
import smtplib, ssl

port = 587
smtp_server = "smtp.gmail.com"
sender_password = "<YOUR_PASSWORD>"
sender_email = "<YOUR_EMAIL>"
receiver_email = "<EMAIL_WHERE_YOU_WANT_TO_SEND>"

def send_email(message):
   context = ssl.create_default_context()
   try:
       server = smtplib.SMTP(smtp_server,port)
       server.ehlo() # Can be omitted
       server.starttls(context=context) # Secure the connection
       server.ehlo() # Can be omitted
       server.login(sender_email, sender_password)
       server.sendmail(sender_email, receiver_email, message)
       server.close()
   except:
       print(" failed to send email ")

username = "<YOUR_LAMBDATEST_USERNAME>"
access_key = "<YOUR__LAMBDATEST_ACCESS_KEY>"

class TestClass():
   def setup_method(self):
       desired_caps = {
           "build": "your build name",
           "name": "your test name",
           "platform": "Windows 10",
           "browserName": "Chrome",
           "version": "92.0",
           "selenium_version": "3.13.0",
           "geoLocation": "IN",
           "chrome.driver": "91.0",
       }
       """
       Setup remote driver
               -------
               username and access_key can be found on lt platform

       """
       self.driver = webdriver.Remote(
           command_executor="https://{}:{}@hub.lambdatest.com/wd/hub".format(
               username, access_key
           ),
           desired_capabilities=desired_caps,
       )

   # tearDown runs after each test case
   def teardown_method(self):
       self.driver.quit()

   def test_login(self):
       """
       this function logins you in lambdatest dashboard
       """
       LOGIN_URL = "https://accounts.lambdatest.com/login"
       DASHBOARD_URL = "https://accounts.lambdatest.com/dashboard"
       self.driver.get(LOGIN_URL)
       login_email = self.driver.find_element_by_xpath(
           "//input[@id='email']"
       )
       login_password = self.driver.find_element_by_xpath(
           "//input[@id='password']"
       )
       login_button = self.driver.find_element_by_xpath(
           "//button[@id='login-button']"
       )

       login_email.send_keys(email)
       login_password.send_keys(password)
       login_button.click()

       # if we are on the right page
       if self.driver.current_url == DASHBOARD_URL :
           assert True
           # time to get recent tests
           self.get_recent_tests()
       else:
           print("something went wrong !!")
           assert False

   def get_recent_tests(self):

       titles = self.driver.find_elements_by_class_name("cut-text")
       titles_list = []
       for i in titles:
           titles_list.append(i.text)
       message = ' '.join(titles_list[0::2])
       send_email(message)

Scheduling Selenium with Python Test Script

Making web bot using selenium python isn’t the main use. The scheduler allows to schedule test scripts to be run at a given point in time or under a given condition. For this task, I am using Windows Task Scheduler, which is a Windows GUI application. I am using the Win32com Library for programmatically scheduling the triggers.

Win32com is the API library, a very thin Python wrapper that allows automating Windows Applications as an instance of the Windows Task Scheduler.

Windows Task Scheduler uses COM (Component Object Model) objects. It allows controlling windows applications from another program or script. COM Objects are defined in the Windows registry. As COM objects are “objects”, they can be accessible to the test script and can be interacted with in a programmatic manner.

Here is the run down of the Python code for running the developed test script in Windows task scheduler:

Defining win32com client from pywin32 library:

scheduler = win32com.client.Dispatch('Schedule.Service')
scheduler.Connect()
root_folder = scheduler.GetFolder('\')
task_def = scheduler.NewTask(0)
Setting up few important Arguments

start time: Set the start time for the script

start_time = datetime.datetime.now() + datetime.timedelta(minutes=1)

Repetition.Duration: Time duration (in days) for which the program execution needs to be repeated. In this particular case, I want to run this script for 10 days.

#Repeat for 10 days
num_of_days = 10
trigger.Repetition.Duration = "P"+str(num_of_days)+"D"

Repetition.Interval: This variable tells after how much time the program needs to be executed. Here I have set to run the script after every 6 hours.

#For every 6 hour
trigger.Repetition.Interval = "PT6H"

action.Path: Path to the Python interpreter or pytest binary.

action.Path = r'C:\Users\vinayak\selenium_test\env\Scripts\pytest.exe'

action.Arguments: Path of the script that has to be executed.

action.Arguments = r'C:\Users\vinayak\selenium_test\main.py'

For writing our scheduler code, let’s create a scheduler.py file.

import datetime

import win32com.client

scheduler = win32com.client.Dispatch('Schedule.Service')

scheduler.Connect()

root_folder = scheduler.GetFolder('\')

task_def = scheduler.NewTask(0)

  
  
  Start time of script


start_time = datetime.datetime.now() + datetime.timedelta(minutes=1)

  
  
  for running it one time


TASK_TRIGGER_DAILY = 1

trigger = task_def.Triggers.Create(TASK_TRIGGER_DAILY)

  
  
  Repeat for 10 day


num_of_days = 10

trigger.Repetition.Duration = "P"+str(num_of_days)+"D"

  
  
  For every 6 hour


trigger.Repetition.Interval = "PT6H" 

trigger.StartBoundary = start_time.isoformat()

  
  
  Create action


TASK_ACTION_EXEC = 0

action = task_def.Actions.Create(TASK_ACTION_EXEC)

action.ID = 'TRIGGER BATCH'

action.Path = r'C:\Users\vinayak\selenium_test\env\Scripts\pytest.exe'

action.Arguments = r'C:\Users\vinayak\selenium_test\main.py'

  
  
  Set parameters


task_def.RegistrationInfo.Description = 'Test Task'

task_def.Settings.Enabled = True

task_def.Settings.StopIfGoingOnBatteries = False

  
  
  Register task


  
  
  If task already running, it will be updated


TASK_CREATE_OR_UPDATE = 6

TASK_LOGON_NONE = 0

root_folder.RegisterTaskDefinition(

   'Test Task',  # Task name

   task_def,

   TASK_CREATE_OR_UPDATE,

   '',  # No user

   '',  # No password

   TASK_LOGON_NONE

)

It’s A Wrap

Selenium webdriver with Python is used for applications like scraping websites or testing applications. But automation can also be used in many other ways as we have used it in our particular use-case. In this Selenium Python tutorial on web bot using selenium with python, we have seen how to create automated web bot using Selenium with Python.

In the same way, Windows Task Scheduler can be used in many ways, not just to schedule tasks for Windows applications, it also allows you to run scripts in the background without having any need to remember them.

The Ultimate Selenium Python Cheat Sheet for Test Automation

Vinayak Sharma — Tue, 20 Sep 2022 14:29:46 +0000

Python is one of the most popular programming languages for Selenium web automation since it provides a simplified syntax and lets you perform more with much less code! Thus, Python and Selenium form an ideal combination to perform web automation testing!

For starters, Selenium is an open-source framework that is primarily used for automating interactions on the WebElements in the AUT (Application Under Test). Along with Python, Selenium also supports Java, C#, JavaScript, Ruby, and PHP. However, as per my experience, it would be fair to mention that Python will be my preferred language for Selenium web automation.

Like me, more & more developers are picking up Python, which is rated as the third-most popular language as per the Stack Overflow Developer Survey 2021. Therefore, a Selenium Python cheat sheet could serve the purpose of providing insights into the useful Selenium Python APIs for realizing automation of websites (or web apps).

If you’re new to Selenium and wondering what it is then we recommend checking out our guide — What is Selenium?

In this blog, we explore the Python package that provides Python bindings for the Selenium WebDriver. For installing the Selenium WebDriver package, we use the Python Package Index (PyPI). Run the following command on the terminal to install Selenium for Python:

$ pip install selenium

This Python library wraps the Selenium WebDriver and provides methods for automating a range of tasks like filling up the form, logging into a website, clicking on buttons, and more. In addition, you can have a look at the Selenium Python tutorial that deep-dives into the integral aspects of Selenium Python from a web automation testing point of view.

The commands mentioned in this Selenium Python cheat sheet can be used as a handy resource for anyone toying with Selenium and Python to automate web applications. If you need a quick recap of Python with Selenium, check out the tutorial that deep dives into the Selenium WebDriver architecture and highlights integral aspects related to Selenium WebDriver with Python. Let’s kick start our Selenium cheat sheet with Python!

1. Import the Selenium library

Before you can use any Selenium Python commands, you need to import the Selenium WebDriver package.

from selenium import webdriver

2. Driver Initialization with Python

After downloading the corresponding browser driver, you need to start the Selenium WebDriver and browser driver (e.g., ChromeDriver for Chrome, GeckoDriver for Firefox, etc.).

For Chrome

Initialize Chrome WebDriver

driver = webdriver.Chrome()
For Firefox

Initialize Firefox/Gecko WebDriver

driver = webdriver.Firefox()
For Safari

Initialize Safari WebDriver

driver = webdriver.Safari()
For Internet Explorer

Initialize IE WebDriver

driver = webdriver.Ie()

In case the location of the browser driver is not added to the PATH variable (or if it is not in the System Path), you need to add the following arguments:

executable_path: Path to your Selenium WebDriver (binary file)
options: Options regarding the web browsers execution

Example:

driver = webdriver.Chrome(executable_path="/usr/bin/chromedriver/", options=chrome_options )

3. Setting Options in Selenium WebDriver

The Options class in Selenium Python is commonly used in conjunction with Desired Capabilities to customize Selenium WebDriver.

It helps to perform various operations like opening the browser(Chrome, Firefox, Safari, IE, Edge, etc.) in maximized mode, enabling and disabling browser extensions, disabling GPU mode, disabling pop-ups, and more. Therefore, it is important to be well-versed with this section of the Selenium Python cheat sheet since it will help solve Python web automation-specific problems that involve changing browser properties that we mentioned earlier.

For Chrome

a. Importing Chrome options

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

b. Initialization of Chrome options

chrome_options = Options()

c. Adding Desired Capabilities

chrome_options.add_argument("--disable-extensions")

d. Adding Desired Capabilities to a session

driver = webdriver.Chrome(chrome_options=chrome_options)

For Firefox

a. Importing Firefox options

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

b. Initialization of Firefox options

firefox_options = Options()

c. Adding Desired Capabilities

firefox_options.set_headless()

d. Adding Desired Capabilities to session

driver = webdriver.Firefox(options=firefox_options)

4. Finding an element

Locators in Selenium are majorly used for locating WebElements present in the DOM. Appropriate interactions (or actions) are further performed on the located WebElements. Some popular Selenium web locators are ID, Name, Link Text, Partial Link Text, CSS Selectors, XPath, TagName, etc.

Test on Latest Safari Desktop and Mobile Browsers for Cross Browser Compatibility. Test on real Mac machines running real Safari browser online

Locate Elements by the ID attribute

In this method, the element in the DOM is searched using the ID attribute. ID is unique for every element on the page. Thus, an ID can uniquely identify an element. For example, shown below is the use of the ID attribute for locating WebElements on the LambdaTest login page:

Here is how you can use the ID attribute in Selenium Python:

from selenium import webdriver

driver = webdriver.Chrome()
driver.get("[https://www.lambdatest.com](https://www.lambdatest.com)")
email_form = driver.find_element_by_id("testing_form")

Locate Elements by CSS Class

Elements in HTML DOM can also be searched by Class Name, which is stored in the Class attribute of an HTML tag. A class can have many instances; it returns the first element with a matching class.

Here is how Class Name is used for locating the Email Address element on the LambdaTest page:

from selenium import webdriver

driver = webdriver.Chrome()
driver.get(“https://www.lambdatest.com")
first_form_input = driver.find_element_by_class_name(“ form-control “)

Locate Elements by Name

WebElements like input tag have a Name attribute associated with them. Selenium also provides a method to search for WebElements using the NAME attribute. If there are multiple elements of the same name, the first matched element is returned by the method.

Here is the HTML code that contains an input element of Name — name!

<input name="name" type="text" value="user name" />

Shown below is the usage of the Selenium Python method for locating the WebElement using the NAME property:

from selenium import webdriver

driver = webdriver.Chrome()
driver.get("[https://www.lambdatest.com](https://www.lambdatest.com)")
# for selection input with name attribute "name"
name_input = driver.find_element_by_name("name")

Locate Elements by XPath

XPath uses path expressions to select nodes and locate the required WebElement. The find_element_by_xpath() method is used to locate an appropriate element in the document using XPath. You can read the Selenium XPath tutorial to gain deeper insights into using XPath for locating WebElements for Selenium web automation.

Here is how the email element is located using the XPath attribute

from selenium import webdriver

driver = webdriver.Chrome()
driver.get("[https://www.lambdatest.com](https://www.lambdatest.com)")
email_input = driver.find_element_by_xpath("//input[[@name](http://twitter.com/name)='email']")

Locate Element by tag name

This method is used to locate and select WebElements using the HTML tag name. The find_element_by_tag_name() method is used to find tags such as H1, DIV, INPUT, etc. If there are multiple occurrences of the same tag, it returns the first matching tag.

Here is how the email address element is located using the Tag Name:

from selenium import webdriver

driver = webdriver.Chrome()
driver.get("[https://www.lambdatest.com](https://www.lambdatest.com)")
email_input = driver.find_element_by_tag_name("input")

Browser & app testing cloud to perform both exploratory and automated testing web browsers, real devices and operating systems.

Locate Element by Link text or Partial Link Text

It selects elements based on the link text (either complete link text or partial link text). Partial link text does not look for an exact match of the string value since it looks for a string subset (in the link text).

Link text locators in Selenium and partial link text locators work only on links of a given web application.

Here is how you can use the link text locator to locate the desired WebElement on the LambdaTest login page:

from selenium import webdriver

driver = webdriver.Chrome()
driver.get("[https://www.lambdatest.com](https://www.lambdatest.com)")
email_input = driver.find_element_by_link_text('Start Free Testing')

Here is how you can use the partial link text locator to locate the desired WebElement on the LambdaTest login page:

from selenium import webdriver

driver = webdriver.Chrome()
driver.get("[https://www.lambdatest.com](https://www.lambdatest.com)")
email_input = driver.find_element_by_partial_link_text('Start Free')

5. Misc methods for finding elements

Two private methods might be useful for locating page elements in conjunction with the “By” class for selecting attributes.

It is to be noted that there is no difference between find_element_by_tag method and find_element(By.tag) method. By default, find_element_by_tag calls the find_element(By.tag) method.

find_element — It returns the first instance from multiple web elements with a particular attribute in the DOM. The method throws NoSuchElementException if no web elements are matching the required web locator. Check out our blog on common Selenium exceptions to gather a deeper understanding of when a particular Selenium exception is raised.
find_elements — It returns a list of all the instances of WebElements matching a particular attribute. The list is empty in case there are no matching elements in the DOM.

Here are the attributes available for the By class:

ID = “id”
XPATH = “xpath”
NAME = “name”
TAG_NAME = “tag name”
CLASS_NAME = “class name”
LINK_TEXT = “link text”
PARTIAL_LINK_TEXT = “partial link text”

Shown below is an example of find_element method that uses the XPath locator to locate the desired WebElement:

from selenium.webdriver.common.by import By
driver.find_element(By.XPATH, '//input[name()="password"]')
driver.find_elements(By.XPATH, '//input')

6. Opening a URL (or document)

Before performing any operation on the WebElements present on the page, opening the target URL (or test URL) is important. Here are some of the ways to open a URL in Selenium Python:

Test your web and mobile apps on online Emulator Android. Ensure your apps are compatible across latest and legacy Android operating systems, devices, and browsers.

driver.get(URL)

The driver.get() method navigates to the page that is passed as a parameter to the method. Selenium WebDriver will wait until the page has fully loaded, post which it fires an “onload” event before returning control to the test script. You can check out our blog on Selenium Waits in Python to further understand handling waits in Selenium Python.

driver.get("[https://www.lambdatest.com](https://www.lambdatest.com)")

7. Refresh a page

There are scenarios where you would want to refresh the contents on the page. The Refresh method of Selenium WebDriver is used for refreshing a web page.

The driver.refresh() method refreshes the current web page. It does not take any arguments nor returns any value.

driver.refresh()

8. Writing text inside a WebElement

The send_keys() method in Python is used for entering text inside a text element. The text to be entered is passed as an argument to the method. The same method can also be used for simulating key presses on any field (e.g. input fields of a form).

Here is an example usage of the send_keys() method where the email address is passed to the text element on the LambdaTest signup page:

from selenium import webdriver

# create webdriver object
driver = webdriver.Chrome()

# get lambdatest
driver.get("[https://www.lambdatest.com/](https://www.lambdatest.com/)")

# get element 
element = driver.find_element_by_id("useremail")

# send keys 
element.send_keys("[emailid@lambdatest.com](mailto:emailid@lambdatest.com)")

9. Clearing text of a WebElement

The element.clear() method in Selenium Python is used to clear text from fields like input fields of a form, etc.

Here is how the clear method is used for clearing contents in the email input box on the LambdaTest home page:

from selenium import webdriver

# create webdriver object
driver = webdriver.Chrome()

# get lambdatest
driver.get("[https://www.lambdatest.com/](https://www.lambdatest.com/)")

# get element 
element = driver.find_element_by_id("useremail")

# send keys 
element.clear()

10. Clicking a WebElement

The element.click() method in Selenium Python is used to click on an element like anchor tag, button tag, etc.

Here is how a button present on the LambdaTest home page is clicked using the click() method:

from selenium import webdriver

# create webdriver object
driver = webdriver.Chrome()

# get lambdatest
driver.get("[https://www.lambdatest.com/](https://www.lambdatest.com/)")

# get element 
element = driver.find_element_by_id("useremail")

# send keys 
element.send_keys("[emailid@lambdatest.com](mailto:emailid@lambdatest.com)")

# get element 
button_element = driver.find_element_by_link_text("Start Free Testing")

# click the element
button_element.click()

11. Dragging and Dropping a WebElement

Dragging & dropping an object is one of the extensively used scenarios in popular apps (or softwares) like Canva, Google Drive, Trello, Asana, etc. The drag_and_drop(element, target) method in Selenium Python helps in automating the functionality of dragging WebElements from the source and dropping them on target area (or element).

Actions class in Selenium has two methods through which you can perform drag & drop operation in browser compatibility testing. Do check out our detailed blog that deep dives into how to perform drag and drop in Selenium Python.

Here is a simple example that shows the sample usage of drag_and_drop() method:

element = driver.find_element_by_name("source")
target = driver.find_element_by_name("target")

from selenium.webdriver import ActionChains
action_chains = ActionChains(driver)
action_chains.drag_and_drop(element, target).perform()

12. Selecting an option

Select(element) provides useful methods for interacting with drop-downs, selecting elements, and more.

Here is an example of how an element is selected using its index:

from selenium.webdriver.support.ui import Select
select = Select(driver.find_element_by_id('city'))
select.select_by_index(index)
select.select_by_visible_text("text")
select.select_by_value(value)

Here are some of the various ways in which desired element is selected using the select_by_*() method:

13. Navigating between windows

If there are multiple windows, you might need to switch to the right window before performing actions on the WebElements present in the DOM.

driver.switch_to_window(“window_name”)

The switch_to_window() method of Selenium WebDriver lets you switch to the desired window. The window handle is passed as an argument to the switch_to_window() method.

driver.switch_to_window("window_handle")

All the subsequent calls of the WebDriver are now applicable to the window under focus (or the newly switched window).

driver.window_handles

Window_handles property of the WebDriver returns handles of the windows. You can now use the switch_to_window() method to navigate to each window available in the list of window_handles.

for handle in driver.window_handles:
    driver.switch_to_window(handle)

driver.current_window_handle

The current_window_handle() method returns the handle of the current window (or window currently under focus)

handler = driver.current_window_handle

14. Switching to iFrames

Selenium WebDriver can not access or locate the web elements inside an iFrame in the context of the main web page. Hence, you need to switch to an iFrame before accessing the WebElements inside the iframe.

driver.switch_to_frame(“frame_name”)

The switch_to_frame() method in Selenium Python lets you switch the context of WebDriver from the context of the main page. We can also access subframes by separating the path and index with a dot.

driver.switch_to_frame("frame_name.0.child")

driver.switch_to_default_content()

This method allows you to switch back to the context of the main page.

driver.switch_to_default_content()

15. Handling pop-ups and alerts

There are three main types of popups & alerts that are commonly used in web applications:

Simple Alert
Confirmation Alert
Prompt Alert

You have the option to switch to the alert, dismiss the alert, or accept the alert. You can check out our detailed tutorial on handling alerts and pop-ups in Selenium. Though the language used is C#, the fundamentals of alerts and pop-ups remain the same!

driver.switch_to.alert

The switch_to.alert property of WebDriver returns the currently open alert object. You can use the object to accept, dismiss, read its contents, or type into the prompt.

alert_obj = driver.switch_to.alert

alert_obj.accept()

Once you have the handle of the alert window (e.g. alert_obj), the accept() method is used to accept the Alert popup.

alert_obj = driver.switch_to.alert 
alert_obj.accept()

alert_obj.dismiss()

Once you have switched to the alert window (e.g. alert_obj), you can use the dismiss() method to cancel the Alert popup.

alert_obj = driver.switch_to.alert 
alert_obj.accept()

alert_obj.text()

This method is used to retrieve the message included in the Alert popup.

alert_obj = driver.switch_to.alert 
msg = alert_obj.text()
print(msg)

16. Getting Page Source

The page_source() method in Selenium WebDriver is used to get the target document’s page source (or test page).

page_source = driver.page_source

17. Navigating browser history

Selenium WebDriver in Python provides some functionalities to move backward and forward in the web browser’s history

driver.forward()

This method allows scripts to navigate one step forward in history.

driver.forward()

driver.back()

This method allows scripts to navigate one step backward in history.

driver.back()

18. Handling Cookies in Selenium

Handling cookies in Selenium WebDriver is one of the common scenarios that you might have to deal with in Selenium web automation. You can perform various operations like add, remove, get cookie name, and more.

driver.add_cookie()

This method helps to set a cookie to a Selenium session. It accepts values in the key-value pair.

# Go to the domain
driver.get("[https://www.lambdatest.com/](https://www.lambdatest.com/)")

# Now set the cookie. 
cookie = {'name' : 'user', 'value' : 'vinayak'}
driver.add_cookie(cookie)

driver.get_cookies()

This method outputs all the available cookies for the current Selenium session.

# Go to the domain
driver.get("[https://www.lambdatest.com/](https://www.lambdatest.com/)")

driver.get_cookies()

driver.delete_cookie()

There is an option to delete a specific cookie or all the cookies associated with the current Selenium session.

# delete one cookie
driver.delete_cookie(cookie)
# delete all cookies
driver.delete_all_cookies()

Watch this video to understand how you can handle Cookies and perform different operations like deleting, getting the parameter values, and adding them to Selenium WebDriver using Java.

19. Setting Window Size

The set_window_size() method is used to set the browser window’s size to desired dimensions (in height and width).

# Setting the window size to 1200 * 800

driver.set_window_size(1200, 800)

Configuring TimeOuts in Selenium WebDriver

When the browser loads a page, the WebElements inside the page may load at various time intervals. This might create complications when interacting with the dynamic elements present on the page.

If an element is not present in the DOM of the web page, the locate method will raise an exception. Waits in Selenium lets you add delays (in ms or seconds) between the actions performed between loading the page and locating the required WebElement.

Implicit wait and Explicit wait are the two major ways you can add delays in Selenium Python code for handling dynamic WebElements on the page.

Implicit Wait in Selenium Python

An implicit wait informs the Selenium WebDriver to examine the DOM for a particular amount of time when trying to find the WebElement that is not immediately available for access.

By default, implicit wait is set as zero. However, once we define implicit wait, it is set for the lifetime of the WebDriver object. Check out our detailed tutorial that demonstrates the usage of Implicit wait in Selenium Python in greater detail.

from selenium import webdriver

driver = webdriver.Chrome()

driver.implicitly_wait(10) # in seconds

driver.get("https://www.lambdatest.com/")

element = driver.find_element_by_id("testing_form")

Explicit Wait in Selenium Python

Explicit wait in Selenium Python is used when we want to wait for a particular condition to happen before proceeding further in the code.

There are some convenient methods provided by the Selenium WebDriver that let you wait until a particular condition is satisfied. For example, explicit waits can be achieved using the webdriverWait class combined with Expected Conditions in Selenium.

Here are some of the Expected Conditions that can be used in conjunction with Explicit wait in Selenium Python:

presence_of_all_elements_located
text_to_be_present_in_element
text_to_be_present_in_element_value
frame_to_be_available_and_switch_to_it
invisibility_of_element_located
title_is
title_contains
presence_of_element_located
visibility_of_element_located
visibility_of
element_located_selection_state_to_be
alert_is_present
element_to_be_clickable
staleness_of
element_to_be_selected
element_located_to_be_selected
Element_selection_state_to_be

Shown below is an example that demonstrates the usage of explicit wait where a non-blocking wait of 10 seconds is performed until the required WebElement is located (using its ID attribute):

from selenium import webdriver

from selenium.webdriver.common.by import By

from selenium.webdriver.support.ui import WebDriverWait

from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()

driver.get("https://www.lambdatest.com/")

try:

    element = WebDriverWait(driver, 10).until(

        EC.presence_of_element_located((By.ID, "testing_form"))

    )

except:

    print(“some error happen !!”)

Capturing Screenshots

During the process of Selenium web automation, you might want to capture the screenshot of the entire page or screenshot of a particular WebElement.

This is specifically used in scenarios where you want to check what went wrong in the test execution. Capture screenshots of WebElement using Selenium Python when you want to check which particular WebElement has created issues in the test execution process.

The save_screenshot() method of Selenium WebDriver is used for capturing screenshots of a web page in Python.

capture_path = 'C:/capture/your_desired_filename.png'

driver.save_screenshot(capture_path)

Conclusion

Get started with Selenium Python testing & that too free!!! Python is one of the most popular languages, and there is no denial that you can run complex Selenium operations with a few lines of code. In this Selenium Python cheat sheet, we covered some of the widely used Selenium Python commands primarily used for cross browser compatibility testing.

This Selenium cheat sheet can be used as a guide (or reference) for quickly referring to the commands that might be of interest for your test code. I hope that you find this Selenium Python cheat sheet useful, do let me know if you come across any Selenium Python command that should be a part of the sheet.

Happy Automation Testing With Python!

Most Exhaustive XPath Locators Cheat Sheet

Vinayak Sharma — Mon, 19 Sep 2022 14:07:20 +0000

The Selenium framework lets you interact with the WebElements in the DOM. For realizing the interaction(s), it is important to choose the appropriate locator from the available Selenium web locators. As per my opinion, Selenium web locators can be considered as the backbone of any web automation script.

There are many options for web locators in Selenium — ID, Name, XPath, LinkText, Partial LinkText, Tag Name, and more. XPath is one of the widely preferred web locators, as it easily traverses through the DOM elements and attributes to identify an object.

If you plan to use an XPath web locator to shape your test automation scripts, you might not have to look further than this comprehensive XPath cheat sheet. In addition, you can keep this XPath cheat sheet handy in scenarios where you want to have a quick look at the XPath syntax or other aspects related to this web locator.

In this Selenium locators cheat sheet, we have demonstrated the XPath locator in the Java language. So, let’s get started with this exhaustive XPath cheat sheet that will come in super handy in Selenium automation testing.

What is XPath Web Locator in Selenium

XPath, which stands for XML Path Language, uses ‘path-like’ syntax to identify and navigate nodes in an XML or HTML document. It follows the XSLT(eXtensible Stylesheet Language Transformations) standard, which is predominantly used to navigate WebElements and attributes.

In order to navigate through the HTML document, the DOM (HTML tags, JavaScript, etc.) is used as a map where all the WebElements exist along with the WebElement that we intend to locate using the appropriate web locator.

You can check out our blog on locators in Selenium WebDriver to check other web locators that we have not covered in this XPath cheat sheet.

This article explains the emulator vs simulator vs real device differences, the learning of which can help you select the right mobile testing solution for your business.

Different types of XPath in Selenium

You can locate the desired WebElement in the DOM, either using the absolute path or using a relative path with respect to another element. In this part of the XPath cheat sheet, we look at the different ways in which you can use an XPath locator for locating a WebElement.

Here are the two main types of XPath in Selenium:

Absolute XPath

As the name indicates, an absolute XPath contains the complete path from the root element to the desired element. The downside of using absolute XPath is that any changes made in the path of the element, HTML path, etc., results in the failure of the XPath.

Absolute Xpath begins with the single forward-slash(/), which means selecting the element from the document’s root node.

Syntax — Absolute Xpath

/html/body/div[x]/div[y]/

Relative XPath

In contrast to absolute XPath, a relative XPath starts by referencing the element we want to locate relative to a particular location. This means that the element is positioned relative to its normal position.

Since the path of XPath is relative, it starts with a double forward-slash (//). As mentioned earlier in this Selenium locators cheat sheet, relative XPath is usually preferred for Selenium automation testing since it is not a complete path from the root element.

Hence, any changes in the page design or the DOM hierarchy will have a minimal (to no) impact on the existing XPath selector.

Syntax — Relative Xpath

//tagname[[@attribute](http://twitter.com/attribute)='value']

The XPath syntax

XPath uses path expressions to select nodes (or node-sets) in an HTML document. As shown below, the element or tag is selected by following a path or series of steps:

Here are some of the useful path expressions that you can keep in handy as a part of the XPath cheat sheet:

Perform cross browser compatibility testing across all Safari browsers for Windows, and many more real browsers and operating systems.

XPath Expressions in Selenium

XPath expression is a pattern that is used for selecting a set of nodes. These patterns are used by XSLT for performing transformations. XPointer also uses these patterns for addressing purposes.

XPath uses a path expression to select the node or a list of nodes from an XML or HTML document.

Syntax — XPath Expression

//tagname[[@attribute](http://twitter.com/attribute)='value']

Prefixes

Prefixes in XPath are used at the beginning of the XPath expression.

Here are the examples of different kinds of Steps in XPath:

Selecting Nodes

In this part of the XPath cheat sheet, we look at the different ways to select a node (or WebElement) in the document using XPath.

Predicates

Predicates in XPath are used to find a specific node that contains a designated value. It uses syntax enclosed in square brackets.

The table shown below in this Selenium locators cheat sheet shows some examples of predicates, along with different ways of selecting WebElements using those predicates.

Chaining order

Chaining order in XPath indicates the importance of order in an Xpath expression. The meaning of XPath changes with the change in order. For example a[1][@href=’/’] and a[@href=’/’][1] are different.

Indexing

Indexing in XPath can be done using [ ] with a number that specifies the node we intend to select. It can also be done using functions like last() or position() that specify the index of the elements.

Nesting predicates

XPath can also be nested to find the targeted node (or WebElement).

Example — Nested Predicates

//section[.//a[[@id](http://twitter.com/id)='btn']]

This example will return

if it has an anchor tag descendant with an ID value of ‘btn.’

Selecting Unknown Nodes

Wildcards can also be used with an XPath locator to find unknown HTML document elements.

Selecting Several Paths

Using the ‘|’ operator in the XPath expression, you can select several paths.

Below in this Selenium locators cheat sheet are examples that demonstrate the usage of the ‘|’ operator.

Browser test & app testing cloud to perform both exploratory and automated testing across 3000+ different browsers, real devices and operating systems.

XPath Selectors in Selenium

Selectors are used to ‘select’ certain parts of the HTML document specified by the element’s XPath. It’s the same as Axes in XPath.

Descendant selectors

The descendant selectors indicate all the children of the current node, all children of children, etc. Attribute and namespace nodes are not included in descendant selectors. The parent of an attribute node is its element node, and on the other hand, attribute nodes are not the offspring of their parents.

Attribute Selectors

The XPath attribute selector matches elements based on the presence or value of a given attribute.

Siblings

Sibling in Selenium WebDriver is used for fetching a WebElement that is a sibling to a parent element. In this part of the XPath cheat sheet, we look at how to fetch elements using siblings in Selenium WebDriver.

Using different Operators

You can use different operators like NOT, text matches (i.e., string, substring, etc.), union, and more to locate elements using operators and XPath in Selenium.

Axes in XPath

Axes in XPath represent a relationship from the current element to its relative element on the tree using either the absolute XPath or relative XPath. Its relationship to elements is identified as a parent, child, sibling, etc.

They are named Axes because they refer to the axis on which elements lie relative to a particular WebElement. Shown below is the syntax of Axes in XPath:

axisname::element[attribute]

This part of the Selenium locators cheat sheet shows the axis names and their relationship with the elements.

Element	XPath
self	For referring current elements
parent	For referring parent of the current node
preceding	Selects all elements that appear before the current element in the document
preceding-sibling	Selects all siblings before the current elements
namespace	Selects all namespace elements of the current element
ancestor	Selects all ancestors with a relationship like a parent, grandparent, etc. relative to the current element
ancestor-or-self	Selects the current element as well as all the ancestors of that element
attribute	Selects all attributes of the current node
child	Selects all children of the current element
descendant	Selects all children, grandchildren, etc. of the current element
descendant-or-self	Selects all descendants of the current elements and the current element itself
following	Selecting everything in the document after the closing tag
following-sibling	Selecting all siblings after the current element

Here are some of the ways in which siblings in XPath can be used to find WebElements for Selenium automation testing.

Operators in XPath

An XPath expression can return either a node-set(div,a,li), a string, a Boolean (true or false), or a number. As a result, XPath provides many operators to manipulate their return values so that you get the desired value.

In this part of XPath cheat sheet, we look at the list of operators that can be used with XPath expressions:

Functions in XPath

XPath provides a number of core libraries that bring in functionalities to deal with node sets, strings, booleans, and numbers.

Syntax Demonstration

button[text()="Submit"]
    //button/text()
    //ol/li[position()=2]

Node functions

There are seven kinds of nodes in XPath — element, attributes, namespaces, comments, document nodes, text, and processing instructions. First, let’s look at the permitted operations on nodes in further detail in this section of the XPath cheat sheet.

name()

This function provides the name of the element

Example:

//[starts-with(name(), 'h')]

text()

This function Selects a text node.

Example:

//button[text()="Submit"]
    //button/text()

position()

This function provides the position of the element.

/div/a[position()<3]

last()

This function selects the last node relative to the current element.

Example:

/div/a[last()]

comment()

This function selects the elements, which are comments.

Example:

/div/comment()

String functions

XPath string functions let you access elements using string type functions (e.g., contains, starts-with, etc.)

contains()

This function is used to see if a certain element contains a particular string

Example:

font[contains([@class](http://twitter.com/class),"nav")]

starts-with()

This function is used to check if our element starts with a certain text.

Example:

font[starts-with([@class](http://twitter.com/class),"nav")]

ends-with()

Same as starts-with(), look (or check) at the end of the string.

Example:

font[ends-with([@class](http://twitter.com/class),"nav")]

substring-before()

This function returns a string that is part of the input string before a given substring.

Example:

substring-before("01/02", "/")

Output: 01

substring-after()

Same as substring-before(), returns a string that is part of the input string after a given substring.

Example:

substring-after("01/02", "/")

Math functions

You can use mathematics-type functions like floor, ceil, sum, etc., to find elements using the XPath selector.

Here are the math functions that we have covered in our XPath cheat sheet:

ceiling(number)

This function returns the smallest integer value greater than or equal to the decimal number.

floor(number)

This function returns the largest integer value less than or equal to the decimal number.

round(decimal)

This function returns a number that is the nearest integer to the given number.

sum(elements)

This function returns a number that is the sum of the numeric values of each element in a given element-set.

Boolean functions

Boolean operators can also be used to find elements using the XPath selector.

not()

This function is used to check whether certain conditions are met or not.

Example:

button[not(starts-with(text(),"Submit"))]

Browser console to locate XPath

Most modern web browsers (e.g., Chrome, Firefox, etc.) have a built-in debugging tool, which includes a feature that allows us to evaluate or validate XPath/CSS selectors. These tokens execute in the console panel, thereby providing you an option to validate the XPath selector.

Here is a screenshot of the Debug Console in Chrome:

In this example, we have used $x(“//div”). With this, it selects all div elements from the DOM. An exception is thrown in case there is no matching node or elements in the DOM.

Obtain XPath using JQuery

JQuery supports all basic kinds of XPath expressions; major ones are listed below in this part of the XPath cheat sheet:

Bonus Examples

Conclusion

XPath is one of the widely used web locators in Selenium. However, coming up with an efficient XPath is an art that largely depends on the document’s structure and the current position of the WebElement.

One of the most important things is finding the nearest unique WebElement relative to the target element and using the best technique to locate the WebElement. We covered the major aspects of the XPath locator in this exhaustive XPath cheat sheet.

This Selenium locators cheat sheet can be used as a handy resource when you are performing Selenium automation testing.

When it comes to Selenium automation testing, you can leverage the capabilities offered by cloud-based Selenium Grid like LambdaTest which lets you run tests on different browsers, platforms, and device combinations.

Do share your views on how you would use this XPath cheat sheet for Selenium automation testing. Leave your feedback in the comments section…

Happy Testing!

Tutorial on Python Read Config File Using Selenium [With Example]

Vinayak Sharma — Thu, 08 Sep 2022 14:32:34 +0000

Before starting with this tutorial on Python read config file let’s understand Selenium. Selenium is one of the best automation tools used for running automated tests through which you can perform relevant actions with the WebElements on the page. The major advantage of Selenium is that it supports a number of popular programming languages (e.g., Python, Java, JavaScript, C#, PHP, and Ruby).

When working with Selenium WebDriver, we need to follow the best practices to make the most out of the Selenium automation framework. One of them could be adding useful parameters like platform, browsers, browser versions, or device names in a manner that scaling up, organizing, and debugging the tests becomes easy. In addition, it should involve minimal changes in the test implementation! This is where configuration files come in handy, as configuration data can be added to those files with super ease. This, in turn, helps in improved maintenance of the test code.

In this Selenium Python tutorial, we deep dive into the python read config file using Selenium.

How to Create Configuration Files With Constant Parameters Using Python

Configuration files usually contain information that is organized in key:value pairs, where value is recognized by key. Configuration files usually have extensions like .ini, .cnf, .conf, .cfg, and .cf. There are several types of modules and techniques to create and parse configuration files in Python.

Here are some of the commonly used configuration files in Python:

Python Configuration Files
JSON Configuration Files
YAML Configuration Files
INI Configuration Files
XML Configuration Files

Run at scale automation tests on our online Appium grid or debug your web and mobile apps manually on the most powerful Android Online Emulator

How to Create python read config file

Though Python files are not used to store configuration information, the best part is that storing configuration in Python files is a simple task. The major advantage of this approach is the ease in the management of configuration variables . For example, to store these configurations, we can create a config.py file.

drivers_config = {
    "URL": "localhost.com",
    "FileName": "name.json",
}

# implementation comes here ...

You can use this config file in the actual implementation as shown below:

import config as cfg

driver = webdriver.Firefox()
driver.get(cfg.drivers_config["URL"])

As seen above, we simply imported the config file in the code before using the configuration parameters present in it. Now that we covered on python read config file, we now move to create JSON config files using the Python JSON module.

How to Create JSON configuration file

JSON (JavaScript Object Notation) is a lightweight data-interchange format. It provides information in a human-readable format, making it easy for machines to parse the information stored in the JSON file. Shown below is an example that demonstrates how to use Python’s JSON module for creating a config.json file.

{
    "drivers_config":{
        "URL":"localhost.com",
        "FileName" : "file.json",
      },
}

You can use this config file in the actual implementation as demonstrated below:

import json

# reading config file
file = open("config.json",)
data = json.load(file)

# using config file
print(data["drivers_config"]["URL"])

How to create YAML configuration file

YAML (YAML Ain’t Markup Language) is a human-readable data serialization language. In Python, we use the YAML module to parse the YAML file.

drivers_config:
 URL: localhost.com
 FileName: file.json
other:
     Variable: “data “

Here is how you can read the YAML config file in Python:

import yaml

file = open("config.yml", "r") 
cfg = yaml.load(file)



print(cfg["drivers_config"]["URL"])
# this will print "localhost.com"

print(cfg["drivers_config"])
# this will print

# drivers_config
# {
#     “URL” : “localhost.com”   
#     “NameFile”: “file.json”
# }

Use android emulator for iOS to make sure your app works on mobile devices with different operating systems before releasing it.

How to create INI configuration file

INI configuration file is a text-based structured syntax file that comprises key-value pairs.

[drivers_config]
URL=localhost.com
FileName=file.json

[other]
# other variables

To read this config file in Python, we will use the ConfigParser module. Run the below command on the terminal to install the ConfigParser module:

$ pip install configparser

Here is how you can parse the INI configuration file in Python:

import configparser

file = open("config.ini","r")

config = configparser.RawConfigParser(allow_no_value=True)
config.read_file(file)

print( config.get("drivers_config","URL") )
# output will be localhost.com

How to create XML configuration file

XML (Extensible Markup Language) lets you define rules for encoding documents that are easy to understand by humans and machines. The beautifulsoup module in Python is used for leveraging the advantages of XML configuration files.

Run the below command on the terminal to install the beautifulsoup module:

$ pip install bs4



<config>
    <drivers_config>
        <URL>localhost.com</URL>
        <FileName>file.json</FileName>
    </drivers_config>
</config>

To parse this file, we will use the bs4 or beautifulsoup module in the implementation. Once parsed, the Python file will look like this:

from bs4 import BeautifulSoup as bs

file = open('config.xml','r')
# parsing xml file
data = bs(file)

print(data.drivers_config.url.text)

Each parsing technique has its share of advantages & disadvantages. Hence, it is recommended to choose a technique that meets the project's needs and requirements.

In the remaining part of the Python read config file tutorial, we would use the INI configuration file since INI is the widely preferred configuration file format by Python developers.

Writing Selenium scripts for testing “add” functionality on a cloud Selenium Grid

So far in this tutorial on Python read config file, we have covered the essentials of python read config file. So, how can you leverage the benefits offered by configuration files in Python to make Selenium tests more readable and maintainable? Well, first, you need to move to a scalable and reliable cloud-based Selenium Grid so that you can run Selenium tests across different browsers and platform combinations.

LambdaTest is a cloud-based platform that enables you to perform live, interactive, and automated cross browser testing across 2000+ browsers and operating systems online. There are umpteen benefits of cloud testing, something that cannot be achieved by running Selenium tests on a local test infrastructure. Porting existing Python code to the LambdaTest Selenium Grid involves minimal changes, making it easy for QA engineers to leverage the platform in the shortest possible time.

For demonstrating Python config file usage with an example, we would use the LambdaTest ToDo App on which the relevant tests are performed. The desired browser capabilities are generated using the LambdaTest capabilities generator.

Here is the set of operations performed on the LambdaTest Sample ToDo App:

Mark the first item and second items as complete. For this, we select the corresponding element using the Name locator (i.e., li1 and li2). You can check out our learning hub on Selenium Locators to ensure that you follow the best practices when using the appropriate web locators.

check_box_one = driver.find_element_by_name("li1")
check_box_one.click()

check_box_two = driver.find_element_by_name("li2")
check_box_two.click()
1. Find the text element using ID and enter the details of the new item using SendKeys in Selenium.
Enter item in textfield

textfield = driver.find_element_by_id("sampletodotext")
textfield.send_keys("text we need to add")

Click on add button

add_button = driver.find_element_by_id("addbutton")
add_button.click()
1. After typing values in the text box, we will click the add button.
2. For taking a screenshot, we will use the save_screenshot method provided by Selenium. You can read our in-depth blog that covers taking screenshots using Python and Selenium.
saving screenshot

driver.save_screenshot(filename.png)
1. Now comes the best part — creating an INI configuration file for storing the essential configuration parameters:) Create a configuration file named config.ini. This is the complete project structure:

Now, create a config.ini file with the configurations that would be used in the test. The username and access key for using the LambdaTest Selenium Grid are available in the LambdaTest Profile Section.

[USER_INFO]
LT_USERNAME=YOUR_LT_USERNAME
LT_ACCESS_KEY=YOUR_LT_ACCESS_KEY

[SCREENSHOT_FILE_INFO]
FILENAME=filename.png

You could either pass this configuration file as a Command-Line argument or directly define it in the code.

We will see both methods for parsing the configuration file:

Pre-defining configuration file details in the implementation
Passing config file name as a command-line argument

Though PyUnit is the default test automation framework in Python, QA engineers prefer PyTest for Selenium automation testing. To get started with PyTest, check out the below video from LambdaTest YouTube Channel.

Predefined configuration file name in Python implementation

In our case, we have named the Python configuration file as config.ini. This file will be parsed to read the relevant configuration information for usage in the code.

Here is how we open and read contents from the config file:

## getting config
with open("config.ini","r") as file:
    config = configparser.RawConfigParser(allow_no_value=True)
    config.read_file(file)

Here is the entire implementation where we read the LT_USERNAME and LT_ACCESS_KEY from the configuration file. The read information is further used in the code for accessing the LambdaTest Grid. We have the PyUnit framework for the entire demonstration. Please refer to our PyUnit Framework Tutorial to get more information on the said framework.

import configparser
import unittest
from selenium import webdriver

# getting config
with open("config.ini", "r") as file:
    config = configparser.RawConfigParser(allow_no_value=True)
    config.read_file(file)

username = config.get("USER_INFO", "LT_USERNAME")
access_key = config.get("USER_INFO", "LT_ACCESS_KEY")

class FirstSampleTest(unittest.TestCase):
    # Generate capabilities here [https://www.lambdatest.com/capabilities-generator/](https://www.lambdatest.com/capabilities-generator/)
    # setUp runs before each test case and
    def setUp(self):
        desired_caps = {
            "build": config.get("CAPABILITIES", "build"),
            "name": config.get("CAPABILITIES", "name"),
            "platform": config.get("CAPABILITIES", "platform"),
            "browserName": config.get("CAPABILITIES", "browserName"),
            "version": config.get("CAPABILITIES", "version"),
            "selenium_version": config.get("CAPABILITIES", "selenium_version"),
            "visual": config.get("CAPABILITIES", "visual"),
            "geoLocation": config.get("CAPABILITIES", "geoLocation"),
            "chrome.driver": config.get("CAPABILITIES", "chrome_driver"),
        }
      """
       Setup remote driver
            -------
            username and access_key can be found on lt platform

"""
        self.driver = webdriver.Remote(
            command_executor="[@hub](https://{}:{}<a href=).lambdatest.com/wd/hub">https://{}:{}[@hub](http://twitter.com/hub).lambdatest.com/wd/hub".format(
                username, access_key
            ),
            desired_capabilities=desired_caps,
        )

# tearDown runs after each test case
    def tearDown(self):
        self.driver.quit()

# """ You can write the test cases here """
    def test_unit_user_should_able_to_add_item(self):
        # try:
        driver = self.driver

# Url
        driver.get("[https://lambdatest.github.io/sample-todo-app/](https://lambdatest.github.io/sample-todo-app/)")

# Click on checkbox
        check_box_one = driver.find_element_by_name("li1")
        check_box_one.click()

# Click on checkbox
        check_box_two = driver.find_element_by_name("li2")
        check_box_two.click()

# Enter item in textfield
        textfield = driver.find_element_by_id("sampletodotext")
        textfield.send_keys(config.get("ITEM_TO_ADD", "ITEM_VALUE"))

# Click on add button
        add_button = driver.find_element_by_id("addbutton")
        add_button.click()

# navigation to list to find the username to added item
        custom_xpath = (
            "//*[contains(text()," + config.get("ITEM_TO_ADD", "ITEM_VALUE") + ")]"
        )
        added_item = driver.find_element_by_xpath(custom_xpath).text

# saving screenshot
        driver.save_screenshot(config.get("SCREENSHOT_FILE_INFO", "FILENAME"))

if __name__ == "__main__":
    unittest.main()

Take a look at the aspects of simulation and explore various ways to use iOS Simulator on PC

Pass configuration file name as a Command-Line argument

For accepting the command-line argument(s), we will use the sys module in Python to pass the configuration name.

Here are some of the Python modules that can be used for passing command-line argument(s):

Using sys module
Using argparse module
Using optparse module and more

For demonstration, we have used the sys module to read the command-line argument(s):

import configparser
import unittest
import sys
from selenium import webdriver

class sampleTest(unittest.TestCase):
    # default file name in case no arg passed
    config_file = "config.ini"

def setUp(self):
        self.config = configparser.RawConfigParser(allow_no_value=True)
        self.config.read(self.config_file)
        username = self.config.get("USER_INFO", "LT_USERNAME")
        access_key = self.config.get("USER_INFO", "LT_ACCESS_KEY")

## get CAPABILITIES from [https://www.lambdatest.com/capabilities-generator/](https://www.lambdatest.com/capabilities-generator/)

desired_caps = {
            "build": self.config.get("CAPABILITIES", "build"),
            "name": self.config.get("CAPABILITIES", "name"),
            "platform": self.config.get("CAPABILITIES", "platform"),
            "browserName": self.config.get("CAPABILITIES", "browserName"),
            "version": self.config.get("CAPABILITIES", "version"),
            "selenium_version": self.config.get("CAPABILITIES", "selenium_version"),
            "visual": self.config.get("CAPABILITIES", "visual"),
            "geoLocation": self.config.get("CAPABILITIES", "geoLocation"),
            "chrome.driver": self.config.get("CAPABILITIES", "chrome_driver"),
        }

"""
            Setup remote driver
            -------
            username and access_key can be found on lt platform

"""
        self.driver = webdriver.Remote(
            command_executor="[@hub](https://{}:{}<a href=).lambdatest.com/wd/hub">https://{}:{}[@hub](http://twitter.com/hub).lambdatest.com/wd/hub".format(
                username, access_key
            ),
            desired_capabilities=desired_caps,
        )

# tearDown runs after each test case
    def tearDown(self):
        self.driver.quit()

# """ You can write the test cases here """
    def test_unit_user_should_able_to_add_item(self):
        # try:
        driver = self.driver

# Url
        driver.get("[https://lambdatest.github.io/sample-todo-app/](https://lambdatest.github.io/sample-todo-app/)")

# Click on checkbox
        check_box_one = driver.find_element_by_name("li1")
        check_box_one.click()

# Click on checkbox
        check_box_two = driver.find_element_by_name("li2")
        check_box_two.click()

# Enter item in textfield
        textfield = driver.find_element_by_id("sampletodotext")
        textfield.send_keys(self.config.get("ITEM_TO_ADD", "ITEM_VALUE"))

# Click on add button
        add_button = driver.find_element_by_id("addbutton")
        add_button.click()

# navigation to list to find added item
        custom_xpath = (
            "//*[contains(text()," + self.config.get("ITEM_TO_ADD", "ITEM_VALUE") + ")]"
        )
        added_item = driver.find_element_by_xpath(custom_xpath).text

# saving screenshot
        driver.save_screenshot(self.config.get("SCREENSHOT_FILE_INFO", "FILENAME"))

if __name__ == "__main__":
    if len(sys.argv) > 1:
        sampleTest.config_file = sys.argv.pop()
    unittest.main()

Trigger the following command on the terminal to read the command-line arguments:

$ python -m single_test.py config.ini

Visit the Automation Dashboard on LambdaTest to get further information about the execution of tests on the cloud grid:

Conclusion

In this Selenium Java tutorial, we have learnt about python read config file using the common configuration file formats like .cnf, .conf, .cfg, .cf, and .ini. We have also seen how multiple configuration files are read together, and their results can be merged into a single configuration using ConfigParser.

We then performed Selenium automation testing to test the “add” functionality in the to-do list app and take a screenshot to verify our results using a cloud Selenium Grid.

How To Perform Web Scraping with Python and Selenium

Vinayak Sharma — Wed, 07 Sep 2022 18:09:20 +0000

As per Stack Overflow Survey 2020, Python holds the fourth position in the preferred programming languages category. Moreover, it is supported across a wide range of test automation frameworks, including the Selenium framework. There web scraping with python and selenium is one of the most used combinations for smarter data collection and intelligent analysis.

‘Data is the new oil,’ the ever-green quote by Humbly becomes much more relevant if the right methods are used for making the most out of the data. There is a plethora of information (read data) available on the internet, and acting on the right set of data can reap significant business benefits. Putting the right data collection methods in implementation can bring useful insights. On the other hand, incorrect data collection methods can result in obtuse data.

Web scraping, surveys, questionnaires, focus groups, oral histories, etc., are some of the widely used mechanisms for gathering data that matters! Out of all the data collection methods, web scraping is considered the most reliable and efficient data collection method. For starters, web scraping (also termed web data extraction) is an automatic method for scraping (or obtaining) large amounts of data from websites. Selenium, the popular test automation framework, can be extensively used for scraping web pages. You can also learn more about what is Selenium? In this Selenium Python tutorial, we look at web scraping using Selenium and Python.

We have chosen Python — the popular backend programming language for demonstrating web page scraping. Along with scraping information from static web pages, we will also look into web scraping of dynamic pages using python and selenium.

What is Web Scraping

Web Scraping, also known as “Crawling” or “Spidering,” is a technique for web harvesting, which means collecting or extracting data from websites. Here, we use bots to extract content from HTML pages and store it in a database (or CSV file or some other file format). Scraper bot can be used to replicate entire website content, owing to which many varieties of digital businesses have been built around data harvesting and collection.

Web scraping with Python can help us extract an enormous volume of data about customers, products, people, stock markets, etc. The data has to be put to ‘optimal use’ for the betterment of the service. There is a debate whether web scraping is legal or not, the fact is that web scraping can be used for realizing legitimate use cases.

Test on Latest Safari browser online and Mobile Browsers for Cross Browser Compatibility.

Use Cases for Web Scraping with Python

Here are some of the valid (or authorized) use cases of web scraping in Python (and other Selenium-supported programming languages):

Search Engines: Several Search bot engines crawl billions of websites and analyze their content for gathering meaningful search results. Search engine crawling is often called spidering. Spiders navigate through the web by downloading web pages and following links on these pages to find new pages available for their users. Then, they rank them according to different factors like keywords, content uniqueness, page freshness, and user engagement.
E-commerce (Price Comparison): Price comparison websites use bots to fetch price rates from different e-commerce websites. Web scraping with Python is a reliable and efficient method of getting product data from target e-commerce sites according to your requirements. They acquire data by either building in-house web scraping methodologies or employing a DaaS (Data As A Service) provider that’ll provide the requisite data.
Sentiment Analysis: Market research companies use web scraping with python for sentiment analysis. This kind of analysis helps companies gain customer insights, along with helping them understand how their customers behave to particular brands and products.
Job Postings: Job listings for details about job openings and interviews are scraped from a collection of websites. The scraped information is then listed in one place so that it is seamlessly accessible to the users.

Legality of Web Scraping with Python?

This is a debatable topic since it entirely depends on the intent of web scraping and the target website from which the data is being scraped. Some websites allow web scraping while several don’t. To see whether a website permits web scraping or not, we have to look at the website’s “robots.txt” file. We can find this file by adding “/robots.txt” at the end of the URL that you want to scrape.

For example, if we want to scrape the LambdaTest website, we have to see the “robots.txt” file, which is at the URL https://www.lambdatest.com/robots.txt

User-agent: *
Allow: /

Sitemap: [https://www.lambdatest.com/sitemap.xml](https://www.lambdatest.com/sitemap.xml)

Browser & app testing cloud to perform both exploratory and automated testing across 3000+ different browsers, real devices and operating systems.

Web Scraping with Python

Getting started with web scraping in Python is easy since it provides tons of modules that ease the process of scraping websites. Here are some of the modules that you should be aware of to realize web scraping with Python:

Requests Library for Web Scraping

The requests library is used for making several types of HTTP requests like getting GET, POST, PUT, etc. Because of its simplicity and efficiency of use, it has a motto of “HTTP for Humans.”

But, we can’t directly parse HTML using the requests library. It uses the lxml library to parse HTML.

$ pip install requests

Beautiful Soup Library for Web Scraping

BeautifulSoup Library is one of the widely-used Python libraries for web scraping. It works by creating a parse tree for parsing HTML and XML documents. Beautiful Soup automatically transforms incoming documents to Unicode and outgoing documents to UTF-8. It uses a custom parser to implement idiomatic navigation methods to search and transform the parse tree.

$ pip install beautifulsoup4

Scrapy Framework for Web Scraping

Scrapy is a web scraping framework created by Pablo Hoffman and Shane Evans, co-founders of Scrapinghub. It is a full-fledged web scraping tool that does all the heavy lifting and provides spider bots to crawl various websites and extract the data. With Scrapy, we can create spider bots, host them on Scrapy Hub, or use their APIs. It allows us to develop fully functional spiders in a few minutes. We can also add pipelines to process and store data.

Scrapy allows making the asynchronous request, which means it makes multiple HTTP requests simultaneously. This method saves a lot of time and increases our efficiency of scraping.

$ pip install Scrapy

Note: To further ease down the process of writing small tests, Python offers various tools and frameworks. Whether you are a Python beginner or an experienced programmer, pytest helps you write the tests you need and have them run in a reliable manner. For a quick overview on getting started with pytest, check out the video below from LambdaTest YouTube Channel.

Static and Dynamic Web Scraping with Python and Selenium

There is a difference between static web pages and dynamic web pages. In a static web page, the content remains the same until someone changes them manually. On the other hand, the dynamic web page content of the page can differ for different visitors (e.g., content can change as per the geolocation, user profile, etc.). This increases its time complexity as dynamic web pages can render at the client-side, unlike static web pages, which render at the server-side.

The static web page content or HTML documents are downloaded locally, and data can be scraped using relevant scripts. On the other hand, dynamic web page content (or data) is generated uniquely for every request after the initial page load request. For that case, we need to perform the following several actions using the manual approach:

For this purpose, we need to automate websites, the same can be achieved using Selenium WebDriver.

In this article, look at how to debug websites using dev tools in Safari.

Scraping Dynamic Web Page using Python and Selenium

Here are the prerequisites for realizing web scraping with Python and Selenium:

Beautifulsoup for scraping HTML content for websites:

$ pip install beautifulsoup4

Parsing HTML content of websites:

$ pip install lxml

Selenium for automation:
Installing Selenium using pip

$ pip install selenium

Install Selenium using conda

$ conda install -c conda-forge selenium

Importing modules for web scraping with Python and Selenium

For demonstration, we would be using the LambdaTest Grid. Cloud-based Selenium Grid on LambdaTest lets you run Selenium automation tests on 2,000+ browsers and operating systems online. You can perform parallel testing at scale using the cloud-based Grid. Once you create an account on LambdaTest, make a note of the user-name & access-key from the LambdaTest profile section.

import os
from selenium.webdriver.common.keys import Keys
from bs4 import BeautifulSoup as bs
from lxml import html
from selenium import webdriver
from selenium.webdriver.common.by import By

// get your user key from LambdaTest platform and import using environment variables
// username = os.environ.get("LT_USERNAME")
// access_key = os.environ.get("LT_ACCESS_KEY")

// Username and Access Key assigned as String variables
username = "user_name"
access_key = "access_key"

Now that we have imported all modules let’s get our hands dirty with web scraping with Python and Selenium.

Locating elements for web scraping with Python and selenium

We would scrap the Blog Titles from the LambdaTest Blog Page. For this, we search for a particular topic and enter the required topic in the search bar.

The following Selenium Locators can be used for locating WebElements on the web page under test:

Tag name
Class name
IDs
XPath
CSS selectors

Here is an example of the usage of Selenium web locators to locate the search box on the page:

In this case, we would use the XPath method driver.find_element(By.XPATH) to locate the search box on the page.

driver.get("[https://www.lambdatest.com/blog/](https://www.lambdatest.com/blog/)")

searchBarXpath = "/html[1]/body[1]/section[1]/div[1]/form[1]/label[1]/input[1]"

# searching topic

textbox = driver.find_element(By.XPATH, searchBarXpath)
textbox.send_keys(topic)
 textbox.send_keys(Keys.ENTER)
 source = driver.page_source

As we have got the content, we can parse it using lxml and beautifulsoup.

Scraping titles using beautifulsoup

After parsing HTML source using lxml’s html.parser, we will find all h2 tags with class “blog-titel” and anchor tags inside them as these anchor tags contain the blog titles.

title_list = [] 
soup = bs(source,"html.parser")
for h2 in soup.findAll("h2", class_="blog-titel"):
     for a in h2.findAll("a",href=True):
            title_list.append(a.text)

return title_list

Putting it all together

Let’s combine the code to get the output.

import os
from selenium.webdriver.common.keys import Keys
from bs4 import BeautifulSoup as bs
from lxml import html
from selenium import webdriver
from selenium.webdriver.common.by import By

// get your user key from LambdaTest platform and import using environment variables
// username = os.environ.get("LT_USERNAME")
// access_key = os.environ.get("LT_ACCESS_KEY")

// Username and Access Key assigned as String variables
username = "user_name"
access_key = "access_key"

class blogScraper:
    # Generate capabilities from here: [https://www.lambdatest.com/capabilities-generator/](https://www.lambdatest.com/capabilities-generator/)
    def setUp(self):
        capabilities = {
                "build" : "your build name",
                "name" : "your test name",
                "platform" : "Windows 10",
                "browserName" : "Chrome",
                "version" : "91.0",
                "selenium_version" : "3.11.0",
                "geoLocation" : "IN",
                "chrome.driver" : "91.0",
                "headless" : True
            }
        self.driver = webdriver.Remote(
           command_executor="[@hub](https://{}:{}<a href=).lambdatest.com/wd/hub">https://{}:{}[@hub](http://twitter.com/hub).lambdatest.com/wd/hub".format(username, access_key),
           desired_capabilities= capabilities)

def tearDown(self):
        self.driver.quit()

def scrapTopic(self,topic):
        driver = self.driver

# Url
        driver.get("[https://www.lambdatest.com/blog/](https://www.lambdatest.com/blog/)")

searchBarXpath = "/html[1]/body[1]/section[1]/div[1]/form[1]/label[1]/input[1]"

        # searching topic
        textbox = driver.find_element_by_xpath(searchBarXpath)
        textbox.send_keys(topic)
        textbox.send_keys(Keys.ENTER)

        source = driver.page_source
        # scraping title
        title_list = [] 
        soup = bs(source,"html.parser")
        for h2 in soup.findAll("h2", class_="blog-titel"):
            for a in h2.findAll("a",href=True):
                title_list.append(a.text)

        return title_list

if __name__ == "__main__":
    obj = blogScraper()
    obj.setUp()
    print(obj.scrapTopic("scrap"))
    obj.tearDown()

Here is the execution output:

['Scraping Dynamic Web Pages Using Selenium And C#', '9 Of The Best Java Testing Frameworks For 2021', 'The Best Alternatives to Jenkins for Developers', '10 Of The Best Chrome Extensions - How To Find XPath in Selenium', 'How To Take A Screenshot Using Python & Selenium?', 'Top 10 Java Unit Testing Frameworks for 2021', 'Agile Vs Waterfall Methodology', 'Why You Should Use Puppeteer For Testing']

Selenium is often essential to extract data from websites using lots of JavaScript as it’s an excellent tool to automate nearly anything on the web.

Here is the execution snapshot of our Python web automation tests on the LambdaTest Automation Dashboard:

Conclusion

In this blog on web scraping with Python and Selenium, we deep-dived into web scraping as a technique that is extensively used by software developers for automating the extraction of data from websites. The purpose of web scraping is to allow companies and enterprises to manage information efficiently. There are a number of applications, such as VisualScrapper, HTMLAgilityPack, etc., that allow users to scrape data from static web pages. On the other hand, Selenium is the most preferred tool for dynamic web page scraping.

Test automation supports a variety of browsers and operating systems. LambdaTest offers a cloud-based Selenium Grid that makes it easy to perform cross browser testing at scale across different browsers, platforms, and resolutions.

Happy Scraping!

Forem: Vinayak Sharma

Selenium with Python Tutorial: Creating Automated Web Bot

Tools For Creating Web Bot using Selenium with Python

Selenium

Python Automation Frameworks

PyUnit (Unittest)

PyTest

Robot Framework

Behave

Task Schedulers in Python

Cron

Windows Task Scheduler

Building Automated Web Bot using Selenium with Python

Creating a Virtual environment and installing dependencies

Folder structure of Test Scripts

Implementation of Automated Selenium Python bot

Generate capabilities from here: https://www.lambdatest.com/capabilities-generator/

setUp runs before each test case and

Sending emails using Selenium for Python

Scheduling Selenium with Python Test Script

Start time of script

for running it one time

Repeat for 10 day

For every 6 hour

Create action

Set parameters

Register task

If task already running, it will be updated

It’s A Wrap

The Ultimate Selenium Python Cheat Sheet for Test Automation

1. Import the Selenium library

2. Driver Initialization with Python

Initialize Chrome WebDriver

Initialize Firefox/Gecko WebDriver

Initialize Safari WebDriver

Initialize IE WebDriver

3. Setting Options in Selenium WebDriver

4. Finding an element

Locate Elements by the ID attribute

Locate Elements by CSS Class

Locate Elements by Name

Locate Elements by XPath

Here is how the email element is located using the XPath attribute

Locate Element by tag name

Locate Element by Link text or Partial Link Text

5. Misc methods for finding elements

6. Opening a URL (or document)

driver.get(URL)

7. Refresh a page

8. Writing text inside a WebElement

9. Clearing text of a WebElement

10. Clicking a WebElement

11. Dragging and Dropping a WebElement

12. Selecting an option

13. Navigating between windows

driver.switch_to_window(“window_name”)

driver.window_handles

driver.current_window_handle

14. Switching to iFrames

driver.switch_to_frame(“frame_name”)

driver.switch_to_default_content()

15. Handling pop-ups and alerts

driver.switch_to.alert

alert_obj.accept()

alert_obj.dismiss()

alert_obj.text()

16. Getting Page Source

17. Navigating browser history

driver.forward()

driver.back()

18. Handling Cookies in Selenium

driver.add_cookie()

driver.get_cookies()

driver.delete_cookie()

19. Setting Window Size

Configuring TimeOuts in Selenium WebDriver

Implicit Wait in Selenium Python

Explicit Wait in Selenium Python

Capturing Screenshots

Conclusion