Forem: Thomas Montfort

Create an Automated Web Bot with Selenium in Python

Thomas Montfort — Sat, 08 May 2021 16:16:13 +0000

Motivation

Due to the pandemic, a lot of activities require signing up online in advance. At my University’s gym, they offer 2 hour slots throughout the day, each of which open 6 hours in advance for reservation. Because the demand for time slots heavily outweigh the amount of slots, the competition for getting a slot has reached a point where you have to be on your computer exactly when they are released.

// Detect dark theme var iframe = document.getElementById('tweet-1359891578173669378-685'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1359891578173669378&theme=dark" }

I decided it was time to combine my coding skills and knack for extreme efficiency (or laziness) to create a bot that would automate signing up for me. My original idea was to have a graphical user interface (GUI) on which I could enter my username and password, and then select the gym location, date, and time for the slot. After submitting the information, there would be a task scheduled 6 hours in advance of the gym time that would reserve a slot for me.

After some research and deliberation, I decided on using the following:

Command line interface (CLI): reserving a gym slot is as easy as opening the command prompt and entering my username and password along with the specific slot I want.
Selenium: this is a Python library that wraps the Selenium Web Driver which can automate web tasks like clicking items and filling out forms.
Windows Task Scheduler: in order to schedule the scripts that reserve a gym slot, I used the Windows Task Scheduler on my computer. Later on, I will cover an option for those with different OSes.

Here is a video of it in action:

You can view my code here.

What Will You Learn?

In this tutorial I will be walking you through how I created my gym web bot, but the topics can be transferable to scheduling and automating web tasks with the use of Selenium in Python. I will be teaching you:

Basics of Selenium
- navigating web pages
- selecting certain elements
- filling out forms
- waiting for elements to load
Implementing schedulers
- Windows Task Scheduler
- API for automating Windows Task Scheduler
- passing arguments to scripts
- alternative for Windows Task Scheduler

Getting Started

In order to get started with this tutorial, I trust that you have python downloaded and a coding editor (I use VSCode) that you prefer.

I would also recommend using an environment (optional) to avoid cluttering your local environment and having issues with dependencies. After you have handled the tasks above, you can switch to your project environment (if you created one) and enter the following command:

pip install selenium

Your next order of business is to download a Web Driver. This is what Selenium uses to access a browser and perform the tasks that you program it to do. I personally prefer Chrome so that is the browser I will be using for this tutorial. You can download the Chrome Web Driver here . Make sure to save the path of the driver (the location of it in your file system) - we will need this later.

You are now ready to automate web tasks!

Basics of Selenium

Once you have a .py file opened and are ready to code, you first need to import webdriver from Selenium and instantiate the Chrome webdriver instance.

from selenium import webdriver

driver = webdriver.Chrome(insert_webdriver_path_here)

Now that we stored the webdriver instance in a variable, we will be able to use its various methods to interact with the web. Our first order of business is to go to a webpage.

driver.get(insert_target_url_here)

To start, I would choose a simple web page like Instagram. For my project, I have a function that determines the url based on the gym location selection.

Navigating a Web Page

Now that we have landed on a web page, we need to be able to navigate to other pages and perform tasks. When arriving on my University’s gym website, I must first login. To do so, I click the login link in the top right corner.

If you are unfamiliar with html, it implements a Document Object Model (DOM) which is a hierarchical structure of elements that have certain attributes. For instance, links are stored in anchor elements with an href attribute which provides the url. Selenium has direct access to the DOM and can access specific elements like a link and perform actions on them like clicking.

Selenium provides a couple of methods for selecting elements listed here . In order to locate elements that you want to interact with you can inspect the webpage with Google Chrome Dev Tools and use the element inspector.

The element inspector is circled in red and the element in the Dev Tools highlighted in blue refers to the login link. The login link has id “loginLink” so I did the following:

login_link = driver.find_element_by_id(“loginLink”)

login_link.click()

Waiting For Elements To Load

Because clicking on the login link redirects the browser to another page, it takes some amount of time for the web page to load along with its elements. If we tried to access an element on the login page right after clicking the login link, we’d experience an error because the page hasn’t loaded yet. Therefore, we need to implement Selenium’s explicit wait function.

Selenium’s WebDriverWait waits for a certain condition to be true before it proceeds to access an element. To wait for the login page to load and access the username form field I wrote:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_condition as EC

username_field = WebDriverWait(driver, 1).until(
    EC.element_to_be_clickable((By.ID, “username”))
)

First, we pass WebDriverWait the webdriver and the maximum amount of time (in seconds) we want the webdriver to wait. Then using the expected_condition we can select the condition we are waiting for and pass it the element we want to select. Once the condition evaluates to true, the WebDriverWait function will return the selected element.

Filling out Forms

Now that we have selected the username form field, we need to have the Web Driver input our username. To write in an element, we take the variable containing the element and use the send_keys() method.

username.send_keys(insert_username_here)

You can access the password field and insert your password with the instructions above. After submitting the login form, I am then redirected back to the web page for registering for a gym slot.

Accessing those Hard to Select Element

Now that I am back to the gym slot reservation page, I have encountered an issue. On the reservation page, there are multiple time slot cards of which I need to click the register button corresponding to the inputted gym time. However, each time slot card has the same attributes so I can’t access the correct time slot with name or id.

This is where using Xpath comes in handy. Xpath stands for XML Path Language and allows you to navigate to certain elements of an XML document with a given path. When using the Google Chrome Dev Tools, you can right click on an element and copy its Xpath. Xpath also allows you to select an element based on the physical text it displays. Therefore, I am able to find the time slot card with the correct time and then navigate to the register button so that I can click it.

To do so I write:

register_btn = driver.find_element_by_xpath(“//*[id=’mainContent’]/ … /div[1]/small[contains(text(), ‘insert_text_to_match_here’]/../../div[2]/button”)

To visualize how I am navigating the DOM tree:

After finding the card with the text that matches that correct gym time, I navigate up the DOM tree back to the card and then traverse down to the register button.

With the above information, you should be able to navigate throughout a website, select elements - even in the most difficult situations, click on them, fill in forms and wait for elements to appear so that you can interact with them.

Implementing Schedulers

A task scheduler allows for you to register scripts to be run given a certain time or condition. In the case of my gym reservation bot, I want it to reserve a gym slot 6 hours in advance of the time slot I selected.

For this, I used the Windows Task Scheduler. It is a Windows Application with a GUI so you can open it, view default scheduled tasks and create your own task. When creating a task, you provide the scheduler a trigger - certain time of day or condition, an action - something to be executed in the command prompt, and conditions - when you want the task to run.

Obviously I don’t want to have to open the app every time to schedule a task, so I need an application programming interface (API) that will allow me to write code to automatically schedule one.

Win32com Library

The win32com is the API library that I choose to automate scheduling tasks on Windows. It is essentially a very thin Python wrapper that allows you to automate Windows Applications which in this instance is the Task Scheduler. To download the library:

pip install pywin32

Once the library is downloaded, you will be able to write code to interact with Task Scheduler. This post shows you exactly the code you need to write to schedule tasks but I’ll show you some small changes to make for my implementation

Start time: set the start time of the task. For me, its 6 hours before the gym slot.
Repetitions: set how many times you want the task to repeat. Since, I want the gym sign up task to only occur once (my schedule varies), I deleted the code for this altogether.
Path: set action.Path to the path of your python interpreter. This will allows you to run python scripts.
Arguments: set action.Arguments to the path of the script you want to execute and input any arguments for the script.
Configure additional conditions:
- DisallowStartIfOnBatteries = False → Will allow task to execute when your computer isn’t charging
- WakeToRun = True → Will wake up your computer when asleep to run a task.

Arg Parser

In order to have my program operate as a CLI, I need to have a way to pass inputs such as the password, username, gym location, etc. to my script. To do this, I used the built-in Python library argparse. Argparse will allow you to define arguments for a python file, and extrapolate them into variables to be used in your program.

For instance, to add an argument for time I wrote the following:

parser.add_argument(‘-t’, type=str, help=’Gym slot starting time in format 00:00’)

I define an argument -t that takes an input of type String and provide a help message that reminds me how to format the time.

When running a python script from the command prompt, to pass the time argument, I write:

python main.py -t 12:30

Then to extrapolate the argument variables I do:

args = parser.parse_args

And to access a specific argument you refer to the name of the argument. For instance, accessing the time argument I passed into the script earlier:

time = args.t

The same can be done for the rest of the variables you need.

Scheduler Alternatives

If you have a Linux or Mac OS then you can use cron. Cron is the same as Windows Task Scheduler in that you can schedule tasks to run at a given time. This post goes into detail about how you can schedule python scripts. However, instead of using an API to schedule tasks, you use the command line. In code, you can use os.system() in a python script in order to have commands executed at the command line.

Putting it All Together

For my gym bot project, I created two .py files, one for scheduling a task and one for executing the task. The scheduler.py file allows me to pass the information necessary to reserve a gym slot. For instance, say I wanted to reserve a spot at Ernie Gym at 4:00pm on May 5th as I do in the video example. I would type in the command prompt:

python scheduler.py --u tjmontfo --pw ******** --d 05/05/2021 --flr ernie -t 16:00

The scheduler.py file will use this information to create a scheduled task at 10:00am on May 5th. When the task is executed, it will run the main.py file which will be passed the username, password, location and time and using the Selenium WebDriver, reserve a gym slot.

This tutorial does not need to be specifically used for creating a gym reservation bot. Selenium in Python has extensive applications like scraping websites for data or testing applications. Also, it is not necessary to use Selenium in tandem with a scheduler like Windows Task Scheduler. Scheduling tasks just allows for scripts to be run in the background without having to remember to run them.

Summary

Thank you for reading through the tutorial, I really hope you gained something from this. In fact, I challenge you to create something using Selenium in Python. It can be as simple as a web bot that logins into your Instagram account and likes a post. Please DM me a video of it in action or a Github repository of your code to my twitter account .

If you enjoyed my in-depth explanation of programming in Python please check out some of my other posts.

Why Your Impression of a Hackathon is Wrong - You Are Missing Out

Thomas Montfort — Mon, 29 Mar 2021 19:59:06 +0000

Addressing the Misconceptions

If you don’t actually know what a hackathon is you are probably imagining the scene from Social Network where Facebook hirees are challenged to a race to hack into a server. The scene takes place in a tightly packed room with a cheering audience and shots being forced down their throats.

While I am unsure if this Facebook story is true, I can assure you that hackathons are nothing like this. The atmosphere is not of the high stake and competitive nature that Social Network portrays but rather an innovative, collaborative and fun environment.

People hear the word “hackathon” and assume that some nefarious criminal activity is occurring. The “hack” part is not referring to the literal definition. It’s supposed to be a play on words hack and marathon. In fact, one of the intentions of hackathons is to create projects in 24-48 hours that solve real world problems. Hackathons encourage projects of this nature with certain prize categories like Best Climate Hack.

Some Definitions:

Hackathon: the event.
Hacker: a person who attends a hackathon.
Hack: the project a group or individual creates at a hackathon.

Lastly, people have this belief that hackathons have very high barriers for entry because you must be a talented coder. First of all, writing code is only a small part of the many elements that go into a successful project. We need entrepreneurs to provide creative ideas, designers to help with logo design and UI, and business people to help pitch the final presentation to the judges.

Also, if you do code but don't have much knowledge or want to learn, attending hackathons are a great way to gain resources and direction to learn.

Most people have a very false image of what hackathons are, but by learning about what they actually are and the benefits they provide, you will not want to pass up on them.

So what is a hackathon then?

Glad you asked. A hackathon is an invention marathon where people from around the country and possibly the globe gather in one area for a weekend. Once assembled, they form teams, share ideas and collaborate to create an original project that addresses a problem. Throughout the hacking, there are many breaks for caffeine, free food, fun games, workshops and networking with reputable companies.

CuseHacks 2020:

You usually arrive at the hackathon on a Saturday where you check-in, meet your team or participate in team-building events to find teammates, and begin working on your project. When the hackathon is coming to a wrap on Sunday, you and your teammates demo your project to judges where you can win prizes for specific categories.

The real magic of a hackathon is the actual projects that can be built in 24 hours. One famous example is GroupMe, where the founders locked themselves in a room during a 2010 hackathon in NYC to code out their communication app idea. The amazing thing is they didn’t even win any prizes but felt so confident about their idea that they quit their full-time jobs to pursue it. This risk paid off a year later when Skype purchased GroupMe for $85 million.

The power of a community of innovators and technical people coming together for a weekend of sharing ideas and knowledge and addressing problems, creates an incredible atmosphere. Hackers are able to consult mentors, professionals in the industry or peers for advice. The abundance of resources allows for lasting connections and great learning opportunities.

Alright, so why should I attend a hackathon?

After learning about what a hackathon actually is you should be convinced but let me just give some more motivation:

Learn industry skills: By attending an event with a community of tech innovators, you will be sure to learn about new emerging technologies that you can incorporate in your own work.
Networking: Hackathons are great for networking because they allow you to talk to employees at top-tier companies like IBM who are passionate and have experience in your field of work unlike recruiters. It also can lead to great first impressions because you can directly showcase your skills through your project.

IBM Table at CuseHacks 2020:

Great community: When attending a hackathon, you can easily meet like-minded individuals. It's a great place to discuss ideas and make great connections.
Resume builder: Hackathons are very attractive on your resume because they display your passion for the industry and provide the opportunity to build a project that you can put on it.
Free stuff: With the reasons mentioned above, you might be skeptic that a hackathon is free. However, not only is attending free, but so is the food, drinks and swag.
They’re fun: It’s basically like having a sleepover with friends where you have the latitude to create whatever you want and spend your time however you choose whether that be actually creating a project or attending workshops and games.

I understand that even with the above reasons, that you may still be hesitant about attending a hackathon. If you have never attended one before and have no friends to accompany you, it might seem daunting to attend. However, I can assert to you that the hackathon community is very welcoming. It’s also not even required to build a project so you can attend only to make friends, learn and abuse the free stuff.

Resources

Now that you have decided you want to attend a Hackathon, you are probably now thinking about where you can find them. Well there are two awesome resources that can lead you in the right direction.

Major League Hacking (MLH):

MLH is the official student hackathon league. They help support 200 weekend long hackathons every year. You are able to find upcoming hackathons through their event page. They also provide other resources for hackers including internship and fellowship opportunities.

Devpost :

Devpost is basically like the social media for hackathons. You have the ability to create an account which you will use to submit your projects for hackathons. Your profile will be populated with all of the projects you submitted including any prizes that you won for specific categories.

Devpost is also a great place to find hackathons as they include more than just the MLH sponsored events and have a great search filter system for location, length and interest

Final Thoughts

Personally, I held some of the same misconceptions I addressed. However, I joined the Innovate Orange club this winter at Syracuse University that hosts CuseHacks . When helping moderate the event this year, I was able to connect with experienced industry professionals and learn about newer technologies like GANs . I only moderated, but it was still fun and beneficial to me!

My teammates and I at CuseHacks 2021:

I think the biggest thing holding back hackathons is the misconception that you need to know how to code. However, we actually need people with different specializations from backgrounds like design, business and entrepreneurship. Trust me, coders often lack the creativity for ideas or the business skills to pitch their project, so those who have these specializations would make a huge impact.

If you have never attended a hackathon before, the best advice is to go in open-minded. While some experienced hackathoners will go in with a project idea and only focus on winning a prize, don’t let those select few scare you. The real reason to attend a hackathon is to meet like minded people and learn.

If you have made it this far, please consider attending a hackathon. They are not just for students. While a lot of hackathons are hosted by Universities, there are also companies like Amazon who will host hackathons for those who are in industry. They also come in different forms like remote or extended periods of time instead of the traditional in-person weekend events.

Feel free to reach out to me on twitter if you have any questions.

The Mystery of Recursion - Understanding How to Implement it in Python

Thomas Montfort — Fri, 05 Feb 2021 16:34:54 +0000

What is Recursion?

If you understand the underlying concept of recursion but struggle to implement it like myself, or you don't even know what it is, then this post if for you.

Recursion is defined as solving a complex problem by breaking the problem into smaller versions of itself that can be solved. In programming, this is done by having a function call itself.

The underlying principle of recursion are these two cases:

Base Case: this case defines the smallest instance of a problem. In certain recursive solutions, there can be multiple base cases but for the purpose of this post's explanation, my example will only incorporate one. When coding, the base case prevents an infinite loop of the function calling itself.
Recursive Case: this case manipulates the problem in order to approach the base case. In programming, this is the case in which the function calls itself.

Put simply, the recursive case will breakdown the problem until it arrives at the base case, after which the the sub solutions build up to solve the original problem.

Do not worry if your understanding of recursion is hazy because the above definitions will make sense when implementing a recursive solution in Python.

The Call Stack

In order to truly understand how recursion is implemented in Python, it is necessary to familiarize yourself with the call stack.

The call stack uses the stack data structure to keep track of local variables and previous function calls. The stack is a last-in, first-out data structure. This means that the last item to be placed in the stack will be the first to be removed. To help visualize, think of a stack of plates:

Whatever plate is placed last on the stack is the first one to be removed.

In terms of programming, the stack has two simple operations: push and pop. Push adds an item onto the stack and pop removes an item from the stack, returning its value.

In python, the call stack consists of frames. The bottom most frame is the global or module frame. This consists of all the global variables. Every time a function is called a new frame is pushed onto the stack containing its local variables and function arguments. When a function call returns, the frame is popped off the stack and its value is passed to where it was called in the previous frame.

A Famous Recursion Example - Factorial

The factorial of a number is defined as so:

4! = 4 * 3 * 2 * 1 = 24

If we were to define the factorial of a number n as a mathematical equation, it would be:

factorial(n) = n * factorial(n-1)

The right side of the equation will be the recursive case of our solution. Notice, how the recursive case takes the original problem, factorial(n), and makes it a simpler problem of multiplying n by whatever factorial(n-1) is.

We can define a function factorial that takes a parameter n. However, if we only defined this function with the above equation, it would call itself infinitely leading to an error.

This is because we did not define a base case. The most basic factorial is the factorial of 1 which equals 1. So we can tell the program that once it reaches the factorial of 1, instead of calling 1 * factorial(0), to just return 1. After which, the Python call stack will handle multiplying out the numbers.

After defining both the recursive and base cases, we arrive at the Python program below:

def factorial(n):
    if n <= 1:
        return 1
    else:
        return n * factorial(n-1)

In Depth Recursive Function Implementation

Before calling the factorial function, the Python call stack would look like:

Then, we call the factorial function passing in an argument of 4.

factorial(4)

A frame for the factorial function would be pushed onto to the call stack, setting the argument n equal to 4.

The Python interpreter will then process the function and since n is not <= 1, it will evaluate n * factorial(n-1). However before the function can return a value, it must evaluate factorial(n-1). This will cause another factorial frame to be pushed on the stack, with the argument n set to 3.

Again, since n is not <= 1, the function will evaluate n * factorial(n-1). Likewise, the function must evaluate factorial(n-1) before returning a value.

This pattern will continue until the factorial frame with the argument n equal to 1 is pushed onto the stack.

When this occurs, the function reaches the base case, so it instead returns 1. As I mentioned early when explaining the call stack, when a function returns, it pops its frame off the stack, transferring its value to where it was called in the previous frame.

Now since the value for factorial(n-1) has been evaluated, the factorial function can evaluate the product and return it to the previous frame, pushing the current frame off the stack.

This pattern continues until we reach the original function call...

As you might have guessed, the factorial function with argument 4 returns a value of 24.

Summary

When writing a recursive solution in Python, think of the problem in terms of a base case and recursive case. Ask yourself: "How can I define a recursive case that will help me reduce the problem towards a base case?"

When trying to understand a recursive solution, I find it best to visualize the call stack in my head. Every time a function is called an item is added onto the stack and any time you encounter the return keyword, it pops an item off the stack and returns its value to the previous function call.

If you have made it this far and have enjoyed my post, please follow me on Hashnode and Twitter. I plan on posting more content and am open to suggestions or requests!