Forem: Shaheer Sarfaraz

Saving time by wasting it - The story of AutoClass

Shaheer Sarfaraz — Fri, 14 Oct 2022 13:08:03 +0000

The Idea

Because of online lessons during the lockdown, my friends and I thought that getting out of bed at 8:00 a.m. to sit in front of a screen with a monotonous teacher teaching an unimportant topic was boring.

This was about to change since it occurred to me that classes typically start at the same time each day. The starting time of a lesson is scheduled in advance and doesn't change much from day to day. This means that instead of waking up at 8 am to take my class, the computer would know it's 8 am, know it's time for a class, and join that class's Zoom meeting code.

The heart of the program does just this. However, some changes still needed to be made to make AutoClass an actually easy (read: usable) program for me and my classmates. The most important of these changes was ensuring you joined the meeting even if something unexpected happened.

Of course, teachers are also humans, and they can't always start their classes strictly at the chime of the hour, so that's a good starting point for where to start handling these edge cases. Technically, the program went through quite a few stages since sometimes, I'd wake up at 12:00 and realize the program failed to join any classes before this time. I'd trace down what went wrong and fix it in the next update.

Over time, I developed AutoClass into utterly impenetrable software that could handle the most extreme edge circumstances. An example is when the teacher forgot to start the lesson until much later than the scheduled time or when the teacher removed me (read: my computer) from the classroom.

All of this combined meant that regular online classes merely served as a testbed for new features and improvements I'd implemented in AutoClass, not as a source of academic learning. My classmates and I started to push the limits of AutoClass and the school's management once the software had proven reliable to a certain extent.

Since I had spent the better part of the previous year working on AutoClass, and dare I say I was really proud of it, I knew I had to include it with my uni apps. However, this presented another problem because our school doesn't have a strictly separate admin and counseling department. This means that I can't write about AutoClass without also revealing that I wasn't taking any of my classes. I had to work around this by entrusting my work to only one of the councilors I trusted a little more to not open my secret, at least until I passed the school.

Development continued until the school shifted to a hybrid approach where some classes were on-campus, with the option to take them online if you so preferred. I would leave AutoClass running at home even when attending the school in person since I wasn't ready to stop the constant progress and improvement. Hence, there was a time when a person named "Shaheer Sarfaraz" was present in a class in person and online. This is, of course, extremely easy for any half-competent admin department to identify as an issue. Thankfully for us, this wasn't an issue :).

There's still the fact that my classmates are pretty observant, and so there were a few times when some of them called me out, saying, "wait, you're here in person, I swear I saw you in the online class as well." I was forced to make something up to ensure my "secret" wasn't discovered. I doubt that anyone was buying my shenanigans, however.

The Consequences

I've paraded that we were skipping classes left, right, and center as wholly positive, which was in the short term, but the long term was soon approaching. Mock exams took place in our school before the A Levels exams, and my entire friend group knew absolutely nothing. Not even what we had to know.

Because we weren't taking any classes, our academic performance declined sharply. We were unable to approach our teachers since, in their eyes, given our exemplary attendance record, we had to be familiar with every subject at the very least. This couldn't be farther from the truth, so we knew we had to study by scraping together all the Indian Youtube Channels to help us out.

Students started blaming AutoClass as a source of their failures. This was amusing because they knew the risks of missing practically all classes in 6 months, yet they blamed my program. Thankfully, at least my friends and I clawed our way out of the hell we created for ourselves.

Technically

AutoClass is also a reasonably efficient program in terms of the technologies used. The program's backbone is written in Python, with a library called pyautogui for automation. The classes are picked using a spreadsheet, so it's easy for anyone to start with their classes and timings. The codebase is easy to work on, with a clear separation of concerns.

After months of putting it off, I also implemented a functional UI to represent the program's state and classes, e.g., a color-coded list of all online classes. This allowed you to view your chosen settings, such as how long you wanted to attempt to join the class or your display colors.

The program's entire history and the various updates it went through are preserved in the git history of the main AutoClass Git Repo.

I've graduated

I'm happy to report, I've now graduated from Benchmark School System. With this comes the end of development of AutoClass since I'm no longer taking online classes.

Throughout, AutoClass has been a very substantial project, and it will always have a place in my heart. Not because of the technical aspect of it, but because of the emotional factor of flying under the radar without flying too close to the sun.

Get In Touch

👋 Hi, I’m @DaKheera47, you may know me as Shaheer Sarfaraz

👀 I’m interested in making web apps in React and Node, and you can view more of my projects here

💻 I'm also passionate about automating everything to improve my day-to-day life

🌱 Currently, I’m working as a Full Stack Web Developer at Project Mirage, a Design And Development Consultancy

📫 I'm free to talk any time on my email address, shaheer30sarfaraz@gmail.com

Why I Created Thousands Google Meet Links and Rate Limited My Account

Shaheer Sarfaraz — Sat, 03 Sep 2022 08:04:41 +0000

The Idea

It's simple really. What if you could have a Google Meeting with a code of your choosing.

As you might already know, creating a new meeting with Google Meet creates a new link with a pattern of random unique letters at the end. The pattern looks like this xxx-xxxx-xxx.

My idea is to create thousands of these URLs in the hope that maybe one pops up with a phrase in it that I liked, maybe my name, maybe my initials, it could be anything.

All in all, It's a project purely for fun. It's fun to just mess around every once in a while!

What I Already Knew

A new link can be generated by visiting a specific url
Links are always unique and random
A Google Account which creates a link is always its host
A meeting link is re-usable after leaving

How To Approach This Problem

There were a few ideas on how to generate these meetings and get their links.

Python + Selenium

Selenium is a web browser automation tool, and there's bindings for the Selenium API in Python

The Good

An advantage of using Selenium would be the ability to hide a browser window completely whilst it generated links using Headless Mode. This would allow the program to run in the background even when you were using your device

The Bad

Accessing a Google account in a Selenium driven window requires you to jump through quite a few hoops because of the possible breach in security this could lead to. This project needed to be built quickly, the quality of the code didn't matter too much because at the end of the day, this project won't ever serve a real purpose
Before this, I'd only ever heard of Selenium and had a glance at the documentation, and never actually used it in a project, so there'd be a learning curve to overcome during this project too

Python + pyautogui

The second idea, and the one I ended up going with was using Python and pyautogui (a package to use a Python script to control the mouse and keyboard).

The Good

There would be no browser compatibility issue, since I would be using my normal browser with its default Google Account.
And, there'd be a easy and quick way to click on elements on screen. Simply take a screenshot of the element you want to click and reference it in your code. This completely avoids going through the pain of finding out which selector is precise enough in Selenium.
Another positive for me was the fact that I had been using pyautogui for a while before this for AutoClass, so using it was almost second nature to me

The Bad

It's slower than Selenium for sure, but absolute speed just wasn't the main goal here.

A Laundry List of Problems

How To Find Out When The URL Has Changed

The way I first imagined the program working was by looking at the URL bar of the browser. With Edge, whenever I joined a meeting, it would show a camera icon, indicating that the camera was being used. This of course would only happen after the page loaded up completely (and hence the link in the URL bar had changed from https://meet.google.com/new to a new URL with the pattern), and since I could easily take a screenshot of anything on the screen and detect its presence and location, I felt this was a good place to start.

Coding this up wasn't an issue, especially with the helper functions like findImageTimeout I had made previously which add some functionality which is actually on the roadmap of the pyautogui package. As the name suggests findImageTimeout attempts to find an image on screen using pyautogui, and keeps attempting to find it until the specified timeout is up.

How To Get The URL From The URL Bar

Now I knew when the URL had updated to the new one, however I still hadn't gotten around to actually getting the URL into a variable. Checking to see if the camera was visible did help me in this case because the new URL is just slightly left of the where the camera icon appears. Now we know how to tell the program where to find the URL, but how to get it into a variable?

Purely by chance, I had stumbled across this exact problem in an earlier project (more on that in another post), and I knew how to solve it now. Using pyautogui, you can send hotkeys like ctrl + c to copy and in Python, you can access the contents of the clipboard in a few different ways.

The one I used was pyperclip because of its simplicity, and reliability.
The second option was using win32, which had extremely verbose syntax, and had to be executed with quite a few pauses with time.sleep() to get it to work reliably, so it far slower, and more clunky to work with.

The Complete Roadmap

So to get the URL from the browser, I can have the program click the URL bar slightly left of the coordinates of the camera icon, which automatically selects all the text in the field, then I can send a ctrl + c using pyautogui to copy the created URL into the clipboard. I can then read the clipboard using pyperclip, and store it in a file.

v1.0

After implementing the very initial concept of the project laid out above, here's some gripes I still had with the program.

Clunky Codebase

The code base at this time was very bodged together, almost feeling like it was different pieces of code just duct taped together without thought. The program wasn't too reliable either, eg if a link failed to load, then the program would be unable to handle it.

Speed

I measured the performance of the script in terms of seconds per link. In its current state, the script was taking around 13-15 seconds per link. This was directly and severely impacted by my (at the time) subpar internet performance, and other factors like when Edge decided to show the camera icon.

v2.0

Here comes the single biggest improvement to the speed of the program.

The Google Meet Homepage has an option to create the link for a later time.

Crucially for us, to create a new link, this option doesn't refresh the whole page. It loads a modal on the page, with just the link and a copy button.

Conveniently, it also has a close button on the modal, which resets the state of the webpage. This means this loop can be repeated extremely quickly.

With this method, I brought the program down to about 2 seconds per link. We're looking at roughly a 750% improvement from our initial starting point of 15 seconds per link.

v2.1

Previously, processes taking up fractions of a second wouldn't have made a significant performance improvement, since one link was taking 15 seconds. However now, with our time down to 2 seconds per link, I had to optimize these processes to improve speed further.

Find Images In Screenshots More Effectively

One of the main areas where I could still improve was finding elements on the screen. Now seems like a good time to mention that just taking a screenshot of a 1080p display takes ~100ms, according to the pyautogui docs. Finding an image within this screenshot would take far more time, and so this was a main point of concern for performance. This can be improved significantly and we'll see how later.

Fractional Improvement

To prevent pyautogui from attempting to find static objects (eg the "close" button and other similar non moving objects) on the entire screen, I had to hard code the exact pixel coordinates of these locations, and clicking on these pre-defined points as opposed to taking time to find them on screen.
There was at least one element I couldn't hard code however, and that was the copy button in the modal. This is because there's no set time for it to appear. It could appear in 300ms, or it could take up to 1000ms, all subject to my internet connection.

I could optimize this further however. Considering we already know that the copy button will appear only in a specific area, we can search for the copy button in that (far smaller) area instead of the entire screen. This makes the search process as efficient as it can be.

All of this improvement resulted in sub-second time for each link generally around 0.75 seconds for every link. The largest performance bottleneck now was my internet connection, and I considered this a good enough place to stop working on it, and also because I got the link I wanted to find, a link with dak in its pattern.

Being Rate Limited by Google

Since this program was creating links automatically (and in sub-second time), of course I left it running overnight. After a few nights of creating links, I ran into a limitation set in place by Google. If I generated more than 5000 links in a span of one hour, I got rate limited.

This was evident because the modal, instead of showing the copy button with a new link, showed an error message. I found out this limitation was account-specific, and switching to another account meant you could go all the way again. I felt quite good about myself since this was one of my first web scraping / web automation projects, and getting rate limited by Google seemed like quite an achievement.

Some Interesting Observations

Google doesn't use the letter L in any of its link patterns
Google avoids common words in its links