Forem: Dmitriy Zub ☀️

SerpApi Changelog, May 2023

Dmitriy Zub ☀️ — Tue, 06 Jun 2023 08:19:31 +0000

Summary of May Updates

A sum-up of SerpApi's latest product improvements, and feature releases 🙂

Features

[Feature] Google Scholar API: add support for "Review Articles"
[Feature] Google Answer Box API: support thumbnail for answer box finance results

Urgent Fixes

Fixes

Google APIs:
Bing API:
- Bing Search API:
  - [Fix] Bing Search API: new layout for local pack
Yahoo API:
- Yahoo Search API:
  - [Fix] Yahoo Organic Results API: scrape shopping results
  - [Fix] Yahoo API: scrape refine this search

API Updates

[Feature] Google Scholar API: add support for "Review Articles"

Documentation | Public Roadmap Issue

[Feature] Google Answer Box API: support thumbnail for answer box finance results

Public Roadmap Issue | Documentation | Playground

Conclusion

If you notice any bugs or possible improvements, then you can directly improve the product by creating a issue in the SerpApi Public Roadmap.

Join us on Twitter | YouTube

💛

Make Direct Async Requests to SerpApi with Python

Dmitriy Zub ☀️ — Wed, 24 May 2023 12:19:59 +0000

Intro

In the previous SerpApi Async Requests with Pagination using Python blog post we've covered how to make Async requests with SerpApi's pagination, how to use Search Archive API and Queue.

In this blog post we'll cover on how to make direct requests to serpapi.com/search.json without using SerpApi's google-search-results Python client.

This way, when making a direct request SerpApi, we can get a slightly faster response time in comparison to Python's client batch async search feature which uses Queue.

For example, 50 requests in less than 10 seconds. Depends on the internet speed.

In the following blog post, we'll cover how to add pagination to the shown code below.

Subject of test: YouTube Search Engine Results API.

Test includes: 50 async search queries.

Code

You can check the code example in the online IDE:

import aiohttp
import asyncio
import json
import time

async def fetch_results(session, query):
    params = {
        'api_key': '...',      # your serpapi api key: https://serpapi.com/manage-api-key
        'engine': 'youtube',   # search engine to parse data from
        'device': 'desktop',   # from which device to parse data
        'search_query': query, # search query
        'no_cache': 'true'     # https://serpapi.com/search-api#api-parameters-serpapi-parameters-no-cache
    }

    async with session.get('https://serpapi.com/search.json', params=params) as response:
        results = await response.json()

    data = []

    if 'error' in results:
        print(results['error'])
    else:
        for result in results.get('video_results', []):
            data.append({
                'title': result.get('title'),
                'link': result.get('link'),
                'channel': result.get('channel').get('name'),
            })

    return data

async def main():
    # 50 queries
    queries = [
        'burly',
        'creator',
        'doubtful',
        'chance',
        'capable',
        'window',
        'dynamic',
        'train',
        'worry',
        'useless',
        'steady',
        'thoughtful',
        'matter',
        'rotten',
        'overflow',
        'object',
        'far-flung',
        'gabby',
        'tiresome',
        'scatter',
        'exclusive',
        'wealth',
        'yummy',
        'play',
        'saw',
        'spiteful',
        'perform',
        'busy',
        'hypnotic',
        'sniff',
        'early',
        'mindless',
        'airplane',
        'distribution',
        'ahead',
        'good',
        'squeeze',
        'ship',
        'excuse',
        'chubby',
        'smiling',
        'wide',
        'structure',
        'wrap',
        'point',
        'file',
        'sack',
        'slope',
        'therapeutic',
        'disturbed'
    ]

    data = []

    async with aiohttp.ClientSession() as session:
        tasks = []
        for query in queries:
            task = asyncio.ensure_future(fetch_results(session, query))
            tasks.append(task)

        start_time = time.time()
        results = await asyncio.gather(*tasks)
        end_time = time.time()

        data = [item for sublist in results for item in sublist]

    print(json.dumps(data, indent=2, ensure_ascii=False))
    print(f'Script execution time: {end_time - start_time} seconds') # ~7.192448616027832 seconds

asyncio.run(main())

Code Explanation

Import libraries:

import aiohttp # to make a request
import asyncio
import json    # for printing data
import time    # to measure execution time

In the fetch_results() function we:

creating search params that will be passed to SerpApi while making request.
making an async session request, passing params and awaiting for each response and storing it to results variable.
checking for 'error' in the results and iterating over 'video_results', and storing extracted data to the data list.
returning list with videos data.

async def fetch_results(session, query):
    params = {
        'api_key': '...',      # your serpapi api key: https://serpapi.com/manage-api-key
        'engine': 'youtube',   # search engine to parse data from
        'device': 'desktop',   # from which device to parse data
        'search_query': query, # search query
        'no_cache': 'true'     # https://serpapi.com/search-api#api-parameters-serpapi-parameters-no-cache
    }

    async with session.get('https://serpapi.com/search.json', params=params) as response:
        results = await response.json()

    data = []

    if 'error' in results:
        print(results['error'])
    else:
        for result in results.get('video_results', []):
            data.append({
                'title': result.get('title'),
                'link': result.get('link'),
                'channel': result.get('channel').get('name'),
            })

    return data

In the second main() function we:

create a list of queries. Could be also txt/csv/json.
open a aiohttp.ClientSession().
iterate over queries and create asyncio tasks.
proceed all of the tasks with asyncio.gather(*tasks).
flatten list with data and store it to the data variable.
print the data.

async def main():
    queries = [
        'burly',
        'creator',
        'doubtful',
        # ...
    ]

    data = []

    async with aiohttp.ClientSession() as session:
        tasks = []
        for query in queries:
            task = asyncio.ensure_future(fetch_results(session, query))
            tasks.append(task)

        start_time = time.time()
        results = await asyncio.gather(*tasks)
        end_time = time.time()

        data = [item for sublist in results for item in sublist]

    print(json.dumps(data, indent=2, ensure_ascii=False))
    print(f'Script execution time: {end_time - start_time} seconds') # ~7.192448616027832 seconds

asyncio.run(main())

Conclusion

As you saw (and possibly tried) this results in quite a fast response times. Additionally, we can add pagination to it, which will be covered in the next blog post.

Join us on Twitter | YouTube

SerpApi Changelog: April, 2023

Dmitriy Zub ☀️ — Wed, 24 May 2023 09:18:13 +0000

Summary of April Updates

A sum-up of SerpApi's latest product improvements, and feature releases 🙂

Features

[Feature] Accounts API: now has a monthly subscription cost
[Feature] Google Shopping API: support "Stores Near You" Block
[Feature] Bing Search API: support Coupon results

Parser Updates

Fixes

Projects built with SerpApi:

API Updates

If you notice any bugs or possible improvements, then you can directly improve the product by creating a issue in the SerpApi Public Roadmap.

Join us on Twitter | YouTube

💛

Scrape Google Scholar in R

Dmitriy Zub ☀️ — Sat, 06 May 2023 07:43:54 +0000

What will be scraped
Prerequisites
Full Code
- Explanation
Google Scholar Python scraper alternatives

Scraping Google Scholar profiles in R can be a powerful tool for academic researchers, librarians, and data analysts.

This blog post will show how to scrape profile data with pagination.

What will be scraped

Prerequisites

In your R CMD install all the needed packages:

install.packages("httr")
install.packages("rvest")
install.packages("jsonlite")
install.packages("purrr")
install.packages("stringr")
install.packages("glue")
install.packages("dplyr")

Full Code

Please keep in mind that I'm not an experienced R user and some of the techniques might be better implemented.

library(httr)
library(rvest)
library(jsonlite)
library(purrr)
library(stringr)
library(glue)
library(dplyr)


scrape_all_profiles_from_university <- function(label, university_name) {
  headers <- c("User-Agent" = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36")

  # remove trailing whitespaces/hidden characters
  label <- trimws(label)
  university_name <- trimws(university_name)

  params <- list(
    view_op = "search_authors",
    mauthors = glue("label:{label} '{university_name}'"),
    hl = "en",
    astart = 0
  )

  # empty list to store future the profile results
  all_profile_results <- list()

  profiles_is_present <- TRUE
  while (profiles_is_present) {
    response <- GET("https://scholar.google.com/citations", query = params, add_headers(.headers = headers))
    page <- read_html(content(response, "text"))
    print(paste0("extracting authors at page #", params$astart))

    profiles <- page %>% html_elements(".gs_ai_chpr")
    profile_results <- map(profiles, function(profile) {
      name <- profile %>% html_element(".gs_ai_name a") %>% html_text()
      link <- paste0("https://scholar.google.com", profile %>% html_element(".gs_ai_name a") %>% html_attr("href"))
      affiliations <- profile %>% html_element(".gs_ai_aff") %>% html_text(trim = TRUE)
      email <- profile %>% html_element(".gs_ai_eml") %>% html_text()
      cited_by <- profile %>% html_element(".gs_ai_cby") %>% html_text() %>% gsub(pattern = "[^0-9]", replacement = "") # Cited by 17143 -> 17143
      interests <- profile %>% html_elements(".gs_ai_one_int") %>% html_text()

      # scalar values instead of single-value vectors
      # or data.frame() could be used instead here
      list(
        profile_name = name[[1]],
        profile_link = link[[1]],
        profile_affiliations = affiliations[[1]],
        profile_email = email[[1]],
        profile_city_by_count = cited_by[[1]],
        profile_interests = interests
      )
    })

    # append profile results to the list
    all_profile_results <- c(all_profile_results, profile_results)

    # pagination
    next_page_button <- page %>% html_elements("button.gs_btnPR") %>% html_attr("onclick")
    if (!is.na(next_page_button)) {
      # extract the "after_author" parameter from the "onclick" attribute of the "Next" button using regex
      # and assign it to the "after_author" URL parameter which is the next token pagination
      # along with "astart" URL param
      params$after_author <- str_match(next_page_button, "after_author\\\\x3d(.*)\\\\x26")[, 2]
      params$astart <- params$astart + 10
    } else {
      profiles_is_present <- FALSE
    }
  }

  # convert to data frame
  all_profile_results <- data.frame(do.call(rbind, all_profile_results), stringsAsFactors = FALSE)

  return(all_profile_results)
}

# Scrape the data
data <- scrape_all_profiles_from_university(label="physics", university_name="Harvard University")

# Select all columns of the data frame using dplyr
all_data <- select(data, everything())

# Extract the email addresses using dplyr
emals <- all_data %>% pull(profile_email)
for (email in emals) {
  cat("- ", name, "\n")
}

Explanation

Import all the needed packages:

library(httr)
library(rvest)
library(jsonlite)
library(purrr)
library(stringr)
library(glue)
library(dplyr)

Next, we create a function with 2 arguments, label and university_name:

scrape_all_profiles_from_university <- function(label, university_name) {
  # ... code
}

The following step is to pass a browser user-agent to act like we're sending request as an actual user, not a bot that sends a request. Check what's your user-agent.

You can read more about bypassing topic from my reducing the chance of being blocked while web scraping blog post.

params list is used to create URL parameters for the reqesut and dynamically pass label and university_name data to the request.

headers <- c("User-Agent" = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36")

# remove trailing whitespaces/hidden characters
label <- trimws(label)
university_name <- trimws(university_name)

params <- list(
  view_op = "search_authors",
  mauthors = glue("label:{label} '{university_name}'"),
  hl = "en",
  astart = 0
)

# empty list to store future the profile results
all_profile_results <- list()

After that, we need to create a while loop that will be used to paginate through all pages dynamically no matter how many there're, it will go through all of them. We can use hardcoded approach (iterate from x to x page) but this is not reliable.

If you want to iterate over x amount of pages, you may create a varibale before the while loop that tells how many iteration should be done and a condition to check for the current iteration.

While making request, we're passing earlier created params and headers and reading HTML content and assigning it to the page variable:

profiles_is_present <- TRUE
while (profiles_is_present) {
  # on each params update, 'response' will be different (new page) and 'page' accordingly
  response <- GET("https://scholar.google.com/citations", query = params, add_headers(.headers = headers))
  page <- read_html(content(response, "text"))
  # ...
}

At this step we're iterating (map()) over HTML containers with .gs_ai_chpr CSS selector and extracting data.

profiles <- page %>% html_elements(".gs_ai_chpr")
  profile_results <- map(profiles, function(profile) {
    name <- profile %>% html_element(".gs_ai_name a") %>% html_text()
    link <- paste0("https://scholar.google.com", profile %>% html_element(".gs_ai_name a") %>% html_attr("href"))
    affiliations <- profile %>% html_element(".gs_ai_aff") %>% html_text(trim = TRUE)
    email <- profile %>% html_element(".gs_ai_eml") %>% html_text()
    cited_by <- profile %>% html_element(".gs_ai_cby") %>% html_text() %>% gsub(pattern = "[^0-9]", replacement = "") # Cited by 17143 -> 17143
    interests <- profile %>% html_elements(".gs_ai_one_int") %>% html_text()

Right after that, we append extracted data from the current iteration and append it to the all_profile_results by concatenating two lists profile_results and profile_results together:

# not really sure if [[1]] is needed
list(
  profile_name = name[[1]],
  profile_link = link[[1]],
  profile_affiliations = affiliations[[1]],
  profile_email = email[[1]],
  profile_city_by_count = cited_by[[1]],
  profile_interests = interests
)
})

# append profile results to the list
all_profile_results <- c(all_profile_results, profile_results)

At this point we get to the actual pagination.

Firstly, we extract onclick attribute from the button HTML element and assign it to next_page_button.
Secondly, we check if (!is.na(next_page_button)) (if button is present) otherwise exit the while loop of no button available.
Thirdly, we extract next page token from the button onclick attribute and pass it to params as a new key.
Lastly, we increment a 10 to a astart under params which is used in combination with after_authour parameter to drive pagination.
- astart 10 = 2nd page, 20 = 3rd page and so on.

next_page_button <- page %>% html_elements("button.gs_btnPR") %>% html_attr("onclick")
if (!is.na(next_page_button)) {
  params$after_author <- str_match(next_page_button, "after_author\\\\x3d(.*)\\\\x26")[, 2]
  params$astart <- params$astart + 10
} else {
  profiles_is_present <- FALSE
}

Finally, we convert all_profile_results to a dataframe:

all_profile_results <- data.frame(do.call(rbind, all_profile_results), stringsAsFactors = FALSE)
return(all_profile_results)

do.call(rbind) will stack all of the data to create a single matrix/data.
stringsAsFactors = FALSE will convert all column type to a string.

And as a final step, here's how we can access the data:

data <- scrape_all_profiles_from_university(label="physics", university_name="Harvard University")

all_data <- select(data, everything())

emals <- all_data %>% pull(profile_email)
for (email in emals) {
  cat("- ", name, "\n")
}

Outputs:

[1] "extracting authors at page #0"
[1] "extracting authors at page #10"
[1] "extracting authors at page #20"
[1] "extracting authors at page #30"
[1] "extracting authors at page #40"
[1] "extracting authors at page #50"
> ...
-  Verified email at neu.edu
-  Verified email at seas.harvard.edu
-  Verified email at physics.harvard.edu
-  Verified email at cfa.harvard.edu
-  Verified email at physics.harvard.edu
-  Verified email at cfa.harvard.edu
-  Verified email at seas.harvard.edu 
-  Verified email at mcb.harvard.edu
-
-  Verified email at mcgill.ca
-  Verified email at cfa.harvard.edu
-  Verified email at bidmc.harvard.edu
-  Verified email at physics.harvard.edu
-  Verified email at physics.harvard.edu
-  Verified email at hsph.harvard.edu
-  Verified email at g.harvard.edu
-  Verified email at fas.harvard.edu
-  Verified email at iisc.ac.in
-  Verified email at fas.harvard.edu
-  Verified email at fas.harvard.edu
-  Verified email at physics.harvard.edu
-  Verified email at fas.harvard.edu
-  Verified email at bwh.harvard.edu
-  Verified email at g.harvard.edu
-  Verified email at seas.harvard.edu
-  Verified email at g.harvard.edu 
-  Verified email at hsph.harvard.edu
-  Verified email at math.harvard.edu
-  Verified email at fas.harvard.edu
-  Verified email at fas.harvard.edu
-  Verified email at cfa.harvard.edu
-  Verified email at harvard.edu
-  Verified email at polytechnique.edu
-  Verified email at seas.harvard.edu
-  Verified email at fas.harvard.edu
-  Verified email at g.harvard.edu
-  Verified email at fas.harvard.edu
-  Verified email at g.harvard.edu
-  Verified email at seas.harvard.edu
-  Verified email at cfa.harvard.edu
-  Verified email at fas.harvard.edu
-  Verified email at fas.harvard.edu 
-  Verified email at college.harvard.edu
-  Verified email at fas.harvard.edu
-  Verified email at go.cambridgecollege.edu
-  Verified email at mgh.harvard.edu
-  Verified email at hsph.harvard.edu
-  Verified email at g.harvard.edu
-
-  Verified email at g.harvard.edu
-  Verified email at college.harvard.edu
-  Verified email at college.harvard.edu
-  Verified email at physics.uoc.gr

Google Scholar Python scraper alternatives

If you want to extract more data from Google Scholar in R but haven't figured it out, you can use a few of the Python alternatives if you're comfortable using it:

scrape-google-scholar-py is a open-source project of mine that aims to extract all the possible data from Google Scholar. In the future I'll port it to R.
scholarly is also an open-source project that extracts data from Google Scholar. The difference between this and mine package is that mine aim to extract all possible pages, while scholarly not.

Join us on Twitter | YouTube

Add a Feature Request💫 or a Bug🐞

How to install Python packages without a requirements.txt file with pipreqs

Dmitriy Zub ☀️ — Wed, 03 May 2023 11:32:16 +0000

Why bother

Say you get a project that doesn't have a requirements.txt file in it and that project has 20+ imports, meaning you need to install 20+ modules manually. Sound not that interesting.

That's when pipreqs comes into play as a "life saver". This tool will scan all scripts/folders in the current working directory (or where you want it to look by providing a path) and installs all the found packages.

Example usage

Solution

create virtual env
activate it
install pipreqs
tell pipreqs to look for files in the current folder "./" and use --encoding utf-8
- wait until requirements.txt is created
install script dependencies from created requirements.txt

Which results in this command:

# windows
python -m venv env && \
source env/Scripts/activate && \
pip install pipreqs && \
pipreqs --encoding utf-8 "./" && \
pip install -r requirements.txt && \
pip freeze > requirements.txt

# linux
python -m venv env && \
source env/source/activate && \
pip install pipreqs && \
pipreqs --encoding utf-8 "./" && \
pip install -r requirements.txt && \
pip freeze > requirements.txt

Low amount of imports example

Let's say you have a script like this:

import requests

response = requests.get('https://serpapi.com/playground')
print(response.html)

Big amount of imports example

The point of it is to show how all the modules install automatically without having an initial requirements.txt file.

Here will have a bigger amount of imports:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import plotly.express as px
import scipy.stats as stats
import statsmodels.api as sm
import sklearn
import yellowbrick
import wordcloud
import nltk
import spacy
import transformers
import streamlit as st

# Load and clean data
data = pd.read_csv('data.csv')
data.dropna(inplace=True)

# Descriptive statistics
print('Data Summary')
print(data.describe())

# Data visualization
sns.histplot(data['age'], kde=False, bins=10)
plt.title('Age Distribution')
plt.show()

px.scatter(data, x='income', y='age', color='gender', title='Income vs. Age')

# Correlation analysis
corr_matrix = data.corr()
sns.heatmap(corr_matrix, annot=True, cmap='coolwarm')
plt.title('Correlation Matrix')
plt.show()

# Statistical analysis
stat, p = stats.ttest_ind(data[data['gender']=='M']['income'], data[data['gender']=='F']['income'])
print(f'T-test: statistic={stat}, pvalue={p}')

# Machine learning
X = data[['age', 'income']]
y = data['gender']
model = sklearn.linear_model.LogisticRegression()
model.fit(X, y)
visualizer = yellowbrick.classifier.classification_report(model, X, y)
visualizer.show()

# Text analysis
text = 'This is a sample text for text analysis'
tokens = nltk.word_tokenize(text)
print(f'Tokenized text: {tokens}')

nlp = spacy.load('en_core_web_sm')
doc = nlp(text)
for token in doc:
    print(token.text, token.pos_)

model = transformers.pipeline('sentiment-analysis')
result = model(text)[0]
print(f'Sentiment analysis: {result["label"]}, score={result["score"]}')

# Streamlit app
st.title('Data Analysis App')
st.write('Data Summary')
st.write(data.describe())

Generated requirements.txt afterward:

altair==4.2.2
attrs==23.1.0
blinker==1.6.2
blis==0.7.9
cachetools==5.3.0
catalogue==2.0.8
certifi==2022.12.7
charset-normalizer==3.1.0
click==8.1.3
colorama==0.4.6
confection==0.0.4
contourpy==1.0.7
cssselect==1.2.0
cycler==0.11.0
cymem==2.0.7
decorator==5.1.1
docopt==0.6.2
entrypoints==0.4
filelock==3.12.0
fonttools==4.39.3
fsspec==2023.4.0
gitdb==4.0.10
GitPython==3.1.31
huggingface-hub==0.14.1
idna==3.4
importlib-metadata==6.6.0
Jinja2==3.1.2
jmespath==1.0.1
joblib==1.2.0
jsonschema==4.17.3
kiwisolver==1.4.4
langcodes==3.3.0
lxml==4.9.2
markdown-it-py==2.2.0
MarkupSafe==2.1.2
matplotlib==3.7.1
mdurl==0.1.2
murmurhash==1.0.9
nltk==3.8.1
numpy==1.24.3
packaging==23.1
pandas==2.0.1
parsel==1.8.1
pathy==0.10.1
patsy==0.5.3
Pillow==9.5.0
pipreqs==0.4.13
plotly==5.14.1
preshed==3.0.8
protobuf==3.20.3
pyarrow==12.0.0
pydantic==1.10.7
pydeck==0.8.1b0
Pygments==2.15.1
Pympler==1.0.1
pyparsing==3.0.9
pyrsistent==0.19.3
python-dateutil==2.8.2
pytz==2023.3
pytz-deprecation-shim==0.1.0.post0
PyYAML==6.0
regex==2023.5.4
requests==2.29.0
rich==13.3.5
scikit-learn==1.2.2
scipy==1.10.1
seaborn==0.12.2
six==1.16.0
smart-open==6.3.0
smmap==5.0.0
spacy==3.5.2
spacy-legacy==3.0.12
spacy-loggers==1.0.4
srsly==2.4.6
statsmodels==0.13.5
streamlit==1.22.0
tenacity==8.2.2
thinc==8.1.10
threadpoolctl==3.1.0
tokenizers==0.13.3
toml==0.10.2
toolz==0.12.0
tornado==6.3.1
tqdm==4.65.0
transformers==4.28.1
typer==0.7.0
typing_extensions==4.5.0
tzdata==2023.3
tzlocal==4.3
urllib3==1.26.15
validators==0.20.0
w3lib==2.1.1
wasabi==1.1.1
watchdog==3.0.0
wordcloud==1.9.1.1
yarg==0.1.9
yellowbrick==1.5
zipp==3.15.0

Current limitations

The only drawback for now, is that it doesn't recognize all packages as there're ~10+ related issues when pipreqs didn't recognize a package.

SerpApi Changelog: March, 2023

Dmitriy Zub ☀️ — Tue, 11 Apr 2023 08:21:48 +0000

Summary of March Updates

A sum-up of SerpApi's latest product improvements, and feature releases 🙂

Features

[Feature] Google Search API: add support to expanded "Top Sites"
[Feature] Google Maps Reviews API: add support to scrape reviews rating
[Feature] Google Local Results API: now sponsored (ad) results is a separate JSON key from regular, organic results
[Feature] Google Shopping Api: add support for "Top Features" section
[Feature] Google Search Results: add support to "Open Hours" answer box
[Feature] Knowledge Graph API: add support to extract "Average cost after aid", "Graduation rate", and "Acceptance rate"
[Feature] Google Shopping API: add support for "Others searched for" Block
[Feature] Google Shopping API: add support to extract price in the parentheses
[Feature] Home Depot API: add support to search by SKU / Product ID
[Feature] Google Search API: add support to a new "Things To Know" section in mobile results
[Feature] Google Search API: add support to extract "Refine this search" block
[Feature] Google Search API: add support for "Things to Know" section on Desktop
[Feature] Google Shopping API: add support for filter blocks "You might like", "People also search for"

Fixes

Projects built with SerpApi:

API Updates

[Feature] Google Search API: add support to expanded "Top Sites"

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Maps Reviews API: add support to scrape reviews rating

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Local Results API: now sponsored (ad) results is a separate JSON key from regular, organic results

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Shopping API: add support for "Top Features" section

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Search Results: add support to "Open Hours" answer box

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Knowledge Graph API: add support to extract "Average cost after aid", "Graduation rate", and "Acceptance rate"

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Shopping API: add support for "Others searched for" Block

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Shopping API: add support to extract price in the parentheses

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Home Depot API: add support to search by SKU / Product ID

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Search API: add support to a new "Things To Know" section in mobile results

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Search API: add support to extract "Refine this search" block

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Search API: add support for "Things to Know" section on Desktop

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Shopping API: add support for filter blocks "You might like", "People also search for"

Try in the Playground | Documentation | Public Roadmap Issue

Conclusion

If you notice any bugs or possible improvements, then you can directly improve the product by creating a issue in the SerpApi Public Roadmap.

Join us on Twitter | YouTube

💛

SerpApi Changelog: February, 2023

Dmitriy Zub ☀️ — Mon, 27 Mar 2023 06:17:17 +0000

Summary of February Updates

A sum-up of SerpApi's latest product improvements, and feature releases 🙂

Features

[Feature] Google Maps API: now scraping experiences section of results
[Feature] Google Jobs API: now scraping job highlight section
[Feature] Google Search API: now scraping date for Google images results
[Feature] Knowledge Graph API: now scraping working at and typical company salaries in Knowledge Graph
[Feature] Google Reverse Image Search: remove forced q parameter
[Feature] [Documentation]: now supports keyboard shortcuts for 'Search' menu
[Feature] Google Jobs Listing API: now scraping website and other data on the page
[Feature] Google Shopping API: now support deals related to your search
[Feature] Google Shopping API: now scraping second hand condition tags
[Feature] Google Shopping API: now scraping number of compare prices options on Google Shopping
[Feature] Google Shopping API: now scraping related searches

Fixes

Projects built with SerpApi:

API Updates

[Feature] Google Maps API: now scraping experiences section of results

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Jobs API: now scraping job highlight section

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Search API: now scraping date for Google images results

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Knowledge Graph API: now scraping working at and typical company salaries

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Reverse Image Search: remove forced `q` parameter from `serpapi_link`

Now it is possible to use Google Reverse Image API without a q parameter. And forced q parameter was also removed from the image_sizes array serpapi_link key.

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] [Documentation]: now supports keyboard shortcuts for 'Search' menu

Now you can search through the docs page using new "Search" menu.

Public Roadmap Issue

[Feature] Google Jobs Listing API: now scraping website and other data on the page

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Shopping API: now support deals related to your search

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Shopping API: now scraping second hand condition tags

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Shopping API: now scraping number of compare prices options on Google Shopping

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Shopping API: now scraping related searches

Try in the Playground | Documentation | Public Roadmap Issue

Conclusion

If you notice any bugs or possible improvements, then you can directly improve the product by creating a issue in the SerpApi Public Roadmap.

Join us on Twitter | YouTube

💛

SerpApi Changelog: January, 2023

Dmitriy Zub ☀️ — Wed, 15 Mar 2023 12:40:57 +0000

Summary of January Updates ⭐

Latest product improvements and feature releases of January 2023, which are still relevant and important, even though they were supposed to be published one month ago.

[Feature] Google Finance API: add new parser to extract Quote (ticker) data
[Feature] Google Answer Box API: now parsing extra information in local_time type
[Feature] Google Immersive Products API: now parsing immersive products block
[Feature] Google Maps API: now users can make search by place_id
[Feature] Knowledge Graph API: now parsing menu filter tabs
[Feature] Google Shopping API: now support mobile and tablet versions
[Fix] Knowledge Graph API: see_results_about not parsed correctly
[Fix] Yandex Search API: respond with no results
[Fix] Google Shopping API: now parsing source's direct link instead of Google's redirect link
[Fix] Google Finance API: fix extracting Graph item
[Fix] Google Search API: fix empty displayed_link key for mobile ads searches
[Fix] Google Search API: fix empty rating_count returned for local_ads
[Fix] Google Search API: fix parsing the number of pages for PDF result in the date key
[Fix] Google Trends API: fix parsing interest by region results
[Fix] Google Search API: fix inline_videos results being included into organic_results on mobile
[Fix] Google Search API Mobile: fix extracting Knowledge Graph tabs
[Fix] Apple Reviews API: fix maximum page depth limit
[Fix] Google Maps Photos API: User information now can be returned via photo_meta_serpapi_link key
[Fix] Google Shopping API: fix parsing CSS values in shopping_results

Slight changes:

[Google Knowledge Graph] tabs links are not parsed for some cases.
[Google Local "Places Results API] Returning Unexpected Results When Pagination Params are Supplied.
[Bing Search API] empty snippet for some organic_results.
[Google Shopping API] fix results basic queries such as "dog supplies".

Projects built with SerpApi:

eBay-Walmart Dropshipping tool in Python
eBay-Walmart Dropshipping tool in JavaScript
symbolicai
langchain-visualizer
Shazam-like telegram bot

API Updates

[Feature] Google Finance API: add new parser to extract Quote (ticker) data

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Answer Box API: now parsing extra information in `local_time` type

Fixed bug with local_time type.

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Immersive Products API: now parsing immersive products block

Added the ability to scrape and return to immersive products.

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Maps API: now users can make search by `place_id`

Now searching for a specific place could be done using place_id.

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Knowledge Graph API: now parsing menu filter tabs

Combined sections to return data for Knowledge Graph menu items.

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Shopping API: now support mobile and tablet versions

Try in the Playground | Documentation | Public Roadmap Issue

dimitryzub marked this conversation as resolved.

[Fix] Knowledge Graph API: `see_results_about` not parsed correctly

This update will allow you to extract all the data from the new layout for 'Knowledge Graph'.

Try in the Playground | Documentation | Public Roadmap Issue

[Fix] Yandex Search API: respond with no results

Fix Yandex Search API for responding results.

Try in the Playground | Documentation | Public Roadmap Issue

[Fix] Google Shopping API: now parsing source's direct link instead of Google's redirect link

Try in the Playground | Documentation | Public Roadmap Issue

[Fix] Google Finance API: fix extracting Graph item

Try in the Playground | Documentation | Public Roadmap Issue

[Fix] Google Search API: fix empty `displayed_link` key for mobile ads searches

Try in the Playground | Documentation | Public Roadmap Issue

[Fix] Google Search API: fix empty `rating_count` returned for `local_ads`

Added the ability to parse ratings_count for local ads.

Try in the Playground | Documentation | Public Roadmap Issue

[Fix] Google Search API: fix parsing the number of pages for PDF result in the `date` key

Try in the Playground | Documentation | Public Roadmap Issue

[Fix] Google Trends API: fix parsing interest by region results

Try in the Playground | Documentation | Public Roadmap Issue
dimitryzub marked this conversation as resolved.

[Fix] Google Search API: fix `inline_videos` results being included into `organic_results` on mobile

We fixed wrong direct links to video and debug hashed Google link to correct link.

Try in the Playground | Documentation | Public Roadmap Issue

[Fix] Google Search API Mobile: fix extracting Knowledge Graph tabs

Try in the Playground | Documentation? | Public Roadmap Issue

[Fix] Apple Reviews API: fix maximum page depth limit

Try in the Playground | Documentation | Public Roadmap Issue

[Fix] - Google Maps Photos API: User information now can be returned via `photo_meta_serpapi_link` key

We should be able to get the user information in Google Maps Photos API, according to our documentation. Right now it is returned.

Try in the Playground | Documentation | Public Roadmap Issue

[Fix] Google Shopping API: fix parsing CSS values in `shopping_results`

Try in the Playground | Documentation | Public Roadmap Issue

[Fix] Google Search API: answer_box show correct value in anwer_box in type field

Fixed an issue with uncorrect value in answer_box.

Try in the Playground | Documentation | Public Roadmap Issue

New Projects Made with SerpApi

eBay-Walmart Dropshipping tool in Python

dropshipping-tool-demo is a tool that compares products from Walmart and eBay to find a profit.

eBay-Walmart Dropshipping tool in JavaScript

Basic website SerpApi JS demo that shows how to create a Walmart and eBay dropshipping tool: https://app-comparison.netlify.app/

symbolicai

symbolicai - building applications with LLMs at its core through our Symbolic API leverages the power of classical and differentiable programming in Python

langchain-visualizer

langchain-visualizer - visualization and debugging tool for LangChain workflows.

Shazam like Telegram bot built in Python

A tutorial to create and deploy a telegram bot that takes in music in real-time, and helps you to search the title and the singer

Conclusion

If you notice any bugs or possible improvements, then you can directly influence the work by creating a problem in the SerpApi Public Roadmap.

Join us on Twitter | YouTube

💛

Implement SEO Cron Jobs with Replit and SerpApi Code

Dmitriy Zub ☀️ — Wed, 01 Mar 2023 10:22:01 +0000

Intro

This blog post will go over how to use an online IDE called Replit and how it could be used as a host option to host your SerpApi code to run cron jobs for your SEO, data extraction script, or anything else.

What is SerpApi?

SerpApi provides a convenient gateway to access raw data from Google and other search engines through JSON responses, meaning whatever you see in Google or other engines you can see in the response JSON.

It has support for 9 languages (and 2 coming soon, Swift and C++). Playground page to play around and experiment.

But why Replit?

Well, it's simpler from pretty much all sides than other similar tools (such as pythonanywhere) and it keeps only getting better and better, at least for now.

A few words about "Always on" feature

Please keep in mind that this example requires at least a Hacker plan that Replit offers, which currently costs $7/month. This plan is required to use of the "Always on" feature. The Replit plan costs $2 more than a pythonanywhere hacker plan, I think it's worth it.

Setting up Replit account and Repl

First, make sure you've registered an account, and then choose a Hacker plan:

Next, we need to create a repl where all code will be added:

Choose Python as a Template, name your repl however you like, make it public or private, and create the repl:

After the repl has been created, here are the main things we'll be touching, the same as in your IDE or text editor pretty much:

Install Packages

Now to the code itself. Add google-search-results under the "Packages" tool and install it:

Add a schedule package that will do a Cron job:

Code

Here's a very simple example code that runs every 5 seconds until N time and extracts the position of the target website.

Have a look at how the following example runs on Replit, in case you want (you need to change os.getenv("API_KEY") to your SerpApi key).

from serpapi import GoogleSearch
import os
import time
from datetime import datetime, time as datetime_time
import schedule


def job():
    TARGET_KEYWORD = 'coffee'
    TARGET_WEBSITE = 'healthline.com'
    QUERY = 'coffee tips'

    params = {
        "api_key": os.getenv("API_KEY"),             # your serpapi api key, https://serpapi.com/manage-api-key
        "engine": "google",                          # serpapi pareser engine
        "location": "Austin, Texas, United States",  # location of the search
        "q": QUERY,                                  # search query
        "gl": "us",                                  # country of the search, https://serpapi.com/google-countries
        "hl": "en",                                  # language of the search, https://serpapi.com/google-languages 
        "num": 100                                   # 100 results
    }

        search = GoogleSearch(params)
        results = search.get_dict()

        matched_results = []

        for result in results["organic_results"]:
            if TARGET_KEYWORD in result["title"].lower() and TARGET_WEBSITE in result["link"]:
                matched_results.append({
                    "position": result["position"],
                    "match_title": result["title"],
                    "match_link": result["link"]
                })

        print(datetime.now().strftime("%H:%M"))
        print(matched_results)

if __name__ == "__main__":
    scheduled_off_time = datetime_time(6, 11) # 6 hours, 11 minutes. Repl server local time
    schedule.every(5).seconds.until(scheduled_off_time).do(job)

    while True:
        schedule.run_pending()
        print("job pending ⌛")
        time.sleep(4)

        current_time = datetime.now().time()

        if current_time > scheduled_off_time:
            schedule.clear()
            print("Done ✅")
            break

Full execution output

job pending ⌛
job pending ⌛
job pending ⌛
https://serpapi.com/search
06:01
[{'position': 75, 'match_title': '8 Ways to Make Your Coffee Super Healthy - Healthline', 'match_link': 'https://www.healthline.com/nutrition/8-ways-to-make-your-coffee-super-healthy'}]
job pending ⌛
job pending ⌛
job pending ⌛
https://serpapi.com/search
06:01
[{'position': 75, 'match_title': '8 Ways to Make Your Coffee Super Healthy - Healthline', 'match_link': 'https://www.healthline.com/nutrition/8-ways-to-make-your-coffee-super-healthy'}]
job pending ⌛
job pending ⌛
job pending ⌛
https://serpapi.com/search
06:01
[{'position': 75, 'match_title': '8 Ways to Make Your Coffee Super Healthy - Healthline', 'match_link': 'https://www.healthline.com/nutrition/8-ways-to-make-your-coffee-super-healthy'}]
job pending ⌛
job pending ⌛
job pending ⌛
https://serpapi.com/search
06:01
[{'position': 75, 'match_title': '8 Ways to Make Your Coffee Super Healthy - Healthline', 'match_link': 'https://www.healthline.com/nutrition/8-ways-to-make-your-coffee-super-healthy'}]
job pending ⌛
job pending ⌛
job pending ⌛
https://serpapi.com/search
06:01
[{'position': 75, 'match_title': '8 Ways to Make Your Coffee Super Healthy - Healthline', 'match_link': 'https://www.healthline.com/nutrition/8-ways-to-make-your-coffee-super-healthy'}]
job pending ⌛
job pending ⌛
job pending ⌛
https://serpapi.com/search
06:01
[{'position': 75, 'match_title': '8 Ways to Make Your Coffee Super Healthy - Healthline', 'match_link': 'https://www.healthline.com/nutrition/8-ways-to-make-your-coffee-super-healthy'}]
job pending ⌛
job pending ⌛
job pending ⌛
https://serpapi.com/search
06:01
[{'position': 75, 'match_title': '8 Ways to Make Your Coffee Super Healthy - Healthline', 'match_link': 'https://www.healthline.com/nutrition/8-ways-to-make-your-coffee-super-healthy'}]
job pending ⌛
job pending ⌛
job pending ⌛
https://serpapi.com/search
06:01
[{'position': 75, 'match_title': '8 Ways to Make Your Coffee Super Healthy - Healthline', 'match_link': 'https://www.healthline.com/nutrition/8-ways-to-make-your-coffee-super-healthy'}]
job pending ⌛
job pending ⌛
job pending ⌛
Done ✅

Returned result:

[
   {
      "position":75,
      "match_title":"8 Ways to Make Your Coffee Super Healthy - Healthline",
      "match_link":"https://www.healthline.com/nutrition/8-ways-to-make-your-coffee-super-healthy"
   }
]

The one thing I want to emphasize is that make sure you see what the current time at the repl server is by running the command below.

This is to make sure you see what the server's local time is as it might be different from your local one.

from datetime import datetime
datetime.now().strftime("%H:%M:%S") # or without %S, %M

# 05:35:02

Setting up "Always on" feature

In addition to enabling the "Always on" feature, you can also enable "Boost repl" if you need more RAM, CPU, and storage:

Yay 🎉 Now your script will be always active and run until a certain time 🙂

With this approach, you can add tools such as Google Cloud Storage to extract the data and immediately upload it to the cloud bucket, and a lot other things.

Join us on Twitter | YouTube

Add a Feature Request💫 or a Bug🐞

8 Most Popular Python HTML Web Scraping Packages with Benchmarks

Dmitriy Zub ☀️ — Wed, 01 Feb 2023 10:41:38 +0000

Intro

This blog post will cover Python web scraping packages in terms of their speed, ease of use, and personal investigations. This blog post won't cover what webscraping is and how parsers work.

By the end of this blog post, you'll understand or have a better understanding of which parser/browser automation to use for your purpose.

This blog post is split into two parts:

Python HTML parsers and their benchmarks.
Python browser automation packages and their benchmarks.

The first section will be about speed comparison and ease of use. The second section will be specifically about Python's browser automation packages, memory consumption, and speed.

📌Please note that I'm not very advanced in benchmarking, feel free to comment on what feels wrong or confusing.

Used Hardware

Component	Specs
RAM	16GB
CPU	Intel 12th Gen i7-12700H, 14 cores, 4.7 MHz

Python Parsers Benchmarks

This section will compare 5 Python HTML parsers:

html.parser
html5lib
lxml
parsel
selectolax (HTML, Lexbor backends)

There're four tests with saved locally Google Organic search results HTML file:

100 iterations with 10 repeats.
500 iterations with 10 repeats.
1000 iterations with 10 repeats.
3000 iterations with 5 repeats.

Combined Chart Results

100 Iteration Results

Raw Data

[
   {
      "bs4_html_parser":{
         "raw":[
            4.3710737,
            3.8684199,
            3.8580279,
            3.8806571,
            3.7997756,
            3.8502996,
            3.7788926,
            3.840815,
            3.8426696,
            3.8693173
         ],
         "min":3.78,
         "max":4.37,
         "mean":3.9,
         "standard_deviation":0.17
      },
      "bs4_lxml_parser":{
         "raw":[
            3.4638336,
            3.4571164,
            3.4321829,
            3.3861049,
            3.3953615,
            3.3972832,
            3.4319847,
            3.4485809,
            3.4614765,
            3.4190135
         ],
         "min":3.39,
         "max":3.46,
         "mean":3.43,
         "standard_deviation":0.03
      },
      "bs4_html5lib_parser":{
         "raw":[
            9.2249861,
            9.203607,
            9.1739324,
            9.1611666,
            9.2419121,
            9.2673369,
            9.3951571,
            9.346496,
            9.3287497,
            9.3132658
         ],
         "min":9.16,
         "max":9.4,
         "mean":9.27,
         "standard_deviation":0.08
      },
      "parsel_parser":{
         "raw":[
            2.679202,
            2.6082995,
            2.6354441,
            2.6149692,
            2.5950211,
            2.5903689,
            2.6174196,
            2.6494348,
            2.5917189,
            2.6739687
         ],
         "min":2.59,
         "max":2.68,
         "mean":2.63,
         "standard_deviation":0.03
      },
      "lxml_parser":{
         "raw":[
            0.8436189,
            0.8283767,
            0.8420748,
            0.8255317,
            0.827338,
            0.8243309,
            0.8134264,
            0.8111648,
            0.8125786,
            0.8133089
         ],
         "min":0.81,
         "max":0.84,
         "mean":0.82,
         "standard_deviation":0.01
      },
      "selectolax_html_parser":{
         "raw":[
            0.18461,
            0.1845551,
            0.18227,
            0.1875291,
            0.1873954,
            0.1860093,
            0.19331,
            0.1936247,
            0.1898201,
            0.1888448
         ],
         "min":0.18,
         "max":0.19,
         "mean":0.19,
         "standard_deviation":0.0
      },
      "selectolax_lexbor_parser":{
         "raw":[
            0.1461264,
            0.1471886,
            0.1477886,
            0.148007,
            0.1501829,
            0.144999,
            0.1466642,
            0.1459928,
            0.148036,
            0.1476213
         ],
         "min":0.14,
         "max":0.15,
         "mean":0.15,
         "standard_deviation":0.0
      }
   }
]

500 Iteration Results

Raw Data

[
   {
      "bs4_html_parser":{
         "raw":[
            20.2197304,
            19.8741745,
            19.2251858,
            19.3863994,
            19.5261737,
            19.3143987,
            19.295948,
            19.5619137,
            19.3965105,
            19.5783358
         ],
         "min":19.23,
         "max":20.22,
         "mean":19.54,
         "standard_deviation":0.3
      },
      "bs4_lxml_parser":{
         "raw":[
            17.4766226,
            17.6523627,
            17.3481226,
            17.5603945,
            17.6367459,
            17.5745645,
            17.5120047,
            17.7037503,
            17.0720811,
            17.1215897
         ],
         "min":17.07,
         "max":17.7,
         "mean":17.47,
         "standard_deviation":0.22
      },
      "bs4_html5lib_parser":{
         "raw":[
            46.0190557,
            46.1886938,
            46.9830729,
            46.2576209,
            46.2123261,
            46.0697103,
            46.0612127,
            46.1187136,
            48.670253,
            47.7965513
         ],
         "min":46.02,
         "max":48.67,
         "mean":46.64,
         "standard_deviation":0.91
      },
      "parsel_parser":{
         "raw":[
            13.4378014,
            13.3847556,
            13.5100824,
            13.3942492,
            13.4126319,
            13.3633435,
            13.4016935,
            13.4611343,
            13.3859031,
            13.37833
         ],
         "min":13.36,
         "max":13.51,
         "mean":13.41,
         "standard_deviation":0.04
      },
      "lxml_parser":{
         "raw":[
            4.3305542,
            4.3557282,
            4.2665298,
            4.3775716,
            4.379404,
            4.3783643,
            4.3159682,
            4.3932252,
            4.43205,
            4.391286
         ],
         "min":4.27,
         "max":4.43,
         "mean":4.36,
         "standard_deviation":0.05
      },
      "selectolax_html_parser":{
         "raw":[
            0.9595209,
            0.9527344,
            0.9630006,
            0.9670192,
            0.9511656,
            0.9588291,
            0.9580593,
            0.9678788,
            0.940953,
            0.9192121
         ],
         "min":0.92,
         "max":0.97,
         "mean":0.95,
         "standard_deviation":0.01
      },
      "selectolax_lexbor_parser":{
         "raw":[
            0.7347563,
            0.7880385,
            0.7712946,
            0.803693,
            0.7808561,
            0.7661754,
            0.7927354,
            0.8141454,
            0.7798704,
            0.7893399
         ],
         "min":0.73,
         "max":0.81,
         "mean":0.78,
         "standard_deviation":0.02
      }
   }
]

1000 Iteration Results

Raw Data

[
   {
      "bs4_html_parser":{
         "raw":[
            39.1850317,
            38.8625512,
            38.9220413,
            38.52645,
            39.5012472,
            39.0812714,
            40.2874018,
            39.1607705,
            38.9987914,
            38.9538794
         ],
         "min":38.53,
         "max":40.29,
         "mean":39.15,
         "standard_deviation":0.47
      },
      "bs4_lxml_parser":{
         "raw":[
            35.340024,
            35.5665838,
            35.4370601,
            35.4310116,
            35.1316697,
            35.1764227,
            35.5666647,
            35.2332891,
            35.0854779,
            35.7218053
         ],
         "min":35.09,
         "max":35.72,
         "mean":35.37,
         "standard_deviation":0.21
      },
      "bs4_html5lib_parser":{
         "raw":[
            96.2746173,
            95.1716108,
            94.9976969,
            95.376245,
            95.5432712,
            96.4209762,
            94.7473696,
            95.6874917,
            95.34009,
            95.1788754
         ],
         "min":94.75,
         "max":96.42,
         "mean":95.47,
         "standard_deviation":0.53
      },
      "parsel_parser":{
         "raw":[
            27.1312845,
            26.8943964,
            32.5753347,
            47.7647626,
            40.4320701,
            27.203478,
            27.4574984,
            26.7016569,
            26.305777,
            28.7976675
         ],
         "min":26.31,
         "max":47.76,
         "mean":31.13,
         "standard_deviation":7.28
      },
      "lxml_parser":{
         "raw":[
            8.8578888,
            8.8009302,
            9.0597153,
            8.7252489,
            8.7669332,
            8.7292014,
            8.8213478,
            8.7004821,
            8.6868534,
            8.7972479
         ],
         "min":8.69,
         "max":9.06,
         "mean":8.79,
         "standard_deviation":0.11
      },
      "selectolax_html_parser":{
         "raw":[
            1.9501049,
            2.0294157,
            2.0303961,
            2.0856484,
            2.0366627,
            2.0423066,
            2.0580174,
            2.0388072,
            2.0790543,
            2.0126756
         ],
         "min":1.95,
         "max":2.09,
         "mean":2.04,
         "standard_deviation":0.04
      },
      "selectolax_lexbor_parser":{
         "raw":[
            1.6442625,
            1.6344478,
            1.6930229,
            1.6476543,
            1.6561128,
            1.6467175,
            1.6648112,
            1.6683134,
            1.6984335,
            1.6524308
         ],
         "min":1.63,
         "max":1.7,
         "mean":1.66,
         "standard_deviation":0.02
      }
   }
]

3000 Iteration Results

186% faster than bs4 with lxml backend 🙂

Raw Data

[
   {
      "bs4_html_parser":{
         "raw":[
            136.3143269,
            113.9956494,
            114.0446546,
            121.5711856,
            171.5756146
         ],
         "min":114.0,
         "max":171.58,
         "mean":131.5,
         "standard_deviation":24.18
      },
      "bs4_lxml_parser":{
         "raw":[
            217.9561803,
            216.2607809,
            192.2252402,
            143.3660566,
            123.4686671
         ],
         "min":123.47,
         "max":217.96,
         "mean":178.66,
         "standard_deviation":43.11
      },
      "bs4_html5lib_parser":{
         "raw":[
            316.4003634,
            285.8371232,
            284.3877244,
            287.4475911,
            285.5628814
         ],
         "min":284.39,
         "max":316.4,
         "mean":291.93,
         "standard_deviation":13.72
      },
      "parsel_parser":{
         "raw":[
            77.7046641,
            78.2331405,
            78.0829456,
            78.017651,
            78.879304
         ],
         "min":77.7,
         "max":78.88,
         "mean":78.18,
         "standard_deviation":0.43
      },
      "lxml_parser":{
         "raw":[
            24.8936843,
            24.9198819,
            24.8717849,
            24.8783482,
            25.0041617
         ],
         "min":24.87,
         "max":25.0,
         "mean":24.91,
         "standard_deviation":0.05
      },
      "selectolax_html_parser":{
         "raw":[
            5.466414,
            5.4846724,
            5.4738816,
            5.4023584,
            5.4395063
         ],
         "min":5.4,
         "max":5.48,
         "mean":5.45,
         "standard_deviation":0.03
      },
      "selectolax_lexbor_parser":{
         "raw":[
            4.6869728,
            4.7313916,
            4.9446686,
            4.7700584,
            4.8052036
         ],
         "min":4.69,
         "max":4.94,
         "mean":4.79,
         "standard_deviation":0.1
      }
   }
]

Personal Investigations

html5lib is the slowest parser of all of the mentioned parsers.
BeautifulSoup html.parser and lxml parser has a very slight change in results. Interesting that in 3000 iterations bs4 lxml backend was slower. Didn't understand why.
It's interesting that lxml parser is used in both bs4 and parsel but parsel outperforms bs4 with lxml backend.
selectolax HTML and Lexbor parsers are incredibly fast. Besides speed, I've noticed (not measured in this blog post) that the Lexbor parser used almost half less memory compared to all other parsers.
lxml was the hardest to use as it required XPath knowledge. This point is subjective, however, worth noting. Writing XPath takes some time and some additional knowledge. If you were writing XPath before, then there'll be no issues.
selectolax in terms of syntax is very similar to both bs4 and parsel, which is handy. One thing to keep in mind is that selectolax currently doesn't support XPath.

Python Parsers Benchmark Code

import timeit
import lxml # installing from wheel if on Python 3.11 https://stackoverflow.com/a/74546098/15164646
import json
from lxml import html
from bs4 import BeautifulSoup
from parsel import Selector
from selectolax.parser import HTMLParser
from selectolax.lexbor import LexborHTMLParser

import pandas as pd
import numpy as np


def get_html():
    # locally saved HTML
    with open('google-search-test.html', mode='r', encoding='utf-8') as html_file:
        html_page = html_file.read()
    return html_page

def bs4_html_parser(html_page):
    soup = BeautifulSoup(html_page, 'html.parser')

    for result in soup.select('.tF2Cxc'):
        title = result.select_one('.DKV0Md').text
        snippet = result.select_one('.lyLwlc').text
        link = result.select_one('.yuRUbf a')['href']
        displayed_link = result.select_one('.tjvcx').text

    for related_search in soup.select('.s8bAkb'):
        related_title = related_search.select_one('.AB4Wff').text
        related_link = related_search['href']

    for video in soup.select('.dFd2Tb'):
        video_title = video.select_one('.DKV0Md').text
        video_lenght = video.select_one('.R4Cuhd .J1mWY div')['aria-label']

    for knowledge_graph in soup.select(".rVusze"):
        if knowledge_graph.select_one(".kno-fv"):
            knowledge_graph_key = knowledge_graph.select_one('.w8qArf .fl').text
            knowledge_graph_value = knowledge_graph.select_one('.kno-fv').text
            knowledge_graph_value_link = knowledge_graph.select_one('.kno-fv a')['href']

def bs4_lxml_parser(html_page):
    soup = BeautifulSoup(html_page, 'lxml')

    for result in soup.select('.tF2Cxc'):
        title = result.select_one('.DKV0Md').text
        snippet = result.select_one('.lyLwlc').text
        link = result.select_one('.yuRUbf a')['href']
        displayed_link = result.select_one('.tjvcx').text

    for related_search in soup.select('.s8bAkb'):
        related_title = related_search.select_one('.AB4Wff').text
        related_link = related_search['href']

    for video in soup.select('.dFd2Tb'):
        video_title = video.select_one('.DKV0Md').text
        video_lenght = video.select_one('.R4Cuhd .J1mWY div')['aria-label']

    for knowledge_graph in soup.select(".rVusze"):
        if knowledge_graph.select_one(".kno-fv"):
            knowledge_graph_key = knowledge_graph.select_one('.w8qArf .fl').text
            knowledge_graph_value = knowledge_graph.select_one('.kno-fv').text
            knowledge_graph_value_link = knowledge_graph.select_one('.kno-fv a')['href']

def bs4_html5lib_parser(html_page):
    soup = BeautifulSoup(html_page, 'html5lib')

    for result in soup.select('.tF2Cxc'):
        title = result.select_one('.DKV0Md').text
        snippet = result.select_one('.lyLwlc').text
        link = result.select_one('.yuRUbf a')['href']
        displayed_link = result.select_one('.tjvcx').text

    for related_search in soup.select('.s8bAkb'):
        related_title = related_search.select_one('.AB4Wff').text
        related_link = related_search['href']

    for video in soup.select('.dFd2Tb'):
        video_title = video.select_one('.DKV0Md').text
        video_lenght = video.select_one('.R4Cuhd .J1mWY div')['aria-label']

    for knowledge_graph in soup.select(".rVusze"):
        if knowledge_graph.select_one(".kno-fv"):
            knowledge_graph_key = knowledge_graph.select_one('.w8qArf .fl').text
            knowledge_graph_value = knowledge_graph.select_one('.kno-fv').text
            knowledge_graph_value_link = knowledge_graph.select_one('.kno-fv a')['href']

def parsel_parser(html_page):
    selector = Selector(text=html_page)

    for result in selector.css('.tF2Cxc'):
        title = result.css('.DKV0Md::text').get()
        snippet = result.css('.lyLwlc::text').get()
        link = result.css('.yuRUbf a::attr(href)').get()
        displayed_link = result.css('.tjvcx::text').get()

    for related_search in selector.css('.s8bAkb'):
        related_title = related_search.css('.AB4Wff::text').get()
        related_link = related_search.attrib['href']

    for video in selector.css('.dFd2Tb'):
        video_title = video.css('.DKV0Md::text').get()
        video_lenght = video.css('.R4Cuhd .J1mWY div::attr(aria-label)')

    for knowledge_graph in selector.css(".rVusze"):
        if knowledge_graph.css(".kno-fv").get():
            knowledge_graph_key = knowledge_graph.css('.w8qArf .fl::text').get()
            knowledge_graph_value = knowledge_graph.css('.kno-fv::text').get()
            knowledge_graph_value_link = knowledge_graph.css('.kno-fv a::attr(href)').get()

def lxml_parser(html_page):
    tree = lxml.html.fromstring(html_page)

    for result in tree.xpath('//div[contains(@class, "tF2Cxc")]'):
        title = result.xpath('.//h3[contains(@class, "DKV0Md")]/text()')[0]
        snippet = result.xpath('.//div[contains(@class, "lyLwlc")]/text()')
        link = result.xpath('.//div[contains(@class, "yuRUbf")]/a/@href')[0]
        displayed_link = result.xpath('.//cite[contains(normalize-space(@class), "tjvcx")]')[0].text_content()

    for related_search in tree.xpath('//a[contains(@class, "s8bAkb")]'):
        related_title = related_search.xpath('.//div[contains(@class, "OhScic")]')[0].text_content()
        related_link = related_search.xpath('.//@href')[0]

    for video in tree.xpath('//div[contains(@class, "dFd2Tb")]'):
        video_title = video.xpath('.//h3[contains(@class, "DKV0Md")]/text()')[0]
        video_lenght = video.xpath('.//div[contains(@class, "J1mWY")]/div/@aria-label')[0]

    # returns a lists of data
    knowledge_graph_key = tree.xpath('//span[contains(@class, "w8qArf")]/a[contains(@class, "fl")]/text()')
    knowledge_graph_value = tree.xpath('//span[contains(@class, "kno-fv")]/a/text()')
    knowledge_graph_value_link = tree.xpath('//span[contains(@class, "kno-fv")]/a/@href')

def selectolax_html_parser(html_page):
    parser = HTMLParser(html_page)

    for result in parser.css('.tF2Cxc'):
        title = result.css_first('.DKV0Md').text()
        snippet = result.css_first('.lyLwlc').text()
        link = result.css_first('.yuRUbf a').attrs['href']
        displayed_link = result.css_first('.tjvcx').text()

    for related_search in parser.css('.s8bAkb'):
        related_title = related_search.css_first('.AB4Wff').text()
        related_link = related_search.attrs['href']

    for video in parser.css('.dFd2Tb'):
        video_title = video.css_first('.DKV0Md').text()
        video_lenght = video.css_first('.R4Cuhd .J1mWY div').attrs['aria-label']

    for knowledge_graph in parser.css(".rVusze"):
        if knowledge_graph.css_first(".kno-fv"):
            knowledge_graph_key = knowledge_graph.css_first('.w8qArf .fl').text()
            knowledge_graph_value = knowledge_graph.css_first('.kno-fv').text()
            knowledge_graph_value_link = knowledge_graph.css_first('.kno-fv a').attrs['href']

def selectolax_lexbor_parser(html_page):
    parser = LexborHTMLParser(html_page)

    for result in parser.css('.tF2Cxc'):
        title = result.css_first('.DKV0Md').text()
        snippet = result.css_first('.lyLwlc').text()
        link = result.css_first('.yuRUbf a').attrs['href']
        displayed_link = result.css_first('.tjvcx').text()

    for related_search in parser.css('.s8bAkb'):
        related_title = related_search.css_first('.AB4Wff').text()
        related_link = related_search.attrs['href']

    for video in parser.css('.dFd2Tb'):
        video_title = video.css_first('.DKV0Md').text()
        video_lenght = video.css_first('.R4Cuhd .J1mWY div').attrs['aria-label']

    for knowledge_graph in parser.css(".rVusze"):
        if knowledge_graph.css_first(".kno-fv"):
            knowledge_graph_key = knowledge_graph.css_first('.w8qArf .fl').text()
            knowledge_graph_value = knowledge_graph.css_first('.kno-fv').text()
            knowledge_graph_value_link = knowledge_graph.css_first('.kno-fv a').attrs['href']


if __name__ == '__main__':
    html_page = get_html()

    parsers_benchmark_data = {
        'bs4_html_parser': [],
        'bs4_lxml_parser': [],
        'bs4_html5lib_parser': [],
        'parsel_parser': [],
        'lxml_parser': [],
        'selectolax_html_parser': [],
        'selectolax_lexbor_parser': []
    }

    iterations = 1000
    repeat = 10

    for parser_name in parsers_benchmark_data:
        benchmark_data = timeit.repeat(
            stmt=f'{parser_name}(html_page)',
            setup=f'from __main__ import {parser_name}, html_page',
            number=iterations,
            repeat=repeat
        )

        parsers_benchmark_data[parser_name].append({
            'raw': benchmark_data,
            'min': round(min(benchmark_data), 2),
            'max': round(max(benchmark_data), 2),
            'mean': round(sum(benchmark_data) / len(benchmark_data), 2),
            'standard_deviation': round(np.std(benchmark_data, ddof=1), 2),
        })
    print(json.dumps(parsers_benchmark_data, indent=2))
    pd.DataFrame(data=parsers_benchmark_data).to_json('browser-automation-benchmark-data-10-iterations.json', orient='records')

Python Browser Automation

The test was fairly basic to see memory consumption, and speed. Each script makes a request to Google Search and paginates through 10 pages.

Browser Automation Profiling Results

Tests were run with the following command:

$ mprof run <file_name>.py
$ mprof plot # to plot the results (matplotlib is required)

Personal Investigations

Surprisingly selenium was a little faster than requests-html. Or maybe not surprisingly :)
playwright was the slowest.
selenium had a hard time with complex CSS selectors, for example .d6cvqb a[id=pnnext] (.get_attribute() also didn't work). That's why for in range() was used instead. I know it can be done with XPath but I wanted to use CSS selectors only, although requests-html uses only XPath.
Memory consumption was pretty much identical between the 3 packages.

Browser Automation Code

from playwright.sync_api import sync_playwright
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from requests_html import HTMLSession

from memory_profiler import memory_usage
from memory_profiler import profile

@profile 
def playwright_parser():
    # https://stackoverflow.com/questions/73043207/how-to-start-playwright-outside-with-without-context-managers
    playwright = sync_playwright().start()
    chromium = playwright.chromium
    browser = chromium.launch(headless=True, timeout=0)
    page = browser.new_page()

    page_num = 0
    while True:
        page.goto(f'https://www.google.com/search?q=minecraft&gl=us&hl=en&start={page_num}')

        for result in page.query_selector_all('.tF2Cxc'):
            title = result.query_selector('.DKV0Md').inner_text()
            try:
                snippet = result.query_selector('.lyLwlc').inner_text()
            except: snippet = None
            link = result.query_selector('.yuRUbf a').get_attribute('href')
            displayed_link = result.query_selector('.tjvcx').inner_text()

        for related_search in page.query_selector_all('.s8bAkb'):
            related_title = related_search.query_selector('.AB4Wff').inner_text()
            related_link = related_search.get_attribute('href')

        for video in page.query_selector_all('.dFd2Tb'):
            video_title = video.query_selector('.DKV0Md').inner_text()
            video_lenght = video.query_selector('.R4Cuhd .J1mWY div').get_attribute('aria-label')

        for knowledge_graph in page.query_selector_all(".rVusze"):
            if knowledge_graph.query_selector(".kno-fv"):
                knowledge_graph_key = knowledge_graph.query_selector('.w8qArf .fl').inner_text()
                knowledge_graph_value = knowledge_graph.query_selector('.kno-fv').inner_text()
                knowledge_graph_value_link = knowledge_graph.query_selector('.kno-fv a').get_attribute('href')

        if page.query_selector('.d6cvqb a[id=pnnext]'):
            page_num += 1
        else:
            break
    browser.close()
    playwright.stop()

@profile
def selenium_parser(): 
    service = Service(executable_path='<full_path_to>\chromedriver.exe')
    options = webdriver.ChromeOptions()
    options.add_argument('--headless')
    options.add_experimental_option('excludeSwitches', ['enable-logging'])

    driver = webdriver.Chrome(service=service, options=options)

    for page_num in range(0, 11):
        driver.get(f'https://www.google.com/search?q=minecraft&gl=us&hl=en&start={page_num}')

        for result in driver.find_elements(By.CSS_SELECTOR, '.tF2Cxc'):
            title = result.find_element(By.CSS_SELECTOR, '.DKV0Md').text
            try:
                snippet = result.find_element(By.CSS_SELECTOR, '.lyLwlc').text
            except: snippet = None
            link = result.find_element(By.CSS_SELECTOR, '.yuRUbf a').get_attribute('href')
            displayed_link = result.find_element(By.CSS_SELECTOR, '.tjvcx').text

        for related_search in driver.find_elements(By.CSS_SELECTOR, '.s8bAkb'):
            related_title = related_search.find_element(By.CSS_SELECTOR, '.AB4Wff').text
            related_link = related_search.get_attribute('href')

        for video in driver.find_elements(By.CSS_SELECTOR, '.dFd2Tb'):
            video_title = video.find_element(By.CSS_SELECTOR, '.DKV0Md').text
            video_lenght = video.find_element(By.CSS_SELECTOR,'.R4Cuhd .J1mWY div').get_attribute('aria-label')
        for knowledge_graph in driver.find_elements(By.CSS_SELECTOR, '.rVusze'):
            if knowledge_graph.find_element(By.CSS_SELECTOR, '.kno-fv'):
                knowledge_graph_key = knowledge_graph.find_element(By.CSS_SELECTOR, '.w8qArf .fl').text
                knowledge_graph_value = knowledge_graph.find_element(By.CSS_SELECTOR, '.kno-fv').text
                knowledge_graph_value_link = knowledge_graph.find_element(By.CSS_SELECTOR, '.kno-fv a').get_attribute('href')
    driver.close()

@profile    
def requests_html_parser():
    session = HTMLSession()

    page_num = 0
    while True:
        soup = session.get(f'https://www.google.com/search?q=minecraft&gl=us&hl=en&start={page_num}')

        for result in soup.html.xpath('//div[contains(@class, "tF2Cxc")]', first=False):
            title = result.xpath('.//h3[contains(@class, "DKV0Md")]/text()', first=True)
            snippet = result.xpath('.//div[contains(@class, "lyLwlc")]/text()', first=True)
            link = result.xpath('.//div[contains(@class, "yuRUbf")]/a/@href', first=True)
            displayed_link = result.xpath('.//cite[contains(normalize-space(@class), "tjvcx")]', first=True).text

        for related_search in soup.html.xpath('//a[contains(@class, "s8bAkb")]', first=False):
            related_title = related_search.xpath('.//div[contains(@class, "OhScic")]', first=True).text
            related_link = related_search.xpath('.//@href', first=True)

        for video in soup.html.xpath('//div[contains(@class, "dFd2Tb")]', first=False):
            video_title = video.xpath('.//h3[contains(@class, "DKV0Md")]/text()', first=True)
            video_lenght = video.xpath('.//div[contains(@class, "J1mWY")]/div/@aria-label', first=True)

        knowledge_graph_key = soup.html.xpath('//span[contains(@class, "w8qArf")]/a[contains(@class, "fl")]/text()', first=False)
        knowledge_graph_value = soup.html.xpath('//span[contains(@class, "kno-fv")]/a/text()', first=False)
        knowledge_graph_value_link = soup.html.xpath('//span[contains(@class, "kno-fv")]/a/@href', first=False)

        if soup.html.xpath('//td[contains(@class, "d6cvqb")]/a/@id', first=True):
                page_num += 1
        else:
            break

if __name__ == '__main__':
    playwright_parser()
    requests_html_parser()
    selenium_parser()

Conclusions

If you need to scrape data from a dynamic page that doesn't require clicking, scrolling and similar things but still requires rendering JavaScript, try requests-html. It uses pure XPath as lxml and should be faster than the other two browser automations.
If you need to do complex page manipulation on the dynamic page, try to use playwright or selenium.
If you scraping non-dynamic pages (rendered via JavaScript), try selectolax over bs4, lxml or parsel. It's a lot faster, uses less memory, and has almost identical syntax to parsel or bs4. A hidden gem I would say.
If you need to use XPath in your parser, try to use either lxml or parsel. parsel is built on top of lxml and translates every CSS query to XPath and can combine (chain) CSS and XPath queries. However, lxml is faster.

Making YouTube Mentions Tracker in Python

Dmitriy Zub ☀️ — Tue, 10 Jan 2023 12:23:36 +0000

Intro
App Demo
Virtual Environment and Libraries Installation
Full Code
Code Explanation
Links

Intro

This blog post is created to show you how SerpApi was integrated into this small app with an explanation of how the app functionates and SerpApi's role in it. We'll go from top to bottom of the code shown below.

What about the demo itself, this demo project was inspired by a tool called Mention.

This project does kind of the same thing but for YouTube videos only. It transcribes videos (even if the video is without captions) and finds the target keyword and returns a URL with the timestamp.

The app doesn't scan/detect new videos all the time via cron job or something similar.

App Demo

Live demo:

youtube-mentions-tracker.streamlit.app

Virtual Environment and Libraries Installation

If you want to follow along with your API key (instead of mine), follow these steps:

Clone repository:

$ git clone https://github.com/dimitryzub/youtube-mention-tracker.git

Install dependencies:

$ cd youtube-mention-tracker && pip install -r requriements.txt

Add SerpApi API key for the current shell and all processes started from the current shell:

# used to parse youtube videos, has a plan of 100 free searches
$ export SERPAPI_API_KEY=<your-api-key>

Run the app (this will open streamlit in your browser):

$ cd youtube-tracker && streamlit run tracker.py

Additionally, you might need to install FFmpeg. To reduce text duplication, have a look at how to Install FFmpeg on Windows, Mac, Linux Ubuntu, and Debian from VideoProc or any other resource on the internet that shows how to do it.

Full Code

from urllib.parse import (parse_qsl, urlsplit)
from serpapi import YoutubeSearch
from pytube import YouTube
import pytube.exceptions as exceptions
from queue import Queue
import streamlit as st
import whisper
import re, os, time
import subprocess
import pathlib
from stqdm import stqdm
import math, base64
import pandas as pd
from dotenv import load_dotenv, find_dotenv


load_dotenv(find_dotenv())

VIDEOS_DOWNLOAD_PATH = pathlib.Path(__file__).parent.resolve() / 'videos' 

if VIDEOS_DOWNLOAD_PATH.exists():  
    subprocess.run(['rm', '-rf', f'{VIDEOS_DOWNLOAD_PATH}']) # remove videos on each new run

# create videos folder if not exist. Temporary store videos.
if not VIDEOS_DOWNLOAD_PATH.exists():
    #TODO: change `videos/` to VIDEOS_DOWNLOAD_PATH or to subprocess.run(['rm', '-rf', f'{VIDEOS_DOWNLOAD_PATH}']) command  
    subprocess.run(['rm', '-r', 'videos/'])
    subprocess.run(['mkdir', 'videos/'])


def main():
    # TODO: add a blog post link to "SerpApi Demo Project"
    footer_modified = '''
            <style>
                #MainMenu {visibility: hidden;}
                footer {visibility: hidden;}
                .footer {
                    position: fixed;
                    left: 0;
                    bottom: 0;
                    width: 100%;
                    background-color: white;
                    color: black;
                    text-align: center;
                    font-size: 8px;
                }
            </style>
            <div class="footer">
                <p align="center" style="padding-top: 200px;">SerpApi Demo Project (<a href="https://github.com/dimitryzub/youtube-mention-tracker">repo</a>)<br>Made with <a href="https://streamlit.io/">Streamlit</a>, <a href="http://serpapi.com/">SerpApi</a>, <a href="https://github.com/pytube/pytube">PyTube</a>, <a href="https://github.com/openai/whisper">Whisper</a> 🧡</p>
            </div>
    '''
    st.markdown(footer_modified, unsafe_allow_html=True) 

    st.title('📺YouTube Videos Mention Tracker')
    st.markdown(body="This demo parses, downloads and transcribes YouTube videos to find target mention(s) inside the video(s). It's similar to [Mention](https://mention.com/en/) but for videos.")

    if 'visibility' not in st.session_state:
        st.session_state.visibility = 'visible'
        st.session_state.disabled = False

    SEARCH_QUERY: str = st.text_input(label='Search query', placeholder='Data Nerd', help='Multiple search queries is not supported')      # used to parsing youtube videos
    TARGET_KEYWORD: str = st.text_input(label='Target keyword', placeholder='SQL', help='Multiple target keywords is not supported')  # used for in the speech recognition
    NUMBER_OF_VIDEOS_TO_ANALYZE: int = st.slider(label='Number of videos to analyze', min_value=1, max_value=20, help='By default, extracted videos will be no longer than 20 minutes. Its done by filtering YouTube results with [`sp`](https://serpapi.com/youtube-search-api#api-parameters-advanced-youtube-parameters-sp) URL parameter.')
    SAVE_OPTION = st.selectbox(label='Choose file format to save', options=(None, 'CSV'), help='By default data wont be saved. Choose CSV format if you want to save the results to CSV file.')
    # PAGINATION = st.checkbox(label='Enable pagination') # increase amount of videos to parse

    # submit button
    col1, col2, col3 , col4, col5 = st.columns(5)
    with col1:
        pass
    with col2:
        pass
    with col4:
        pass
    with col5:
        pass
    with col3:
        submit_button_holder = st.empty()
        submit_search = submit_button_holder.button(label='Find mentions') # centered button

    if submit_search and not SEARCH_QUERY:
        st.error(body='Looks like you click a button without a search query. Please enter a search query 👆')
        st.stop()

    if submit_search and SEARCH_QUERY and NUMBER_OF_VIDEOS_TO_ANALYZE:
        search_queue = Queue()

        params = {
            'api_key': os.getenv('SERPAPI_API_KEY'),  # https://serpapi.com/manage-api-key
            'engine': 'youtube',                      # search engine
            'sp': 'EgIYAw%253D%253D',                 # filter to control length of the video: https://serpapi.com/youtube-search-api#api-parameters-advanced-youtube-parameters-sp
            'device': 'desktop',                      # device type
            'search_query': SEARCH_QUERY,             # search query
            'async': True                             # async batch requests
        }

        search = YoutubeSearch(params)                # where data extraction happens
        results = search.get_dict()                   # JSON -> Python dict

        search_queue.put(results)

        videos = []

        with st.spinner(text='Parsing YouTube Videos...'):
            while not search_queue.empty():
                result = search_queue.get()
                search_id = result['search_metadata']['id']

                # print(f'Get search from archive: {search_id}')
                search_archived = search.get_search_archive(search_id) # where all extracted data is stored and accessed

                # print(f"Search ID: {search_id}, Status: {search_archived['search_metadata']['status']}")

                if re.search(r'Cached|Success', search_archived['search_metadata']['status']):
                    for video_result in search_archived.get('video_results', []):
                        if video_result.get('title') not in videos:
                            # can't contain emojies in the file name
                            title = video_result.get('title') \
                                .replace('|', '')\
                                .replace('/', '')\
                                .replace('?', '')\
                                .replace(':', '')\
                                .replace('<', '')\
                                .replace('>', '')\
                                .replace('\\','')\
                                .replace('', '') \
                                .replace('*', '')

                            absolute_title_path = pathlib.Path(title).absolute()
                            print(absolute_title_path)

                            videos.append({
                                'title': title,
                                'link': video_result.get('link'),
                                'file_path': f"{absolute_title_path}.mp4"
                            })

                        if len(videos) == NUMBER_OF_VIDEOS_TO_ANALYZE:
                            print(f'downloading {len(videos)} videos')
                            break

                    # increase amount of videos to parse
                    # if PAGINATION:    
                    #     if 'next' in search_archived.get('serpapi_pagination', {}):
                    #         search.params_dict.update(dict(parse_qsl(urlsplit(search_archived['serpapi_pagination']['next']).query)))

                    #         new_page_result = search.get_dict() # new results from updated (next) page
                    #         search_queue.put(new_page_result)   # add to queue results from updated (next) page
                else:
                    # print(f'Requeue search: {search_id}')
                    search_queue.put(result)

        parsing_is_success = st.success('Done parsing 🎉')
        time.sleep(1)

        if parsing_is_success:
            parsing_is_success.empty()

            with st.spinner(text='Downloading YouTube Videos...'):
                download_info = st.info(body='Note: downloading speed depends on video length (the longer the video the more time it take to download) and your internet speed 📡')

                # https://discuss.streamlit.io/t/stqdm-a-tqdm-like-progress-bar-for-streamlit/10097
                for video in stqdm(videos):
                    youtube_downloader = YouTube(url=video['link'])
                    print(f'Downloading {video["link"]}')

                    try: 
                        # download only audio from youtube video
                        youtube_downloader.streams \
                            .filter(only_audio=True) \
                            .first() \
                            .download(
                                output_path=VIDEOS_DOWNLOAD_PATH,
                                filename=video['file_path']
                                )
                    except exceptions.LiveStreamError: 
                        print(f"Video {video['link']} is a livestream, couldn't download.")
                        pass
                    except exceptions.VideoUnavailable:
                        print(f'Video {video["link"]} unavailable, skipping.')
                        pass

            downloading_videos_success = st.success('Done downloading 🎉')

            time.sleep(5)
            downloading_videos_success.empty()
            download_info.empty()
            submit_button_holder.empty()

        # transcribe with whisper 
        transcript_data = []

        if VIDEOS_DOWNLOAD_PATH.exists():
           with st.spinner(text='Transcribing Videos...'):
                transcript_note = st.info(body='Note: it may take some time to process all audio files, especially if video is 20+ minutes long.')

                # iterate through video files to transcribe
                model = whisper.load_model('base')

                for video in videos:
                    transcribed_audio = model.transcribe(video["file_path"], fp16=False)

                    # generic check to check if transcibe text is present to do further tasks
                    # it could be [] or str: "... ... ... ... ...", if it's a song with no text
                    if transcribed_audio['text']:
                        for segment in transcribed_audio['segments']:
                            if TARGET_KEYWORD in segment['text'].lower():
                                transcript_data.append({
                                    'video_title': video['title'],
                                    'timestamp_url': f'{video["link"]}&t={math.floor(segment["start"])}', # <url>&t=488s
                                    # 'timestamp': segment['start'],
                                    'text': segment['text']
                                })
                                # print(segment)
                                # print(f'Found target keyword from {video["title"]}, timestamp: {segment["start"]}')
                                # print(f'Text: {segment["text"]}')
                    else: pass

        transcipt_success = st.success('Done transcribing 🎉')

        time.sleep(4)
        transcript_note.empty()
        transcipt_success.empty()

        transcribed_results_info = st.markdown(body='#### Transcribed results')
        TRANSCRIPT_TABLE = st.table(data=transcript_data)

        time.sleep(3)

        # start over
        with col1:
            pass
        with col2:
            pass
        with col4:
            pass
        with col5:
            pass
        with col3:
            start_over_button_holder = st.empty()
            start_over_button = st.button(label='Start over') # centered button

        if SAVE_OPTION == 'CSV' and transcript_data: 
            # save to CSV with Pandas
            save_csv(df=transcript_data) # draw a download CSV anchor tag  

            start_over_info_holder = st.empty()
            start_over_info_holder.error(body="To rerun the script, click on the 'Start over' button, or refresh the page.")

            if start_over_button:  
                subprocess.run(['rm', '-rf', f'{VIDEOS_DOWNLOAD_PATH}'])

                TRANSCRIPT_TABLE.empty()
                transcribed_results_info.empty()
                start_over_button_holder.empty() 
                start_over_info_holder.empty()

        if SAVE_OPTION == 'CSV' and not transcript_data:
            TRANSCRIPT_TABLE.empty()
            transcribed_results_info.empty()

            no_data_holder = st.empty()
            no_data_holder.error('No target keyword found. Click *Start Over* button and try different keyword.')

            if start_over_button:  
                subprocess.run(['rm', '-rf', f'{VIDEOS_DOWNLOAD_PATH}'])

                no_data_holder.empty()
                start_over_button_holder.empty()

        if SAVE_OPTION is None and transcript_data:
            start_over_info_holder = st.empty()
            start_over_info_holder.error(body="To rerun the script, click on the 'Start over' button, or refresh the page.")

            if start_over_button:  
                subprocess.run(['rm', '-rf', f'{VIDEOS_DOWNLOAD_PATH}'])

                TRANSCRIPT_TABLE.empty()
                transcribed_results_info.empty()
                start_over_button_holder.empty() 
                start_over_info_holder.empty()

        if SAVE_OPTION is None and not transcript_data:
            TRANSCRIPT_TABLE.empty()
            transcribed_results_info.empty()

            no_data_holder = st.empty()
            no_data_holder.error('No target keyword found. Click *Start Over* button and try different keyword.')   

            if start_over_button:  
                subprocess.run(['rm', '-rf', f'{VIDEOS_DOWNLOAD_PATH}'])

                TRANSCRIPT_TABLE.empty()
                no_data_holder.empty()
                transcribed_results_info.empty()
                start_over_button_holder.empty()

def save_csv(df): 
    # https://stackoverflow.com/a/68071190/15164646
    csv_file = pd.DataFrame(data=df).to_csv(index=False)
    b64_csv_file = base64.b64encode(csv_file.encode()).decode()

    href = f'<a href="data:file/csv;base64,{b64_csv_file}" download="youtube-transcript.csv" >Download CSV</a>'
    st.markdown(href, unsafe_allow_html=True)


if __name__ == '__main__':
    main()

Code Explanation

After a bunch of imports, we need to load SerpApi API key:

load_dotenv(find_dotenv())

The dotenv will load any globally available envs or from the .env file. In our case, we have used export SERPAPI_API_KEY=<your-api-key> to store API key in the current terminal session, meaning when the terminal will be closed, API key no longer be active.

Next, we define videos download path using pathlib. This path will be used to temporarily store downloaded videos from YouTube that further be used in the transcript:

VIDEOS_DOWNLOAD_PATH = pathlib.Path(__file__).parent.resolve() / 'videos'

After that, we check if videos/ path exists or not:

if VIDEOS_DOWNLOAD_PATH.exists():  
    subprocess.run(['rm', '-rf', f'{VIDEOS_DOWNLOAD_PATH}']) # remove videos on each new run

# create videos folder if not exist. Temporary store videos.
if not VIDEOS_DOWNLOAD_PATH.exists():
    # remove folder and create new one afterwards.
    # wihtout subprocess.run(['rm', '-r', 'videos/']) can't be created, saying that it already exists. 
    subprocess.run(['rm', '-r', 'videos/'])
    subprocess.run(['mkdir', 'videos/'])

Next, we define a main function where everything happens and right away I was modifying the default streamlit footer (optional) with custom HTML that passed to st.markdown, and added <h1> title and description of the ap:

def main():
    footer_modified = '''
            <style>
                #MainMenu {visibility: hidden;}
                footer {visibility: hidden;}
                .footer {
                    position: fixed;
                    left: 0;
                    bottom: 0;
                    width: 100%;
                    background-color: white;
                    color: black;
                    text-align: center;
                    font-size: 8px;
                }
            </style>
            <div class="footer">
                <p align="center" style="padding-top: 200px;">SerpApi Demo Project (<a href="https://github.com/dimitryzub/youtube-mention-tracker">repo</a>)<br>Made with <a href="https://streamlit.io/">Streamlit</a>, <a href="http://serpapi.com/">SerpApi</a>, <a href="https://github.com/pytube/pytube">PyTube</a>, <a href="https://github.com/openai/whisper">Whisper</a> 🧡</p>
            </div>
    '''
    st.markdown(footer_modified, unsafe_allow_html=True)

    st.title('📺YouTube Videos Mention Tracker')
    st.markdown(body="This demo parses, downloads and transcribes YouTube videos to find target mention(s) inside the video(s). It's similar to [Mention](https://mention.com/en/) but for videos.")

Next is to define streamlit session state. I've used this to hide or unhide certain widgets:

if 'visibility' not in st.session_state:
    st.session_state.visibility = 'visible'
    st.session_state.disabled = False

After that, I defined a few input fields, a slider, and a select box. Pagination left commented as it just takes too long to transcribe videos:

SEARCH_QUERY: str = st.text_input(label='Search query', placeholder='Data Nerd', help='Multiple search queries is not supported')      # used to parsing youtube videos
TARGET_KEYWORD: str = st.text_input(label='Target keyword', placeholder='SQL', help='Multiple target keywords is not supported')  # used for in the speech recognition
NUMBER_OF_VIDEOS_TO_ANALYZE: int = st.slider(label='Number of videos to analyze', min_value=1, max_value=20, help='By default, extracted videos will be no longer than 20 minutes. Its done by filtering YouTube results with [`sp`](https://serpapi.com/youtube-search-api#api-parameters-advanced-youtube-parameters-sp) URL parameter.')
SAVE_OPTION = st.selectbox(label='Choose file format to save', options=(None, 'CSV'), help='By default data wont be saved. Choose CSV format if you want to save the results to CSV file.')
# PAGINATION = st.checkbox(label='Enable pagination') # increase amount of videos to parse

Here I was creating a centered button (with kind of weird way I found on streamlit Discourse forum)

 # submit button
col1, col2, col3 , col4, col5 = st.columns(5)
with col1:
    pass
with col2:
    pass
with col4:
    pass
with col5:
    pass
with col3:
    submit_button_holder = st.empty()
    submit_search = submit_button_holder.button(label='Find mentions') # centered button

if submit_search and not SEARCH_QUERY:
    st.error(body='Looks like you click a button without a search query. Please enter a search query 👆')
    st.stop()

submit_button_holder is used to hide or unhide widget.
st.stop is used to stop the script if user didn't provide any search query.

If the user entered a search query and hits "Find mentions" button, the following code executes:

if submit_search and SEARCH_QUERY and NUMBER_OF_VIDEOS_TO_ANALYZE:
    search_queue = Queue()

    params = {
        'api_key': os.getenv('SERPAPI_API_KEY'),  # https://serpapi.com/manage-api-key
        'engine': 'youtube',                      # search engine
        'sp': 'EgIYAw%253D%253D',                 # filter to control length of the video: https://serpapi.com/youtube-search-api#api-parameters-advanced-youtube-parameters-sp
        'device': 'desktop',                      # device type
        'search_query': SEARCH_QUERY,             # search query
        'async': True                             # async batch requests
    }

    search = YoutubeSearch(params)                # where data extraction happens
    results = search.get_dict()                   # JSON -> Python dict

    search_queue.put(results)

    videos = []

    with st.spinner(text='Parsing YouTube Videos...'):
        while not search_queue.empty():
            result = search_queue.get()
            search_id = result['search_metadata']['id']

            # print(f'Get search from archive: {search_id}')
            search_archived = search.get_search_archive(search_id) # where all extracted data is stored and accessed

            # print(f"Search ID: {search_id}, Status: {search_archived['search_metadata']['status']}")

            if re.search(r'Cached|Success', search_archived['search_metadata']['status']):
                for video_result in search_archived.get('video_results', []):
                    if video_result.get('title') not in videos:
                        # can't contain emojies in the file name
                        title = video_result.get('title') \
                            .replace('|', '')\
                            .replace('/', '')\
                            .replace('?', '')\
                            .replace(':', '')\
                            .replace('<', '')\
                            .replace('>', '')\
                            .replace('\\','')\
                            .replace('', '') \
                            .replace('*', '')

                        absolute_title_path = pathlib.Path(title).absolute()

                        videos.append({
                            'title': title,
                            'link': video_result.get('link'),
                            'file_path': f"{absolute_title_path}.mp4"
                        })

                    if len(videos) == NUMBER_OF_VIDEOS_TO_ANALYZE:
                        print(f'downloading {len(videos)} videos')
                        break

                # increase amount of videos to parse
                # if PAGINATION:    
                #     if 'next' in search_archived.get('serpapi_pagination', {}):
                #         search.params_dict.update(dict(parse_qsl(urlsplit(search_archived['serpapi_pagination']['next']).query)))

                #         new_page_result = search.get_dict() # new results from updated (next) page
                #         search_queue.put(new_page_result)   # add to queue results from updated (next) page
            else:
                # print(f'Requeue search: {search_id}')
                search_queue.put(result)

    parsing_is_success = st.success('Done parsing 🎉')
    time.sleep(1)

Quite a few things happening. First of all, this is a SerpApi part. It extracts YouTube URLs for further usage. Additionally, this example shows SerpApi's async approach instead of sync, meaning each result are not being waited to be completed which improves parsing speed quite a lot. However, this only affects when the PAGINATION option is enabled.

If you want to see a comparison, I've written a blog post specifically about it: SerpApi Async Requests with Pagination using Python. In short, I've record a 434% speed improvement when making requests with YouTube API. I believe speed could be improved further.

Here I defined a search_queue for async searches. Defined a params with all the search parameters, and at the end put() results to a search_queue.

search_queue = Queue()

params = {
    'api_key': os.getenv('SERPAPI_API_KEY'),  # https://serpapi.com/manage-api-key
    'engine': 'youtube',                      # search engine
    'sp': 'EgIYAw%253D%253D',                 # filter to control length of the video: https://serpapi.com/youtube-search-api#api-parameters-advanced-youtube-parameters-sp
    'device': 'desktop',                      # device type
    'search_query': SEARCH_QUERY,             # search query
    'async': True                             # async batch requests
}

search = YoutubeSearch(params)                # where data extraction happens
results = search.get_dict()                   # JSON -> Python dict

search_queue.put(results)

This portion of the code below does actual parsing. We're looping through a search_queue until it's empty(). Then check if the search status is either "Cached" or "Success", and extract the title and link + creating an absolute title path from the title.

A lot of replace methods are used to remove forbidden characters for filenames. If the search status is not "Success" it will put it to the search_queue at the bottom to process it again:

videos = []

with st.spinner(text='Parsing YouTube Videos...'):
    while not search_queue.empty():
        result = search_queue.get() # access search result from the queue
        search_id = result['search_metadata']['id']

        # print(f'Get search from archive: {search_id}')
        search_archived = search.get_search_archive(search_id) # where all extracted data is stored and accessed
        # print(f"Search ID: {search_id}, Status: {search_archived['search_metadata']['status']}")

        if re.search(r'Cached|Success', search_archived['search_metadata']['status']):
            for video_result in search_archived.get('video_results', []):
                if video_result.get('title') not in videos:
                    # can't contain emojies in the file name
                    title = video_result.get('title') \
                        .replace('|', '')\
                        .replace('/', '')\
                        .replace('?', '')\
                        .replace(':', '')\
                        .replace('<', '')\
                        .replace('>', '')\
                        .replace('\\','')\
                        .replace('', '') \
                        .replace('*', '')

                    absolute_title_path = pathlib.Path(title).absolute()

                    videos.append({
                        'title': title,
                        'link': video_result.get('link'),
                        'file_path': f"{absolute_title_path}.mp4"
                    })

                if len(videos) == NUMBER_OF_VIDEOS_TO_ANALYZE:
                    print(f'downloading {len(videos)} videos')
                    break

            # increase amount of videos to parse
            # if PAGINATION:    
            #     if 'next' in search_archived.get('serpapi_pagination', {}):
            #         search.params_dict.update(dict(parse_qsl(urlsplit(search_archived['serpapi_pagination']['next']).query)))

            #         new_page_result = search.get_dict() # new results from updated (next) page
            #         search_queue.put(new_page_result)   # add to queue results from updated (next) page
        else:
            # print(f'Requeue search: {search_id}')
            search_queue.put(result)

parsing_is_success = st.success('Done parsing 🎉')
time.sleep(1)

After all the parsing is done, we need to download videos from returned URLs using pytube. Here we only download audio with filter(only_audio=True) option as this is what we only need for transcript and defining output_path to create videos/ path with filename that was done at the parsing stage, i.e absolute filename.

if parsing_is_success:
    parsing_is_success.empty() # hides text

    with st.spinner(text='Downloading YouTube Videos...'):
        download_info = st.info(body='Note: downloading speed depends on video length (the longer the video the more time it take to download) and your internet speed 📡')

        # https://discuss.streamlit.io/t/stqdm-a-tqdm-like-progress-bar-for-streamlit/10097
        for video in stqdm(videos):
            youtube_downloader = YouTube(url=video['link'])
            print(f'Downloading {video["link"]}')

            try: 
                # download only audio from youtube video
                youtube_downloader.streams \
                    .filter(only_audio=True) \
                    .first() \
                    .download(
                        output_path=VIDEOS_DOWNLOAD_PATH,
                        filename=video['file_path']
                        )
            except exceptions.LiveStreamError: 
                print(f"Video {video['link']} is a livestream, couldn't download.")
                pass
            except exceptions.VideoUnavailable:
                print(f'Video {video["link"]} unavailable, skipping.')
                pass

    downloading_videos_success = st.success('Done downloading 🎉')

    time.sleep(5)
    # hiding all previously used widgets
    downloading_videos_success.empty()
    download_info.empty()
    submit_button_holder.empty()

Next is the main part, audio transcription with whisper. Here I'm checking for video path existence (and not handling exceptions in any way 👀), load_model() for a transcript, and iterating over video file paths to get audio for the transcript.

If a transcribed text was found, I'm iterating over ['segments'] which contains all the needed info such as found text, and at what point it was found (really handy to generate YouTube timestamp URL):

transcript_data = []

if VIDEOS_DOWNLOAD_PATH.exists():
    with st.spinner(text='Transcribing Videos...'):
        transcript_note = st.info(body='Note: it may take some time to process all audio files, especially if video is 20+ minutes long.')

        # iterate through video files to transcribe
        model = whisper.load_model('base')

        for video in videos:
            transcribed_audio = model.transcribe(video["file_path"], fp16=False)

            # generic check to check if transcibe text is present to do further tasks
            # it could be [] or str: "... ... ... ... ...", if it's a song with no text
            if transcribed_audio['text']:
                for segment in transcribed_audio['segments']:
                    if TARGET_KEYWORD in segment['text'].lower():
                        transcript_data.append({
                            'video_title': video['title'],
                            'timestamp_url': f'{video["link"]}&t={math.floor(segment["start"])}', # <url>&t=488
                            # 'timestamp': segment['start'],
                            'text': segment['text']
                        })
            else: pass # use better exception handling than me

transcipt_success = st.success('Done transcribing 🎉')

time.sleep(4)
# hiding widgets
transcript_note.empty()
transcipt_success.empty()

transcribed_results_info = st.markdown(body='#### Transcribed results')
TRANSCRIPT_TABLE = st.table(data=transcript_data) # rendering table with found transcript results

And almost the final chuck of the code is to reset results to start a new search. Here I'm creating a new button to start the search again. Button with label creation could be done with the custom created function, for example create_button(label: str, button_name, button_holder_name) or something similar to reduce code duplication.

Then I'm checking SAVE_OPTIONs defined when a user submitted search. If it's CSV, then I'm calling the addition function to render the download button with found data in the CSV. If nothing was found, an error appears.

When the start_over_button gets hit, all video files in the videos/ directory get deleted for a new search:

 # start over
with col1:
    pass
with col2:
    pass
with col4:
    pass
with col5:
    pass
with col3:
    start_over_button_holder = st.empty()
    start_over_button = st.button(label='Start over') # centered button

if SAVE_OPTION == 'CSV' and transcript_data: 
    # save to CSV with Pandas
    save_csv(df=transcript_data) # draw a download CSV anchor tag  

    start_over_info_holder = st.empty()
    start_over_info_holder.error(body="To rerun the script, click on the 'Start over' button, or refresh the page.")

    if start_over_button:  
        subprocess.run(['rm', '-rf', f'{VIDEOS_DOWNLOAD_PATH}'])

        TRANSCRIPT_TABLE.empty()
        transcribed_results_info.empty()
        start_over_button_holder.empty() 
        start_over_info_holder.empty()

if SAVE_OPTION == 'CSV' and not transcript_data:
    TRANSCRIPT_TABLE.empty()
    transcribed_results_info.empty()

    no_data_holder = st.empty()
    no_data_holder.error('No target keyword found. Click *Start Over* button and try different keyword.')

    if start_over_button:  
        subprocess.run(['rm', '-rf', f'{VIDEOS_DOWNLOAD_PATH}'])

        no_data_holder.empty()
        start_over_button_holder.empty()

if SAVE_OPTION is None and transcript_data:
    start_over_info_holder = st.empty()
    start_over_info_holder.error(body="To rerun the script, click on the 'Start over' button, or refresh the page.")

    if start_over_button:  
        subprocess.run(['rm', '-rf', f'{VIDEOS_DOWNLOAD_PATH}'])

        TRANSCRIPT_TABLE.empty()
        transcribed_results_info.empty()
        start_over_button_holder.empty() 
        start_over_info_holder.empty()

if SAVE_OPTION is None and not transcript_data:
    TRANSCRIPT_TABLE.empty()
    transcribed_results_info.empty()

    no_data_holder = st.empty()
    no_data_holder.error('No target keyword found. Click *Start Over* button and try different keyword.')   

    if start_over_button:  
        subprocess.run(['rm', '-rf', f'{VIDEOS_DOWNLOAD_PATH}'])

        TRANSCRIPT_TABLE.empty()
        no_data_holder.empty()
        transcribed_results_info.empty()
        start_over_button_holder.empty()

def save_csv(df): 
    # https://stackoverflow.com/a/68071190/15164646
    csv_file = pd.DataFrame(data=df).to_csv(index=False)
    b64_csv_file = base64.b64encode(csv_file.encode()).decode()

    href = f'<a href="data:file/csv;base64,{b64_csv_file}" download="youtube-transcript.csv" >Download CSV</a>'
    st.markdown(href, unsafe_allow_html=True)

In the end I've added a if __name__ == '__main__' idiom which protects users from accidentally invoking the script when they didn't intend to, and call the main function which will run the whole script:

if __name__ == '__main__':
    main()

Pagination Techniques to Scrape Data from any Website in Python

Dmitriy Zub ☀️ — Fri, 16 Dec 2022 08:26:54 +0000

Intro
Dynamic vs Hardcoded Pagination
Prerequisites
Types of pagination
Token Pagination
- Dynamic Pagination with Token based Websites
Non Token Pagination
- Dynamic Pagination with Non Token based Websites
Click Pagination
- Dynamic Pagination with Clicks
Scroll or JavaScript Evaluation
- Dynamic Pagination with Scrolls
Conclusion

Intro

In this blog post will go over most frequent pagination techniques that could be applied to perform dynamic pagination on any website. This blog post is ongoing and will be updated if new techniques will be discovered.

Dynamic vs Hardcoded Pagination

What the heck is dynamic pagination?

Well, it's simply a way to paginate through all available pages without you knowing how many there are, it will just go through them all:

while True:
    requests.get('<website_url>')
    # data extraction code

    # condition to paginate to the next page or to exit pagination

Hardcoded approach differs by explicitly writing the N number of pages we want to paginate over:

# hardcoded way to paginate from 1 to 25th page
for page_num in range(1, 26):
    requests.get('<website_url>')
    # data extraction code

It's an easy approach if we need to extract data from N number of pages. But what if we need to extract all pages from several, say categories on the same website, and if each category contains different number of pages?

The thing is, when using hardcoded approach, we'll come to the point where we need to update page numbers to meet requirements for every page which is not particularly satisfying :-)

Dynamic pagination exit condition

Let's also stop for a second and see what another difference is dynamic while True pagination and for page_num in range(...) approach.

Have you noticed comment in the dynamic pagination: "condition to paginate to the next page or to exit pagination"?

This means that whenever we use a dynamic pagination we always need some condition to exit the infinite loop. It could be: element disappeared, previous page number is different than current, height of the elements are the same, etc.

Prerequisites

If you want to try along, let's create a separate environment first.

If you're on Linux:

python -m venv env && source env/bin/activate

If you're on Windows and using Git Bash:

python -m venv env && source env/Scripts/activate

Next, install needed libraries if you want to try yourself:

$ pip install requests bs4 parsel playwright

requests: make a request to a website.
bs4: HTML parser.
parsel: Another HTML parser, faster than bs4, used in a few examples.
playwright: modern browser automation.

For playwright, if you're on Linux, you also need to install additional dependencies:

$ sudo apt-get install -y libnss3 libnspr4 libatk1.0-0 libatk-bridge2.0-0 libcups2 libdrm2 libxkbcommon0 libxcomposite1 libxdamage1 libxfixes3 libxrandr2 libgbm1 libatspi2.0-0 libwayland-client0

After that we need to install chromium (or other browsers):

$ playwright install chromium

Types of pagination

There're four most frequent types of pagination:

token pagination using unique token.
- For example: SAOxcijdaf#Ad21
non-token pagination using digits.
- For example: 1,2,3,4,5.
click pagination.
- For example: clicking on the next page button until button disappears.
scroll or JavaScript evaluation pagination.
- For example: scrolling page until no more reviews left. Same could be done (evaluate) with JS code.

📌Those types of pagination could be combined with one or another.

For example, non-token and token pagination values need to be updated at the same time to paginate to the next page. This is how Google Scholar Profile pagination works without using browser automation.

Another example of combined pagination is combining scrolls with clicks as they become needed (when a certain button appears).

Token Pagination

Token pagination is when website generates a token responsible for retrieving next page data. It could look something like this: 27wzAGn-__8J.

This token most likely will be passed as URL parameter, for example on Google Scholar Profiles page it looks like this:

#                                                     ▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼
https://scholar.google.com/citations?mauthors=biology&after_author=27wzAGn-__8J

In some cases this token need to be combined with other parameters. For example, Google Scholar Profile page has:

astart parameter which responsible for page number.
after_author parameter that holds next page token responsible for next page.

#                                                     ▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼▼
https://scholar.google.com/citations?mauthors=biology&after_author=27wzAGn-__8J&astart=10

Dynamic Pagination with Token based Websites

Dynamic pagination on token based websites happens through parsing next page token which can be located in the:

Page source (CTRL + U), somewhere in the inline JSON.
HTML element attribute.
XHR request.

Here's a code snippet from my StackOverflow answer of paginating through all Google Scholar Profiles in Python:

from bs4 import BeautifulSoup
import requests, lxml, re

params = {
    "view_op": "search_authors", # profiles tab
    "mauthors": "blizzard",      # search query
    "hl": "en",                  # language of the search
    "astart": 0                  # page number
}

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
}

authors_is_present = True
while authors_is_present:

    html = requests.get("https://scholar.google.com/citations", params=params, headers=headers, timeout=30)
    soup = BeautifulSoup(html.text, "lxml")

    for author in soup.select(".gs_ai_chpr"):
        name = author.select_one(".gs_ai_name a").text
        link = f'https://scholar.google.com{author.select_one(".gs_ai_name a")["href"]}'
        affiliations = author.select_one(".gs_ai_aff").text
        email = author.select_one(".gs_ai_eml").text
        try:
            cited_by = re.search(r"\d+", author.select_one(".gs_ai_cby").text).group() # Cited by 17143 -> 17143
        except: cited_by = None

        print(f"extracting authors at page #{params['astart']}.",
                name,
                link,
                affiliations,
                email,
                cited_by, sep="\n")

    # if next page token exists, we extract next page token form HTML node attribute
    # and increment `astart` parameter +10
    if soup.select_one("button.gs_btnPR")["onclick"]:
        params["after_author"] = re.search(r"after_author\\x3d(.*)\\x26", str(soup.select_one("button.gs_btnPR")["onclick"])).group(1)  # -> XB0HAMS9__8J
        params["astart"] += 10
    else:
        authors_is_present = False

Non Token Pagination

Non-token pagination is simply when you increment page number by an N number. It could be incremented by 1, 10 (Google Search), 11 (Bing Search), 100 (Google Scholar Author Articles), or other number depending on how pagination on certain website functionates.

As mentioned above, non-token pagination could be combined with token in order to perform pagination.

Dynamic Pagination with Non Token based Websites

To identify if website uses non-token based pagination is simple. Keep an eye on URL parameters, see if there're any digits associated with URL parameters, and see if they are changing.

For example, Google Search has a start parameter:

# first page (no start parameter, or could be manually set to 0, first page)
https://www.google.com/search?q=sushi

# second page                         ▼▼▼▼▼▼▼▼
https://www.google.com/search?q=sushi&start=10

# third page                          ▼▼▼▼▼▼▼▼
https://www.google.com/search?q=sushi&start=20

A code example of non-token pagination is Google Search results. The following code scrapes all results from all pages.

from bs4 import BeautifulSoup
import requests, lxml

# https://docs.python-requests.org/en/master/user/quickstart/#passing-parameters-in-urls
params = {
    "q": "sushi",       # search query
    "hl": "en",         # language
    "gl": "us",         # country of the search, US -> USA
    "start": 0,         # number page by default up to 0
}

# https://docs.python-requests.org/en/master/user/quickstart/#custom-headers
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
}

page_num = 0

while True:
    page_num += 1
    print(f"{page_num} page:")

    html = requests.get("https://www.google.com/search", params=params, headers=headers, timeout=30)
    soup = BeautifulSoup(html.text, 'lxml')

    for result in soup.select(".tF2Cxc"):
        title = f'Title: {result.select_one("h3").text}'
        link = f'Link: {result.select_one("a")["href"]}'
        try:
            description = f'Description: {result.select_one(".VwiC3b").text}'
        except: 
            description = None

        print(title, link, description, sep="\n", end="\n\n")

    # if arrow button with attribute 'pnnext' is present -> paginate
    if soup.select_one('.d6cvqb a[id=pnnext]'):
        params["start"] += 10
    else:
        break

Click Pagination

As you understand, it performs a click and can only be used with browser automation such as playwright or selenium because these libraries provide method to click on a given element.

Dynamic Pagination with Clicks

All we need to do is to find the button or whatever element responsible for next page button via CSS selector or XPath.

After that we need to perform a click() method:

# page = playwright.chromium.launch(headless=True).new_page()
# page.goto('<website_url>')
page.query_selector('.Jwxk6d .u4ICaf button').click(force=True)

Scroll or JavaScript Evaluation

The scroll pagination technique requires scrolls in order to perform pagination. Scrolls could be either top-bottom or side to side depending on how a website works.

Dynamic Pagination with Scrolls

There're three frequent methods that I use to perform scrolls with either playwright or selenium:

Pressing a keyboard button to perform a scroll down:

# page = playwright.chromium.launch(headless=True).new_page()
# page.goto('<website_url>')
page.keyboard.press('END') # scrolls to possible end of the page

Evaluating JavaScript code to perform a scroll down:

# page = playwright.chromium.launch(headless=True).new_page()
# page.goto('<website_url>')
page.evaluate("""let scrollingElement = (document.scrollingElement || document.body);
              scrollingElement.scrollTop = scrollingElement scrollHeight;""")

📌We have to keep in mind that whenever we use a scroll pagination, we always need to perform a condition check that checks height of a certain element before and after scroll.

If height before and after scroll is the same, this will be a signal that there's more space for additional scrolls:

last_height = page.evaluate('() => document.querySelector(".fysCi").scrollTop')  # 2200

    while True:
        print("scrolling..")
        page.keyboard.press("End")
        time.sleep(3)

        new_height = page.evaluate('() => document.querySelector(".fysCi").scrollTop') # 2800

        if new_height == last_height:
            break
        else:
            last_height = new_height

Here's a complete example from one of my blog posts with step-by-step explanation about scraping all Google Play App Reviews in Python that uses click, keyboard press, and evaluate to check for current height:

import time, json, re
from parsel import Selector
from playwright.sync_api import sync_playwright


def run(playwright):
    page = playwright.chromium.launch(headless=True).new_page()
    page.goto("https://play.google.com/store/apps/details?id=com.collectorz.javamobile.android.books&hl=en_GB&gl=US")

    user_comments = []

    # if "See all reviews" button present
    if page.query_selector('.Jwxk6d .u4ICaf button'):
        print("the button is present.")

        print("clicking on the button.")
        page.query_selector('.Jwxk6d .u4ICaf button').click(force=True)

        print("waiting a few sec to load comments.")
        time.sleep(4)

        last_height = page.evaluate('() => document.querySelector(".fysCi").scrollTop')  # 2200

        while True:
            print("scrolling..")
            page.keyboard.press("End")
            time.sleep(3)

            new_height = page.evaluate('() => document.querySelector(".fysCi").scrollTop')

            if new_height == last_height:
                break
            else:
                last_height = new_height

    selector = Selector(text=page.content())
    page.close()

    print("done scrolling. Exctracting comments...")
    for index, comment in enumerate(selector.css(".RHo1pe"), start=1):

        comment_likes = comment.css(".AJTPZc::text").get()   

        user_comments.append({
            "position": index,
            "user_name": comment.css(".X5PpBb::text").get(),
            "user_avatar": comment.css(".gSGphe img::attr(srcset)").get().replace(" 2x", ""),
            "user_comment": comment.css(".h3YV2d::text").get(),
            "comment_likes": comment_likes.split("people")[0].strip() if comment_likes else None,
            "app_rating": re.search(r"\d+", comment.css(".iXRFPc::attr(aria-label)").get()).group(),
            "comment_date": comment.css(".bp9Aid::text").get(),
            "developer_comment": {
                "dev_title": comment.css(".I6j64d::text").get(),
                "dev_comment": comment.css(".ras4vb div::text").get(),
                "dev_comment_date": comment.css(".I9Jtec::text").get()
            }
        })

    print(json.dumps(user_comments, indent=2, ensure_ascii=False))


with sync_playwright() as playwright:
    run(playwright)

Example from the another blog post of mine that shows how to scrape all Naver Video results:

from playwright.sync_api import sync_playwright
import json

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)
    page = browser.new_page()
    page.goto("https://search.naver.com/search.naver?where=video&query=minecraft")

    video_results = []

    not_reached_end = True
    while not_reached_end:
        page.evaluate("""let scrollingElement = (document.scrollingElement || document.body);
                                 scrollingElement.scrollTop = scrollingElement scrollHeight;""")

        if page.locator("#video_max_display").is_visible():
            not_reached_end = False

    for index, video in enumerate(page.query_selector_all(".video_bx"), start=1):
        title = video.query_selector(".text").inner_text()
        link = video.query_selector(".info_title").get_attribute("href")
        thumbnail = video.query_selector(".thumb_area img").get_attribute("src")
        channel = None if video.query_selector(".channel") is None else video.query_selector(".channel").inner_text()
        origin = video.query_selector(".origin").inner_text()
        video_duration = video.query_selector(".time").inner_text()
        views = video.query_selector(".desc_group .desc:nth-child(1)").inner_text()
        date_published = None if video.query_selector(".desc_group .desc:nth-child(2)") is None else \
            video.query_selector(".desc_group .desc:nth-child(2)").inner_text()

        video_results.append({
            "position": index,
            "title": title,
            "link": link,
            "thumbnail": thumbnail,
            "channel": channel,
            "origin": origin,
            "video_duration": video_duration,
            "views": views,
            "date_published": date_published
        })

    print(json.dumps(video_results, indent=2, ensure_ascii=False))

    browser.close()

In the Naver pagination example, have a look at the if a condition that exits an infinite loop:

if page.locator("#video_max_display").is_visible():
    not_reached_end = False

Conclusion

Keep an eye on URL parameters. If something is changing when pagination if performed, then it could be a sign that those parameters can be used to perform pagination programmatically.
Try to find the next page tokens in the page source.
If nothing can be found from the points above, use either click or scroll pagination, or both.

Hope you found it useful. Let me know if something is still confusing.

Forem: Dmitriy Zub ☀️

SerpApi Changelog, May 2023

Summary of May Updates

API Updates

[Feature] Google Scholar API: add support for "Review Articles"

[Feature] Google Answer Box API: support thumbnail for answer box finance results

Conclusion

Make Direct Async Requests to SerpApi with Python

Intro

Code

Code Explanation

Conclusion

SerpApi Changelog: April, 2023

Summary of April Updates

API Updates

[Feature] Accounts API: now has a monthly subscription cost

[Feature] Google Shopping API: now support "Stores Near You" Block

[Feature] Bing Search API: now support Coupon results

Scrape Google Scholar in R

What will be scraped

Prerequisites

Full Code

Explanation

Google Scholar Python scraper alternatives

How to install Python packages without a requirements.txt file with pipreqs

Why bother

Example usage

Solution

Low amount of imports example

Big amount of imports example

Current limitations

SerpApi Changelog: March, 2023

Summary of March Updates

API Updates

[Feature] Google Search API: add support to expanded "Top Sites"

[Feature] Google Maps Reviews API: add support to scrape reviews rating

[Feature] Google Local Results API: now sponsored (ad) results is a separate JSON key from regular, organic results

[Feature] Google Shopping API: add support for "Top Features" section

[Feature] Google Search Results: add support to "Open Hours" answer box

[Feature] Knowledge Graph API: add support to extract "Average cost after aid", "Graduation rate", and "Acceptance rate"

[Feature] Google Shopping API: add support for "Others searched for" Block

[Feature] Google Shopping API: add support to extract price in the parentheses

[Feature] Home Depot API: add support to search by SKU / Product ID

[Feature] Google Search API: add support to a new "Things To Know" section in mobile results

[Feature] Google Search API: add support to extract "Refine this search" block

[Feature] Google Search API: add support for "Things to Know" section on Desktop

[Feature] Google Shopping API: add support for filter blocks "You might like", "People also search for"

Conclusion

SerpApi Changelog: February, 2023

Summary of February Updates

API Updates

[Feature] Google Maps API: now scraping experiences section of results

[Feature] Google Jobs API: now scraping job highlight section

[Feature] Google Search API: now scraping date for Google images results

[Feature] Knowledge Graph API: now scraping working at and typical company salaries

[Feature] Google Reverse Image Search: remove forced `q` parameter from `serpapi_link`

[Feature] [Documentation]: now supports keyboard shortcuts for 'Search' menu

[Feature] Google Jobs Listing API: now scraping website and other data on the page

[Feature] Google Shopping API: now support deals related to your search

[Feature] Google Shopping API: now scraping second hand condition tags

[Feature] Google Shopping API: now scraping number of compare prices options on Google Shopping

[Feature] Google Shopping API: now scraping related searches

Conclusion

SerpApi Changelog: January, 2023

Summary of January Updates ⭐

API Updates

[Feature] Google Finance API: add new parser to extract Quote (ticker) data

[Feature] Google Answer Box API: now parsing extra information in `local_time` type

[Feature] Google Immersive Products API: now parsing immersive products block

[Feature] Google Maps API: now users can make search by `place_id`

[Feature] Knowledge Graph API: now parsing menu filter tabs

[Feature] Google Shopping API: now support mobile and tablet versions

[Fix] Knowledge Graph API: `see_results_about` not parsed correctly

[Fix] Yandex Search API: respond with no results

[Fix] Google Shopping API: now parsing source's direct link instead of Google's redirect link

[Fix] Google Finance API: fix extracting Graph item

[Fix] Google Search API: fix empty `displayed_link` key for mobile ads searches

[Fix] Google Search API: fix empty `rating_count` returned for `local_ads`

[Fix] Google Search API: fix parsing the number of pages for PDF result in the `date` key

[Fix] Google Trends API: fix parsing interest by region results