Forem: Darshan Khandelwal

How to scrape Google AI Overviews using Python

Darshan Khandelwal — Fri, 13 Feb 2026 09:44:18 +0000

Google’s AI Overviews are changing how search results are displayed, delivering AI-generated summaries at the very top of the SERP. While these responses are helpful for users, they’re tricky to capture programmatically.

Whether you’re analyzing your brand presence, tracking how Google summarizes answers, or just experimenting with AI-generated content, scraping these overviews can unlock powerful insights.

In this tutorial, I will show you how you can scrape AI overviews from Google search using Python and Scrapingdog’s Google Search API.

What are AI Overviews?

AI Overviews are Google’s experimental feature that uses generative AI to answer search queries directly at the top of the search results page.

Instead of just showing blue links, Google summarizes information from various websites into a concise, natural-language response, often citing sources underneath.

Google introduced this feature in May 2024, initially for users in the United States. It was later rolled out to other countries as well. Currently, AI Overviews are available only in English, though Google may expand language support in the future.

Why Scrape Google AI Overviews?

Google’s AI Overviews are transforming how people consume information. Instead of relying completely on blue links, users now get AI-generated answers at the very top of search results, often without clicking anything.

This shift presents both a challenge and an opportunity.

Here’s why scraping AI Overviews matters:

📢 Track Brand Mentions
Know when your brand (or your competitor’s) is referenced in Google’s AI responses, even if you’re not ranking #1 organically.

🧠 Understand Search Intent Better
AI Overviews often reflect Google’s best guess at user intent. Scraping them gives you a window into how Google “thinks”.

But in the past, many answers from AI overviews have raised eyebrows too, and that made people question the reliability of AI-generated summaries and highlighted the need for better accuracy verification before deploying these features at scale.

✍️ Content & SEO Research
Identify what types of answers Google prefers, what sources it cites, and how it summarizes complex topics. Since AI overviews reduce organic traffic, it becomes necessary for any brand to appear in AI overviews.

🔍 Competitive Intelligence
Learn which companies or products consistently show up in AI-generated summaries and why. AI overviews have changed how this intelligence is gathered and how it impacts the overall SEO industry.

Why use Scrapingdog for scraping AI overviews?

Scraping AI Overviews isn’t easy, and they don’t appear on every search. They’re dynamically rendered, and they’re often wrapped in JavaScript-heavy containers. Traditional scrapers break. Headless browsers are slow and expensive.

That’s where Scrapingdog’s Google Search API stands out:

⚡ Fast & Scalable: Avoid setting up your own browsers and proxies. Scrapingdog handles that useless stuff so that you keep collecting the data at scale without hassle.

📦 Includes Source Attribution: Extracts citations, reference links, and summary text from the AI Overview box.

🔁 Works with All Search Parameters: Supports pagination, country targeting, device type, and more.

🧪 Great for Experiments & Monitoring: Track when Overviews appear, how they change, and which sites are cited.

Whether you’re building a dashboard to monitor brand mentions in Overviews or analyzing how AI rewrites search content, Scrapingdog gives you a reliable, high-speed way to access this data that too without browser automation and proxy setup nightmares.

Prerequisite
Python 3.x should be available on your machine. If not, then you can download it from here.
Install requests library for making HTTP connections with the API.
Account on Scrapingdog. You will get 1000 free credits on signup.

How to Scrape AI Overviews

To scrape Google AI overviews, you have to pass a sample query to Scrapingdog’s Google Search API. For this tutorial, we are going to use the query what is AI overview. The Google search page will look like this for this query.

Now, pass the query to the Google Search Scraper Playground.

As you can see in the above image, after typing the query in the input field, you will get a ready-to-use Python code. You can just copy this code and paste it into your Python file.

import requests

api_key = "your-api-key"
url = "https://api.scrapingdog.com/google"

params = {
    "api_key": api_key,
    "query": "what is AI overview",
    "country": "us",
    "advance_search": "true",
    "domain": "google.com"
}

response = requests.get(url, params=params)

if response.status_code == 200:
    data = response.json()
    print(data)
else:
    print(f"Request failed with status code: {response.status_code}")

Do not forget to use your own API key in the above code.

Here’s a brief explanation of the code in points:

API Configuration:

Sets up the ScrapingDog API key for authentication
Defines the endpoint URL (https://api.scrapingdog.com/google) for Google SERP scraping
api_key: Your authentication credential from ScrapingDog
query: The search term to scrape (“what is AI overview”)
country: Target location for search results (US in this case)
advance_search: Enables the extraction of AI Overviews and other advanced features.
domain: Specifies which Google domain to scrape (google.com)
Once you run this code, you will get this neat JSON response.

AI overview response from Scrapingdog

Within the JSON response, you will get an object ai_overview which will have the data from the AI overview section. Look pretty aesthetic, right?🔥

Now, in some cases, you might not get this data. For example, check this JSON response from the Google Search API.

Handling the ScrapingDog AI overview Extension Link:

When AI Overviews aren’t immediately available in the main response, ScrapingDog provides a fallback mechanism through the scrapingdog_link field.

How It Works:

The initial response may contain a url (original Google URL) and a scrapingdog_link (extension API endpoint)
If the AI Overview is missing from the primary JSON response, use the scrapingdog_link to retrieve it
Make a simple GET request to this link with no additional parameters needed
The response will contain the AI Overview data in JSON format
Critical Timing:

⚠️ 60-Second Expiration: You must make the GET request to scrapingdog_link within 60 seconds of receiving it. After this window, the link expires and becomes invalid.

Usage Example:

# If AI overview is not in main response
if 'scrapingdog_link' in data:
    extension_response = requests.get(data['scrapingdog_link'])
    ai_overview_data = extension_response.json()

This two-step approach ensures you can always capture AI Overviews, even when they require additional rendering time from Google’s servers.

Key Takeaways

ScrapingDog’s SERP API simplifies the complex process of extracting AI Overviews by handling proxy management, anti-bot detection, and browser fingerprinting automatically
The advance_search parameter unlocks rich SERP features, including AI Overviews, featured snippets, Ads and knowledge panels
The two-step approach with scrapingdog_link ensures you never miss AI Overview data, even when it requires extended rendering time
With just a few lines of Python code, you can monitor AI Overviews at scale for competitive analysis, content strategy, and SEO optimization

Conclusion

Scraping Google AI Overviews has become essential for staying competitive in today’s SEO landscape. As Google continues to prioritize AI-generated content at the top of search results, understanding how your content appears or doesn’t appear in these overviews is crucial for visibility and traffic. With the help of Python and powerful SERP APIs from Scrapingdog, we were able to achieve our goal without the hassle of setting up a browser or proxy.

FAQs (Frequently Asked Questions)

What are Google AI Overviews?
Google AI Overviews are AI-generated summaries displayed at the top of Google search results. They combine information from multiple websites and include source links.
Why are AI Overviews hard to scrape?
AI Overviews are dynamically rendered using JavaScript and don’t appear for every query, making traditional HTML scrapers unreliable.
How can I scrape Google AI Overviews using Python?
You can scrape AI Overviews using Scrapingdog’s Google Search API by sending a search query and enabling advanced search features.
What can I use scraped AI Overviews for?
Scraped AI Overviews can be used for SEO research, brand monitoring, and competitor analysis.

How to Scrape Google Trends Using Node.js and Export Data to CSV (2026)

Darshan Khandelwal — Mon, 09 Feb 2026 11:52:42 +0000

The world is moving very fast and analyzing trends becomes very crucial. No matter what industry you work in, if you have real-time trends data it can give you a competitive advantage over others.

Google Trends provides in-depth data on any trend around the globe. It collects this data through Google Search Results. If you want to prepare a report on growing trends within multiple industries, scraping Google Trends would be quite efficient for building reports.

This article will scrape Google Trends data using Nodejs and Scrapingdog’s Google Trends API.

Setup & Installation

For extracting trends data we are going to use Nodejs. If it is not installed on your machine, you can download it here.

Then create a folder by any name you like.

mkdir trends
cd trends

Initialize the package.json file to establish a node project and create a js file. I am naming the file as trends.js.

npm init

Now, we have to install four libraries that will be used in the course of this article.

npm i axios fs chartjs-node-canvas fast-csv

axios is for making an HTTP connection with the API.
fs is for reading the JSON data received from the API.
chartjs-node-canvas generates the graph using Chart.js.
fast-csv saves the data to a CSV file.

The final step would be to sign up for the trial pack of Scrapingdog. The trial pack comes with 1000 free credits that can be used for testing any API from Scrapingdog.

Scraping Google Trends with Nodejs

Let’s say you work in a consumer electronic manufacturing sector and you want to analyze whether the usage of Air purifier will going to increase or not. To analyze consumer interests and demands we will scrape the trends data.

Here’s a small video go through on how you can use Scrapigndog to scrape Google Trends.
https://youtu.be/mSlENTgSmSg

Before we start coding, it would be great to read the documentation. This will help us understand the role of every parameter. We are going to analyze the trends of the keyword Air Purifier from 01 January 2021 to 01 January 2026 in India.

Let’s dive into our Scrapingdog’s dashboard and fill in the fields.

The best part is that after filling out the form you will get a ready-made code on the right.

Just copy this code and paste it into your working environment.

//trends.js

const axios = require('axios');

const api_key = 'Your-API-Key';
const url = 'https://api.scrapingdog.com/google_trends/';

const params = {
  api_key: api_key,
  query: 'Air purifier',
  language: 'en',
  geo: 'IN',
  region: '0',
  data_type: 'TIMESERIES',
  tz: '',
  cat: '0',
  gprop: '',
  date: '2021-01-01 2025-01-01'
};

axios
  .get(url, { params: params })
  .then(function (response) {
    if (response.status === 200) {
      const data = response.data;
      console.log(data);
    } else {
      console.log('Request failed with status code: ' + response.status);
    }
  })
  .catch(function (error) {
    console.error('Error making the request: ' + error.message);
  });

The code is very simple but let me explain you step by step.

If you see the params object you will find that we have almost passed 10 parameters. Let me explain to you the meaning of each parameter.

api_key is the API key for your Scrapingdog account.
query is the term you want to search for.
language is the language of the result.
geo is the country you are targeting.
region is the place you are targeting using the geo parameter.
data_type defines the type of search you want to do.
tz is the time zone.
cat is used for defining the search category.
gprop is used to sort results by property.
date is the date range. If you run this code you will get this data.

According to Google Trends, value means this…

That means higher the value of value higher the demand.

Analyze the trends

Let’s plot a graph using chartjs-node-canvas. This will help us visualize the demand. We have to modify the code a little to make it the code look more readable.

const axios = require('axios');
const fs = require("fs");
const { ChartJSNodeCanvas } = require("chartjs-node-canvas");
const fastCsv = require("fast-csv");
const path = require("path");

let data;
const api_key = 'your-api-key';
const url = 'https://api.scrapingdog.com/google_trends/';

const params = {
  api_key: api_key,
  query: 'Air purifier',
  language: 'en',
  geo: 'IN',
  data_type: 'TIMESERIES',
  tz: '',
  category: '0',
  gprop: '',
  date: '2021-01-01 2025-01-01'
};
let TrendsData

async function fetchData() {
  try{
    TrendsData = await axios.get(url, { params: params })
    if (TrendsData.status === 200) {
      data = TrendsData.data["interest_over_time"]["timeline_data"];
      let dates = data.map(entry => entry["date"]);
      let values = data.map(entry => parseInt(entry["values"][0]["value"]));
      console.log("Data is Extracted, now ploting the graph");

      let graphStatus = await generateChart(dates,values)
      if(graphStatus){
        console.log('graph plotting is complete')
      }


    } else {
      console.log('Request failed with status code: ' + TrendsData.status);
    }
  }catch(err){
    console.error('Error making the request: ' + err.message);
  }
}



async function generateChart(dates,values) {
      const width = 800;
      const height = 400;

      const chartCanvas = new ChartJSNodeCanvas({ width, height });

      const configuration = {
          type: "line",
          data: {
              labels: dates,
              datasets: [
                  {
                      label: "Search Interest in 'Air Purifier'",
                      data: values,
                      borderColor: "blue",
                      fill: false,
                      tension: 0.3,
                  },
              ],
          },
          options: {
              responsive: false,
              plugins: {
                  title: {
                      display: true,
                      text: "Google Trends: Air Purifier Search Interest Over Time",
                      font: { size: 16 },
                  },
              },
              scales: {
                  x: { title: { display: true, text: "Date" } },
                  y: { title: { display: true, text: "Search Interest" } },
              },
          },
      };

      // Generate and save chart as an image
      const imagePath = path.join(__dirname, "trends_chart.png");
      const imageBuffer = await chartCanvas.renderToBuffer(configuration);
      fs.writeFileSync(imagePath, imageBuffer);

      console.log(`Chart saved as ${imagePath}`);
      return true
  }

fetchData()

After data extraction is over we call the generateChart function. After setting the height and width of the graph we are creating an object that will design the graph. Once the plotting is done we save the graph as a PNG file by the name trends_chart.png.

Let’s run the code and see what it looks like.

It clearly shows that trends started picking up after September 2024 and peaked around November 2024. If you notice the entire graph you will notice that the graph peaks around the same time interval. It starts growing by September and peaks in the month of November.

Key Findings:

Highest Search Interest: 100 (Peak in November 2024)
Lowest Search Interest: 5 (Observed in September 2023)
Average Search Interest: 10.98 across all data points

Storing the data into a CSV file

Here we will use fast-csv library to simply store the data into a CSV file.

const axios = require('axios');
const fs = require("fs");
const { ChartJSNodeCanvas } = require("chartjs-node-canvas");
const fastCsv = require("fast-csv");
const path = require("path");

let data;
const api_key = 'your-api-key';
const url = 'https://api.scrapingdog.com/google_trends/';

const params = {
  api_key: api_key,
  query: 'Air purifier',
  language: 'en',
  geo: 'IN',
  data_type: 'TIMESERIES',
  tz: '',
  category: '0',
  gprop: '',
  date: '2021-01-01 2025-01-01'
};
let TrendsData

async function fetchData() {
  try{
    TrendsData = await axios.get(url, { params: params })
    if (TrendsData.status === 200) {
      data = TrendsData.data["interest_over_time"]["timeline_data"];
      let dates = data.map(entry => entry["date"]);
      let values = data.map(entry => parseInt(entry["values"][0]["value"]));
      console.log("Data is Extracted, now ploting the graph");

      let graphStatus = await generateChart(dates,values)
      if(graphStatus){
        console.log('graph plotting is complete')
      }

      let csvStatus = await saveCSV(dates,values)
      if(csvStatus){
        console.log('CSV file is created')
      }
    } else {
      console.log('Request failed with status code: ' + TrendsData.status);
    }
  }catch(err){
    console.error('Error making the request: ' + err.message);
  }
}



async function generateChart(dates,values) {
      const width = 800;
      const height = 400;

      const chartCanvas = new ChartJSNodeCanvas({ width, height });

      const configuration = {
          type: "line",
          data: {
              labels: dates,
              datasets: [
                  {
                      label: "Search Interest in 'Air Purifier'",
                      data: values,
                      borderColor: "blue",
                      fill: false,
                      tension: 0.3,
                  },
              ],
          },
          options: {
              responsive: false,
              plugins: {
                  title: {
                      display: true,
                      text: "Google Trends: Air Purifier Search Interest Over Time",
                      font: { size: 16 },
                  },
              },
              scales: {
                  x: { title: { display: true, text: "Date" } },
                  y: { title: { display: true, text: "Search Interest" } },
              },
          },
      };

      // Generate and save chart as an image
      const imagePath = path.join(__dirname, "trends_chart.png");
      const imageBuffer = await chartCanvas.renderToBuffer(configuration);
      fs.writeFileSync(imagePath, imageBuffer);

      console.log(`Chart saved as ${imagePath}`);
      return true
  }

async function saveCSV(dates,values) {
    const csvPath = path.join(__dirname, "air_purifier_trends.csv");
    const writeStream = fs.createWriteStream(csvPath);
    const csvStream = fastCsv.format({ headers: true });

    csvStream.pipe(writeStream);
    dates.forEach((date, index) => {
        csvStream.write({ Date: date, Interest: values[index] });
    });

    csvStream.end();

    console.log(`CSV saved at ${csvPath}`);

    return true;
}


fetchData()

Here we will use fast-csv library to simply store the data into a CSV file.

fs.createWriteStream(csvPath) creates a writable stream to write data to the CSV file. Pipes (.pipe()) the formatted CSV data into the writable file stream (writeStream), allowing for efficient data writing.

Once you run the code you will see a CSV file inside your folder.

Observations

🔹Marketing & Sales Timing:

The best time to promote air purifiers is October–December when search interest is at its highest.
Advertising campaigns should be intensified in September to capture early buyers before peak demand.

🔹 Content Strategy for SEO:

Publish seasonal blog posts about air quality concerns and solutions before the peak season (August–September).
Focus on health benefits, pollution reports, and expert recommendations in October–November.

🔹 Product Launches & Discounts:

Introduce limited-time promotions in October-November to capitalize on peak interest.
Offer special bundles or discounts around Black Friday and Cyber Monday.

Conclusion

Tracking Google Trends data for ‘Air Purifier’ searches provides valuable insights into consumer behavior, seasonal demand, and market opportunities. By leveraging Scrapingdog’s Google Trends API and automating data extraction with Node.js, we successfully visualized the search interest over time.

By storing and analyzing Google Trends data, businesses can predict consumer demand, refine their marketing efforts, and stay ahead of competitors. Future improvements could include correlating this data with pollution levels, e-commerce sales, or weather patterns to gain deeper insights.

Frequently Asked Questions (FAQs)

1: How can I scrape Google Trends data using Node.js?
You can scrape Google Trends using Node.js by integrating the Scrapingdog API. Fetch trends data for any keyword, export it to a CSV file, and visualize it using libraries like Chart.js or chartjs-node-canvas.

2: Do I need an API key to scrape Google Trends with Scrapingdog?
Yes, Scrapingdog requires an API key. Sign up for a free trial to get 1000 credits, which allows you to test scraping Google Trends and integrate it with your Node.js projects.

3: What are the benefits of scraping Google Trends data for businesses?
Scraping Google Trends helps businesses identify seasonal demand, track consumer interests, optimize marketing campaigns, and make data-driven decisions to stay ahead of competitors.

How To Scrape Google Search Results using Python in 2026

Darshan Khandelwal — Wed, 04 Feb 2026 08:41:25 +0000

Google Scraping is one of the best methods to get comprehensive data from SERPs, as it provides insights into trends, competition, and consumer behavior.

Being one of the largest search engines, Google contains enormous data valuable for businesses and researchers.

However, to efficiently and effectively scrape Google search results, your data pipeline must be robust, scalable, and capable of handling dynamic changes in Google’s structure.

Whether you are looking to build your own LLM model or you are trying to gain some insight from the market, a Google search scraper would be needed.

In this read, we will build a Google search result scraper from scratch using Python and the BeautifulSoup library, enabling you to automate data extraction and gain actionable insights from search engine data.

But let’s see some common use cases one can have to use a Google scraper.

Top Use Cases for Scraping Google Search Results in 2026

Analyze the latest market trends.
Train LLM Models
Scrape Google Ads data.
Price competitive intelligence.
Build a Rank Tracking System/Tool.
Extract Emails by Scraping Google Search Results.

Why Python for Scraping Google Search Results?

Python is a widely used & simple language with built-in mathematical functions & hence is considered one of the best languages for scraping. Web scraping with Python is one of the most demanding skills in 2026 because AI is on a boom. It is also flexible and easy to understand even if you are a beginner. Plus the community is very big which helps if you face any syntax error during your initial days of coding.

Many forums like StackOverflow, GitHub, etc already have the answers to the errors you might face while coding when you scrape Google search results.

You can do countless things with Python but for now, we will learn web scraping Google search results with it.

Requirements

I hope Python is already installed on your computer, if not then you can download it from here. Create a folder to keep Python scripts in it.

mkdir google

We will need to install two libraries.

selenium– It is a browser automation tool. It will be used with Chromedriver to automate the Google Chrome browser. You can download the Chrome driver from here.
BeautifulSoup– This is a parsing library. It will be used to parse important data from the raw HTML data.
pandas– This library will help us store the data inside a CSV file.

pip install beautifulsoup4 selenium pandas

Now, create a Python file. We will write our script in this file. I am naming the file as search.py.

Why Selenium

As you know, Google recently released an update stating that JavaScript rendering is required to access Google pages. Therefore, a normal GET request through HTTP clients like requests will not work anymore.

Using Selenium we can run headless browsers which can execute javascript like a real user.

Scraping Google with Python and Selenium

In this article, we are going to scrape this page. Of course, you can pick any Google query. Before writing the code let’s first see what the page looks like and what data we will parse from it.

The page will look different in different countries.

We are going to extract the link, title, and description from the target Google page. Let’s first create a basic Python script that will open the target Google URL and extract the raw HTML from it.

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
import time
from bs4 import BeautifulSoup

# Set path to ChromeDriver (Replace this with the correct path)
CHROMEDRIVER_PATH = "D:/chromedriver.exe"  # Change this to match your file location

# Initialize WebDriver with Service
service = Service(CHROMEDRIVER_PATH)
options = webdriver.ChromeOptions()


options.add_argument("--window-size=1920,1080")  # Set window size


driver = webdriver.Chrome(service=service, options=options)

# Open Google Search URL
search_url = "https://www.google.com/search?q=lead+generation+tools&oq=lead+generation+tools"

driver.get(search_url)

# Wait for the page to load
time.sleep(2)

page_html = driver.page_source
print(page_html)

Let me briefly explain the code

First, we have imported all the required libraries. Here selenium.webdriver is controlling the web browser and time is for sleep function.
Then we have defined the location of our chromedriver.
Created an instance of chromedriver and declared a few options.
Then using .get() function we open the target link.
Using .sleep() function we are waiting for the page to load completely.
Then finally we extract the HTML data from the page.
Let’s run this code.

Yes yes, I know you got a captcha. Here I want you to understand the importance of using options arguments. While scraping Google you have to use — disable-blink-features=AutomationControlled. This Chrome option hides the fact that a browser is being controlled by Selenium, making it less detectable by anti-bot mechanisms. This also hides fingerprints.

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
import time
from bs4 import BeautifulSoup

# Set path to ChromeDriver (Replace this with the correct path)
CHROMEDRIVER_PATH = "D:/chromedriver.exe"  # Change this to match your file location

# Initialize WebDriver with Service
service = Service(CHROMEDRIVER_PATH)
options = webdriver.ChromeOptions()


options.add_argument("--window-size=1920,1080")  # Set window size
options.add_argument("--disable-blink-features=AutomationControlled")

driver = webdriver.Chrome(service=service, options=options)

# Open Google Search URL
search_url = "https://www.google.com/search?q=lead+generation+tools&oq=lead+generation+tools"

driver.get(search_url)

# Wait for the page to load
time.sleep(2)

page_html = driver.page_source
print(page_html)

As expected we were able to scrape Google with that argument. Now, let’s parse it.

Parsing HTML with BeautifulSoup

Before parsing the data we have to find the DOM location of each element.

All the organic results have a common class Ww4FFb. All these organic results are inside the div tag with the class dURPMd.

The link is located inside the a tag.

The title is located inside the h3 tag.

The description is located inside the div tag with the class VwiC3b. Let’s code it now.

page_html = driver.page_source
obj={}
l=[]
soup = BeautifulSoup(page_html,'html.parser')

allData = soup.find("div",{"class":"dURPMd"}).find_all("div",{"class":"Ww4FFb"})
print(len(allData))
for i in range(0,len(allData)):
    try:
        obj["title"]=allData[i].find("h3").text
    except:
        obj["title"]=None

    try:
        obj["link"]=allData[i].find("a").get('href')
    except:
        obj["link"]=None

    try:
        obj["description"]=allData[i].find("div",{"class":"VwiC3b"}).text
    except:
        obj["description"]=None

    l.append(obj)
    obj={}



print(l)

In the allData variable, we have stored all the organic results present on the page. Then using the for loop we are iterating over all the results. Lastly, we are storing the data inside the object obj and printing it.

Once you run the code you will get a beautiful JSON response like this.

Finally, we were able to scrape Google and parse the data.

Storing data to a CSV file

We are going to use the pandas library to save the search results to a CSV file.

The first step would be to import this library at the top of the script.

import pandas as pd

Now we will create a pandas data frame using list l

df = pd.DataFrame(l)
df.to_csv('google.csv', index=False, encoding='utf-8')

Again once you run the code you will find a CSV file inside your working directory.

Complete Code

You can surely scrape many more things from this target page, but currently, the code will look like this.

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
import time
from bs4 import BeautifulSoup
import pandas as pd


# Set path to ChromeDriver (Replace this with the correct path)
CHROMEDRIVER_PATH = "D:/chromedriver.exe"  # Change this to match your file location

# Initialize WebDriver with Service
service = Service(CHROMEDRIVER_PATH)
options = webdriver.ChromeOptions()


options.add_argument("--window-size=1920,1080")  # Set window size
options.add_argument("--disable-blink-features=AutomationControlled")

driver = webdriver.Chrome(service=service, options=options)

# Open Google Search URL
search_url = "https://www.google.com/search?q=lead+generation+tools&oq=lead+generation+tools"

driver.get(search_url)

# Wait for the page to load
time.sleep(2)

page_html = driver.page_source

soup = BeautifulSoup(page_html,'html.parser')
obj={}
l=[]
allData = soup.find("div",{"class":"dURPMd"}).find_all("div",{"class":"Ww4FFb"})
print(len(allData))
for i in range(0,len(allData)):
    try:
        obj["title"]=allData[i].find("h3").text
    except:
        obj["title"]=None

    try:
        obj["link"]=allData[i].find("a").get('href')
    except:
        obj["link"]=None

    try:
        obj["description"]=allData[i].find("div",{"class":"VwiC3b"}).text
    except:
        obj["description"]=None

    l.append(obj)
    obj={}

df = pd.DataFrame(l)
df.to_csv('google.csv', index=False, encoding='utf-8')

print(l)

Well, this approach is not scalable because Google will block all the requests after a certain number of requests. We need some advanced scraping tools to overcome this problem.

Limitations of scraping Google search results with Python

Although the above approach is great if you are not looking to scrape millions of pages. But if you want to scrape Google search at scale, then the above approach will fall flat, and your data pipeline will stop working immediately. Here are a few reasons why your scraper will be blocked.

Since we are using the same IP for every request, Google will ban your IP, which will result in the blockage of the data pipeline.
Along with IPs, we need quality headers and multiple browser instances, which are absent in our approach.
The solution to the above problem will be using a Google Search API like Scrapingdog. With Scrapingdog, you don’t have to worry about proxy rotations or retries. Scrapingdog will handle all the hassle of proxy and header rotation and seamlessly deliver the data to you.

You can scrape millions of pages without getting blocked with Scrapingdog. Let’s see how we can use Scrapingdog to scrape Google at scale.

Scraping Google Search Results with Scrapingdog

Now, that we know how to scrape Google search results using Python and Beautifulsoup, we will look at a solution that can help us scrape millions of Google pages without getting blocked.

We will use Scrapingdog’s Google Search Result Scraper API for this task. This API handles everything from proxy rotation to headers.

You just have to send a GET request and in return, you will get beautiful parsed JSON data.

This API offers a free trial and you can register for that trial from here. After registering for a free account you should read the docs to get the complete idea of this API.

import requests
api_key = "Paste-your-own-API-key"
url = "https://api.scrapingdog.com/google/"
params = {
"api_key": api_key,
"query": "lead generation tools",
"results": 10,
"country": "us",
"page": 0
}
response = requests.get(url, params=params)
if response.status_code == 200:
  data = response.json()
  print(data)
else:
  print(f"Request failed with status code: {response.status_code}")

The code is pretty simple. We are sending a GET request to https://api.scrapingdog.com/google/ along with some parameters. For more information on these parameters, you can again refer to the documentation.

Once you run this code you will get a beautiful JSON response.

In this JSON response, you will get People also ask for data & related search data as well. So, you are getting full data from Google, not just organic results.

What if I need search results from a different country?
As you might know, Google shows different results in different countries for the same query. Well, I just have to change the country parameter in the above code.

Let’s say you need results from the United Kingdom. For this, I just have to change the value of the country parameter to gb (ISO code of UK).

You can even extract 100 search results instead of 10 by just changing the value of the results parameter.

Here’s a video tutorial on how to use Scrapingdog’s Google SERP API.⬇️
https://youtu.be/W1yyt6VnEmk

**Note: **Scrapingdog has recently launched an all-in-one search engine API.

This API gives filtered data from all major search engines (Google + Bing + Yahoo + DuckDuckGo).

We are calling it Universal SERP API. The advantage of using this API is that it allows you to take data in one API call, you don’t need to filter out repetitive results, and it is economical.

How To Scrape Google Ads using Scrapingdog's Search API

You can use the same API to extract competitors’ AD results as well!!

In the documentation, you can read about the 'advance_search' parameter. This parameter allows you to get advanced SERP results, and Google Ads results are there in it.

I have made a quick tutorial on this, too, to make you understand how the Scrapingdog’s Google SERP API can be used to get ADs data.⬇️

https://youtu.be/NRsnOKkOEh4

Is there an official Google SERP API to extract search results?

Google offers its API to extract data from its search engine. It is available at this link for anyone who wants to use it. However, the usage of this API is minimal due to the following reasons: –

The API is very costly — For every 1000 requests you make, it will cost you around $5, which doesn’t make sense as you can do it for free with web scraping tools.

The API has limited functionality — It is made to scrape only a small group of websites, although by doing changes to it you can scrape the whole web again which would cost you time.

Limited Information — The API is made to provide you with little information, thus any data extracted may not be useful.

Scrape Google Search Data Easily with Our Google Sheets Add-On (For Non- Developers)

If you are a non-developer and wanted to scrape the data from Google, here is a good news for you.

We have recently launched a Google Sheet add-on Google Search Scraper.

Here is the video 🎥 tutorial for this action.
https://youtu.be/IznWFH3oI0o

Here are 5 crisp key takeaways:

Google Search results can be scraped to extract titles, URLs, descriptions, rankings, and other SERP insights useful for SEO, research, and market analysis.
A basic Python setup using tools like Selenium and BeautifulSoup works for learning and small-scale scraping.
Google actively detects scraping through IP behavior, headers, and browser fingerprints, making DIY scraping unreliable at scale.
Scaling manual scrapers requires handling proxies, retries, CAPTCHA, and rendering. This adds significant engineering overhead.
Using a SERP API like Scrapingdog simplifies the process by returning structured search data without worrying about blocks or infrastructure.

Conclusion

In this article, we saw how we can scrape Google results with Python and BS4. Then we used Google SERP API for scraping Google search results at scale without getting blocked.

Google has a sophisticated anti-scraping wall that can prevent mass scraping, but Scrapingdog can help you by providing a seamless data pipeline that never gets blocked. Scrapingdog also provides a Bing Search API ,Baidu Search API to scrape search results from this search engine.

If you like this article, please do share it on your social media accounts. If you have any questions, please contact me at info@scrapingdog.com.

Scrapingbee vs ScraperAPI vs Scrapingdog: Which One Is Better For Your Web Scraping Needs

Darshan Khandelwal — Wed, 21 Jan 2026 08:37:29 +0000

When it comes to web scraping at scale, choosing the right scraping API can make all the difference, whether you’re tracking prices, extracting SERP data, or collecting leads. Among the top contenders in the market are Scrapingbee, ScraperAPI, and Scrapingdog, each promising fast, reliable, and hassle-free data extraction. But how do they stack up in terms of performance, features, and cost?

In this article, we’ll break down their differences and help you decide which one best fits your scraping needs.

Criteria
We will test each product and compare them based on:

Speed
Success rate
Support
Scalability
Developer friendly
We will test each API on these target websites.

We will use this Python code to test various APIs.

import requests
import time
import random
import urllib.parse

# List of search terms
amazon_urls = ["https://www.amazon.com/dp/B0CVSJ8YBX/","https://www.amazon.es/s?k=monitores","https://www.amazon.com.br/dp/B0DGXR6FRP/","https://www.amazon.in/s?k=air+conditioner","https://www.amazon.es/s?k=rigoberta+bandini"]

walmart_urls = ["https://www.walmart.com/ip/Genuine-John-Deere-OEM-Filter-Kit-LG180/982108275","https://www.walmart.com/ip/Crest-3D-Whitestrips-Glamorous-White-At-Home-Teeth-Whitening-Kit-14-Treatments/46480251","https://www.walmart.com/ip/Crest-Pro-Health-Toothpaste-Advanced-White-for-Teeth-Whitening-2-Pack/1554705966","https://www.walmart.com/ip/Crest-Pro-Health-Toothpaste-Clean-Mint-4-3-oz/1315357918","https://www.walmart.com/ip/Equate-Ibuprofen-Tablets-200-mg-Pain-Reliever-Fever-Reducer-40-Count/891884626"]

ebay_url = ["https://www.ebay.com/itm/306335193840","https://www.ebay.com/itm/186754878239","https://www.ebay.com/itm/176068005601","https://www.ebay.com/itm/183759998583","https://www.ebay.com/itm/316882974542"]

glassdoor_urls = ["https://www.glassdoor.com/Job/new-york-python-jobs-SRCH_IL.0%2C8_IC1132348_KO9%2C15.htm?clickSource=searchBox","https://www.glassdoor.com/Reviews/Glassdoor-Reviews-E100431.htm","https://www.glassdoor.com.au/Salary/Reserve-Bank-of-Australia-Salaries-E8214.htm","https://www.glassdoor.com/Job/texas-us-data-engineer-jobs-SRCH_IL.0%2C8_IS1347_KO9%2C22.htm?includeNoSalaryJobs=true","https://www.glassdoor.co.in/Overview/Working-at-Eni-Spa-EI_IE3164.11,18.htm"]

google_serp_terms = ["shoes","burger","corona","cricket","tennis"]

base_url = "https://api.example.com/scrape"
API_key='Your-api-key'

total_requests = 10
success_count = 0
total_time = 0

for i in range(total_requests):
    try:
        search_term = random.choice(google_serp_terms)

        params = {
        "api_key": API_key,

        "query": search_term
        }

        # url = base_url.format(query=search_term)

        start_time = time.time()
        response = requests.get(base_url,params=params)
        end_time = time.time()

        request_time = end_time - start_time
        total_time += request_time

        if response.status_code == 200:
            success_count += 1
        print(f"Request {i+1}: '{search_term}' took {request_time:.2f}s | Status: {response.status_code}")

    except Exception as e:
        print(f"Request {i+1} with '{search_term}' failed due to: {str(e)}")

# Final Stats
average_time = total_time / total_requests
success_rate = (success_count / total_requests) * 100

print(f"\n Total Requests: {total_requests}")
print(f" Successful: {success_count}")
print(f" Average Time: {average_time:.2f} seconds")
print(f" Success Rate: {success_rate:.2f}%")

Scrapingbee

This web scraping tool can be used for scraping any website at scale.

Details

They also provide 1000 free credits on signup.
Per scrape cost starts from $0.000196 and drops below $0.000075 with a higher volume.
Documentation is very clear and developer-friendly.
Support is only available through email.

Testing the API with Amazon

Testing the API with Glassdoor

Testing the API with eBay

Testing the API with Walmart

Testing the API with Google

Test Summary

While testing Scrapingbee on Amazon, we got a 100% success rate with an average response time of just 4.96 seconds.
0% success rate with an average response time of 4.99 seconds on Glassdoor.
100% success rate with an average response time of 8.98 seconds on eBay.
40% success rate with an average response time of 7.35 seconds on Walmart.
A 90% success rate with a 16.28s average response time on Google, at that cost, only makes sense if budget isn’t a concern and performance doesn’t matter.

ScraperAPI

ScraperAPI is one of the oldest players in this industry, providing robust solutions for scraping websites.

Details

ScraperAPI provides 5000 free credits on signup.
The per-scrape cost starts from $0.00049 and drops below $0.000095 with higher volume.
Documentation is very clear and can be easily integrated.
Support is available through email only.

Testing the API with Amazon

Testing the API with Glassdoor

Testing the API with eBay

Testing the API with Walmart

Testing the API with Google

Test Summary

While testing ScraperAPI on Amazon, we got a 100% success rate with an average response time of just 40.65 seconds. Response time was too high
100% success rate with an average response time of 20.48 seconds on Glassdoor.
100% success rate with an average response time of 8.28 seconds on eBay.
100% success rate with an average response time of 18.89 seconds on Walmart.
With an 80% success rate and an average response time of 27.25 seconds, this API falls short of being reliable for SERP scraping.

Scrapingdog

Scrapingdog offers one of the best web scraping APIs. From a general web scraper to dedicated endpoints for multiple websites, this API becomes the best choice for web scraping.

Details

Once you sign up, you get 1000 free credits for testing.
Per scrape cost starts from $0.0002 and drops below $0.000063 with a higher volume.
Scrapingdog provides clear documentation, and any developer can integrate the API very easily into their working environment. New video tutorials and blogs are regularly published to support you along the way.
Customer support is available 24*7 to help you resolve any query related to the services offered.

Testing the API with Amazon

Testing the API with Glassdoor

Testing the API with eBay

Testing the API with Walmart

Testing API with Google

Test Summary

While testing Scrapingdog on Amazon, we achieved a 100% success rate with an average response time of just 5.48 seconds.
Achieved a 100% success rate with an average response time of 5.57 seconds on Glassdoor.
Achieved a 100% success rate with an average response time of 5.91 seconds on eBay.
Achieved a 100% success rate with an average response time of 4.48 seconds on Walmart.
Scraped Google with a flawless 100% success rate and an impressively fast average response time of just 1.25 seconds.

Speed Comparison

Let’s visualize the time taken by the APIs on each domain.

Here are 3 key observations from the response time data:

Scrapingdog consistently outperforms both ScraperAPI and Scrapingbee in terms of speed across all tested websites, especially on Google, with a remarkable 1.25s response time.
ScraperAPI shows major latency, particularly with Amazon and Google, clocking over 40s and 27s, respectively, which could hinder performance in real-time applications.
Scrapingbee is inconsistent — while fast on Amazon (4.96s), it failed on Glassdoor (0% success) and showed poor performance on Walmart (40% success despite a moderate 7.35s time), making it unreliable for diverse scraping needs.

Price Comparison

Scrapingdog offers the most budget-friendly option, making it ideal for scaling projects affordably.
Despite the competitive pricing, Scrapingdog maintained strong performance in both speed and reliability across tests.
While ScraperAPI and Scrapingbee performed decently, their slightly higher costs may be better suited for specific use cases or integrations.

Conclusion

Choosing the right web scraping API isn’t just about cost, it’s about reliability, speed, and scalability. After testing Scrapingdog, ScraperAPI, and ScrapingBee across five high-demand websites (Amazon, Glassdoor, eBay, Walmart, and Google), the differences were clear.

Scrapingdog consistently delivered a 100% success rate across all tested websites with the fastest average response times. When paired with its highly affordable $0.058 per 1K requests on the Premium plan, it offers exceptional value, especially for users who care about both performance and budget.

ScraperAPI performed well in terms of success rate but suffered from noticeably slower response times across most websites, which could become a bottleneck for high-frequency use cases. Its $0.095/1K requests rate is also the highest among the three.

ScrapingBee showed inconsistent reliability, including a 0% success rate on Glassdoor and 40% on Walmart, despite faster response times on some sites. Priced at $0.0748/1K requests, it may not justify the cost for critical scraping needs without performance guarantees.

In short:

If you’re looking for speed, reliability, and cost-efficiency, Scrapingdog stands out.
ScraperAPI could be a solid choice if you’re okay with slower responses.
ScrapingBee may require careful evaluation depending on your target domains.
💡 Pro Tip: Before committing to any provider, always test their free or trial tier to see how it handles your real-world scraping needs.

Additional Resources

Search Engine Scraping Tutorial With ScrapingDog

Darshan Khandelwal — Tue, 13 Jan 2026 10:26:56 +0000

Search engines are where the world’s information lives and scraping them opens up endless opportunities for research, analysis, and automation. Whether it’s tracking rankings, gathering keyword data, analyzing competitors, or extracting search insights across multiple platforms, having structured search results at scale can be incredibly valuable.

In this tutorial, we’ll walk you through how to scrape Google, Bing, Yahoo, Baidu, and DuckDuckGo step-by-step using Scrapingdog’s Search Engine Scraping APIs. You’ll learn how to set up requests, handle responses, and extract useful data like titles, URLs, snippets, and more all without worrying about CAPTCHAs or IP blocks.

By the end of this guide, you’ll have a working blueprint to scrape multiple search engines effortlessly and integrate real-time search data into your own apps or dashboards.

There is a bonus section at the end of this article where I will show how you can extract data from all the major search engine with just a single API call.

Why scrape Search Engines?

Search engines are the pulse of the internet, they reveal what people are searching for, which brands dominate visibility, and how information trends evolve. Scraping them gives you direct access to this live search intelligence, which can be applied across multiple use cases.

Here’s why businesses and developers scrape search engines:

Keyword Research & SEO Tracking- Collect SERP data to analyze keyword trends, monitor rankings, and track competitors’ visibility.
Market & Competitor Insights- Understand how rivals position themselves across search platforms and identify emerging topics or products.

Content and News Monitoring- Extract real-time updates from search results to feed dashboards or alert systems.
Data-Driven Applications- Power custom tools like price trackers, sentiment analysis systems, and AI models with fresh, search-based data.
Automation- Instead of manually checking results, APIs automate the process — saving hours of repetitive work. In short, scraping search engines lets you turn public search results into actionable data, enabling smarter decisions across SEO, marketing, and analytics.

Why use Scrapingdog to Scrape Search Engines?

When it comes to scraping search engines like Google, Bing, Baidu, or DuckDuckGo, Scrapingdog simplifies what’s usually a painful, error-prone process. Traditional scraping often fails due to IP bans, CAPTCHAs, and constant layout changes but Scrapingdog handles all of that for you.

Here’s why it’s the smarter choice:

No Proxy or Headless Setup Needed- You don’t have to manage rotating proxies, browsers, or user agents, Scrapingdog does it automatically.
Supports All Major Search Engines- Scrapingdog’s API endpoints lets you extract results from Google, Bing, Baidu, and DuckDuckGo with consistent response structure.
High-Speed, High-Success Rate- Built-in infrastructure ensures 99% success with low latency, even for heavy workloads.
JSON Response Ready for Integration- You get clean, structured data directly usable in your app or data pipeline.
Free Trial for Developers- Start scraping instantly with 1,000 free credits, no complex setup or long sign-up process. In short, Scrapingdog gives you developer-friendly access to real-time search data, without worrying about bans or browser management.

How to Scrape Search Engines With ScrapingDog

We’ll test dedicated APIs for scraping Google, Bing, DuckDuckGo, and Baidu, one by one using Python(Before we begin testing the the APIs I hope you have Python 3.x installed on your machine). And just when you think you’ve seen it all, I’ll introduce an API that can pull results from all these search engines in a single call. Sounds interesting? Let’s dive in.

Scraping Google search results with Scrapingdog
Once you sign up and access the dashboard, you’ll find the Google SERP Scraping API displayed right there on the dashboard.

To scrape Google search results you can pass any randome query. For this tutorial, I’ll be using the query “search engine scraping”.

With the Google scraper you will get this complete data in JSON format.

Once I pass this query in the scraper I will get a python code which I can just copy and paste it in my python environment to scrape Google.

import requests

api_key = "your-api-key"
url = "https://api.scrapingdog.com/google"

params = {
    "api_key": api_key,
    "query": "search engine scraping",
    "country": "us",
    "advance_search": "true",
    "domain": "google.com"
}

response = requests.get(url, params=params)

if response.status_code == 200:
    data = response.json()
    print(data)
else:
    print(f"Request failed with status code: {response.status_code}")

Once you run this code you will get this beautiful JSON response.

You will get everything right from Ads, AI overview to organic search results within this JSON response.

If you don’t need such a detailed response and are only interested in organic search data, you can use the Google Light Search API instead.

import requests

api_key = "your-api-key"
url = "https://api.scrapingdog.com/google"

params = {
    "api_key": api_key,
    "query": "search engine scraping",
    "country": "us",
    "advance_search": "false",
    "domain": "google.com",
    "language": "en"
}

response = requests.get(url, params=params)

if response.status_code == 200:
    data = response.json()
    print(data)
else:
    print(f"Request failed with status code: {response.status_code}")

You will get this JSON response with the above code.

This API is economical and its latency is also very low compared to the advance search API.

Scraping Bing search results with Scrapingdog

Scrapingdog also provides a dedicated endpoint for scraping Bing at scale. To test this API just pass search engine scraping to the Bing scraper

Copy the python code from the dashboard and paste it in your Python file.

import requests

api_key = "your-api-key"
url = "https://api.scrapingdog.com/bing/search"

params = {
    "api_key": api_key,
    "query": "search engine scraping"
}

response = requests.get(url, params=params)

if response.status_code == 200:
    data = response.json()
    print(data)
else:
    print(f"Request failed with status code: {response.status_code}")

Once you run this code you will get this JSON response.

Scraping DuckDuckGo with Scrapingdog

DuckduckGo is another search engine which is widely used in many countries. You can scrape this search engine to create your own seo tool. Let’s see how this can be scraped with the help of Scrapingdog’s scraping APIs.

We will use DuckduckGo scraper API to scrape search results in JSON format. Again we will use the same query search engine scraping. If you search this query on DuckDuckGo, it will render this.

Now, to scrape this you have to pass the query to the scraper and copy the python code from the dashboard.

import requests

api_key = "your-api-key"
url = "https://api.scrapingdog.com/duckduckgo/search"

params = {
    "api_key": api_key,
    "query": "search engine scraping"
}

response = requests.get(url, params=params)

if response.status_code == 200:
    data = response.json()
    print(data)
else:
    print(f"Request failed with status code: {response.status_code}")

Here we are making a GET request to https://api.scrapingdog.com/duckduckgo/search along with the basic query parameters. Once you run this code you will get this JSON response.

You got the title, link, snippet and other relevant data. This is how you can scrape millions of search pages on daily basis with Scrapingdog.

Scraping Baidu with Scrapingdog

Baidu is a dominant search engine in China and scraping this search engine can provide you with valuable insight about Chinese market.

In this section we will learn to scrape Baidu with the help of Baidu Scraping API. You will find the scraper over here. We will use the same technique used before with other search engines.

We will make a GET request to https://api.scrapingdog.com/baidu/search to extract the search result data in JSON format.

import requests

api_key = "your-api-key"
url = "https://api.scrapingdog.com/baidu/search"

params = {
    "api_key": api_key,
    "query": "search engine scraping"
}

response = requests.get(url, params=params)

if response.status_code == 200:
    data = response.json()
    print(data)
else:
    print(f"Request failed with status code: {response.status_code}")

Once you run this code you will get this JSON data.

This data will have everything from links to title. Of course, you can pass more query parameter to the API according to your requirements.

Earlier in this article, I mentioned an API capable of fetching data from all major search engines with a single request. Now it’s time to put that into action. We’ll be using Scrapingdog’s Universal Search API for this.

How to Scrape all major search engines with one API.

Universal Search API fetches results from all major search engines in a single request, allowing you to collect data efficiently without making separate API calls for each engine.

You can access this scraper from here. To access this API we are going to make a GET request to https://api.scrapingdog.com/search

import requests

url = "https://api.scrapingdog.com/search"
params = {
    "api_key": "your-api-key",
    "query": "search engine scraping",
    "country": "us",
    "language": "en"
}

response = requests.get(url, params=params)
print(response.json())

It is a very clean python code and once you run this you will get this JSON response.

With this we are going to wrap this article.

Conclusion

Scraping search engines doesn’t have to be complicated or risky. With Scrapingdog, you get a simple, reliable, and scalable way to extract data from Google, Bing, Baidu, and DuckDuckGo.

Whether you’re tracking keyword rankings, building a research tool, or analyzing market trends, Scrapingdog saves you hours of setup and maintenance. No rotating proxies, no browser automation, just clean, structured data ready to use.

If you haven’t tried it yet, sign up for the free pack and start scraping search engine data instantly with your first 1,000 free credits.

Additional Resources
Search Engine Scraping: Challenges, Use Cases & Tools

How To Scrape Baidu Search Results using Python

How To Scrape Google Search Results using Python in 2026

How to Scrape Google AI Mode Using Python

Darshan Khandelwal — Mon, 12 Jan 2026 10:32:25 +0000

Google’s recent integration of generative AI into search results, known as “AI mode,” provides summarized answers directly on the search page. For businesses, SEOs, and data analysts, scraping these AI-generated answers can unlock valuable insights, track content visibility, and monitor shifts in Google’s approach.

This article provides a clear, step-by-step guide on how to utilize Scrapingdog’s Google AI mode scraper API with Python to scrape Google’s AI-generated search results efficiently.

Why Scrape Google’s AI Results?

SEO Analysis- Understand how your content is reflected in Google’s AI summaries.
Competitor Monitoring- Keep track of competitor presence in AI-generated answers.
Content Research- Gather structured answers to feed AI content creation and research processes.
Market Intelligence- Gain insights into trends and shifts in AI-based search behaviors.

Requirements

I hope you have already installed Python on your computer; if not, then you can install it from here. Next, create a folder to keep all your project files. Let’s name it scraper.

mkdir scraper

Now, inside this folder, install the requests library. Using this library, we are going to make a GET request to the host website.

pip install requests

Create a Python file by any name you like. I am naming the file as aimode.py.

Now, sign up for the free pack on Scrapingdog. You’ll get 1,000 free credits to test any API.

Scraping Google AI Results with Python and Scrapingdog

Once you are signed up, you will be redirected to your dashboard, which looks like this.

I have passed a sample query, What is llm model? and instantly, a Python code appeared on the right. Just copy this and run it in your Python environment.

import requests

api_key = "your-api-key"
url = "https://api.scrapingdog.com/google/ai_mode"

params = {
    "api_key": api_key,
    "query": "what is llm model?",
    "country": "us",
}

response = requests.get(url, params=params)

if response.status_code == 200:
    data = response.json()
    print(data)
else:
    print(f"Request failed with status code: {response.status_code}")

The code is pretty simple, but let me explain to you how it works.

Imports the requests library to make HTTP requests.
Set your Scrapingdog API key and the endpoint URL for scraping Google AI mode results.
We have defined the required parameters.
Sends a GET request to the Scrapingdog AI Mode Scraper API with the set parameters.
If the request is successful (status code 200), it prints the parsed JSON data (Google’s AI response).
If the request fails, it prints an error message with the HTTP status code.
Let’s execute this script and see the results.

I got a beautiful JSON response from the API. Just like this, you can scrape answers from Google AI mode with the help of Scrapingdog for any number of queries.

If you want to understand more about this API, then you can refer to this video.

Conclusion

Scraping Google’s AI mode doesn’t need to be complicated. With Scrapingdog’s specialized Google AI scraping endpoint and Python, you can effortlessly capture valuable AI-generated insights at scale.

Whether you’re monitoring brand visibility, performing competitive research, or generating structured data for content creation, Scrapingdog provides a reliable, scalable solution. Get started today and elevate your data scraping workflows with powerful, actionable insights.

Additional Resources

Building A Simple Google AI Mode Query Tracker using Scrapingdog

Scrape Google Search Results Using Python

Darshan Khandelwal — Sat, 28 Dec 2024 21:34:21 +0000

Scrape Google Search Results Using Python

Google holds an immense volume of data for businesses and researchers. It performs over 8.5 billion daily searches and commands a 91% share of the global search engine market.

Since the debut of ChatGPT, Google data has been utilized not only for traditional purposes like rank tracking, competitor monitoring, and lead generation but also for developing advanced LLM models, training AI models, and enhancing the capabilities of Natural Language Processing (NLP) models.

Scraping Google, however, is not easy for everyone. It requires a team of professionals and a robust infrastructure to scrape at scale.

In this article, we will learn to scrape Google Search Results using Python and BeautifulSoup. This will enable you to build your own tools and models that are capable of leveraging Google’s data at scale.

Let’s get started!

What are Google Search Results?

Google Search Results are the listings that appear on Google based on the user query entered in the search bar. Google heavily utilizes NLP to understand these queries and present users with relevant results. These results often include featured snippets in addition to organic results, such as the latest AI overviews, People Also Ask sections, Related Searches, and Knowledge Graphs. These elements provide summarized and related information to users based on their queries.

Applications Of Scraping Google Search Data

Google Search Data has various applications:

Building a rank and keyword tracker for SEO purposes.
Searching for local businesses.
Building LLM engines.
Discovering exploding topics for potential trends in the future.

Why Python for scraping Google?

Python is a versatile and robust language that provides a powerful HTTP handshake configuration for scraping websites that other languages may struggle with or have lower success rates. As the popularity of AI models trained on web-scraped data grows, Python’s relevance in web-scraping topics continues to rise within the developer community.

Additionally, beginners looking to learn Python as a web scraping skill can understand it easily due to its simple syntax and code clarity. Plus, it has huge community support on platforms like Discord, Reddit, etc., which can help with any level of problem you are facing.

This scalable language excels in web scraping performance and provides powerful frameworks like Scrapy, Requests, and BeautifulSoup, making it a superior choice for scraping Google and other websites compared to other languages.

Scraping Google Search Results With Python

This section will teach us to create a basic Python script to retrieve the first 10 Google search results.

Requirements

To follow this tutorial we need to install the following libraries:

Requests — To pull HTML data from the Google Search URL.
BeautifulSoup — To refine HTML data in a structured format.

Setup

The setup is simple. Create a Python file and install the required libraries to get started.

Run the following commands in your project folder:

    touch scraper.py

And then install the libraries.

    pip install requests
    pip install beautifulsoup4

Process

We are done with the setup and have all the stuff to move forward. We will use the Requests library in Python to extract the raw HTML and the BeautifulSoup to refine it and get the desired information.

But what is “desired information” here?

The filtered data would contain this information:

Title
Link
Displayed Link
Description
Position of the result

Let us import our installed libraries first in the scraper.py file.

    from bs4 import BeautifulSoup
    import requests

Then, we will make a GET request on the target URL to fetch the raw HTML data from Google.

headers={'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.361681276786'}
    url='https://www.google.com/search?q=python+tutorials&gl=us'
    response = requests.get(url,headers=headers)
    print(response.status_code)

Passing headers is important to make the scraper look like a natural user who is just visiting the Google search page for some information.

The above code will help you in pulling the HTML data from the Google Search link. If you got the 200 status code, that means the request was successful. This completes the first part of creating a scraper for Google.

In the next part, we will use BeautifulSoup to get out the required data from HTML.

    soup = BeautifulSoup(response.text, ‘html.parser’)

This will create a BS4 object to parse the HTML response and thus we will be able to easily navigate inside the HTML and find any element of choice and the content inside it.

To parse this HTML, we would need to first inspect the Google Search Page to check which common pattern can be found in the DOM location of the search results.

So, after inspecting we found out that every search result is under div container with the class g. This means, we just have to run a loop over each div container with g class to get the information inside it.

Before writing the code, we will find the DOM location for the title, description, and link from the HTML.

If you inspect the title, you’ll find that it is contained within an h3 tag. From the image, we can also see that the link is located in the href attribute of the anchor tag.

The displayed link or the cite link can be found inside the cite tag.

And finally, the description is stored inside a div container with the class VwiC3b.

Wrapping all these data entities into a single block of code:

organic_results = []
        i = 0

        # Parse organic results with error handling
        for el in soup.select(".g"):
            try:
                title = el.select_one("h3").text if el.select_one("h3") else "No title"
                displayed_link = el.select_one(".byrV5b cite").text if el.select_one(".byrV5b cite") else "No displayed link"
                link = el.select_one("a")["href"] if el.select_one("a") else "No link"
                description = el.select_one(".VwiC3b").text if el.select_one(".VwiC3b") else "No description"

                organic_results.append({
                    "title": title,
                    "displayed_link": displayed_link,
                    "link": link,
                    "description": description,
                    "rank": i + 1
                })
                i += 1
            except Exception as e:
                print(f"Error parsing element: {e}")

        print(organic_results)

We declared an organic results array and then looped over all the elements with g class in the HTML and pushed the collected data inside the array.

Running this code will give you the desired results which you can use for various purposes including rank tracking, lead generation, and optimizing the SEO of the website.

[
      {
        "title": "Python Tutorial",
        "displayed_link": "https://www.w3schools.com \u203a python",
        "link": "https://www.w3schools.com/python/",
        "description": "Learn Python. Python is a popular programming language. Python can be used on a server to create web applications. Start learning Python now.",
        "rank": 1
      },
      {
        "title": "The Python Tutorial \u2014 Python 3.13.1 documentation",
        "displayed_link": "https://docs.python.org \u203a tutorial",
        "link": "https://docs.python.org/3/tutorial/index.html",
        "description": "This tutorial introduces the reader informally to the basic concepts and features of the Python language and system. It helps to have a Python interpreter handy\u00a0...",
        "rank": 2
      },
     ....
    ]

So, that’s how a basic Google Scraping script is created.

However, there is a CATCH. We still can’t completely rely on this method as this can result in a block of our IP by Google. If we want to scrape search results at scale, we need a vast network of premium and non-premium proxies and advanced techniques that can make this possible. That’s where the SERP APIs come into play!

Scraping Google Using ApiForSeo’s SERP API

Another method for scraping Google is using a dedicated SERP API. They are much more reliable and don’t let you get blocked in the scraping process.

The setup for this section would be the same, just we need to register on ApiForSeo to get our API Key which will provide us with access to its SERP API.

Getting API Credentials From ApiForSeo

After activating the account, you will be redirected to the dashboard where you will get your API Key.

You can also copy the code from the dashboard itself.

Setting Up our code for scraping search results

Then, we will create an API request on a random query to scrape data through ApiForSeo SERP API.

 import requests

    api_key = "APIKEY"
    url = "https://api.apiforseo.com/google_search"

    params = {
        "api_key": api_key,
        "q": "elon+musk",
        "gl": "us",
    }

    response = requests.get(url, params=params)

    if response.status_code == 200:
        data = response.json()
        print(data)
    else:
        print(f"Request failed with status code: {response.status_code}")

You can try any other query also. Don’t forget to put your API Key into the code otherwise, you will receive a 404 error.

Running this code in your terminal would immediately give you results.

 "organic_results": [
        {
          "title": "Elon Musk - Wikipedia",
          "displayed_link": "https://en.wikipedia.org › wiki › Elon_Musk",
          "snippet": "Elon Reeve Musk is a businessman known for his key roles in the space company SpaceX and the automotive company Tesla, Inc. His other involvements include ...Musk family · Tesla Roadster · Tesla, SpaceX, and the Quest... · Maye Musk",
          "link": "https://en.wikipedia.org/wiki/Elon_Musk",
          "extended_sitelinks": [
            {
              "title": "Musk family",
              "link": "https://en.wikipedia.org/wiki/Musk_family"
            },
            {
              "title": "Tesla Roadster",
              "link": "https://en.wikipedia.org/wiki/Elon_Musk%27s_Tesla_Roadster"
            },
            {
              "title": "Tesla, SpaceX, and the Quest...",
              "link": "https://en.wikipedia.org/wiki/Elon_Musk:_Tesla,_SpaceX,_and_the_Quest_for_a_Fantastic_Future"
            },
            {
              "title": "Maye Musk",
              "link": "https://en.wikipedia.org/wiki/Maye_Musk"
            }
          ],
          "rank": 1
        },
        {
          "title": "Elon Musk - Forbes",
          "displayed_link": "https://www.forbes.com › profile › elon-musk",
          "snippet": "Real Time Net Worth · Elon Musk cofounded seven companies, including electric car maker Tesla, rocket producer SpaceX and artificial intelligence startup xAI.Will Elon Musk’s Silicon Valley... · Forbes Real Time Billionaires · Tesla · Peter Thiel",
          "link": "https://www.forbes.com/profile/elon-musk/",
          "extended_sitelinks": [
            {
              "title": "Will Elon Musk’s Silicon Valley...",
              "link": "https://www.forbes.com/sites/gregorme/2024/12/11/will-elon-musks-silicon-valley-playbook-work-in-government/"
            },
            {
              "title": "Forbes Real Time Billionaires",
              "link": "https://www.forbes.com/real-time-billionaires/"
            },
            {
              "title": "Tesla",
              "link": "https://www.forbes.com/companies/tesla/"
            },
            {
              "title": "Peter Thiel",
              "link": "https://www.forbes.com/profile/peter-thiel/"
            }
          ],
          "rank": 2
        },
    .....
    ]

The above data contains various points, including titles, links, snippets, descriptions, and featured snippets like extended sitelinks. You will also get advanced feature snippets like People Also Ask For, Knowledge Graph, Answer Boxes, etc., from this API.

Conclusion

The nature of business is evolving at a rapid pace. If you don’t have access to data about ongoing trends and your competitors, you risk falling behind emerging businesses that make data-driven strategic decisions at every step. Therefore, it is crucial for a business to understand what is happening in its environment, and Google can be one of the best data sources for this purpose.

In this tutorial, we learned how to scrape Google search results using Python. If you found this blog helpful, please share it on social media and other platforms.

Thank you!

E-commerce v/s Retail: The Battle For Future

Darshan Khandelwal — Fri, 18 Oct 2024 19:10:00 +0000

Since the internet boom and the launch of online retail, US E-commerce sales have been growing exponentially and have reached a trillion dollar mark in 2022. However, that doesn’t mean any decrease in sales or the presence of physical stores.

According to NRF, nearly eighty percent of retail sales in the US are still happening from physical stores, highlighting this with the fact that physical stores were opened at an unexpected pace after the 2020 pandemic.

We do agree, that the convenience offered by E-commerce is undeniable allowing people to engage and access the product from anywhere and everywhere. However, the experience provided by brick-and-mortar stores is unmatched by online retail.

So, where does the future lie — retail or e-commerce?

The Difference Between Retail and E-commerce

Retail stores refer to traditional physical or brick-and-mortar stores with in-store purchases and physical interaction with the product. Popular examples include retail chains like Walmart, Best Buy, and The Home Depot.

On the other hand, e-commerce involves the selection of products from online stores, with digital transactions and home delivery. Amazon, eBay, and Target are some of the popular companies deriving the majority of e-commerce sales in the US.

The key differences between these two are the shopping experience — handling the product directly in physical stores versus viewing it online in e-commerce — and pricing, as online products are often priced lower than those in physical stores.

E-commerce vs. Retail for Businesses

For businesses, looking to expand their foothold, choosing between e-commerce and retail should be considered on various factors including the market gap, buying behavior, and whether the audience they are selling to is willing to purchase the product or not.

A product that belongs to a particular niche and is difficult to access with physical stores can select e-commerce as the best option through which it can cater to a global audience. However, the product which relies on physical demonstrations, and a localized network — can select retail as its preferred solution.

The broader reach in e-commerce can help you scale your business, however, the shipping costs, delivery delays, etc can make a dent in your profit. Physical stores are based on impulsive buying, face-to-face customer engagement, and physical interaction with the product which fosters customer trust and loyalty. But it also comes up with challenges like higher operation costs which makes it hard for smaller businesses to grow and survive in the market.

Let’s understand each of these factors in detail for a clear understanding.

Cost and Infrastructure

The first thing that crosses your mind when starting a new business is the initial cost of setting it up — How will I gather funds for the initial months? How long it will take to get the first customer? How long it will take to generate the first profit?

These questions are legit for an entrepreneur and the answers to these questions are necessary to be informed regarding future decisions.

Retailers

The investment cost for retail depends on various factors including real estate, rent, labor salaries, etc which makes it quite expensive to set up compared to an e-commerce store.

Moreover, a brick-and-mortar store is associated with recurring and long-term costs beyond the initial setup including property maintenance, insurance, and inventory management.

E-commerce

The initial investment required to set up an e-commerce business is relatively low. It requires setting up an e-commerce platform, domain, hosting, and other marketing and automated tools necessary for increasing the visibility of your platform.

Operations

Operations are a crucial part of running a business and they may vary for different types of professions and can not be put into one category. Similarly, there may be some differences and similarities between retail and e-commerce with regard to operational complexity. Let’s discuss them also:

Retailers

Retailers face increasing operational complexity, managing everything from suppliers to customers. This involves extensive processes, including inventory management, logistics, marketing, and customer service.

E-commerce

Handling operations in e-commerce is relatively easier than in retail. Many tasks can be automated, including:

Inventory management, which involves tracking various channels in real-time, such as website stock, warehouse stock, and third-party suppliers.
Logistical supply to customers, which can be automated through delivery services.
Customer service can be managed online without any face-to-face interaction.

However, this doesn’t mean you should completely rely on automation. Real-time monitoring is essential, and managing all the above operations still requires time and resources.

Marketing and Customer Acquisition

A marketing channel is important for businesses of any type and size. It nurtures the growth and development of the business allowing it to acquire leads and convert them into potential customers.

Retailers

Retail marketing is more of a local and physical thing relying mainly on in-store and off-store marketing tactics.

**In-store marketing **refers to strategies employed by retailers within the vicinity of their physical stores to influence the purchasing decisions of the consumers. This involves eye-catching product displays, point-of-purchase displays, promotions, and discounts aimed at encouraging immediate purchases.

Off-store marketing refers to the marketing strategies that take place outside the vicinity of the physical store to increase footfall and brand awareness. Word of Mouth, banners on the street and transport system, and sponsoring events are some of the strategies brought into play by retailers to expand their business reach.

E-commerce

E-commerce businesses heavily rely on digital marketing. SEO, social media marketing including ads and influencer marketing, and email campaigns are some of the factors that help businesses in their customer acquisition.

E-commerce vs. Retail for Customers

E-commerce offers numerous advantages and time-saving benefits for consumers. Shoppers can browse products online, compare prices across different options, and secure the best deal from the comfort of their homes while simply waiting for delivery. However, this doesn’t mean e-commerce has completely surpassed the retail market. In fact, retail still holds many advantages over e-commerce businesses.

Customer Experience

Customer experience plays a crucial role in shaping a brand’s success. Every interaction with a customer has the potential to strengthen or weaken the relationship they have with the company.

Retail

Face-to-face interaction is necessary for a brand to create a true connection with its customers. This personal interaction helps in the comprehension of customer’s emotions and feelings, which in turn fosters trust and strengthens the relationship with the brand.

The proactive support provided by the physical stores including immediate assistance, hands-on demonstrations, and checkout assistance adds to the experience of customers and is much better than dealing with potential issues on the chat.

E-commerce

Physical support shouldn’t be underestimated. However, the support provided by e-commerce businesses tends to be more reactive. Customers can reach out 24/7 via email, phone, or chat, without being limited by store hours.

Shopping Experience

According to a report by PWC, 32% of all customers will stop doing business with the brand they are loyal to after a bad customer experience(source).

Therefore, delivering a pleasant and positive shopping experience is crucial for both retail and e-commerce stores. There’s no reason a customer would choose to buy from a place that complicates the purchasing process.

Retail

The retail shopping experience is more grounded in physical interaction. Customers can touch, feel, and try out products before buying, making the purchasing process more interactive and immediate.

This leverage over e-commerce allows retailers not only to facilitate shopping at their stores but also to connect with customers on a personal level.

E-commerce

Though e-commerce doesn’t allow for direct interaction with customers, it enhances their shopping experience by offering key advantages, *convenience *— 24/7 shopping and customer support, *efficiency *— easy product comparisons and data-driven personalization, and *flexibility *— the ability to browse and purchase from anywhere at any time.

Convenience

Convenience in retail and e-commerce refers to how much easier and faster customers can complete the shopping process — From finding products to making a purchase.

Retail

Retailers may not be able to provide the level of convenience compared to e-commerce. But, there are some options to consider that amplify the experience, including immediate product access, personalized service, and physical interaction, which help level up the convenience offered by retail.

E-commerce

The concept of E-commerce is based on convenience. It allows consumers to look for products from anywhere at any time, compare prices from different online retailers, read reviews from previous customers, and apply offers to multiple payment options, which makes it easier to get the best product from the market.

What to choose — Retail Or E-commerce?

These factors are important to consider before choosing between retail and e-commerce for your business:

Business Model

Knowing the business model is important in determining which option would be best in the future. Retail is an ideal option if your business relies on customer experience, local demand, and physical interaction with the products.

However, selecting e-commerce would be the best decision if your company is dealing in B2B and C2C. It also works best for businesses that are looking to scale at a global level or sell niche products on high margins.

Target Audience

Identifying the target audience before starting a business is the most important step. Is your audience tech-savvy enough to prefer e-commerce? Does your audience prefer an in-person shopping experience over the convenience of online shopping?

Operational Costs

Retail in general requires a huge upfront investment, and monthly recurring expenses, including salaries, maintenance, insurance, and rent. An e-commerce business comes with a comparatively lower cost than retail with an upfront investment in domain, website hosting, logistics, and marketing.

Conclusion

Both retail and e-commerce have their own plus points and assuming that retail will not be dominated or it will be dead after some time is a very naive decision. The best answer for the retail vs e-commerce debate would be to ask yourself the motive of the business or what it is trying to achieve.

I hope you like this blog. Feel free to message me anything you need clarification on. Follow me on Twitter. Thanks for reading!

Best Price Monitoring Tools in 2024

Darshan Khandelwal — Fri, 27 Sep 2024 21:21:28 +0000

With the availability of multiple e-commerce platforms, customers can easily compare the pricing over several platforms before deciding to purchase. However, if you are a retailer, you cannot compare the pricing of each product with that of your competitors. This is not a practical solution and you would waste a lot of time!

As we know, the e-commerce industry has been facing cut-throat competition, which is making getting ahead of the competition harder than ever. This is where price monitoring tools come into play. These tools are specifically designed to track and monitor the changes and strategies implemented by the competitors in real time.

Over time, price monitoring tools have become much more advanced and offer many features like automated price tracking, automated alerts, and insights into pricing trends. With the right tool, retailers can not only stay competitive but also predict market shifts and respond before competitors do.

What is Price Monitoring

Price Monitoring is the process of analyzing prices of products and services using various techniques including web scraping, machine learning, and data analysis. It involves creating large spiders that collect products from multiple platforms such as Amazon, Walmart, etc, to create pricing strategies, track stock availability, sentimental analysis on customer reviews, and much more.

Best Price Monitoring Tools in 2024

Here are the top 10 Price Monitoring tools of 2024 that will help you empower your e-commerce strategy:

EcommerceAPI

EcommerceAPI is a dedicated API made for e-commerce platforms to scrape real-time product data at scale from multiple channels at a time. Its robust infrastructure is capable of handling millions of API calls and is backed by an advanced infrastructure that returns you with readymade JSON data from websites like Amazon, Walmart, Flipkart, etc to help you extract any information about the product and its pricing easily.

It has dedicated documentation for each of the APIs, and 24/7 active support to help you out with any problem.

Features:

Country and postal code targeting.
Supports custom changes in the API and has integrations in all major languages.
Supports featured snippets like sponsored results, videos, etc.

Pricing:

Pricing starts from 30$ with 150k credits.

Prisync

Prisync is one of the best price monitoring tools out there in the market. It is SAAS-based price-tracking software offering a free 14-day trial to test their services. They can monitor an unlimited number of competitors irrespective of the plan.

Features:

Real-time Price Monitoring.
Provides historical trends to check when the competitor can change its pricing.
Stock Availability Monitoring.

Pricing:

The pricing starts from 199$ with a limit of 100 products.

Price2Spy

Price2Spy is a popular feature-packed price-monitoring tool that monitors and accesses the competitor's product pricing and also collects data over time to provide its users with historical pricing data so you can set your product pricing strategically to attract customers.

Their dashboard has advanced features where you can find the most up-to-date pricing data and send price alerts over email.

Features:

Dynamic Pricing Functionality.
Price Alert over the email.

Pricing:

Pricing starts from 67.95$ in which a user can add 500 **product URLs and up to **50 competitors.

Skuuudle

Skuuuddle is a well-trusted British price intelligence and monitoring company that lets you get details about more than millions of products across multiple marketplaces and allows you to set dynamic repricing rules in response to the current market conditions.

Features:

Large Scale Ecommerce Scraping.
MAP Monitoring is also available.

Pricing:

You need to consult with customer support for the pricing model and solutions you need.

Competera

Competera.ai is a powerful AI-driven platform that reshapes the complex part of pricing into data-fueled science. It is trusted by major retailers worldwide who have increased their profit margins by 8% by mastering the pricing game with precision and intelligence using well-calibrated strategies designed for increasing profit margins, winning over customers, and keeping you ahead of the competition.

Pricing is the most important part of a product. Too high, you will be losing customers, or too low you will be leaving money on the table. But Competera’s AI stands out by analyzing large amounts of data including, historical sales trends and competitor pricing patterns — and provides AI-based price feedback that helps to elucidate the customers.

Features:

AI-powered price optimization.
Focuses on long-term sustainable growth strategies.
Real-time price adjustments based on competitor moves.

Pricing:

Consultation is required before getting the pricing.

Conclusion

Offering quality products is not sufficient to win the war in the market, it is also about winning the delicate dance of pricing and keeping up with current e-commerce trends. With the above best price monitoring tools you’re well informed by data and acting with precision. Each tool comes with its own unique features and can help you navigate the right path in this high tide of competition.

In this tutorial, we explored the best price monitoring tools in the market. I hope you like this blog. Feel free to message me anything you need clarification on. Follow me on Twitter. Thanks for reading!

Additional Resources

Scraping E-commerce Platforms with Python

Darshan Khandelwal — Fri, 27 Sep 2024 21:18:56 +0000

Scraping E-commerce Platforms with Python

The online retail or e-commerce industry has seen a fast pace of growth since its launch. This boom is also witnessing the rise of 10-minute delivery models, revolutionizing how customers interact with e-commerce platforms.

This article will explore how to scrape data from e-commerce platforms using Python and the E-commerce Data API.

What is E-commerce Scraping?

E-commerce web scraping involves extracting publicly available data from e-commerce platforms such as Amazon, Walmart, and Flipkart. This data can be used to compare prices, track competitors, understand customer preferences, make data-driven decisions, and stand out in the fierce competition.

It offers various use cases for businesses to grow their digital presence, including price monitoring, market trends forecasting, price prediction, product data enrichment, and more.

So far, we have covered the basics of e-commerce scraping. Let us now explore how we can implement it with Python.

Is it legal to scrape E-Commerce platforms?

In short, it is legal to scrape e-commerce platforms as long as the data being extracted is publicly available. The data generally used by businesses includes product information, customer reviews, and pricing data, which is available to everyone and is completely legal to scrape since you are not accessing any private information of the platform or its users.

However, it is important to respect the website’s terms of service before extracting the data and to avoid overloading their servers with excessive requests, which can severely affect not only the website but also your data collection process.

Prerequisites

For those who have not installed Python, you can download it from here. After downloading Python, we will install the libraries we will use in this project.

pip install requests

So, now we are done with the setup. Let’s create a new file in our project folder and start the project.

Building the Scraper

To build our scraper, we need to first import the library we installed earlier.

import requests

As we will use the EcommerceAPI to retrieve the data, you will need an API Key from its dashboard to collect the data. If you haven’t registered already, you can sign up to get the API Key and 1000 free credits for testing purposes.

After successfully registering, you can add the API Key to your code.

api_key = "xxxx8977ac"

For the sake of this tutorial, we will be scraping Walmart.

Making an API request on EcommerceAPI is straightforward. You just need to pass the API key and the platform URL to scrape the results.

base_url = "https://api.ecommerceapi.io/walmart_search"

params = {
    "api_key": api_key,
    "url": "https://www.walmart.com/search?q=football"
}

Now that we have the base URL and parameters ready, we will establish an HTTP GET connection using Python’s Requests library.

response = requests.get(base_url, params=params)

print(response.json())

This will return you the meta information and the search results from the Walmart search page. However, we only need the list of products from the search results to access the pricing information.

If you examine the returned response, you will find that the products are within the search results array. Let’s access it.

data = response.json()

search_results = data.get('search_results', [])  

print(search_results)

This will give you the following output:

Alternatively, you can loop through each item to retrieve the pricing and other details of the product.

    # Extract and print the current_price for each product
for product in search_results:

    for item in product['item']:
        print(f"Product Title: {item['title']}")
        print(f"Current Price: {item['current_price']}")
        print('-' * 40)

Easier isn’t it? You don’t even need to parse complex HTML structures; the ready-made JSON data is available to you within seconds.

Here is the complete code:

import requests

api_key = "xxxx8977ac"

base_url = "https://api.ecommerceapi.io/walmart_search"

params = {

    "api_key": api_key,

    "url": "https://www.walmart.com/search?q=football"

}

response = requests.get(base_url, params=params)

data = response.json()

search_results = data.get('search_results', [])

print(search_results)

for product in search_results:

for item in product['item']:
    print(f"Product Title: {item['title']}")
    print(f"Current Price: {item['current_price']}")
    print('-' * 40)

Conclusion

The web scraping community has developed various techniques to extract data from e-commerce platforms, making it easier than ever. Techniques include bypassing CAPTCHAs or other blocking mechanisms using configurations that help our IP address avoid getting blocked by the website.

However, if you need to perform this method at scale, relying on a single IP with basic infrastructure may not suffice. In such cases, using an e-commerce scraper API would be ideal. It helps you collect data at scale without facing obstructions and at an economical price.

In this article, we learned how to use Python for scraping e-commerce platforms. With this basic technique, you can develop your scraper to perform data extraction at scale.

Best Amazon Scraper APIs To Check Out in 2024

Darshan Khandelwal — Sun, 09 Jun 2024 10:44:29 +0000

The E-commerce Industry was valued at 25.93 trillion $ and is expected to rise at 18.4% CAGR from 2024 to 2030(source). With this exponential rise of the e-commerce industry, Amazon has become the industry leader with a current market capitalization of more than 1.93 trillion $.

From this mammoth size, one can estimate the volume of products this particular company is handling per day and the huge load of data stored in their data centers including user data, product data, and many other things. However, getting this data from Amazon may not be an easy task for a lot of web scraping experts.

This is where Amazon Scraper APIs come into play. They are highly scalable and allow users to extract data from Amazon without facing any blockage issues.

In this article, we will list the 5 best-performing Amazon Scraper APIs that can be utilized for large-scale scraping to gather product information.

The Top 5 Amazon Scraper APIs in 2024

1. EcommerceAPI

EcommerceAPI tops our list of the best-performing Amazon Scraper APIs. Its highly scalable and robust API can be efficiently used for scraping Amazon Search, Product, and Reviews Pages.

It is the first dedicated provider in the industry to offer scraping services exclusively for e-commerce platforms, including Amazon, Walmart, Google Shopping, and more to come. It also supports the much-needed country-level targeting, allowing users to get accurate and precise results.

Here is the list of Amazon APIs offered by the EcommerceAPI:

Amazon Search API — To get the list of products from the Amazon Search Engine.
Amazon Product API — To get the product data in detail including its pricing, description, ratings and reviews, and much more.
Amazon Reviews API — To get customer reviews for a particular product.
Amazon Autocomplete API — To get the search suggestions for a given search query on Amazon.

Features:

Comprehensive documentation and integration with every major language.
It supports country-level targeting for all the marketplaces.
Structured JSON output for each API is available and it also offers custom changes in the API on customer demand.
Supports extra featured snippets on Amazon including sponsored brand videos, sponsored brand results, and brands related to that search.

Pricing:

Pricing starts from 30$ and offers 150k credits making it the most economical Amazon API in the market.

2. Scrapingdog

Scrapingdog is a web scraping company that also specializes in delivering Amazon Scraping Services to its customers. Their Amazon Scraper is based on a huge network of more than 40M+ residential and datacenter IPs with a higher success rate.

Scrapingdog’s Amazon Scraper API also supports country-level and postal-level targeting. Moreover, they already have a huge infrastructure to defy any anti-bot mechanism present on the website, as stated on their website, “You can focus on using the data, not collecting it”.

Scrapingdog parses data from Amazon Search and Product page and offers the following features:

Country and postal level targeting.
Structured JSON output.
It also supports featured snippets such as sponsored brands and brands related to the search.

Pricing:

Pricing starts from 40$ and offers 200k credits.

3. Oxylabs

Oxylabs is another big data provider in the market that provides access to valuable Amazon data, such as pricing, product information, or reviews, on a large scale. Their API can be used for various purposes including Price and Product Monitoring, competitor monitoring, etc.

Additionally, their scraper is regularly maintained and they have created a parser for different types of pages on Amazon so that their scraper doesn’t break on layout changes.

Not only Amazon, but Oxylabs covers all the major e-commerce platforms including eBay, Etsy, Walmart, Flipkart, and many more.

Features:

Real-time data from any country.
JSON output for each type of layout is available.
Supports JS rendering.

Pricing:

Pricing starts from 49$ and offers 17.5k credits only.

4. BrightData

Every person in the data scraping industry may have heard of the name BrightData. They are the biggest web scraping and proxy provider company in the industry, which offers dedicated scrapers for every major website known for collecting data by developers worldwide, including Amazon.

You can use both their proxies or dedicated API to scrape Amazon. However, the dedicated is preferred more due to the in-built bot-defying system.

The only disadvantage with their API is that they don’t provide structured JSON output making it harder to access the data and maintain the parser for each layout on the user end.

Features:

Country and city-level targeting.
JSON output.
The Amazon dataset is also available.

Pricing:

The pricing structure is not clearly defined however, it starts from 0.001 $ per record scraped.

5. Smartproxy

Smarproxy is another proxy provider that offers a dedicated section for scraping e-commerce platforms including Amazon. Their Amazon API is known for its speed and quality and can be used just with a simple POST request.

Smarproxy not only provides data in JSON format but also CSV files for non-developers to test their API.

Features:

Country and city-level targeting.
JSON output and CSV output.
Featured sponsored results are available.

Pricing:

Pricing starts from 30$ and offers 15k credits.

Conclusion

Amazon is generally scraped at a huge scale by businesses as the e-commerce industry is always changing dynamically. The best solution is that can handle millions of requests without any blockage and can serve you with precise and accurate data.

Additional Resources

Understanding E-commerce API: Developing an Application for Live Price Tracking

Darshan Khandelwal — Thu, 23 May 2024 12:32:09 +0000

An Application Programming Interface (API) is a way for computers to communicate with each other. It helps businesses maintain coordination between their system software to increase overall efficiency and productivity.

APIs play a significant role in the e-commerce industry by providing online retailers with crucial information about their competitors’ pricing and product strategy, allowing them to adjust their marketing strategy in real time.

In this article, we will explore what an e-commerce API is and how we can develop a real-time price-tracking application to maintain a competitive stature in the market.

What is E-commerce API?

In retrieving product information, an e-commerce data API can be defined as a set of rules and protocols that extract product information, including pricing, descriptions, features, and other relevant information, from e-commerce platforms.

E-commerce APIs are crucial for online retailers and businesses because they help streamline marketing and business operations, optimizing logistics to align supply with current market demand.

The Need For Live Price Tracking

The Price Tracking service is the need of the current hour for online retailers and businesses. It enables you to collect key pricing changes and optimize your pricing accordingly. Analyzing trends in pricing fluctuations resulting from holidays, special offers, or other dynamic changes can help forecast how pricing metrics will evolve.

Price Tracking also has the powerful capability to check if competitors are out of stock on any items, and if so, it can track the duration for which these items have been unavailable. This information can be used to adjust product prices strategically, maximizing profits and achieving a significant competitive advantage in terms of sales and margins.

Additionally, thanks to real-time price tracking, customers can secure the best deals and receive alerts about significant price drops on big-ticket items. Even a small percentage decrease in these products can lead to substantial savings. Customers can also capitalize on trend forecasting by utilizing historical price charts offered by price tracking tools.s

Features of Price Tracking Apps

Live Price Tracking Apps come with a variety of features designed to enhance the overall shopping experience for the customers:

Real-time Alerts: This feature offers significant advantages for both retailers and customers! Live tracking apps can send immediate notifications about price drops, ensuring consumers secure the best-saving deals. Even a slight change in pricing can potentially drive customers away, and without utilizing software, it’s challenging to make dynamic changes. This is where price tracking becomes essential for retailers, alerting them to every price drop and enabling live adjustments to be made.

Historical Price Data: The E-commerce industry is experiencing significant shifts in product pricing, leading to noticeable differences within short timeframes. By tracking historical price data for various products, retailers can analyze pricing trends and better understand consumer behavior. This insight enables them to optimize pricing strategies effectively and stay competitive in the market.

Price Comparison: Price comparison is one of the main uses of live price tracking apps. It allows retailers and consumers to look for the same product across different retailers. This helps retailers find out which online businesses are selling the same product with lower pricing while for customers, it allows them to find the lowest possible pricing.

Market Dynamics: Historical pricing data provided by price-tracking apps assists retailers in identifying patterns of pricing changes over time. These fluctuations often result from shifts in demand caused by supply chain disruptions, seasonal variations, and competitor actions. Analyzing these historical graphs helps in predicting potential future trends, enabling retailers to adjust pricing strategies accordingly.

Steps to build a Live Price Tracking APP

Building a live price-tracking app involves several steps before deployment. Here’s a step-by-step guide to help you get started:

Goals and Requirements

Thorough market research is essential when determining the types of products or services that the app will track. This includes researching competitor applications to identify the features they offer and analyzing customer reviews to gain a better understanding of customer expectations. Additionally, identifying the target audience that will use the app is crucial, as it will influence the app’s design to meet their specific requirements.

APIs Selection

Now that you have analyzed and listed out the features necessary for your application, select the APIs that will provide you with live data from E-commerce Platforms consistently without any downtime. There are both official APIs and third-party APIs available; however, the official APIs of Amazon, Walmart, and eBay are much more expensive to use than third-party APIs, and they also don’t offer any customization or flexibility with the data.

Designing the UI

After selecting the APIs, begin designing a user-friendly UI that will enable users to view beautiful historical charts, add products to a tracking list, receive notifications when there is a price drop, and easily navigate through different pages using well-organized sidebars or footer menus, as well as other relevant designs.

Integrate APIs and Prototype Testing

Integrate the selected APIs in your application and fetch the real-time data, parse it at the backend transfer it to the front end, and show it to the user. Similarly, using the fetched data test the application functionality includes price drop notifications, pricing charts, and real-time price changes.

Deployment

After completing testing, ensure that your application is bug-free and does not cause any disruptions to the user experience. Finally, deploy your application on platforms such as the Google Play Store and Apple App Store to reach your target audience and gather their feedback to further enhance the app.

Conclusion

In summary, live price tracking using e-commerce APIs helps retailers and digital businesses gain a more precise understanding of market dynamics. This not only benefits retailers but also customers, who can secure the lowest possible price for their desired products. Such tools have become essential for businesses to sustain growth in a competitive environment.