Forem: Hilman Ramadhan

Grok AI API tutorial - from Chat to Web Search

Hilman Ramadhan — Mon, 30 Mar 2026 02:03:37 +0000

The xAI Grok API provides access to powerful frontier models like Grok 4 series, supporting chat completions (text + vision), image generation, tool calling (function calling + built-in tools like web search), and more advanced features.

Quick Intro

Sign up at https://x.ai/api
Generate an API key from the console
Install: pip install xai-sdk
Set env var: export XAI_API_KEY="your_key_here"
Models list: https://docs.x.ai/developers/models

I'll share some samples in Python.

Basic Chat API Call

Let's first prepare our project before making the API call

Install the xai-sdk

pip install xai-sdk

Set env var: export XAI_API_KEY="your_key_here" or use .env file

Now, create a new file and this basic setup:

import os
from xai_sdk import Client
from xai_sdk.chat import user, system

from dotenv import load_dotenv
load_dotenv()
XAI_API_KEY =  os.environ.get("XAI_API_KEY")

client = Client(api_key=XAI_API_KEY)

Ensure you can print out your XAI_API_KEY correctly at this stage.

Next, let's call the chat function

...
model = "grok-4-1-fast-non-reasoning"
chat = client.chat.create(model=model)
chat.append(system("You are Grok, a highly intelligent, helpful AI assistant."))
chat.append(user("How can I be a good developer?"))

response = chat.sample()
print(response.content)

Feel free to switch the model based on your needs or preferences.

Here is an example output:

Image Generation API

Let's see how to generate an image with Grok API. We'll need to use the "grok-imagine-image" model for this.

...
response = client.image.sample(
    model="grok-imagine-image",  
    prompt="detective cat searching on website"
)

print(f"Generated image: {response.url}")

The output is a URL like this:

Video Generation API

Generating a video is as easy as generating an image with Grok API. We'll need to use the "grok-imagine-video" model for this.

response = client.video.generate(
    prompt="A glowing crystal-powered rocket launching from the red dunes of Mars, ancient alien ruins lighting up in the background as it soars into a sky full of unfamiliar constellations",
    model="grok-imagine-video",
    duration=10,
    aspect_ratio="16:9",
    resolution="720p",
)
print(response.url)

You can set the duration, aspect ratio, and resolution.

The xAI Grok API features powerful tool calling capabilities, allowing Grok to go far beyond simple text generation. It can take real actions such as performing web searches, running code, retrieving information from your own data sources, or invoking any custom functions you've defined.

Let's start by calling a custom function, as it'll help us call any internal or external API or function.

Let's say we want to call a function to look for an item's price. First, we need to define the function, such as adding the name, description, and parameters.

...
import json
from xai_sdk.chat import user, tool, tool_result
...

# Define tools
tools = [
    tool(
        name="get_item_price",
        description="Get the price of an item from the store",
        parameters={
            "type": "object",
            "properties": {
                "item_name": {"type": "string", "description": "Name of the item to get the price for"},
            },
            "required": ["item_name"]
        },
    ),
]

Upon calling the client method, we now need to include the tool we declared above.

chat = client.chat.create(
    model="grok-4.20-reasoning",
    tools=tools,
)
chat.append(user("What is the price of a laptop?"))
response = chat.sample()

print("========= response ===========")
print(response)
print("==========================")

Important: At this stage, Grok doesn't care if we have the actual function to check the price or not. The AI simply wants to know "what tools are available" for them to use.

Try to run the code to see the output from the chat call.

As you can see, Grok can detect the tool we need to call. You can see it from outputs > message > tool_calls . It consists of the name of the function and the arguments that are extracted from the user's prompt, so it'll be dynamic.

Function call simulation

Next, let's create a fake function to call. In real life, it could be a call to a database or APIs.

def get_item_price(item_name):
    prices = {
        "laptop": 999.99,
        "smartphone": 499.99,
        "headphones": 199.99,
    }
    return {"item_name": item_name, "price": prices.get(item_name, "Item not found")}

Following up from the latest code, we can check if the response has a "tool_calls" object or not. If so, we'll call the actual function we just declared above.

# Handle tool calls
if response.tool_calls:
    chat.append(response)
    for tc in response.tool_calls:
        args = json.loads(tc.function.arguments)
        result = get_item_price(args["item_name"])
        chat.append(tool_result(json.dumps(result)))


    response = chat.sample()

print(response.content)

We need to loop through the tool_calls object
We need to extract the argument to pass to the function
Call the actual function alongside the argument value
Add the information back to our chat method

Now, calling the chat.sample() method, will include all the information we received from calling the "fake function" before.

Let's try with a different prompt

chat.append(user("I need to buy two laptops and a smartphone. Can you tell me how much that will cost?"))

Here is the result

Web Search API

Grok can access real-time information through this feature, so you can get up-to-date content. Unlike the function calling above, we don't need to declare a custom function, as it's an internal tool. Here is a simple example:

import os
from xai_sdk import Client
from xai_sdk.chat import user
from xai_sdk.tools import web_search

from dotenv import load_dotenv
load_dotenv()
XAI_API_KEY =  os.environ.get("XAI_API_KEY")

client = Client(api_key=XAI_API_KEY)

chat = client.chat.create(
    model="grok-4.20-reasoning",  # reasoning model
    tools=[web_search()],
    include=["verbose_streaming"],
)

chat.append(user("Grok VS OpenAI API"))

is_thinking = True

for response, chunk in chat.stream():
    for tool_call in chunk.tool_calls:
        print(f"\nCalling tool: {tool_call.function.name} with arguments: {tool_call.function.arguments}")
    if response.usage.reasoning_tokens and is_thinking:
        print(f"\rThinking... ({response.usage.reasoning_tokens} tokens)", end="", flush=True)
    if chunk.content and is_thinking:
        print("\n\nFinal Response:")
        is_thinking = False
    if chunk.content and not is_thinking:
        print(chunk.content, end="", flush=True)

print("\n\nCitations:")
print(response.citations)

Use tools=[web_search()]
To show what's happening in the process, we use include=["verbose_streaming"],
is_thinking variable is to check if the process is still running (a boolean variable)

As you can see, it'll perform several searches on the internal with different queries. It'll then visit a specific URL after that to get more context.

Allowed domains

You can search only in specific domains using allowed_domains.

 tools=[
        web_search(allowed_domains=["grokipedia.com"]),
    ],

Exclude Domains

Vice versa, you can exclude specific domains

chat = client.chat.create(
    model="grok-4.20-reasoning",
    tools=[
        web_search(excluded_domains=["grokipedia.com"]),
    ],
)

Better Web Search API

While you can specifically choose the domain, the keyword Grok uses to find answers on the internet is random. For example, when I'm asking for "Top 3 pizza restaurants from Google Maps in Boston. Share some reviews and ratings for each place."

This is what I saw from the thinking process:

It needs to perform multiple queries before returning the answer.

Another sample, when asking simply for 3 images:

It runs on multiple pages, and unfortunately, the links are not valid. Grok may hallucinate at this point.

Web Search API alternative

In some cases, AI-generated keywords are fine, but if you're building an app where you want efficiency and full control of the process, the native "Web Search Tool" can be replaced with a simple API call to a specific API your app needs. For example, to find answers on the internet, SerpApi provides 100+ APIs for you.

Need a generic Google answer? We have:

- Google Search API

- Google AI Overview

- Google AI Mode

and more!

See how SerpApi is the Web Search API for your AI apps, LLM, and agents.

Using Grok API with SerpApi

To get some idea of how SerpApi works, feel free to test the results on our playground. You can play with different parameters and directly see the JSON sample we return.

Sample case

Let's say we want to find images via Google Image API like this:

Step 1: Preparation

You can register for free at serpapi.com to get your API key.

Step 2: Parsing keyword

Let's say we need 3 images from Google. Since users can type anything, we need to parse the keyword, as SerpApi simply performs a search using a particular keyword.

USER_QUERY = "Show me 3 cute cat images from the internet"

# Step 1: Ask Grok to extract a search keyword from the user's natural language
keyword_chat = client.chat.create(model="grok-3-fast")
keyword_chat.append(system("Extract the most relevant search keyword or phrase from the user's message. Reply with only the keyword, nothing else."))
keyword_chat.append(user(USER_QUERY))

keyword_response = keyword_chat.sample()
search_keyword = keyword_response.content.strip()
print(f"Extracted keyword: {search_keyword}")

Step 3: Search via SerpApi

We now have the keyword. Let's run a search on SerpApi

# Step 2: Search via SerpAPI using simple requests (Google Images)
serpapi_params = {
    "api_key": SERPAPI_API_KEY,
    "engine": "google_images",
    "q": search_keyword,
    "hl": "en",
    "gl": "us",
}

serpapi_url = "https://serpapi.com/search"
serpapi_response = requests.get(serpapi_url, params=serpapi_params)
results = serpapi_response.json()

At this stage, you already have the answers you're looking for.

Step 4: (Optional) Filter results

Sometimes, we don't need all the information. It's good to filter it programmatically first, so we don't use too many tokens.

For example, I'm only interested in the top 5 answers:

image_results = results.get("images_results", [])
[:5]
formatted_results = "\n".join(
    f"- {img.get('title', 'No title')}: {img.get('original', img.get('thumbnail', 'No URL'))}"
    for img in image_results
)
print(f"\nSerpAPI results:\n{formatted_results}")

We can also format the answer as a bonus.

Step 5: (Optional) Reply in a natural language

Depending on your application, you may want to answer the user back in natural language. We just need to pass the answers above back to the AI:

# Step 3: Feed results back to Grok for a final response
final_chat = client.chat.create(model="grok-3-fast")
final_chat.append(system("You are a helpful assistant. Use the provided search results to answer the user's question."))
final_chat.append(user(f"User question: {USER_QUERY}\n\nSearch results from SerpAPI:\n{formatted_results}\n\nPlease answer the user's question based on these results."))

final_response = final_chat.sample()
print(f"\nFinal Response:\n{final_response.content}")

Final result:

Sidenote

It's also possible to call the API with the OpenAI SDK. Sample:

from openai import OpenAI

client = OpenAI(
    api_key=os.getenv("XAI_API_KEY"),
    base_url="https://api.x.ai/v1",
)

How to scrape 100 search results on Google (num parameter)

Hilman Ramadhan — Thu, 25 Sep 2025 01:48:54 +0000

Finally, there is a solution to scrape 100 search results from Google after they dropped the num parameter: introducing the Google Fast Light API

Google recently discontinued support for showing up to 100 results per page in search, a parameter that many SEO professionals, data companies, and researchers have relied on for years. This change disrupts established workflows, making large-scale data collection, competitor analysis, and keyword research less efficient.

API to scrape 100 search results from Google

Introducing Google Light Fast API

With this API, you can scrape 100 search results from Google with one single request. This API only contains the organic_results , which makes it faster compared to our regular Google Search API.

You can access this API with a single GET request:

https://serpapi.com/search.json?engine=google_light_fast&q=coffee&api_key=YOUR_API_KEY&num=100

You can replace coffee with any keyword you want to search for.
To get your API key, you can register for free at SerpApi.

Scraping other rich results from Google

If you need the complete Google results on the first page, you can still use our Google Search API in order to scrape AI Overview answer, knowledge graph, and so on. So, if you need both 100 organic results and complete rich results , you only need to perform two searches:

1 to get the 100 organic results via the new Google Fast Light API
2 to get the rich results on the Google Search API.

No need to paginate the results or run 10 searches anymore.

Conclusion

The removal of the num=100 (&num=100) parameter from Google has a lot of impacts on many SEO companies that need to serve ranking data for their clients. Many search API providers fail to scrape 100 search results, which results in many errors and misinformation on their platform.

This issue was also mentioned a couple of times on X (Twitter) by many search engine professionals,

Barry Schwartz - founder of the Search Engine Roundtable

Brodie Clark - Independent SEO consultant

SEO companies looking for a way to track their clients’ SEO rankings programmatically, as well as AI companies seeking more real-time data from Google, should try this new API.

How to scrape Tripadvisor (2025 Tutorial)

Hilman Ramadhan — Fri, 19 Sep 2025 02:17:42 +0000

Scraping Tripadvisor listings allows you to gather detailed information about various attractions, hotels, or restaurants, including descriptions, amenities, pricing, and user-generated ratings. This data can be instrumental in understanding market trends, identifying competitors, and optimizing your own listings or offerings in the travel industry.

Luckily, you can do this easily using our brand new Tripadvisor API. We'll see how to scrape it in cURL, Python, and JavaScript. We can scrape restaurants, things to do, hotels, destinations, vacation rentals, and forum information.

Available data on the Tripadvisor Search API

Here is the list of data you can retrieve from this Tripadvisor Search API:

title
description
rating
reviews
location
thumbnail
highlighted overview

This is perfect if you need to collect place data from Tripadvisor.

How to scrape the Tripadvisor website?

Now, let's see how to use this simple API from SerpApi to collect the data!

Get your API Key

First, ensure you register at serpapi.com to get your API Key. You can get 250 free searches per month. You can use this API Key to access all of our APIs, including the Tripadvisor Search API.

Available parameters

On top of running the basic search, you can see all of our available parameters here.

cURL Implementation

Here is the basic implementation in cURL:

curl --get https://serpapi.com/search \
 -d api_key="YOUR_API_KEY" \
 -d engine="tripadvisor" \
 -d q="Rome"

The q parameter is responsible for the search query.

Sample response from cURL request

Scrape Tripadvisor search results in Python

Next, let's see how to scrape the Tripadvisor search results in Python.

Preparation for accessing the SerpApi API in Python

Create a new main.py file
Install requests with:

pip install requests

Here is what the basic setup looks like:

import requests
SERPAPI_API_KEY = "YOUR_REAL_SERPAPI_API_KEY"

params = {
    "api_key": SERPAPI_API_KEY, #replace with your real API Key
    # soon
}

search = requests.get("https://serpapi.com/search", params=params)
response = search.json()
print(response)

With these few lines of code, we can access all of the search engines available at SerpApi, including the Tripadvisor Search API.

import requests
SERPAPI_API_KEY = "YOUR_SERPAPI_API_KEY"

params = {
    "api_key": SERPAPI_API_KEY, 
    "engine": "tripadvisor",
    "q": "indonesia"
}

search = requests.get("https://serpapi.com/search", params=params)
response = search.json()
print(response)

To make it easier to see the response, let's add indentation to the output.

import json

# ...
# ...
# all previous code

print(json.dumps(response, indent=2))

Running this Python file should show you the listing for that keyword from Tripadvisor website.

Print specific information

Let's say we only need the title, description, rating, and reviews. This is how we can print specific columns from the locations results:

# ...
search = requests.get("https://serpapi.com/search", params=params)
response = search.json()

for result in response.get("locations", []):
    title = result.get("title")
    rating = result.get("rating")
    reviews = result.get("reviews")
    description = result.get("description")

    print(f"Title: {title}")
    print(f"Rating: {rating}")
    print(f"Reviews: {reviews}")
    print(f"description: {description}")
    print("-" * 10)

Export data to a CSV file

Let's see how to export this Tripadvisor product data into a CSV file in Python

# ...

import csv

with open("tripadvisor_results.csv", "w", newline="", encoding="utf-8") as csvfile:
    fieldnames = ["title", "rating", "reviews", "description"]
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)

    writer.writeheader()
    for result in response.get("locations", []):
        writer.writerow({
            "title": result.get("title"),
            "rating": result.get("rating"),
            "reviews": result.get("reviews"),
            "description": result.get("description")
        })
print("Data exported successfully.")

JavaScript implementation

Finally, let's see how to scrape the Tripadvisor search results in JavaScript.

Install the serpapi package:

npm install serpapi

Run a basic query:

const { getJson } = require("serpapi");
getJson({
  engine: "tripadvisor",
  api_key: API_KEY, // Put your API Key
  q: "indonesia"
}, (json) => {
  console.log(json["locations"]);
});

Other programming languages

While you can use our APIs using a simple GET request with any programming language, you can also see our ready-to-use libraries here: SerpApi Integrations.

How to customize the search?

We provide many filters to customize your search.

tripadvisor_domain: Parameter defines the Tripadvisor domain to use. It defaults to tripadvisor.com. Head to Tripadvisor domains for a full list of supported domains.
lat and lon: Specify the lat and lon GPS coordinates when performing the search

Example using a different Tripadvisor domain:

curl --get https://serpapi.com/search \
 -d api_key="YOUR_API_KEY" \
 -d engine="tripadvisor" \
 -d q="cafe" \
 -d tripadvisor_domain="wwww.tripadvisor.ca"

Advanced Parameters:

ssrc: This parameter specifies the search filter you want to use for the Tripadvisor search.

Available options:

a - All Results

r - Restaurants

A - Things to Do

h - Hotels

g - Destinations

v - Vacation Rentals

f - Forums

How to paginate the results?

You can scrape beyond the first page using the offset and limit parameter.

limit defines how many results you want to receive per page. By default, it's 30. Maximum value is 100.
Offset skips the given number of results. For example, 0 for the first page, 30 for 2nd page, 60 for 3rd page, and so on. Or if you're using 100 as the limit, then the 2nd page will be 100, 3rd page will be 200, and so on.

Frequently Asked Questions (FAQs)

Is it legal to scrape the Tripadvisor website?

Scraping publicly available data from websites like Tripadvisor is generally permitted under U.S. law.

How much does it cost?

Register at serpapi.com to start for free. If you want to scale, we offer tiered plans based on your usage.

Why do you need to scrape Tripadvisor listing?

Scraping Tripadvisor listings can provide valuable data for market analysis, competitive insights, and understanding customer preferences in the travel and hospitality industry. This information can help businesses optimize their offerings and improve customer engagement.

Closing

That's it! Thank you very much for reading this blog post. You can play around for free on our playground here.

Scrape YouTube videos in Python

Hilman Ramadhan — Fri, 08 Aug 2025 08:10:37 +0000

Scraping YouTube videos enables developers and businesses to extract detailed YouTube video metadata at scale, including titles, descriptions, view counts, thumbnails, channel names, related videos, and comments/replies. It streamlines what would otherwise require complex scraping and anti‑blocking measures.

Get your API Key

First, ensure to register at SerpApi to get your API Key. You can get 250 free searches per month. Use this API Key to access all of our APIs, including the YouTube Video API.

Available parameters

In addition to running the basic search, you can view all YouTube Video API parameters here.

How to scrape YouTube video data with Python

Create a new main.py file
Install requests with:

pip install requests

Here is what the basic setup looks like:

import requests
SERPAPI_API_KEY = "YOUR_REAL_SERPAPI_API_KEY"

params = {
    "api_key": SERPAPI_API_KEY, #replace with the actual API Key
    # soon
}

search = requests.get("https://serpapi.com/search", params=params)
response = search.json()
print(response)

With these few lines of code, we can access all of the search engines available at SerpApi, including the YouTube Video API.

import requests
SERPAPI_API_KEY = "YOUR_SERPAPI_API_KEY"

params = {
    "api_key": SERPAPI_API_KEY, 
    "engine": "youtube_video",
    "v": "j3YXfsMPKjQ" # YouTube video ID
}

search = requests.get("https://serpapi.com/search", params=params)
response = search.json()
print(response)

To make it easier to see the response, let's add indentation.

import json

# ...
# ...
# all previous code

print(json.dumps(response, indent=2))

Here is the result:

Here is how to scrape the comments on a video.

From the request we performed previously, you should be able to see this in the response.

We can scrape the "Top comments" or "Newest first" comments using the token that is available for each. We can put this token in the next_page_token parameter.

Here is an example:

params = {
    "api_key": SERPAPI_API_KEY, 
    "engine": "youtube_video",
    "v": "j3YXfsMPKjQ", # YouTube video ID
    "next_page_token": "Eg0SC2ozWVhmc01QS2pRGAYyOCIRIgtqM1lYZnNNUEtqUTABeAIwAUIhZW5nYWdlbWVudC1wYW5lbC1jb21tZW50cy1zZWN0aW9u"
}

search = requests.get("https://serpapi.com/search", params=params)
response = search.json()
print(json.dumps(response, indent=2))

Here is the result:

If you're interested in scraping the reply on the comment, you can repeat the same action, but this time using the replies_next_page_token.

Bonus

You can also scrape YouTube search results using our YouTube Search API. Here is how to scrape YouTube search results using Python.

Why Use It?

Real‑time data: Fetch up‑to‑date video details.
Rich metadata access: Retrieve comments and nested replies, related videos, and viewer stats in one request.
Automated proxy and captcha handling: SerpApi manages the complexities of scraping so you can focus on analysis.

Key Features

Feature: Comprehensive video metadata
- Benefit: Includes title, description, duration, chapters, views, publishing date, channel, thumbnails
Feature: Comments & replies
- Benefit: Access nested comments for engagement analysis
Feature: Related videos
- Benefit: Discover context and competition around a video

Build a SERP rank tracker app with this API

Hilman Ramadhan — Mon, 23 Sep 2024 07:48:06 +0000

If you're interested in building a SERP ranking tracker app, you'll love our API. SerpApi provides a simple API to access live data from various search engines, including Google, Bing, DuckDuckGo, Yahoo, and others. It enables you to build an app like a SERP ranking tracker.

The idea

To get the ranking position of a website, we need to access the organic results and check where the domain first appears. The organic results data is available through our API. Here are the three APIs we're going to use:

Google Search API: Scrape the results from Google search.
Bing Search API: Scrape the results from the Bing search.
DuckDuckGo Search API: Scrape the results from the DuckDuckGo search.

API Design

We'll create a single endpoint where people can receive the ranking results from the above search engine using this parameter:

Domain name (String): The website domain we want to track.
Keywords (Array): List of keywords that we want to search for.
Engines (Object): List of search engines we want to search on. We can also adjust the parameter details based on the API. Refer to the relevant documentation to check the available parameters for each APIs.

POST: /api/rankings

Data:
 - domain (string)
 - keywords (array[string])
 - engines ((array[name, params]))

Example:
{
  "domain": "archive.org",
  "keywords": ["internet archive"],
  "engines": [
    {
      "name": "google",
      "params": {
        "domain": "google.com",
        "gl": "es"
      }
    }
  ]
}

The source code is available on GitHub; feel free to take a look at the detailed implementation here:
https://github.com/hilmanski/rank-tracker-api/

Let's write the code!

I'll use Nodejs for this API; feel free to use other languages/frameworks.

Install Express

Let's use Express to help us clean up the code structure.

npm i express --save

Export your API Key

You can either export your API key in a terminal like the sample below or save it on an .env file.

export SERPAPI_API_KEY=YOUR_ACTUAL_API_KEY

Basic route

Prepare the POST endpoint with relevant parameters

const express = require('express')
const app = express()
const port = 3000

app.use(express.json());

app.post('/api/rankings', async(req, res) => {
  const { keywords, engines, domain } = req.body;

  // detail implementation later

  res.json({ keywords, engines });
})

app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

Validate the input type

Make sure API users use the correct types.

app.post('/api/rankings', async(req, res) => {
  const { keywords, engines, domain } = req.body;

   // Validate keywords
  if (!Array.isArray(keywords) || !keywords.length) {
    return res.status(400).json({ error: 'Keywords and engines must be arrays.' });
  }

  // Validate engines
  for (const engine of engines) {
    if (typeof engine !== 'object' || !engine.name) {
      return res.status(400).json({ error: 'Each engine must be an object with a "name" property.' });
    }
    if (engine.params && typeof engine.params !== 'object') {
      return res.status(400).json({ error: 'Engine "params" must be an object.' });
    }
  }

  // coming soon

})

Run parallel search

Since we're enabling multiple keywords and multiple search engines, we need to run the function in parallel to save us some time.

// Parallel search
  const results = await Promise.all(engines.map(async engine => {
    const rankings = await Promise.all(keywords.map(async keyword => {
      return await getRanking(keyword, engine, cleanDomain);
    }));

    // map keywords - rankings in one array
    const rankingResults = keywords.map((keyword, index) => {
      return [keyword, rankings[index]];
    });

    console.log(rankingResults);

    return { domain, engine, rankingResults };
  }))

*getRanking method coming soon.

GetRanking method implementation

Here is the function that is responsible for running the search for each search engine.

const suportEngines = ['google', 'bing', 'duckduckgo'];

async function getRanking(keyword, engine, domain) {
  const engineName = engine.name.toLowerCase();

  if(!suportEngines.includes(engineName)) {
      console.error(`Error: Engine ${engineName} is not supported.`);
      return;
  }

  return new Promise(async (resolve, reject) => {
      switch(engineName) {
          case 'google':
            resolve(await searchGoogle(keyword, engine.params, domain))
          break;
          case 'bing':
            resolve(await searchBing(keyword, engine.params, domain))
          break;
          case 'duckduckgo':
            resolve(await searchDuckDuckGo(keyword, engine.params, domain))
          break;
          default:
          break;
      }
  })
}

We'll use the native fetch method in NodeJS to request the actual SerpApi endpoint for each search engine.

API to access ranking position in Google

function searchGoogle(keyword, params, domain) {
  let endpoint = `https://serpapi.com/search?q=${keyword}&engine=google&num=100&api_key=${SERPAPI_API_KEY}`
  if(params) {
      endpoint += `&${new URLSearchParams(params).toString()}`
  }

  return fetch(endpoint)
    .then(response => response.json())
    .then(data => {
      const organic_results = data.organic_results;
      let ranking = organic_results.findIndex(result => result.link.includes(domain))
      return ranking + 1;
    })
    .catch(error => {
      console.error(error);
    });
}

API to access ranking position in Bing

function searchBing(keyword, params, domain) {
  let endpoint = `https://serpapi.com/search?q=${keyword}&engine=bing&count=50&api_key=${SERPAPI_API_KEY}`
  if(params) {
      endpoint += `&${new URLSearchParams(params).toString()}`
  }

  return fetch(endpoint)
    .then(response => response.json())
    .then(data => {
      const organic_results = data.organic_results;
      let ranking = organic_results.findIndex(result => result.link.includes(domain))
      return ranking + 1;
    })
    .catch(error => {
      console.error(error);
    });
}

API to access ranking position in DuckDuckGo

function searchDuckDuckGo(keyword, params, domain) {
  let endpoint = `https://serpapi.com/search?q=${keyword}&engine=duckduckgo&api_key=${SERPAPI_API_KEY}`
  if(params) {
      endpoint += `&${new URLSearchParams(params).toString()}`
  }

  return fetch(endpoint)
    .then(response => response.json())
    .then(data => {
      const organic_results = data.organic_results;
      let ranking = organic_results.findIndex(result => result.link.includes(domain))
      return ranking + 1;
    })
    .catch(error => {
      console.error(error);
    });
}

Final Endpoint

Here's what our endpoint looks like

app.post('/api/rankings', async(req, res) => {
  const { keywords, engines, domain } = req.body;

  // Validate keywords
  if (!Array.isArray(keywords) || !keywords.length) {
    return res.status(400).json({ error: 'Keywords and engines must be arrays.' });
  }

  // Validate engines
  for (const engine of engines) {
    if (typeof engine !== 'object' || !engine.name) {
      return res.status(400).json({ error: 'Each engine must be an object with a "name" property.' });
    }
    if (engine.params && typeof engine.params !== 'object') {
      return res.status(400).json({ error: 'Engine "params" must be an object.' });
    }
  }

  // CLean up domain
  // Since people can include https:// or http:// or a subdomain, strip all of it?
  const cleanDomain = domain.replace(/^https?:\/\//, '').replace(/\/$/, '');

  // Parallel search
  const results = await Promise.all(engines.map(async engine => {
    const rankings = await Promise.all(keywords.map(async keyword => {
      return await getRanking(keyword, engine, cleanDomain);
    }));

    // map keywords - rankings in one array
    const rankingResults = keywords.map((keyword, index) => {
      return [keyword, rankings[index]];
    });

    console.log(rankingResults);

    return { domain, engine, rankingResults }
  }))

  res.json(results);
})

Test the API

Let's try out this API via cURL.

No parameter example

curl -X POST http://localhost:3000/api/rankings \
  -H 'Content-Type: application/json' \
  -d '{
    "keywords": ["internet archive"],
    "domain": "archive.org",
    "engines": [
      {
       "name": "google"
     }
    ]
  }'

Using multiple keywords and search engine parameter sample

curl -X POST http://localhost:3000/api/rankings \
  -H 'Content-Type: application/json' \
-d '{
    "keywords": ["internet archive", "digital library archived internet"],
    "domain": "archive.org",
    "engines": [
      {
        "name": "google",
        "params": {
            "google_domain": "google.co.id",
            "gl": "id"
        }
      }
    ]
  }

Using multiple search engines

curl -X POST http://localhost:3000/api/rankings \
  -H 'Content-Type: application/json' \
  -d '{
    "keywords": ["internet archive", "digital archive", "internet library"],
    "domain": "archive.org",
    "engines": [
    {
      "name": "google",
      "params": {
        "domain": "google.com",
        "gl": "es"
      }
    },
    {
      "name": "Bing",
      "params": {
        "cc": "gb"
      }
    },
{
      "name": "duckduckgo",
      "params": {
        "kl": "uk-en"
      }
    }
  ]
  }'

That's it! I hope you like this post. Please let us know if you have any questions. Feel free to also contribute to this project on GitHub.

Create a super fast AI assistant with Groq (Without a database)

Hilman Ramadhan — Thu, 16 May 2024 02:47:59 +0000

Last week, I tried to build a voice AI assistant using OpenAI AI assistant. It takes a while to generate a response, which is not suitable for a voice assistant. So, I'm looking for an alternative to make my assistant faster. That's how I found out about Groq. This post will cover how I build an AI assistant using Groq.

Pros and Cons summary

Pro:

Easy to implement with only one API (Groq API).

Respond is fast.

Cons:

The longer we chat, the higher the chance that we might lose some context along the way.

What is Groq?

Groq is a service that provides a super fast engine to run AI applications. It's not an AI model! We can run different AI models like Llama, Mixtral, Gemma and more!

Ref: Why Groq?

How I build a fast AI assistant

Many AI models exist, but only OpenAI offers an easy way to implement a chat-like experience using the Assistants API. By default, these models won't know or understand the context of our previous chat. So, we have to re-explain everything if we want the AI to understand the context of each message.

There are some alternatives out there, such as using LangChain chat history. But I prefer to find a simple way (*with the caveat, of course). Luckily, I found some ideas on the internet (Thank you, Internet!).

The idea below can be implemented for any AI model/engine, not just Groq. You can try this with OpenAI itself, Mixtral, Claude, and so on.

Here is the flow:

The user sends the initial message
The AI responds to the message
We ask AI to summarize the conversation
We send the response and summary back to the user
The user will send the summary back later alongside the new message
AI now will reply based on the fresh message and with help of the conversation summary to provide some context.

The caveat of this method

By summarizing a conversation, we may lose some information along the way. That's why it's a good idea in certain cases to store the message history on a database (Vector database).

One way I can reduce this shortage is by attaching the recent reply from AI. I've also read an article that suggests keeping the latest 2-3 conversations and providing them as additional context later.

Code implementation

I'll use NodeJs for this tutorial. Feel free to use any language you want. The final code is available at GitHub:

GitHub -assistants-api-with-groq-ai

Install dependencies

npm i express groq-sdk dotenv --save

Express for creating a route for the endpoint
Groq-sdk is the official package for using Groq in Javascript
dotenv to store our API key safely.

Add API Key

Create a new .env file. Add your Groq API key in this file like this:

GROQ_API_KEY=YOUR_GROQ_API_KEY

Make sure to sign up to Groq and get your API key here.

Basic Setup

Let's create a new index.jsfile, and we'll write everything in this file. We prepare one endpoint called chat where we'll send these parameters:

- message: user's message

- latestReply: The latest reply from AI

- messageSummary: The conversation summary so far

In this endpoint, we'll do two things:

- Respond to new user message (with latestReply and messageSummary as context)

- Create a new conversation summary by providing the fresh reply from AI.

const express = require('express');

// Express Setup
const app = express();
app.use(express.json());
const port = 3000

require("dotenv").config();
const { GROQ_API_KEY } = process.env;

// GROQ Setup
const Groq = require("groq-sdk");
const groq = new Groq({
    apiKey: GROQ_API_KEY
});

async function chatWithGroq() { } // soon
async function summarizeConversation() { } // soon

app.post('/chat', async (req, res) => {
    const { message, latestReply, messageSummary } = req.body;

    // request chat completion
    const reply = await chatWithGroq(message, latestReply, messageSummary)

    // request chat summary
    const summary = await summarizeConversation(message, reply, messageSummary)

    // Always return chat history/summary
    res.send({
        reply,
        summary
    })
})

app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

Chat with Groq method

Here is the chatWithGroq method implementation:

async function chatWithGroq(userMessage, latestReply, messageHistory) {
    let messages = [{
        role: "user",
        content: userMessage
    }]

    if(messageHistory != '') {
        messages.unshift({
            role: "system",
            content: `Our conversation's summary so far: """${messageHistory}""". 
                     And this is the latest reply from you """${latestReply}"""`
        })
    }

    console.log('original message', messages)

    const chatCompletion = await groq.chat.completions.create({
        messages,
        model: "llama3-8b-8192"
    });

    const respond = chatCompletion.choices[0]?.message?.content || ""
    return respond
}

We only provide a conversation summary when we have one (look at the if statement). So, it won't be included in our first message.

Conversation summary method

Here is the summarizeConversation method implementation:

async function summarizeConversation(message, reply, messageSummary) {
    let content = `Summarize this conversation 
                    user: """${message}""",
                    you(AI): """${reply}"""
                  `

    // For N+1 message
    if(messageSummary != '') {
        content = `Summarize this conversation: """${messageSummary}"""
                    and last conversation: 
                    user: """${message}""",
                    you(AI): """${reply}"""
                `
    }

    const chatCompletion = await groq.chat.completions.create({
        messages: [
            {
                role: "user",
                content: content
            }
        ],
        model: "llama3-8b-8192"
    });

    const summary = chatCompletion.choices[0]?.message?.content || ""
    console.log('summary: ', summary)
    return summary
}

In this method, we ask the AI to create a summary based on the latest summary and recent reply.

Demo Time!

You can use any API client, like Postman, Thunder (VS Code), etc.

Don't forget to run your program with node index.js

Create a POST request for the /chat endpoint and provide message endpoint and provide the first message parameter.

We can display the reply from the response on our user interface. This is the actual reply to our message.

We'll save the summary for the next request.

Now, this is how the JSON looks like for the N+1 message:

The next messages should include the latestReply and messageSummary as parameters.

message: *Don't forget to add a new message. This is you talking to the AI. Notice that I use here on my question, to validate that the AI knows what's the previous context here.
latestReply: Send the latest reply from AI (from previous response)
messageSummary: Send the conversation summary so far (from previous response)

Here is the result to this request:

As you can see, the AI knows that when I said here I was talking about Indonesia. You can try to send a follow-up message (create a new request) by asking something like "Can you tell me more about number 4?" as an example. But don't forget that we always need to update the latestReply and summaryConversation on each request.

To return the response and summarize the conversation, I only need to wait around 2s. This is much faster than using OpenAI AI assistants.

Reference:

- Build a smart AI voice assistant

- Basic tutorial: Assistants API by OpenAI

Build an AI Voice assistant like Siri (use OpenAI AI Assistant)

Hilman Ramadhan — Tue, 30 Apr 2024 00:23:48 +0000

Hi! Today, we'll learn how to build an AI Voice assistant like Siri that can understand what we say and speak back to us.

Demo AI Voice assistant

Here's what we're going to build:

This post will focus on implementing it as a web application. Therefore, we will use HTML for the interface and Javascript for the voice features. You might want to adjust this if you build for another platform (mobile, desktop, etc.).

Regardless of the platform, knowing the components of how to build this will help us along the way.

Basic Voice AI assistant Structure

Just like other applications, we will have this basic structure:

Here is what it looks like on our app:

You can replace each of these elements with the more advanced option. I'm trying to stick with what we already have in the browser. These are the alternatives:

Voice input: AssemblyAI or OpenAI Whisper
Voice output: Elevenlabs or OpenAI Whisper

Code Tutorial on how to build a Voice AI assistant

We'll split our codebase into two parts, one for the front end and one for the back end.

First, we just want to make sure that we can get a text from the user's voice and read a text out loud; there is no AI or conversation involved yet.

Here is the final code result of this post:
GitHub - hilmanski/simple-ai-voice-assistant-openai-demo

Step 1: Basic frontend Views

Since we're building a web application, we'll use HTML.

- We need two buttons to start and stop the recording

- A div to display the text

<button id="record">Record</button>
<button id="stop">Stop</button>

<div id="output">Output</div>

(Optional) If you want to copy the style I implemented:

<style>
body {
    margin: 50px auto;
    width: 500px;
}

#output {
    margin-top: 20px;
    border: 1px solid #000;
    padding: 10px;
    height: 200px;
    overflow-y: scroll;
}

#output p:nth-child(even) {
    background-color: #f8f6b1;
}
</style>

Step 2: Listen to speech

Let's trigger the actions from Javascript:

<script>
    // Set up
    const SpeechRecognition = window.SpeechRecognition || window.webkitSpeechRecognition;
    const SpeechGrammarList = window.SpeechGrammarList || window.webkitSpeechGrammarList;
    const SpeechRecognitionEvent = window.SpeechRecognitionEvent || window.webkitSpeechRecognitionEvent;

    const recognition = new SpeechRecognition();
    const speechRecognitionList = new SpeechGrammarList();

    recognition.grammars = speechRecognitionList;
    recognition.continuous = true;
    recognition.lang = 'en-US';
    recognition.interimResults = false;
    recognition.maxAlternatives = 1;


    // Start recording
    document.getElementById('record').onclick = function() {
        recognition.start();
    }

    // Stop recording
    document.getElementById('stop').onclick = function() {
        recognition.stop();
        console.log('Stopped recording.');
    }

    // Output
    recognition.onresult = async function(event) {
        // Get the latest transcript 
        const lastItem = event.results[event.results.length - 1]
        const transcript = lastItem[0].transcript;
        document.getElementById('output').textContent = transcript;
        recognition.stop(); 
        // await sendMessage(transcript); // we'll implement this later
    }

    recognition.onspeechend = function() {
        recognition.stop();
    }
</script>

Step 3: Backend structure

We're using NodeJS/Express for the backend. Make sure to install the necessary packages in your new directory (separate from the frontend code):

npm init -y #initialize NPM package
npm init express cors dotenv openai --save
touch index.js #create a new empty file

const express = require('express');
const cors = require('cors')

// Setup Express and allow CORS
const app = express();
app.use(express.json());
app.use(cors()) // allow CORS for all origins

// Main route
app.post('/message', async (req, res) => {
    const { message } = req.body;
    res.json({ message: 'Received: ' + message });
});

// Start the server
const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
  console.log(`Server is running on port ${PORT}`);
});

Run node js in your terminal to run the server. Your application is running on localhost:3000.

We only have one route, which is /message , which receives a message from the client and echoes it back. This is to ensure that the input and output parts are running smoothly.

Step 4: Send a message from frontend

Let's add a fetch method to send the message to that endpoint.

Update your recognition.onresult event to call the sendMessage function like this:

recognition.onresult = async function(event) {
    // Get the latest transcript 
    const lastItem = event.results[event.results.length - 1]
    const transcript = lastItem[0].transcript;
    document.getElementById('output').textContent = transcript;

    await sendMessage(transcript); // New addition
}

Now, let's declare the sendMessage method:

async function sendMessage(message) {
    const response = await fetch('http://localhost:3000/message', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json'
        },
        body: JSON.stringify({ message })
    });

    const data = await response.json();
    console.log(data);

    speak(data.message);
}

function speak(message) {
    if (synthesis) {
        const utterance = new SpeechSynthesisUtterance(message);
        synthesis.speak(utterance);
    }
}

I also added a new method called speak to use the Web Speech API, specifically SpeechSynthesis method to speak. The SendMessage will hit the endpoint we prepared previously.

Step 5: Try the App

Now, run your HTML file, you can use something like VS Code Live server.

press record
say anything

AI Voice assistant web application screenshot

Now, you should see your message echo back to you (it's coming from the server). Press the record button again to send a different message.

Now, we have an app that can listen and speak to us. Let's dive into the AI part!

Smart AI assistant

We'll use the OpenAI Assistants API as the brain.

Step 1: Install the OpenAI package

We've previously installed the OpenAI package for NodeJS. Make sure you've installed it as well.

Step 2: Grab your API Key

Get your API Key from the OpenAI dashboard. Create a new .env file and paste it there.

OPENAI_API_KEY=YOUR_API_KEY_FROM_OPENAI

Step 3: Import OpenAI and dotenv

require("dotenv").config();
const OpenAI = require('openai');
const { OPENAI_API_KEY } = process.env;

// Set up OpenAI Client
const openai = new OpenAI({
    apiKey: OPENAI_API_KEY,
});

Step 4: Implement assistants API

If you're not familiar with the assistants API, I suggest to read this introduction blog post first:

[

Assistant API by OpenAI (Basic Tutorial)

Learn the basics of Assistant API by OpenAI. We’ll create a super simple coding example to understand the barebone of Assistant API: https://serpapi.com/blog/assistant-api-openai-beginner-tutorial/

We'll use the same logic and code for this tutorial (with some updates soon). You can also get the code sample for the API assistants here:

GitHub - hilmanski/assistant-API-openai-nodejs-sample: A simple example for Assistant API by OpenAI using NodeJS

Note: I won't explain and show the whole code here.

Step 5: Add the assistants API

We need to tell the AI assistants that they will act as general helpers who can help us with anything.

*Make sure to update the assistant_id key on your code.

Step 6: Assign a thread ID

Since the API needs a thread unique ID for each conversation, I'll add a new fetch request on our frontend to automatically ask for a thread id on the first visit. The ID will also updated on browser refresh.

Reminder: we have this route to create a thread ID from the OpenAI assistant tutorial

// Open a new thread
app.get('/thread', (req, res) => {
    createThread().then(thread => {
        res.json({ threadId: thread.id });
    });
})

Let's add this block to our HTML file (frontend part)

let threadId = null;

// onload
window.onload = function() {
    fetch('http://localhost:3000/thread')
        .then(response => response.json())
        .then(data => {
            console.log(data);
            assistant_id = data.threadId;
        });
}

We'll adjust our sendMessage method to attach the threadId when sending a message.

 async function sendMessage(message) {
    const response = await fetch('http://localhost:3000/message', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json'
        },
        body: JSON.stringify({ message, threadId }) // <- update here
    });

    // continue

Step 7: Return the last message only

Let's update our checkingStatus method to only return the latest message from the AI

async function checkingStatus(res, threadId, runId) {
    const runObject = await openai.beta.threads.runs.retrieve(
        threadId,
        runId
    );

    const status = runObject.status;
    console.log(runObject)
    console.log('Current status: ' + status);

    if(status == 'completed') {
        clearInterval(pollingInterval);

        const messagesList = await openai.beta.threads.messages.list(threadId);
        const lastMessage = messagesList.body.data[0].content[0].text.value

        res.json({ message: lastMessage });
    }
}

Our voice assistant is ready!

Step 8: Show all messages

If you want to add all previous transcriptions to the user interface, here is the code to collect all the previous messages in the div.

First, every time we speak:

recognition.onresult = async function(event) {
    // Get the latest transcript 
    const lastItem = event.results[event.results.length - 1]
    const transcript = lastItem[0].transcript;

    // Update: Append new text to div
    const newText = "<p>" + transcript + "</p>";
    document.getElementById('output').insertAdjacentHTML("afterbegin", newText);

    recognition.stop();
    await sendMessage(transcript);
}

Second, every time we got a response from the AI:

async function sendMessage(message) {
    console.log('Sending message: ', threadId);
    const response = await fetch('http://localhost:3000/message', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json'
        },
        body: JSON.stringify({ message, threadId })
    });

    const data = await response.json();

    // update: add new text here
    const newText = "<p>" + data.message + "</p>";
    document.getElementById('output').insertAdjacentHTML("afterbegin", newText);

    speak(data.message);
}

Make sure to re-run your NodeJS app. Now try to have a chat with your AI!

Things we can improve

There are several things we can improve for this voice assistant

AI Instruction

Since we're building a voice assistant, it shouldn't explain things too long for a simple question. We can adjust the instructions like this:

You're a general AI assistant that can help with anything. You're a voice assistant, so don't speak too much, make it clear and concise.

Voice and listening

We use native browser API to listen and speak. Better alternatives exist, such as Elevenlabs, AssemblyAI, and more.

Knowledge Limitation

We're using one of the OpenAI models, where the knowledge is cut off at a particular year. We can expand its knowledge by providing PDF files or connecting the assistant API to the internet.

Beautiful Soup: Web Scraping with Python (Beginner friendly)

Hilman Ramadhan — Tue, 05 Mar 2024 02:22:05 +0000

Imagine you're looking for a shiny new monitor to perfect your setup. You've found the perfect model on a popular shopping site, but there's a catch – it's just a bit outside your budget. You know prices fluctuate, often dropping during sales or special promotions, but checking the website daily is a tedious task that wastes your valuable time.

Instead of manually monitoring the site, we can build a simple yet effective program that checks the price for you. This is where web scraping comes into play, and Beautiful Soup is your ally!

What is beautiful soup?

Beautiful Soup is a Python library designed to help you easily extract information from web pages by parsing HTML and XML documents.

Link: Beautiful soup

Beautiful Soup is a versatile tool that can be used to extract all kinds of data from web pages, not just price information. Whether you're interested in headlines from a news website, comments from a forum, product details from an e-commerce site, or any other information, Beautiful Soup can help you automate the extraction process efficiently.

You might also interested in reading this post
Learn web scraping in Python for beginner.

Prerequisites:

Basic understanding of Python.
Python is installed on your machine.
PIP for installing Python packages.

Here's a basic tutorial on web scraping with Python. We will use two popular libraries: requests for making HTTP requests and Beautiful Soup for parsing HTML.

Step 1: Install Necessary Libraries

First, you need to install the requests and BeautifulSoup libraries. You can do this using pip:

pip install requests beautifulsoup4

Step 2: Import Libraries

In your Python script or Jupyter Notebook, import the necessary modules:

import requests
from bs4 import BeautifulSoup

Step 3: Make an HTTP Request

Choose a website you want to scrape and send a GET request to it. For this example, let's scrape Google's homepage.

url = 'https://google.com'
response = requests.get(url)

Step 4: Parse the HTML Content

Once you have the HTML content, you can use Beautiful Soup to parse it:

soup = BeautifulSoup(response.text, 'html.parser')

Step 5: Extract Data

Now, you can extract data from the HTML. Let's say you want to extract all the headings:

headings = soup.find_all('div')
for heading in headings:
    print(heading.text.strip())

Step 6: Handle Errors

Always make sure to handle errors like bad requests or connection problems:

if response.status_code == 200:
    # Proceed with scraping
    # ...
else:
    print("Failed to retrieve the web page")

Notes

We need two primary tools to perform web scraping in Python: HTTP Client and HTML Parser.

An HTTP API Client to fetch web pages. e.g. requests, urllib, pycurl or httpx
An HTML parser to extract data from the fetched pages. e.g. Beautiful Soup, lxml, or pyquery

Inspect the page first!

If we want to scrape specific data from a website, we need to know where this data is located.

Websites are built using HTML, which organizes content in a nested structure of elements like headings, paragraphs, and divs. Each element can have attributes like class and id, which are useful for identifying specific page parts.

To effectively use Beautiful Soup to extract data, you need to inspect the website and identify the elements that contain the data you're interested in. This process involves using a web browser's developer tools to look at the HTML structure of the page.

By understanding how the content is organized, you can write more precise and efficient Beautiful Soup code to target exactly what you need, avoiding unnecessary processing and ensuring you get accurate data.

Real-world example using Beautiful Soup

Let's practice! For this example, we're going to scrape the price of the item in this website: https://webscraper.io/test-sites/e-commerce/allinone/computers

Step 1: Inspect the page

Remember always to inspect the page first. We can do this by right-clicking on the page and clicking inspect. You can see the HTML structure in the elements tab.

I can see the price is wrapped in the card element, and specifically on the price CSS class.

Step 2: Make an HTTP request

In your Python script, import the necessary modules (Make sure to install BeautifSoup first!):

import requests
from bs4 import BeautifulSoup

Choose a website you want to scrape and send a GET request.

url = 'https://webscraper.io/test-sites/e-commerce/allinone/computers'
response = requests.get(url)

Step 3: Parse the HTML Content

Once you have the HTML content, you can use Beautiful Soup to parse the price information:

soup = BeautifulSoup(response.text, 'html.parser')

Now we're going to scrape all data from the items, not just price.

# Empty list to hold the data
products = []

# Find all product wrapper divs
for card in soup.find_all('div', class_='product-wrapper'):
    # Extract title, description, price, rating, and number of reviews
    title = card.find('a', class_='title').text.strip()
    description = card.find('p', class_='description').text.strip()
    price = card.find('h4', class_='price').text.strip()
    rating = len(card.find('div', class_='ratings').find_all('span', class_='ws-icon-star'))
    reviews = card.find('p', class_='review-count').text.strip().split(' ')[0]  # Assuming "X reviews"

    # Append the data to the products list
    products.append({
        'title': title,
        'description': description,
        'price': price,
        'rating': rating,
        'reviews': reviews
    })

# Display the extracted data
for product in products:
    print(product)

You should be able to see this data:

Now, you can expand your code to save this data on your database or CSV files and compare it later after running the script every day (Pro tip: You can use CRON to automate this process).

Parsing data when there is no ID or class provided

If the HTML structure you're working with lacks specific class or id attributes to easily identify elements, you can still use Beautiful Soup to navigate and extract data based on the hierarchical structure of the HTML.

For this example, I'll simplify the previous HTML structure, removing specific class names and id attributes and then demonstrate how to extract the same information.

Here's a simplified version of the HTML structure:

<div>
    <div>
        <div>
            <img src="/path/to/image1.png">
            <div>
                <h4>$107.99</h4>
                <h4>Galaxy Tab 3</h4>
                <p>7", 8GB, Wi-Fi, Android 4.2, Yellow</p>
            </div>
            <div>
                <p>14 reviews</p>
                <p>Rating: ★★★☆☆</p>
            </div>
        </div>
    </div>
    <!-- Repeat for other products -->
</div>

In this scenario, you can still extract data by carefully navigating the structure. Let's write Python code using Beautiful Soup to do this:

from bs4 import BeautifulSoup

html_content = """
<div>
    <div>
        <div>
            <img src="/path/to/image1.png">
            <div>
                <h4>$107.99</h4>
                <h4>Galaxy Tab 3</h4>
                <p>7", 8GB, Wi-Fi, Android 4.2, Yellow</p>
            </div>
            <div>
                <p>14 reviews</p>
                <p>Rating: ★★★☆☆</p>
            </div>
        </div>
    </div>
    <div>
        <div>
            <img src="/path/to/image2.png">
            <div>
                <h4>$50.01</h4>
                <h4>Second I phone</h4>
                <p>Blue night</p>
            </div>
            <div>
                <p>10 reviews</p>
                <p>Rating: ★★☆☆☆</p>
            </div>
        </div>
    </div>
</div>
"""

# Parse the HTML content
soup = BeautifulSoup(html_content, 'html.parser')

# Empty list to hold the data
products = []

# Assuming each product is within the third-level div
for div in soup.div.find_all('div', recursive=False):
     # Extract the image, price, title, description, reviews, and rating
    img_src = div.div.img['src']
    price = div.div.find_next('h4').text
    title = div.div.find_next('h4').find_next_sibling('h4').text
    description = div.div.find('p').text
    reviews = div.div.find_all('p')[1].text.split(' ')[0]
    rating = div.div.find_all('p')[2].text.count('★')

    # Append the product data to the products list
    products.append({
        'img_src': img_src,
        'price': price,
        'title': title,
        'description': description,
        'reviews': reviews,
        'rating': rating
    })

# Display the extracted data
for product in products:
    print(product)

It demonstrates how you can still extract data without specific class names or ids, by carefully navigating the HTML tree and using the relationships between elements.

How to scrape a website with Python (Beginner tutorial)

Hilman Ramadhan — Fri, 23 Feb 2024 00:30:31 +0000

If you've ever been curious about how to extract valuable data from websites, you're in the right place. Web scraping is a powerful tool for gathering information from the internet, and Python, with its rich ecosystem of libraries, makes this task easy for us.

In this blog post, we'll cover:

List of tools we can use for web scraping with Python.
Simple web scraping for static websites.
Using Selenium for dynamic content or Javascript-heavy site/
MechanicalSoup to automate some task in browser.

We have a lot of libraries in Python that we can use for scraping data from a website. Here is some of it:

Category: HTTP Libraries
- Tool/Library: Requests
- Description: Simple HTTP library for Python, built for human beings.
Category:
- Tool/Library: urllib
- Description: A module for fetching URLs included with Python.
Category:
- Tool/Library: urllib3
- Description: A powerful, user-friendly HTTP client for Python.
Category:
- Tool/Library: httpx
- Description: A fully featured HTTP client for Python 3, which provides sync and async APIs, and support for both HTTP/1.1 and HTTP/2.
Category: Parsing Libraries
- Tool/Library: Beautiful Soup
- Description: A library for pulling data out of HTML and XML files.
Category:
- Tool/Library: lxml
- Description: Processes XML and HTML in Python, supporting XPath and XSLT.
Category:
- Tool/Library: pyquery
- Description: A jQuery-like library for parsing HTML.
Category: Web Drivers
- Tool/Library: Selenium
- Description: An automated web browser, useful for complex scraping tasks.
Category:
- Tool/Library: Splinter
- Description: Open-source tool for testing web applications.
Category: Automation Tools
- Tool/Library: Scrapy
- Description: An open-source web crawling and scraping framework.
Category:
- Tool/Library: MechanicalSoup
- Description: A Python library for automating interaction with websites.
Category: Data Processing
- Tool/Library: pandas
- Description: A fast, powerful, flexible and easy-to-use data analysis tool.
Category: JavaScript Support
- Tool/Library: Pyppeteer (Python port of Puppeteer)
- Description: A tool for browser automation and web scraping.

Feel free to suggest if you know any other tools out there!

Step by Step basic web scraping tutorial in Python

Here's a basic tutorial on web scraping in Python. For this example, we will use two popular libraries: requests for making HTTP requests and Beautiful Soup for parsing HTML.

Prerequisites:

Basic understanding of Python.
Python is installed on your machine.
PIP for installing Python packages.

Step 1: Install Necessary Libraries

First, you need to install the requests and BeautifulSoup libraries. You can do this using pip:

pip install requests beautifulsoup4

Step 2: Import Libraries

In your Python script or Jupyter Notebook, import the necessary modules:

import requests
from bs4 import BeautifulSoup

Step 3: Make an HTTP Request

Choose a website you want to scrape and send a GET request to it. For this example, let's scrape Google's homepage.

url = 'https://google.com'
response = requests.get(url)

Step 4: Parse the HTML Content

Once you have the HTML content, you can use Beautiful Soup to parse it:

soup = BeautifulSoup(response.text, 'html.parser')

Step 5: Extract Data

Now, you can extract data from the HTML. Let's say you want to extract all the headings:

headings = soup.find_all('div')
for heading in headings:
    print(heading.text.strip())

Step 6: Handle Errors

Always make sure to handle errors like bad requests or connection problems:

if response.status_code == 200:
    # Proceed with scraping
    # ...
else:
    print("Failed to retrieve the web page")

Notes

We need two primary tools to perform web scraping in Python: HTTP Client and HTML Parser.

An HTTP API Client to fetch web pages. e.g. requests, urllib, pycurl or httpx
An HTML parser to extract data from the fetched pages. e.g. Beautiful Soup, lxml, or pyquery

Here is a concrete example on how to use these tools on a real world use case: How to scrape Google search results with Python

Step by Step scraping dynamic content in Python

What if the content you want to scrape is not loaded initially? Sometimes, the data hides behind a user interaction. To scrape dynamic content in Python, which often involves interacting with JavaScript, you'll typically use Selenium.

Unlike the requests and BeautifulSoup combination, which works well for static content, Selenium can handle dynamic websites by automating a web browser.

Prerequisites:

Basic knowledge of Python and web scraping (as covered in the previous lesson).
Python is installed on your machine.
Selenium package and a WebDriver installed.

Step 1: Install Selenium

First, install Selenium using pip:

pip install selenium

Step 2: Download WebDriver

You'll need a WebDriver for the browser you want to automate (e.g., Chrome, Firefox). For Chrome, download ChromeDriver. Make sure the WebDriver version matches your browser version. Place the WebDriver in a known directory or update the system path.

Step 3: Import Selenium and Initialize WebDriver

Import Selenium and initialize the WebDriver in your script.

from selenium import webdriver

driver = webdriver.Chrome()

Step 4: Fetch Dynamic Content

Open a website and fetch its dynamic content. Let's use http://example.com as an example.

url = 'https://google.com'
driver.get(url)

Step 5: Print title

Here is an example of how to get a certain element on the page.

print(driver.title)

Try to run this script. You'll see a new browser pop up and open the page.

Step 6: Interact with the Page (if necessary)

If you need to interact with the page (like clicking buttons or filling forms), you can do so:

text_box = driver.find_element(by=By.NAME, value="my-text")
submit_button = driver.find_element(by=By.CSS_SELECTOR, value="button")
submit_button.click()

Step 7: Scrape Content

Now, you can scrape the content. For example, to get all paragraphs:

paragraphs = driver.find_elements_by_tag_name('p')
for paragraph in paragraphs:
    print(paragraph.text)

Step 8: Close the Browser

Once done, don't forget to close the browser:

driver.quit()

Additional Tips:

Selenium can perform almost all actions that you can do manually in a browser.
For complex web pages, consider using explicit waits to wait for elements to load.
Remember to handle exceptions and errors.

Here is a video tutorial on using Selenium for automation in Python by NeuralNine on Youtube.

A basic example of web scraping using MechanicalSoup

MechanicalSoup is a Python library for web scraping that combines the simplicity of Requests with the convenience of BeautifulSoup. It's particularly useful for interacting with web forms, like login pages. Here's a basic example to illustrate how you can use MechanicalSoup for web scraping:

Please note that MechanicalSoup doesn't handle javascript loaded content. That's a task for Selenium 😉

Prerequisites:

Python is installed on your machine.
Basic understanding of Python and HTML.

Step 1: Install MechanicalSoup

You can install MechanicalSoup via pip:

pip install mechanicalsoup

Step 2: Import MechanicalSoup

In your Python script, import MechanicalSoup:

import mechanicalsoup

Step 3: Create a Browser Object

MechanicalSoup provides a Browser class, which you'll use to interact with web pages:

browser = mechanicalsoup.StatefulBrowser()

Step 4: Make a Request

Let's say you want to scrape data from a simple example page. You can use the Browser object to open the URL:

url = 'https://google.com'
print(browser.get(url))

Step 5: Parse the HTML Content

The page variable now contains the response from the website. You can access the BeautifulSoup object via browser.page:

page = browser.page
print(page)

Step 6: Extract Data

Now, you can extract data using BeautifulSoup methods. For example, to get all paragraphs:

page = browser.page
pTags = page.find_all('p')
print(pTags)

Step 7: Handling Forms (Optional)

If you need to interact with forms, you can do so easily.

Given this HTML content on a page:

<form action="/pages/forms/" class="form form-inline" method="GET">
<label for="q">Search for Teams:  </label>
<input class="form-control" id="q" name="q" placeholder="Search for Teams" type="text"/>
<input class="btn btn-primary" type="submit" value="Search"/>
</form>

To submit a search query on a form:

# Select the form
browser.select_form('form')

# Fill the form with your query
browser['q'] = 'red'

# Submit the form
response = browser.submit_selected()

# Print the URL (assuming the form is correctly submitted and a new page is loaded)
print("Form submitted to:", response.url)

What if you have multiple forms on the page?

select_form and another method in MechanicalSoup usually accept a CSS selector parameter. So, whether it's id or class you can always name it specifically there.

When to use MechanicalSoup (From their documentation)

MechanicalSoup is designed to simulate the behavior of a human using a web browser. Possible use-case include:

- Interacting with a website that doesn’t provide a webservice API, out of a browser.

- Testing a website you’re developing

Why use Python for web scraping?

Python is a popular choice for web scraping for several reasons. Here are the top three:

Seamless Integration with Data Science Tools: After scraping data from the web, you often need to clean, analyze, and visualize this data, which is where Python's data science capabilities come in handy. Tools like Pandas, NumPy, and Matplotlib integrate seamlessly with web scraping libraries, allowing for an efficient end-to-end process. Here's a bit more detail on each:
Rich Ecosystem of Libraries: Python has a vast selection of libraries specifically designed for web scraping, such as Beautiful Soup, Scrapy, Selenium, Requests, and MechanicalSoup. These libraries simplify the process of extracting data from websites, parsing HTML and XML, handling HTTP requests, and even interacting with JavaScript-heavy sites. This rich ecosystem means that Python offers a tool for almost every web scraping need, from simple static pages to complex, dynamic web applications.
Ease of Learning and Use: Python is known for its simplicity and readability, making it an excellent choice for beginners and experienced programmers alike. Its straightforward syntax allows developers to write less code compared to many other programming languages, making the process of writing and understanding web scraping scripts easier and faster. This ease of use is particularly beneficial in web scraping, where scripts can often become complex and difficult to manage.

That's it! I hope you enjoy this tutorial!

Scrape Google Trends with Python (simple API, pytrends alternative)

Hilman Ramadhan — Wed, 31 Jan 2024 03:29:45 +0000

We are surrounded by tons of data. But what's the point if we can't understand it all? That's where Google Trends comes in. Google Trends offers valuable insights into what people search for on the internet. Let's dive in and see how we can programmatically scrape Google Trends data using Python (Psst.. no PyTrends needed!).

Step-by-step scraping Google Trends data with Python

Without further ado, let's start and collect data from Google Trends.

Step 1: Tools

We'll use the new official Python library by SerpApi: serpapi-python .

That's the only tool that we need!

As a side note: You can use this library to scrape search results from other search engines, not just Google.

Usually, you'll write your DIY solution using PyTrends, BeautifulSoup, Selenium, Scrapy, Requests, etc., to scrape Google Trends results. You can relax now since we perform all these heavy tasks for you. So, you don't need to worry about all the problems you might've encountered while implementing your web scraping solution.

Step 2: Setup and preparation

Sign up for free at SerpApi. You can get 100 free searches per month.
Get your SerpApi Api Key from this page.
Create a new .env file, and assign a new env variable with value from API_KEY above. SERPAPI_KEY=$YOUR_SERPAPI_KEY_HERE
Install python-dotenv to read the .env file with pip install python-dotenv
Install SerpApi's Python library with pip install serpapi
Create new main.py file for the main program.

Your folder structure will look like this:

|_ .env
|_ main.py

Step 3: Write the code for scraping Google Trends

Let's say I want to get this result:

- Keyword: 'standing desk'

- Region: 'worldwide'

- Time: 'Past 12 months'

Here is the Python code

import os
import serpapi
from dotenv import load_dotenv

load_dotenv()
api_key = os.getenv('SERPAPI_KEY')

client = serpapi.Client(api_key=api_key)
search =  client.search(
    engine="google_trends",
    q="standing desk",
    api_key=api_key,
  )

print(search)

By default, the Google Trends API will set the location for worldwide and duration of the past 12 months . We'll take a look at how to change these values later. For now, take a look at our perfect result:

Please note, that we only return the response in JSON format. You'll need to build the graph yourself if you need one.

Step 4: Adjust region and timeline

Let's change the settings. I set the region to United States and timeline to Past 30 days .

Let's see how it is represented in our code.

search =  client.search(
    engine="google_trends",
    q="standing desk",
    api_key=api_key,
    geo="US", // Two-code-letter country code
    date="today 1-m" // Timeline for last 1 month
)

The result is perfect as well:

More information about available parameters: https://serpapi.com/google-trends-api

SerpApi Google Trends API features

Using SerpApi, it's possible to grab other parts from Google Trends, like the breakdown by region result, interest by region, related queries, related topics, and what's trending now.

Searching for multiple terms

It's possible to search for multiple terms. You can separate the terms by a comma. Here are the limitations:

- Max 5 terms per search

- Max 100 characters per search

search =  client.search(
    engine="google_trends",
    q="standing desk, ergonomic chair",
    api_key=api_key,
)

Interest by Region

We can grab the data for interest by region. We need to change the data_type to GEO_MAP_0

search = client.search(
    engine="google_trends",
    q="coffee latte",
    date="today 12-m",
    tz="420",
    data_type="GEO_MAP_0", // Important part
    api_key=api_key,
)

Here is the list of possible values for data_type

For example, if you're interested in the related topics, you need to adjust your code to:

search = client.search(
    engine="google_trends",
    q="coffee latte",
    date="today 12-m",
    tz="420",
    data_type="RELATED_TOPICS", // Adjust this accordingly
    api_key=api_key,
)

print(search)

References to our other related API:

Pytrends Tutorial to scrape Google Trends data

There is also another unofficial API for Google Trends for Python. It's called Pytrends. We'll see a basic tutorial on how to implement this.

Please note that the repository has not been very active these last few months. There are also some unadressed issues. Therefore, we recommend using Google Trends API instead.

Install Pytrends

pip install pytrends

Initialize the object. We also initialize pandas so it's easier to read the data.

from pytrends.request import TrendReq
import pandas as pd

pytrend = TrendReq()
pytrend.build_payload(kw_list=['Standing Desk'])

Inside the build_payload method, we can put the keywords list in an array.

Interest Over Time

# Interest over time
pytrend = TrendReq()
pytrend.build_payload(kw_list=['Standing Desk'])
df = pytrend.interest_over_time()
print(df)

Here is the result

There are some other parameters for build_payload method. Unfortunately, I sometimes get error results. Here are the parameters you can play with: year_start, month_start, day_start, hour_start, year_end, month_end, day_end, hour_end.

Interest by Region

Here is how to split the trends by country and sort the top 10 results.

pytrend = TrendReq()
pytrend.build_payload(kw_list=['Standing Desk'])
df = pytrend.interest_by_region(resolution='COUNTRY')
print(df.sort_values(by='Standing Desk', ascending=False).head(10))

Related topics

Here is how to show related topics using Pytrends:

df = pytrend.related_topics()
print(df)

Related queries

Here is how to show related topics using Pytrends:

df = pytrend.related_queries()
print(df)

Trending searches

Here is how to display trending searches by Region from Google Trends

#Trending Searches
df = pytrend.trending_searches(pn='united_states') # trending searches in real time for United States
print(df.head)

The following API methods are available for PyTrends:

Interest Over Time: returns historical, indexed data for when the keyword was searched most as shown on Google Trends' Interest Over Time section.
Multirange Interest Over Time: returns historical, indexed data similar to interest over time but across multiple time date ranges.
Historical Hourly Interest: returns historical, indexed, hourly data for when the keyword was searched most, as shown on Google Trends' Interest Over Time section. It sends multiple requests to Google, each retrieving one week of hourly data. It seems like this would be the only way to get historical, hourly data.
Interest by Region: This returns data for where the keyword is most searched, as shown in Google Trends' Interest by Region section.
Related Topics: returns data for the related keywords to a provided keyword shown on Google Trends' Related Topics section.
Related Queries: returns data for the related keywords to a provided keyword shown on Google Trends' Related Queries section.
Trending Searches: returns data for the latest trending searches shown on Google Trends' Trending Searches section.
Top Charts: returns the data for a given topic shown in Google Trends' Top Charts section.
Suggestions: Return a list of additional suggested keywords that can be used to refine a trend search.

Thank you for reading this blog post! I hope it can help you to gather data from the Google Trends site.

We have a Google Trends playground where you can play with our API: https://serpapi.com/playground?engine=google_trends

See you in the next post!

Image data parsing: From Image to data (Using Vision API)

Hilman Ramadhan — Thu, 18 Jan 2024 00:24:45 +0000

The AI becomes ~~scarier~~ better every day. OpenAI now offers the vision API, which allows you to extract information from an image.

We'll learn how to use Vision API by OpenAI in a simple image and extract data from complex images.

_We experimented with parsing HTML raw data with AI before, feel free to read the blog post: Web scraping experiment with AI (Parsing HTML with GPT-4)

Vision API tutorial step-by-step

Let's start with setting up a project to test the Vision API. I'll be using Javascript (Nodejs) in this sample, but feel free to use any language you're comfortable with.

Preparation

Create a new directory and initialize NPM

mkdir openai-vision-api && cd openai-vision-api 
npm init -y // NPM init
npm install openai dotenv --save  // Install openai and dotenv package

Add API Key

Get your API Key from openAI dashboard, and put it in the .env file. Feel free to create a new .env file.

OPENAI_API_KEY=YOUR_API_KEY

Basic code setup

Create a new index.js file and import related packages and create a new openai instance

require("dotenv").config();
const OpenAI = require('openai');

const { OPENAI_API_KEY } = process.env;

const openai = new OpenAI({
  apiKey: OPENAI_API_KEY,
});

Add vision API method

Here is how to call a vision API in your code

async function main() {
  const response = await openai.chat.completions.create({
    model: "gpt-4-vision-preview",
    messages: [
      {
        role: "user",
        content: [
          { type: "text", text: "What’s in this image?" },
          {
            type: "image_url",
            image_url: {
              "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
            },
          },
        ],
      },
    ]
  });
  console.log(response.choices[0].message.content);
}
main();

Now run the program with

node index.js

Here is the result:

Parsing data from complex image with Vision API

We saw it worked with a simple image. Now, let's try for a complex one. I'm going to take a screenshot from Google Shopping results.

I'll upload this image, to use the public URL on our Vision API

I need to update two things: first, the token parameter since the response should be longer. Second is the prompt, to tell exactly what I want from the AI.

async function main() {
  const response = await openai.chat.completions.create({
    model: "gpt-4-vision-preview",
    messages: [
      {
        role: "user",
        content: [
          { type: "text", text: "Please share the detail information of each item on this product on a nice structure JSON" },
          {
            type: "image_url",
            image_url: {
              "url": "https://i.ibb.co/F8nGWk5/Clean-Shot-2024-01-17-at-13-46-43.png",
            },
          },
        ],
      },
    ],
    max_tokens: 1000 // Add more token
  });
  console.log(response.choices[0].message.content);
}

Here is the result

The result is very good! but here is the catch:

- The response is not always consistent (structure wise). I believe we can solve this by adjusting our prompt

- The time taken for this particular image is range between 10+ to 20+ seconds. (It's just the parsing time, not the scraping time).

Can we use this as a web scraping solution?

As you might know, parsing data is just a part of web scraping. There are other things involved like proxy rotation, solving captcha, and so on. So we can't say that vision API is a web scraping solution.

Here is the idea though, of how to use this as part of our web scraping solution:

- Create a scraping solution, for example using Puppeteer in Javascript to take a screenshot .

- Upload the image to a public URL or get the base64 code

- Pass this image to the vision API method parameter like the one we provided above.

- Return the results in a nice structured way.

- (Bonus) If you want to have a consistent data structure, you might want to learn about function calling by OpenAI.

Summary

It's very fun to experiment with OpenAI features like vision API and see the possibility to help us with web scraping and parsing.

In above example, where we try to parse the Google Shopping results page data, it's still far from ready for production, compare to the Google Shopping API, which only take 1-3s to scrape and return the Google Shopping page in a consistent structured format.

FAQ

How much does vision API cost?

Model gpt-4-1106-vision-preview costs $0.01 / 1K tokens for the input and $0.03/1K tokens for the output.

Does it support function calling?

Not right now, the gpt-4-1106-vision-preview haven't support function calling yet (Per 17th January 2024).

Reference: OpenAI Vision API

Scraping the full snippet from Google search result

Hilman Ramadhan — Tue, 02 Jan 2024 00:28:59 +0000

Sometimes, you see truncated text on a Google search result like this (...) . Google doesn't always display the meta description of a website. Sometimes, it gets a snippet of relevant text to the search query, which could truncate the text.

Wonder how you can get the entire snippet of this search result? Let's dive in!

The Idea

The idea is to visit the page URL and scrape part of the relevant text until the next period sign or the whole paragraph.

But before that, we need to find the Google search results list. Therefore, we will use Google search API by SerpApi to scrape the Google SERP.

You can use any programming language you want, but I'll use Go Lang for this sample.

Scraping Google SERP list with Go lang

First, let's get the Google organic results.

Step 1:

Get your SerpApi key

Sign up for free at SerpApi.
Get your SerpApi Api Key from this page.

Step 2:

Create a new Go Lang project

mkdir fullsnippet && cd fullsnippet // Create a new folder and move 
touch main.go // Create a new go file

Step 3:

Install Golang SerpApi package

go mod init project-snippet // Initialize Go Module
go get -u github.com/serpapi/serpapi-golang // Install Go lang package by SerpApi

Step 4:

This is how to get the organic_results from Google SERP

package main

import (
    "fmt"
    "github.com/serpapi/serpapi-golang"
)

const API_KEY = "YOUR_API_KEY"

func main() {
    client_parameter := map[string]string{
        "engine": "google",
        "api_key": API_KEY,
    }
    client := serpapi.NewClient(client_parameter)

    parameter := map[string]string{ 
        "q": "why the sky is blue", // Feel free to change with any keyword
    }

    data, err := client.Search(parameter)
    fmt.Println(data["organic_results"])

    if err != nil {
        fmt.Println(err)
    }
}

We've received each result's title, description, link, and other information.

Collect only specific data

We can collect and display only specific data in a variable like this

data, err := client.Search(parameter)

type OrganicResult struct {
    Title string
    Snippet string
    Link string
}

var organic_results []OrganicResult

for _, result := range data["organic_results"].([]interface{}) {
    result := result.(map[string]interface{})
    organic_result := OrganicResult{
        Title: result["title"].(string),
        Snippet: result["snippet"].(string),
        Link: result["link"].(string),
    }

    organic_results = append(organic_results, organic_result)
}

Scraping the individual page

SerpApi focuses on scraping search results. That's why we need extra help to scrape individual sites. We'll use GoColly package.

Install package

go get -u github.com/gocolly/colly/v2

Code for scraping individual site

Add this code inside the loop

// Scrape each of the link
c := colly.NewCollector()

c.OnHTML("body", func(e *colly.HTMLElement) {
    rawText := e.Text
    fmt.Println("Raw text in entire body tag:", rawText)
})

// Handle any errors
c.OnError(func(r *colly.Response, err error) {
    fmt.Println("Request URL:", r.Request.URL, "failed with response:", r, "\nError:", err)
})

// Start scraping
c.Visit(organic_result.Link)

If you need the whole text of each site, you can return the rawText from above. Then you're done.

But if you need only the snippet part until the next period (the entire sentence), we will continue to the following function.

Scraping only the relevant text

Here's the pseudocode on returning only the relevant full snippet.

Find $partialSnippet in the rawText
Find the position of $partialSnippet
Find the next (closest) period after that partial snippet
Return the whole snippet

Here is the Go Lang code

fullSnippet := findSentence(rawText, snippet)
return fullSnippet

The findSentence method

func findSentence(rawText string, searchText string) string {

    // 1. Replace all whitespaces with a single space
    re := regexp.MustCompile(`\s+`) 
    fullText := re.ReplaceAllString(rawText, " ")

    // 2. Replace all backtik ’ into ' at rawText
    re1 := regexp.MustCompile(`’`)
    fullText = re1.ReplaceAllString(fullText, "'")

    // 3. Find the start index of searchText
    startIndex := strings.Index(fullText, searchText)
    if startIndex == -1 {
        return "Text not found"
    }

    // 4. Calculate the end index of the snippet
    snippetEndIndex := startIndex + len(searchText)

    // 5. Find the end of the sentence after the snippet
    endOfSentenceIndex := strings.Index(fullText[snippetEndIndex:], ".")
    if endOfSentenceIndex == -1 {
        // Return the rest of the text from snippet if not found
        return fullText[startIndex:]
    }

    // Adjust to get the correct index in the full text
    endOfSentenceIndex += snippetEndIndex + 1

    return fullText[startIndex:endOfSentenceIndex]
}

Here is the result

You can create a conditional logic (if statement) to only perform this when the snippet has "..." (three dots in the end).

Full code sample

Here is the full code sample in GitHub: https://github.com/hilmanski/serpapi-fullsnippet-golang

Warning

Here are a few potential issues and solutions with our method.

Different snippet format

This might not work when Google displays a snippet list, where the snippet comes from some headings or key points. We'll need to write a different logic for this.

Adding proxy

To prevent getting blocked when scraping the individual site, you can add proxies to the GoColly.

As a reminder, You don't need to worry about getting block for scraping the Google search itself when using SerpApi.

Reference: https://go-colly.org/docs/examples/proxy_switcher/

Sample proxy switcher

package main

import (
    "bytes"
    "log"

    "github.com/gocolly/colly"
    "github.com/gocolly/colly/proxy"
)

func main() {
    // Instantiate default collector
    c := colly.NewCollector(colly.AllowURLRevisit())

    // Rotate two socks5 proxies
    rp, err := proxy.RoundRobinProxySwitcher("socks5://127.0.0.1:1337", "socks5://127.0.0.1:1338")
    if err != nil {
        log.Fatal(err)
    }
    c.SetProxyFunc(rp)

    // Print the response
    c.OnResponse(func(r *colly.Response) {
        log.Printf("%s\n", bytes.Replace(r.Body, []byte("\n"), nil, -1))
    })

    // Fetch httpbin.org/ip five times
    for i := 0; i < 5; i++ {
        c.Visit("https://httpbin.org/ip")
    }
}

I hope it helps you to collect more data for your Google SERP!