Forem: Oleg Kulyk

Robots are not that bad

Oleg Kulyk — Wed, 13 Aug 2025 15:29:12 +0000

I don’t chase blue links anymore. I chase the sentence people trust.

One night, I asked a friend a simple question, and they wondered if ChatGPT could answer it instead. The reply was calm and useful, quoting someone else - not me. A brand name sat there like a signature. It wasn’t a banner. It was better. It was the line in the answer.

I went home and rewired my pages for that moment.

Two honest sentences at the top that actually answer the question. Clean the schema on products and FAQs so a machine can see what a human sees. A tiny JSON feed with stable IDs and updated timestamps, so the model doesn’t have to guess. I left my name near the facts, not as a pitch but as a source.

A week later, I tried again. Same question, same calm tone — and this time the answer borrowed my words. My brand was in the sentence. There was a small “visit” button, but even when people didn’t click, they came later with the right spelling and the exact model in mind. Zero-click wasn’t zero-intent; it was planted intent.

Being the source changes the game. You don’t fight for position ten blue links down — you live upstream, where the answer is written. You become the footnote, and footnotes are sticky. People repeat them. Assistants cite them. Aggregators copy them with your canonical intact. The traffic that does arrive is weirdly decisive: fewer tabs, more checkouts. It feels like the sale started somewhere else and finished on your page.

Do I still care about crawlers? Yes, because the concierge asks them for directions. So I keep a polite lane open: robots.txt that says “this way,” feeds that are light and cacheable, and an API key for the hungry ones. If a new agent knocks and says Please, I give them the good version of my truth.

Now, when I publish, I ask: “Would I quote this?” If the answer is yes, I ship it that way - answer-first, machine-readable, signed with my name. The rest is compounding.

BTW, check out ScrapingAnt - we're doing some cool stuff.

Proper AI data extraction without manual HTML parsing

Oleg Kulyk — Wed, 27 Sep 2023 08:48:20 +0000

In my previous article, I've described the basic concept behind the AI data extraction using ChatGPT and some readers contacted back to me with questions and complains that the mentioned approach is not that good.

They were entirely right.

The main issues are:

Improper ChatGPT AI model usage, which leads to poor extraction quality
Limited input size, which doesn't allow processing something bigger than example.com

So, let's dig deeper and find out how to address those issues and implement proper AI data extraction.

1. Improper ChatGPT AI Model Usage

The first issue was the improper use of the ChatGPT AI model, leading to poor extraction quality. This mainly happened because not all pages follow the same HTML structure, "confusing the model".

The best way to overcome this is to use Chat Models from OpenAI, like GPT-3.5 or GPT-4.

This way, you can set up your HTML extraction using a similar code:

response = openai.ChatCompletion.create(
    model="gpt-3.5-turbo",
    messages=[
        {"role": "user", "content": "Extract <your parameters> from <HTML code>"},
    ]
)

Also, you could use more capable models and apply more sophisticated layouts.

2. Input size

The second issue was the limitation in input size, which barred the processing of more extensive websites.

To tackle this, divide the larger content into smaller, manageable chunks and then process them sequentially or in parallel. This method will allow the AI model to handle larger datasets effectively without being overwhelmed.

Or use GPT-4 model 🙂

What's the cost?

Using powerful GPT-3.5 or GPT-4 models for data extraction from web pages is an over-engineered solution, so paying for such an amount of tokens could be not that economically efficient (usually, it is still cheaper than supporting your parsers and paying the support team of developers).

So, the solution should fit exactly the purpose to be the most cost-efficient.

AI extractor from ScrapingAnt

The Beta version of the solution is here!

ScrapingAnt team developed unique AI Data Extractor technology, which is suitable for any-size websites and requires only 2 parameters: URL and free-form input!

A JSON with automatically converted input parameter names would appear as the output. Which is near magic!

It's time to turn any web site into API with JSON format!

The best thing I like about this tool is that it fully supports nesting and types, so you don't need to struggle with loops like in other no-code tools, but write it down:

product title, price(number), full description, reviews(list: review title, review content)

Such extraction input could be sent to ScrapingAnt AI API for some Amazon URL like: https://www.amazon.com/dp/B0725MVKCZ/

To receive a similar output:

{
  "productTitle":"MREs (Meals Ready-to-Eat) Genuine U.S. Military Surplus Assorted Flavor (6-Pack)",
  "price":60,
  "fullDescription":"MREs (Meals Ready-to-Eat) Genuine U.S. Military Surplus Assorted Flavor (6-Pack). Brand: MRE. Number of Items: 1. Flavor: Assorted. Item Weight: 0.01 Ounces. Size: 6 Count (Pack of 1). Long shelf life when stored per manufacturer's directions. 2012 or newer Pack Date. Genuine US War Fighter Rations are the ultimate survivalist, Prepper & outdoor enthusiast Meal. Ideal for hunting, camping, hiking, fishing, boating, and emergency food supply. Designed for maximum endurance and nutrition with average 1250 calories per meal.",
  "reviews":[
    {
      "reviewTitle":"Will buy from Food Dude again",
      "reviewContent":"Great selection! I ordered 6 MRE and they are all different and a nice selection. I also ordered a First Strike MRE and you get what you pay for. This one is a Monster and it’s packed to the hilt. Not only will I buy from this vendor again i already ordered 4 more and will be getting a case on payday. Thanks so much for the lightning fast shipping. Great Seller,Great product!"
    },
    {
      "reviewTitle":"AWESOME!",
      "reviewContent":"This is the second time I've purchased this product, and if I could leave 6 stars, I would! Not only are the meals exactly as described, and arrived in good time, but the seller communication is phenomenal. I made a special request on my second purchase and my message was answered very quickly, with my request being fulfilled perfectly. I highly recommend this seller, and the MREs themselves. They're great for camping, emergency rations, or just to have instead of cooking or eating out!"},
...]}

This is exactly the technology I've been looking for for the past 2 years, as you no longer need to write your HTML parsers and work with Cheerio or BeautifulSoup!

The extractor itself is available to test using the UI request generator. And it's free to try without a credit card 🙃

Also, you can check more at documentation pages.

I'm excited to share this technology with this community and would appreciate your comments and feedback!

Leverage the Power of ChatGPT API and web scraping API to Parse Web Pages Efficiently

Oleg Kulyk — Sat, 18 Mar 2023 16:43:53 +0000

Introduction

Web scraping is an essential process for many businesses, researchers, and developers who want to extract valuable data from the internet. With the rapid growth of web technologies, the need for efficient and reliable web scraping tools has become increasingly important. In this article, we'll discuss how to parse web pages using the ChatGPT API and the ScrapingAnt API, two powerful tools designed to make web scraping easier and more efficient.

The ChatGPT API, developed by OpenAI, is a powerful language model that can process and generate human-like text. By leveraging its capabilities, you can enhance your web scraping processes and improve the quality of extracted data. On the other hand, ScrapingAnt API is a web scraping API that provides access to headless browsers, enabling users to extract web data efficiently.

Let's dive into the process of parsing web pages using these two powerful APIs.

ChatGPT data extraction

1. Setting up the ChatGPT API

Before using the ChatGPT API, you need to acquire an API key. Follow these steps to set up the API:

a) Sign up for an OpenAI account at https://beta.openai.com/signup/.

b) Navigate to the API key section and generate your API key.

c) Install the OpenAI Python library using pip:

pip install openai

d) Import the library in your Python script and configure the API key:

import openai

openai.api_key = "your_api_key_here"

2. Setting up the ScrapingAnt API

To use the ScrapingAnt API, follow these steps:

a) Sign up for a ScrapingAnt account at https://scrapingant.com/.

b) Obtain your API key from the dashboard.

c) Install the requests library to make HTTP requests in Python:

pip install requests

3. Parsing Web Pages with ChatGPT API and ScrapingAnt API

Now that we have both APIs set up, let's see how to use them together to parse web pages:

a) Make an API request to ScrapingAnt to scrape the desired web page:

import requests

url_to_scrape = "https://example.com"
scrapingant_api_key = "your_scrapingant_api_key_here"

response = requests.get(f"https://api.scrapingant.com/v1/general?url={url_to_scrape}&x-api-key={scrapingant_api_key}")

b) Extract the HTML content of the web page:

html_content = response.json()["content"]

c) Use the ChatGPT API to parse the HTML content and extract the desired information:

def parse_html_with_chatgpt(html, extraction_instructions):
    prompt = f"Parse the following HTML content and {extraction_instructions}:\n\nHTML:\n{html}\n\nAnswer:"

    response = openai.Completion.create(
        engine="text-davinci-002",
        prompt=prompt,
        max_tokens=100,
        n=1,
        stop=None,
        temperature=0.5,
    )

    answer = response.choices[0].text.strip()
    return answer

# Example: Extract the main heading from the HTML content
extraction_instructions = "extract the main heading"
main_heading = parse_html_with_chatgpt(html_content, extraction_instructions)
print(main_heading)

By combining the power of the ChatGPT API and the ScrapingAnt API, you can effectively parse web pages and extract valuable information with ease. This approach is versatile and can be adapted to various web scraping tasks, making it a powerful solution for businesses, researchers and data experts.

Conclusion

In this article, we have demonstrated the potential of combining the ChatGPT API and the ScrapingAnt API to parse web pages and extract valuable information efficiently. By leveraging the strengths of both APIs, developers can simplify the web scraping process, reduce the need for complex and time-consuming manual coding, and enhance the quality of extracted data.

Whether you are a business owner, researcher, or developer, the combination of ChatGPT and ScrapingAnt APIs can help you streamline your web scraping projects, allowing you to focus on what matters most - turning the extracted data into valuable insights and driving better decision-making. Embrace the power of these two APIs and revolutionize your web scraping endeavors today!

How to make POST, PUT and DELETE requests using Puppeteer?

Oleg Kulyk — Tue, 09 Nov 2021 13:48:10 +0000

Making POST, PUT, and DELETE requests is a crucial web scraping and web testing technique. Still, this functionality is not included in Puppeteer's API as a separate function.

Let's check out the workaround for this situation and create a helper function to fix this out.

Why do we need POST while web scraping?

POST request is one of several available HTTP request types. By design, this method is used to send data to a web server for processing and possible storage.

POST requests usage is one of the main ways to send form data during login or registration. It is also one of the ways to send any data to the web server.

Earlier, one of the main implementation patterns of login or registration was to send the form data with the required authorization parameters via POST request and get the protected content as a response to this request (along with cookies to avoid re-entering the authentication and authorization data).

Nowadays SPAs (Single Page Applications) also use POST requests to send data to API, but such requests usually return only necessary data to update web page and not the whole page.

Thus, many sites use POST requests for client-server communication and this requires the ability of sending POST requests while web scraping.

Unfortunately, Puppeteer developers haven't introduced the native way of making requests other than GET, but it's not a big deal for us to create a workaround.

Interception of the initial request

The idea behind our approach is quite simple - we need to change the request type while opening the page, so we can send POST data along with opening a page.

To do that, we have to intercept the request using page.on('request') handler.

We're going to use HTTPBin which can help us with our solution testing.

Let's check out the simple JS snippet which just opens HTTPBin's POST endpoint:

const puppeteer = require('puppeteer');
const TARGET_URL = 'https://httpbin.org/post';

(async () => {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    await page.goto(TARGET_URL);
    console.log(await page.content());
})();

The result is, definitely, not what we're trying to achieve:

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 3.2 Final//EN"><html><head><title>405 Method Not Allowed</title>
</head><body><h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>
</body></html>

So, let's add a request interception:

const puppeteer = require('puppeteer');
const TARGET_URL = 'https://httpbin.org/post';
const POST_JSON = { hello: 'I like ScrapingAnt' };

(async () => {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    await page.setRequestInterception(true);
    page.once('request', request => {
        request.continue({ method: 'POST', postData: JSON.stringify(POST_JSON), headers: request.headers });
    });
    await page.goto(TARGET_URL);
    console.log(await page.content());
})();

This time our POST request has been successfully executed:

<html><head></head><body><pre style="word-wrap: break-word; white-space: pre-wrap;">{
  "args": {}, 
  "data": "{\"hello\":\"I like ScrapingAnt\"}", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept-Encoding": "gzip, deflate, br", 
    "Accept-Language": "en-US", 
    "Content-Length": "30", 
    "Host": "httpbin.org", 
    "Sec-Fetch-Dest": "document", 
    "Sec-Fetch-Mode": "navigate", 
    "Sec-Fetch-Site": "none", 
    "Sec-Fetch-User": "?1", 
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/93.0.4577.0 Safari/537.36", 
    "X-Amzn-Trace-Id": "Root=1-61757548-7608d72817d01768524a3298"
  }, 
  "json": {
    "hello": "I like ScrapingAnt"
  }, 
  "origin": "7.133.7.133", 
  "url": "https://httpbin.org/post"
}
</pre></body></html>

Unfortunately, this code is only a proof-of-concept, not a complete solution, because any request made by the browser will be converted to a POST request.

Let's improve our code and make it production-ready.

Puppeteer's extension

The next big step of our HTTP requests Puppeteer journey is to create something reusable to avoid code duplication while making non-GET requests.

To achieve that, let's create a function gotoExtended:

async function gotoExtended(page, request) {
    const { url, method, headers, postData } = request;

    if (method !== 'GET' || postData || headers) {
        let wasCalled = false;
        await page.setRequestInterception(true);
        const interceptRequestHandler = async (request) => {
            try {
                if (wasCalled) {
                    return await request.continue();
                }

                wasCalled = true;
                const requestParams = {};

                if (method !== 'GET') requestParams.method = method;
                if (postData) requestParams.postData = postData;
                if (headers) requestParams.headers = headers;
                await request.continue(requestParams);
                await page.setRequestInterception(false);
            } catch (error) {
                log.debug('Error while request interception', { error });
            }
        };

        await page.on('request', interceptRequestHandler);
    }

    return page.goto(url);
}

The usage of this function is straightforward and simple:

const puppeteer = require('puppeteer');
const TARGET_URL = 'https://httpbin.org/post';
const POST_JSON = { hello: 'I like ScrapingAnt' };
const headers = { header_1: 'custom_header' };

(async () => {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    await gotoExtended(page, { url: TARGET_URL, method: 'POST', postData: JSON.stringify(POST_JSON), headers });
    console.log(await page.content());
})();

It is also possible to use this helper function with any available HTTP method and custom headers.

Conclusion

Having the ability to send any request type using Puppeteer allows web scraping specialists to extend their capabilities and improve their data crawlers' performance by skipping unnecessary steps like form filling.

As usual, we recommend you extend your web scraping knowledge using our articles:

Web Scraping with Javascript (NodeJS) - JavaScript libraries to scrape data
Download image with Javascript (NodejS) - how to download files using NodeJS
Get all text from the page using Puppeteer - how to extract all the text from a page using Puppeteer
HTML Parsing Libraries - JavaScript - JavaScript HTML parsing libraries overview

Happy Web Scraping, and don't forget to cover your code with unit-tests 👷

How to download images with NodeJS?

Oleg Kulyk — Thu, 26 Aug 2021 18:59:04 +0000

Working with images in NodeJS extends your web scraping capabilities, from downloading the image with an URL to retrieving photo attributes like EXIF. How to achieve the image download and obtain the data?

Let's start our walk through the several methods used to download images in NodeJS.

Download an image using `http.request`

Our image downloading journey starts with the default NodeJS HTTP(S) client. Needless to say that this is the most used library to get the data across the backend Javascript community. Also, it's a default way to download any file type.

Our goal is to create a function that can download and save the image. This function should have 2 parameters input - url and filepath.

url will be used to specify the remote image path (URL or path at the server) and filepath - path to be downloaded in (where to save the image). So, the empty function will be the following:

function downloadImage(url, filepath) {

}

Let's keep this signature across all the file downloading methods, so we'll be able to substitute the content of the function without changing the output. Also, it is helpful for unit testing and clean coding.

The vanilla downloading code will have a following look:

const fs = require('fs');
const client = require('https');

function downloadImage(url, filepath) {
    client.get(url, (res) => {
        res.pipe(fs.createWriteStream(filepath));
    });
}

We're using https.get function to process the file downloading from the server, while fs streaming allows us to save the file to the defined path.

https module is used here to process the encrypted https requests (as I assume, that most of the Internet is secured with SSL). Otherwise, https should be replaced with http without any extra coding.

Still, this function requires some extra modification. It doesn't notify us about success or failure, and we won't track the processing end. So let's fix this situation by promisifying it.

const fs = require('fs');
const client = require('https');

function downloadImage(url, filepath) {
    return new Promise((resolve, reject) => {
        client.get(url, (res) => {
            if (res.statusCode === 200) {
                res.pipe(fs.createWriteStream(filepath))
                    .on('error', reject)
                    .once('close', () => resolve(filepath));
            } else {
                // Consume response data to free up memory
                res.resume();
                reject(new Error(`Request Failed With a Status Code: ${res.statusCode}`));

            }
        });
    });
}

Voila! Our function returns a promise which allows us to track the process completion and the status.

This function usage is well known for most Javascript developers:

downloadImage('https://upload.wikimedia.org/wikipedia/en/thumb/7/7d/Lenna_%28test_image%29.png/440px-Lenna_%28test_image%29.png', 'lena.png')
    .then(console.log)
    .catch(console.error);

Let's move forward and check out another popular option.

The modern way - Axois download image and any file

axios is a simple and modern promise based HTTP client that can be used for client-side and server-side applications.

It is another favored method for downloading data in Javascript.

To install axios you can use npm or your favorite package manager like yarn:

npm install axios

Then we're able to replace our function internal to get the same functionality. Also, we're going to add async/await flavor to our code.

const fs = require('fs');
const Axios = require('axios')

async function downloadImage(url, filepath) {
    const response = await Axios({
        url,
        method: 'GET',
        responseType: 'stream'
    });
    return new Promise((resolve, reject) => {
        response.data.pipe(fs.createWriteStream(filepath))
            .on('error', reject)
            .once('close', () => resolve(filepath)); 
    });
}

As I've mentioned before, we can change the entire function content while keeping the behavior persistent.

Still, it's Javascript so that you can resolve every specific task with a separate module.

Be specific - use a separate NodeJS download module

As I've mentioned before, Javascript allows you to resolve most of the tasks with a separate module, and image downloading using NodeJS is not an exclusion from this rule.

Meet image-downloader

It's a Node module for downloading image to disk from a given URL.

It can be installed by execution of the following command:

npm install image-downloader

This kind of library allows you to solve your specific task with the smallest possible amount of code. To demonstrate this, we will rewrite our function for the module usage:

const download = require('image-downloader');

function downloadImage(url, filepath) {
    return download.image({
       url,
       dest: filepath 
    });
}

Pretty terse, isn't it?

Conclusion

As always, each of these methods has its pros and cons. Still, such a variety of available ways of an image download allows you to pick up the best one. I'd recommend only one approach - avoid bloating the codebase with many libraries and stick up to one HTTP client.

If you're looking for even more ways to download images and files from the web with Javascript, I suggest you check out the article, Javascript Web Scraping: HTTP clients.

Web Scraping with Javascript (NodeJS) - JavaScript libraries to scrape data
HTML Parsing Libraries - JavaScript - JavaScript HTML parsing libraries overview

Happy Web Scraping, and don't forget to enable GZIP compression in your HTTP client for the proxy traffic saving 💰

Block resources with Playwright

Oleg Kulyk — Mon, 21 Jun 2021 16:55:20 +0000

This article will expose how to block specific resources (HTTP requests, CSS, video, images) from loading in Playwright. Playwright is Puppeteer's successor with the ability to control Chromium, Firefox, and Webkit. So I'd call it the second one of the most widely used web scraping and automation tools with headless browser support.

Why block resources

Block resources from loading while web scraping is a widespread technique that allows you to save time and costs.

For example, when you crawl a resource for product information (scrape price, product name, image URL, etc.), you don't need to load external fonts, CSS, videos, and images themselves. However, you'll need to extract text information and direct URLs for media content for most cases.

Also, such improvements will:

speed up your web scraper
increase number of pages scraped per minute (you'll pay less for your servers and will be able to get more information for the same infrastructure price)
decrease proxy bills (you won't use proxy for irrelevant content download)

Intercept requests with Playwright

Request interception

Since Playwright is a Puppeteer's successor with a similar API, it can be very native to try out using the exact request interception mechanism. Also, from the documentation for both libraries, we can find out the possibility of accessing the page's requests.

const playwright = require('playwright');

(async () => {
    const browser = await playwright['chromium'].launch();
    const context = await browser.newContext();
    const page = await context.newPage();
    page.on('request', (request) => {
       console.log(`Request: ${request.url()} to resource type: ${request.resourceType()}`);
    });
    await page.goto('https://amazon.com');

    await browser.close();
})();

So, the output will provide information about the requested resource and its type.

Request: https://amazon.com/ to resource type: document
Request: https://www.amazon.com/ to resource type: document
......
Request: https://m.media-amazon.com/images/I/41Kf0mndKyL._AC_SY200_.jpg to resource type: image
Request: https://m.media-amazon.com/images/I/41ffko0T3kL._AC_SY200_.jpg to resource type: image
Request: https://m.media-amazon.com/images/I/51G8LfsNZzL._AC_SY200_.jpg to resource type: image
Request: https://m.media-amazon.com/images/I/41yavwjp-8L._AC_SY200_.jpg to resource type: image
......
Request: https://m.media-amazon.com/images/S/sash/2SazJx$EeTHfhMN.woff2 to resource type: font
Request: https://m.media-amazon.com/images/S/sash/ozb5-CLHQWI6Soc.woff2 to resource type: font
Request: https://m.media-amazon.com/images/S/sash/KwhNPG8Jz-Vz2X7.woff2 to resource type: font

Still, according to Playwright's documentation, the Request callback object is immutable, so you won't be able to manipulate the request using this callback.

Let's check out the Playwright's suggestion about this situation:


  /**
   * Emitted when a page issues a request. The [request] object is read-only. In order to intercept and mutate requests, see
   * [page.route(url, handler)](https://playwright.dev/docs/api/class-page#pagerouteurl-handler) or
   * [browserContext.route(url, handler)](https://playwright.dev/docs/api/class-browsercontext#browsercontextrouteurl-handler).
   */
  on(event: 'request', listener: (request: Request) => void): this;

Cool. Let's use page.route for the request manipulations.

Route interception

The concept behind using page.route interception is very similar to Puppeteer's page.on('request'), but requires indirect access to Request object using route.request.

const playwright = require('playwright');

(async () => {
    const browser = await playwright['chromium'].launch();
    const context = await browser.newContext();
    const page = await context.newPage();
    await page.route('**/*', (route) => {
        return route.request().resourceType() === 'image'
            ? route.abort()
            : route.continue()
    });
    await page.goto('https://amazon.com');
    await page.screenshot({ path: 'amazon_no_images.png' });

    await browser.close();
})();

So, we're using intercepting routes and then indirectly accessing the requests behind these routes.

As a result, you will see the website images not being loaded.

All the supported resource types can be found below:

- stylesheet 
- image 
- media 
- font 
- script 
- texttrack 
- xhr 
- fetch 
- eventsource 
- websocket 
- manifest 
- other

Also, you can apply any other condition for request prevention, like the resource URL:

const playwright = require('playwright');

(async () => {
    const browser = await playwright['chromium'].launch();
    const context = await browser.newContext();
    const page = await context.newPage();
    await page.route('**/*', (route) => {
        return route.request().url().endsWith('.jpg')
            ? route.abort()
            : route.continue()
    });
    await page.goto('https://amazon.com');
    await page.screenshot({ path: 'amazon_no_jpg_images.png' });

    await browser.close();
})();

Improve SPA page scraping speed (Vue.js, React.js, etc.)

Since the start of my web scraping journey, I've found pretty neat the following exclusion list that improves Single-Page Application scrapers and decreases scraping time up to 10x times:

const playwright = require('playwright');

const RESOURCE_EXCLUSTIONS = ['image', 'stylesheet', 'media', 'font','other'];

(async () => {
    const browser = await playwright['chromium'].launch();
    const context = await browser.newContext();
    const page = await context.newPage();
    await page.route('**/*', (route) => {
        return RESOURCE_EXCLUSTIONS.includes(route.request().resourceType())
            ? route.abort()
            : route.continue()
    });
    await page.goto('https://amazon.com');

    await browser.close();
})();

Such code snippet prevents binary and media content loading while providing all required dynamic web page load.

Summary

Request interception is a basic web scraping technique that allows improving crawler performance and saving money while doing data extraction at scale.

To save more money, you can check out the web scraping API concept. It already handles headless browser and proxies for you, so you'll forget about giant bills for servers and proxies.

Also, those articles might be interesting for you:

Web Scraping with Javascript (NodeJS) - to learn more about web scraping with Javascript
How to download a file with Playwright? - downloading files with Playwright (Javascript)
How to submit a form with Playwright? - submitting forms with Playwright (Javascript)

Happy Web Scraping, and don't forget to enable caching in your headless browser 💾

How to submit a form with Playwright?

Oleg Kulyk — Sun, 30 May 2021 16:05:54 +0000

In this article, we'll take a look at how to submit forms using Playwright. This knowledge might be beneficial while scraping the web, as it allows to get the information from the target web page, which requires providing parameters before.

We're going to check out several different topics about the form submission:

how to fill an input and click a button using Playwright API
how to upload a file to a file input using Playwright API
hot to fill an input and click a button using an internal Javascript execution

Automating form submission

Let's start with automating Google search using Playwright.

By visiting Google.com and revealing HTML sources, we may observe required selectors for both the search input field and the search button.

The text input has the name attribute q, while the form submission button has the name btnK.

To make such an inspection, open the browser's Dev Tools and inspect the form. To know more about Developer Tools inspection, check out our web scraping guide.

Those selectors will help us get control over the input and button elements, so we'll automate Google search.

Let's find something in Google. Like ScrapingAnt is awesome 😀

const playwright = require('playwright');

(async () => {
    const browser = await playwright['chromium'].launch();
    const context = await browser.newContext();
    const page = await context.newPage();
    await page.goto('https://google.com');
    await page.type('input[name=q]', 'ScrapingAnt is awesome');
    await page.click('input[name=btnK]:visible');
    await page.waitForTimeout(5000);
    await page.screenshot({ path: 'scrapingant.png' });

    await browser.close();
})();

And the result is the expected one:

The exact input fill was implemented using page.type method with a proper CSS selector.

The more exciting part of this form submission is related to form click. While the Google.com page has several buttons with this name, we have to pick the button capable of being clicked. To select it, we've used a CSS pseudo-selector :visible.

Submitting a form with attachments

Sometimes web scraper or browser automation tool may require submitting a file as a part of the form submission.

Let's take a look at how we can upload a file using Playwright. We will use this demo form from W3 school to demonstrate file attachments in the scope of the HTML forms.

We will upload a previous screenshot we've made using Playwright and will create a new one:

const playwright = require('playwright');

(async () => {
    const browser = await playwright['chromium'].launch();
    const context = await browser.newContext();
    const page = await context.newPage();
    await page.goto('https://www.w3schools.com/howto/howto_html_file_upload_button.asp');
    await page.setInputFiles('input[type=file]', 'scrapingant.png');
    await page.screenshot({ path: 'scrapingant.png' });

    await browser.close();
})();

As a result, we may observe a prepared file to be sent using the HTML form.

Playwright's API is capable to upload files to the inputs using page.setInputFiles method.

It makes it possible to use a file from the filesystem, several files, remove a file from the input and even use a buffer to pass the data into the file input.

Submitting a form using Javascript

Sometimes you might not be able to access a Playwright API (or any other API like Puppeteer's one), but you'll be able to execute a Javascript snippet in the context of the scraped page. For example, ScrapingAnt web scraping API provides such ability without dealing with the browser controller itself.

So, luckily, it also possible to fill a form using the Javascript code inside the page. Let's rewrite a Google Search sample using it:

const playwright = require('playwright');

(async () => {
    const browser = await playwright['chromium'].launch();
    const context = await browser.newContext();
    const page = await context.newPage();
    await page.goto('https://google.com');
    await page.evaluate(() => {
        document.querySelector('input[name=q]').value = 'ScrapingAnt is awesome';
        document.querySelector('input[name=btnK]:not([hidden])').click();
    });
    await page.waitForTimeout(5000);
    await page.screenshot({ path: 'scrapingant.png' })

    await browser.close();
})();

The mentioned code doesn't use Playwright API to fill inputs or click a button. Instead, it uses an internal page context to grab the DOM element using a query selector (document.querySelector) and manipulate it.

Also, you might observe that the pseudo-selector :visible has been replaced by :not([hidden]), which is supported and can be used in such case (:visible is not).

Conclusion

As you can see, there are a lot of possible ways to fill a form using Playwright. Some of them require basic knowledge of Playwright's API, while the other - not, which makes it possible to use the same form submitting code across the Playwright, Puppeteer, or the web scraping API.

I'm highly recommending to check out the following resources to know more about the Playwright and form submission:

Happy Web Scraping, and don't forget to wait for the selector before the data extraction start 🕒

How to download a file with Playwright?

Oleg Kulyk — Wed, 26 May 2021 12:22:53 +0000

In this article, we will share several ideas on how to download files with Playwright. Automating file downloads can sometimes be confusing. You need to handle a download location, download multiple files simultaneously, support streaming, and even more. Unfortunately, not all the cases are well documented. Let's go through several examples and take a deep dive into Playwright's APIs used for file download.

Downloading a file after the button click

The pretty typical case of a file download from the website is leading by the button click. By the fast Google'ing of the sample files storages I've found the following resource: https://file-examples.com/

Let's use it for further code snippets.

Our goal is to go through the standard user's path while the file download: select the appropriate button, click it and wait for the file download. Usually, those files are download to the default specified path. Still, it might be complicated to use while dealing with cloud-based browsers or Docker images, so we need a way to intercept such behavior with our code and take control over the download.

To click a particular button on the web page, we must distinguish it by the CSS selector. Our desired control has a CSS class selector .btn.btn-orange.btn-outline.btn-xl.page-scroll.download-button or simplified one .download-button:

Let's download the file with the following snippet and check out a path of the downloaded file:

const playwright = require('playwright');
const pageWithFiles = 'https://file-examples.com/index.php/sample-video-files/sample-avi-files-download/';

(async () => {
    const browser = await playwright['chromium'].launch();
    const context = await browser.newContext({ acceptDownloads: true });
    const page = await context.newPage();
    await page.goto(pageWithFiles);
    const [ download ] = await Promise.all([
        page.waitForEvent('download'), // wait for download to start
        page.click('.download-button')
    ]);
    // wait for download to complete
    const path = await download.path();
    console.log(path);
    await browser.close();
})();

This code snippet shows us the ability to handle file download by receiving the Download object that is emitted by page.on('download') event.

Note

Browser context must be created with the acceptDownloads set to true when user needs access to the downloaded content. If acceptDownloads is not set, download events are emitted, but the actual download is not performed and user has no access to the downloaded files.

After executing this snippet, you'll get the path that is probably located somewhere in the temporary folders of the OS.

For my case with macOS, it looks like the following:

/var/folders/3s/dnx_jvb501b84yzj6qvzgp_w0000gp/T/playwright_downloads-wGriXd/87c96e25-5077-47bc-a2d0-3eacb7e95efa

Let's define something more reliable and practical by using saveAs method of the download object. It's safe to use this method until the complete download of the file.

const playwright = require('playwright');
const pageWithFiles = 'https://file-examples.com/index.php/sample-video-files/sample-avi-files-download/';
const reliablePath = 'my-file.avi';

(async () => {
    const browser = await playwright['chromium'].launch();
    const context = await browser.newContext({ acceptDownloads: true });
    const page = await context.newPage();
    await page.goto(pageWithFiles);
    const [ download ] = await Promise.all([
        page.waitForEvent('download'), // wait for download to start
        page.click('.download-button')
    ]);
    // save into the desired path
    await download.saveAs(reliablePath);
    // wait for the download and delete the temporary file
    await download.delete()
    await browser.close();
})();

Awesome!

The file will be downloaded to the root of the project with the filename my-file.avi and we don't have to be worried about copying it from the temporary folder.

But can we simplify it somehow? Of course. Let's download it directly!

Direct file download

You've probably mentioned that the button we're clicked at the previous code snippet already has a direct download link:

<a href="https://file-examples-com.github.io/uploads/2018/04/file_example_AVI_480_750kB.avi" download="file_example_AVI_480_750kB.avi" class="btn btn-orange btn-outline btn-xl page-scroll download-button">Download sample AVI file</a>

So we can use the href value of this button to make a direct download instead of using Playwright's click simulation.

To make a direct download, we'll use two native NodeJS modules, fs and https, to interact with a filesystem and file download.

Also, we're going to use page.$eval function to get our desired element.

const playwright = require('playwright');
const https = require('https');
const fs = require('fs');

const pageWithFiles = 'https://file-examples.com/index.php/sample-video-files/sample-avi-files-download/';
const reliablePath = 'my-file.avi';

(async () => {
    const browser = await playwright['chromium'].launch();
    const context = await browser.newContext({ acceptDownloads: true });
    const page = await context.newPage();
    await page.goto(pageWithFiles);
    const file = fs.createWriteStream(reliablePath);
    const href = await page.$eval('.download-button', el => el.href);
    https.get(href, function(response) {
        response.pipe(file);
    });
    await browser.close();
})();

The main advantage of this method is that it is faster and simple than the Playwright's one. Also, it simplifies the whole flow and decouples the data extraction part from the data download. Such decoupling makes available decreasing proxy costs, too, as it allows to avoid using proxy while data download (when the CAPTCHA or Cloudflare check already passed).

Downloading multiple files in parallel

While preparing this article, I've found several similar resources that claim single-threaded problems while the multiple files download.

NodeJS indeed uses a single-threaded architecture, but it doesn't mean that we have to spawn several processes/threads in order to download several files in parallel.

All the I/O processing in the NodeJS is asynchronous (when you're making the invocation correctly), so you haven't to worry about parallel programming while downloading several files.

Let's extend the previous code snippet to download all the files from the pages in parallel. Also, we'll log the events of the file download start/end to ensure that the downloading is processing in parallel.

const playwright = require('playwright');
const https = require('https');
const fs = require('fs');

const pageWithFiles = 'https://file-examples.com/index.php/sample-video-files/sample-avi-files-download/';
const reliablePath = 'my-file.avi';

(async () => {
    const browser = await playwright['chromium'].launch();
    const context = await browser.newContext({ acceptDownloads: true });
    const page = await context.newPage();
    await page.goto(pageWithFiles);
    const hrefs = await page.$$eval('.download-button', els => els.map(el => el.href));
    hrefs.forEach((href, index) => {
        const filePath = `${reliablePath}-${index}`;
        const file = fs.createWriteStream(filePath);
        file.on('pipe', (src) => console.log(`${filePath} started`));
        file.on('finish', (src) => console.log(`${filePath} downloaded`));
        https.get(href, function(response) {
            response.pipe(file);
        });
    });
    await browser.close();
})();

As expected, the output will be similar to the following:

my-file.avi-0 started
my-file.avi-1 started
my-file.avi-3 started
my-file.avi-2 started
my-file.avi-0 downloaded
my-file.avi-1 downloaded
my-file.avi-2 downloaded
my-file.avi-3 downloaded

Voilà! The NodeJS itself handles all the I/O concurrency.

Conclusion

Downloading a file using Playwright is smooth and a simple operation, especially with a straightforward and reliable API. Hopefully, my explanation will help you make your data extraction more effortless, and you'll be able to extend your web scraper with file downloading functionality.

I'd suggest further reading for the better Playwright API understanding:

Happy web scraping, and don't forget to change the fingerprint of your browser 🕵️

Turn Any Website Into An API with AutoScraper and FastAPI

Oleg Kulyk — Sat, 24 Apr 2021 09:38:35 +0000

In this article, we will learn how to create a simple e-commerce search API with multiple platform support: eBay and Amazon. AutoScraper and FastAPi provide the ability to create a powerful JSON API for the date. With Playwright's help, we'll extend our scraper and avoid blocking by using ScrapingAnt's web scraping API.

Let's satisfy the requirements

To complete the whole tutorial we'll need the following libraries:

FastAPI - to implement the API
Uvicorn - to serve the API
AutoScraper - to parse data into structured format
Playwright or ScrapingAnt API client - for blocking avoidance and web scraping at scale

To install all the requirements via pip just run the following command (Python 3.7+ required):

pip install fastapi 'uvicorn[standard]' autoscraper playwright scrapingant-client
playwright install

Not all the described libraries are required for a complete project, but I'd suggest installing them to try different ways of data extraction.

Scrape one page, and you'll scrape them all

Let's start from the most interesting part - creating a smart scraper to fetch data from Amazon's search result page. For a simple case, we'll use the title, price and product link as a required data for extraction. To generate a product link for Amazon's product we'll need to get a unique product id like B077XTPWZ5.

Using AutoScraper, it would be easily done by just providing some sample data:

from autoscraper import AutoScraper

url = 'https://www.amazon.com/s?k=baking+mat'

wanted_dict = {
    'title': [
        'Amazon Basics Silicone, Non-Stick, Food Safe Baking Mat - Pack of 2',
        'Amazon Basics Silicone, Non-Stick, Food Safe Baking Mat - Pack of 3',
    ],
    'price': ['$15.06', '$11.55'],
    'url': ['B0725GYNG6', 'B077XTPWZ5']
}

scraper = AutoScraper()
scraper.build(url=url, wanted_dict=wanted_dict)

# results grouped per rule to we'll know which one to use
scraper.get_result_similar(url, grouped=True)

The code above tells us how to apply several samples when the structure of items on search result pages is different, and we want the scraper to learn them all, which is redundant for the current case and added to show this cool feature.

If you want to copy and run this code, you may need to update the wanted_dict with your values.

From the output, we’ll know which rule corresponds to which data:

{
  "rule_iqkj": ["Amazon Basics Silicone, Non-Stick, Food Safe Baking Mat - Pack of 2", "Amazon Basics Silicone, Non-Stick, Food Safe Baking Mat - Pack of 3", "Amazon Basics Silicone, Non-Stick, Food Safe Baking Mat - Pack of 4", "Amazon Basics Silicone, Non-Stick, Food Safe Baking Mat, Macaron - Pack of 2", "Amazon Basics Anti-Fatigue Mat Marbleized Composite Mat 1/2' Thick 2X3 Black/Gray/White"], 
  "rule_0emm": ["Amazon Basics Silicone, Non-Stick, Food Safe Baking Mat - Pack of 2", "Amazon Basics Silicone, Non-Stick, Food Safe Baking Mat - Pack of 3", "Amazon Basics Silicone, Non-Stick, Food Safe Baking Mat - Pack of 4", "Amazon Basics Silicone, Non-Stick, Food Safe Baking Mat, Macaron - Pack of 2", "Amazon Basics Anti-Fatigue Mat Marbleized Composite Mat 1/2"], 
  "rule_91nj": ["Amazon Basics Silicone, Non-Stick, Food Safe Baking Mat - Pack of 2", "Amazon Basics Silicone, Non-Stick, Food Safe Baking Mat - Pack of 3", "Amazon Basics Silicone, Non-Stick, Food Safe Baking Mat - Pack of 4", "Amazon Basics Silicone, Non-Stick, Food Safe Baking Mat, Macaron - Pack of 2", "Amazon Basics Anti-Fatigue Mat Marbleized Composite Mat 1/2' Thick 2X3 Black/Gray/White"], 
  "rule_pkc7": ["Amazon Basics Silicone, Non-Stick, Food Safe Baking Mat - Pack of 2", "Amazon Basics Silicone, Non-Stick, Food Safe Baking Mat - Pack of 3", "Amazon Basics Silicone, Non-Stick, Food Safe Baking Mat - Pack of 4", "Amazon Basics Silicone, Non-Stick, Food Safe Baking Mat, Macaron - Pack of 2", "Amazon Basics Anti-Fatigue Mat Marbleized Composite Mat 1/2"], 
  "rule_vv2c": ["Amazon Basics Silicone, Non-Stick, Food Safe Baking Mat - Pack of 3", "Silpat Premium Non-Stick Silicone Baking Mat, Half Sheet Size, 11-5/8 x 16-1/2"], 
  "rule_o7dc": ["$15.06", "$11.55", "$14.99", "$25.87", "$12.05", "$13.99", "$45.49"], 
  "rule_y9sp": ["$15.06", "$11.55", "$14.99", "$25.87", "$12.05", "$13.99", "$45.49"], 
  "rule_4psi": ["$15.06", "$11.55", "$14.99", "$12.05", "$13.99", "$24.99", "$14.99", "$9.19", "$11.50", "$25.87", "$19.99", "$22.25", "$10.36", "$13.99", "$17.97", "$13.99", "$15.47", "$17.00", "$7.99", "$23.28", "$9.99", "$24.99", "$10.99", "$3.49", "$4.99", "$6.99", "$9.99", "$15.99", "$19.99", "$8.99", "$14.99", "$14.99", "$16.92", "$8.26", "$41.27", "$30.00", "$27.66", "$17.95", "$17.90"], 
  "rule_t1vk": ["$15.06", "$11.55", "$14.99", "$12.05", "$13.99", "$24.99", "$14.99", "$9.19", "$11.50", "$25.87", "$19.99", "$22.25", "$10.36", "$13.99", "$17.97", "$13.99", "$15.47", "$17.00", "$7.99", "$23.28", "$9.99", "$24.99", "$10.99", "$3.49", "$4.99", "$6.99", "$9.99", "$15.99", "$19.99", "$8.99", "$14.99", "$14.99", "$16.92", "$8.26", "$41.27", "$30.00", "$27.66", "$17.95", "$17.90"], 
  "rule_vjnv": ["B0725GYNG6", "B077XTPWZ5", "B07DZVRJBZ", "B077XR38W3", "B07Q1CJSWL"], 
  "rule_zho2": ["B0725GYNG6"], 
  "rule_3h0m": ["B0725GYNG6", "B077XTPWZ5", "B07DZVRJBZ", "B077XR38W3", "B07Q1CJSWL"], 
  "rule_h12x": ["B077XTPWZ5", "B00008T960"], 
  "rule_he5c": ["B077XTPWZ5"]
}

After analyzing the output, let’s keep only desired rules, remove the rest, and save our model:

scraper.keep_rules(['rule_iqkj', 'rule_7wqn', 'rule_vjnv', 'rule_y9sp'])

# Give it a file path
scraper.save('amazon-search')

Rule names will be re-generated each script run. Please, use REPL, Jupyter or save model without cleaning first. You'll be able to open and clean the model from the redundant rules later.

Let's do the same trick for eBay:

from autoscraper import AutoScraper

url = 'https://www.ebay.com/sch/i.html?_nkw=baking+mat'
wanted_list = ['Durable Silicone Baking Mat Non-Stick Pastry Cookie Sheet Oven Baking Mat Liner', '$2.15', 'https://www.ebay.com/itm/Durable-Silicone-Baking-Mat-Non-Stick-Pastry-Cookie-Sheet-Oven-Baking-Mat-Liner/202362695368?hash=item2f1dc1aec8:g:S3UAAOSwwpdW62sK']

scraper = AutoScraper()
scraper.build(url=url, wanted_list=wanted_list)

scraper.get_result_similar(url, grouped=True)

... and the execution hangs.

What is the reason? Why it not worked?

AutoScraper is a good tool for automatic intelligent parsing. Still, it does not have capabilities for proper data extraction from the page, so the URL's direct data extraction can not work for some sites.

Let's try the workaround for the learning phase only: save the required page as HTML via the browser and then open via Python to pass the data into AutoScraper:

from autoscraper import AutoScraper
from pathlib import Path

# page content from https://www.ebay.com/sch/i.html?_nkw=baking+mat saved from the browser
html = Path('baking_mat_ebay.html').read_text()

wanted_list = ['Durable Silicone Baking Mat Non-Stick Pastry Cookie Sheet Oven Baking Mat Liner', '$2.14', 'https://www.ebay.com/itm/Durable-Silicone-Baking-Mat-Non-Stick-Pastry-Cookie-Sheet-Oven-Baking-Mat-Liner/202362695368?hash=item2f1dc1aec8:g:S3UAAOSwwpdW62sK']

scraper = AutoScraper()
scraper.build(html=html, wanted_list=wanted_list)

scraper.get_result_similar(html=html, grouped=True)

scraper.set_rule_aliases({'rule_9otk': 'title', 'rule_7n5a': 'price', 'rule_buz1': 'url'})

// No rules will be cleaned this time, all of them works fine

# Give it a file path
scraper.save('ebay-search')

How to extract data and avoid anti-scraping techniques? Let's try out two different approaches:

Playwright

Playwright is a high-level API over headless Chrome, Firefox and Webkit browser. It means that you can automate the action that we made above (open the page's HTML), and this action will not trigger the website's anti-scraping mechanism. Let's create a function that does content extraction:

from playwright.sync_api import sync_playwright

def get_webpage_content_playwright(url):
    with sync_playwright() as p:
        browser = p.chromium.launch()
        page = browser.new_page()
        page.goto(url)
        content = page.content()
        browser.close()
        return content

This function allows us to get the page's HTML content without being blocked and detected as a bot.

Also, we'll need to apply rotating proxies to avoid rate-limiting for the web scraping at scale. Check out how to use a proxy in Playwright.

Let's check out the pros and cons of using Playwright for web scraping:

Pros:

On-premise or cloud solution with no third party API
Full browser control
Customisable proxy pool
Community support

Cons:

Low scalability
High proxy prices
Complicated maintenance
High expenses for infrastructure to run CPU intense task

ScrapingAnt API

ScrapingAnt web scraping API is a web interface for performing data extraction at a scale that already uses headless browsers and rotating proxies inside. Only one action is needed - execute an API request with a target URL. Let's create the same function with using ScrapingAnt API:

from scrapingant_client import ScrapingAntClient

client = ScrapingAntClient(token='<YOUR-SCRAPINGANT-API-TOKEN>')

def get_webpage_content_scrapingant(url):
    return client.general_request(url).content

To get you API token, please, visit Login page to authorize in ScrapingAnt User panel. It's free.

Pros:

Simple usage
Always reproducible results
Low maintenance and usage price
Large proxy pool
High data extraction success rate

Cons:

Third-party API usage
Paid when scraping web pages at scale

Learning phase notes

Model creation should be implemented in exactly the same way you're going to use for other pages scraping. If you're planning to use AutoScraper with url param - you should also create the model with passing url param. The different browsers may create different layouts, so a Playwright or a desktop Chrome model can not parse HTML retrieved with a direct URL propagation or content retrieved via ScrapingAnt web scraping API.

To save HTML for learning, you can use the following snippets:

Playwright

from pathlib import Path
from playwright.sync_api import sync_playwright

url = 'https://www.ebay.com/sch/i.html?_nkw=baking+mat'

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto(url)
    Path('baking_mat_ebay.html').write_text(page.content())
    browser.close()

ScrapingAnt

from pathlib import Path
from scrapingant_client import ScrapingAntClient

url = 'https://www.ebay.com/sch/i.html?_nkw=baking+mat'
client = ScrapingAntClient(token='<YOUR-SCRAPINGANT-API-TOKEN>')
html = client.general_request(url=url).content

Path('baking_mat_ebay.html').write_text(html)

Create your own API

It's time to glue-up everything together and create our e-commerce API based on Amazon and eBay. The final code contains both data extraction functions, so you'll be able to modify and use your favorite:

from autoscraper import AutoScraper
from fastapi import FastAPI
from scrapingant_client import ScrapingAntClient
from typing import Optional
from playwright.sync_api import sync_playwright

ebay_scraper = AutoScraper()
amazon_scraper = AutoScraper()
ebay_scraper.load('ebay-search')
amazon_scraper.load('amazon-search')

client = ScrapingAntClient(token='<YOUR-SCRAPINGANT-API-TOKEN>')

app = FastAPI()


def get_ebay_result(search_query):
    url = 'https://www.ebay.com/sch/i.html?_nkw=%s' % search_query
    html = get_webpage_content_scrapingant(url)
    result = ebay_scraper.get_result_similar(html=html, group_by_alias=True)
    return _aggregate_result(result)


def get_amazon_result(search_query):
    url = 'https://www.amazon.com/s?k=%s' % search_query
    html = get_webpage_content_scrapingant(url)
    result = amazon_scraper.get_result_similar(html=html, group_by_alias=True)
    result['url'] = [f'https://www.amazon.com/dp/{i}' for i in result['url']]
    return _aggregate_result(result)


def get_webpage_content_scrapingant(url):
    return client.general_request(url).content


def get_webpage_content_playwright(url):
    with sync_playwright() as p:
        browser = p.chromium.launch()
        page = browser.new_page()
        page.goto(url)
        content = page.content()
        browser.close()
        return content


def _aggregate_result(result):
    """
    :param result {
        'title': ['name1', 'name2']
        'url': ['url1', 'url2']
        'price': ['price1', 'price2']
    }

    :returns [
        {'title': 'name1', 'url': 'url1', 'price': 'price1'},
        {'title': 'name2', 'url': 'url2', 'price': 'price2'},
    ]
    """
    final_result = []
    items_count = len(result['title'])
    for i in range(items_count):
        final_result.append({alias: result[alias][i] for alias in result})
    return final_result

@app.get("/")
def search_api(q: Optional[str] = None):
    return dict(result=get_ebay_result(q) + get_amazon_result(q))

Execute the following command to run the API server:

uvicorn main:app --reload

By running this code, the API server will be up listening on port 8080. So let’s test our API by opening http://127.0.0.1:8080/?q=baking in our browser:

And finally, we're having our own custom multi-sourced e-commerce API. Just replace baking in the URL with your desired search query to get its results.

Conclusion

This tutorial is intended for personal and educational use. I hope this article is useful and helps to grab all the provided code snippets to a fully functional and production-grade application.

I'd recommend following the links below to extend your knowledge and have a deep dive into the project parts:

Happy web scraping, and don't forget to check websites policies regarding the scraping bots 😉

How to use a proxy in Playwright

Oleg Kulyk — Mon, 08 Mar 2021 18:36:58 +0000

Playwright is a high-level API to control and automate headless Chrome (Chromium), Firefox, and Webkit. It can be considered as an extended Puppeteer, as it allows using more browser types to automate modern web apps testing and scraping. Playwright API can be used in JavaScript & TypeScript, Python, C# and, Java. In this article, we are going to show how to set up a proxy in Playwright for all the supported browsers.

Configuring proxy in Playwright

Playwright can be considered as Puppeteer's successor with a similar API, so many developers prefer to use it for a single page applications data extraction and anti-scraping avoidance while automating their data mining tasks. On the other hand, it has a different way to set up a proxy parameters than Puppeteer. Before the Jun 2020, it was a huge problem to make proxy works across all the browsers, but, luckily, the API been unified to pass proxy options via a browser's launch method. Let's try it out for all the browsers:

Launch `proxy` option

It's possible to pass proper proxy settings inside proxy property in options object for browserType.launch method:

const playwright = require('playwright');

const launchOptions = {
    proxy: {
        server: '222.165.235.2:80'
    }
};

(async () => {
  for (const browserType of ['chromium', 'firefox', 'webkit']) {
    const browser = await playwright[browserType].launch(launchOptions);
    const context = await browser.newContext();
    const page = await context.newPage();
    await page.goto('https://httpbin.org/ip');
    console.log(await page.textContent("*"));
    await browser.close();
  }
})();

As a result, you'll observe a similar output:

{
  "origin": "222.165.235.2"
}

{
  "origin": "222.165.235.2"
}

{
  "origin": "222.165.235.2"
}

As you can observe, all the browsers have different ways to pass proxy settings. For example, Firefox requires passing profile configuration file to set up browser proxy.

Command line arguments (only for Chromium)

It's also possible to pass proxy settings via command-line arguments as we do it with Puppeteer. Below you can find the example for Chromium proxy options:

const playwright = require('playwright');
const launchOptions = {
    args: [ '--proxy-server=http://222.165.235.2:80' ]
};

(async () => {
  for (const browserType of ['chromium']) {
    const browser = await playwright[browserType].launch(launchOptions);
    const context = await browser.newContext();
    const page = await context.newPage();
    await page.goto('https://httpbin.org/ip');
    console.log(await page.textContent("*"));
    await browser.close();
  }
})();

Other browsers also allow you to set up proxy parameters in their native way, but the behavior may differ between operating systems and browser versions.

How to specify proxy settings for a separate page or request

By using the methods above you'll be able to set up proxy settings for the whole browser session, not for request or the page. In our previous article we have shared info about setting up your own rotation proxy server and separating each request with using it.

Reducing the complexity

In order to simplify your web scraper and have more time for data mining tasks itself, you might want to get rid of the infrastructure hell and just focus on what you really want to achieve (extract the data).

ScrapingAnt API provides the ability to scrape the target page with only one API call. All the proxies rotation and cloud headless Chrome rendering already handled by the API side. You can check out how simple it is with the ScrapingAnt Javascript client:

const ScrapingAntClient = require('@scrapingant/scrapingant-client');

const client = new ScrapingAntClient({ apiKey: '<YOUR-SCRAPINGANT-API-KEY>' });

// Check the proxy address
client.scrape('https://httpbin.org/ip')
    .then(res => console.log(res))
    .catch(err => console.error(err.message));

With ScrapingAnt API, you can forget about headless browsers infrastructure and maintenance. You can use it for free, follow here to sign in and get your API token.

How to use rotating proxies with Puppeteer

Oleg Kulyk — Sun, 07 Mar 2021 11:52:00 +0000

Puppeteer is a high-level API to control headless Chrome. Most things that you can do manually in the browser can be done using Puppeteer, so it quickly became one of the most popular web scraping tool in Node.js and Python. Many developers use it for single-page applications (SPA) data extraction as it allows executing client-side Javascript. In this article, we are going to show how to set up a proxy in Puppeteer and how to spin up your own rotating proxy server.

Configuring proxy in Puppeteer

For requesting the target site via a proxy server we just should specify the --proxy-server launch parameter with a proper proxy address. For example, http://10.10.10.10:8080

const puppeteer = require('puppeteer');

(async() => {

  const browser = await puppeteer.launch({
     args: [ '--proxy-server=http://10.10.10.10:8080' ]
  });

  const page = await browser.newPage();
  await page.goto('https://httpbin.org/ip');
  await browser.close();
})();

As a result, httpbin should respond with a JSON, that contains the exact proxy server address, so the code above can be used for further proxy IP address testing:

{
  "origin": "10.10.10.10"
}

Pretty simple, isn't it? The only one downside of this approach, that the defined proxy server will be used for all the requests from the browser start, and for changing the proxy server the browser should be relaunched by puppeteer.launch with a new proxy IP address.

Rotate proxy servers by your own

To avoid ban while web scraping you need to use different proxies and rotate them. In case of implementing your custom IP pool you'll need to re-launch your headless Chrome each time with a new proxy server settings. How to implement proxy rotation by each browser request?

The answer is pretty simple - you may intercept each request with your own proxy rotation tool! That kind of tool will handle proxy rotation for the browser, and you'll be able to save the precious time while web scraping.

To spin up proxy rotation server you may use the handy library proxy-chain and ScrapingAnt free proxies list:

const proxies = {
  'session_1': 'http://185.126.200.167:3128',
  'session_2': 'http://116.228.227.211:443',
  'session_3': 'http://185.126.200.152:3128',
};

const server = new ProxyChain.Server({
  port: 8080,
  prepareRequestFunction: ({ request }) => {
      // At this point of code we should decide what proxy
      // to use from the proxies list.
      // You can chain your browser requests by header 'session-id'
      // or just pick a random proxy from the list
      const sessionId = request.headers['session-id'];
      const proxy = proxies[sessionId];
      return { upstreamProxyUrl: proxy };
  }
});

server.listen(() => console.log('Rotating proxy server started.'));

The only disadvantage of this method is that you have to handle a bigger codebase and have a deep dive into networking, proxy management, and maintenance.

One API call solution

In order to simplify the web scraper and have more space while scraping at scale, you might want to get rid of the infrastructure pain and just focus on what you really want to achieve (extract the data).

ScrapingAnt API provides the ability to scrape the target page with only one API call. All the proxies rotation and headless Chrome rendering already handled by the API side. You can check out how simple it is with the ScrapingAnt Javascript client:

const ScrapingAntClient = require('@scrapingant/scrapingant-client');

const client = new ScrapingAntClient({ apiKey: '<YOUR-SCRAPINGANT-API-KEY>' });

// Check the proxy rotation
client.scrape('https://httpbin.org/ip')
    .then(res => console.log(res))
    .catch(err => console.error(err.message));

Or with a plain Javascript request to API (a bit more boilerplate code):

var http = require("https");

var options = {
   "method": "POST",
   "hostname": "api.scrapingant.com",
   "port": null,
   "path": "/v1/general",
   "headers": {
       "x-api-key": "<YOUR-SCRAPINGANT-API-KEY>",
       "content-type": "application/json",
       "accept": "application/json",
       "useQueryString": true
   }
};

var req = http.request(options, function (res) {
   var chunks = [];

   res.on("data", function (chunk) {
       chunks.push(chunk);
   });

   res.on("end", function () {
       var body = Buffer.concat(chunks);
       console.log(body.toString());
   });
});

req.write(JSON.stringify({
    url: 'https://httpbin.org/ip',
}));
req.end();

With ScrapingAnt API, you can forget about any complications with IP rotation, and the internal anti-scraping avoiding mechanisms will help you to not be detected by Cloudflare. You can use it for free, follow here to sign in and get your API token.

Forem: Oleg Kulyk

Robots are not that bad

Proper AI data extraction without manual HTML parsing

1. Improper ChatGPT AI Model Usage

2. Input size

What's the cost?

AI extractor from ScrapingAnt

Leverage the Power of ChatGPT API and web scraping API to Parse Web Pages Efficiently

Introduction

ChatGPT data extraction

1. Setting up the ChatGPT API

2. Setting up the ScrapingAnt API

3. Parsing Web Pages with ChatGPT API and ScrapingAnt API

Conclusion

How to make POST, PUT and DELETE requests using Puppeteer?

Why do we need POST while web scraping?

Interception of the initial request

Puppeteer's extension

Conclusion

How to download images with NodeJS?

Download an image using http.request

The modern way - Axois download image and any file

Be specific - use a separate NodeJS download module

Conclusion

Block resources with Playwright

Why block resources

Intercept requests with Playwright

Request interception

Route interception

Improve SPA page scraping speed (Vue.js, React.js, etc.)

Summary

How to submit a form with Playwright?

Automating form submission

Submitting a form with attachments

Submitting a form using Javascript

Conclusion

How to download a file with Playwright?

Downloading a file after the button click

Note

Direct file download

Downloading multiple files in parallel

Conclusion

Turn Any Website Into An API with AutoScraper and FastAPI

Let's satisfy the requirements

Scrape one page, and you'll scrape them all

Playwright

ScrapingAnt API

Learning phase notes

Playwright

ScrapingAnt

Create your own API

Conclusion

How to use a proxy in Playwright

Configuring proxy in Playwright

Launch proxy option

Command line arguments (only for Chromium)

How to specify proxy settings for a separate page or request

Reducing the complexity

How to use rotating proxies with Puppeteer

Configuring proxy in Puppeteer

Rotate proxy servers by your own

One API call solution

Download an image using `http.request`

Launch `proxy` option