Forem: Extract by Zyte

Build Scrapy spiders in 23.54 seconds with this free Claude skill

John Rooney — Mon, 30 Mar 2026 17:50:39 +0000

I built a Claude skill that generates Scrapy spiders in under 30 seconds — ready to run, ready to extract good data. In this post I'll walk through what I built, the design decisions behind it, and where I think it can go next.

What it does

The skill takes a single input: a category or product listing URL. From there, Claude generates a complete, runnable Scrapy spider as a single Python script. No project setup, no configuration files, no boilerplate to write. Just a script you can run immediately.

Here's what that looks like in practice. I opened Claude Code in an empty folder with dependencies installed, activated the skill, and said: "Create a spider for this site" — and pasted a URL.

Within seconds, the script was generated. I ran it, watched the products roll in, piped the output through jq, and had clean structured product data. Start to finish: under a minute.

Why a single-file script, not a full Scrapy project?

Scrapy is usually a full project — multiple files, lots of moving parts, a proper setup process. Running it from a script instead is generally discouraged for production work, but for this use case it's actually the right call.

The goal here is what I'd call pump-and-dump scraping: give Claude a URL, get a spider, run it for a couple of days, move on. It's not designed to scrape millions of products every day for years. For that kind of scale you need proper infrastructure, robust monitoring, and serious logging. This isn't that — and that's intentional.

What you do get, even in the single-file approach, is almost everything Scrapy offers: middleware, automatic retries, and concurrency handling. You'd have to build all of that yourself with a plain requests script. Scrapy gives it to you for free, even when running from a script.

The key design decision: AI extraction

The other major call I made was to lean entirely on Zyte API's AI extraction rather than generating CSS or XPath selectors.

Specifically, the skill uses two extraction types chained together: productNavigation on the category or listing page, which returns product URLs and the next page link, and product on each product URL, which returns structured product data including name, price, availability, brand, SKU (stock keeping unit), description, images, and more.

This means the spider doesn't need to know anything about the structure of the site it's crawling. There are no selectors to generate, no schema to define, no user confirmation step. The AI on Zyte's end handles all of that. It does cost slightly more than a raw HTTP request, but given how little time it takes to go from URL to working spider, the trade-off makes sense.

I've hardcoded httpResponseBody as the extraction source — it's faster and more cost-efficient than browser rendering. If a site is JavaScript-heavy and you're not getting the data you need, you can switch to browserHtml with a one-line change. The spider logs a warning to remind you of this.

The use case is deliberately narrow

This skill is designed for e-commerce sites, and only e-commerce sites. That's not a limitation I stumbled into — it's a feature.

Because the scope is narrow, the spider structure is simple and predictable: category pages with pagination, product links, and detail pages. Zyte API's productNavigation and product extraction types handle this reliably. Widening the scope to arbitrary crawling would require a lot more of Scrapy's machinery and would quickly exceed what makes sense for a lightweight script like this.

What it doesn't do: deep subcategory crawling, link discovery, or full-site crawls. If a page renders all its products without pagination, that works fine — the next page just returns nothing.

Logging and output

I replaced Scrapy's default logging with Rich logging, which gives cleaner terminal output. Scrapy's logs are verbose in ways that aren't useful when you're running a short-lived script — I wanted something concise enough that if something went wrong, it would be obvious at a glance.

Output goes to a .jsonl file named after the spider, alongside a plain .log file. Both are derived from the spider name, which is itself derived from the domain. Run example_com.py, get example_com.jsonl and example_com.log.

Where this goes next

The immediate next step I have in mind is selector-based extraction as an alternative path — useful for sites where the AI extraction isn't quite right, or where you want more control over exactly what gets pulled.

The longer-term vision is running this fully agentically. URLs get submitted somewhere — a queue, a database table, a form — an agent picks them up, builds the spider, and maybe runs a quick validation. The spider then goes into a pool to be run on a schedule, and data lands in a database rather than a flat file. Give Claude access to a virtual private server (VPS) via terminal and most of this is achievable without much extra infrastructure. The skill is already the hard part.

Download the skill

The skill is free to download and use. It's a single .skill file you can install directly into Claude Code. You'll need:

pip install scrapy scrapy-zyte-api rich
export ZYTE_API_KEY=your_key_here

Scrapy 2.13 or above is required for AsyncCrawlerProcess.

The link to the repo and the skill download are in the video description, and here. If you've built something similar, or have thoughts on the design decisions — especially around the extraction approach or the logging setup — I'd love to hear it in the comments. GitHub links to your own scrapers are very welcome too.

If you're interested in more agentic scraping patterns, I also built a Claude skill that helps spiders recover from excessive bans — you can watch that video here.

I Built a Self-Healing Web Scraper to Auto-Solve 403s

John Rooney — Mon, 16 Mar 2026 11:09:31 +0000

Web scraping has a recurring enemy: the 403. Sites add bot detection, anti-scraping tools update their challenges, and scrapers that worked fine last week start silently failing. The usual fix is manual — check the logs, diagnose the cause, update the config, redeploy. I wanted to see if an agent could handle that loop instead.

So I built a self-healing scraper. After each crawl, a Claude-powered agent reads the failure logs, probes the broken domains with escalating fetch strategies, and rewrites the config automatically. By the next run, it's already fixed itself.

How it works

The project has two parts: a scraper and a self-healing agent.

The scraper

main.py is a straightforward Python scraper driven entirely by a config.json file. Each domain entry tells the scraper which URLs to fetch and how to fetch them:

{
  "id": "books",
  "zyte": true,
  "browser_html": false,
  "urls": ["https://www.bookstocsrape.co.uk/products/..."]
}

There are three fetch modes:

Direct — a plain requests.get(). Fast, free, works for sites that don't block bots.
Zyte API (httpResponseBody) — routes the request through Zyte's residential proxy network. Good for sites that block datacenter IPs.
Zyte API (browserHtml) — spins up a real browser via Zyte, executes JavaScript, and returns the fully-rendered DOM. Required for sites using JS-based bot challenges.

Every request is logged to scraper.log in the same format:

2026-03-14 09:12:01 url=https://... domain_id=scan status=200

If a request throws any exception, it's recorded as a 403. That keeps the log clean and gives the agent a consistent signal to act on.

The self-healing agent

agent.py is a Claude-powered agent that runs after each crawl. It uses the Claude Agent SDK and has access to three tools: Read, Bash, and Edit — enough to operate completely autonomously.

The agent works through a staged process:

Read the log — finds every domain that returned a 403
Cross-reference the config — skips domains already configured to use Zyte
Stage 1 probe — uses the zyte-api CLI to fetch one URL per failing domain with httpResponseBody, then inspects the page <title>
Challenge detection — if the title contains phrases like "Just a moment", "Checking your browser", or "Verifying you are human", the page is flagged as a bot challenge
Stage 2 probe — challenge pages are re-probed using browserHtml, which runs a real browser to bypass JS-based detection
Config update — the agent edits config.json directly, setting zyte: true and/or browserHtml: true for domains that now work

The next crawl automatically uses the right fetch strategy. No manual intervention needed.

Design decisions

Config-driven, not code-driven

Everything lives in config.json. Adding a new domain is a one-liner, and the scraper doesn't need to know anything about individual sites — it just reads the config and follows instructions. The agent writes to the same file, so the loop closes itself naturally.

Graduated fetch strategy

Not every site needs an expensive browser render. By escalating from direct to httpResponseBody to [browserHtml](https://www.zyte.com/zyte-api/headless-browser/) only when necessary, I keep costs manageable. Browser renders are slower and consume more API credits — reserving them for sites that actually need them makes a meaningful difference at scale.

Letting the agent handle the heuristics

The challenge detection logic — matching titles against known bot-detection phrases — is exactly the kind of fuzzy heuristic that's tedious to maintain as code but natural for a language model to reason about. Claude also handles edge cases gracefully: if the zyte-api CLI isn't installed, if the log is empty, if a domain is already correctly configured. A rule-based script would need explicit handling for every one of those scenarios.

The limitations

It's worth being honest about where this approach falls short.

It's reactive, not proactive. The agent only runs after a failed crawl. If a site starts blocking mid-run, those URLs fail silently until the next cycle.

Title-based detection is fragile. Most bot-challenge pages say "Just a moment…" — but a legitimate site could theoretically use that phrase. A false positive would cause the scraper to wastefully use browser rendering where it isn't needed.

One URL per domain. The agent probes only the first failing URL for each domain. Different URL patterns on the same domain can have different bot-detection behaviour, which this doesn't account for.

No rollback. Once the config is updated, there's no way to detect if a Zyte setting later stops working and revert it automatically.

Cost opacity. The scraper logs HTTP status codes, not Zyte API credit consumption. There's no visibility into what each domain actually costs to fetch.

Where I'd take it next

Smarter challenge detection. Rather than keyword-matching on the title, the agent could read the full page HTML and make a more nuanced call — is this a product page, a login wall, or a soft block with a CAPTCHA? Each requires a different response.

Proactive monitoring. A lightweight probe running daily against each configured domain, independent of the main crawl, would let the agent update the config before a full scrape run hits a known-bad configuration.

Per-URL config. Right now zyte and browser_html are set at the domain level. Some sites serve static product pages on one path and JS-rendered category pages on another — granular per-URL settings would handle that cleanly.

Structured data extraction. Right now parse_page only pulls the page title. The natural next step is structured product extraction — price, availability, name, images — either via CSS selectors in the config or Zyte's product extraction type, which uses ML models to parse product data from any page.

Multi-agent parallelism. The self-healing loop is currently a single agent. As the config grows, a coordinator could spawn one subagent per failing domain, each running its own probe pipeline concurrently. The Claude Agent SDK supports subagents natively, so this would be a relatively small change.

The core idea is simple: a scraper that observes its own failures and reconfigures itself. What I found interesting about building it wasn't the scraping itself — it was seeing how little scaffolding the agent actually needed. Three tools, a clear task, and it handles the diagnostic work that would otherwise fall to me.

How I get Claude to build HTML parsing code the way I want it

John Rooney — Wed, 11 Mar 2026 07:13:06 +0000

Getting HTML off a page is only the first step. Once you have it, the real work begins: pulling out the data that actually matters — product names, prices, ratings, specifications — in a clean, structured format you can actually do something with.

That's what the parser skill is for. If you haven't read the introduction to skills in our fetcher post, it's worth a quick look first. But the short version is this: a skill is a SKILL.md file that gives Claude precise, reusable instructions for using a specific tool. The parser skill is one of three that together form a complete web scraping pipeline.

zytelabs / claude-webscraping-skills

claude-webscraping-skills

A collection of claude skills and other tools to assist your web-scraping needs.

video explanations:

https://youtu.be/HH0Q9OfKLu0 https://youtu.be/P2HhnFRXm-I

Other claude tools for web scraping

zyte-fetch-page-content-mcp-server

A Model Context Protocol (MCP) server that runs locally using docker desktop mcp-toolkit and help you extracts clean, LLM-friendly content from any webpage using the Zyte API. Perfect for AI assistants that need to read and understand web content. by Ayan Pahwa

Improve Claude Code WebFetch with Zyte API

When Claude encounters a WebFetch failure, it reads the CLAUDE.md instructions and makes a curl request to the Zyte API endpoint. The API returns base64-encoded HTML, which Claude decodes and processes just like it would with a normal WebFetch response. by Joshua Odmark

Claude skills vs MCP vs Web Scraping CoPilot

View on GitHub

What is a skill?

A skill is a small markdown file that tells Claude how to use a specific script or tool — what it does, when to use it, and step-by-step how to run it. Claude reads the file and follows the instructions as part of a broader workflow, with no manual intervention required.

Skills are composable by design. The fetcher skill hands raw HTML to the parser skill, which hands structured JSON to the compare skill. Each one does one job well, and they're built to work together.

The parser skill's front matter sets out its purpose immediately:

---
name: parser
description: "Extracts structured product data from raw HTML. Tries JSON-LD via"
Extruct first, falls back to CSS selectors via Parsel.
---

Two methods, one fallback. That single description line captures the entire logic of the skill.

What the parser skill does

The parser skill takes raw HTML as input and returns a structured JSON object. It uses two extraction methods in sequence, trying the more reliable one first and falling back to the more flexible one if needed.

The primary method uses Extruct to find JSON-LD data embedded in the page. JSON-LD is a structured data format that many modern sites include in their HTML specifically to make their content machine-readable — it's used for search engine optimisation and data portability. When it's present, Extruct can read it cleanly and reliably, with no need to write or maintain selectors.

If no usable JSON-LD is found, the skill falls back to Parsel, which uses CSS selectors to locate data heuristically across the page. This is more flexible but inherently tied to the page's visual structure, which can change.

When to use it

## When to use
Use this skill when you have raw HTML and need to extract structured data from
it — product details, prices, specs, ratings, or any page content.

In practice, that means the parser skill is almost always the second step in a pipeline — running immediately after the fetcher skill has retrieved your HTML. It works with any page type, and handles the most common product fields out of the box.

How it works

## Instructions
1. Save the HTML to a temporary file `page.html`
2. Run `parser.py` against it:
   python parser.py page.html

3. The script outputs a JSON object. Check the `method` field:
   - "extruct" — clean structured data was found, use it directly
   - "parsel" — fell back to CSS selectors, review fields for completeness
4. If key fields are missing from the Parsel output, ask the user which fields
   they need and re-run with --fields:
   python parser.py page.html --fields "price,rating,brand"

5. Return the parsed JSON to the conversation for use in the Compare skill.

The method field in the output is particularly useful. It tells you immediately how the data was extracted and how much trust to place in it. An "extruct" result is clean and stable. A "parsel" result is worth reviewing, especially if you're working with an unusual page layout.

The --fields flag is a practical escape hatch. Rather than requiring you to dig into the script when key data is missing, it lets you specify exactly what you need and re-run — a much more efficient loop.

Why prefer Extruct?

The notes section of the skill file makes this explicit:

## Notes
- Always prefer the Extruct path — it is more stable and requires no maintenance
- Parsel selectors are generated heuristically and may need adjustment for
  unusual page layouts
- Run once per page; pass all outputs together into the Compare skill

Parsel selectors break when sites redesign. JSON-LD, by contrast, is structured data the site publishes independently of its visual layout. A site can completely overhaul its design and its JSON-LD will often remain untouched. That stability is worth prioritising wherever possible.

What comes next

Once you've run the parser skill across all your target pages, you have a set of structured JSON objects ready to compare. That's where the compare skill picks up — generating tables, summaries, and side-by-side analysis from the extracted data.

Do you need a skill?

The parser skill works well when the data you need maps cleanly onto fields that Extruct or Parsel can find — product names, prices, ratings, and similar structured attributes that sites commonly expose through JSON-LD or consistent HTML patterns. For that category of task, the skill is fast to apply and requires no custom code.

See our post about Skills vs MCP vs Web Scraping Copilot (our VS Code Extension):

zyte.com

But not every extraction problem fits that mould. If you're working with pages that don't include JSON-LD and have highly irregular layouts, Parsel's heuristic selectors may return incomplete or inconsistent results, and you'll spend time debugging field by field. In those cases, a purpose-built extraction script using Parsel or BeautifulSoup directly — with selectors you've written and tested against the specific target — will be more reliable.

For larger-scale or more complex extraction work, Zyte API's automatic extraction capabilities go further still. Rather than relying on selectors at all, automatic extraction uses AI to identify and return structured data from a page without requiring you to specify fields or maintain selector logic. If you're extracting data from many different site structures, or you need extraction to keep working through site redesigns without manual intervention, that's a more robust foundation than a skill-based approach. The parser skill is best understood as a practical middle ground: fast to use, good enough for a wide range of common cases, and easy to slot into a pipeline — but not a replacement for extraction tooling built for scale or resilience.

I gave Claude access to a web scraping API

John Rooney — Wed, 11 Mar 2026 07:10:43 +0000

If you've worked with Claude for any length of time, you've probably noticed it can do a lot more than answer questions. With the right setup, it can take actions — running scripts, processing files, working through multi-step workflows autonomously. Skills are what make that possible.

What is a skill?

A skill is a small, self-contained instruction set that tells Claude how to use a specific tool or script to accomplish a well-defined task. Technically, it's a markdown file — a SKILL.md — that describes what a tool does, when to reach for it, and exactly how to run it. Claude reads that file and follows the instructions as part of a larger workflow.

Skills are designed to be composable. Each one does one thing well, and they're built to hand off to each other. The fetcher skill retrieves HTML. The parser skill extracts data from it. The compare skill turns multiple parsed outputs into a structured comparison. Together, they form a complete scraping pipeline — and Claude orchestrates the whole thing.

See our skills here: https://github.com/zytelabs/claude-webscraping-skills

The skill format looks like this:

---
name: fetcher
description: "Fetches raw HTML from a URL using httpx, with automatic fallback to Zyte API if blocked."
---

That front matter is what Claude uses to match the right skill to the right task. The description is deliberately precise: it tells Claude not just what the skill does, but how it does it, so Claude can reason about whether it's the right tool for the job.

What the fetcher skill does

The fetcher skill's job is exactly what it sounds like: given a URL, fetch the raw HTML and return it. It uses httpx as its primary HTTP client — a modern, performant Python library well suited to scraping workloads.

What makes it more than a simple wrapper is the fallback logic. A significant number of sites actively block automated requests. Without a fallback, a blocked request just fails, and you're left manually diagnosing why. The fetcher skill handles this automatically. If a request comes back with a BLOCKED status, it retries via Zyte API, which provides built-in unblocking. Most of the time, you get your HTML without ever needing to intervene.

When to use it

The skill's SKILL.md is explicit about this:

## When to use
Use this skill when the user provides one or more URLs and asks you to fetch,
retrieve, scrape, or get the HTML or page content.

In practice, that means any time you're starting a scraping or data extraction task and you have a URL to work from. It's the entry point for the pipeline.

How it works

The instructions in the skill file are straightforward:

## Instructions
1. Run `fetcher.py` with the URL as an argument:
   python fetcher.py <url>

2. If the script returns a successful HTML response, return the HTML to the
   conversation for use in the next step.
3. If the script returns a `BLOCKED` status, re-run with the `--zyte` flag:
   python fetcher.py <url> --zyte

4. Inform the user if a URL could not be fetched after both attempts.

The two-step process keeps things efficient. httpx is fast and lightweight, so it handles the majority of requests without needing to route through Zyte API. The fallback only kicks in when it's needed. If both attempts fail, Claude surfaces that to you clearly rather than silently moving on.

For multiple URLs, the script runs once per URL — there's no batching — so Claude loops through a list sequentially.

Transparency about failure

One detail worth highlighting is the final instruction: inform the user if a URL could not be fetched after both attempts. This might seem obvious, but it reflects a design principle worth being explicit about. A skill that silently drops failed URLs would produce incomplete data downstream, and you might not notice until you're looking at a comparison table with missing rows. Surfacing failures immediately keeps the pipeline honest.

What comes next

The fetcher skill's output is raw HTML — exactly what the parser skill expects as its input. The two are designed to be used in sequence. Once you have the HTML, the parser skill takes over, extracting structured data through JSON-LD or CSS selectors depending on what the page contains.

That handoff is documented in the skill's notes:

## Notes
- For multiple URLs, run the script once per URL
- Pass the raw HTML output into the Parser skill for extraction

The pipeline continues from there.

Do you need a skill?

Skills are a good fit when you have a well-defined, repeatable task that benefits from consistent behaviour across many runs. Fetching HTML from a URL is a clear example: the inputs and outputs are predictable, the fallback logic is always the same, and packaging that into a skill means Claude applies it reliably without you having to re-explain the process each time.

Read our break down of Skills vs MCP vs Web Scraping Copilot here - our VS Code extension

That said, skills aren't always the right tool. If you only need to fetch a handful of pages once, asking Claude to write a quick httpx script directly may be faster and more flexible. Similarly, if your target sites have unusual behaviour — rate limiting, JavaScript rendering, login walls, or multi-step navigation — a bespoke Scrapy spider built with Zyte API gives you far more control than a general-purpose fetch wrapper. Scrapy's middleware architecture, item pipelines, and scheduling make it better suited to large-scale or complex crawls where you need precise control over every aspect of the request cycle.

The fetcher skill sits in the middle: more structured than an ad hoc script, less complex than a full Scrapy project. It's the right choice when you want Claude to handle straightforward retrieval as part of a larger automated workflow, without the overhead of setting up and maintaining a dedicated spider.

I built a Claude Code skill that screenshots any website (and it handles anti-bot sites too)

Ayan Pahwa — Fri, 06 Mar 2026 14:40:32 +0000

TLDR;

Automate screenshot capture for any URL with JavaScript rendering and anti-ban protection — straight from your AI assistant.

Taking a screenshot of a webpage sounds trivial, until you need to do it at scale. Modern websites throw every obstacle imaginable in your way: JavaScript-rendered content that only appears after a React bundle loads, bot-detection systems that serve blank pages to automated headless browsers, geo-blocked content, and CAPTCHAs that appear the moment traffic patterns look non-human. For a handful of URLs you can get away with Puppeteer or Playwright. For hundreds or thousands? You need infrastructure built for the job.

The Zyte API was designed specifically for this problem. It handles JavaScript rendering, anti-bot fingerprinting, rotating proxies, and headless browser management so you don't have to and what better way to do it straight from your LLM supplying the URLs? Hence I created this zyte-screenshots Claude Skill, which you can use to trigger the entire workflow- API call, base64 decode, PNG save on your filesystem, all just by chatting with Claude.

In this tutorial, we'll walk through exactly how the skill works, how to set it up, and how to use it to capture production-quality screenshots of any URL.

Why Use the Zyte API for Screenshots?

Before diving into the skill itself, it's worth understanding what makes the Zyte API uniquely suited to screenshot capture at scale.

1. Full JavaScript Rendering

Single-page applications built with React, Vue, Angular, or Next.js don't serve their content in the raw HTML response, they render it client-side after the page loads. Tools that capture the raw HTTP response will get a blank shell. Zyte's screenshot endpoint fires a real headless browser, waits for the DOM to fully settle, then captures the final rendered state.

2. Anti-Bot and Anti-Ban Protection

Enterprise-grade sites use fingerprinting libraries to detect automation. They check TLS fingerprints, browser headers, canvas rendering patterns, mouse movement entropy, and dozens of other signals. Zyte's infrastructure is battle-tested to pass these checks so your screenshots won't return a "Access Denied" page.

3. Scale Without Infrastructure

Managing a fleet of headless browser instances, proxy rotation, retries, and residential IP pools is a serious engineering investment. Zyte abstracts all of this into a single API call.

4. One API, Any URL

Whether the target is a static HTML page, a JS-heavy SPA, a behind-login dashboard (with session cookies), or a geo-restricted site, the same API call structure works. The skill you're about to install uses this endpoint.

What Is the zyte-screenshots Claude Skill?

Claude Skills are reusable instruction packages that extend Claude's capabilities with domain-specific workflows. The zyte-screenshots skill teaches Claude how to:

Accept a URL from the user in natural language
Read the ZYTE_API_KEY environment variable
Construct and execute the correct curl command against https://api.zyte.com/v1/extract
Pipe the JSON response through jq and base64 --decode to produce a PNG file
Derive a clean filename from the URL (e.g. https://quotes.toscrape.com becomes quotes.toscrape.png)
Report the exact file path and describe what's visible in the screenshot in one sentence

In practice, this means you can open Claude, say "screenshot https://example.com", and have a pixel-perfect PNG on your filesystem in seconds, no browser, no script, no Puppeteer config.

Prerequisites

Before installing the skill, make sure you have the following:

Tools

curl: Pre-installed on macOS and most Linux distributions. On Windows, use WSL or Git Bash.
jq: A lightweight JSON processor. Install via brew install jq (macOS) or sudo apt install jq (Ubuntu/Debian).
base64: Standard on all Unix-like systems.
Claude desktop app with Skills support enabled.

A Zyte API Key

Sign up at zyte.com and navigate to your API credentials. The free tier includes enough credits to get started with testing. Copy your API key, you'll set it as an environment variable.

💡 Pro tip: Set your ZYTE_API_KEY in your shell profile (~/.zshrc or ~/.bashrc) so it's always available: export ZYTE_API_KEY="your_key_here" or pass it along your prompt

Installing the Skill

Step 1: Download the Skill from GitHub

The skill is open source and available at github.com/apscrapes/claude-zyte-screenshots. Download the latest release ZIP from the repository's Releases page, or clone it directly:

git clone https://github.com/apscrapes/claude-zyte-screenshots.git

Step 2: Import into Claude

Open the Claude desktop app or go to Claude.ai
Navigate to Settings → Skills
Click Import Skill
Select the SKILL.md file from the downloaded ZIP
Claude will confirm the skill has been installed

Step 3: Set Your API Key

In your terminal, export your Zyte API key or pass it along your prompt:

export ZYTE_API_KEY="your_zyte_api_key_here"

To make this permanent across terminal sessions, add the export line to your ~/.zshrc or ~/.bashrc file.

Using the Skill: Step-by-Step

Basic Screenshot

Once the skill is installed and your API key is set, using it is as simple as asking Claude:

Screenshot https://quotes.toscrape.com using zyte-screenshots skill

Claude will:

Verify ZYTE_API_KEY is available in the environment
Derive the output filename: quotes.toscrape.png
Execute the following command:

curl -s https://api.zyte.com/v1/extract \
  -u "$ZYTE_API_KEY": \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://quotes.toscrape.com",
    "screenshot": true
  }' \
| jq -r '.screenshot' \
| base64 --decode > quotes.toscrape.png

Verify the file is non-empty
Open and inspect the image
Return the file location and a one-line visual description

Example output from Claude:

✅ Screenshot captured!

📁 Location: /home/you/quotes.toscrape.png

🖼️  What's in it: A clean light-themed page listing inspirational quotes
   with author attributions and tag filters in the sidebar.

Under the Hood: How the API Call Works

Let's break down the exact curl command the skill executes:

curl -s https://api.zyte.com/v1/extract \
  -u "$ZYTE_API_KEY": \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://target-site.com",
    "screenshot": true
  }' \
| jq -r '.screenshot' \
| base64 --decode > output.png

curl -s — Silent mode; suppresses progress output.

-u "$ZYTE_API_KEY": — HTTP Basic Auth. Zyte uses the API key as the username with an empty password.

-H "Content-Type: application/json" — Tells the API to expect a JSON body.

-d '{...}' — The JSON request body. Setting screenshot: true instructs Zyte to return a base64-encoded PNG of the fully rendered page.

| jq -r '.screenshot' — Extracts the raw base64 string from the JSON response.

| base64 --decode — Decodes the base64 string into binary PNG data.

> output.png — Writes the binary data to a PNG file.

The Zyte API handles everything in between — spinning up a headless Chromium instance, loading the page with real browser fingerprints, waiting for JavaScript execution to complete, and rendering the final DOM to a pixel buffer.

This was a fun weekend project I put together, let me know your thoughts on our Discord and feel free to play around with it. I'd also love to know if you create any useful claude skills or mcp server, so say hi on our discord.

Tags: web scraping • Zyte API • screenshots at scale • JavaScript rendering • anti-bot • Claude AI • Claude Skills • automation • headless browser • site APIs

Why your Python request gets 403 Forbidden

John Rooney — Wed, 04 Mar 2026 09:11:29 +0000

If you’ve had your HTTP request blocked despite using correct headers, cookies, and clean IPs, there’s a chance you are running into one of the simplest forms of blocking, and one of the most confusing for beginners.

Chances are, you will recognise the problem. You found the hidden API, and your request works perfectly in Postman... but it fails instantly within your Python code.

It’s called TLS fingerprinting. But the good news is, you can solve it. In fact, when I showed this to some developers at Extract Summit, they couldn’t believe how straightforward it was to fix.

CAPTION: “I copied the request -> matching headers, cookies and IP, but it still failed?”

Your TLS fingerprint

Let’s start with a question. How do the servers and websites know you’ve moved from Postman to making the request in Python? What do they see that you can’t? The key is your TLS fingerprint.

To use an analogy: We’ve effectively written a different name on a sticker and stuck it to our t-shirt, hoping to get past the bouncer at a bar.

Your nametag (headers) says "Chrome."

But your t-shirt logo (TLS handshake) very obviously says "Python."

It’s a dead giveaway. This mismatch is spotted immediately. We need to change our t-shirt to match the nametag.

To understand how they spot the “logo”, we need to look at the initial “Client Hello” packet. There are three key pieces of information exchanged here:

Cipher suites: The encryption methods the client supports.
TLS extensions: Extra features (like specific elliptic curves).
Key exchange algorithms: How they agree on a password.

This is because Python’s requests library uses OpenSSL, while Chrome uses Google's BoringSSL. While they share some underlying logic, their signatures are notably different. And that’s the problem.

OpenSSL vs. BoringSSL

The root cause of this mismatch lies in the underlying libraries.

Python’s requests library relies on OpenSSL, the standard cryptographic library found on almost every Linux server. It is robust, predictable, and remarkably consistent.

Chrome, however, uses BoringSSL, Google’s own fork of OpenSSL. BoringSSL is designed specifically for the chaotic nature of the web and it behaves very differently.

The biggest giveaway between the two is a mechanism called GREASE (Generate Random Extensions And Sustain Extensibility).

[
  "TLS_GREASE (0xFAFA)",
   ....
]

Chrome (BoringSSL) intentionally inserts random, garbage values into the TLS handshake - specifically, in the cipher suites and extensions lists. It does this to "grease the joints" of the internet, ensuring that servers don't crash when they encounter unknown future parameters.

This is one of the key changes

Chrome: Always includes these random GREASE values (e.g., 0x0a0a).
Python (OpenSSL): Never includes them. It only sends valid, known ciphers.

So, when an anti-bot system sees a handshake claiming to be "Chrome 120" but lacking these random GREASE values, it knows instantly that it is dealing with a script. It’s not just that your shirt has the wrong logo; it’s that your shirt is too clean.

JA3 hash

Anti-bot companies take all that handshake data and combine it into a single string called a JA3 fingerprint.

Salesforce invented this years ago to detect malware, but it found its way into our industry as a simple, effective way to fingerprint HTTP requests. Security vendors have built databases of these fingerprints.

It is relatively straightforward to identify and block any request coming from Python’s default library because its JA3 hash is static and well-known.

This code snippet would yield the below JSON response.

def get_ja3_info():
    url = "https://tls.peet.ws/api/clean"
    with requests.Session() as session:
        response = session.get(url)
        response.raise_for_status()
        data = response.json()
        print(json.dumps(data))

Note the lack of akamai_hash

{
  "ja3": 
"771,4866-4867-4865-49196-49200-49195-49199-52393-52392-49188-49192-49187-49191-159-158-107-103-255,0-11-10-16-22-2
3-49-13-43-45-51-21,29-23-30-25-24-256-257-258-259-260,0-1-2",
  "ja3_hash": "a48c0d5f95b1ef98f560f324fd275da1",
  "ja4": "t13d1812h1_85036bcba153_375ca2c5e164",
  "ja4_r": 
"t13d1812h1_0067,006b,009e,009f,00ff,1301,1302,1303,c023,c024,c027,c028,c02b,c02c,c02f,c030,cca8,cca9_000a,000b,000
d,0016,0017,002b,002d,0031,0033_0403,0503,0603,0807,0808,0809,080a,080b,0804,0805,0806,0401,0501,0601,0303,0301,030
2,0402,0502,0602",
  "akamai": "-",
  "akamai_hash": "-",
  "peetprint": 
"772-771|1.1|29-23-30-25-24-256-257-258-259-260|1027-1283-1539-2055-2056-2057-2058-2059-2052-2053-2054-1025-1281-15
37-771-769-770-1026-1282-1538|1||4866-4867-4865-49196-49200-49195-49199-52393-52392-49188-49192-49187-49191-159-158
-107-103-255|0-10-11-13-16-21-22-23-43-45-49-51",
  "peetprint_hash": "76017c4a71b7a055fb2a9a5f70f05112"
}

Putting the above JA3 hash into ja3.zone clearly shows this is a python3 request, using urllib3:

What’s the solution?

As mentioned, simply changing headers and IP addresses won’t make a difference, as these are not part of the TLS handshake. We need to change the ciphers and Extensions to be more like what a browser would send.

The best way to achieve this in Python is to swap requests for a modern, TLS-friendly library like curl_cffi or rnet.

Here is how easy it is to switch to curl_cffi:

from curl_cffi import requests
# note the impersonate argument & import above
def get_ja3_info():
    url = "https://tls.peet.ws/api/clean"
    with requests.Session() as session:
        response = session.get(url, impersonate="chrome")
        response.raise_for_status()
        data = response.json()
        print(json.dumps(data))

"akamai_hash": "52d84b11737d980aef856699f885ca86"

CAPTION: Note - I searched via the akamai_hash here as the fingerprint from the JA3 hash wasn’t in this particular database.

By adding that impersonate parameter, you are effectively putting on the correct t-shirt.

Summary

Make curl_cffi or rnet your default HTTP library in Python. This should be your first port of call before spinning up a full headless browser.

A simple change (which brings benefits like async capabilities) means you don’t fall foul of TLS fingerprinting. curl-cffi even has a requests-like API, meaning it's often a drop-in replacement.

Hybrid scraping: The architecture for the modern web

John Rooney — Wed, 25 Feb 2026 17:05:36 +0000

If you scrape the modern web, you probably know the pain of the JavaScript challenge.

Before you can access any data, the website forces your browser to execute a snippet of JavaScript code. It calculates a result, sends it back to an endpoint for verification, and often captures extensive fingerprinting data in the process.

Once you pass this test, the server assigns you a session cookie. This cookie acts as your "access pass." It tells the website, "This user has passed the challenge," so you don’t have to re-run the JavaScript test on every single page load.

For web scrapers, this mechanism creates a massive inefficiency.

It looks like you are forced to use a headless browser (like Puppeteer or Playwright) for every single request just to handle that initial check. But browsers are heavy, they are slow and they consume massive amounts of RAM and bandwidth.

Running a browser for thousands of requests can quickly become an infrastructure nightmare. You end up paying for CPU cycles just to render a page when all you wanted was the JSON payload.

The solution: Hybrid scraping

The answer to this problem is a technique I’ve started calling hybrid scraping.

This involves using the browser only to open the initial request, grab the cookie, and create a session. Once you have them, you extract that session data and hand it over to a standard, lightweight HTTP client.

This architecture gives you the access of a browser with the speed and efficiency of a script.

Implementing this in Python

To build this in Python, we need two specific packages:

A browser: We will use ZenDriver, a modern wrapper for headless Chrome that handles the "undetected" configuration for us.
HTTP client: We will use rnet, a Rust-based HTTP client for Python.

But why rnet? Well, within the initial TLS handshake where the client/server “hello” is sent, the information traded here can be fingerprinted, taking in things like the TLS version and the ciphers available for encryption. This can be hashed into a fingerprint and profiled.

Python’s requests package, which uses urllib from the standard library, has a very distinctive TLS fingerprint, containing ciphers (amongst other things) that aren’t seen in a browser. This makes it very easy to spot. Both rnet, and other options such as curl-cffi, are able to send a TLS fingerprint similar to that of a browser. This reduces the chances of our request being blocked.

Here is how we assemble the pipeline.

Step 1: Load the page (The handshake)

First, we define our browser logic. Notice that we are not trying to parse HTML here. Our only goal is to visit the site, pass the initial JavaScript challenge, and extract the session cookies.

import zendriver as zd
import asyncio

async def get_cookies():
    """
    Use ZenDriver to launch a browser, navigate to the page, 
    and retrieve the cookies.
    """

    browser = await zd.start()

    # Hit the homepage to trigger the check
    await browser.get("https://auto.hylnd7.com")

    # Wait briefly for the JS challenge to complete
    await asyncio.sleep(1) 

    # Extract the cookies
    requests_style_cookies = await browser.cookies.get_all()
    await browser.stop()

    return requests_style_cookies

What’s happening here:

We launch the browser, visit the site, and wait just one second for the JS challenge to run. Once we have the cookies, we call browser.stop(). This is the most important line: we do not want a browser instance wasting resources when we don’t need it.

Step 2: Use the cookies

Now that we have the "access pass," we can switch to our lightweight HTTP client. We take those cookies and inject them into the rnet client headers.

from rnet import Client, Emulation

async def http_request_rnet(cookies=None):
    """
    Make a fast request using RNet with the borrowed cookies.
    """
    headers = {
        "referer": "https://auto.hylnd7.com/",
    }

    # Format the browser cookies into a simple HTTP header string
    if cookies:
        cookie_list = []
        for cookie in cookies:
            cookie_list.append(f"{cookie.name}={cookie.value}")
        headers["Cookie"] = "; ".join(cookie_list)

    # We use Emulation.Chrome142 to change the TLS Fingerprint.
    # This is site dependent - but worth using
    client = Client(emulation=Emulation.Chrome142, headers=headers)

    response = await client.get("https://auto.hylnd7.com/api/products?page=1&limit=8")
    return response

What’s happening here:

We convert the browser's cookie format into a standard header string. Note the “Emulation.Chrome142” parameter. We are layering two techniques here: hybrid scraping (using real cookies) and TLS fingerprinting (using a modern HTTP client). This double-layer approach covers all our bases.

(Note: Many HTTP clients have a cookie jar that you could also use; for this example, sending the header directly worked perfectly).

Step 3: Run the code

Finally, we tie it together. For this demo, we use a simple argparse flag to show the difference with and without the cookie.

async def main(use_cookies: bool):
    cookies = None

    # The Decision Logic
    if use_cookies:
        cookies = await get_cookies() # Run the heavy browser

    # Always run the fast HTTP client
    resp = await http_request_rnet(cookies)

    status_code = resp.status
    print("Status Code:", status_code)

    if status_code == 200:
        print("Response Body:", await resp.json())
    else:
        print("Request blocked")

Get the complete script

Want to run this yourself? We’ve put the full, copy-pasteable script (including the argument parsers and imports) in the block below.

uv init
uv add zendriver rnet rich
# linux/mac
source .venv/bin/activate\
# windows
.venv\Scripts\activate

import argparse
import asyncio

import zendriver as zd
from rnet import Client, Emulation
from rich import print


async def http_request_rnet(cookies=None):
    """
    Make an HTTP GET request using rnet with the provided cookies. Cookies are sent in the headers. Note for this site we need the referer too.
    Return the Response Object.
    """
    headers = {
        "referer": "https://auto.hylnd7.com/",
    }

    if cookies:
        cookie_list = []
        for cookie in cookies:
            # Adjust based on the actual structure of the cookie object from zendriver
            # If it's a dict: cookie['name'], cookie['value']
            # If it's an object: cookie.name, cookie.value
            cookie_list.append(f"{cookie.name}={cookie.value}")
        headers["Cookie"] = "; ".join(cookie_list)

    client = Client(emulation=Emulation.Chrome142, headers=headers)
    response = await client.get("https://auto.hylnd7.com/api/products?page=1&limit=8")
    return response

async def get_cookies():
    """
    Use zendriver to launch a browser, navigate to a page, and retrieve cookies.
    """
    browser = await zd.start()
    await browser.get("https://auto.hylnd7.com")
    await asyncio.sleep(1)
    requests_style_cookies = await browser.cookies.get_all()
    await browser.stop()
    return requests_style_cookies

async def main(use_cookies: bool):
    cookies = None
    if use_cookies:
        cookies = await get_cookies()

    resp = await http_request_rnet(cookies)
    status_code = resp.status
    print("Status Code:", status_code)

    if status_code == 200:
        print("Response Body:", await resp.json())

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Make HTTP request with optional browser cookies")
    parser.add_argument(
        "--cookies",
        type=lambda x: x.lower() == "true",
        default=False,
        help="Set to 'true' to launch browser and get cookies, 'false' to skip (default: false)"
    )
    args = parser.parse_args()
    asyncio.run(main(args.cookies))

Pros and Cons of Hybrid Scraping

Feature	Pros	Cons
Efficiency	Reduces RAM usage massively compared to pure browser scraping.	Higher complexity: You must manage two libraries (zendriver and rnet) and the glue code.
Speed	HTTP requests complete in milliseconds. Browsers take seconds.	State management: You need logic to handle cookie expiry. If the cookie dies, you must "wake up" the browser.
Access	You get the verification of a real browser without the drag.	Maintenance: You are debugging two points of failure: the browser's ability to solve the challenge, and the client's ability to fetch data.

Final thoughts

For smaller jobs, it might be easier to just use the browser; the benefits won’t necessarily outweigh the extra complexity required.

But for production pipelines, this approach is the standard. It treats the browser as a luxury resource: used only when strictly necessary to unlock the door, so the HTTP client can do the real work. It’s this session and state management that allows you to scrape harder-to-access sites effectively and efficiently.

If building this orchestration layer yourself feels like too much overhead, this is exactly what the Zyte API handles internally. We manage the browser/HTTP switching logic automatically, so you just make a single request and get the data.

Raspberry Pi & E-ink scrapes & displays the price of Gold today

Ayan Pahwa — Tue, 24 Feb 2026 08:39:31 +0000

They say that “data is the new oil”, but there’s another hot commodity that’s setting markets alight - precious metals.

In the last 12 months, the value of gold has surged about 75%, while silver has boomed more than 200%. That’s why I, like a growing number of others, now trade in the metal markets.

These days, it is possible to buy digital versions of precious metals. But I think of myself as a collector - I like to buy real, solid coins or bullions whenever I get a chance.

In the last two years, I have acquired a small collection of gold bullions and silver coins, which have appreciated healthily. But I am not planning to sell and book a profit just yet. In fact, I want to buy more, especially when there’s a dip in the price.

There’s just one problem that hits this hobby - prices of actual physical gold and silver bullions are very different in the retail market from stock exchanges’ spot prices and keeping track of them manually is cumbersome specially with a full time job.

To take advantage of the dips and price arbitrage, I need to automate my decisions. To buy gold old-style, I need a key resource from the modern trading toolset - data.

Turn data into gold

The All-India Gem And Jewellery Domestic Council (GJC), a national trade federation for the promotion and growth of trade in gems and jewellery across, is the go-to site listing latest retail rates for gold and silver.

Alas, it doesn’t offer an API to access that data. But fear not - with web scraping skills and Zyte API, I can extract these prices quickly and regularly.

And I can do it using some of the tech I love to tinker with.

I call it ExtractToInk - a custom project that pulls the latest prices on a two-inch, 250x122 e-ink display powered by a retired Raspberry Pi (total cost under US$50).

This is the story of how I power my quest for rapid riches using cheap old hardware and the world’s best web scraping engine - and how you can, too.

Mining for data

Like many modern sites, GJC’s includes both JavaScript rendering for HTML and protection mechanisms- technologies that can break brittle traditional scraping solutions.

This project connects all the dots:

Web → Extract → Parse → Render → Physical display

Tech stack

Hardware

Raspberry Pi (tested on Pi Zero 2 W), it should run on any Raspberry Pi Board
Pimoroni Inky pHAT (Black, SSD1608)

Software

Python 3
Zyte API: to get rendered HTML
BeautifulSoup: to parse HTML
Pillow and Inky Python libraries: for e-ink display stuff

Now let’s get building.

Step 1: Prepare hardware

Setup your Raspberry Pi. In my case, I am using Raspberry Pi OS booted from the SD card.

Depending on which display you use, it most probably will be connected to the Pi over i2c bus or SPI bus protocol - so, enable your display type by entering:

sudo raspi-config

Now attach your e-ink display and do a quick reboot

You might need to install libraries to use your e-ink display.

Step 2: Fetching rendered HTML with Zyte API

The source site, GJC, renders prices dynamically, using JavaScript - something which can make plain HTTP requests unreliable.

No problem. By accessing the page through Zyte API, we can set browserHTML mode to return the page content as though rendered in an actual browser.

Instead of fighting JavaScript, we let Zyte handle it.

html = requests.post(`  
    `"https://api.zyte.com/v1/extract",`  
    `auth=(ZYTE_API_KEY, ""),`  
    `json={"url": URL, "browserHtml": True},`  
`).json()["browserHtml"]

Note: there is no Selenium here, and no headless browsers. This is much more reliable for production-style scraping

Step 3: Parsing with CSS selectors

Once we have clean HTML, parsing becomes straightforward.

Gold prices

Let’s locate the actual prices in the page mark-up.

for row in soup.select(".gold_rate table tr"):`  
    `label = row.select_one("td strong")`  
    `values = row.select("td strong")`

    `if not label or len(values) < 2:`  
        `continue`

    `text = label.get_text(strip=True)`  
    `priceText = values[1].get_text()`

    `if "Standard Rate Buying" in text:`  
        `goldBuying = re.search(r"\d[\d,]*", priceText).group(0)`

    `if "Standard Rate Selling" in text:`  
        `goldSelling = re.search(r"\d[\d,]*", priceText).group(0)

We’re deliberately using:

CSS selectors (easy to find from your browser’s DevTools).
Minimal regular expressions (only for numeric extraction).
Defensive checks to avoid brittle parsing.

Silver prices

Silver appears outside the main table, so we filter it carefully:

for strong in soup.select("p > strong"):`  
    `text = strong.get_text(" ", strip=True)`

    `if "Standard Rate Selling" in text and not strong.find_parent("table"):`  
        `silver = re.search(r"\d[\d,]*", text).group(0)`  
        `break

Step 4: Rendering for e-ink

For this project, I did not want to pipe data into a web dashboard on a computer monitor.

E-ink is always-on, low power, distraction-free and perfect for “ambient information” like this.

So, it’s a great fit for data like prices, weather, status indicators and system health.

But e-ink displays are not normal screens.

They are typically black-and-white, have high contrast and are slow to refresh.

What’s more, no two e-ink displays are made the same way. Every vendor has different support packages so, whichever you end up using, make sure to read the documentation and change the code accordingly.

In my case, I am using Pimoroni inky PHat. The supplied Python library has great built-in examples to get you quickly up and running. I used the helper function to render texts on the display, ex, the build in draw.text() function comes handy:

Draw silver selling price

    draw.text((x, y), f"Silver : {silverPrice}", fill=(0, 0, 0), font=fontBig)

Taking it furtherSection about the finished product

I built this project to use web data thoughtfully, connecting it to the physical world, and building pipelines that feel calm, reliable, and purposeful. When I am at my work-desk the project actively tells me the current prices so I can buy new coins if I see a price drop.

I can further extend this to place automatic orders on the website and secure me a coin at my desired strike price.

If you want to take this further, you could also:

Run it via cron every 10 minutes : The website I am targeting only refreshes prices twice a day, so my cron job runs every 12 hours, but, if you need faster data, you can scrape a site with more real-time updates.
Add more commodities or currencies.
Turn it into a systemd service to run at start time.
Swap e-ink for another output (PDF, LED, dashboard).

If you’re exploring Zyte API, or looking for real-world scraping examples beyond CSVs and JSON files, this project is a great place to start.

You can get my code in the ExtractToInk GitHub repository now.

Holiday Gift Guide 2025: For Developers, Web Scrapers & Everyone In Between

Ayan Pahwa — Fri, 19 Dec 2025 07:55:12 +0000

It’s that time of the year when the coffee gets stronger, commits get messier, and everyone agrees to finally refactor that script in January. And let’s be honest, most of us won’t. But while it lasts let’s celebrate the season of enjoying laid back family time and exchanging gifts.
And to make gifting a little easier, we asked the Zyte team and community to share what they would love to receive. So if you’ve got a developer, a web scraper, or someone who just really enjoys arguing with APIs in your life… Here's your cheat sheet.
Grab a hot drink, settle into your favourite debugging position, and let’s dive in.

Disclaimer : It's not sponsored. All the recommendations are from community and author's personal experience using these products, hence no URL is provided but you should be able find these easily.

1. Book: Soul of a New Machine

For the dev who loves origin stories. It’s a classic engineering tale that reminds us why we fell in love with building things in the first place.

2. Logitech MX Master mouse

Smooth scrolling. Perfect ergonomics. Side buttons that feel like cheat codes for productivity. This is the mouse equivalent of finally finding that one undocumented API endpoint you needed all year.

3. AeroPress + Coffee Subscription

If coffee is the real runtime powering your favourite developer, this combo is basically a performance upgrade. AeroPress means fast, clean brews; fresh beans mean they might actually fix that bug before lunch.

4. Spider Plush Toy

For every web scraper who proudly identifies as “part human, part spider.” it sits silently on your desk reminding you to obey robots.txt (…most of the time).

5. Git Merch Based on Their Commit History

A mug that says “I survived the merge conflict of 2025.”
A T-shirt celebrating their 3,000-day streak.
Or maybe a gentle reminder that “WIP” is not a personality.
Funny, personal, and guaranteed to make them smile during standup.
Link : https://gitmerch.com/ (not-sponsored), just enter their github username

6. Nothing Headphones

Sleek, transparent, and great for tuning out noisy offices or noisy families. Perfect for deep work, debugging, or pretending you can’t hear someone asking, “Can you take a quick look at this?”

7. Mechanical Keyboard (NuPhy / Keychron)

Developers don’t just type on these; they perform.
Clicky keys, gorgeous layouts, RGB that could guide aircraft, what’s not to love? Warning: once they switch, they’ll never stop talking about actuation force.

8. Bambu Lab A1 Mini 3D Printer

For the dev who already has too many hobbies.
They’ll print brackets, cable holders, figurines, replacement parts… and things no one can identify but everyone politely admires. A creator’s playground in miniature.

9. BenQ Monitor Light Bar

Immediate upgrade to any desk setup. Reduces eye strain, looks clean, and helps developers see their code clearly even during late-night “just one more function” sessions without taking extra space on their desk.

10. 100W GaN Charger

Tiny but absurdly powerful, just like that one script they wrote at 3 AM that still runs in production. A GaN charger keeps everything powered: laptop, phone, tablet, e-reader, existential dread, everything.

If you love someone who spends their days coaxing data out of websites, obsessing over keyboard switches, or whispering sweet nothings to their terminal, this list has something that will make them light up.
Here’s to a warm, restful holiday season… and to fewer bugs in 2026. Happy gifting, happy coding, and as always, happy scraping! 🕷️✨

The AI Web Scraper: One Workflow to Scrape Anything (n8n Part 3)

Lakshay Nasa — Tue, 16 Dec 2025 14:00:59 +0000

Welcome to the finale of our n8n web scraping series!

In Part 1, we covered the basics: fetching a single page and parsing it with CSS selectors.
In Part 2, we tackled the tricky mechanics: pagination loops, infinite scroll, and network capture.

But if you’ve been following along, you know there is still one massive headache in web scraping: "New Site = New Workflow."

Every time you want to scrape a different website, you have to open the browser inspector, hunt for new <div> classes, debug why price_color isn't working, and rewrite your entire flow. It’s exhausting.

Today, we changed that.

In this final part, we are going to build an Automated AI Scraper - a single n8n workflow that can scrape almost anything (Online Stores, Article/News Sites, Job Boards, and more) without you changing a single node.

Whether you are a developer looking to save hours of coding, or a non-technical user who just needs the data without the headache, this tool is designed for you.

Watch the Walkthrough 🎬

💡 TL;DR: Want to start scraping immediately? We have packaged this entire workflow into a ready-to-use template. 👉 Automate Data Extraction with Zyte AI (Products, Jobs, Articles & More)

The Concept: "AI-Driven Architecture”

So, we will make n8n stop looking for specific elements (CSS Selectors) and start caring about what we actually want (Data).

We are leveraging Zyte’s AI-powered extraction. Instead of saying "Find the text inside .product_pod h3 a, we simply send the URL and say product: true. The AI analyzes the visual layout of the page and figures it out, even if the website changes its code tomorrow.

To handle every scenario, we designed three distinct pipelines:

The "AI Extraction" Pipeline (Automatic)
This is the core of the workflow. You simply select a Category (e.g., E-commerce, Article etc) and a Goal, and the workflow automatically routes you to one of two paths:
- Direct Extraction (Fast): If you just need data from the current page (like a "Single Product" or a "Simple List"), the workflow sends a single smart request. No loops, no waiting.
- The "Two-Phase" Architecture (For AI crawling): If your goal involves "All Pages" or "Visiting Items," the workflow activates a robust recursive loop:
  - Phase 1 (The Crawler): It maps out the URLs you need (looping through pagination or grabbing item links from a list).
  - Phase 2 (The Scraper): It visits every mapped URL one by one to extract the rich details you asked for.
The "SERP" Pipeline (For SEO Data)
Need search rankings? We included a dedicated path for Search Engine Results Pages (SERP). It uses the specific serp schema to automatically extract organic results, ads, and knowledge panels without you needing to parse complex HTML.
The "Manual" Mode (For Raw Control)
Sometimes you don't need AI. We added a "General" path that gives you raw browserHtml, HTTP responses, or Screenshots so you can parse specific data yourself.

Let’s get building.

Step 1: The Control Center

In previous parts, we hardcoded URLs into our nodes. For this tool, that won’t work. We need a flexible User Interface.

1. The Main Interface (Form Trigger)

We use an n8n Form Trigger node as the entry point. This turns your workflow into a clean web app that anyone on your team can use.

The Main Form collects three key inputs:

Target URL (Text): The website you want to scrape.
Site Category (Dropdown): Options like Online Store, Article/News, Job Post , General & More .
Zyte API Key (Password): Securely input the key so it isn't hardcoded in the workflow.

2. Smart Routing (The Switch Node)

This is where the magic happens. Immediately after the form, we use a Switch Node ("Route by Category") that directs the traffic into distinct lanes.

This logic is crucial because different categories require different inputs:

The AI Lane (Store, News, Jobs): If you select a structured category, the workflow routes you to a Secondary Form asking for your "Extraction Goal" (e.g., Scrape this page vs. Crawl ALL pages).
The SEO Lane: If you select "SERP (Search Engine Results)," it bypasses extra forms and goes straight to the specialized SERP scraper.
The Manual Lane (General): If you select "General," it routes you to a different Manual Options Form where you can choose specific technical actions (e.g., Take Screenshot, Get Browser HTML, Network Capture).

This architecture ensures you only see options relevant to your goal.

Step 2: Pipeline 1 – AI Extraction

If the user selects E-commerce, News/Blog/Article, or Jobs, they enter the AI pipeline.

1. The "AI Extraction Goal Form" (Refining Scope)

Since "scraping" can mean anything from checking one price to archiving an entire blog, we present a secondary form here to define the scope. You simply tell the workflow what you need: a quick Single Item lookup, a List from the current page, or a full Multi-Page Crawl.

2. The Brain (Config Generator)

We place a Code Node (the "Zyte Config Generator") to translate your form choices into technical instructions.

For instance,

If you selected "Online Store" → It maps to the Zyte schema product.
If you select "Article Site" → It maps to articles.
If you choose "Get List" → It targets productList (or articleList) to extract an array of items.
If you choose "Crawl All Pages" → It switches the target to productNavigation (or articleNavigation) to activate the crawler loop.

3. The 5 Strategies

Based on your "Extraction Goal," the workflow automatically routes to one of 5 specific branches:

A. Single Item:

Fast execution. Scrapes details of one URL.

We send a single request to the Zyte API with our specific target schema (e.g., product: true). The AI analyzes the page layout and returns a structured JSON object with the price, name, and details instantly.

B. List (Current Page):

Returns a clean JSON array of items found on the provided URL.

Similar to the single item strategy, but instead of asking for one object, we request a List schema (like productList). The AI identifies the repeating elements on the page and returns them as a clean array.

💡 Design Note: You might notice that the nodes for Strategy 1 and 2 look identical. That is because the heavy lifting (choosing between product vs productList) is actually handled upstream by the Config Generator.

🧑‍💻 Best Practice: In your own production automations, you should usually combine these into a single node to keep your canvas clean. However, for this template, we kept them separate. This makes the logic visually intuitive and allows you to add specific post-processing (like a unique filter) to the List path without accidentally breaking the Single Item path.

C. Details (Current Page):

A hybrid approach. It scans the current list, finds item links, and visits them one by one.

We use a two-step logic: first, we request a navigation schema to identify all item links on the current page. Then, we split that list and use a loop to visit each URL individually to extract the full details.

D. Crawl List (All Pages):

Activates the Crawler (Phase 1) to loop through pagination and build a massive master list.

This enables the pagination loop. The workflow fetches the current page's list, saves the items to a global "Backpack" (memory), detects the "Next Page" link automatically, and loops back to repeat the process until it reaches the end.

E. Crawl Details (All Pages):

The ultimate mode. It crawls all pages (Phase 1) AND visits every single item found (Phase 2).

This uses our robust "Two-Phase" architecture. Phase 1 loops through pagination specifically to map out every item URL. Once the map is complete, Phase 2 takes over to visit every single URL one by one and extract the deep data.

Step 3: Pipeline 2 – SERP (Search Engine Results)

If you select "Search Engine Results" in the main form, the workflow takes a direct path to the SERP Node.

This is a single HTTP Request node configured with the serp schema.

Input: Your target Search URL (e.g., a query on a search engine).
Output: Structured JSON containing organic results, ad positions, and knowledge panels.

It is the fastest way to get reliable SERP data for rank tracking or brand monitoring, handling complex layouts automatically.

Step 4: Pipeline 3 – Manual / General Mode

What if sometimes you need to scrape a unique dashboard, a niche directory, or just want to debug the raw HTML yourself. That’s why we included the "Manual" path.

If you select "General / Other" in the form, you are presented with a secondary form offering 5 raw tools:

Browser HTML: Returns the full rendered DOM (great for the custom parsing logic we built in Part 1).
HTTP Response Body: Useful for API endpoints.
Network Capture: Intercepts background XHR/Fetch requests (as we learned in Part 2).
Infinite Scroll: Automatically scrolls to the bottom before capturing HTML. (see the infinite scroll guide in Part 2)
Screenshot: Returns a PNG snapshot of the page. (view the setup steps here).

This ensures your scraping tool is never useless, even on the most obscure websites.

The Result & Output

Regardless of which pipeline you chose (AI, SERP, or Manual), all data converges at a final Data Collector node.

We use a Convert to File node to transform that JSON into a clean CSV file or Image file (for screenshots), ready for download directly in the browser

Get the Workflow

We have packaged this entire logic – the forms, the smart routing, the crawler loops, and the safety checks, into a single template you can import right now from the n8n community.

👉 Automate Data Extraction with Zyte AI (Products, Jobs, Articles & More)

Wrapping Up

And that’s a wrap on our n8n web scraping series! 🎬

From building your first simple scraper in Part 1, to mastering pagination in Part 2, we have now arrived at the ultimate goal: an Intelligent Scraper that adapts to the web so you don't have to.

You now have a tool that:

Gets You Data With Ease: Automatically extracts structured fields (like prices, images, and articles) without you needing to hunt for CSS selectors or manage CAPTCHAs.
Reduces Maintenance: Adapts to layout changes automatically.
Gives You Control: Lets you switch between AI automation and manual debugging instantly.

This template is ready for you to fork, modify, and deploy.

Thanks for joining us on this journey! If you build something cool, or if you run into a challenge that stumps you, come share it in the Extract Data Community.

Happy scraping! 🚀🕷️

Build Your Own Holiday Deal Tracker with Python, Zyte API & IFTTT 🎁

Ayan Pahwa — Fri, 21 Nov 2025 11:06:18 +0000

(A simple, developer-friendly project to help you catch Black Friday, Cyber Monday, and Christmas deals before anyone else)

Watch the video tutorial :

It’s that time of the year again, everyone’s hunting for discounts, limited-time deals, and that one item you’ve been keeping an eye on all year. You know the drill: tabs open everywhere, price tracker sites breaking under traffic, browser extensions that promise magic but fail right when The Deal drops.

So this year, I decided to build something different, something that actually works when I need it the most.

A personal price-alert system, powered by:

Python 🐍
Zyte’s Automatic Extraction API 🕷️
An IFTTT mobile notification trigger 📱

And honestly? It ended up being one of the simplest, useful and most reliable holiday project I’ve made in a long time.

❌ No HTML parsing.
❌ No complex CSS selectors.
❌ No brittle scrapers that break during peak traffic.
✅Just a clean API call, structured product data, and a push notification when the price hits your set target.

Let me walk you through the whole thing :

🎁 Why Build Your Own Deal Tracker?

Because holiday deals don’t wait for anyone. Every year I get messages from friends asking,

“Bro, what’s the best price tracker? Nothing seems to work today.”

And they’re right. Most free services throttle, break, or go down completely during Black Friday and Cyber Monday because everyone hits them at once.

Meanwhile, with a few lines of Python and Zyte’s AI-powered extraction, you can build something that:

Works only for you
Deals with anti-ban, anti-bot and captchas
Checks as often as you want (make it run on your pc, github-actions, server or a raspberry pi)
Doesn’t rely on brittle selectors
Doesn’t choke on JavaScript
Doesn’t get rate-limited
Sends you a mobile ping instantly when price drops

It’s the perfect mix of practical dev fun + a tool you’ll actually use.

🎄 Why Zyte?

Scraping e-commerce sites for price data is usually a pain: blocking, JavaScript rendering, cookie rules, bot detection, dynamic DOMs, infinite variations in product page layouts... you get it.

The magic here is Zyte’s Automatic Extraction.

You literally tell AI powered Zyte API:

{ 
"url": "<product-url>", 
"product": true 
}

and it returns structured fields like:

price
currency
product name
sku
images
stock availability
description

You don’t write selectors. You don’t parse HTML. You don’t chase CSS changes. You just get clean data. And that makes this holiday project absurdly simple.

🎅 What We’re Building

A script that:

Takes:
a product URL
a target price
your Zyte API key
your IFTTT Webhook key
Fetches structured product data using Zyte API
Checks if the product price is ≤ your target
If yes → triggers an IFTTT event

Your phone instantly notifies you with product URL:
“Your product dropped to your target price. Go get it!”

You can run this:

manually (when you want)
on a cron job (in the background)
in GitHub Actions
on a Raspberry Pi
on a cloud function Your call.

🛠 Setup

Clone your project

git clone https://github.com/apscrapes/zyte-sale-alert.git
cd zyte-sale-alert

Create your virtual environment

python3 -m venv venv
source venv/bin/activate

Install dependencies

pip install -r requirements.txt

Create and add secrets in .env file

ZYTE_API_KEY=your_zyte_key_here (obtained from zyte)
IFTTT_KEY=your_webhooks_key (obtained from IFTTT, see next step)
IFTTT_EVENT=price_drop (name of your ifttt applet, see next step)
TARGET_PRICE=149.99 (example)
PRODUCT_URL=https://example.com/product/123

Create an IFTTT Webhook Applet

a. Download IFTTT mobile app, it’s paid but you get a 7-day trial and it has tons of automation you can build using no-code, so i think it’s worth it.

b. Click create new automation / applet

c. In “IF” field, select “Webhooks” > Add Event Name

d. In “THEN” field select Notification > App Notification > JSON Payload > High Priority

Note : You can instead of notification can also set other automation like getting an email for example

e. Here's how it should look like when it's successfully created:

Once the automation is created add your IFTTT_KEY and IFTTT_EVENT to your .env file

Set product parameters The main script is at src/pricedrop.py, edit the following variables to add your:

PRODUCT_URL = “ ”
DESIRED_PRICE = 250

PRODUCT_URL “ ” is the URL of the product you want to track and DESIRED_PRICE is the price at which you want to be notified, that is it.

Run the project

python src/pricedrop.py

You can run it manually or set up a cronjob to run at regular intervals. Whenever the price drops equal or below your target price you’ll get a notification from the IFTTT app on your phone with product URL so you can order it right away.

Let’s understand how it works.

🧩 Core Logic (Short Version)

resp = client.get({"url": url, "product": True})

“product : True” tells zyte API that the webpage we’re scraping has a product so the Machine Learning powered scrapper gets you all the relevant parameters like price, quantity, description, currency etc.

And that’s literally all you need.

The reason this works so beautifully is Zyte is handling:
JS rendering
blocking
retries
browser simulation
extraction logic
AI-powered field detection

In-detail :

Importing all the necessary libraries

import os
import sys
import requests
from zyte_api import ZyteAPI
from dotenv import load_dotenv

Setting required variables

PRODUCT_URL = "https://outdoor.hylnd7.com/product/a1b2c3d4-e5f6-4a7b-8c9d-000000000293"
DESIRED_PRICE = 250

Here we've setup a sample product link whose price we want to track and a price at which if it goes below we want to be notified.

Loading the API keys from .env file

load_dotenv()

# from Zyte API
ZYTE_API_KEY = os.getenv("ZYTE_API_KEY")

# from IFTTT Service applets
EVENT_NAME= os.getenv("EVENT_NAME")
IFTTT_KEY= os.getenv("IFTTT_KEY")

if not ZYTE_API_KEY:
    print("ERROR: ZYTE_API_KEY not found in environment.")
    sys.exit(1)

In the project root directory, create a .env file and add ZYTE API Key you'll get after logging into zyte.com and IFTTT webhook API key you get after creating the automation applet

Function to trigger the mobile notification :

def trigger_ifttt(event_name, key, value1):
    url = f"https://maker.ifttt.com/trigger/{event_name}/json/with/key/{key}"

    payload = {
        "value1": value1,
    }

What this funciton is doing is basically making an API call to IFTTT and IFTTT applet is set so whenever the API calls comes with payload it sends mobile notification with that payload, which in this case is product URL so you can directly click and open the product page and buy it before it goes out of stock, SMART right? 😉

Scraping init

client = ZyteAPI(api_key=ZYTE_API_KEY)

    payload = {
        "url": PRODUCT_URL,
        "product": True,          
    }

    resp = client.get(payload)

Making a GET request on Zyte API with product : True, we're asking zyte to treat the URL as product page and thus it's uses it's ML capabilities to fetch product relevant details, price in this case.

Compare price to SETPOINT

if price_float <= DESIRED_PRICE:
            trigger_ifttt(EVENT_NAME, IFTTT_KEY, value1 = PRODUCT_URL)

If price of the product reaches to or below our target price it will call the IFTTT function, thus triggering the notification.

🌟 Make It Even Better

You can extend this to:

Track multiple URLs
Log daily prices to CSV
Plot graphs
Send WhatsApp alerts
Push to a Telegram bot
Use GitHub Actions to check every hour
Deploy as a Streamlit dashboard

Zyte handles the extraction. You build the magic on top.

🧘 Final Thoughts

I love projects like this because they hit the sweet spot between:

seasonal usefulness
real-world scraping challenges
a clean developer experience
a fun weekend build

If you're new to the world of web scraping like me, this shows how powerful the right tools can be. If you're experienced, it’s refreshing to skip the boilerplate and let Zyte handle the messy parts.

And honestly, there’s something fun about getting a custom alert on your phone saying:
“Hey, that gadget you wanted all year just dropped to your target price.”

Happy building, happy holidays, and happy deal-hunting! 🎄🎁
Let me know what you end up tracking.

Join Zyte Discord to share what you're building or get any support :
https://discord.com/invite/DwTnbrm83s

@iayanpahwa

n8n Web Scraping || Part 2: Pagination, Infinite Scroll, Network Capture & More

Lakshay Nasa — Mon, 17 Nov 2025 18:15:13 +0000

This is Part 2 of our n8n web scraping series with Zyte API. If you’re new here, check out Part 1 first, it covers the basics: fetching pages, extracting HTML with the HTML node, cleaning + normalizing results, & exporting CSV/JSON.

Pagination
Infinite Scroll
Geolocation support
Screenshots from browser rendering
Capturing network requests
Handling cookies, sessions, headers & IP type

Let’s Begin!

In this part, we’ll explore some important scraping practices and nodes, along with a few hands on tricks that make your web scraping journey smoother.

Everything you learn here will also lay the foundation for our 3rd & final part, where we will build a universal scraper capable of scraping any website with minimal configuration.

Let’s start by taking the same workflow we built in Part 1, & extend it. Beginning with Pagination and Infinite Scroll.

Pagination across pages

A website can navigate in multiple ways & our scraper needs to adapt accordingly.

N8N gives us a default Pagination Mode inside the HTTP Request node under Options, and while it sounds convenient, it didn’t behave reliably in my experience for typical web scraping use cases.

After testing several patterns, the approach below is the one that has worked most consistently in my workflows.

💬 If you’re stuck or want to share your own approach, lets discuss it in the Extract Data Discord.

Step 1: Page Manager Node

Before calling the HTTP Request node, we introduce a small function called Page Manager, exactly what the name suggests: a node that controls the page number.

Add a Code node (JavaScript) and paste:

// Page Manager Function ( We use this node as both starter and incrementer)
const MAX_PAGE = 100;

// n8n provides `items` array. If no items => first run
let current = 0;
if (Array.isArray(items) && items.length > 0) {

  // Check for .json.page (from this node's first run)
  // OR .json.Page (from the Normalizer node's output)
  const p = items[0].json?.page || items[0].json?.Page;

  current = (typeof p !== 'undefined' && p !== null && p !== '') ? Number(p) || 0 : 0;
}

let next;
if (current === 0) {
  next = 1; // first run (still 1)
} else {
  next = current + 1;
}

if (next > MAX_PAGE) return []; // safety stop

return [{ json: { page: next } }];

What this does:

On the first run, it starts with page = 1.
Every time the loop returns here, it increments to the next page.
There’s a built in safety MAX_PAGE so you don’t accidentally infinite loop. ( Adjust accordingly )

Now update your URL in old HTTP Request node to use the page variable:

URL: https://books.toscrape.com/catalogue/page-{{ $json.page }}.html

This makes the node fetch the correct page each time.

The rest of the workflow remains the same, till the second HTML Extract node (where we parsed the book name, URL, price, rating etc. in Part 1).

Step 2: Modify the Normalizer Function Node to Save Results Across Pages

In Part 1, our Step 7 code simply cleaned and normalized items for one page.

Now we need it to do two things:

Normalize the results (same as before)
Store the results from every page inside n8n’s global static data bucket. Think of it like temporary workflow memory.

Update the node, code with:

// --- Normalizer (Code node) ---
// Get the global workflow static data bucket
const workflowStaticData = $getWorkflowStaticData('global');

// initialize storage if needed
workflowStaticData.workBooks = workflowStaticData.workBooks || [];

// normalization logic (kept minimal version)
const base = 'https://books.toscrape.com/';
const normalized = items.map(item => {
  const urlRel = item.json.url || '';
  const imgRel = item.json.image || '';
  const ratingClass = item.json.rating || '';
  const ratingParts = ratingClass.split(' ');
  const rating = ratingParts.length > 1 ? ratingParts[ratingParts.length - 1] : '';

  return {
    name: item.json.name || '',
    url: base + urlRel.replace(/^(\.\.\/)+/, ''),
    image: base + imgRel.replace(/^(\.\.\/)+/, ''),
    price: item.json.price || '',
    availability: (item.json.availability || '').toString().trim(),
    rating
  };
});

// append to global storage
workflowStaticData.workBooks.push(...normalized);

// return control info for IF node (not the items)
const currentPage = $('Page Manager').first().json.page || 1;
return [{
  json: {
    itemsFound: normalized.length,
    nextHref: $json.nextHref || null,
    Page: currentPage
  }
}];

We normalize the data exactly like Part 1.
Then we push all normalized items into workflowStaticData.workBooks.
Instead of returning the items themselves, we return only a small control object.
This object is used by the IF node to decide whether we continue scraping or stop.

Step 3: IF Node (Stop Scraping or Continue)

Add an IF node with two conditions and OR Type:

Condition 1:
{{ $json.itemsFound }} is equal to 0

Meaning → The current page returned no items → we’ve reached the end.

Condition 2:
{{ $json.Page }} is greater than or equal to YOUR_MAX_PAGE

Meaning → Stop when you reach the max page number you set.

Together these conditions help the workflow decide:

IF → True
Stop scraping and move to the export step.

IF → False
Go back to the Page Manager, increment the page number, and keep scraping.

This creates a complete and safe pagination loop.

Step 4: Collect All Results and Export

When the IF node returns True, add one more small Code node before the Convert To File node:

// Get the global data
const workflowStaticData = $getWorkflowStaticData('global');

// Get the array of books, or an empty array if it doesn't exist
const allBooks = workflowStaticData.workBooks || [];

// Return all the books as standard n8n items
return allBooks.map(book => ({ json: book }));

What this one does:

Pulls everything we stored in the temporary memory.
Returns it as normal n8n items.
These go straight into Convert To File → CSV.

And that’s the entire pagination workflow.

Infinite Scroll

This one is much simpler.

Some websites load content as you scroll, there's no traditional page numbers.
The Zyte API supports browser actions, which makes this easy.

Just add one line to our original cURL command:

curl \
   --user YOUR_ZYTE_API_KEY_GOES_HERE: \
   --header 'Content-Type: application/json' \
   --data '{"url": "https://quotes.toscrape.com/scroll", "browserHtml": true, "actions": [{ "action": "scrollBottom" }]}' \
   https://api.zyte.com/v1/extract

Why this works

Zyte API loads the page in a headful browser session.
It scrolls to the bottom, triggering all JavaScript that loads additional items.
Then it returns the final, fully loaded browserHtml.
You can parse this HTML normally using the same nodes from Part 1.

Geolocation

Some websites return different data depending on your region.
Zyte API makes this super simple by allowing you to specify a geolocation.

Use this inside an HTTP Request node:

curl \
   --user YOUR_ZYTE_API_KEY_GOES_HERE: \
   --header 'Content-Type: application/json' \
   --data '{"url": "http://ip-api.com/json", "browserHtml": true, "geolocation": "AU" }' \
   https://api.zyte.com/v1/extract

Setting "geolocation": "AU" makes Zyte perform the browser request from that region, check the list of all available CountryCodes.
Many websites use region based content (pricing, currencies, language, product availability), so this is extremely helpful.

Screenshots

If you’d like to grab a screenshot of what the browser rendered, you can do that too.

cURL:

curl \
   --user YOUR_ZYTE_API_KEY_GOES_HERE: \
   --header 'Content-Type: application/json' \
   --data '{"url": "https://toscrape.com", "screenshot": true }' \
   https://api.zyte.com/v1/extract

It will return the screenshot as Base64 data.

To convert it into a proper image (PNG, JPEG, etc.) → Use Convert To File node in n8n.

Important:

n8n often converts boolean values like true into "true" when importing via cURL.
Fix it by clicking the gear icon → Add Expression → {{true}}.

Or switch body mode to Using JSON and paste:

{
  "url": "https://toscrape.com",
  "screenshot": true
}

Network Capture

Many modern websites load content through background API calls rather than raw HTML.
And you can just capture those network activity during rendering.

Example:

curl \
   --user YOUR_ZYTE_API_KEY_GOES_HERE: \
   --header 'Content-Type: application/json' \
   --data '{"url": "https://quotes.toscrape.com/scroll", "browserHtml": true,  "networkCapture": [
        {
            "filterType": "url",
            "httpResponseBody": true,
            "value": "/api/",
            "matchType": "contains"
        }]}' \
   https://api.zyte.com/v1/extract

This returns a networkCapture array with all responses whose URL contains /api/.

Understanding above Parameters

filterType: "url" ⟶ filter network requests by URL
value: "/api/" ⟶ look for URLs containing /api/
matchType: "contains" ⟶ pattern match style
httpResponseBody: true ⟶ include the response body (Base64)

Extracting data from the captured network response

You can decode the Base64 response in two easy ways:

1. Using a Function node (Python)

(You can also use JS if you prefer)

# Get the network capture data
capture = _input.first().json["networkCapture"][0]

# Decode base64 and parse JSON
import base64
import json

decoded_data = base64.b64decode(capture["httpResponseBody"]).decode('utf-8')
data = json.loads(decoded_data)

# Return the result
return [{
    "json": {
        "quotes": data["quotes"],
        "firstAuthor": data["quotes"][0]["author"]["name"]
    }
}]

→ This method decodes the Base64 encoded HTTP response, parses it as JSON, and gives you structured data directly, very reliable and readable.

2. Using Edit Field Node (No code)

In this method, you still need to parse your data

Add an Edit Fields node
Mode: Add Field
Name: decodedData
Type: String
Value:

{{ $json.networkCapture[0].httpResponseBody.base64Decode().parseJson() }}

→ This takes the Base64 content, decodes it, parses JSON, and puts the result under decodedData automatically.

Cookies, sessions, headers & IP type (quick guide)

When you move from toy sites to real sites, a few extra controls matter a lot: which IP type you use, whether you keep a session, and what cookies or headers you send.

Zyte API exposes all these as request fields and you can use them the same way we used browserHtml, networkCapture or actions above (via curl → Import in n8n HTTP Request node → Adjust Fields as needed → Extract).

To keep this guide focused, we won’t dive into all code examples here, but here’s one small one for _ Setting a cookie and getting it back_ ( requestCookies ) just to show how it integrates.

Cookies (viarequestCookies/ responseCookies) ➜ Useful when a website relies on cookies for preferences, language, or maintaining continuity between requests.

curl \
   --user YOUR_ZYTE_API_KEY_GOES_HERE: \
   --header 'Content-Type: application/json' \
   --data '{ "url": "https://httpbin.org/cookies", "browserHtml": true,
    "requestCookies": [{ "name": "foo",
            "value": "bar",
            "domain": "httpbin.org"
        }]
}' \
   https://api.zyte.com/v1/extract

⟶ This example uses requestCookies, but responseCookies works the same way, you simply read cookies from one request and pass them into the next.

Learn more on cookies.

Everything else below (sessions, ipType, custom headers) plugs in the same way.

Sessions
➜ Sessions bundle IP address, cookie jar, and network settings so multiple requests look consistently related. Helpful for multi step interactions, region based content, or sites that hate stateless scraping..
Docs: Sessions
Custom Headers
➜ Add a User Agent, Referer, or any custom metadata the target site expects: simply define them inside the HTTP Request node headers.
Docs: Headers
IP Type (datacenter vs residential)
➜ Some sites vary content based on IP type. Zyte API automatically selects the best option, but you can override it with ipType.
Docs: IP Types

All of these follow the same pattern we’ve already used above.

Where This Takes Us Next

And that’s it for Part 2! 🎉

We covered a lot more than just pagination, from infinite scroll & geolocation to screenshots, network capture, and the key request fields you’ll use while scraping sites.

What we learned isn’t a complete workflow on its own, but it builds the foundation you’ll use again and again in your scraping workflows.

In Part 3, we’ll take everything one step further and combine these patterns into a universal scraper: a reusable, configurable template that can adapt to almost any site with minimal changes.

Thanks for following along, and feel free to share your workflow, questions, or improvements in the Extract Data Community.
Happy scraping! 🕸️✨