Forem: Robert N. Gutierrez

Building a Crate & Barrel Price Monitor Dashboard with Python & Playwright

Robert N. Gutierrez — Tue, 07 Apr 2026 11:02:22 +0000

E-commerce pricing is rarely static. On sites like Crate & Barrel, the cost of a sofa or dining set can fluctuate significantly based on seasonal sales, clearance events, or inventory shifts. Checking a page manually every day is a chore you're likely to forget, often missing the sale entirely.

This guide covers building an automated Crate & Barrel price monitor. We’ll use an open-source Playwright scraper to extract data, build a wrapper to track price history, and create a visual dashboard using Pandas and Matplotlib. By the end, you’ll have a system that watches prices and visualizes trends so you can buy when the price is lowest.

Prerequisites & Setup

We’ll use the Crateandbarrel.com-Scrapers repository as a foundation.

1. Clone the Repository

Open your terminal and clone the base scrapers:

git clone https://github.com/scraper-bank/Crateandbarrel.com-Scrapers.git
cd Crateandbarrel.com-Scrapers

2. Install Dependencies

You'll need Python 3.8 or higher. Install the libraries for browser automation and data visualization:

pip install playwright pandas matplotlib playwright-stealth
playwright install chromium

3. Get a ScrapeOps API Key

Crate & Barrel uses anti-bot measures that often block standard headless browsers. To ensure the monitor runs reliably, we’ll use ScrapeOps to handle proxy rotation and browser headers.

You can sign up for a free ScrapeOps API key here.

Step 1: Understanding the Base Scraper

In the cloned repository, navigate to the Playwright implementation:
python/playwright/product_data/scraper/crateandbarrel_scraper_product_data_v1.py.

This script handles the heavy lifting of data extraction. It uses a PROXY_CONFIG dictionary to route requests through ScrapeOps:

# python/playwright/product_data/scraper/crateandbarrel_scraper_product_data_v1.py

PROXY_CONFIG = {
    "server": "http://residential-proxy.scrapeops.io:8181",
    "username": "scrapeops",
    "password": "YOUR_SCRAPEOPS_API_KEY"
}

The core of this scraper is the extract_data function. It parses JSON-LD (structured data embedded in the page) to find the most accurate price, name, and availability. If the price drops, the scraper captures both the current price and the preDiscountPrice.

Step 2: Configuring the Target List

Instead of scraping the whole site, we'll focus on specific items. Create a configuration file named products.py to store your target URLs.

# products.py

TARGET_PRODUCTS = [
    "https://www.crateandbarrel.com/lounge-deep-3-piece-sectional-sofa/s533475",
    "https://www.crateandbarrel.com/tate-walnut-gateleg-dining-table/s454238",
    "https://www.crateandbarrel.com/direction-ash-wood-floor-lamp/s582497"
]

Step 3: Building the Monitor Script

The base scraper runs once and saves a JSONL file. For a price tracker, we need a wrapper that appends data to a master CSV file every time it runs. Create monitor.py and import the logic from the repository.

import asyncio
import pandas as pd
from datetime import datetime
import os
from playwright.async_api import async_playwright
from products import TARGET_PRODUCTS
from python.playwright.product_data.scraper.crateandbarrel_scraper_product_data_v1 import extract_data, PROXY_CONFIG

async def run_monitor():
    results = []
    scrape_date = datetime.now().strftime("%Y-%m-%d")

    async with async_playwright() as p:
        # Launch browser with ScrapeOps proxy
        browser = await p.chromium.launch(proxy=PROXY_CONFIG)
        page = await browser.new_page()

        for url in TARGET_PRODUCTS:
            print(f"Checking price for: {url}")
            await page.goto(url, wait_until="domcontentloaded")

            data = await extract_data(page)

            if data:
                results.append({
                    "date": scrape_date,
                    "name": data.name,
                    "price": data.price,
                    "url": url
                })

        await browser.close()

    # Save to CSV
    df_new = pd.DataFrame(results)
    file_path = "price_history.csv"

    if not os.path.isfile(file_path):
        df_new.to_csv(file_path, index=False)
    else:
        df_new.to_csv(file_path, mode='a', header=False, index=False)

    print(f"Updated price history for {len(results)} items.")

if __name__ == "__main__":
    asyncio.run(run_monitor())

By appending to price_history.csv, we create a time-series dataset. Each row contains the date, product name, and price, which serves as the foundation for the dashboard.

Step 4: Automating the Run

To catch daily price changes, the script needs to run regularly.

Option A: Simple Loop

For quick testing, wrap the execution in a loop:

if __name__ == "__main__":
    while True:
        asyncio.run(run_monitor())
        print("Waiting 24 hours for next check...")
        time.sleep(86400)

Option B: Cron Job

In production, use a system scheduler like Cron. To run the monitor every morning at 8:00 AM, add this to your crontab (crontab -e):

0 8 * * * /usr/bin/python3 /path/to/monitor.py

Step 5: Visualizing the Data

We can turn the CSV file into a trend chart. Create dashboard.py using Pandas to clean the data and Matplotlib to plot it.

import pandas as pd
import matplotlib.pyplot as plt
import os

def generate_dashboard():
    if not os.path.exists("price_history.csv"):
        print("No data found. Run the monitor first!")
        return

    df = pd.read_csv("price_history.csv")
    df['date'] = pd.to_datetime(df['date'])
    df['price'] = pd.to_numeric(df['price'])

    plt.figure(figsize=(10, 6))

    for product in df['name'].unique():
        product_data = df[df['name'] == product]
        plt.plot(product_data['date'], product_data['price'], marker='o', label=product)

    plt.title("Crate & Barrel Price Trends")
    plt.xlabel("Date")
    plt.ylabel("Price ($)")
    plt.legend(loc='upper left', bbox_to_anchor=(1, 1))
    plt.grid(True, linestyle='--', alpha=0.7)
    plt.tight_layout()

    plt.savefig("price_dashboard.png")
    plt.show()

if __name__ == "__main__":
    generate_dashboard()

This script groups data by product name. If you track three items, you’ll see three lines. When Crate & Barrel launches a sale, a sharp dip in the line chart is your signal to buy.

Practical Tips

Anti-Bot Protection: Crate & Barrel uses advanced detection. Running this script without a proxy or with basic requests will likely result in a 403 Forbidden error. Playwright, playwright-stealth, and ScrapeOps residential proxies are necessary for reliable monitoring.
Handling Out-of-Stock: The base scraper extracts an availability field. You can modify monitor.py to record when an item is out of stock, as prices often change when items are replenished.
Data Integrity: If Crate & Barrel changes their page layout, the CSS selectors might break. Periodically check price_history.csv to ensure prices aren't returning as 0.0 or None.

To Wrap Up

A custom price monitor provides a data-driven advantage. Instead of relying on luck, you can track the actual market behavior of the furniture you want.

Key Takeaways:

Playwright handles dynamic content and JSON-LD extraction better than static scrapers.
ScrapeOps bypasses anti-bot walls that would otherwise block automated checks.
Pandas simplifies turning a CSV into a time-series analysis.

Try expanding this project by adding an email notification system using Python's smtplib. You can set a target price in products.py and have the script email you the moment the price hits your threshold.

How to Repair Broken Wayfair Scrapers Instantly Using AI-Generated Selectors

Robert N. Gutierrez — Mon, 30 Mar 2026 11:27:47 +0000

Maintaining a Wayfair scraper often feels like a full-time job. If you’ve ever inspected a Wayfair product page, you’ve seen the "class name soup": dynamic, obfuscated strings like sc-12345 or _1a7uksta that change almost every time the site deploys a new build.

For developers, this means a CSS selector that worked perfectly yesterday might return null today. Traditionally, fixing this involves manually inspecting the new DOM, finding the new path to the price or product title, and pushing a code update. This manual cycle is slow, expensive, and fragile.

There is a better way. By using the architecture found in the Wayfair Scraper Bank to decouple extraction logic, you can use the ScrapeOps AI Scraper Generator to swap out broken code in seconds.

The Anatomy of a Breakage

When a scraper breaks on a site like Wayfair, it rarely crashes the whole script. Instead, you experience DOM Drift. The page still loads, your proxies are working, but your data object comes back empty:

{
  "name": null,
  "price": 0.0,
  "brand": ""
}

Wayfair frequently updates its structural hierarchy. A product title once held in an h1 with a specific class might move into a div or a new section. Because Wayfair uses auto-generated CSS classes, targeting them is a losing game.

To survive these updates, you need a workflow that detects these "soft failures" and allows you to replace the parsing logic without touching your browser or proxy configuration.

Step 1: Decoupling Extraction from Crawling

The first step to a self-healing scraper is modularity. If browser navigation, proxy rotation, and data parsing are all tangled in one giant function, fixing a single selector requires a full redeploy of your infrastructure.

The Wayfair Playwright Scraper in the repository uses a clean separation of concerns. The browser logic handles how to get to the page, while a standalone extract_data function handles what to collect.

# Simplified architecture
async def extract_data(page: Page) -> Optional[ScrapedData]:
    # This is the Extraction Layer.
    # If Wayfair updates their DOM, only change this function.
    name = await page.locator("h2[data-test-id='ListingCardName']").inner_text()
    return ScrapedData(name=name, ...)

async def main():
    # This is the Infrastructure Layer.
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page()
        await page.goto("https://www.wayfair.com/...")

        # Pass the page object to the extraction layer
        data = await extract_data(page)

By isolating the extraction logic, you create a plug-and-play system. When the DOM changes, you don't need to fix the scraper; you just generate a new extract_data function.

Step 2: Implementing Automated Validation

You can't fix what you don't know is broken. To automate repairs, add a validation check at the end of the pipeline to flag any item missing critical fields.

In the repository's DataPipeline class, you can implement a simple check:

# Inside the main loop or DataPipeline class
data = await extract_data(page)

if not data or not data.name or data.price == 0:
    logger.error(f"Validation Failed for {url}. Selectors may be obsolete.")

    # Save the HTML to feed to the AI for a fix
    html_snapshot = await page.content()
    with open("failed_page.html", "w") as f:
        f.write(html_snapshot)

    # Trigger an alert or move to the repair workflow

By catching these errors programmatically, you shift from reactive maintenance to a proactive approach.

Step 3: The Hot-Swap Workflow

Once you identify that extract_data is failing, use the ScrapeOps AI Scraper Generator to generate a fix.

Fetch the HTML: Use the failed_page.html file saved in Step 2.
Generate: Upload the HTML to the ScrapeOps AI tool.
Select Framework: Choose the language that matches your implementation, such as Python Playwright or Node.js Cheerio.
The Fix: The AI analyzes the new Wayfair DOM and provides a fresh extract_data function.

The AI is particularly effective at spotting Wayfair’s JSON-LD data, which is a hidden script tag containing structured data. This is far more stable than CSS classes.

Example: Before and After

Old code might have targeted a specific class:

# Old, broken selector
name = await page.locator(".ProductTitle-123").inner_text()

The AI-generated fix often pivots to the more robust JSON-LD pattern:

# New, robust logic
json_content = await page.locator("script[type='application/ld+json']").text_content()
data = json.loads(json_content)
name = data.get("name")

Step 4: Applying the Fix

Because the repository uses a ScrapedData dataclass, the rest of your pipeline expects a specific format. When you paste the new AI-generated function into your script, the rest of your code remains untouched as long as the function returns that ScrapedData object.

Here is how to apply the fix in wayfair_scraper_product_data_v1.py:

# Replace the old extract_data function with the AI-generated version

async def extract_data(page: Page) -> Optional[ScrapedData]:
    # AI detected Wayfair moved data to a new schema
    # ... new logic here ...
    return ScrapedData(
        name=new_name_logic,
        price=new_price_logic,
        productId=new_id_logic,
        url=page.url
    )

Recommended Approaches for Wayfair

While the hot-swap workflow is fast, you can reduce the frequency of breakages by following the patterns established in the repository:

Prioritize JSON-LD: Always check for application/ld+json first. Wayfair uses this for SEO, making it less likely to change than visual CSS classes.
Use Regex for Prices: Prices on Wayfair can appear in various formats like "$10.00" or "USD 10". Use the repository's clean_price and detect_currency helpers to handle these variations consistently.
Handle Soft Anti-Bot: Wayfair sometimes serves a simplified layout to suspected bots. Using ScrapeOps Residential Proxies helps ensure you see the same DOM as a regular user.

To Wrap Up

Maintenance doesn't have to be the bottleneck of a scraping project. By adopting a modular architecture and using AI for selector repair, you can turn a tedious debugging session into a quick update.

Key Takeaways:

Isolate logic: Keep browser navigation separate from HTML parsing.
Validate data: Log errors when critical fields return empty.
Use AI: Let the ScrapeOps AI Scraper Generator write selectors for you when the site updates.

Stop Silent Scraper Failures: Using Pydantic for Instant Layout Change Detection

Robert N. Gutierrez — Tue, 17 Mar 2026 10:44:14 +0000

You’ve likely experienced the "Monday Morning Surprise." You check your database after a weekend of automated scraping only to find thousands of new rows where the price column is empty, the product_name is "None," and the stock_count is zero.

The script didn't crash, your proxies worked perfectly, and the status codes were all 200 OK. But because the website owner changed a single CSS class from .price-value to .item-price, your scraper spent 48 hours collecting digital garbage.

In the world of Scraper Reliability Engineering (SRE), this is a silent failure. While traditional error handling focuses on network stability, Pydantic allows you to treat scraped data like a strict API contract. This approach detects layout changes the second they happen, ensuring your data pipeline remains untainted.

Why `try/except` Isn't Enough

Most developers write defensive scrapers, wrapping extraction logic in try/except blocks to prevent the process from crashing when a single element is missing. While this keeps the script running, it often leads to silent data corruption.

Consider this common pattern:

def extract_product(element):
    try:
        return {
            "name": element.query_selector(".title").inner_text(),
            "price": float(element.query_selector(".price").inner_text().replace("$", ""))
        }
    except Exception:
        return None # The silent killer

If the .price selector fails, this function returns None. The main loop sees the None, skips the entry, and continues. You’ve lost data and won't know why until you manually inspect the logs days later. You need to move away from "just don't crash" and toward "fail fast when the contract is broken."

Step 1: Defining the Data Contract with Pydantic

To build a reliable scraper, you must first define what "good data" looks like. Pydantic is a data validation library that uses Python type hints to enforce schemas. Instead of working with raw dictionaries, you map scraped data to a BaseModel.

Using Product Hunt as an example, we can define a model for the product name, upvote count, and primary tags.

from pydantic import BaseModel, Field
from typing import List

class ProductHuntItem(BaseModel):
    name: str = Field(min_length=1)
    upvotes: int = Field(ge=0) # Must be an integer >= 0
    tags: List[str]
    rank: int

By defining this model, you’ve created a data contract. If you try to pass a string like "1.2k" into upvotes, which expects an int, Pydantic will immediately raise a ValidationError.

Step 2: Integrating Validation with Playwright

When integrating this into a scraper using Playwright, the goal is to extract raw data into a dictionary and immediately attempt to instantiate the Pydantic model.

import asyncio
from playwright.async_api import async_playwright
from pydantic import ValidationError

async def scrape_product_hunt():
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page()
        await page.goto("https://www.producthunt.com/")

        products = await page.query_selector_all('[data-test="post-item"]')
        valid_items = []

        for index, product in enumerate(products[:10]):
            try:
                # 1. Extract raw data
                raw_data = {
                    "name": await product.query_selector('[data-test="post-name"]').inner_text(),
                    "upvotes": await product.query_selector('[data-test="vote-button"]').inner_text(),
                    "tags": [await product.query_selector('[data-test="post-tag"]').inner_text()],
                    "rank": index + 1
                }

                # 2. Validate with Pydantic
                item = ProductHuntItem(**raw_data)
                valid_items.append(item)

            except ValidationError as e:
                print(f"Layout Change Detected at item {index}: {e.json()}")
            except Exception as e:
                print(f"Generic extraction error: {e}")

        await browser.close()
        return valid_items

if __name__ == "__main__":
    asyncio.run(scrape_product_hunt())

In this workflow, you extract data as a "dirty" dictionary first. When you call ProductHuntItem(**raw_data), Pydantic performs the type checking. If the website changes its structure, such as the upvote button containing text like "Free" instead of a number, the script catches it immediately.

Step 3: Catching the Drift

Imagine Product Hunt updates their UI. The upvote count, which used to be a simple integer in the HTML, is now formatted as "2,450".

A standard Python int() conversion would fail, and a basic scraper might just store None. Pydantic provides a clear error trace instead:

{
  "type": "int_parsing",
  "loc": ["upvotes"],
  "msg": "Input should be a valid integer, unable to parse string as an integer",
  "input": "2,450"
}

This is shift-left testing for data extraction. You catch the error at the point of ingestion rather than weeks later during analysis.

Step 4: Advanced Cleaning with Validators

Websites are messy. Sometimes a layout change doesn't mean the data is gone, but that the format has shifted. Pydantic's @field_validator allows you to fix minor issues automatically before validation.

This updated model handles commas in numbers and "k" suffixes:

from pydantic import BaseModel, Field, field_validator

class ProductHuntItem(BaseModel):
    name: str
    upvotes: int

    @field_validator('upvotes', mode='before')
    @classmethod
    def clean_upvotes(cls, v: any) -> int:
        if isinstance(v, str):
            v = v.replace(',', '').strip().lower()
            if 'k' in v:
                return int(float(v.replace('k', '')) * 1000)
            return int(v)
        return v

By using mode='before', you intercept the raw data before Pydantic tries to force it into an integer. This makes the scraper resilient to cosmetic changes while maintaining a strict floor for data quality. If the input is "Add to Cart," the validator will fail and alert you to a total layout shift.

Step 5: The Error Budget

In production web scraping, 100% success is rarely possible. A single item might fail validation because of a unique ad placement or a weird edge case. You don't want to wake up an engineer for one failed row.

Instead, implement an error budget. Track the ratio of validation failures to successful extractions.

class ScrapeSession:
    def __init__(self):
        self.success_count = 0
        self.failure_count = 0
        self.threshold = 0.15  # 15% error budget

    def log_result(self, success: bool):
        if success:
            self.success_count += 1
        else:
            self.failure_count += 1

    def check_reliability(self):
        total = self.success_count + self.failure_count
        if total == 0: return

        error_rate = self.failure_count / total
        if error_rate > self.threshold:
            self.trigger_alert(error_rate)

    def trigger_alert(self, rate):
        print(f"CRITICAL: Scraper reliability dropped to {1-rate:.2%}. Layout change likely.")

# Usage in loop
session = ScrapeSession()
try:
    ProductHuntItem(**raw_data)
    session.log_result(True)
except ValidationError:
    session.log_result(False)

session.check_reliability()

If 2% of items fail, it is likely noise. If 20% fail, the website has likely changed its DOM structure and your error budget is spent. This is the moment to trigger a Slack notification or a PagerDuty alert.

Summary

Moving from basic scripts to Scraper Reliability Engineering requires a mindset shift. By treating scraped data as a typed contract rather than a loose collection of strings, you protect downstream applications from garbage data.

Fail Fast: Use Pydantic to catch layout changes during extraction.
Define Contracts: Create models that represent the ideal state of your data.
Automate Cleaning: Use Pydantic validators to handle predictable noise like currency symbols.
Monitor Ratios: Use an error budget to distinguish between minor glitches and systemic layout changes.

Implementing these patterns ensures you spend less time cleaning broken databases and more time building features with data you can trust.

How to Build an AliExpress Price Tracker with Python and Playwright

Robert N. Gutierrez — Sat, 14 Mar 2026 12:00:00 +0000

AliExpress is a goldmine for competitive pricing, but those prices are notoriously volatile. Between "Choice Day" deals, flash sales, and dynamic pricing based on inventory, a product that costs $20 today might be $12 tomorrow or $30 next week. For dropshippers or market analysts, manual price checking is a losing game.

Automating this process is the only way to stay competitive. By building a custom price monitor, you can track competitor SKUs daily, identify genuine discounts versus "fake" sales, and build a historical dataset for better purchasing decisions.

This guide demonstrates how to build a production-ready price monitoring pipeline using Python, Playwright, and the open-source Aliexpress.com-Scrapers repository. We will move from raw data extraction to a fully automated system that visualizes price trends over time.

Prerequisites

To follow along, you’ll need a few things set up in your environment:

Python 3.8+ installed on your machine.
A ScrapeOps API Key: AliExpress uses aggressive anti-bot measures. We will use the ScrapeOps Proxy to bypass CAPTCHAs and blocks. You can get a free API key here.
Basic Python Knowledge: You should be comfortable with lists, dictionaries, and installing packages via pip.

Setup: Cloning the Scraper Engine

Rather than writing a scraper from scratch, we can use the Aliexpress.com-Scrapers repository. This repo contains optimized selectors and logic for handling the complex AliExpress DOM.

First, clone the repository and navigate to the Playwright implementation:

git clone https://github.com/scraper-bank/Aliexpress.com-Scrapers.git
cd Aliexpress.com-Scrapers/python/playwright/product_data

Next, install the required dependencies:

pip install playwright playwright-stealth pandas matplotlib schedule
playwright install chromium

Step 1: Understanding the Scraper Logic

Before modifying the code, let's look at how the core engine works. Inside scraper/aliexpress_scraper_product_data_v1.py, there is a ScrapedData dataclass. This acts as the blueprint for the extracted information:

@dataclass
class ScrapedData:
    name: str = ""
    price: Optional[Any] = None
    preDiscountPrice: Optional[Any] = None
    productId: str = ""
    url: str = ""
    # ... other fields

The script uses Playwright because AliExpress relies heavily on JavaScript to render prices. A simple requests call often returns empty tags, but Playwright loads the full browser environment. It also integrates with the ScrapeOps Proxy to rotate residential IPs, which prevents the monitor from being blacklisted during extended operation.

Step 2: Customizing for Price Monitoring

To turn this from a one-time scraper into a monitor, we need to add timestamps and persistent storage. We want to append every new price check to a single file rather than overwriting it.

Modify the ScrapedData class and the DataPipeline in a new file called monitor_engine.py:

from datetime import datetime
from dataclasses import dataclass, asdict, field
import json

@dataclass
class ScrapedData:
    name: str = ""
    price: float = 0.0
    preDiscountPrice: float = 0.0
    productId: str = ""
    url: str = ""
    # Add a timestamp so we can plot data over time
    scraped_at: str = field(default_factory=lambda: datetime.now().isoformat())

class PriceHistoryPipeline:
    def __init__(self, filename="price_history.jsonl"):
        self.filename = filename

    def save(self, data: ScrapedData):
        with open(self.filename, "a", encoding="utf-8") as f:
            f.write(json.dumps(asdict(data)) + "\n")

Using JSONL (JSON Lines) format ensures the data is streamable. If you track 100 products for a year, the file will grow large. JSONL allows you to read it line-by-line without loading the entire history into memory.

Step 3: Automating the Scan

Now we need a wrapper script to run the scraper against a specific list of competitor URLs at a set interval. This example uses the schedule library to run a scan every day at 9:00 AM.

Create a file named run_monitor.py:

import schedule
import time
import asyncio
from datetime import datetime
from playwright.async_api import async_playwright
from scraper.aliexpress_scraper_product_data_v1 import extract_data

# The URLs you want to track
COMPETITOR_URLS = [
    "https://www.aliexpress.com/item/100500123.html",
    "https://www.aliexpress.com/item/100500456.html"
]

async def track_prices():
    print(f"Starting price check at {datetime.now()}")
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()

        pipeline = PriceHistoryPipeline()

        for url in COMPETITOR_URLS:
            try:
                # Use the extraction logic from the repo
                data = await extract_data(page, url)
                if data:
                    pipeline.save(data)
                    print(f"Tracked: {data.name[:30]}... - ${data.price}")
            except Exception as e:
                print(f"Error tracking {url}: {e}")

        await browser.close()

# Schedule the job
schedule.every().day.at("09:00").do(lambda: asyncio.run(track_prices()))

while True:
    schedule.run_pending()
    time.sleep(60)

Step 4: Visualizing the Data

Raw JSON files are useful for storage, but trends are easier to spot visually. Use Pandas to clean the data and Matplotlib to generate a price trend graph.

The scraper often returns prices as strings, such as "$12.50," so these must be cleaned before plotting.

import pandas as pd
import matplotlib.pyplot as plt

def plot_trends():
    # Load the JSONL file
    df = pd.read_json('price_history.jsonl', lines=True)

    # Convert scraped_at to datetime objects
    df['scraped_at'] = pd.to_datetime(df['scraped_at'])

    # Ensure price is a float (remove currency symbols if present)
    if df['price'].dtype == object:
        df['price'] = df['price'].str.replace(r'[^\d.]', '', regex=True).astype(float)

    # Pivot data: Rows = Date, Columns = Product Name, Values = Price
    pivot_df = df.pivot(index='scraped_at', columns='name', values='price')

    # Plotting
    pivot_df.plot(figsize=(10, 6), marker='o')
    plt.title('AliExpress Competitor Price Trends')
    plt.ylabel('Price (USD)')
    plt.xlabel('Date')
    plt.grid(True)
    plt.legend(loc='upper left', prop={'size': 8})
    plt.savefig('price_trends.png')
    plt.show()

if __name__ == "__main__":
    plot_trends()

This script generates a multi-line chart where each line represents a product. This makes it easy to see exactly when a competitor lowered their price or when a "limited time" sale actually ended.

Common Obstacles

When monitoring AliExpress at scale, you will likely encounter these three issues:

Selector Drift: AliExpress frequently updates its CSS classes, changing .product-price to something like .price--currentPrice--123. If the scraper returns None, inspect the page in your browser and update the selectors in extract_data.
Currency Mismatches: Depending on the proxy location, AliExpress might show prices in EUR or GBP. Force the currency by appending ?cur=USD to your target URLs in COMPETITOR_URLS.
Bot Detection: If failure rates increase, check the ScrapeOps dashboard. You may need to enable render_js: True in your proxy settings to handle advanced "Slide to Verify" CAPTCHAs.

To Wrap Up

Building a custom price tracker provides a data-driven edge in the e-commerce market. By combining the extraction logic of the Aliexpress.com-Scrapers repository with a simple automation loop, you can move from manual guesswork to automated intelligence.

Key Takeaways:

Playwright is necessary for handling AliExpress's dynamic JavaScript rendering.
JSONL is a reliable format for long-term time-series data storage.
Proxies are required to avoid persistent blocking; residential rotation is the recommended approach.

As a next step, consider adding a notification layer using smtplib or a Discord Webhook. Setting a threshold, such as "Alert me if Product A drops below $15," makes the monitor truly proactive. For more advanced scraping patterns, explore the other implementations in the ScrapeOps Scraper Bank.

How to Export Depop Listings to a Shopify CSV Using Python

Robert N. Gutierrez — Sun, 08 Mar 2026 15:19:00 +0000

For many resellers, moving from a marketplace like Depop to a dedicated Shopify store is a major milestone. However, growth often hits a technical wall: inventory migration. Manually copy-pasting titles, descriptions, and high-resolution images for hundreds of items is a bottleneck that prevents you from scaling.

This guide solves that problem by building an automated pipeline. We’ll use Python to scrape your Depop listings, transform that data into a Shopify-compatible format using Pandas, and generate a CSV file ready for instant import.

Prerequisites

Before starting, ensure you have the following:

Python 3.8+ installed on your machine.
A ScrapeOps API Key: Depop uses aggressive anti-bot measures. We use the ScrapeOps proxy aggregator to bypass these blocks. You can sign up for a free ScrapeOps account here.
Pandas & Playwright: These libraries handle data manipulation and browser automation.

Setup Instructions

Clone the Depop.com-Scrapers repository, which contains the core scraping logic:

git clone https://github.com/scraper-bank/Depop.com-Scrapers.git
cd Depop.com-Scrapers/python/playwright
pip install playwright playwright-stealth pandas
playwright install chromium

Step 1: Extracting Product Data from Depop

To get the data, we’ll use the Playwright implementation from the repository. While the base script is designed for single URLs, we can wrap it to process multiple listings.

Create a script named run_scraper.py. This script iterates through your Depop product URLs and saves the raw data into a JSONL (JSON Lines) file. JSONL is efficient because it allows us to append data line-by-line, preventing data loss if the script is interrupted.

import asyncio
from playwright.async_api import async_playwright
from scraper.depop_scraper_product_data_v1 import extract_data, DataPipeline

# Replace with your ScrapeOps API Key
API_KEY = "YOUR_SCRAPEOPS_API_KEY"

async def run_export(urls):
    async with async_playwright() as p:
        # Use a headless browser with stealth to mimic a real user
        browser = await p.chromium.launch(headless=True)
        pipeline = DataPipeline(jsonl_filename="depop_raw_data.jsonl")

        for url in urls:
            page = await browser.new_page()
            try:
                print(f"Scraping: {url}")
                await page.goto(url, wait_until="domcontentloaded", timeout=60000)
                # Wait 2 seconds for dynamic images to load
                await asyncio.sleep(2) 

                data = await extract_data(page)
                if data:
                    pipeline.add_data(data)
            except Exception as e:
                print(f"Error scraping {url}: {e}")
            finally:
                await page.close()

        await browser.close()

if __name__ == "__main__":
    my_listings = [
        "https://www.depop.com/products/example-item-1/",
        "https://www.depop.com/products/example-item-2/"
    ]
    asyncio.run(run_export(my_listings))

Step 2: Mapping the Data

Depop’s data structure differs significantly from Shopify’s import requirements. We need to map the fields carefully.

Depop Field (JSONL)	Shopify CSV Header	Transformation Needed
`name`	`Title`	Direct map
`name`	`Handle`	URL-friendly slug (e.g., "Vintage Red Tee" becomes `vintage-red-tee`)
`description`	`Body (HTML)`	Convert plain text to basic HTML line breaks
`price`	`Variant Price`	Ensure numeric format
`images`	`Image Src`	Extract the first URL from the image list

Shopify also requires specific columns like Vendor, Published, and Status. We will add these as constants in our transformation script.

Step 3: The Transformation Script

Create convert_to_shopify.py. This script uses Pandas to load the JSONL file, apply the mapping logic, and export the final CSV.

import pandas as pd
import json
import re

def slugify(text):
    """Converts a title into a URL-friendly handle."""
    text = text.lower()
    return re.sub(r'[^\w\s-]', '', text).strip().replace(' ', '-')

def format_html(text):
    """Converts plain text description to basic HTML."""
    if not text: return ""
    return text.replace('\n', '<br>')

def transform_depop_to_shopify(input_file, output_file):
    items = []
    with open(input_file, 'r', encoding='utf-8') as f:
        for line in f:
            items.append(json.loads(line))

    df = pd.DataFrame(items)
    shopify_df = pd.DataFrame()

    # Core Mapping Logic
    shopify_df['Handle'] = df['name'].apply(slugify)
    shopify_df['Title'] = df['name']
    shopify_df['Body (HTML)'] = df['description'].apply(format_html)
    shopify_df['Vendor'] = 'My Depop Shop'
    shopify_df['Type'] = df['category']
    shopify_df['Tags'] = df['brand']
    shopify_df['Published'] = 'TRUE'

    # Pricing & Inventory
    shopify_df['Variant Price'] = df['price']
    shopify_df['Variant Inventory Tracker'] = 'shopify'
    shopify_df['Variant Inventory Qty'] = 1
    shopify_df['Variant Fulfillment Service'] = 'manual'

    # Image Handling: Grab the first image URL
    shopify_df['Image Src'] = df['images'].apply(lambda x: x[0]['url'] if (isinstance(x, list) and len(x) > 0) else '')

    shopify_df['Status'] = 'active'

    # Export to CSV with UTF-8 encoding for emojis
    shopify_df.to_csv(output_file, index=False, encoding='utf-8-sig')
    print(f"Successfully exported {len(shopify_df)} items to {output_file}")

if __name__ == "__main__":
    transform_depop_to_shopify('depop_raw_data.jsonl', 'shopify_import.csv')

Why Pandas?

Pandas handles missing data gracefully. If a Depop listing is missing a brand or category, you can easily fill those gaps with defaults like "Unbranded" or "Clothing" using .fillna() or .apply() without the script crashing.

Step 4: Handling Images and Variations

A common hurdle in migration is high-resolution images. The Playwright scraper is tuned to find the src attributes pointing to Depop's CDN.

If you have multiple images per product, Shopify requires a separate row for each additional image. In these rows, the Handle remains the same, but other columns are left blank. While this script grabs the primary image, you can expand it using df.explode('images') if you need every photo transferred.

Step 5: Importing into Shopify

Once the shopify_import.csv is ready, follow these steps:

Log in to your Shopify Admin.
Go to Products and click Import.
Upload your shopify_import.csv.
If you are updating existing products with the same handle, check the box to Overwrite any current products.
Preview the import. Verify that the Title, Price, and Image URL are correctly mapped.
Click Import Products.

Troubleshooting

403 Forbidden Errors: If Depop blocks your scraper, check your ScrapeOps API key. Depop is highly sensitive to IP reputation, so using a proxy is necessary.
Missing Handles: Shopify will fail if a Handle is empty. If you have non-Latin titles, ensure the slugify function produces a valid string.
Character Encoding: If your descriptions contain emojis, the encoding='utf-8-sig' argument in the Pandas to_csv function prevents "garbage" characters from appearing in Shopify.

To Wrap Up

By automating the extraction and transformation of your listings, you turn a multi-day manual task into a 60-second process. You can further improve this by using a profile scraper to automatically gather all listing URLs, creating a completely hands-off migration tool.

From Scrape to Feed: Building a Google Merchant Center CSV from Zappos Data

Robert N. Gutierrez — Fri, 06 Mar 2026 13:32:00 +0000

Scraping product data is often only the first half of a larger business objective. Whether you are building a price comparison engine, a dropshipping store, or a market research tool, raw data is rarely useful in its initial state. If you want to list products on Google Shopping, you face a significant hurdle: Google Merchant Center (GMC) has incredibly strict formatting requirements.

Raw scraped data from Zappos typically arrives as a nested JSON or JSONL file. Google expects a clean CSV or XML feed with specific headers like availability, image_link, and price formatted exactly to their specifications.

This guide bridges that gap. We’ll use a production-ready Zappos scraper to extract data and then build a Python transformation pipeline using pandas to convert that raw data into a valid, GMC-compliant product feed.

Prerequisites & Setup

Ensure you have Python installed on your machine. We will use the playwright library for extraction and pandas for data transformation.

First, clone the Zappos.com-Scrapers repository, which contains the logic needed to handle Zappos' complex page structure.

git clone https://github.com/scraper-bank/Zappos.com-Scrapers.git
cd Zappos.com-Scrapers
pip install pandas playwright playwright-stealth
playwright install chromium

You will also need a ScrapeOps API Key. Zappos employs advanced anti-bot measures that can block standard requests. ScrapeOps provides proxy rotation and request optimization to ensure your scrapers reach the page. You can sign up for a free API key here.

Step 1: Extracting Raw Data from Zappos

We will use the Playwright implementation found in the repository. This script navigates to a product page, handles dynamic content loading, and extracts structured data into a JSONL (JSON Lines) format.

Open python/playwright/product_data/scraper/zappos_scraper_product_data_v1.py, insert your API key into the API_KEY variable, and run the scraper:

cd python/playwright/product_data/scraper
python zappos_scraper_product_data_v1.py

The Raw Output

The scraper produces a zappos_com_product_page_scraper_data_[timestamp].jsonl file. Each line looks something like this:

{
  "name": "Nike Air Zoom Pegasus 40",
  "productId": "9848135",
  "price": 129.95,
  "availability": "in_stock",
  "images": [{"url": "https://m.media-amazon.com/images/I/71...jpg", "alt_text": ""}],
  "url": "https://www.zappos.com/p/nike-air-zoom..."
}

While this is structured, it isn't ready for Google. The price is a float without a currency code, the images are trapped in a list, and the availability format doesn't match Google's requirements.

Step 2: Gap Analysis (JSONL vs. GMC Requirements)

To build a successful feed, we must map our scraped fields to the Google Product Data Specification. Here is how the raw data stacks up against Google's requirements:

Field	Scraped Data Example	Google Requirement	Transformation
ID	`9848135`	Alphanumeric ID	Rename to `id`
Title	`Nike Air Zoom...`	Max 150 chars	Rename to `title`
Price	`129.95`	`129.95 USD`	Format as string + currency
Availability	`in_stock`	`in stock`	Replace underscore with space
Image Link	List of objects	Single URL string	Extract first URL in list
Condition	Missing	`new`, `refurbished`, `used`	Add default value `new`

Step 3: Building the Transformation Script

Create a new Python script named build_feed.py in your project root. We’ll use pandas to load the JSONL file and clean the data.

Loading and Mapping

First, load the data and rename the basic columns to match Google’s headers.

import pandas as pd
import json

# 1. Load the JSONL data
data = []
with open('zappos_data.jsonl', 'r') as f:
    for line in f:
        data.append(json.loads(line))

df = pd.DataFrame(data)

# 2. Basic Column Mapping
df = df.rename(columns={
    'productId': 'id',
    'name': 'title',
    'url': 'link'
})

Formatting Challenges

Next, we need to handle price and image_link. Google is very particular about these formats.

# 3. Format Price (Google requires: [Price] [Currency])
df['price'] = df['price'].apply(lambda x: f"{float(x):.2f} USD")

# 4. Extract Image Link
# Zappos provides a list of objects; Google wants a single primary URL string.
def get_main_image(image_list):
    if isinstance(image_list, list) and len(image_list) > 0:
        return image_list[0].get('url', '')
    return ''

df['image_link'] = df['images'].apply(get_main_image)

# 5. Map Availability
# Google expects 'in stock' or 'out of stock' without underscores
availability_map = {
    "in_stock": "in stock",
    "out_of_stock": "out of stock"
}
df['availability'] = df['availability'].map(availability_map).fillna('out of stock')

# 6. Add Static Required Fields
df['condition'] = 'new'

Step 4: Exporting the Final Feed

Finally, select only the columns Google Merchant Center expects and export the file. While Google accepts Tab-Separated Values (TSV) and XML, CSV is the easiest format for smaller feeds.

# Select only the required columns
final_columns = [
    'id', 'title', 'description', 'link', 
    'image_link', 'availability', 'price', 'condition', 'brand'
]

# Ensure we only export what's needed
final_df = df[final_columns]

# Export to CSV
final_df.to_csv('google_merchant_feed.csv', index=False)

print(f"Successfully converted {len(final_df)} products to google_merchant_feed.csv")

When you open google_merchant_feed.csv in Excel or Google Sheets, you will see a formatted table ready for upload.

Practical Tips and Gotchas

When running this pipeline at scale, keep these factors in mind:

Character Limits: Google truncates titles after 150 characters. Zappos titles are usually short, but if you scrape descriptions, make sure they stay under Google's 5,000-character limit.
Image Quality: The ScrapeOps Zappos scraper is configured to find high-resolution images. In the extract_data function within the repo, a regex replaces thumbnail dimensions (e.g., _AC_SR1840,1472_) to ensure Google receives high-quality assets.
GTINs and MPNs: For "Buy on Google" or highly competitive categories, Google requires GTINs (UPCs). These aren't always visible on the Zappos storefront, but they can sometimes be found in the __INITIAL_STATE__ JSON blob extracted by the scraper.

To Wrap Up

Converting raw scraped data into a platform-specific feed is a vital skill for e-commerce data engineers. By combining the extraction power of the ScrapeOps Zappos scraper with the data manipulation strengths of pandas, you can automate the entire path from a web page to a live product ad.

Key Takeaways:

Raw data is just a starting point: Always perform a gap analysis between your scraped output and your target platform's requirements.
Cleaning is essential: Use Python to handle currency formatting, image URL extraction, and specific status mapping.
Don't reinvent the wheel: Use the production-ready implementations in the Zappos.com-Scrapers repository to save development time.

Bulletproof Data Pipelines: Adding Schema Validation to Nike Scrapers

Robert N. Gutierrez — Wed, 04 Mar 2026 01:00:00 +0000

Imagine running a large-scale scraper for days, only to discover that 40% of your dataset has a price of 0 or a null product name. The script didn't crash, your logs show "200 OK," and your database is growing—but the data is worthless.

In web scraping, this is a soft break. Unlike a hard crash, where a changed HTML selector throws an error and stops the script, a soft break happens when extraction logic fails silently, returning empty strings or default values instead of real data.

This guide explains how to prevent these silent failures by adding strict schema validation to the Nike.in Scrapers repository. We’ll use Zod for Node.js and Pydantic for Python to transform fragile scrapers into reliable data pipelines.

Prerequisites

To follow along, you'll need:

A basic understanding of Node.js or Python.
The Nike.in Scrapers repository cloned locally.
Node.js (v16+) or Python (3.8+) installed.

The Anatomy of a Soft Break

The current implementation in the repository is vulnerable because it lacks strict checks. If we look at the Node.js Puppeteer scraper at node/puppeteer/product_data/scraper/nike_scraper_product_data_v1.js, we see a common pattern in the extractData function:

const parsePrice = (priceText) => {
    const match = priceText.match(/[\d,.]+/);
    if (!match) return 0; // Defaulting to 0 if regex fails
    const clean = match[0].replace(/,/g, '');
    return parseFloat(clean) || 0;
};

If Nike changes their price HTML class from .css-nqh3vz to something else, priceText becomes an empty string. The regex fails, and the function returns 0.

To your database, 0 is a valid number. To your business logic, a Nike shoe costing $0 is a critical error. Without validation, your JSONL output ends up looking like this:

{"name": "Nike Air Max", "price": 0, "url": "https://nike.in/...", "availability": "in_stock"}
{"name": "", "price": 12995, "url": "https://nike.in/...", "availability": "in_stock"}

This data is valid JSON, but it's invalid information.

Strategy: Validation at the Pipeline Level

Avoid cluttering extraction logic with hundreds of if/else checks. Instead, treat the DataPipeline class as a gatekeeper.

The repository architecture uses a DataPipeline to handle deduplication and file writing. By inserting a validation step here, we ensure that no garbage data ever touches persistent storage.

The Flow:
Scraper -> Extractor -> Validator (New) -> Writer

If the data fails validation, we log the error and drop the record. If too many records fail, we can trigger a "Circuit Breaker" to stop the scraper entirely.

Implementation 1: Node.js with Zod

Zod is a schema declaration and validation library. It’s ideal for Node.js scrapers because it allows us to define the "Product" structure in a single, reusable schema.

1. Install Zod

Navigate to your Node.js project folder and install the package:

npm install zod

2. Define the Schema

In node/puppeteer/product_data/scraper/nike_scraper_product_data_v1.js, define a schema that mirrors the expected output. This ensures the price is a positive number and the productId is not empty.

const { z } = require('zod');

const ProductSchema = z.object({
    productId: z.string().min(1, "Product ID is missing"),
    name: z.string().min(3, "Product name is too short"),
    price: z.number().positive("Price must be greater than 0"),
    currency: z.string().length(3),
    url: z.string().url(),
    availability: z.enum(["in_stock", "out_of_stock"])
});

3. Update the DataPipeline

Modify the addData method in the DataPipeline class to use this schema.

class DataPipeline {
    constructor(outputFile = CONFIG.outputFile) {
        this.itemsSeen = new Set();
        this.outputFile = outputFile;
        this.errorCount = 0; 
    }

    async addData(scrapedData) {
        // Validate the data
        const validation = ProductSchema.safeParse(scrapedData);

        if (!validation.success) {
            this.errorCount++;
            console.error(`❌ Validation Failed for ${scrapedData.url}:`, validation.error.format());
            return; // Skip writing this item
        }

        if (!this.isDuplicate(validation.data)) {
            const jsonLine = JSON.stringify(validation.data) + '\n';
            await fs.promises.appendFile(this.outputFile, jsonLine, 'utf8');
            console.log('✅ Saved valid item:', validation.data.name);
        }
    }
}

Implementation 2: Python with Pydantic

For Python developers, Pydantic is the standard choice. The Python scrapers in the repo currently use standard dataclasses. We can upgrade these to Pydantic BaseModels for automatic validation.

1. Define the Pydantic Model

Replace the existing @dataclass with a Pydantic model. Pydantic handles type coercion automatically, such as converting a string "12995" to a float 12995.0.

from pydantic import BaseModel, Field, field_validator

class ScrapedData(BaseModel):
    productId: str = Field(min_length=1)
    name: str = Field(min_length=1)
    price: float = Field(gt=0)
    currency: str = Field(default="INR")
    url: str
    availability: str

    @field_validator('availability')
    @classmethod
    def validate_availability(cls, v: str) -> str:
        if v not in ['in_stock', 'out_of_stock']:
            return 'out_of_stock'
        return v

2. Integrate into the Pipeline

Update the add_data method to catch ValidationError.

class DataPipeline:
    def add_data(self, raw_data: dict):
        try:
            # Trigger validation logic
            validated_data = ScrapedData(**raw_data)

            data_dict = validated_data.model_dump()
            if not self.is_duplicate(data_dict):
                with open(self.jsonl_filename, mode="a", encoding="UTF-8") as f:
                    f.write(json.dumps(data_dict) + "\n")
                logger.info(f"Saved: {validated_data.name}")
        except Exception as e:
            logger.error(f"❌ Data Validation Error: {e}")

Error Budgets: The Circuit Breaker Pattern

Validation filters out single bad records, but what if the entire site layout changes? If 100% of your requests fail validation, you should stop the scraper to save on proxy costs and compute resources.

Implement a circuit breaker by tracking the failure rate. If the rate exceeds a specific threshold, such as 50% of the last several items, kill the process.

// Inside DataPipeline.addData (Node.js)
this.totalProcessed++;
if (!validation.success) {
    this.errorCount++;
    const failureRate = this.errorCount / this.totalProcessed;

    if (this.totalProcessed > 10 && failureRate > 0.5) {
        console.error("🚨 CRITICAL: Failure rate above 50%. Site layout likely changed. Aborting.");
        process.exit(1); 
    }
    return;
}

Closing the Loop: Regenerating Scrapers

When validation triggers a failure alert or trips the circuit breaker, your extraction logic is likely outdated.

Instead of manually hunting for new CSS selectors, use the ScrapeOps AI Scraper Generator. You can feed the failing Nike URL into the generator to receive updated selectors.

Validate: Catch the break immediately with Zod or Pydantic.
Detect: Trigger an alert when the failure rate spikes.
Regenerate: Use ScrapeOps to get new selectors and resume the pipeline.

Summary

Strict schema validation is what separates a simple script from a professional data pipeline. By moving validation into the DataPipeline class, you ensure high-quality data without overcomplicating your extraction logic.

Silent failures are more dangerous than crashes because they corrupt your dataset over time.
Zod and Pydantic enforce data types and business rules at runtime.
Circuit Breakers protect resources by stopping scrapers when site layouts change significantly.
Verify everything: Never let data hit your database without validation.

To practice, try cloning the Nike.in-Scrapers repository and adding Zod validation to the product_search scrapers to ensure every result contains a valid price.

The 5-Minute Repair: A Workflow for Surviving Site Redesigns

Robert N. Gutierrez — Sun, 01 Mar 2026 02:45:00 +0000

It’s Monday morning, and your Slack is already blowing up. The product data that usually flows into your database from Dermstore.com has suddenly dried up. You check the logs, and the scraper is still "working"—it connects, it scrolls, and it exits without error—but every product price is 0 and every name is an empty string.

Dermstore, like many major e-commerce platforms, frequently updates its frontend. A simple change from <span class="product-price"> to <div class="price-v2"> is all it takes to break a traditional scraper. Most developers respond by diving into a multi-hour session of manual DOM inspection and trial-and-error regex.

There is a better way. By using a modular "Hot-Swap" workflow, you can repair broken extraction logic in under five minutes without touching your networking, proxy, or browser automation infrastructure. This guide shows you how to use the Dermstore.com-Scrapers repository to maintain your data pipelines with minimal friction.

Prerequisites

To follow this workflow, you need:

A basic understanding of Node.js or Python.
A cloned copy of the repository: git clone https://github.com/scraper-bank/Dermstore.com-Scrapers.git
Familiarity with CSS selectors or Cheerio/Playwright locators.

Phase 1: Diagnosing the Break

Before fixing the scraper, you need to know exactly how it broke. Web scrapers typically fail in one of two ways: Network Blocks or Layout Changes.

If you see a 403 Forbidden or 429 Too Many Requests error in your console, your proxy or anti-bot headers are the problem. If the script runs successfully but produces "hollow" data, the site has redesigned its layout.

In the Dermstore repository, look at the output generated in your .jsonl files. A silent failure looks like this:

{
  "name": "",
  "price": 0,
  "brand": "",
  "url": "https://www.dermstore.com/example-product.production",
  "availability": "out_of_stock"
}

If the url is correct but the name and price are missing, the extraction logic is outdated. The scraper reached the page, but it couldn't read it.

Phase 2: The Bottleneck of Manual Debugging

The traditional fix is slow. You open the browser, right-click "Inspect," find the new class name, update your code, and run the whole script again. If Dermstore has heavy anti-bot protections, just loading the page to test your new selector can take a minute per attempt.

We solve this by isolating the "Extraction Block." In our repository, the code is split between Infrastructure (Puppeteer/Playwright setup, proxy rotation, retries) and Logic (the extractData function).

By treating the extraction function as a standalone "brain," you can swap it out without re-testing the entire body of the scraper.

Phase 3: The Hot-Swap Workflow

Let’s walk through the repair process using the Node.js Puppeteer implementation located at node/puppeteer/product_data/scraper/dermstore_scraper_product_data_v1.js.

1. Locate the Target Function

Open the script and find the extractData function. This is a "pure" function: it takes the HTML (via Cheerio) and returns a JSON object.

/**
 * Extract structured data from HTML using Cheerio
 */
function extractData($, url) {
    try {
        // Extraction logic lives here
        const priceText = $("#product-price").text(); 
        outputData.price = parsePrice(priceText);

        return outputData;
    } catch (error) {
        console.error('Error:', error);
        return null;
    }
}

2. Regenerate the Logic

Instead of manually hunting for selectors, use an AI scraper builder or the ScrapeOps AI Generator to generate a new extraction function based on the updated Dermstore URL.

3. The Hot-Swap

Once you have the new logic, do not replace your entire file. Copy only the internal logic of the new extractData function and paste it into your existing script.

By keeping the DataPipeline class and the puppeteer-extra configuration untouched, you ensure your proxy settings and duplicate detection remain active. You are simply upgrading the logic while keeping the engine running.

4. Handle Schema Changes

Dermstore often moves data into application/ld+json scripts. If the new site version uses JSON-LD, your swapped function should prioritize it, as shown in the repo's implementation:

let jsonData = null;
$("script[type='application/ld+json']").each((i, el) => {
    const data = JSON.parse($(el).text());
    if (data["@type"] === "Product") {
        jsonData = data; // Use the structured data if available
    }
});

Phase 4: Regression Testing with Example Data

How do you know your repair didn't change the data format? If your database expects price, but your new logic returns product_price, your pipeline will break further downstream.

The repository includes an example-data folder for every scraper. Use the product_data.json file as a baseline.

Run your repaired scraper on a single Dermstore URL.
Compare the keys in your new .jsonl output against node/puppeteer/product_data/example-data/product_data.json.
Ensure that aggregateRating, brand, and productId still exist and follow the same types. For example, price should be a float, not a string.

Phase 5: Python vs. Node.js Implementation

This workflow isn't limited to JavaScript. If you use the Playwright Python implementation at python/playwright/product_data/scraper/dermstore_scraper_product_data_v1.py, the pattern is identical.

The Python implementation uses an async def extract_data(page: Page) function. While the syntax differs, the modular philosophy remains:

Component	Node.js (Puppeteer)	Python (Playwright)
Automation	`puppeteer.launch()`	`async_playwright()`
Logic Block	`function extractData($, url)`	`async def extract_data(page)`
Data Output	`DataPipeline.addData()`	`DataPipeline.add_data()`

Because both implementations in the repo follow this structure, you can apply the hot-swap method regardless of your preferred language.

To Wrap Up

Web scraper maintenance doesn't have to be a chore. By isolating your extraction logic from your browser infrastructure, you turn a complex debugging session into a simple "swap and verify" procedure.

Key Takeaways:

Diagnose First: Distinguish between network blocks and selector breaks by checking for empty data versus error codes.
Modularize: Keep your extractData function separate from your proxy and browser setup.
Verify Schema: Use the repository's example-data to ensure your fix doesn't break your database downstream.
Automate Repairs: Use AI tools to generate selectors, then manually hot-swap them into your proven infrastructure.

To get started, clone the Dermstore Scrapers repository and run the existing scripts. When the next site update hits, you’ll be ready to fix it in five minutes flat.

Automating Market Research on Dermstore: Scrape Categories and Monitor Inventory Trends

Robert N. Gutierrez — Fri, 27 Feb 2026 01:30:00 +0000

The beauty and skincare market moves at a breakneck pace. For Revenue Operations (RevOps) teams and e-commerce managers, keeping tabs on competitor pricing, stock levels, and brand positioning on major retailers like Dermstore is a full-time job. Manual browsing is tedious and results in stale data. By the time you’ve mapped a category, prices have changed or a top-seller has gone out of stock.

Implementing a data-on-demand strategy solves this. Instead of manual checks, automated scrapers pull a complete snapshot of any Dermstore category in minutes. This allows you to spot inventory gaps, analyze "Share of Shelf," and adjust bidding strategies in real time.

This guide uses the open-source Dermstore.com-Scrapers repository to build an automated market research pipeline using Python and Playwright.

Why Scrape Product Categories?

Before looking at the code, it helps to understand why this data is a goldmine for competitive intelligence.

1. Inventory Monitoring and Conquesting

"Conquesting" is a strategy where you bid on a competitor's keywords the moment they run out of stock. By scraping the availability field across a category like "Sun Care," you can identify top-tier products that are currently unavailable. This is your signal to increase ad spend on your own equivalent product to capture frustrated customers.

2. Share of Shelf Analysis

In physical retail, brands fight for eye-level shelf space. In e-commerce, the "shelf" is the first page of a category. By scraping category listings, you can calculate how many SKUs your brand has on page one versus your competitors. If Brand X has 40 SKUs and you have five, you have a visibility problem.

3. Pricing Architecture

Automated scraping allows you to map the price points of an entire category. You can quickly calculate the average price for "Vitamin C Serums" and see if your product is positioned as a luxury, mid-tier, or budget option compared to the live market.

Prerequisites and Setup

To follow this tutorial, you need:

Python 3.8+ installed.
A ScrapeOps API Key (the free tier works for this).
Basic familiarity with the terminal.

First, clone the repository and install the dependencies for the Playwright implementation:

# Clone the repository
git clone https://github.com/scraper-bank/Dermstore.com-Scrapers.git
cd Dermstore.com-Scrapers

# Install Python dependencies
pip install playwright playwright-stealth
playwright install chromium

Step 1: Configuring the Product Category Scraper

We will focus on the Playwright implementation in python/playwright/product_category/. This scraper navigates category pages, handles pagination, and extracts structured data.

Open python/playwright/product_category/scraper/dermstore_scraper_product_category_v1.py. You need to add your API key and define your target category URL.

# python/playwright/product_category/scraper/dermstore_scraper_product_category_v1.py

API_KEY = "YOUR_SCRAPEOPS_API_KEY"

# Define the category you want to research
# Example: Sun Care category
TARGET_URL = "https://www.dermstore.com/skin-care/sun-care.list"

# The PROXY_CONFIG uses ScrapeOps to bypass anti-bot measures
PROXY_CONFIG = {
    "server": "http://residential-proxy.scrapeops.io:8181",
    "username": "scrapeops",
    "password": API_KEY
}

The scraper uses ScrapeOps Residential Proxies. This is necessary because Dermstore employs sophisticated anti-bot protections like Cloudflare. Without proxy rotation and optimized headers, the script will likely be blocked after a few requests.

Step 2: Running the Scraper

Run the script from your terminal. The scraper visits the category page, extracts all products across subsequent pages, and saves them to a JSONL file.

python python/playwright/product_category/scraper/dermstore_scraper_product_category_v1.py

We use JSONL (JSON Lines) because it is efficient for data pipelines. Each line is a standalone JSON object, so you can process files containing thousands of products without loading the entire dataset into memory at once.

Step 3: Interpreting the Data Signals

The output file contains objects representing each product. Here is how to map technical fields to business metrics:

JSON Field	Business Metric	Insight
`availability`	Inventory Status	Identifies "out_of_stock" items for ad conquesting.
`price`	Current Market Price	Monitors for sudden price drops or promotional shifts.
`rating`	Customer Sentiment	High-rated items with low stock are prime targets for replacement.
`isSponsored`	Competitor Spend	Detects which brands are paying for top-row placement.

A typical data object looks like this:

{
  "name": "Supergoop! Unseen Sunscreen SPF 40",
  "price": 38.00,
  "availability": "in_stock",
  "isSponsored": false,
  "rating": 4.8
}

Step 4: Visualizing the Data

Now for the RevOps analysis. We can use pandas to analyze the JSONL file and find which brands are potentially losing sales due to stockouts.

import pandas as pd
import json

# Load the scraped data
data = []
with open('dermstore_com_product_category_page_data_TIMESTAMP.jsonl', 'r') as f:
    for line in f:
        data.append(json.loads(line))

df = pd.DataFrame(data)

# 1. Calculate Share of Shelf (Top 10 Brands by SKU count)
share_of_shelf = df['brand'].value_counts().head(10)
print("Share of Shelf (Top 10 Brands):")
print(share_of_shelf)

# 2. Identify Out of Stock Competitors
oos_competitors = df[df['availability'] != 'in_stock']
print("\nOut of Stock Opportunities:")
print(oos_competitors[['brand', 'name', 'price']])

This script provides an immediate list of products missing from the shelf, allowing a marketing team to pivot their strategy quickly.

Reliable Scraping with ScrapeOps

Scraping major e-commerce sites isn't as simple as sending a GET request. These sites use advanced fingerprinting to detect automation. The Dermstore.com-Scrapers repository handles this complexity by integrating with ScrapeOps.

When you use the PROXY_CONFIG provided in the repo, the request follows this path:

Your Script: Sends a request to the ScrapeOps Proxy Gateway.
ScrapeOps: Rotates through residential IPs and selects browser headers that mimic a real user.
Dermstore: Receives a legitimate request from a residential user and returns the data.
Data Extraction: The Playwright logic parses the HTML and returns clean JSON.

Residential proxies are essential for e-commerce. Datacenter IPs are easily flagged, but residential IPs provide the reputation scores needed to navigate Cloudflare or DataDome challenges.

To Wrap Up

Automating market research on Dermstore transforms data from a static report into a competitive tool. By using the Dermstore.com-Scrapers repository, you can move away from manual browsing and toward a reactive, data-driven strategy.

Key Takeaways:

Automate Categories: Use category scrapers to monitor "Share of Shelf" and pricing trends across entire product lines.
Spot Gaps: Use the availability field to find "Out of Stock" signals for ad conquesting.
Scale Reliably: Use ScrapeOps proxy rotation to handle anti-bot measures and ensure your pipeline remains stable.

For your next steps, consider setting up a Cron job to run these scrapers daily or piping the JSONL output into a visualization tool like Tableau to create a live competitive dashboard.

Building a Costco Price Tracker with Python, Playwright, and Streamlit

Robert N. Gutierrez — Mon, 23 Feb 2026 05:27:16 +0000

In e-commerce, prices are rarely static. Retail giants like Costco adjust prices based on inventory levels, seasonal promotions, and regional demand. If you are a developer hunting for deals or an analyst monitoring competitive intelligence, a single snapshot of a price isn't enough. You need historical data to identify trends and predict future drops.

This guide walks through building a production-ready Costco price tracker. Using the open-source Costco.com-Scrapers repository, we will extract product data with Playwright, store it with timestamps, and visualize the history in a Streamlit dashboard.

Prerequisites

Before starting, ensure you have:

Python 3.8+
ScrapeOps API Key: Costco employs aggressive anti-bot measures. We’ll use ScrapeOps to handle proxy rotation and browser headers. You can get a free API key here.
Terminal access: Comfort working within a virtual environment.

Setup and Installation

Clone the repository and install the dependencies:

# Clone the repository
git clone https://github.com/scraper-bank/Costco.com-Scrapers.git
cd Costco.com-Scrapers/python/playwright

# Create and activate a virtual environment
python -m venv venv
source venv/bin/activate  # On Windows use `venv\Scripts\activate`

# Install requirements
pip install playwright playwright-stealth streamlit pandas
playwright install chromium

Part 1: Data Extraction

The foundation of the tracker is the Playwright scraper located at python/playwright/product_data/scraper/costco_scraper_product_data_v1.py. This script uses playwright-stealth and structured data extraction to avoid detection.

A standard scraper only captures the current price. To track trends, we need to modify the script to include a timestamp. Update the ScrapedData dataclass and the extraction logic to ensure every record is ready for time-series analysis.

Modifying for Time-Series Data

Import the datetime module and add a scraped_at field to the data object:

from datetime import datetime

@dataclass
class ScrapedData:
    # ... existing fields ...
    scraped_at: str = ""  # Add this field

async def extract_data(page: Page) -> Optional[ScrapedData]:
    # ... existing extraction logic ...
    res = ScrapedData()

    # Inject the current timestamp
    res.scraped_at = datetime.now().isoformat()

    # Extract name, price, etc.
    # Playwright targets the JSON-LD schema for accuracy
    scripts = await page.locator("script[type='application/ld+json']").all_text_contents()
    # ...
    return res

Adding datetime.now().isoformat() ensures that every time the scraper runs, the price is anchored to a specific moment. This allows the dashboard to plot price changes over time.

Part 2: Handling Anti-Bot Measures

Scraping Costco repeatedly from a local IP usually results in a 403 Forbidden error or a CAPTCHA. The repository addresses this by integrating with the ScrapeOps Residential Proxy Aggregator.

Residential proxies route requests through real home IP addresses, making the scraper look like a regular shopper. In your costco_scraper_product_data_v1.py file, configure your ScrapeOps credentials:

# ScrapeOps Residential Proxy Configuration
PROXY_CONFIG = {
    "server": "http://residential-proxy.scrapeops.io:8181",
    "username": "scrapeops",
    "password": "YOUR_SCRAPEOPS_API_KEY"
}

The script uses these credentials to authenticate every request. ScrapeOps handles the rotation automatically, so you don't need to manage proxy lists manually.

Part 3: Automating the Tracker

To build a useful history, the scraper needs to run periodically, such as once every 24 hours. Instead of overwriting files, we use the JSONL (JSON Lines) format.

The DataPipeline class in the repository appends a new line to the file for every successful scrape:

class DataPipeline:
    def __init__(self, jsonl_filename="price_history.jsonl"):
        self.jsonl_filename = jsonl_filename

    def add_data(self, scraped_data: ScrapedData):
        with open(self.jsonl_filename, mode="a", encoding="UTF-8") as output_file:
            json_line = json.dumps(asdict(scraped_data), ensure_ascii=False)
            output_file.write(json_line + "\n")

You can automate this with a Cron job on Linux/Mac or a Scheduled Task on Windows. Here is a basic shell script (run_tracker.sh) to trigger the scrape:

#!/bin/bash
# run_tracker.sh
python costco_scraper_product_data_v1.py --urls "https://www.costco.com/product-1.html" "https://www.costco.com/product-2.html"

Part 4: Building the Dashboard

With price_history.jsonl collecting data, we can visualize it using Streamlit. Create a file named app.py:

import streamlit as st
import pandas as pd
import json

st.set_page_config(page_title="Costco Price Tracker", layout="wide")
st.title("🛒 Costco Price Intelligence Dashboard")

# Load data from JSONL
@st.cache_data(ttl=3600)
def load_data():
    data = []
    with open('price_history.jsonl', 'r') as f:
        for line in f:
            data.append(json.loads(line))
    df = pd.DataFrame(data)

    # Data Cleaning
    df['date'] = pd.to_datetime(df['scraped_at'])
    df['price'] = pd.to_numeric(df['price'], errors='coerce')
    return df

try:
    df = load_data()

    # Sidebar for filtering
    product_list = df['name'].unique()
    selected_product = st.sidebar.selectbox("Select a Product", product_list)

    # Filtered Data
    filtered_df = df[df['name'] == selected_product].sort_values('date')

    # Metrics
    current_price = filtered_df['price'].iloc[-1]
    min_price = filtered_df['price'].min()

    col1, col2 = st.columns(2)
    col1.metric("Current Price", f"${current_price:,.2f}")
    col2.metric("All-Time Low", f"${min_price:,.2f}", delta=float(current_price - min_price), delta_color="inverse")

    # Price History Chart
    st.subheader(f"Price Trend: {selected_product}")
    st.line_chart(filtered_df.set_index('date')['price'])

except FileNotFoundError:
    st.error("No data found. Run the scraper first to generate 'price_history.jsonl'.")

Why this approach works:

Efficiency: JSONL allows the app to read data line-by-line, which is more memory-efficient than loading a single massive JSON array.
Context: Streamlit metrics show immediately if the current price is a deal compared to the historical low.
Speed: The @st.cache_data decorator ensures the UI stays responsive even as the dataset grows.

Next Steps

Once the pipeline is stable, you can extend it with several features:

Price Drop Alerts: Integrate SendGrid or Slack notifications to alert you when a price hits a specific threshold.
Availability Tracking: The scraper already extracts stock status. You can plot this to see how often high-demand items sell out.
Database Migration: If the JSONL file grows to thousands of entries, consider switching the DataPipeline to SQLite or PostgreSQL for faster querying.

To wrap up, combining Playwright with Streamlit creates a powerful tool for monitoring market trends. You can find more Costco scraping implementations in the Costco.com-Scrapers repository.

Dynamic Selector Fallbacks: How to Scrape E-commerce Sites That Change Frequently

Robert N. Gutierrez — Fri, 20 Feb 2026 07:49:12 +0000

The most frustrating moment for any web scraping developer is waking up to a broken scraper. You might spend hours mapping out the perfect selectors for a site like Ulta.com, only for their front-end team to push a minor UI update that renames .Price-v1 to .Price-v2. Suddenly, your database fills with null values and your price alerts go silent.

In e-commerce, layouts change constantly. Relying on a single, rigid CSS selector leads to high maintenance costs and data gaps. To build professional data pipelines, we need to move away from fragile scraping and toward a Dynamic Selector Fallback strategy.

This guide explores how to implement robust fallbacks and proactive monitoring, using the Ulta.com-Scrapers repository as a real-world case study.

The Problem: Fragility in Generated Code

Most scrapers fail silently. They don't crash with a loud error; they simply return empty strings or zeros because a specific class name no longer exists in the DOM.

Consider this common pattern found in the initial version of node/cheerio-axios/product_data/scraper/ulta_scraper_product_data_v1.js:

// Rigid selector example
const valText = $(".ReviewStars__Content").first().text().trim();
if (valText) aggRating.ratingValue = parseFloat(valText) || 0.0;

This code is vulnerable. If Ulta’s developers rename .ReviewStars__Content to .ReviewStars__Value or move the rating into a data attribute, this line returns 0.0. You won't know it's broken until you realize your average rating report has plummeted. This cycle of fixing and breaking is exhausting and prevents you from scaling.

Strategy 1: The Priority List

The first step toward reliability is acknowledging that there is often more than one way to find a piece of data. We can define a Priority List, which is an array of potential selectors for each field.

We start by creating a helper function, extractWithFallback, which iterates through candidates until it finds a valid result.

/**
 * Attempts to extract text using a list of selectors.
 * Returns the text and the index of the selector that worked.
 */
function extractWithFallback($, selectors) {
    for (let i = 0; i < selectors.length; i++) {
        const value = $(selectors[i]).first().text().trim();
        if (value && value.length > 0) {
            return { value, index: i };
        }
    }
    return { value: null, index: -1 };
}

Why this works

Using this approach for critical fields like price or product name creates a safety net:

Primary Selector: The most specific, current class name (e.g., .ProductPricing__price).
Secondary Selector: Older known classes or broader containers (e.g., .price, .pal-c-Price).
Metadata: E-commerce sites often include data in <meta> tags or JSON-LD scripts that change less frequently than the visual UI.

Strategy 2: The "Hail Mary" (Text-Based Matching)

When specific class names fail entirely, you need a backup plan. This involves looking for data based on its structure or content rather than its CSS class.

For example, if you can't find the price via a class, search for elements containing a currency symbol or use regular expressions on the entire page body.

// A "Hail Mary" approach using text-based matching in Cheerio
const hailMaryPrice = $("span:contains('$')").first().text();

// Or using Regex on a broader text block
const bodyText = $("body").text();
const priceMatch = bodyText.match(/\$\s?(\d+[.,]\d{2})/);
const price = priceMatch ? priceMatch[1] : null;

Note: These methods can produce false positives, such as accidentally grabbing the price of a related product. They should always be the last item in your fallback array.

Strategy 3: The Deprecation Warning System

The secret to low-maintenance scraping is knowing when a scraper is starting to fail. You can turn your fallback system into a proactive monitoring tool.

If the primary selector fails but a secondary selector succeeds, the scraper is technically still working, but it’s on life support. Log this as a warning to stay ahead of the curve.

const priceResult = extractWithFallback($, [
    ".pal-c-Price--salePrice", // Primary
    ".Price",                  // Secondary
    ".product-price"           // Legacy
]);

if (priceResult.index > 0) {
    console.warn(`DEPRECATION WARNING: Primary selector failed for Price. Fallback index ${priceResult.index} used at ${url}`);
    // This could also be sent to a monitoring service like Sentry or Slack
}

This transforms maintenance from an emergency into a routine task. You can update the selectors during normal business hours rather than scrambling when the data stops flowing.

Tutorial: Refactoring the Ulta Scraper

Let's apply these concepts to the extractData function in the Ulta repository. We will refactor the rating and price logic to be significantly more robust.

Step 1: Define the Helper

Add a safe extraction helper that handles the logic of checking multiple selectors and logging warnings.

function safeExtract($, selectors, fieldName) {
    const result = extractWithFallback($, selectors);
    if (result.index > 0) {
        console.warn(`[Monitoring] ${fieldName} used fallback selector: ${selectors[result.index]}`);
    }
    return result.value;
}

Step 2: Refactor the Rating Logic

The original code relies heavily on ReviewStars__Content. We will expand this to check JSON-LD first, as it is the most stable source, then multiple CSS classes.

function extractRating($, jsonData) {
    const ratingObj = { ratingValue: 0.0, reviewCount: 0 };

    // 1. Try JSON-LD (Most stable)
    if (jsonData?.aggregateRating) {
        ratingObj.ratingValue = parseFloat(jsonData.aggregateRating.ratingValue) || 0;
        ratingObj.reviewCount = parseInt(jsonData.aggregateRating.reviewCount) || 0;
        if (ratingObj.ratingValue > 0) return ratingObj;
    }

    // 2. Fallback to CSS Selectors
    const valText = safeExtract($, [
        ".ReviewStars__Content", 
        ".pal-c-Ratings__numericalRatingAfter",
        ".Text-ds--body-3"
    ], "RatingValue");

    ratingObj.ratingValue = parseFloat(valText) || 0.0;
    return ratingObj;
}

Step 3: Refactor the Price Logic

Price is the most critical field. We’ll combine JSON-LD, specific classes, and a regex match as the final fallback.

function extractPrice($, jsonData) {
    // Priority 1: Schema.org JSON-LD
    if (jsonData?.offers?.price) {
        return parseFloat(jsonData.offers.price);
    }

    // Priority 2: Known CSS Classes
    const priceText = safeExtract($, [
        ".pal-c-Price--PDP span",
        ".Price",
        ".ProductPricing"
    ], "Price");

    if (priceText) {
        const match = priceText.match(/[\d,.]+/);
        if (match) return parseFloat(match[0].replace(/,/g, ''));
    }

    // Priority 3: Hail Mary Regex
    const bodyMatch = $.html().match(/\"price\"\s*:\s*\"?(\d+\.\d{2})\"?/);
    return bodyMatch ? parseFloat(bodyMatch[1]) : 0.0;
}

To Wrap Up

By implementing dynamic selector fallbacks, you trade a few milliseconds of execution time for a massive increase in scraper longevity. Instead of your pipeline breaking the moment a developer changes a class name, your system adapts and alerts you to the change.

Key Takeaways:

Use Priority Lists: Always have a Plan B and Plan C for critical data points.
Use JSON-LD: Metadata scripts are often more stable than the visual DOM.
Log Fallbacks: Treat a successful fallback as a warning to schedule maintenance before a total failure occurs.
Stay Defensive: Use helper functions like safeExtract to keep your main scraping logic clean.

For the full suite of Ulta scraping tools, visit the Ulta.com-Scrapers GitHub repository. To scale your operations, use a ScrapeOps API key to manage proxy rotation and avoid blocks.

Building a 'Data-on-Demand' Microservice: Wrapping Alibaba Scrapers for Internal Tools

Robert N. Gutierrez — Tue, 17 Feb 2026 06:17:56 +0000

You’ve built a powerful Puppeteer script that scrapes product data from Alibaba. It works perfectly in your terminal, outputting clean JSON every time you run it. But then your RevOps manager asks for that same data to populate a live dashboard in Retool or a Google Sheet.

Business users don’t live in the terminal, and they certainly don't want to manage Node.js environments or npm dependencies just to check competitor pricing. To bridge this gap, you need to move the logic out of a standalone script and into a Node.js scraping API.

This guide explores how to transform a local Alibaba scraper into a production-ready microservice using Express.js. We’ll cover modular refactoring, concurrency management, and how to secure your endpoint so internal teams can access live data on demand.

High-Level Architecture

Before writing code, it helps to understand the request lifecycle. Traditional scraping is often scheduled, running once a day or week. "Data-on-Demand" is synchronous. A user clicks a button and expects a response within seconds.

The flow works like this:

The Client: An internal tool or a simple curl command sends a GET request to your server with a search query.
The Express API: Validates the request and checks for an API key.
The Scraper Module: Spawns a Puppeteer instance, navigates Alibaba, and extracts the data.
The Response: The server sends structured JSON back to the client and closes the browser instance.

Because browsers are resource-heavy, stability is the priority. Spawning 50 Chrome instances simultaneously will crash a standard VPS, so we need to implement safeguards.

Phase 1: Refactoring the Scraper for Modularity

Most CLI scrapers use a "flat" structure where logic runs immediately. To use this in an API, wrap it in an asynchronous function that returns data instead of just logging it to the console.

Create a file named scraper.js. This module will export a single function, scrapeAlibaba, to handle the browser lifecycle.

const puppeteer = require('puppeteer');

/**
 * Scrapes Alibaba search results for a given keyword.
 * @param {string} keyword - The product to search for.
 * @param {number} limit - Maximum number of products to return.
 */
async function scrapeAlibaba(keyword, limit = 10) {
  // Use --no-sandbox for compatibility with many Linux environments
  const browser = await puppeteer.launch({
    headless: "new",
    args: ['--no-sandbox', '--disable-setuid-sandbox']
  });

  try {
    const page = await browser.newPage();

    // Set a realistic user agent to avoid immediate detection
    await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36');

    const searchUrl = `https://www.alibaba.com/trade/search?SearchText=${encodeURIComponent(keyword)}`;
    await page.goto(searchUrl, { waitUntil: 'networkidle2', timeout: 60000 });

    // Wait for the product cards to load
    await page.waitForSelector('.list-no-v2-main-list-element', { timeout: 10000 });

    const products = await page.evaluate((maxItems) => {
      const items = Array.from(document.querySelectorAll('.list-no-v2-main-list-element'));
      return items.slice(0, maxItems).map(item => ({
        title: item.querySelector('.search-card-e-title span')?.innerText || 'N/A',
        price: item.querySelector('.search-card-e-price-main')?.innerText || 'N/A',
        minOrder: item.querySelector('.search-card-e-v2-min-order')?.innerText || 'N/A',
        link: item.querySelector('a.search-card-e-slider__link')?.href || ''
      }));
    }, limit);

    return products;
  } catch (error) {
    console.error(`Scraping failed for ${keyword}:`, error.message);
    throw error; 
  } finally {
    await browser.close();
  }
}

module.exports = { scrapeAlibaba };

By returning the products array and using a finally block to ensure browser.close() always runs, the scraper becomes a predictable, reusable unit.

Phase 2: Building the Express Server

To expose this function via an HTTP endpoint, use Express.js for its simplicity and middleware ecosystem.

First, install the packages:
npm install express puppeteer

Then, create server.js:

const express = require('express');
const { scrapeAlibaba } = require('./scraper');

const app = express();
const PORT = process.env.PORT || 3000;

app.get('/api/search', async (req, res) => {
  const { q, limit } = req.query;

  if (!q) {
    return res.status(400).json({ 
      error: "Missing search query. Use ?q=keyword" 
    });
  }

  try {
    const results = await scrapeAlibaba(q, parseInt(limit) || 10);

    res.json({
      success: true,
      query: q,
      timestamp: new Date().toISOString(),
      count: results.length,
      data: results
    });

  } catch (error) {
    res.status(500).json({ 
      success: false, 
      error: "Failed to fetch data from Alibaba",
      details: error.message 
    });
  }
});

app.listen(PORT, () => {
  console.log(`Alibaba Microservice running on http://localhost:${PORT}`);
});

Navigating to http://localhost:3000/api/search?q=mechanical+keyboard will now trigger a real-time scrape and return a JSON payload.

Phase 3: Handling Concurrency and Stability

The code above works for one user, but Puppeteer is resource-heavy. Each browser instance can consume 100MB to 500MB of RAM. If ten people hit the API at once, the server might run out of memory.

Limiting Concurrency

Use a simple counter to ensure you only run a set number of browsers at once. This prevents the server from being overwhelmed.

let activeRequests = 0;
const MAX_CONCURRENT_SCRAPES = 3;

app.get('/api/search', async (req, res) => {
  if (activeRequests >= MAX_CONCURRENT_SCRAPES) {
    return res.status(503).json({ error: "Server busy. Try again later." });
  }

  activeRequests++;
  try {
    const results = await scrapeAlibaba(req.query.q);
    res.json({ data: results });
  } finally {
    activeRequests--; 
  }
});

Proxy Integration

Alibaba protects its data aggressively. Sending too many requests from a single IP will lead to CAPTCHAs. For a reliable service, use rotating residential proxies.

Update the puppeteer.launch configuration in scraper.js:

const browser = await puppeteer.launch({
  args: [
    `--proxy-server=http://YOUR_PROXY_ADDRESS:PORT`,
    '--no-sandbox'
  ]
});

const page = await browser.newPage();
await page.authenticate({
  username: 'YOUR_USERNAME',
  password: 'YOUR_PASSWORD'
});

Phase 4: Error Handling and Response Formatting

A good API returns context. When building for internal tools, map scraping failures to appropriate HTTP status codes:

400 (Bad Request): No search term provided.
404 (Not Found): The scrape finished, but Alibaba returned zero results.
503 (Service Unavailable): The server is at capacity or the IP is blocked.

Standardizing the response format makes it easier for other developers to use the API. A consistent structure like { success: boolean, data: [], error: string } works best.

Phase 5: Security and Integration

Never leave a scraping API open to the public. Since Puppeteer consumes significant resources, an unprotected endpoint can lead to high infrastructure costs or a crashed server.

Adding API Key Middleware

A simple middleware function can protect the route:

const API_KEY = "your-super-secret-key";

const authenticate = (req, res, next) => {
  const userKey = req.headers['x-api-key'];
  if (userKey && userKey === API_KEY) {
    next();
  } else {
    res.status(401).json({ error: "Unauthorized. Valid API key required." });
  }
};

app.get('/api/search', authenticate, async (req, res) => { ... });

Real-World Usage

With this API in place, the RevOps team can use a tool like Retool. They can drag a "Button" component onto a canvas, set it to trigger a REST API request to your microservice, and display the results in a table. They get live Alibaba data without writing any code.

To Wrap Up

Building a "Data-on-Demand" microservice transforms web scraping from a developer task into a company-wide asset. By wrapping Puppeteer in an Express API, you provide a clean, structured interface for non-technical stakeholders.

As you scale, consider these improvements:

Caching: Use Redis to store search results for 24 hours to reduce proxy costs.
Task Queues: For longer scrapes, move from a synchronous API to a background job system like BullMQ.
Headless Management: Use services like Browserless.io to run Chrome instances on external infrastructure, keeping your API server lightweight.

These patterns ensure your data extraction pipelines are accessible, resilient, and professional.

Forem: Robert N. Gutierrez

Building a Crate & Barrel Price Monitor Dashboard with Python & Playwright

Prerequisites & Setup

1. Clone the Repository

2. Install Dependencies

3. Get a ScrapeOps API Key

Step 1: Understanding the Base Scraper

Step 2: Configuring the Target List

Step 3: Building the Monitor Script

Step 4: Automating the Run

Option A: Simple Loop

Option B: Cron Job

Step 5: Visualizing the Data

Practical Tips

To Wrap Up

How to Repair Broken Wayfair Scrapers Instantly Using AI-Generated Selectors

The Anatomy of a Breakage

Step 1: Decoupling Extraction from Crawling

Step 2: Implementing Automated Validation

Step 3: The Hot-Swap Workflow

Example: Before and After

Step 4: Applying the Fix

Recommended Approaches for Wayfair

To Wrap Up

Stop Silent Scraper Failures: Using Pydantic for Instant Layout Change Detection

Why try/except Isn't Enough

Step 1: Defining the Data Contract with Pydantic

Step 2: Integrating Validation with Playwright

Step 3: Catching the Drift

Step 4: Advanced Cleaning with Validators

Step 5: The Error Budget

Summary

How to Build an AliExpress Price Tracker with Python and Playwright

Prerequisites

Setup: Cloning the Scraper Engine

Step 1: Understanding the Scraper Logic

Step 2: Customizing for Price Monitoring

Step 3: Automating the Scan

Step 4: Visualizing the Data

Common Obstacles

To Wrap Up

How to Export Depop Listings to a Shopify CSV Using Python

Prerequisites

Setup Instructions

Step 1: Extracting Product Data from Depop

Step 2: Mapping the Data

Step 3: The Transformation Script

Why Pandas?

Step 4: Handling Images and Variations

Step 5: Importing into Shopify

Troubleshooting

To Wrap Up

From Scrape to Feed: Building a Google Merchant Center CSV from Zappos Data

Prerequisites & Setup

Step 1: Extracting Raw Data from Zappos

The Raw Output

Step 2: Gap Analysis (JSONL vs. GMC Requirements)

Step 3: Building the Transformation Script

Loading and Mapping

Formatting Challenges

Step 4: Exporting the Final Feed

Practical Tips and Gotchas

To Wrap Up

Bulletproof Data Pipelines: Adding Schema Validation to Nike Scrapers

Prerequisites

The Anatomy of a Soft Break

Strategy: Validation at the Pipeline Level

Implementation 1: Node.js with Zod

1. Install Zod

2. Define the Schema

3. Update the DataPipeline

Implementation 2: Python with Pydantic

1. Define the Pydantic Model

2. Integrate into the Pipeline

Error Budgets: The Circuit Breaker Pattern

Closing the Loop: Regenerating Scrapers

Summary

The 5-Minute Repair: A Workflow for Surviving Site Redesigns

Prerequisites

Phase 1: Diagnosing the Break

Why `try/except` Isn't Enough