Forem: NexGenData

Academic Research Made Easy: How to Bulk Search Google Scholar Papers

NexGenData — Fri, 15 May 2026 04:28:05 +0000

Academic Research Made Easy: How to Bulk Search Google Scholar Papers

The literature review process hasn't changed much in 20 years. You identify a research topic. You go to Google Scholar. You search for related papers. You find 20 relevant results. You click through each one, read abstracts, record citations, check how many times it's been cited. You repeat this for 15-20 different search queries. After a week of work, you have a spreadsheet with 200 papers and you're still not confident you've found the landmark papers in your field.

For PhD students, this is months of work. For researchers working on grants, it's time that doesn't advance the research itself. For data scientists building new models, it's weeks spent on context that could be done in hours if you had the right tools.

Modern research shouldn't involve manual clicking and copying spreadsheets. It should involve structured data extraction, intelligent analysis, and systematic discovery. You should be able to search 50 queries, extract 500 papers, analyze citation patterns, and identify the most influential work—all in a single automated process.

That's where intelligent Google Scholar scraping comes in. Instead of spending a week on literature review, you spend a day analyzing results to understand the research landscape.

The Literature Review Time Problem

Academic research depends on understanding the current state of knowledge. What's already been done? What gaps exist? Who are the key researchers? What methodologies are standard? These are foundational questions that require thorough literature review.

But the process is brutally inefficient:

Manual searching: You're typing queries into a web interface one at a time, clicking through results slowly
No bulk processing: You can't easily search 50 related queries at once; you do them serially
Limited context: You see papers one at a time without comparing them systematically
Citation analysis: Understanding which papers are landmarks requires manually checking citation counts across dozens of results
No systematic discovery: You might miss important papers because they use slightly different terminology than your search query

The researcher with better literature review data makes better research decisions. The researcher who wastes weeks on inefficient searches loses that advantage.

For grant writing, this is especially critical. Reviewers expect you to demonstrate deep knowledge of the field. A researcher who can show they've reviewed 300 papers and identified the key gaps in knowledge has more credibility than one who references 30 papers.

For PhD students, literature review can consume 2-3 months. That time should be spent on novel research, not on clicking through search results.

The Solution: Bulk Google Scholar Research

Instead of one search at a time, you can use an intelligent scraper to execute dozens of searches at once, extract comprehensive metadata for each result, and synthesize everything into a research analysis.

The Apify Google Scholar Scraper is designed specifically for this: bulk searches across multiple queries, extraction of all relevant metadata (title, authors, citation count, publication year, abstract), and when tracker mode is enabled, synthesis of that data into a research landscape analysis.

Think of it as having a research assistant who spends a week clicking through Google Scholar while you focus on understanding the landscape, and then presents you with a comprehensive map of the field.

Instead of "I found papers about X," you get "Here are 500 papers about X, organized by citation influence, publication year, and research institution, with the top 20 landmark papers highlighted."

How It Works: Bulk Query Configuration

Here's what a realistic research configuration looks like:

{
  "queries": [
    "machine learning interpretability",
    "explainable AI XAI",
    "LIME SHAP interpretability",
    "neural network feature importance",
    "saliency maps deep learning",
    "attention mechanisms interpretability",
    "model transparency black box",
    "feature attribution methods",
    "causal inference machine learning",
    "adversarial robustness interpretability"
  ],
  "filters": {
    "yearFrom": 2018,
    "yearTo": 2026,
    "citedBy": "all"
  },
  "dataPoints": [
    "title",
    "authors",
    "publicationYear",
    "citationCount",
    "abstract",
    "journalName",
    "publicationVenue",
    "keywords",
    "doi"
  ],
  "trackerMode": true,
  "outputFormat": "json"
}

The scraper then:

Executes all 10 searches in sequence
Extracts complete metadata for each result
Organizes papers by relevance, citation count, and publication date
When tracker mode is enabled, synthesizes the data into a research analysis showing:
- Citation tiers (landmark papers, frequently cited, emerging)
- Venue breakdown (which conferences/journals dominate the field)
- Author networks (who are the key researchers)
- Topic evolution (how the field is evolving over time)
Flags the most important papers based on citation patterns and recency

The result is a comprehensive research landscape in minutes instead of weeks.

Sample Output: Research Landscape Analysis

Here's what the tracker summary produces:

{
  "research_landscape": {
    "analysis_date": "2026-04-05",
    "queries_executed": 10,
    "total_papers_found": 487,
    "date_range": "2018-2026",
    "citation_tiers": {
      "landmark_papers": [
        {
          "rank": 1,
          "title": "Why Should I Trust You?: Explaining the Predictions of Any Classifier",
          "authors": ["Ribeiro, M. T.", "Singh, S.", "Guestrin, C."],
          "year": 2016,
          "citation_count": 8743,
          "venues": ["KDD 2016"],
          "impact": "foundational_framework",
          "relevance": "LIME method definition"
        },
        {
          "rank": 2,
          "title": "A Unified Approach to Interpreting Model Predictions",
          "authors": ["Lundberg, S. M.", "Lee, S. I."],
          "year": 2017,
          "citation_count": 7456,
          "venues": ["NeurIPS 2017"],
          "impact": "foundational_framework",
          "relevance": "SHAP method definition"
        }
      ],
      "frequently_cited_papers": [
        {
          "title": "Interpretable Explanations of Black Boxes by Meaningful Perturbation",
          "authors": ["Fong, R. C.", "Vedaldi, A."],
          "year": 2017,
          "citation_count": 2145,
          "impact": "influential_methodology"
        }
      ],
      "emerging_papers": [
        {
          "title": "Mechanistic Interpretability via Causal Graphs",
          "authors": ["Smith, J.", "Johnson, K."],
          "year": 2025,
          "citation_count": 23,
          "impact": "novel_approach",
          "momentum": "rapidly_cited"
        }
      ]
    },
    "venue_analysis": {
      "top_venues": [
        {
          "venue": "NeurIPS",
          "papers_count": 87,
          "total_citations": 125000,
          "avg_citations_per_paper": 1437
        },
        {
          "venue": "ICML",
          "papers_count": 61,
          "total_citations": 94000,
          "avg_citations_per_paper": 1541
        },
        {
          "venue": "ICLR",
          "papers_count": 53,
          "total_citations": 78000,
          "avg_citations_per_paper": 1472
        }
      ],
      "journal_venues": [
        {
          "venue": "Journal of Machine Learning Research",
          "papers_count": 24,
          "avg_citations_per_paper": 1200
        }
      ]
    },
    "author_network": {
      "most_prolific_authors": [
        {
          "author": "Ribeiro, M. T.",
          "papers_count": 12,
          "avg_citations": 2500,
          "research_focus": "Local interpretability methods"
        }
      ],
      "key_institutions": [
        {
          "institution": "University of Washington",
          "papers_count": 34,
          "citation_impact": "very_high"
        }
      ]
    },
    "temporal_trends": {
      "publication_by_year": [
        {"year": 2018, "papers": 28, "avg_citations_at_age": 650},
        {"year": 2019, "papers": 45, "avg_citations_at_age": 480},
        {"year": 2025, "papers": 67, "avg_citations_at_age": 45}
      ],
      "research_evolution": "Field rapidly accelerating; shift from post-hoc interpretability toward mechanistic understanding"
    },
    "research_gaps": [
      {
        "gap": "Limited work on GPU-efficient interpretation for edge devices",
        "evidence": "Only 3 papers found addressing this across all queries",
        "opportunity": "Novel research direction"
      }
    ]
  }
}

This is immediately useful. You can see that LIME and SHAP are the foundational frameworks. You can see that NeurIPS and ICML are the key venues. You can see the key authors and institutions. You can identify emerging directions and research gaps.

Practical Use Cases: Who Needs This

PhD Students: You need to write a 30-page literature review chapter. Instead of spending 12 weeks reading papers manually, you run the scraper, get 400+ papers organized by importance, read the top 30 landmark papers, and write your chapter in 4 weeks. You've saved 2 months and your chapter is more comprehensive.

Researchers Writing Grants: Grant reviewers expect you to demonstrate deep knowledge of the field. Bulk Google Scholar research lets you cite 100+ papers and show you've comprehensively reviewed the landscape. This strengthens your application.

Data Scientists Building Novel Models: You're about to start a new project. Before you code, you need to understand what methods already exist, what's the current best practice, and where gaps might be. Bulk search gives you this context in hours instead of weeks.

Literature Review Services: You're providing systematic reviews for organizations. Automated bulk search accelerates your process, reduces costs, and improves comprehensiveness. You can review more papers, faster.

Product Managers in AI/ML: You're evaluating emerging research areas to inform product strategy. Bulk search and analysis tells you where academia is moving, which techniques are gaining traction, and where innovation is happening.

Workflow: From Search to Insight

Here's how this works in practice:

Define Your Query Set: Identify 15-20 search queries covering your research area from multiple angles (terminology variations, specific methods, application areas, etc.)
Execute Bulk Search: Run the scraper with all queries at once. This takes 10-15 minutes for 400+ papers.
Review Tracker Output: Spend 30 minutes reviewing the analysis. Identify landmark papers, key venues, top authors, and research gaps.
Read Strategically: Start with the top 10-20 landmark papers. These form your foundation. Then read a sample from each citation tier to understand the breadth.
Iterate: Identify missing areas from the initial search. Run a second search targeting those gaps. Merge all results.
Synthesize: Write your literature review, grant proposal, or research context based on comprehensive data rather than intuition.

What would take 4-6 weeks manually takes 2-3 weeks with intelligent automation. More importantly, your results are more comprehensive and more rigorous.

Getting Started: Access the Actor

The Apify Google Scholar Scraper is available at: https://apify.com/nexgendata/google-scholar-scraper?fpr=2ayu9b

Start with a research area you know well. Define 10-15 search queries. Run the bulk search once and review the results. You'll immediately see patterns and landmark papers you might have missed through manual searching. Once you see the value, apply this to your actual research questions.

The Research Advantage

Manual literature review is a bottleneck in academic work. Researchers spend weeks on administrative work that a computer could do in an afternoon. The researcher who can reduce literature review time from 8 weeks to 2 weeks gains a 6-week advantage on actual research.

Bulk Google Scholar research isn't about cutting corners on rigor—it's about removing administrative burden so you can focus on understanding and creating knowledge rather than on clicking through search results.

The papers you need to read are on Google Scholar right now. The patterns in research direction exist in the data. The gaps in the field are visible if you analyze the right dataset. Bulk search and systematic analysis give you access to all of that.

Your literature review doesn't have to take months. It shouldn't. You have better things to do than manually search Google Scholar. Automate the search and invest your time in understanding and advancing the field.

Scraping Redfin for Real Estate Investment Analysis: A Complete Guide

NexGenData — Thu, 14 May 2026 15:10:14 +0000

Scraping Redfin for Real Estate Investment Analysis: A Complete Guide

Real estate investing isn't about intuition or luck. It's about data. Which neighborhoods are appreciating fastest? Where are days-on-market trending shorter (indicating strong demand)? Which properties are overpriced relative to comparable sales? Where are investors already moving?

Redfin has all this data publicly available. Their site shows prices, sold prices, market trends, property characteristics, and neighborhood statistics. But manually reviewing 50-100 properties to analyze a neighborhood takes hours. And by the time you finish your spreadsheet, the market has moved and properties that looked good yesterday are no longer available.

Professional real estate investors, property flippers, and commercial real estate agents don't do this manually. They have systems that extract hundreds of properties, analyze them against investment criteria, and flag opportunities automatically. You can build the same system in a weekend using intelligent scraping.

The Real Estate Investor's Data Problem

You're looking for your next investment. Maybe you're a buy-and-hold investor looking for property with strong appreciation potential. Maybe you're a flipper looking for undervalued properties with renovation upside. Maybe you're a commercial agent evaluating neighborhood trends for clients.

The traditional approach: open Redfin, search a neighborhood or zip code, and manually review listings. You jot down prices, look at sold comps, maybe calculate price per square foot. This works for 10-20 properties. For systematic analysis of 100+ properties across multiple neighborhoods? It doesn't scale.

The data problems compound:

Timing: By the time you finish analyzing, good properties are already sold
Consistency: You might analyze one neighborhood thoroughly but skip another because it's tedious
Scope: You can only analyze properties you manually look at
Signals: You miss patterns that only become obvious with hundreds of data points

The investment opportunity you miss because you were too slow to analyze the data costs you far more than automation would ever cost.

The Solution: Automated Redfin Property Analysis

Instead of manually reviewing properties, you can use an intelligent scraper to extract comprehensive property and market data from Redfin. The Apify Redfin Scraper is designed to pull property listings, sold comparables, price history, and market statistics—everything you need for investment analysis.

When you enable tracker mode, the scraper doesn't just dump property listings. It synthesizes data to create a neighborhood-level investment analysis. You get insights like average price per square foot trends, days-on-market patterns, price appreciation rates, and supply/demand indicators. You get flagged opportunities—properties that appear underpriced relative to comparables or neighborhoods showing strong momentum.

It's like having a research team that analyzes neighborhoods and properties while you sleep, feeding you only the opportunities that match your investment criteria.

How It Works: Configuration and Output

Here's what a typical configuration looks like for analyzing an investment area:

{
  "searchLocation": "Austin, TX",
  "filters": {
    "minPrice": 250000,
    "maxPrice": 650000,
    "minBeds": 2,
    "minBaths": 1.5,
    "minSqft": 1200,
    "propertyTypes": ["Single Family", "Townhouse"]
  },
  "dataPoints": [
    "price",
    "beds",
    "baths",
    "sqft",
    "daysOnMarket",
    "pricePerSqft",
    "yearBuilt",
    "saleHistory",
    "taxHistory",
    "priceHistory"
  ],
  "trackerMode": true,
  "outputFormat": "json"
}

The scraper then:

Searches for properties matching your criteria
Extracts detailed information for each property
Fetches comparable sales in the area
Analyzes neighborhood trends (appreciation, days-on-market trends, supply levels)
Flags opportunities based on investment signals
Returns everything as structured JSON

The entire process is hands-off. You set it and let it run.

Sample Output: Investment-Grade Analysis

Here's what the tracker summary looks like:

{
  "investment_analysis": {
    "market_summary": {
      "location": "Austin, TX (78701-78704 zip codes)",
      "analysis_date": "2026-04-05",
      "properties_analyzed": 287,
      "market_health": {
        "avg_price": 425000,
        "median_price": 410000,
        "avg_price_per_sqft": 185,
        "price_per_sqft_trend": "up_3.2_percent_3mo",
        "days_on_market_avg": 18,
        "days_on_market_trend": "declining",
        "inventory_months": 1.8,
        "market_velocity": "strong_buyers_market"
      }
    },
    "neighborhood_tiers": {
      "hot_markets": [
        {
          "neighborhood": "East Austin (East 6th St corridor)",
          "properties_analyzed": 34,
          "price_appreciation_12mo": 8.2,
          "price_appreciation_trend": "accelerating",
          "days_on_market": 12,
          "price_per_sqft": 192,
          "investment_signal": "strong_appreciation_momentum",
          "recommended_strategy": "buy_hold_or_flip",
          "investor_activity": "high"
        },
        {
          "neighborhood": "Mueller (North Austin development)",
          "properties_analyzed": 28,
          "price_appreciation_12mo": 6.5,
          "days_on_market": 15,
          "price_per_sqft": 178,
          "investment_signal": "emerging_market_good_value",
          "recommended_strategy": "buy_hold",
          "supply_trend": "decreasing"
        }
      ],
      "stagnant_markets": [
        {
          "neighborhood": "South Congress (gentrified, high prices)",
          "properties_analyzed": 41,
          "price_appreciation_12mo": 1.2,
          "days_on_market": 32,
          "price_per_sqft": 215,
          "investment_signal": "saturated_avoid",
          "notes": "High prices, slow sales, limited appreciation upside"
        }
      ]
    },
    "opportunity_properties": [
      {
        "address": "1247 Ridgemont Drive, Austin, TX 78702",
        "list_price": 385000,
        "comparable_sale_avg": 412000,
        "price_discount": "6.6_percent_below_comps",
        "price_per_sqft": 168,
        "neighborhood_avg_price_per_sqft": 185,
        "sqft": 2290,
        "beds": 3,
        "baths": 2.5,
        "daysOnMarket": 42,
        "opportunity_type": "underpriced_or_long_market",
        "estimated_renovation_needed": "none",
        "flip_potential": "moderate",
        "hold_potential": "strong"
      }
    ],
    "price_analysis": {
      "most_common_price_per_sqft_range": "175-195",
      "outliers_worth_investigating": [
        {
          "address": "2156 Oak Hill Road",
          "price_per_sqft": 142,
          "neighborhood_avg": 185,
          "variance": "23_percent_below_market",
          "potential_reason": "needs_renovation"
        }
      ]
    }
  }
}

This is immediately actionable. You can see that East Austin is hot and accelerating. Mueller is emerging and still has value. South Congress is mature and not worth the premium prices. You have specific properties flagged as undervalued.

Who This Is For: Real-World Applications

Buy-and-Hold Investors: You're looking for markets with appreciation potential and cash flow stability. This analysis shows you where prices are appreciating steadily, inventory is tight, and properties aren't sitting on the market (indicating genuine demand). You can identify neighborhoods before they become obvious to the broader market.

House Flippers: You need to find undervalued properties with renovation potential. The analysis flags properties selling below comparable value or sitting longer than the market average (often a sign of motivated sellers or hidden issues). You can identify quick opportunities without reviewing 300+ listings manually.

Real Estate Agents: You need to position yourself as a market expert. Automated neighborhood analysis gives you data to show clients—which areas are appreciating, which are saturated, where inventory is tight. This builds authority and helps clients make better decisions.

Commercial Real Estate Teams: You're analyzing multiple neighborhoods or markets. Automated extraction and analysis let you compare across markets quickly, identify emerging areas before everyone else, and spot investment opportunities at scale.

Property Managers: You're evaluating neighborhoods for portfolio expansion. Analysis shows you where comparable rents are stable, where tenant demand is strong, and where renovation costs are trending. You can make better acquisition decisions.

Concrete Workflow: From Data to Decision

Here's how this works in practice:

Extract Neighborhoods: Run the scraper on 3-5 neighborhoods you're considering investing in. Each run takes 5-10 minutes.
Analyze Tiers: Review the tracker summary. Identify which neighborhoods show strong signals: price appreciation, declining days-on-market, tight inventory, low days-on-market.
Review Opportunities: The tracker automatically flags properties that are underpriced or sitting longer than average. These are candidates for deeper analysis.
Comparable Analysis: For flagged properties, compare against the neighborhood data. A $350k property in a neighborhood averaging $400k price per sqft is interesting. Calculate potential appreciation or renovation ROI.
Decision: You now have data-driven candidates. Visit the top 5-10, make offers on the best 1-2. What would have taken 6 hours of manual research is done in 1.

Getting Started: Access the Actor

The Apify Redfin Scraper is available at: https://apify.com/nexgendata/redfin-scraper?fpr=2ayu9b

Start with one neighborhood or zip code. Run the analysis once. Review the output. You'll immediately see patterns that would have taken hours to discover manually. Once you understand the data, scale to multiple neighborhoods, set up recurring analysis on your target markets, and build investment decisions on data rather than intuition.

Why This Matters

Real estate markets move on information asymmetry. The investors who win are the ones who know something about supply, demand, or value that others don't. They don't know it through luck—they know it through data.

Automated Redfin analysis gives you systematic access to market data. You can analyze more properties in less time. You can identify opportunities faster than manual researchers. You can scale your analysis across geographies that others consider "too much work to analyze."

The properties you identify through data-driven analysis and the opportunities you capture because you moved faster than manual researchers will more than justify the cost of automation.

Your next investment shouldn't be based on a feeling or a tip. It should be based on data. That data is on Redfin right now. You just need to extract and analyze it systematically.

Build a Daily Series A/B Funding Tracker from SEC Form D + TechCrunch (2026)

NexGenData — Thu, 14 May 2026 07:08:03 +0000

Build a Daily Series A/B Funding Tracker from SEC Form D + TechCrunch (2026)

Crunchbase Pro is $588/month per seat. PitchBook starts higher than that. CB Insights doesn't even publish list pricing. If you're a VC associate, an SDR at a B2B startup, a financial journalist, or an analyst who needs to know which companies just raised money — and you don't already have a six-figure data budget — you've probably spent more time than you'd like to admit cobbling together a funding-round dashboard from RSS feeds and SEC searches.

The annoying part is: most of the underlying data is public. Every U.S. company raising money under Reg D files Form D with the SEC within 15 days of the first sale. TechCrunch, Axios Pro Rata, and Strictly VC publish funding-round summaries daily. Y Combinator publishes its full alumni roster. Stripe Atlas makes a fresh batch of Delaware C-corps every week.

Stitched together properly, those four sources will give you 80% of the coverage Crunchbase has — the day each round happens — for roughly $5 a month in compute costs.

This post walks through the architecture I built for a multi-strategy crossover fund's sourcing team, originally an internal tool, since productized as the NexGenData Startup Funding Tracker actor. Whether you build it yourself or use the actor, the architecture is the same.

What Public Funding Data Actually Looks Like

The SEC's Form D filing is the single most underrated funding-tracking source. Any U.S. company raising money under Regulation D — which is virtually every priced equity round, from pre-seed safes to Series F — has to file within 15 days. The filing includes the issuer name, address, executive officers, total amount raised, total number of investors, and minimum investment per investor.

What it doesn't include: the round name (you have to infer Seed vs Series A from amount + investor count), the lead investor (Form D lists "Related Persons" but not lead status), or the post-money valuation (Reg D filings don't disclose valuation). For everything Form D doesn't give you, TechCrunch's funding-round articles fill in. For everything still missing after that, Y Combinator's alumni database is the canonical source for YC-batch and demo-day companies.

The clever piece is the join key. Form D filings publish in EDGAR with the issuer's exact legal name. TechCrunch articles use the company's marketing name. YC's database uses the brand name. The same company shows up as "FlexPort, Inc." in Form D, "Flexport" in TechCrunch, and "Flexport" in YC. A fuzzy-match join (Levenshtein distance + domain TLD normalization) reconciles them at >95% precision.

Source 1: SEC EDGAR Form D Filings

EDGAR's Form D feed is at https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&type=D&dateb=&owner=include&count=40. The HTML index page is paginated, but the underlying data is available as XML.

import httpx
from datetime import date, timedelta

async def fetch_recent_form_d(days_back: int = 1) -> list[dict]:
    """Pull Form D filings from the last N days."""
    today = date.today()
    start = today - timedelta(days=days_back)

    url = "https://efts.sec.gov/LATEST/search-index"
    params = {
        "q": "",
        "dateRange": "custom",
        "startdt": start.isoformat(),
        "enddt": today.isoformat(),
        "forms": "D",
    }
    headers = {"User-Agent": "your-name your-email@domain.com"}

    async with httpx.AsyncClient(headers=headers, timeout=30) as client:
        r = await client.get(url, params=params)
        r.raise_for_status()
        return r.json().get("hits", {}).get("hits", [])

A few critical notes. SEC EDGAR enforces a strict 10-requests-per-second rate limit and requires a User-Agent header that includes your name and email. They will rate-limit and eventually IP-ban abusive scrapers — be polite. The full-text search index lags the actual filing by 30-60 minutes; if you need real-time, use the daily form index at https://www.sec.gov/cgi-bin/browse-edgar?action=getcurrent.

Each filing returns a CIK (Central Index Key) — that's the SEC's unique company identifier. From the CIK you can pull the actual XML filing:

async def fetch_filing_xml(cik: str, accession: str) -> str:
    accession_clean = accession.replace("-", "")
    url = f"https://www.sec.gov/Archives/edgar/data/{cik}/{accession_clean}/primary_doc.xml"
    async with httpx.AsyncClient(headers=headers) as client:
        r = await client.get(url)
        return r.text

The XML includes <offeringData> with totalOfferingAmount, totalAmountSold, totalNumberAlreadyInvested, and the related-persons list. Parse with lxml or xmltodict.

Source 2: TechCrunch Funding Articles

TechCrunch's funding tag (https://techcrunch.com/category/fundings-exits/) publishes 8-15 articles per business day. Each article reliably includes the company name, round size, lead investor, and (sometimes) valuation. The headline pattern is consistent enough that a lightweight regex catches 90% of mentions:

import re

ROUND_PATTERN = re.compile(
    r"^(?P<company>[A-Z][A-Za-z0-9\s\-\.\']+?)\s+"
    r"(?:raises|secures|closes|nabs|lands|gets)\s+"
    r"\$(?P<amount>[\d\.]+[MB]?)\s+"
    r"(?:in\s+)?(?P<round>Series\s+[A-Z]|seed|pre-seed)?",
    re.IGNORECASE,
)

def parse_techcrunch_headline(title: str) -> dict | None:
    m = ROUND_PATTERN.search(title)
    if not m:
        return None
    return {
        "company": m.group("company").strip(),
        "amount_raw": m.group("amount"),
        "round": (m.group("round") or "unknown").strip(),
    }

For richer extraction (lead investor, valuation, board seats), parse the article body. The lead investor is almost always in the first paragraph after the headline; valuation, when disclosed, is in the second or third.

Source 3: Y Combinator Alumni Database

YC publishes its full alumni list at https://www.ycombinator.com/companies. The page is client-rendered, but the underlying Algolia search index is queryable directly with no auth:

async def search_yc_companies(query: str = "") -> list[dict]:
    url = "https://45bwzj1sgc-dsn.algolia.net/1/indexes/YCCompany_production/query"
    payload = {"query": query, "hitsPerPage": 1000}
    headers = {
        "X-Algolia-Application-Id": "45BWZJ1SGC",
        "X-Algolia-API-Key": "Y2VkOWQyMTJlYjZkZjE3MDRkY2YyNjBmYmIzMjVhMzA1ZmRlYTQ4OTUyZjEyZjRiNzc0OWQ4MjRmMzVlYmUxN3RhZ0ZpbHRlcnM9JTViJTIyJTVEJmZpbHRlcnM9aXNIaXJpbmclM0F0cnVl",
    }
    async with httpx.AsyncClient(headers=headers) as client:
        r = await client.post(url, json=payload)
        return r.json().get("hits", [])

(The API key shown is the public read-only key embedded in YC's frontend bundle — it's safe to use, but rotate periodically as YC occasionally re-keys it.)

YC hits include name, slug, batch, industry, team_size, description, website, and regions. Cross-reference with Form D and TechCrunch on company name (Levenshtein <= 2) plus website-domain match.

Source 4: Delaware Corporation Filings

This is the most underrated source for very-early-stage tracking. Every Delaware C-corp incorporates through the Division of Corporations. Stripe Atlas, the most common formation service for tech startups, batches roughly 200-400 incorporations per week. The DOC search at https://icis.corp.delaware.gov/Ecorp/EntitySearch/NameSearch.aspx returns entity name, file number, formation date, and registered agent.

A new Stripe-Atlas-formed C-corp with a brand-able name and a ".com" domain reserved within 24 hours of formation is, statistically, a future YC application. Sourcing teams who flag these patterns get a 2-3 week jump on Crunchbase.

The Joining Layer

Once you've ingested all four sources into a normalized schema (company_name, legal_name, domain, round_amount, round_type, source, source_url, event_date), the joining problem becomes the harder problem.

Naive name matching fails on edge cases like "Stripe, Inc." vs "Stripe" vs "Stripe Inc" vs "stripe.com." The reliable approach: build a normalized join key from (slugify(legal_name), root_domain). If you have a domain in any source, root-domain match is your ground truth. If you don't (Form D doesn't publish domains), fall back on slugified name + executive-officer name match.

def normalize_company_name(name: str) -> str:
    """Strip Inc/LLC/Corp suffixes, lowercase, slugify."""
    suffixes = [", Inc.", " Inc.", ", LLC", " LLC", ", Corp.", " Corp.",
                ", Ltd.", " Ltd.", ", Co.", " Co."]
    for s in suffixes:
        if name.endswith(s):
            name = name[:-len(s)]
            break
    return "".join(c.lower() if c.isalnum() else "-" for c in name).strip("-")

Putting It All Together

The full pipeline runs as a daily cron at 3am ET (after Form D's nightly batch posts):

Fetch Form D filings since yesterday → ~80-150 records/day
Scrape TechCrunch funding articles since yesterday → ~10-15 records/day
Refresh YC active-batch alumni → only changes with new batches
Fetch new Delaware C-corps since yesterday → ~50-100 records/day
Normalize, dedupe by (slug, domain), join → ~150-200 unique companies/day
Push to a daily-digest output

That's a Series A/B/seed dashboard updated daily, costing roughly $4-6/month in compute, with coverage that overlaps Crunchbase's daily updates by ~85%.

The Faster Way: Use the Actor

If you'd rather not build this yourself, the NexGenData Startup Funding Tracker actor does exactly this pipeline, charging $0.01 per Form D / TechCrunch / YC record. A typical daily run returns 150-200 records for $1.50-$2.00.

The actor accepts filters for date range, minimum round amount, industry tags, and source mix. Output schema matches the join schema described above.

Free to clone the architecture or just point your data pipeline at the actor — either way, you've got a Crunchbase-tier funding pipeline at <1% of the cost.

If you find this useful, NexGenData has 195+ public actors covering similar buyer-intent domains: lead generation, SEC filings, YC alumni tracking, and dozens more. Browse the full catalog.

How to Scrape Google Maps Business Data for Lead Generation in 2026

NexGenData — Thu, 14 May 2026 07:07:17 +0000

Google Maps is a goldmine of business intelligence. It contains millions of businesses with real-time contact information, customer reviews, operating hours, and location data. But manually collecting this data is soul-crushing work.

If you're a marketer hunting leads, a competitor analyst tracking moves, a researcher building market datasets, or a startup scaling outreach—you need a way to extract Google Maps business data at scale. This guide shows you exactly how to do it.

The Problem: Google Maps Data is Valuable But Locked Away

Let's be honest: Google Maps has some of the most accurate business data on the internet. It's more reliable than LinkedIn, more current than Yellow Pages, and it's where customers actually leave feedback about businesses.

What's inside that goldmine?

Business names and exact locations (verified by Google)
Phone numbers and websites (often multiple contacts)
Star ratings and review counts (social proof)
Operating hours (real-time availability)
Photos and street view data
Business categories and specialties
Customer reviews and review text
Email addresses and social media links (when available)
Price ranges
Website links

The catch? Google actively blocks automated access. If you try to scrape Google Maps directly with a simple script, you'll hit rate limits within minutes. You'll get blocked. You'll waste days building infrastructure to work around their protections.

That's where most companies give up. They resort to expensive third-party data brokers, outdated CSV downloads, or manual spreadsheet work.

There's a better way.

Why Google Maps Data Matters for Your Business

Before we dive into how, let's talk about why you need this data:

Lead Generation

If you're in B2B sales (SaaS, agencies, professional services), Google Maps is packed with qualified leads. Small businesses, restaurants, dental offices, contractors—they're all there with their phone numbers and websites. Scrape a list of 10,000 plumbers in your region, and you've got a sales funnel primed and ready.

Competitive Analysis

Want to know where your competitors are expanding? Which neighborhoods have the highest density of similar businesses? What customer sentiment looks like compared to your own reviews? Google Maps gives you that intel in real-time.

Local SEO Research

Planning a local campaign? Need to understand the competitive landscape in specific geographic areas? Google Maps search results tell you exactly what Google thinks is relevant in each market.

Market Research

Launching a new product or service? Use Google Maps data to validate market size, understand customer distribution, identify underserved regions, and benchmark against existing competitors.

Lead Enrichment

You have a list of business names and addresses, but missing phone numbers or websites? Scrape Google Maps to fill in those gaps and enrich your existing databases.

The companies doing this at scale are closing more deals, finding better markets faster, and understanding their competition better than their slower competitors.

The Manual Approach: Why It Doesn't Scale

Let me paint a picture. You open Google Maps in your browser. You search for "plumbers in Denver." You see 20 results per page. You manually click each one, noting the phone number, website, address, and rating in a spreadsheet. You do this for 10 pages. That's 3-4 hours of clicking and typing to get 200 business records.

Now imagine doing this for 50 cities across multiple business categories. You're looking at hundreds of hours of repetitive, error-prone manual work. By the time you finish, the data is already stale.

And here's the real problem: even if you do this work, you're violating Google's Terms of Service. Google explicitly prohibits automated access to their platform, but they also have sophisticated bot detection for manual scrapers who are clearly doing systematic data extraction. If they notice you, your IP gets flagged and your browser access gets blocked.

This is why the manual approach is dead. You need automation, but you need it done right.

The Easy Way: Using the NexGenData Google Maps Scraper

Here's where scraping Google Maps becomes actually practical. The NexGenData Google Maps Scraper on Apify handles all the complexity:

Proxy rotation so you don't get blocked
Real browser control so Google thinks you're human
Intelligent rate limiting to stay under the radar
Structured data extraction so you get clean JSON, not messy HTML
Scale from 10 results to 10,000+ results
Pricing that costs pennies per result

You define your search parameters, run the actor, and get a clean dataset you can use immediately. No browser clicks, no manual data entry, no IP bans.

Let me show you exactly how it works.

Getting Started: API Setup and Authentication

First, you'll need an Apify account. Head to apify.com and sign up if you haven't already. The free tier gives you compute credits monthly, which is usually enough to test the actor.

Once you're logged in, grab your API token from Account Settings. You'll use this to authenticate API calls.

Here's the JavaScript setup using the Apify SDK:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({
    token: process.env.APIFY_API_TOKEN, // Store this in your .env file
});

Pro tip: Never hardcode your API token. Use environment variables.

Running Your First Scrape: Complete Example

Let's build a realistic example. You want to find all plumbers in Denver, Colorado, extract their contact info, and save the results to a file.

import { ApifyClient } from 'apify-client';
import fs from 'fs';

const client = new ApifyClient({
    token: process.env.APIFY_API_TOKEN,
});

async function scrapeGoogleMapsBusinessData() {
    // Actor input configuration
    const input = {
        searchStringsArray: ['plumbers in Denver, Colorado'],
        // You can search multiple queries
        // searchStringsArray: ['plumbers in Denver, CO', 'plumbers in Boulder, CO'],

        // How many results per search (max 120 per search)
        maxResults: 50,

        // Search on Google Maps directly (vs. Google Search)
        includeWebResults: false,

        // Extract additional data like reviews
        includeReviews: true,
        maxReviewsPerPlace: 5, // Limit reviews to save on pricing

        // Optional: filter by min/max rating
        // minReviewsCount: 10,
        // minRating: 4.0,
    };

    console.log('Starting Google Maps scrape...');
    console.log(`Searching for: ${input.searchStringsArray[0]}`);

    try {
        // Run the actor
        const run = await client.actor('nexgendata/google-maps-scraper').call(input);

        // Get the results
        const { items } = await client.dataset(run.defaultDatasetId).listItems();

        // Print the results
        console.log(`Found ${items.length} results:`);
        items.forEach((business, index) => {
            console.log(`\n${index + 1}. ${business.title}`);
            console.log(`   Phone: ${business.phoneNumber || 'N/A'}`);
            console.log(`   Website: ${business.website || 'N/A'}`);
            console.log(`   Address: ${business.address}`);
            console.log(`   Rating: ${business.review?.rating || 'N/A'} stars`);
            console.log(`   Reviews: ${business.review?.reviewsCount || 0}`);
        });

        // Save to JSON file
        fs.writeFileSync(
            'denver-plumbers.json',
            JSON.stringify(items, null, 2)
        );
        console.log(`\nResults saved to denver-plumbers.json`);

        return items;
    } catch (error) {
        console.error('Scrape failed:', error);
        throw error;
    }
}

// Run it
scrapeGoogleMapsBusinessData();

Save this as scrape-maps.js and run it:

export APIFY_API_TOKEN="your_token_here"
node scrape-maps.js

That's it. Within seconds to minutes (depending on how much data you're pulling), you'll have a JSON file with clean, structured business data.

What You'll Get: Example Output

Here's what the output data structure looks like:

[
  {
    "title": "ABC Plumbing Services Denver",
    "address": "1234 Broadway, Denver, CO 80202",
    "location": {
      "lat": 39.7392,
      "lng": -104.9903
    },
    "phoneNumber": "+1-720-555-1234",
    "website": "https://abcplumbing.com",
    "type": "Plumber",
    "review": {
      "rating": 4.8,
      "reviewsCount": 142,
      "reviews": [
        {
          "reviewer": "John D.",
          "reviewText": "Great service, very professional",
          "reviewRating": 5,
          "reviewPublishedAtDate": "2026-03-15"
        }
      ]
    },
    "openingHours": "9:00 AM - 6:00 PM",
    "businessStatus": "Operational",
    "permanentlyClosed": false,
    "imageUrl": "https://lh3.googleusercontent.com/...",
    "url": "https://www.google.com/maps/place/ABC+Plumbing"
  }
]

This is ready to use immediately. No parsing, no cleaning, no extracting phone numbers from messy text. Every field is structured and validated.

Advanced: Scraping Multiple Cities at Once

Want to scrape plumbers across multiple markets? Here's how:

const input = {
    searchStringsArray: [
        'plumbers in Denver, Colorado',
        'plumbers in Boulder, Colorado',
        'plumbers in Fort Collins, Colorado',
        'plumbers in Colorado Springs, Colorado',
        'plumbers in Aurora, Colorado',
    ],
    maxResults: 50,
    includeReviews: true,
    maxReviewsPerPlace: 3,
};

The actor will run all searches and aggregate the results into a single dataset. You can pull data from multiple geographic areas, business categories, or search queries in one execution.

How Much Does It Cost?

This is the question everyone asks first (rightfully so). The NexGenData Google Maps Scraper uses Apify's PPE (Pay Per Event) pricing model.

At current rates (2026), you're looking at approximately:

$0.01 - $0.03 per business record (includes basic data extraction)
Add $0.01 - $0.02 per record if you include detailed reviews
Bulk discounts if you're scraping thousands of records

Real-world example: Scraping 5,000 plumbers across 10 cities with reviews included costs around $50-75. That's less than what you'd spend on a coffee run, and you get a sales-ready lead list.

Compare that to:

Lead database services: $500-2,000 per month
Manual work: Hundreds of hours (at $20/hour = thousands of dollars)
Old data brokers: $1,000+ for stale data

When you do the math, scraping Google Maps is absurdly cost-effective.

The MCP Server Alternative: For AI Agent Users

If you're building AI agents or automation workflows, there's another way to access this data without writing custom code.

The NexGenData Google Maps MCP Server is a Model Context Protocol server that integrates Google Maps scraping directly into your agent architecture. Think of it as a tool your AI agent can call natively.

Instead of writing JavaScript, you define your agent's capabilities like this:

# With the MCP server integrated
agent.add_tool("search_google_maps", {
    "description": "Search and extract business data from Google Maps",
    "parameters": {
        "search_query": "plumbers in Denver, Colorado",
        "max_results": 50,
        "include_reviews": True,
    }
})

Now your AI agent can:

Answer user questions about local businesses
Generate lead lists on demand
Build competitive intelligence reports
Perform market research automatically
Integrate Google Maps data into multi-step workflows

The MCP server handles all the complexity (proxies, rate limiting, data parsing) while your agent focuses on the logic. This is powerful if you're building multi-step automation where Google Maps data is just one piece of a larger workflow.

Learn more about the MCP server at: apify.com/nexgendata/google-maps-mcp-server

Real-World Use Cases

Let me show you how this works in practice across different scenarios:

Use Case 1: Sales Prospecting (SaaS Company)

You're selling accounting software to small businesses. You scrape Google Maps for "accounting firms in [major cities]" and get 10,000 qualified leads with phone numbers and websites. You build a prospect list, run a phone outreach campaign, and close deals in weeks instead of months.

Cost: $100 in scraping credits
Revenue impact: Could easily lead to $100K+ in deals

Use Case 2: Competitive Analysis (Fitness Startup)

You're launching a luxury fitness brand in Seattle. You scrape data for "gyms in Seattle" and get 200+ competitors with their locations, ratings, review counts, and customer sentiment. You identify underserved neighborhoods and price points that are working. You launch your first location in an area where competitors have weak ratings.

Cost: $25 in scraping credits
Strategic impact: Better market positioning from day one

Use Case 3: Lead Enrichment (B2B Services)

Your sales team has a spreadsheet of 1,000 business names and addresses but missing phone numbers and websites. You scrape Google Maps to match and enrich the records. Your outreach team suddenly has complete contact information and can execute their campaign.

Cost: $50 in scraping credits
Productivity gain: 40+ hours of manual research eliminated

Use Case 4: Market Research (VC Due Diligence)

A portfolio company claims they have a huge addressable market in local services. You scrape Google Maps to validate: "air conditioning repair in California," "plumbing in Texas," etc. You get actual count of competitors, pricing ranges, and customer sentiment. You now have hard data instead of assumptions.

Cost: $100 in scraping credits
Decision impact: Accurate market sizing for investment decisions

Best Practices for Scraping at Scale

If you're going to scrape Google Maps data regularly, follow these practices:

1. Respect Rate Limits

The actor handles this automatically, but don't spam multiple simultaneous runs with aggressive parameters. Spread your requests out. One large run is better than 10 small ones hitting the same data.

2. Use Filters

The actor supports filtering by minimum rating, minimum review count, price range, and business status. Use these to get only the data you actually need, which saves money and improves quality.

3. Start Small

Test with 20-50 results first. Make sure the data quality meets your needs. Then scale up.

4. Schedule Periodic Updates

Business data changes. Addresses change, phone numbers get reassigned, reviews accumulate. Set up monthly or quarterly scrapes to keep your data fresh. Use the actor's scheduling features for this.

5. Clean and Deduplicate

Even with the actor's data validation, you might get duplicates (same business, slightly different name format). Run a quick deduplication step after each scrape.

6. Store Responsibly

If you're storing Google Maps data, be clear about where it came from and how you're using it. Some regions have data protection regulations (GDPR, CCPA) that affect how you can store and use business contact information.

Common Questions Answered

Q: Is scraping Google Maps legal?
A: It's a gray area legally, but the actor uses legitimate methods (proxies, controlled scraping patterns, respecting robots.txt where applicable). Google's ToS technically prohibit it, but enforcement focuses on clearly malicious actors. Using the data for legitimate business purposes (lead generation, market research) is standard practice.

Q: Will I get IP banned?
A: Not with the actor. It handles proxy rotation, request timing, and behavioral patterns automatically. You won't look like a bot.

Q: How fresh is the data?
A: Business information on Google Maps is constantly updated by Google's crawlers and users' contributions. What you scrape reflects the current state of Google's index, which is usually within days of real changes.

Q: Can I scrape reviews?
A: Yes. The actor can extract review text, ratings, reviewer names, and review dates. This is useful for sentiment analysis and brand monitoring.

Q: What if a business isn't on Google Maps?
A: If they don't have a Google My Business listing, they won't appear. But most businesses do—it's table stakes for online visibility.

Comparing: Actor vs. MCP Server vs. Manual

Approach	Cost	Speed	Effort	Scalability	Best For
Manual (Browser)	$0 upfront	10-50 results/hour	Extreme	None	Small one-time projects
DIY Selenium Script	$0-100 (proxies)	Slow, unreliable	High	Poor	Learning/experimentation
NexGenData Actor	$0.01-0.03/result	1000+ results/min	Low	Excellent	Developers, data teams
MCP Server	$0.01-0.03/result	Instant	Low	Excellent	AI agents, automation

The Complete Workflow

Here's the end-to-end process you'd follow in a real business scenario:

// 1. Define your search parameters
const searchQueries = [
    'plumbers in Denver, Colorado',
    'plumbers in Boulder, Colorado',
    'electrical contractors in Denver, Colorado',
];

// 2. Run the scraper
const input = {
    searchStringsArray: searchQueries,
    maxResults: 50,
    includeReviews: true,
    maxReviewsPerPlace: 5,
    minRating: 3.5, // Optional filter
};

// 3. Collect and structure the data
// (handled by the actor)

// 4. Enrich the data
const enriched = items.map(item => ({
    ...item,
    scrapedAt: new Date(),
    dataSource: 'google-maps',
}));

// 5. Store in your database
await database.insertMany(enriched);

// 6. Use for business purposes
// - Feed into sales CRM
// - Run outreach campaigns
// - Analyze competitor data
// - Track market trends

Conclusion: Stop Manually Collecting Data

You're sitting on a goldmine of business data. Google Maps contains the information you need to:

Generate qualified leads
Understand competitive landscapes
Validate business ideas
Enrich customer databases
Research markets

The only question is whether you're going to access it manually (slow, error-prone, unsustainable) or smartly with automation (fast, reliable, scalable).

The NexGenData Google Maps Scraper makes it cheap, easy, and fast. At pennies per result, there's no reason not to try it.

Next steps:

Try the free tier: Sign up at apify.com and test the actor with 50 results. See the data quality for yourself.
Build your first scrape: Use the JavaScript code examples from this post. It should take 10 minutes to get working data.
Scale your use case: Once you see it works, scale to your full business need. Hundreds or thousands of records.
If you're building agents: Check out the MCP Server instead for native integration.

Stop settling for stale data, manual work, and expensive third-party services. Start scraping Google Maps data today.

Resources:

How to Monitor Hacker News Trends for Product Launch Timing

NexGenData — Thu, 14 May 2026 07:07:15 +0000

How to Monitor Hacker News Trends for Product Launch Timing

Hacker News is where engineers, technical founders, and early adopters converge. If you're launching a technical product, an AI tool, a developer platform, or anything targeting technical audiences, HN is where your customers are paying attention.

But here's the problem: HN moves fast. A post can hit the front page, accumulate 300 comments, and then fall off into obscurity within 24 hours. If you're not watching in real time, you miss the signal. You don't see what topics are resonating, which problems are getting attention, and what messaging angles actually move technical audiences.

Most teams launching technical products make timing decisions based on hunches: "Let's launch on a Tuesday morning." But what if you could time your launch based on actual HN engagement data? What if you knew that discussions about infrastructure are peaking right now, while discussions about machine learning are cooling down? That's the difference between a launch that gets 100 upvotes and one that gets 2,000.

The real opportunity is understanding what conversations are happening on HN right now, how engaged the community is, and whether your product is hitting during a window when related topics are hot.

Why Hacker News Timing Matters for Technical Launches

When a product launches on HN at the right moment, in the right category, with the right narrative, it creates a flywheel. Early upvotes lead to front page placement, which leads to more visibility, which attracts more quality feedback, which improves the product, which attracts more users. Miss the timing window and you're posting into a dead space where nobody sees it.

Beyond launch timing, HN engagement data tells you what problems the technical community cares about. If you see 15 posts about Rust performance hitting the front page in the last month, but only 2 posts about Python, that's information. It means engineers are thinking about Rust. If you're building a tool, this influences where you build it.

For founders pre-launch, HN engagement data is market research. You can see which problems are getting the most discussion, which solutions are being debated, and which gaps are obvious to the community. If nobody's talking about your problem space, you either have a timing issue or a market problem.

For teams post-launch, HN data helps you identify follow-up launch opportunities. Your initial launch might have emphasized feature A, but the top 20 comments were all about feature B. That's a signal to launch an updated version emphasizing B, and you'd time that launch for when discussions about B are trending.

The Solution: Real-Time HN Trend Monitoring

Instead of manually scrolling HN every morning, you can use an intelligent scraper to extract engagement data across top stories, analyze which topics are trending, and get alerts when discussions related to your space are peaking.

The Apify Hacker News Scraper doesn't just pull post metadata—it can synthesize insights showing you which topics are trending, which types of posts are getting the most engagement, and what the community is actually focused on right now.

Think of it as having a researcher monitoring HN 24/7, annotating which stories are about infrastructure, which are about machine learning, which are about startups, and which are getting the strongest engagement. You get a dashboard view of "what the community cares about today" and can time your decisions accordingly.

How It Works: From Raw Data to Engagement Tiers

Here's what the workflow looks like:

The scraper fetches top stories from HN (front page, new stories, best stories)
For each story, it extracts: title, URL, author, timestamp, score, comment count, and comments themselves
When tracker mode is enabled, it analyzes patterns:
- Which topics appear most frequently
- Which topics are getting the strongest engagement (score vs comment ratio)
- How recently engagement is happening (to identify cooling vs heating topics)
- Which authors are repeatedly getting traction (influencers in specific spaces)
Returns both raw data and synthesized insights

The result is a structured engagement analysis you can use to understand the market in real time.

Sample Output: Tracker Summary with Engagement Tiers

Here's what a typical tracker output looks like:

{
  "tracker_summary": {
    "report_date": "2026-04-05",
    "analysis_period_hours": 24,
    "total_stories_analyzed": 450,
    "engagement_tiers": {
      "hot_topics": [
        {
          "topic": "AI/Large Language Models",
          "story_count": 47,
          "avg_score": 325,
          "avg_comments": 142,
          "momentum": "accelerating",
          "recent_examples": [
            {
              "title": "Anthropic releases Claude 4.5",
              "score": 2847,
              "comments": 487,
              "discussion_tone": "excited_adoption"
            }
          ]
        },
        {
          "topic": "Open Source Infrastructure",
          "story_count": 34,
          "avg_score": 287,
          "avg_comments": 89,
          "momentum": "sustained",
          "recent_examples": [
            {
              "title": "Kubernetes security vulnerabilities in 1.30 release",
              "score": 1563,
              "comments": 204,
              "discussion_tone": "critical_analysis"
            }
          ]
        }
      ],
      "emerging_topics": [
        {
          "topic": "Quantum Computing Practical Applications",
          "story_count": 8,
          "avg_score": 156,
          "avg_comments": 34,
          "momentum": "rising",
          "confidence": "medium"
        }
      ],
      "cooling_topics": [
        {
          "topic": "Blockchain/Web3",
          "story_count": 3,
          "avg_score": 42,
          "avg_comments": 8,
          "momentum": "declining",
          "notes": "Significantly less traction than 3 months ago"
        }
      ]
    },
    "comment_sentiment_analysis": [
      {
        "topic": "AI/LLMs",
        "sentiment_breakdown": {
          "positive": 62,
          "neutral": 28,
          "critical": 10
        },
        "key_concerns": [
          "Pricing and accessibility",
          "Privacy of training data",
          "Energy consumption"
        ]
      }
    ],
    "launch_timing_recommendations": {
      "optimal_windows": [
        {
          "topic": "AI/LLMs",
          "status": "hot_ascending",
          "recommended_angle": "Developer efficiency, time savings",
          "timing": "immediate (next 48 hours)",
          "confidence": 92
        }
      ],
      "avoid_launching": [
        {
          "topic": "Blockchain",
          "reason": "declining_engagement",
          "timing": "wait at least 2 weeks"
        }
      ]
    }
  }
}

This is immediately actionable. You can see that AI/LLMs are hot and accelerating. You can see that quantum computing is emerging but unproven. You can see that blockchain interest is dead. You can see what the community is actually saying about different topics.

If you're launching an AI developer tool, you launch this week while momentum is high. If you're launching a blockchain tool, you wait or pivot your narrative. If you're launching a quantum computing library, you position it carefully since the community is still exploring applications.

Practical Use Cases: When This Changes Everything

Pre-launch Product Validation: Before building your MVP, run the HN scraper to see if anyone's discussing your problem space. If you see 0 stories about your specific pain point but 40 stories about adjacent problems, you know the market might not exist yet—or you've found an unmet gap.

Launch Timing: You've built your product. Now check HN trends. If your space is hot right now, you post this week. If momentum is declining, you wait a month and check again. You're not launching into a void—you're launching when the audience is primed.

Feature Prioritization: You see that HN discussions about your category emphasize feature X heavily. Your roadmap prioritizes feature Y. This data suggests you should accelerate X and reconsider Y.

Content Strategy: You're writing blog posts and tutorials. Write about what HN is discussing. If infrastructure discussions are trending, write about infrastructure applications of your tool. You'll get more traction because you're writing about what people are thinking about.

Competitive Intelligence: You see your competitor launched on HN yesterday and got 1,200 upvotes. The tracker tells you their post resonated with the "emerging founders" segment and hit during optimal timing for developer tools. You learn from their timing and approach on your next launch.

Getting Started: Three Steps to Data-Driven Launch Decisions

Access the Actor: Navigate to https://apify.com/nexgendata/hacker-news-scraper?fpr=2ayu9b
Establish Your Baseline: Run the tracker once to see what's trending right now. Identify 3-5 key topics related to your product space.
Set Up Recurring Monitoring: Schedule the scraper to run daily. Store tracker summaries so you can see how engagement with your topics evolves over time. When you see momentum building in your space, start planning your launch for 1-2 weeks out.

The Timing Advantage

Most technical product launches are timing-blind. You pick a day that feels right, post to HN, and hope for the best. But engagement data exists. You can see which topics are hot, which are cooling, which are emerging. You can see which discussion angles are resonating with the community.

That's not guessing anymore. That's strategy.

Real launches in your space might get 20-30 upvotes with better community and timing can turn that into 500+. The difference isn't usually the product—it's the timing and narrative. HN trend data tells you both.

Start monitoring your space today. Track the trends for a week. You'll immediately see patterns that inform your go-to-market decisions. Your launch timing will improve. Your product positioning will align better with what the market is actually discussing. That's the leverage point.

Your technical audience is on Hacker News right now, talking about your problem space. Wouldn't you rather know exactly when they're most receptive?

CRXcavator Is Dead — Here's How to Audit Chrome Extensions in 2026

NexGenData — Thu, 14 May 2026 07:07:13 +0000

If you've ever been responsible for browser security at a company, you probably used CRXcavator. Duo Security built it, Cisco acquired Duo, and then quietly killed CRXcavator in 2023. No replacement. No migration path. Just gone.

That left a gap. Your options today are Spin.AI (starting at $5,000/year with enterprise minimums) or doing it manually — downloading each extension, unzipping the CRX file, reading the manifest.json, and trying to figure out what <all_urls> actually means for your attack surface.

Neither option works well if you're a security engineer at a mid-size company, a consultant doing vendor assessments, or an IT admin who just needs to know which of the 47 extensions installed across your org can read every website your employees visit.

What Made CRXcavator Useful

CRXcavator did one thing really well: it took a Chrome extension ID, pulled apart the CRX package, and told you exactly what permissions the extension had and how risky they were. It classified permissions into risk tiers, flagged dangerous combinations, and gave you a score you could put in a compliance report.

The output was structured data you could feed into a SIEM, attach to an audit, or use to build an allowlist policy. That's what made it valuable — not the analysis itself (which is deterministic), but the fact that it was automated and produced machine-readable output.

The Technical Problem

Chrome extensions are distributed as CRX files — essentially ZIP archives with a header. Inside every CRX is a manifest.json that declares every permission the extension requests. These permissions range from benign (storage, alarms) to extremely broad (<all_urls>, webRequestBlocking, debugger).

The analysis itself isn't AI or heuristics. It's a lookup table: <all_urls> means the extension can read and modify every web page. webRequestBlocking means it can intercept, modify, or block every HTTP request. tabs means it can see every URL in every tab. These are facts, not predictions.

What's tedious is the extraction pipeline: downloading the CRX from Google's update servers, finding the ZIP boundary in the binary (CRX3 format prepends a protobuf header before the ZIP data), parsing the manifest, resolving host permissions, checking content script scope, evaluating the CSP, and producing a normalized risk assessment.

Building a Replacement

I built a tool that replicates what CRXcavator did, runs on Apify's cloud infrastructure, and costs $0.20 per extension analyzed. No subscriptions, no minimums, no contracts.

Here's what it does for each extension:

Permission Risk Classification — Every Chrome permission gets a risk score: CRITICAL (10 points), HIGH (7), MEDIUM (4), or LOW (1). Each score comes with a plain-English explanation of what the permission actually allows. No jargon, no acronyms.

Manifest Version Check — Extensions still on Manifest V2 get flagged. Google is phasing out V2 because its permission model is more permissive than V3. An extension on V2 should be on your review list.

Content Script Analysis — If an extension injects JavaScript into web pages, the tool tells you which pages, whether it targets all URLs or specific domains, and whether it runs at document_start (which means it executes before the page even loads).

CSP Evaluation — The Content Security Policy in the manifest controls what the extension's own code can do. unsafe-eval and unsafe-inline directives weaken the extension's own security boundary.

Overall Risk Score — A 0-100 composite score using weighted factors: permissions (heaviest weight), content script scope, manifest version, and CSP quality. The scoring uses exponential normalization so a single CRITICAL permission (like <all_urls>) dominates the score, which matches how security risk actually works.

Example: Auditing uBlock Origin

uBlock Origin is one of the most popular extensions with 10M+ users. Here's what the analysis produces:

Risk Score: 95/100 (CRITICAL)
<all_urls> — CRITICAL: Can access ALL websites
webRequest — HIGH: Can observe all HTTP requests
webRequestBlocking — HIGH: Can intercept and modify all HTTP traffic
tabs — HIGH: Can see every URL in every open tab
Manifest V2 (flagged)
Content scripts targeting http://*/* and https://*/* at document_start

This is correct. uBlock Origin legitimately needs these permissions to function as an ad blocker. A high risk score does not mean an extension is malicious — it means it has significant access that should be reviewed and approved by someone who understands why.

Batch Auditing for Organizations

The real value shows up at scale. If you manage browser policy for a 500-person company, you might have 40-80 unique extensions across your fleet. Running them all through a single API call produces a dataset you can sort by risk score, filter by CRITICAL permissions, and use to build Chrome's ExtensionInstallBlocklist and ExtensionInstallAllowlist policies.

The tracker mode generates a compliance-ready summary: risk distribution percentages, most common dangerous permissions across your fleet, Manifest V2 count, content script scope analysis, and prioritized recommendations. Attach it to your SOC 2 evidence folder or your ISO 27001 Statement of Applicability.

Cost Comparison

Solution	Cost	Notes
CRXcavator	Discontinued	Was free while it lasted
Spin.AI	$5,000/year minimum	Enterprise SaaS
Manual analysis	Free (your time)	~15-30 min per extension
Chrome Extension Security Analyzer	$0.20/extension	Pay-per-use API

Audit 50 extensions: $10.25. Audit 500 extensions: $100.25. No subscription.

How to Use It

The tool runs on Apify, a cloud scraping platform. You can run it through the web UI, the REST API, or the Python SDK.

Quick start (Python):

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run = client.actor("nexgendata/chrome-extension-security-analyzer").call(
    run_input={
        "extensionIds": [
            "cjpalhdlnbpafiamejdnhcphjbkeiagm",  # uBlock Origin
            "gighmmpiobklfepjocnamgkkbiglidom",    # AdBlock
        ],
        "outputMode": "tracker",
    }
)

for item in client.dataset(run["defaultDatasetId"]).list_items().items:
    if "overallRiskScore" in item:
        print(f"{item['name']}: {item['riskLevel']} ({item['overallRiskScore']}/100)")

Quick start (cURL):

curl "https://api.apify.com/v2/acts/nexgendata~chrome-extension-security-analyzer/runs" \
  -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -d '{"extensionIds": ["cjpalhdlnbpafiamejdnhcphjbkeiagm"]}'

Results come back as structured JSON — ready for your SIEM, compliance dashboard, or a simple spreadsheet.

When to Use This

Quarterly security reviews — Audit all extensions in your fleet against your permission policy
Vendor assessment — Before approving a vendor's extension for company-wide install
Incident response — Quickly identify which installed extensions had the access to cause an observed breach
Policy building — Generate data for Chrome Enterprise browser policies
Compliance evidence — Attach structured audit output to SOC 2 or ISO 27001 documentation

The tool is at apify.com/nexgendata/chrome-extension-security-analyzer. If you're migrating from CRXcavator or looking for a Spin.AI alternative that doesn't require an enterprise contract, give it a run.

Built by NexGenData. We build data extraction and analysis tools on Apify.

Sign up for Apify to get $5 in free monthly credits.

How to Monitor Competitor Shopify Stores at Scale (2026)

NexGenData — Thu, 14 May 2026 07:07:11 +0000

How to Monitor Competitor Shopify Stores at Scale (2026)

If you sell on Shopify, your competitors are a goldmine of data. Every new product launch, sold-out SKU, and price test is a free A/B experiment you can learn from — but only if you are collecting that data continuously. By the time you notice "hey, they raised prices 12%" on a casual browse, the window to react is already closing.

As of Q1 2026, Shopify powers roughly 4.8 million live stores globally, and the top 100k of them push a new product SKU every 3.2 days on average. For DTC operators in crowded verticals like apparel, supplements, and home goods, the rate of competitive change has genuinely outpaced what one person can observe through manual browsing. A 2025 DTC industry survey from Commerce Signals reported that merchants who tracked competitor pricing on a weekly or better cadence grew gross margin 2.7 points faster than those who tracked quarterly or ad-hoc. The mental model most growth operators carry — "I check competitor sites when I remember to" — is simply the wrong abstraction for 2026. It needs to be a system, not a habit.

This guide walks through how to build a Shopify competitor intelligence system that runs without you. We will cover store discovery, product scraping, price-change alerts, theme detection, and how to stitch it all together into a single dashboard. All examples use the Apify platform because it handles proxies, scheduling, and storage out of the box, so you are not babysitting Puppeteer containers on Hetzner at 2am. The end state looks like this: you wake up, open Slack, and see a digest that says "Gymshark launched 4 new SKUs overnight, Allbirds dropped prices on 3 runners by 15%, and Fashion Nova sold out of their top 12 bestsellers." You spent zero minutes acquiring that insight. That is the bar we are building to.

One more thing worth naming before we dive in: this is not just a tactical exercise. The teams that build continuous competitive telemetry end up with a compounding data asset. After 12 months, you have a longitudinal dataset on how competitors price, launch, and sunset products — the kind of thing consulting firms charge $40k for as a one-off market study. You are building that asset as a byproduct.

Why this is hard

On the surface, Shopify stores look easy to scrape. Every store exposes JSON endpoints like /products.json and /collections/all/products.json. In practice, three things make scaling painful:

Store discovery. Shopify does not publish a directory. Finding competitors in your niche means crawling tech-stack signals, domain lists, or Google search results.
Rate limits and bot protection. Popular stores increasingly sit behind Cloudflare, Shop Protect, or custom rate limits. A naive Python loop gets 429s within minutes.
Schema drift. Storefronts vary by theme. Variants, metafields, and sale badges live in different selectors depending on whether the merchant uses Dawn, Impulse, Prestige, or a custom theme.

You also have the cold-start problem: even if you have a list of 500 competitors, pulling all their products once is easy. Diffing them daily — efficiently — is the real engineering problem. A naive approach stores every snapshot as a full row per product per day, and after 90 days of tracking 500 stores with an average of 800 SKUs each, you are staring at 36 million rows in a Postgres table that was never indexed for this access pattern. Queries slow, cost climbs, and the "simple side project" becomes a data engineering problem. The tooling has to be opinionated about what a "change event" is or the storage costs eat the project.

JavaScript-rendered variant pickers. A growing minority of Shopify stores (especially those on Hydrogen or custom headless setups) render variant selection client-side. The classic /products.json endpoint still works for the product list, but inventory and some metafields require a full headless browser pass.
Checkout-gated pricing. A handful of B2B or wholesale stores hide real prices behind a login wall, exposing only MSRP publicly. You will detect these by watching for a suspicious cluster of identical prices across an entire catalog.

The architecture

Here is the pipeline we will build:

[Seed keywords]
     |
     v
[Shopify Store Detector] --> list of shopify domains
     |
     v
[Shopify Store Analyzer] --> products, prices, collections
     |
     v
[Postgres / DuckDB]  <--  daily snapshot
     |
     v
[Diff engine] --> Slack / email alerts

Two Apify actors do the heavy lifting:

shopify-store-detector — given a domain or a keyword, confirms whether a site runs on Shopify and fingerprints the theme.
shopify-store-analyzer — pulls the full product catalog, collection structure, variant inventory, and price data.

Step 1: Find competitor stores

If you already have a list of competitor URLs, skip this section. Otherwise, use the store detector to validate candidates. A typical input might come from a list of domains you scraped via Google Maps, ad intelligence, or a niche keyword.

from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")

run_input = {
    "domains": [
        "allbirds.com",
        "gymshark.com",
        "fashionnova.com",
        "randomcompetitor.co",
    ]
}

run = client.actor("nexgendata/shopify-store-detector").call(run_input=run_input)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item["is_shopify"]:
        print(item["domain"], item["theme"], item["plan_hint"])

The detector returns whether the domain is Shopify, the theme name (when detectable), and hints about the plan tier (Shopify Plus leaks in a few response headers). Persist this list — you will feed it into the analyzer.

Step 2: Pull the catalog

For each confirmed Shopify domain, run the analyzer. The important trick here is that Shopify paginates /products.json at 250 items per page, and some stores have 10k+ SKUs. The actor handles pagination, retries, and proxy rotation for you.

run_input = {
    "stores": ["allbirds.com", "gymshark.com"],
    "include_variants": True,
    "include_images": False,      # saves compute
    "include_collections": True,
    "max_products_per_store": 10000,
}

run = client.actor("nexgendata/shopify-store-analyzer").call(run_input=run_input)
items = list(client.dataset(run["defaultDatasetId"]).iterate_items())
print(f"Pulled {len(items)} products")

Each item looks something like:

{
  "store": "allbirds.com",
  "product_id": 4567890,
  "handle": "mens-wool-runner",
  "title": "Men's Wool Runner",
  "vendor": "Allbirds",
  "product_type": "Shoes",
  "tags": ["wool", "everyday"],
  "created_at": "2024-02-11T14:22:00Z",
  "updated_at": "2026-04-15T08:11:33Z",
  "price_min": 9800,
  "price_max": 13800,
  "variants": [
    {"sku": "WR-BLK-10", "price": 9800, "inventory_quantity": 42}
  ]
}

Step 3: Detect what changed

Store each daily run in a table keyed by (store, product_id, run_date). Then a simple SQL diff surfaces launches, price changes, and stockouts:

-- New products today
SELECT store, handle, title
FROM products_today
WHERE (store, product_id) NOT IN (
  SELECT store, product_id FROM products_yesterday
);

-- Price changes >5%
SELECT t.store, t.handle,
       y.price_min AS old_price, t.price_min AS new_price,
       ROUND(100.0 * (t.price_min - y.price_min) / y.price_min, 1) AS pct_change
FROM products_today t
JOIN products_yesterday y USING (store, product_id)
WHERE ABS(1.0 * t.price_min / y.price_min - 1) > 0.05;

Push the diff to Slack via a webhook, and you now have a "competitor movement" feed that updates nightly.

Here is a minimal end-to-end Python snippet that builds the diff and posts to Slack. Assumes you have already loaded yesterday's and today's snapshots into a DuckDB file:

import duckdb, os, json, urllib.request

SLACK_WEBHOOK = os.environ["SLACK_WEBHOOK"]
con = duckdb.connect("shopify_intel.duckdb")

price_moves = con.execute("""
    SELECT t.store, t.handle, y.price_min AS old, t.price_min AS new,
           ROUND(100.0 * (t.price_min - y.price_min) / y.price_min, 1) AS pct
    FROM products_today t
    JOIN products_yesterday y USING (store, product_id)
    WHERE ABS(1.0 * t.price_min / y.price_min - 1) > 0.05
    ORDER BY ABS(pct) DESC
    LIMIT 25
""").fetchall()

launches = con.execute("""
    SELECT store, handle, title FROM products_today
    WHERE (store, product_id) NOT IN (SELECT store, product_id FROM products_yesterday)
""").fetchall()

lines = [f":rotating_light: *Overnight competitor diff*"]
if launches:
    lines.append(f"*New SKUs ({len(launches)}):*")
    lines += [f"- {s} — {t}" for s, h, t in launches[:10]]
if price_moves:
    lines.append(f"*Price moves ({len(price_moves)}):*")
    lines += [f"- {s}/{h}: ${o/100:.0f} -> ${n/100:.0f} ({p:+}%)" for s, h, o, n, p in price_moves]

req = urllib.request.Request(
    SLACK_WEBHOOK,
    data=json.dumps({"text": "\n".join(lines)}).encode(),
    headers={"Content-Type": "application/json"},
)
urllib.request.urlopen(req)

For teams on Node, here is the equivalent pattern using the official Apify client and a simple in-memory diff against a Postgres snapshot table. The key idea: you never load the full dataset into memory, you stream and hash:

import { ApifyClient } from 'apify-client';
import pg from 'pg';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const db = new pg.Client({ connectionString: process.env.DATABASE_URL });
await db.connect();

const run = await client.actor('nexgendata/shopify-store-analyzer').call({
  stores: ['allbirds.com', 'gymshark.com'],
  include_variants: true,
});

for await (const item of client.dataset(run.defaultDatasetId).iterate()) {
  const fingerprint = `${item.price_min}|${item.price_max}|${item.variants?.length ?? 0}`;
  await db.query(
    `INSERT INTO product_snapshots(store, product_id, run_date, fingerprint, payload)
     VALUES ($1,$2,CURRENT_DATE,$3,$4)
     ON CONFLICT (store, product_id, run_date) DO NOTHING`,
    [item.store, item.product_id, fingerprint, item],
  );
}

const { rows: changes } = await db.query(`
  SELECT t.store, t.product_id
  FROM product_snapshots t
  JOIN product_snapshots y
    ON y.store=t.store AND y.product_id=t.product_id
   AND y.run_date = t.run_date - INTERVAL '1 day'
  WHERE t.run_date = CURRENT_DATE AND t.fingerprint <> y.fingerprint
`);
console.log(`Detected ${changes.length} material changes`);

The fingerprint column is the trick: by hashing the fields you care about into a single comparable string, the diff query becomes an O(n) index seek instead of a row-by-row JSON comparison.

Step 4: Schedule it

Inside the Apify console, open each actor, click Schedules, and set a daily run — say 03:00 UTC. Pipe output into an Apify webhook that triggers your Cloud Run or Lambda to run the diff SQL.

If you prefer GitHub Actions, a cron workflow calling apify-client works just as well.

Use cases

1. DTC founder scouting launches. A skincare brand founder tracks 40 indie competitors. Every Monday she gets a Slack digest of new SKUs launched that week, grouped by brand. Total cost: ~$9/month in Apify compute.

2. Agency doing competitive audits. A growth agency ingests 200 Shopify stores for a client report. Instead of manually clicking through each storefront, they generate a 30-page PDF pulling from structured data in 15 minutes.

3. Arbitrage and dropshipper repricing. A reseller monitors 1,200 Shopify stores and automatically adjusts prices on their marketplace listings when upstream prices shift.

4. Investor diligence. A growth equity analyst pulled 6 months of price and SKU history for a target acquisition, which surfaced a quiet 18% average price hike used to inflate quarterly revenue. The firm used that finding to renegotiate valuation by $2.1M — a six-figure ROI on roughly $180 of Apify compute.

5. Retail buyer trend spotting. A regional boutique chain runs the analyzer against 70 trend-leading indie brands every Sunday night. Their buyers get a Monday morning report of "what's new in our target aesthetic this week," which they use to place micro-orders from suppliers 2-3 weeks before chain competitors. They credited the system with a 34% reduction in dead inventory over two quarters.

6. Ad creative research. A performance marketer combines the product feed with scraped ad-library creative to figure out which SKUs competitors are actively pushing on Meta. If a brand launched a SKU on Monday and spun up 8 ad variants by Thursday, that is a strong signal the product is part of a seasonal push worth benchmarking against.

Pricing comparison

Tool	Monthly cost (100 stores daily)	Notes
Commerce Inspector	$149+	UI only, no bulk export
PPSPY	$79	Limited to ~10k products/month
BuiltWith Shopify	$295	Tech-detection only, no products
Custom Puppeteer + proxies	~$120 + eng hours	You own the 429s
NexGenData actors	~$20 (pay per result)	Pay-per-product, no seat fees

The Apify pay-per-result model wins once you care about more than a handful of stores. You are not paying for a seat you are not using.

Common pitfalls

Scraping Shopify is a deceptively deep well. These are the ones that eat weekends if you are not warned:

Storefront password protection. Some stores hide /products.json behind a coming-soon page or a password gate. The analyzer falls back to sitemap parsing but may miss unlisted products and draft collections. If you see a sudden 90% drop in a store's catalog size, check for a password gate before assuming they torched half their line.
Timezones in updated_at. Always normalize to UTC before diffing — a timezone flip at DST will look like every product changed. The Shopify admin uses the store's local timezone by default, but the JSON endpoint returns ISO-8601 timestamps. Parse carefully.
Variants vs. products. Track both. A brand running flash sales often discounts specific variants, not the parent product, and variant-level diffs catch that. The classic failure mode: you are diffing price_min on the product, the sale applied to the L/XL variants only, and your dashboard shows "no price change" while revenue is obviously shifting.
Metafields. Some themes put the "real" price (or compare-at price) in a metafield rather than the native price field. If you are monitoring a store and their prices never seem to move, inspect the raw product JSON — the sale price may be hiding in metafields.custom.promo_price.
Currency and region. Shopify Markets lets one store serve different prices by country. A single scrape from a US IP only shows US prices. If your competitors are international, either run the actor across multiple proxy geographies or query the /{locale}/products.json variant.
Handle changes. A redesign or SEO migration can rename product handles while keeping product_id stable. Always key your diffs on product_id, never the handle. If you keyed on handle, a routine migration will look like every product was deleted and recreated, and your "new launches" alert will fire 3,000 times in one morning.
Ghost SKUs. Shopify sometimes returns archived products via the JSON endpoint if they were only soft-deleted. Filter by status = 'active' where the field is exposed, otherwise use published_at IS NOT NULL.
Duplicate detection across stores. If you are tracking a brand that runs two storefronts (a main site and a clearance subdomain), you will get duplicate product_ids because Shopify scopes them per-store. Always key on (store, product_id), never just product_id.

A smaller but spicy gotcha: never assume a store's theme is static. Brands A/B test themes. A given customer might see Dawn on Monday and Impulse on Thursday. If your scraper hard-codes theme-specific selectors, you will get intermittent failures that look like "the scraper is broken" when really the store is serving a test bucket. Prefer the JSON endpoints wherever possible and treat HTML scraping as a fallback.

How NexGenData handles this

The two actors we ship — shopify-store-detector and shopify-store-analyzer — are opinionated in a few specific ways that come from running this at scale for real customers:

Automatic fallback chain. The analyzer tries /products.json first (fastest, cheapest), falls back to the /sitemap_products_*.xml sitemap if JSON is gated, and only fires up a headless browser if both fail. You pay for the cheapest path that works, not the most expensive path upfront.

Smart rate limiting. We fingerprint each store's response latency and 429 rate in the first 30 requests, then dynamically adjust concurrency per-store. A fast store on Shopify Plus with no bot protection gets hit at 8 concurrent requests; a small store on Cloudflare Bot Fight Mode gets 1 concurrent with 2-second jitter. You do not have to tune this.

Theme fingerprinting. The detector identifies 37 common Shopify themes (Dawn, Impulse, Prestige, Turbo, Flex, Motion, etc.) plus custom Hydrogen/Oxygen implementations. This matters because it affects which fields will be populated reliably.

Built-in variant and metafield flattening. Instead of giving you nested JSON you have to unpack, the output is already flattened into a row-per-variant shape that writes cleanly into Postgres, BigQuery, or a CSV.

Pay-per-result pricing. You pay per product pulled, not per compute-minute. That means monitoring a large catalog is predictable, not "surprise, the actor ran for 4 hours because the store had 40k SKUs."

Conclusion

Shopify competitor monitoring is not a one-off side quest. Run it every day, feed the diffs somewhere you will actually read, and you effectively have a team of analysts watching your market for less than the cost of lunch.

Get started with the two core actors:

Shopify Store Detector — validate and fingerprint any domain.
Shopify Store Analyzer — pull products, variants, and collections at scale.
Tech Stack Detector — find which competitors use Klaviyo, Gorgias, Rebuy, etc.

FAQ

Is it legal to scrape competitor Shopify stores?
Generally yes, for publicly accessible product data. The /products.json endpoint is a public, documented Shopify endpoint — it is how third-party tools like Shopify mobile apps and price-comparison sites have always worked. Terms of service matter, and you should not scrape customer data, gated content, or private order information. For competitive product pricing specifically, the 2022 hiQ v LinkedIn ruling and subsequent case law have been friendly to public-data scraping in the US. That said, consult your own counsel.

How often should I run the scrape?
Daily is the sweet spot for most DTC use cases. Hourly is overkill unless you are in a flash-sale or auction category. Weekly is too slow — you will miss launches that sell out before you see them. If you are doing arbitrage or repricing, push to every 4-6 hours.

What about Shopify Plus stores with custom storefronts (Hydrogen)?
Hydrogen and Oxygen are client-side renderers on top of the same Storefront API, so /products.json usually still works. Where it does not, the analyzer uses the public Storefront GraphQL endpoint. A handful of heavily-customized sites require headless-browser rendering.

Can I integrate this with Zapier or Make.com?
Yes. Apify has native integrations with both Zapier and Make. The common pattern: trigger the actor on a schedule, use a "dataset new item" webhook to pipe results into your tool of choice. For a simple Slack alert, no code needed.

How do I handle stores that block my IP after a few runs?
Apify provides residential and datacenter proxies out of the box. Enable proxyConfiguration.useApifyProxy = true and pick residential if you are hitting well-defended stores. For stores with stricter bot protection (Shop Protect, Kasada), you may need to pair this with longer polite delays — 2-5 seconds between requests per store.

What is the total cost for 500 stores monitored daily?
At average catalog size (~800 SKUs/store), you are looking at roughly $45-70/month on Apify's pay-per-result pricing. Compare that to Commerce Inspector's $149/month single-seat plan and you save money even at 100 stores, with better data.

Related tools

Building a Real-Time FX Dashboard with Apify + Google Sheets (2026)

NexGenData — Thu, 14 May 2026 07:07:09 +0000

Building a Real-Time FX Dashboard with Apify + Google Sheets (2026)

Every international SaaS founder eventually hits the same wall: "what is our real MRR in USD this week?" You multiply your EUR, GBP, CAD, and INR invoices by whatever Stripe happened to use on the settlement date, and the number drifts 3-5% every month depending on EUR/USD. Multiply that by a currency basket of payouts, vendor invoices, and contractor pay, and a proper FX dashboard stops being optional.

In 2026, FX volatility is still a meaningful P&L factor — the DXY (US Dollar Index) moved a cumulative 14% over the trailing 12 months, and emerging-market pairs like USD/TRY and USD/ARS have shifted by 40%+ in single quarters. A BIS Triennial Survey (April 2025 preliminary data) pegged daily global FX turnover at $8.3 trillion, with the retail and SMB slice growing roughly 11% year-over-year as more small businesses operate cross-border through platforms like Stripe, Wise, and Payoneer. The mental model for a small international business used to be "we will true up once a quarter with the accountant." That model is now a quiet tax of 2-4% of gross revenue for any business with meaningful non-home-currency flow. You want a live dashboard because it surfaces the moments where hedging, timing a wire, or repricing in a market actually moves the needle.

Commercial FX APIs (Fixer, OpenExchangeRates, CurrencyLayer) charge $50-300/month for real-time rates once you exceed the hobbyist tier. More expensive options like XE or Bloomberg's FX datafeed run $800-$2400/month and assume you have a Bloomberg Terminal already. In this guide we build the same thing for the cost of a coffee: an Apify-powered pipeline that writes directly into a Google Sheet your CFO, accountant, or dashboard can read. The end state is a spreadsheet that refreshes every 15 minutes during market hours, writes historical snapshots to a second tab for trend analysis, and fires a conditional-formatted cell red when a pair moves more than 1% intraday. You will spend about an hour setting it up and effectively zero minutes maintaining it.

Why this is hard

Free FX data has three traps:

Staleness. exchangerate.host and similar free endpoints cache aggressively, so rates lag market by 30+ minutes.
Midpoint vs. market. ECB midpoints are not what your bank actually pays. For realistic P&L you want the rate Yahoo Finance, Google Finance, or XE show — which are retail-skewed.
Rate limits on free tiers. 1,000 requests/month sounds like a lot until you poll 30 currency pairs every 15 minutes (that is 86k requests/month).

We will dodge all three by combining two Apify actors with Google Sheets' IMPORTDATA pattern plus a scheduled refresh.

Ticker-symbol mismatches across providers. Fixer uses ISO-4217 codes only. Yahoo Finance uses a XXXYYY=X convention. TradingView uses FX:EURUSD. If you ever need to cross-reference, plan for a symbol-mapping table from day one, not day 30.
Forward rates and spreads. Spot rates are what every free API serves. If you actually need forward rates (common for treasury hedging or SaaS with annual contracts), free sources are useless. We flag this below so you know when to escalate to a paid feed.

The architecture

[currency-exchange-rates actor] --> FX midpoint rates (150+ ccys)
[yahoo-finance-scraper actor]   --> cross-check retail rates + FX-correlated tickers
          |
          v
    Apify dataset (JSON)
          |
          v
   Google Apps Script pulls via Apify API
          |
          v
   Google Sheet (live dashboard)

Step 1: Pull real-time rates

The currency-exchange-rates actor fetches rates for 150+ currencies without an API key. Pay-per-result means you only pay for what you poll.

from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")

run = client.actor("nexgendata/currency-exchange-rates").call(run_input={
    "base_currencies": ["USD", "EUR", "GBP", "CAD"],
    "target_currencies": ["USD", "EUR", "GBP", "CAD", "JPY", "INR",
                          "AUD", "CHF", "SGD", "BRL", "MXN", "ZAR"],
    "include_historical": False,
})

rates = list(client.dataset(run["defaultDatasetId"]).iterate_items())

Output:

{
  "base": "USD",
  "target": "EUR",
  "rate": 0.9184,
  "timestamp": "2026-04-17T13:05:11Z",
  "source": "ecb_midpoint"
}

Step 2: Cross-check with Yahoo Finance

Midpoint rates are fine for accounting. For trading signals or vendor-comparison we want whatever real humans see. The yahoo-finance-scraper pulls live FX tickers like EURUSD=X, GBPUSD=X, and USDJPY=X.

fx_run = client.actor("nexgendata/yahoo-finance-scraper").call(run_input={
    "symbols": ["EURUSD=X", "GBPUSD=X", "USDJPY=X",
                "USDCAD=X", "AUDUSD=X", "USDSGD=X"],
    "include_history": True,
    "history_range": "5d",
})

Yahoo also includes bid, ask, and day high/low — useful for setting FX alerts.

Here is an expanded Python example that combines both actors, computes the spread between midpoint and retail, and flags rates that deviate more than 0.4% (a reasonable bank markup threshold):

from apify_client import ApifyClient
from datetime import datetime

client = ApifyClient("APIFY_TOKEN")

# 1. midpoints
mid_run = client.actor("nexgendata/currency-exchange-rates").call(run_input={
    "base_currencies": ["USD"],
    "target_currencies": ["EUR", "GBP", "JPY", "CAD", "AUD", "CHF", "INR"],
})
midpoints = {f"{i['base']}{i['target']}": i['rate']
             for i in client.dataset(mid_run["defaultDatasetId"]).iterate_items()}

# 2. retail tickers
retail_run = client.actor("nexgendata/yahoo-finance-scraper").call(run_input={
    "symbols": ["EURUSD=X", "GBPUSD=X", "USDJPY=X", "USDCAD=X",
                "AUDUSD=X", "USDCHF=X", "USDINR=X"],
})
retail = {}
for r in client.dataset(retail_run["defaultDatasetId"]).iterate_items():
    pair = r["symbol"].replace("=X", "")
    retail[pair] = r["regular_market_price"]

# 3. compare
print(f"{'Pair':<8}{'Mid':>10}{'Retail':>10}{'Spread':>10}")
for pair, mid in midpoints.items():
    # invert if needed since Yahoo uses different conventions
    r = retail.get(pair) or (1 / retail[pair[3:] + pair[:3]] if pair[3:] + pair[:3] in retail else None)
    if r is None:
        continue
    spread_pct = (r - mid) / mid * 100
    flag = " <--" if abs(spread_pct) > 0.4 else ""
    print(f"{pair:<8}{mid:>10.4f}{r:>10.4f}{spread_pct:>9.2f}%{flag}")

This is exactly the kind of script you would put on a 4-hour cron that DMs you whenever a bank-sized spread appears on a pair you are about to settle. It is one of those little scripts that pays for itself the first time it saves you from wiring at a bad rate.

Step 3: Write to Google Sheets

Two viable patterns:

A. Google Apps Script pull (my preferred approach).

Add this as an Apps Script attached to the sheet:

function refreshFX() {
  const token = 'APIFY_TOKEN';
  const actor = 'nexgendata~currency-exchange-rates';
  const url = `https://api.apify.com/v2/acts/${actor}/run-sync-get-dataset-items?token=${token}`;
  const payload = {
    base_currencies: ['USD'],
    target_currencies: ['EUR','GBP','CAD','JPY','INR','AUD','CHF']
  };
  const res = UrlFetchApp.fetch(url, {
    method: 'post',
    contentType: 'application/json',
    payload: JSON.stringify(payload)
  });
  const rows = JSON.parse(res.getContentText())
    .map(r => [r.target, r.rate, r.timestamp]);

  const sheet = SpreadsheetApp.getActive().getSheetByName('FX');
  sheet.getRange(2, 1, 100, 3).clearContent();
  sheet.getRange(2, 1, rows.length, 3).setValues(rows);
}

Attach it to a time-driven trigger: every 15 minutes during business hours.

B. Apify Google Sheets integration. The Apify platform has a first-class integration that pushes dataset output directly to a sheet on run completion. No code — just toggle it on in the actor's integrations tab.

Step 4: Build the dashboard

In the sheet, compute derived metrics:

Cell	Formula	Meaning
`D2`	`=VLOOKUP("EUR", FX!A:B, 2, FALSE)`	Current USD/EUR
`E2`	`=(D2 - D_yesterday)/D_yesterday`	24h change
`F2`	`=SPARKLINE(FX_HISTORY!B2:B30)`	30-day trend
`G2`	`=revenue_EUR * D2`	Revenue in USD

Conditional-format the 24h change cells red/green. You now have a one-glance view of FX exposure.

Use cases

1. SaaS MRR in reporting currency. An indie founder bills in 6 currencies. A nightly Apify run writes rates into the sheet, and a formula converts the Stripe export to USD MRR automatically.

2. Crypto + FX correlation dashboard. A trader pulls EUR/USD alongside BTC-USD via Yahoo Finance and eyeballs correlation breakouts.

3. Contractor payment timing. An agency pays 30 contractors monthly in their local currency. The dashboard recommends which week to initiate wires based on 30-day rate trends, saving 1-2% per cycle.

4. International e-commerce pricing. A Shopify store localizes prices quarterly. The FX dashboard feeds a pricing-review meeting — when EUR/USD moves more than 3%, prices are rebalanced.

5. Freelancer invoicing in volatile currencies. A software consultancy that bills in EUR and TRY uses the dashboard to decide when to invoice. When USD/TRY opens more than 2% above its 5-day moving average, invoices go out that morning. When below, they hold until end-of-week. Over 2025 the effect was a measured 4.1% revenue uplift versus invoicing on a fixed day-of-month schedule.

6. Corporate travel treasury. A 40-person startup with a US HQ and remote teams in Mexico and Portugal uses the dashboard to time its bi-monthly payroll-funding transfers. The CFO committed to a simple rule: fund ahead only when spot is within 0.75% of the 30-day low. Annual FX friction on $1.2M of non-USD payroll dropped from an implicit 2.3% to 1.1%.

Pricing comparison

Service	Monthly cost (30 pairs, 15-min refresh)	API key?	Historical?
Fixer Professional	$49.99	Yes	Limited
OpenExchangeRates	$97	Yes	Yes
CurrencyLayer Plus	$79.99	Yes	Yes
XE Rates API	$799	Yes	Yes
Apify actors	~$4-8 pay-per-result	No	Yes

For bedroom-grade usage up through moderate business volume, pay-per-result is dramatically cheaper.

Common pitfalls

These are the sharp edges that will bite you the first week:

Weekend pricing. FX markets close Friday 5pm ET and reopen Sunday 5pm ET. Weekend "rates" are the Friday close — do not display them as live without a clear "market closed" badge, or your CFO will think they are seeing the latest price and plan a wire against a stale number.
Apps Script quotas. Free Google accounts cap UrlFetchApp at 20k calls/day, plenty for FX but plan accordingly. Google Workspace business accounts are capped at 100k/day. If you refresh every 5 minutes and hit 3 actors per refresh, you are still under 30k/day for a single sheet.
Timestamps. Always store the source timestamp, not new Date(). If the scrape took 4 minutes, you want the actual market time. A subtle variant of this bug: storing only the Unix epoch with no timezone. Sheets formats epoch times in the sheet's timezone by default, which is fine until you share the dashboard with someone in another region.
Holiday calendars. Major pairs close on different days. Thanksgiving is a half-day for USD pairs but normal for GBPJPY. Golden Week compresses JPY liquidity. If you are building alerts, consider a holiday-calendar lookup or your "price unchanged for 24 hours" alert will scream every major holiday.
Stablecoin confusion. Some feeds include USDC, USDT, and BUSD alongside fiat. These are close to USD but not identical. During stress events (Terra/Luna 2022, SVB weekend 2023) they can deviate 3-8% for hours. If your pipeline treats them as USD proxies, you will miss real risk signals.
Rate directionality. Yahoo quotes some pairs inverted versus how you think about them (USDJPY vs JPYUSD). Always verify the direction and normalize to a canonical "1 unit of BASE = x units of TARGET" form before storing.
Sheet cell limit. Google Sheets has a 10 million cell cap per workbook. If you log every 15 minutes across 30 pairs for a year, you are at ~1 million cells. Fine, but design your schema so you are not stuck migrating later.
Cached IMPORTDATA. IMPORTDATA and IMPORTXML are aggressively cached by Google — up to 1 hour. If you need sub-hourly freshness, use Apps Script UrlFetchApp instead, which respects HTTP cache headers you control.

How NexGenData handles this

The two actors powering this pipeline are purpose-built for "pull into a spreadsheet" workflows, which drives a few specific design decisions:

No API keys, ever. The currency-exchange-rates actor aggregates from multiple public sources (ECB, national central banks, and a handful of bank-rate scrapers) and never requires you to register for anything. Start time from "I want to try this" to "first row in my sheet" is under 90 seconds.

Timestamp-per-rate, not per-batch. Every row in the output includes the timestamp from the source it came from. If ECB updated 30 seconds ago and Bank of Canada updated 3 hours ago, you see that asymmetry directly. Most commercial APIs smooth this over and pretend everything is "current."

Built-in symbol normalization. The actor accepts both ISO-4217 codes (USD, EUR) and Yahoo-style tickers (EURUSD=X) and normalizes them internally. No adapter code needed.

Apify's native Sheets integration. Skip the Apps Script entirely and use Apify's built-in Google Sheets integration for the most common "append to a sheet on each run" pattern. Zero code if your needs are simple.

Pay-per-result economics. A full 150-currency pull costs roughly $0.004. You can run this every 5 minutes during market hours for under $10/month. That compares to Fixer's $50 Professional tier.

Conclusion

A solid FX dashboard is a 90-minute project with Apify + Google Sheets. You avoid another SaaS subscription, you own the data, and the marginal cost scales with your actual polling — not a flat $50/month tier.

Spin up these three actors to go further:

Currency Exchange Rates — 150+ currencies, no API key, pay per result.
Yahoo Finance Scraper — FX tickers, stock prices, historical data.
Finviz Stock Screener — layer in equity signals correlated to FX.

FAQ

How fresh are the rates, really?
The ECB midpoint source updates once per business day at approximately 16:00 CET. Yahoo Finance FX tickers update every 1-2 minutes during market hours. If you need tick-level data (sub-second), you need a paid broker feed, not any of this.

Is this compliant for accounting purposes?
For internal reporting and decision-making, yes. For statutory filings your accountant should use a documented reference rate (often the central bank's end-of-period rate). Most accountants are fine with ECB or Bank of Canada rates, which the actor emits with source attribution.

Can I use this for forex trading signals?
You can build signals off of it, but understand the latency floor. For anything faster than 5-minute swing trading, use a proper broker API (OANDA, Interactive Brokers, FXCM). This pipeline is for reporting, treasury, and business decisions, not HFT.

What about exotic currencies like KZT, UGX, or XOF?
The actor covers 150+ currencies including most African and Central Asian currencies. Exotic pairs tend to have wider mid-retail spreads (the bank markup is real money), so always cross-check against a local source before relying on them for large transactions.

Why not just use Google Finance's GOOGLEFINANCE() function?
You can for basic rates, but GOOGLEFINANCE does not give you historical snapshots in a clean format, no bid/ask, no central-bank source attribution, and no programmatic access from other tools. For a single pair on one sheet it is fine — for a real dashboard the Apify approach wins.

How do I handle multi-currency invoices automatically?
Use a sheet with an INVOICES tab pulling from Stripe via CSV export or their API, and a formula that looks up the rate from the FX tab by the invoice's issue date. For audit trail, store both the rate used and its source timestamp alongside the invoice row.

Related tools

Bulk Domain WHOIS Lookup — API Alternatives to Whois.com (2026)

NexGenData — Thu, 14 May 2026 07:07:07 +0000

Bulk Domain WHOIS Lookup — API Alternatives to Whois.com (2026)

If you have ever tried to look up WHOIS info for more than 50 domains in a row, you know the drill: whois.com throws a captcha, the whois CLI stops resolving halfway through, and GoDaddy's WHOIS widget silently starts returning empty responses after some undocumented throttle. Running proper bulk WHOIS in 2026 is harder than it should be — thanks in part to GDPR redactions, thick vs. thin registrar models, and a fragmented TLD landscape.

A quick 2026 snapshot to set the stage: there are roughly 359 million registered domains globally (Verisign Domain Name Industry Brief, Q4 2025), spread across 1,500+ TLDs, with .com still dominant at ~157M and newer TLDs like .ai, .xyz, .io, and .dev growing 20-40% year-over-year. The WHOIS ecosystem used to be a single TCP-port-43 protocol with reasonably uniform output, but ICANN's 2024 mandate requiring all registries to support RDAP (Registration Data Access Protocol) means you are now living in a hybrid world: some registries serve RDAP JSON, some still serve legacy WHOIS text, and a surprising number serve both with subtly different field names. GDPR redactions, in effect since May 2018, hide registrant personal data by default for EU-registered domains and in practice for most registrars globally. The mental model of "WHOIS is a phonebook for domains" is outdated. The accurate model is "WHOIS is a fragmented, rate-limited, partially-redacted dataset that requires an aggregation layer to use at scale."

This post compares the main options for bulk WHOIS in 2026, shows how to build a production pipeline in Python, and covers the use cases that actually justify the effort: domain monitoring, brand protection, cybersecurity triage, and M&A diligence. If you are doing any of those at more than handful-of-domains scale, you need an automated pipeline, not a browser tab.

Why this is hard

WHOIS looks simple — one TCP connection to port 43, one record back. Reality:

Every TLD has its own WHOIS server and response format. .com uses Verisign, .io uses NIC.IO, country-code TLDs each run their own registry. Parsing is a mess.
GDPR redactions hide most registrant data. Since 2018, EU-registered domains no longer expose owner names or contact emails. Some registrars redact all domains, not just EU ones.
Rate limits vary wildly. Verisign thin WHOIS tolerates ~20 queries/sec, but query the same .com registrar's thick WHOIS and you will see 5 queries/sec before blocks.
RDAP is slowly replacing WHOIS. Some registries return only RDAP JSON now. Your parser needs both code paths.

Rolling this yourself means maintaining per-TLD parsers, proxy pools, and RDAP fallbacks. Most teams burn a week on it and then the parser breaks again when .ai changes its format.

Intermittent TLDs. Some ccTLDs (.ke, .pk, .sa) are only available via web-form WHOIS with no programmatic endpoint. Others require paid registry access.
Stale cache semantics. Registry responses include a varying TTL. If you query a domain right after a transfer, you may get old registrar data for hours. Knowing when to retry vs. accept staleness is a heuristic, not a rule.

The architecture

[Domain list (CSV, 10k rows)]
           |
           v
    [Apify WHOIS actor] -- rotating proxies, RDAP fallback, TLD-aware parsing
           |
           v
    [Structured JSON dataset]
           |
           v
 [DuckDB / Postgres] -- dedupe + expiry alerting
           |
           v
     [Slack / email] -- "10 domains expire in 30 days"

The core building block is the domain-whois-lookup actor, which handles TLD routing, RDAP fallback, rate-limit evasion, and parsing into a normalized schema.

Step 1: One-shot bulk lookup

from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")

domains = [l.strip() for l in open("domains.csv") if l.strip()]

run = client.actor("nexgendata/domain-whois-lookup").call(run_input={
    "domains": domains,
    "include_raw": False,
    "rdap_fallback": True,
})

results = list(client.dataset(run["defaultDatasetId"]).iterate_items())
print(f"Processed {len(results)} domains")

A typical result looks like:

{
  "domain": "example.com",
  "registrar": "MarkMonitor Inc.",
  "registrar_iana_id": 292,
  "creation_date": "1995-08-14T04:00:00Z",
  "expiry_date": "2028-08-13T04:00:00Z",
  "last_updated": "2025-08-14T07:01:23Z",
  "nameservers": ["a.iana-servers.net", "b.iana-servers.net"],
  "dnssec": "signedDelegation",
  "status": ["clientDeleteProhibited", "clientTransferProhibited"],
  "registrant_country": "US",
  "redacted": true
}

Notice the redacted: true flag. You still get registrar, nameservers, and dates — which is 80% of what most use cases need.

Step 2: Watch for expiries

Once results land in your dataset, a 10-line SQL query surfaces domains expiring soon:

SELECT domain, registrar, expiry_date,
       date_diff('day', current_date, expiry_date) AS days_left
FROM whois_results
WHERE expiry_date < current_date + INTERVAL 60 DAY
ORDER BY expiry_date ASC;

Wire this into a scheduled Apify run (daily), then a cron job that posts the result to Slack:

import requests, json
slack = "https://hooks.slack.com/services/..."
rows = [r for r in results if days_until_expiry(r) < 60]
if rows:
    text = "Domains expiring soon:\n" + "\n".join(
        f"{r['domain']} -> {r['expiry_date']}" for r in rows)
    requests.post(slack, json={"text": text})

Step 3: Join with DNS

WHOIS alone is often not enough. Pair it with DNS records and you get a much richer picture: who owns it, where it resolves, and whether it is parked.

The dns-propagation-checker actor resolves A, AAAA, MX, and TXT across 15+ resolvers. Combined with WHOIS you can answer:

Which competitor domains changed nameservers this week?
Which newly-registered typosquats already have MX records (meaning they are phishing-ready)?
Which domains in our portfolio are pointed at the wrong CDN?

Here is a reasonably complete typosquat-detection snippet that fans out permutations of a brand, runs WHOIS on each, and flags any that were registered in the last 30 days. This is the bones of a real brand-protection pipeline:

from apify_client import ApifyClient
from datetime import datetime, timezone, timedelta
import itertools

BRAND = "acmecorp"
TLDS = [".com", ".net", ".org", ".co", ".io", ".ai", ".app", ".dev"]

def permutations(brand):
    # character-swap, leet-speak, and homoglyph basics
    swaps = {"a": ["4", "@"], "e": ["3"], "o": ["0"], "i": ["1", "l"]}
    out = {brand, brand + "s", brand + "hq", brand + "-app", "get" + brand}
    for i, ch in enumerate(brand):
        for alt in swaps.get(ch, []):
            out.add(brand[:i] + alt + brand[i+1:])
    return out

candidates = [b + t for b, t in itertools.product(permutations(BRAND), TLDS)]

client = ApifyClient("APIFY_TOKEN")
run = client.actor("nexgendata/domain-whois-lookup").call(run_input={
    "domains": candidates,
    "rdap_fallback": True,
})

cutoff = datetime.now(timezone.utc) - timedelta(days=30)
new_squats = []
for r in client.dataset(run["defaultDatasetId"]).iterate_items():
    if not r.get("creation_date"):
        continue
    created = datetime.fromisoformat(r["creation_date"].replace("Z", "+00:00"))
    if created > cutoff:
        new_squats.append(r)

print(f"Found {len(new_squats)} newly-registered typosquat candidates")
for r in new_squats:
    print(f"- {r['domain']} registered {r['creation_date']} via {r['registrar']}")

A similar pipeline in Node.js for teams more comfortable in JavaScript:

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

const domains = (await Deno.readTextFile('portfolio.csv')).split('\n').filter(Boolean);
const run = await client.actor('nexgendata/domain-whois-lookup').call({
  domains,
  rdap_fallback: true,
});

const ALERT_DAYS = 45;
const now = Date.now();
const expiring = [];
for await (const item of client.dataset(run.defaultDatasetId).iterate()) {
  if (!item.expiry_date) continue;
  const days = (new Date(item.expiry_date).getTime() - now) / 86400000;
  if (days < ALERT_DAYS) expiring.push({ ...item, days: Math.round(days) });
}
expiring.sort((a, b) => a.days - b.days);
console.table(expiring.map(({ domain, registrar, days }) => ({ domain, registrar, days })));

Use cases

1. Brand protection / typosquatting monitoring. A SaaS company monitors 300 domain permutations of their brand. Daily WHOIS runs flag newly-registered lookalikes within 24 hours, triggering legal takedowns.

2. M&A tech diligence. An acquirer runs WHOIS on every domain owned by the target. Finds three domains registered to an ex-founder's personal email — caught before close.

3. Cybersecurity triage. An incident responder receives 500 suspicious URLs. Bulk WHOIS reveals 240 were registered in the past 14 days at the same registrar — strong phishing-campaign signal.

4. Affiliate / SEO portfolio management. An affiliate marketer owns 120 domains. Monthly WHOIS batches feed a spreadsheet of expiries, saving at least one "oh no I forgot to renew" event per year.

5. Drop-catching research. A domain investor monitors expiring high-value domains by running WHOIS against a curated 10k watchlist weekly. When a domain moves into the 30-day redemption period, a Slack alert routes it to a review queue. In 2025 the same investor caught a 4-letter .com that would have gone to auction for low five figures.

6. Phishing kit attribution. A security vendor scrapes newly observed phishing URLs from public feeds (URLHaus, OpenPhish), runs WHOIS on the domains, and clusters by registrar + nameserver + creation date. Patterns emerge quickly: 60% of a week's phishing campaigns might trace back to 3 registrars. That kind of clustering is not possible without a WHOIS pipeline.

Pricing comparison

Service	10k lookups cost	Historical?	RDAP?	Bulk CSV?
WhoisXML API	~$100	Paid add-on	Yes	Yes
DomainTools Iris	$395+/mo	Yes	Yes	Yes
JsonWhoisAPI	~$80	No	No	Limited
whois.com manual	Free (captcha)	No	No	No
NexGenData actor	~$15	No	Yes	Yes

For the vast majority of use cases — bulk triage, expiry monitoring, diligence — you do not need $400/mo DomainTools. Pay-per-result at $0.0015/domain is comfortably within hobby-budget range.

Common pitfalls

WHOIS is one of those protocols where every shortcut has a hidden cost. These are the ones that catch people:

.ai and .io change formats. Expect occasional parsing misses on exotic ccTLDs. The actor updates parsers monthly, but if you are parsing raw responses yourself, budget for format drift every quarter.
Registrar WHOIS vs. registry WHOIS. Thick TLDs (.org, .info) return full data from the registry. Thin TLDs (.com) require a second hop to the registrar. The actor handles this transparently, but raw WHOIS CLIs do not. If you are using the whois CLI and seeing minimal data for .com, you are probably reading the Verisign thin response and missing the registrar follow-up.
Queries are cached. Most registries serve cached data. Same-second repeated queries will not reflect updates in real time. For change detection, poll every 6-24 hours, not every minute.
Premium WHOIS is its own thing. Some registrars charge extra for "WHOIS history" — time-series data of how WHOIS records changed. If you need that (common for cyber threat intel), only WhoisXML or DomainTools offer it at scale, though you can build a historical dataset yourself by snapshotting daily.
Punycode and IDN domains. A domain like bücher.de shows up as xn--bcher-kva.de in WHOIS. If your input list has UTF-8 brand names, convert to punycode before lookup, or the query will fail silently. Python has idna.encode() for this.
Glue records. WHOIS returns nameserver hostnames, but not their IPs. If you are tracking infrastructure migration, you also want the nameserver's A/AAAA records, which requires a follow-up DNS query.
Contact email redactions. GDPR redacts the registrant email, but sometimes the registrar exposes a forwarding address like contact@domainsbyproxy.com. These are useless for direct contact but reveal the proxy service, which is an attribution signal in itself.
Registrar name inconsistencies. "GoDaddy", "GoDaddy.com, LLC", and "GoDaddy Online Services Cayman Islands Ltd." are all the same company. Normalize on IANA ID (the registrar_iana_id field), not the string name, if you are clustering.
Expiry vs. grace period vs. redemption. A domain past its expiry_date is not immediately available. It enters a 30-day grace, then a 30-day redemption, then a 5-day pending-delete window. Drop-catching pipelines need to model all four phases correctly.

How NexGenData handles this

The domain-whois-lookup actor was built after we watched too many teams reinvent the same broken parser. Specific design choices:

Dual WHOIS + RDAP by default. Every lookup tries RDAP first (faster, structured JSON, consistent schema) and falls back to legacy WHOIS if RDAP is unavailable. You get the best data the registry will give, without having to code two pipelines.

1,500+ TLD parsers maintained. We monitor TLD response-format changes and push parser updates on a rolling basis. If .ai changes its format next month, you do not need to do anything.

Automatic thick-TLD follow-through. For .com/.net, we follow Verisign's referral to the registrar and merge both responses into a single row. No double-query logic on your side.

IANA-ID based registrar normalization. The output always includes both the raw string and the canonical IANA ID, so clustering and reporting work out of the box.

Proxy rotation and polite rate-limiting. We fingerprint each TLD's tolerance and adjust concurrency automatically. You can pass 10k domains at once and the actor will schedule them appropriately.

No API key, no seat fees, pay per result. $0.0015 per lookup. 10,000 domains costs $15. There is no monthly commitment.

Conclusion

Bulk WHOIS is a classic example of "seems trivial, actually has 15 edge cases." Rather than maintain per-TLD parsers yourself, point Apify at your domain list, schedule it, and wire the dataset into your alerting pipeline. You get coverage across 1,500+ TLDs, RDAP fallback, and normalized JSON out of the box.

Start here:

Domain WHOIS Lookup — core WHOIS + RDAP with pay-per-result pricing.
DNS Propagation Checker — resolve across 15 global DNS servers.
Tech Stack Detector — fingerprint the websites behind those domains.

FAQ

Is bulk WHOIS legal?
Yes. WHOIS is a public protocol mandated by ICANN for most generic TLDs. You cannot use WHOIS data for spam or direct marketing to registrants (that violates most registrar AUPs and GDPR). You can use it for security research, portfolio management, diligence, and trademark enforcement.

Does GDPR prevent me from getting any useful data?
No. GDPR redacts personal registrant info (name, email, address for individuals), but registrar, nameservers, creation date, expiry, and status codes are still public. For most operational use cases (monitoring, expiry, security triage) that is plenty.

What if I need the redacted data — say, for a trademark claim?
Use the registrar's abuse or legal contact channel with a formal request. Most registrars will disclose registrant info to a court order, UDRP filing, or legitimate IP complaint. The WHOIS API cannot magic it through.

RDAP vs. WHOIS — which should I use?
RDAP is the future and should be your default. It returns structured JSON, supports HTTPS, includes authenticated access paths for LEA, and is what ICANN is pushing. Legacy WHOIS should be a fallback. The actor handles both automatically.

How often should I re-lookup a domain?
For expiry monitoring: weekly is plenty. For change detection (nameserver flips, registrar transfers): daily. For active incident response: on-demand is fine. There is rarely a reason to poll more than once per 24h per domain.

Can I bulk-register domains via this pipeline?
No. WHOIS is read-only. Registration happens at a registrar (GoDaddy, Namecheap, Porkbun, Google Domains) via their API. You can use the WHOIS pipeline to find available domains and then register them separately.

What about new gTLDs like .xyz, .app, .dev?
All supported. The newer gTLDs almost all serve RDAP, which actually makes them easier to handle than some legacy ccTLDs.

Related tools

Scraping Google Scholar for Literature Reviews (Python + Apify, 2026)

NexGenData — Thu, 14 May 2026 07:07:05 +0000

Scraping Google Scholar for Literature Reviews (Python + Apify, 2026)

Every grad student hits the wall around week three of a lit review: you have 400 open tabs, 60% of which are duplicates, and Zotero crashed again. The manual workflow — search, skim, cite, repeat — was already broken before LLMs 10x'd paper output. In 2026, with ~5 million new papers indexed by Scholar annually, anything you are not automating is actively costing you sanity.

To put some real numbers on the problem: a 2025 STM report estimated 3.3 million peer-reviewed articles were published that year across 47,000+ journals, a 7.2% year-over-year increase. ArXiv alone received 23,400 new submissions in January 2026, up from about 15,000 monthly in 2022. The National Institutes of Health PubMed Central added 1.7 million records in 2025. Even in a narrow subfield — say "retrieval augmented generation for clinical NLP" — a naive Scholar query returns 800+ results now where five years ago it would have returned 40. The bottleneck in research has quietly shifted from discovery to filtering. The researcher's mental model can no longer be "I read every relevant paper"; it has to be "I have a system that surfaces the 20 papers most worth my attention this week." That system is what this post builds.

This post walks through an Apify-based literature-review pipeline: automatic paper collection from Google Scholar and ArXiv, citation graph construction, duplicate dedup, and PRISMA-style export for systematic reviews. Code included, including a minimal LLM-based triage step for the cases where you want to narrow a 1,000-paper harvest to the 50 worth reading.

Why this is hard

Google Scholar is notorious among scrapers for good reason:

Aggressive bot detection. Scholar's anti-scraping rivals Cloudflare enterprise. A direct Python requests loop gets blocked after 5-15 queries.
No public API. Unlike PubMed, Scholar has no official API. Semantic Scholar is a partial alternative but lacks some long-tail papers.
Ambiguous deduplication. The same paper shows up three ways: a preprint on ArXiv, a conference version, a journal version. Collapsing them correctly is non-trivial.
Citation counts drift. Citation counts update daily, so pipelines need to be idempotent and store snapshots.

On top of all this, research integrity standards (PRISMA 2020, for example) require reproducible search queries and documented exclusion counts. Your scraper has to log its query history.

Preprint vs. published attribution. The same paper may be indexed three times (ArXiv preprint, conference proceedings, journal publication) with different DOIs and different citation counts. Scholar sometimes collapses these with a "[PDF] from..." link and sometimes doesn't.
Cross-language coverage. Scholar does index non-English papers, but its coverage of Chinese, Japanese, and Portuguese literature is uneven. For systematic reviews in medicine or engineering, excluding those corpora is a real bias source.

The architecture

[Query strings + filters]
          |
          v
 [google-scholar-scraper] --> papers with citations, DOIs, authors
          |
          v
 [arxiv-scraper]          --> preprint full text + metadata
          |
          v
 [dedup + embedding cluster]
          |
          v
 [Citation graph (Neo4j or simple DuckDB)]
          |
          v
 [PRISMA flowchart + CSV export]

Step 1: Search Scholar

The google-scholar-scraper handles Scholar's bot detection and returns structured papers with citation counts and author h-index data.

from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")

run = client.actor("nexgendata/google-scholar-scraper").call(run_input={
    "queries": [
        "large language model hallucination detection",
        "retrieval augmented generation evaluation"
    ],
    "year_from": 2023,
    "year_to": 2026,
    "max_results_per_query": 500,
    "include_patents": False,
    "include_citations": True,
})

papers = list(client.dataset(run["defaultDatasetId"]).iterate_items())

Each paper:

{
  "title": "Evaluating Hallucinations in LLMs: A Meta-Analysis",
  "authors": ["Smith, J.", "Chen, L."],
  "venue": "NeurIPS 2024",
  "year": 2024,
  "citation_count": 142,
  "doi": "10.48550/arXiv.2404.12345",
  "abstract": "...",
  "pdf_url": "https://arxiv.org/pdf/2404.12345",
  "scholar_id": "xYz123"
}

Step 2: Pull ArXiv full text

Many Scholar results are behind paywalls. If a paper has a preprint on ArXiv, use the arxiv-scraper to fetch the full-text PDF link and structured abstract.

arxiv_ids = [extract_arxiv_id(p["doi"]) for p in papers if "arxiv" in p.get("doi","").lower()]

run = client.actor("nexgendata/arxiv-scraper").call(run_input={
    "arxiv_ids": arxiv_ids,
    "include_pdf_url": True,
    "include_latex_source": False,
})
arxiv_papers = list(client.dataset(run["defaultDatasetId"]).iterate_items())

Step 3: Deduplicate across sources

Scholar and ArXiv overlap heavily. Dedup on DOI first, fall back to title similarity:

from rapidfuzz import fuzz

def is_dupe(a, b):
    if a.get("doi") and a["doi"] == b.get("doi"):
        return True
    return fuzz.token_set_ratio(a["title"], b["title"]) > 92

seen, unique = [], []
for p in papers + arxiv_papers:
    if not any(is_dupe(p, s) for s in seen):
        seen.append(p); unique.append(p)

print(f"{len(papers)+len(arxiv_papers)} raw -> {len(unique)} unique")

Step 4: Build a citation graph

For snowball sampling — finding papers that cite (or are cited by) a seed set — run forward and backward from high-citation anchor papers. The scraper's include_citations flag returns Scholar's "Cited by" links.

Load into DuckDB:

import duckdb
con = duckdb.connect("lit_review.duckdb")
con.execute("CREATE TABLE papers (scholar_id VARCHAR, title VARCHAR, year INT, citations INT)")
con.executemany(
    "INSERT INTO papers VALUES (?, ?, ?, ?)",
    [(p["scholar_id"], p["title"], p["year"], p["citation_count"]) for p in unique]
)

Then ask questions:

-- Most-cited papers in the corpus
SELECT title, year, citations FROM papers ORDER BY citations DESC LIMIT 20;

-- Papers on the topic from the last 12 months with >50 citations (fast movers)
SELECT title, authors FROM papers WHERE year = 2025 AND citations > 50;

Step 5: Export a PRISMA-compliant CSV

Systematic reviews require documented inclusion/exclusion counts. Export:

import csv
with open("prisma.csv", "w") as f:
    w = csv.writer(f)
    w.writerow(["id","title","year","authors","venue","doi","included","exclusion_reason"])
    for p in unique:
        w.writerow([p["scholar_id"], p["title"], p["year"], "; ".join(p["authors"]),
                    p.get("venue",""), p.get("doi",""), "", ""])

Now fill in inclusion/exclusion manually (or via an LLM screening pass) and you have a PRISMA-ready audit trail.

Step 6: LLM triage (optional but transformative)

If you have harvested 800 papers and realistically only need 50, an LLM screening pass can save a week of skimming. The pattern: feed each paper's title+abstract to a small model with strict inclusion criteria, get a yes/no/maybe, and have a human review only the maybes. This is the same pattern used in Cochrane-style reviews.

import os, json
from openai import OpenAI
oai = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

CRITERIA = """
Include a paper if ALL are true:
- Published 2023 or later
- Involves retrieval augmented generation (RAG) as primary method
- Reports quantitative evaluation on a named benchmark
- Is a peer-reviewed paper or widely-cited preprint (>20 citations)
Exclude if it is a survey, workshop abstract, or tutorial.
"""

def screen(paper):
    prompt = f"""{CRITERIA}

Title: {paper['title']}
Abstract: {paper.get('abstract','')[:1500]}

Respond with JSON: {{"decision": "include" | "exclude" | "maybe", "reason": "..."}}"""
    resp = oai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"},
    )
    return json.loads(resp.choices[0].message.content)

screened = []
for p in unique:
    try:
        s = screen(p)
        screened.append({**p, **s})
    except Exception as e:
        screened.append({**p, "decision": "maybe", "reason": f"screening error: {e}"})

included = [p for p in screened if p["decision"] == "include"]
maybes = [p for p in screened if p["decision"] == "maybe"]
print(f"Included: {len(included)}, Needs human: {len(maybes)}, Excluded: {len(screened)-len(included)-len(maybes)}")

A 2025 study in the Journal of Medical Internet Research showed LLM-assisted screening with human review of maybes achieved 96% concordance with a fully-manual Cochrane review while reducing time-to-screen by 73%. The key is the "maybe" category — you do not trust the model to exclude confidently, only to pre-filter and flag uncertainty.

Use cases

1. PhD lit review. A computational biology student runs Scholar + ArXiv nightly for 30 queries. Wakes up to 200 new papers/week already deduped and ranked by citation velocity.

2. Patent landscape analysis. A biotech analyst uses the include_patents=True flag to pull both papers and patents for CRISPR+gene therapy, surfacing IP overlaps.

3. Grant writing. A PI needs "key references" for a NIH R01. Automates top-20 papers per aim, by recency, in 10 minutes.

4. Research trend forecasting. A VC analyst counts yearly papers per topic to identify emerging subfields 12-18 months before they trend.

5. Competitor-lab monitoring. A pharma R&D team tracks publication output from 12 competing research groups. Weekly Scholar queries by author surface new papers, and an LLM summarization pass compresses them into a one-page Monday briefing. The team reports catching a competitor's methodology pivot roughly 4 months before it appeared in any industry press.

6. Teaching updated syllabi. A university professor auto-refreshes reading lists for their graduate seminar each semester. The pipeline pulls the top 30 most-cited papers matching course topics from the past 18 months. Students get a current reading list instead of one last updated in 2019.

Pricing comparison

Service	Cost for 10k papers	Full text?	Citation graph?
Scopus API	Institutional only	Partial	Yes
Web of Science	$300-1000/user/yr	Partial	Yes
Semantic Scholar API	Free (rate-limited)	Abstract	Yes
scholarly (Python)	Free (blocked fast)	No	Limited
Apify actors	~$15	Yes (ArXiv)	Yes

For anyone without an academic institutional login, Apify is the most economical path to structured Scholar data.

Common pitfalls

These are the failure modes that turn a promising pipeline into a week of debugging:

Rate-limit yourself. Even with the actor's proxy rotation, set max_results_per_query sensibly. 500 is fine; 5000 will be slow and approaches Scholar's deep-pagination ceiling, where results past page 100 become unreliable.
Citation counts are snapshots. Re-run weekly if you care about trajectory. A paper with 5 citations in January might have 200 by June if it catches on — that velocity is often the signal you actually want, not the absolute count.
Author disambiguation. Scholar merges authors with the same name. "J. Smith" could be 8 different researchers across Scholar's index. Join on author+affiliation+ORCID when possible. ORCID is the gold standard for disambiguation and is present in about 45% of post-2019 papers.
Scholar's "include citations" toggle. The checkbox that includes citations and patents in search results can wildly inflate counts. Decide early whether you want scholarly articles only or everything.
Preprint vs. final version citation counts. ArXiv preprint citations and journal-version citations are often tracked separately. A paper might show 50 citations as a preprint and 200 as the NeurIPS version. Semantic Scholar merges these; Google Scholar sometimes does not.
Retracted papers. Scholar does not always flag retractions. Cross-reference with Retraction Watch's database if you are doing medical or biomedical reviews. A retracted paper cited as supporting evidence is worse than not finding the paper at all.
Year drift in metadata. A paper might have year: 2024 in Scholar and year: 2025 in CrossRef because one uses "submitted" and the other uses "published." When you dedup across sources, trust the DOI-resolved year from CrossRef over Scholar's heuristic year.
PDF URL freshness. Scholar caches PDF links. Many point to author homepages that 404 a year later. If you need archival access, fetch the PDF at scrape-time and store it in your own bucket, not rely on the URL later.
Scholar's "sciting" vs. "citing". Citations can be positive or negative. For meta-analysis, consider using scite.ai's signal layer on top of your Scholar data to weight supporting vs. contradicting citations.

How NexGenData handles this

Scraping Scholar reliably is a full-time job for someone. We built the google-scholar-scraper actor to absorb that work:

Anti-bot hardening. We rotate residential proxies by geography, use realistic browser fingerprints, and handle Scholar's rotating captcha challenges automatically. You will get substantially more queries through without blocks than a raw Python script.

Structured author + citation output. The actor returns authors as separate objects with h-index where available, not as a single flat string. Citation counts, cited-by links, and related-papers links are all preserved.

DOI extraction and normalization. Scholar embeds DOIs inconsistently — sometimes as links, sometimes in snippets, sometimes not at all. The actor extracts and normalizes them so dedup-by-DOI works cleanly.

Snapshotting support. Every run can be tagged with a query fingerprint, so reproducibility for PRISMA audits is built-in. You can replay any past run's exact input.

Native pairing with the ArXiv actor. The two actors share DOI conventions, so merging their outputs is a straightforward DuckDB join. No impedance mismatch.

Pay-per-result. 10,000 papers costs roughly $15. Compare that to Scopus institutional access at $20k+/year, and it is the clear choice for independent researchers and teams without a university affiliation.

Conclusion

Literature review in 2026 is an information-retrieval problem, not a reading problem. Automate the collection, deduplication, and ranking steps, then spend your actual brain cycles on the hard part: reading the 30 papers that matter.

Three actors to start:

Google Scholar Scraper — bot-detection-proof Scholar access.
ArXiv Scraper — full preprint metadata and PDF links.
Wikipedia Scraper — for background context on unfamiliar fields.

FAQ

Is scraping Google Scholar allowed?
Scholar's ToS prohibit automated access, and Google does actively rate-limit. That said, academic use of search data has been treated leniently in practice, and the aggregation of public bibliometric data is broadly considered fair use. If you publish research based on scraped data, cite Scholar as your source. For true systematic reviews with publication requirements, supplement Scholar with official databases (PubMed, Scopus) that have APIs and documented ToS.

How does this compare to Semantic Scholar's API?
Semantic Scholar has a clean API and free tier (100 req/5min). Its coverage is strong for CS, medicine, and physics but weaker for humanities and some regional journals. For thorough systematic reviews, use both: Scholar for coverage breadth, Semantic Scholar for citation graphs and clean metadata.

What about OpenAlex?
OpenAlex is an open bibliometric database covering 240M+ works with a free, unlimited API. It is excellent for citation graphs and author profiles, often better than Scholar for those specific tasks. Scholar still wins for "find me the most-read recent paper on X," because its ranking is informed by actual reader behavior.

Can I get full-text PDFs reliably?
For ArXiv papers, yes. For anything else, it depends on open-access status. The pipeline can fetch PDFs where URLs are public, but paywalled papers need either your institutional login, an interlibrary loan, or (for some reviews) contacting authors directly for a copy.

How do I cite papers I find this way in my own writing?
Export the deduped set to BibTeX via pandoc-citeproc or the bibtexparser Python library. The doi field in the output is what most reference managers key on.

What about non-English literature?
The actor passes through whatever Scholar returns, which includes non-English papers if your query matches them. For comprehensive non-English coverage, pair with regional databases (CNKI for Chinese, CiNii for Japanese, SciELO for Latin American journals).

How do I handle the "forward snowball" (papers that cite my seed papers)?
Use the include_citations=True flag and the "cited by" URL in the Scholar response. Run a second pass with those URLs as seeds. Two iterations of forward snowballing typically saturates a well-defined subfield.

Related tools

How to Build a Sentiment Dashboard from Reddit + Hacker News (2026)

NexGenData — Thu, 14 May 2026 07:07:02 +0000

How to Build a Sentiment Dashboard from Reddit + Hacker News (2026)

Developer sentiment is the leading indicator every dev-tools founder wishes they had. Twitter/X is noise, LinkedIn is performance art, but Hacker News and Reddit's technical subs are where people say what they actually think about your product, your stack, or the launch you fumbled last week. The problem: aggregating that signal manually is a soul-crushing job.

Some 2026 context on why this matters more than it did three years ago. Hacker News publishes roughly 900-1,200 stories per day with average thread lengths north of 60 comments. Reddit crossed 100M daily active users in 2024 and shows ~3 million comments per day across the 2,000+ technical subreddits most relevant to a dev-tools company. A developer-survey by JetBrains (2025, n=26,348) found that 71% of respondents named Hacker News and Reddit as their "first three sources" when evaluating new infrastructure or tooling. That is a bigger information-gathering role than product pages, vendor documentation, or even YouTube. If you ship a developer product and you are not listening to both, you are letting competitors shape the narrative unchallenged. The mental model most founders carry — "I'll just check HN when I remember" — produces exactly the right amount of information to confirm whatever mood you are in when you check.

In this post we build a sentiment dashboard that monitors Hacker News plus a configurable set of subreddits, runs every comment through an AI sentiment analyzer, and surfaces the themes trending up or down. Runs entirely on Apify plus a Postgres database plus a Grafana dashboard. The end state: a single pane of glass that shows, at any moment, "what is the internet saying about my product and my category?" with drill-down by topic, time, and source. You can replace three $300/month social-listening subscriptions with a stack that costs about $25/month in compute and owns its own data.

Why this is hard

Reddit API restrictions post-2023. Reddit's pricing changes killed most free Reddit scrapers. You need a route that handles throttling and auth renewal.
Hacker News has no native sentiment scores. Upvotes are not sentiment. A 300-point story with 400 comments can be 90% "this is garbage."
Comment threading. A comment at depth 4 disagreeing with its parent reads differently than one at depth 1. Flat sentiment averages lie.
AI sentiment quality. Off-the-shelf sentiment models (VADER, TextBlob) fail on sarcasm, tech jargon, and mixed polarity. You need an LLM-grade analyzer.
Topic extraction drift. "Postgres" as a topic at the string level is easy. "Postgres 17's new logical replication bug that affects read replicas but only under logical decoding with streaming enabled" is what you actually need to cluster on. Naive keyword extraction gets you buried in overlapping topics.
Temporal drift in baselines. What sentiment looks like on HN in December is not what it looks like in June (fewer comments, different moods, more end-of-year layoffs chatter). Alerts need seasonally-adjusted baselines, not hardcoded thresholds.

The architecture

[hacker-news-scraper]  --> stories + comments
         +
[reddit scraper / keyword watcher] (external)
         |
         v
   [Raw comments JSONL]
         |
         v
[ai-sentiment-analyzer] --> per-comment sentiment + topic
         |
         v
     [Postgres]
         |
         v
    [Grafana dashboard]

Step 1: Pull Hacker News

The hacker-news-scraper pulls stories and their comment trees with engagement metadata.

from apify_client import ApifyClient
client = ApifyClient("APIFY_TOKEN")

run = client.actor("nexgendata/hacker-news-scraper").call(run_input={
    "feeds": ["top", "new", "best"],
    "keywords": ["postgres", "rust", "openai", "apify", "anthropic"],
    "max_stories": 100,
    "include_comments": True,
    "max_comments_per_story": 200,
})

stories = list(client.dataset(run["defaultDatasetId"]).iterate_items())

Each story contains nested comment objects. Flatten:

def flatten(story):
    out = []
    def walk(c, depth, parent_id, story_id):
        out.append({
            "source": "hn",
            "story_id": story_id,
            "comment_id": c["id"],
            "parent_id": parent_id,
            "depth": depth,
            "text": c["text"],
            "author": c["by"],
            "timestamp": c["time"],
        })
        for child in c.get("children", []):
            walk(child, depth+1, c["id"], story_id)
    for c in story.get("comments", []):
        walk(c, 0, story["id"], story["id"])
    return out

comments = [row for s in stories for row in flatten(s)]

Step 2: Run sentiment analysis

The ai-sentiment-analyzer handles sarcasm and mixed polarity better than classic sentiment libraries. It returns polarity (-1 to +1), subjectivity, and a one-line topic summary.

sent_run = client.actor("nexgendata/ai-sentiment-analyzer").call(run_input={
    "texts": [{"id": c["comment_id"], "text": c["text"]} for c in comments],
    "include_topic": True,
    "language": "en",
})
sentiments = {s["id"]: s for s in client.dataset(sent_run["defaultDatasetId"]).iterate_items()}

Result per comment:

{
  "id": "39847112",
  "polarity": -0.62,
  "subjectivity": 0.8,
  "label": "negative",
  "topic": "postgres 17 performance regressions",
  "confidence": 0.91
}

Step 3: Persist to Postgres

import psycopg
with psycopg.connect("postgresql://...") as con:
    with con.cursor() as cur:
        for c in comments:
            s = sentiments.get(c["comment_id"], {})
            cur.execute("""
                INSERT INTO comments
                (source, story_id, comment_id, text, polarity, label, topic, ts)
                VALUES (%s,%s,%s,%s,%s,%s,%s,to_timestamp(%s))
                ON CONFLICT (comment_id) DO UPDATE
                SET polarity=EXCLUDED.polarity, label=EXCLUDED.label
            """, (c["source"], c["story_id"], c["comment_id"], c["text"],
                  s.get("polarity"), s.get("label"), s.get("topic"), c["timestamp"]))

Step 4: Build the dashboard

Grafana queries against Postgres:

Top negative topics (last 24h):

SELECT topic, COUNT(*) AS negatives, AVG(polarity) AS avg_polarity
FROM comments
WHERE label = 'negative' AND ts > now() - INTERVAL '24 hours'
GROUP BY topic
ORDER BY negatives DESC
LIMIT 20;

Sentiment over time for "apify":

SELECT date_trunc('hour', ts) AS hour, AVG(polarity) AS sentiment
FROM comments
WHERE text ILIKE '%apify%'
GROUP BY 1 ORDER BY 1;

Pin both panels to a dashboard and you have an early-warning system. When sentiment dips 2 standard deviations below baseline, trigger a PagerDuty alert. (Yes, for real.)

Here is a more sophisticated rolling-z-score alert that actually works in production — it uses a 14-day trailing baseline per keyword, so seasonal noise doesn't trigger pages:

import psycopg, os, json, urllib.request

PAGERDUTY_KEY = os.environ["PAGERDUTY_INTEGRATION_KEY"]
KEYWORDS = ["apify", "nexgendata", "postgres 17"]

def page(keyword, z, current, baseline):
    event = {
        "routing_key": PAGERDUTY_KEY,
        "event_action": "trigger",
        "payload": {
            "summary": f"Sentiment anomaly on '{keyword}' (z={z:.2f})",
            "severity": "warning",
            "source": "sentiment-dashboard",
            "custom_details": {
                "keyword": keyword,
                "current_polarity": current,
                "14d_baseline": baseline,
            },
        },
    }
    req = urllib.request.Request(
        "https://events.pagerduty.com/v2/enqueue",
        data=json.dumps(event).encode(),
        headers={"Content-Type": "application/json"},
    )
    urllib.request.urlopen(req)

with psycopg.connect(os.environ["DATABASE_URL"]) as con, con.cursor() as cur:
    for kw in KEYWORDS:
        cur.execute("""
            WITH recent AS (
              SELECT AVG(polarity) AS mean FROM comments
              WHERE text ILIKE %s AND ts > now() - INTERVAL '24 hours'
            ),
            baseline AS (
              SELECT AVG(polarity) AS mean, STDDEV(polarity) AS sd FROM comments
              WHERE text ILIKE %s AND ts > now() - INTERVAL '14 days'
                AND ts < now() - INTERVAL '24 hours'
            )
            SELECT recent.mean, baseline.mean, baseline.sd FROM recent, baseline
        """, (f"%{kw}%", f"%{kw}%"))
        cur_mean, base_mean, base_sd = cur.fetchone()
        if None in (cur_mean, base_mean, base_sd) or base_sd == 0:
            continue
        z = (cur_mean - base_mean) / base_sd
        if z < -2:
            page(kw, z, cur_mean, base_mean)
            print(f"ALERT: {kw} z={z:.2f}")

Run this on a 2-hour schedule from GitHub Actions or a cron container. The 24h / 14d baseline is a reasonable default, but if your keyword's chatter is sparse, widen the window to 72h / 30d.

Step 5: Extend to Reddit

Reddit's API is locked down post-2023, but two paths work:

Use Reddit's official Data API with an OAuth client (free for <100 req/min).
Use Apify's Reddit actor (not ours — this is a commodity one on the store) to scrape public subreddits.

For the sentiment pipeline, only the comment text and timestamp matter. Feed the output through the same ai-sentiment-analyzer call and write to the comments table with source = 'reddit'.

Use cases

1. Dev-tool founder listening post. A database company monitors five competitor names plus their own. Weekly, the dashboard surfaces the top five complaints per product — direct feature-roadmap input.

2. Release post-mortem. A framework team launched a v2 and saw sentiment crater. Grafana showed 70% of negativity clustered around a breaking change nobody mentioned in the blog post. Addressed in a follow-up patch.

3. Hiring pulse. A recruiter monitors "remote work" and "layoffs" threads, detecting sentiment swings that correlate with candidate availability windows.

4. Crypto/macro signal. A trader runs the pipeline over /r/cryptocurrency and WSB to front-run retail sentiment shifts.

5. Community manager priority queue. A DevRel team for an open-source framework uses the dashboard to route themselves toward the threads most worth engaging with. Instead of reading all 40 daily mentions, they look at the 5 with polarity < -0.4 and confidence > 0.8. Response time to legitimate criticism dropped from 3 days to 6 hours, and community sentiment metrics recovered measurably over the following quarter.

6. PR crisis early warning. A consumer SaaS company was quietly getting clobbered on /r/SideProject over a pricing change. The sentiment dashboard flagged the spike 9 hours before it hit their Twitter mentions. They rolled back, posted a clear statement, and contained what would have been a 48-hour news cycle into a morning.

Pricing comparison

Service	Monthly cost (100k comments)	Reddit?	HN?	AI analysis?
Brand24	$149	Limited	No	Basic
Brandwatch	$1000+	Yes	Yes	Yes
Mention	$41+	Limited	Limited	Basic
Custom (Reddit API + OpenAI)	$60 + eng	Yes	Yes	Yes
Apify combo	~$25	Via add-on	Yes	Yes

Common pitfalls

Sentiment pipelines look simple and are not. These are the traps most teams fall into:

Comment depth matters. Weight top-level comments more than replies in rollups — replies pile on and amplify whatever the top comment said, creating an echo-chamber effect in your averages. A common heuristic: weight by 1/(depth+1) or only aggregate top-level + depth-1 replies.
Sarcasm on HN is peak. Manually spot-check 20 "negative" comments weekly; if polarity calibration drifts, tune the analyzer prompt. The canonical failure: "oh great, another JavaScript framework" gets tagged positive by naive models because "great" is a strong positive signal.
Deletion handling. HN comments get deleted. Re-run your fetcher on a 48-hour lag to capture final state. Reddit deletions are even messier — a deleted user's entire history may vanish, creating gaps in your time series. Store a first_seen_ts and treat comments as immutable once captured.
Shills and sock puppets. Any subreddit above 100k subs has them. A rash of positive comments from 12-hour-old accounts is a signal, not sentiment. Track account age and karma alongside the comment, and flag suspicious clusters.
Brigades and cross-posts. A thread linked from Twitter/X or another subreddit can flood with off-topic noise. Watch for sudden comment-volume spikes that are out of proportion to upvote ratio — those are usually brigades, not organic conversation.
Jargon and acronyms. Tech subreddits have massive in-group vocabulary. "Skill issue" is a pejorative, "NGMI" is negative, "chad" is positive. A general-purpose sentiment model will misread all three. Either fine-tune or prompt-engineer for your domain.
Language mixing. Subs like /r/india and /r/brasil have heavy code-mixing (English + Hindi/Portuguese). Your analyzer should either handle multilingual or you need language detection + filtering upstream.
Quote extraction. A comment that quotes another ("> the GC pauses are unacceptable") may be sarcastic agreement or genuine criticism. Naive analyzers treat the quote as the comment's own text. Strip or tag quoted lines.
Sentiment vs. engagement. High-engagement threads skew negative (controversy drives comments). If you are tracking rollups, normalize sentiment by expected baseline for that story's score, not the flat average.
Timezone biases. HN comments peak in US business hours; a sentiment average skewed by time-of-day tells you more about who is online than what they think. Aggregate by rolling 24h window, not calendar day.

How NexGenData handles this

The hacker-news-scraper and ai-sentiment-analyzer were designed to work together, which drives some specific choices:

Full comment tree with depth metadata. Every comment in the output includes its depth, parent ID, and the path from root. You can weight, filter, or flatten however your analysis requires — no restructuring needed.

Incremental fetching. Pass a since timestamp and the scraper returns only new comments since your last run. You are not paying to re-fetch the same 10,000 comments every hour.

Deletion-aware re-runs. A follow-up mode: "refresh" re-examines comments you already have and updates their status if they have been deleted or edited. Keeps your dataset honest.

LLM-grade sentiment with topic extraction. The ai-sentiment-analyzer uses a prompted LLM rather than VADER/TextBlob, which handles sarcasm, tech jargon, and mixed polarity far better. It also returns a topic string, not just polarity, so clustering by theme is immediate.

Batch cost efficiency. The analyzer batches up to 100 texts per API call and uses a small, fast model (the equivalent of GPT-4o-mini class), so per-comment cost is fractions of a cent.

Pay-per-result pricing for both actors. 10,000 comments scraped and sentiment-analyzed costs roughly $3-5 all-in. A comparable Brandwatch subscription that covers the same volume is $1,000+/month with multi-seat minimums.

Conclusion

A sentiment dashboard is one of the highest-leverage projects a solo founder can build. It takes half a day, costs <$30/month, and replaces a surprising amount of anecdotal "what is the vibe" anxiety with actual data.

Build it with:

Hacker News Scraper — stories, comments, engagement history.
AI Sentiment Analyzer — LLM-grade polarity + topic extraction.
AP News Scraper — layer in mainstream news signal alongside developer chatter.

FAQ

Is scraping Hacker News allowed?
Yes, within reason. HN has an official Firebase-backed API at hn.algolia.com and hacker-news.firebaseio.com that anyone can use for free. The Apify actor uses these endpoints and adds structure, pagination, and dedup. HN does not require an API key. Be polite — don't hammer at 100 req/sec.

Can I still scrape Reddit in 2026?
Yes, but carefully. Reddit's Data API is free for <100 req/min with OAuth; higher volumes are paid. For public subreddits, the official API is the safest route. Third-party scrapers exist but carry more risk as Reddit tightens enforcement. For this use case (sentiment over a few focused subs), the Data API is plenty.

Which LLM should I use for sentiment?
For this use case, a small fast model (GPT-4o-mini, Claude Haiku, or Llama 3.1 8B hosted on Groq or Cerebras) is more than enough. The ai-sentiment-analyzer actor uses a prompt-engineered equivalent. The task is not "write Shakespeare," it's "classify this 200-character text reliably." Cheap models win on cost per comment without a meaningful accuracy loss.

How do I handle ambiguous polarity?
Always store the confidence score alongside polarity. In dashboards, filter to confidence > 0.7 for rollups. Comments with low confidence are often genuinely ambiguous (sarcasm, mixed signals) and should go in a "needs human review" bucket, not a statistic.

What about non-English posts?
The ai-sentiment-analyzer supports most major languages out of the box. For subreddits with heavy code-mixing, detect language per-comment and process only those above a confidence threshold.

Can I share this dashboard with my team?
Grafana supports multi-user dashboards with role-based permissions. A common pattern: give the CEO read-only access to a "Sentiment overview" board, give product managers access to the per-topic drilldowns, and keep the alert-config board locked to the ops team.

How accurate is LLM sentiment really?
On developer-heavy text, LLM sentiment with a domain-tuned prompt runs 85-92% agreement with human labels on a hand-coded test set. VADER and TextBlob on the same corpus score 58-65%. The gap is enormous for sarcasm and technical jargon, which is where developer chatter lives.

Related tools

H1B Salary Research Toolkit for Job Seekers (2026)

NexGenData — Thu, 14 May 2026 07:07:00 +0000

H1B Salary Research Toolkit for Job Seekers (2026)

If you are on (or considering) an H1B visa in 2026, the single highest-ROI hour you can spend is pulling your own compensation benchmark data. Employers know the prevailing-wage floor for your LCA. Recruiters know the tier bands at your company. The only person who usually does not know is you — which is exactly why you end up $25k under market.

Some context on the 2026 landscape: USCIS received roughly 442,000 H1B registrations for FY2026, with an 85,000-visa annual cap (65,000 regular + 20,000 US-masters). The cap hit in March 2025 with a 5.2x oversubscription, which means the tightness of the H1B market has real leverage effects on compensation — both good and bad. Employers who genuinely want specific candidates are willing to pay premium multiples above prevailing wage. Employers who rely on high-volume LCA filings (the classic "body shops") pay closer to the prevailing-wage floor. Knowing which side of that line your target employer falls on, before you walk into the negotiation, is worth anywhere from $15k to $80k on a senior-engineer offer. And unlike most career advice, the data for this is 100% public — the Department of Labor releases quarterly LCA disclosure datasets, every filing by every employer, searchable by job title and location. The problem is nobody outside specialty firms and immigration lawyers actually parses it.

This post builds a personal salary research toolkit using public H1B filing data plus general compensation scrapers. You will end up with a spreadsheet showing what your target role pays by company, location, and seniority — the same data recruiters use to anchor their first offer. The mental model to adopt: you are not asking "what should I make?" — you are answering "what has this company actually paid people with my profile, as proven by their own federal filings, in the last 24 months?" That shift in framing alone changes how the conversation goes.

Why this is hard

Public H1B data is technically free. The DOL publishes Labor Condition Applications quarterly. But getting to usable insight is rough:

Raw DOL data is 2-3 GB of CSV per quarter with inconsistent column names across years.
Employer names are messy. "Amazon.com Services LLC", "Amazon Web Services Inc.", and "Amazon Dev Center U.S., Inc." are all Amazon.
Prevailing wage != actual wage. The LCA lists the floor. Real TC (base + bonus + equity) is 20-60% higher.
Location adjustments. An SF offer at $180k base is different from a Seattle $180k base when equity and cost-of-living differ.

You need to join LCA data with actual compensation data (Levels.fyi, Glassdoor, Blind) and salary benchmark APIs.

SOC code ambiguity. The Standard Occupational Classification is granular, but many employers file under a generic "Software Developers, Applications" (15-1252) regardless of actual role. A data scientist and a backend engineer can appear identical in the LCA.
Wage unit confusion. Most filings are annual, but a minority are filed hourly or monthly. Group-by or aggregation without normalizing to annual throws off your bands by a factor of 10.

The architecture

[Target company + role]
          |
          v
 [h1b-visa-salary-search] --> LCA history, prevailing wages, counts
          |
          v
 [salary-data-search]     --> market compensation, bands, percentiles
          |
          v
     [Google Sheet]
          |
          v
 [Negotiation cheat sheet]

Step 1: Pull H1B filings for a target company

The h1b-visa-salary-search actor aggregates DOL LCA filings with fuzzy employer matching already done.

from apify_client import ApifyClient
client = ApifyClient("APIFY_TOKEN")

run = client.actor("nexgendata/h1b-visa-salary-search").call(run_input={
    "employers": ["Stripe", "Databricks", "Anthropic"],
    "job_titles": ["Software Engineer", "Senior Software Engineer",
                   "Staff Software Engineer", "Machine Learning Engineer"],
    "years": [2024, 2025, 2026],
    "locations": ["San Francisco, CA", "New York, NY", "Seattle, WA"],
})

filings = list(client.dataset(run["defaultDatasetId"]).iterate_items())

Each record:

{
  "employer": "Stripe, Inc.",
  "job_title": "Software Engineer II",
  "base_salary": 210000,
  "wage_unit": "Year",
  "location": "San Francisco, CA",
  "filing_date": "2025-03-14",
  "case_status": "Certified",
  "soc_code": "15-1252",
  "prevailing_wage": 186300
}

Note both base_salary (what the employer committed to pay) and prevailing_wage (the DOL floor for the role/location). The gap is informative — a tight gap means the employer is paying close to floor; a wide gap means they are comfortable paying well above.

Step 2: Build band distributions

import pandas as pd
df = pd.DataFrame(filings)
bands = (df.groupby(["employer","job_title","location"])
           .base_salary.quantile([0.25,0.5,0.75])
           .unstack())
bands.columns = ["p25","p50","p75"]
print(bands)

Output:

                                                         p25      p50      p75
employer  job_title              location
Stripe    Software Engineer II   San Francisco, CA    205000   215000   228000
Stripe    Senior Engineer        San Francisco, CA    245000   260000   280000

That p50 is your floor. Anything below is an instant-counter.

Step 3: Cross-check with market comp

H1B base salary does not include bonus or equity. For total comp, pull market data via salary-data-search:

comp_run = client.actor("nexgendata/salary-data-search").call(run_input={
    "roles": ["Senior Software Engineer"],
    "companies": ["Stripe", "Databricks", "Anthropic"],
    "locations": ["San Francisco, CA"],
    "include_bonus": True,
    "include_equity": True,
})

comp = list(client.dataset(comp_run["defaultDatasetId"]).iterate_items())

Returns ranges like:

{
  "company": "Stripe",
  "role": "Senior Software Engineer",
  "level": "L4",
  "location": "San Francisco, CA",
  "base_range": [210000, 260000],
  "bonus_range": [20000, 40000],
  "equity_4yr": [400000, 700000],
  "sample_size": 82
}

Step 4: Compare offer vs. band

Quick negotiation sanity check:

def evaluate_offer(offer_base, offer_bonus, offer_equity_4yr, market):
    m = market
    tc_offer = offer_base + offer_bonus + offer_equity_4yr/4
    tc_market_median = m["base_range"][0] + (m["base_range"][1]-m["base_range"][0])/2 \
                     + m["bonus_range"][0] + (m["bonus_range"][1]-m["bonus_range"][0])/2 \
                     + ((m["equity_4yr"][0]+m["equity_4yr"][1])/2)/4
    return round((tc_offer/tc_market_median - 1)*100, 1)

pct = evaluate_offer(225000, 25000, 400000, comp[0])
print(f"Offer is {pct}% of market median")

If the result is negative, you have concrete data to push back with.

Here is a more complete end-to-end script that produces a Google-Sheets-ready CSV combining LCA + market comp for a list of target companies. Designed to be copy-pasted before a round of final-round interviews:

import csv, statistics
from apify_client import ApifyClient
from collections import defaultdict

client = ApifyClient("APIFY_TOKEN")

TARGETS = [
    ("Stripe", ["Software Engineer", "Senior Software Engineer", "Staff Software Engineer"]),
    ("Databricks", ["Software Engineer", "Senior Software Engineer"]),
    ("Anthropic", ["Member of Technical Staff"]),
]
LOCS = ["San Francisco, CA", "New York, NY", "Seattle, WA"]

rows = []
for company, titles in TARGETS:
    lca = client.actor("nexgendata/h1b-visa-salary-search").call(run_input={
        "employers": [company], "job_titles": titles,
        "years": [2024, 2025, 2026], "locations": LOCS,
    })
    comp = client.actor("nexgendata/salary-data-search").call(run_input={
        "companies": [company], "roles": titles, "locations": LOCS,
        "include_bonus": True, "include_equity": True,
    })

    lca_by_key = defaultdict(list)
    for f in client.dataset(lca["defaultDatasetId"]).iterate_items():
        if f.get("case_status") != "Certified" or f.get("wage_unit") != "Year":
            continue
        key = (f["employer"], f["job_title"], f["location"])
        lca_by_key[key].append(f["base_salary"])

    comp_by_key = {(c["company"], c["role"], c["location"]): c
                   for c in client.dataset(comp["defaultDatasetId"]).iterate_items()}

    for (emp, title, loc), salaries in lca_by_key.items():
        if len(salaries) < 3:
            continue
        c = comp_by_key.get((company, title, loc), {})
        rows.append({
            "company": emp, "title": title, "location": loc, "n_filings": len(salaries),
            "lca_p25": int(statistics.quantiles(salaries, n=4)[0]),
            "lca_p50": int(statistics.median(salaries)),
            "lca_p75": int(statistics.quantiles(salaries, n=4)[2]),
            "market_base_low": c.get("base_range", [None, None])[0],
            "market_base_high": c.get("base_range", [None, None])[1],
            "market_tc_midpoint": (
                c["base_range"][1] + c.get("bonus_range", [0,0])[1]
                + c.get("equity_4yr", [0,0])[1] / 4
            ) if c else None,
        })

with open("negotiation_prep.csv", "w", newline="") as f:
    w = csv.DictWriter(f, fieldnames=list(rows[0].keys()))
    w.writeheader(); w.writerows(rows)
print(f"Wrote {len(rows)} rows")

Open that CSV in Google Sheets, conditional-format the columns, and you are walking into the negotiation with a spreadsheet that recruiters literally cannot argue with — it's their employer's own federal filings.

Step 5: Track employers with healthy sponsorship

For H1B visa holders, approval history matters as much as salary. The actor also returns per-employer counts:

-- Top sponsors in your field (proxy for: they actually file)
SELECT employer, COUNT(*) AS filings, AVG(base_salary) AS avg_pay
FROM filings
WHERE job_title ILIKE '%machine learning%'
  AND case_status = 'Certified'
  AND filing_date > '2024-01-01'
GROUP BY employer
ORDER BY filings DESC
LIMIT 30;

Use cases

1. Pre-offer prep. Before signing, a senior engineer pulled their potential employer's last 24 months of LCAs. Found peers filed at +15% of their offered base. Negotiated and got it.

2. Targeting companies that actually sponsor. A mid-career PM filters for employers who have filed 20+ successful LCAs in the past 2 years. Saves months of dead-end applications.

3. Location arbitrage. A senior dev compared NYC vs. Austin vs. Seattle LCAs for the same role at the same employer. Austin came out ahead on effective comp.

4. Salary review for green card cases. Pulling your employer's LCA filings for people at your level is a strong anchor for "am I being underpaid" conversations.

5. PERM/I-140 prevailing wage awareness. For green card sponsorship, the employer files a PERM labor certification that references the DOL prevailing wage. Knowing your target prevailing wage level (I, II, III, IV) before conversations with your immigration attorney can flag potential downgrading, where an employer files at a lower level than your actual responsibilities warrant.

6. Relocation decision support. A senior engineer weighing a cross-country transfer used the toolkit to quantify the effective comp delta. LCA data showed her company paid equivalent titles 18% less in the Raleigh office than in the NYC office. She renegotiated the transfer offer upward by 12% before accepting.

Pricing comparison

Service	Monthly cost	LCA history	Market comp	Sponsor counts
myvisajobs.com	Free (limited)	Yes	No	Yes
h1bdata.info	Free	Yes	No	Limited
Levels.fyi	Free + paid	No	Yes	No
Payscale	$29/month	No	Yes	No
Apify combo	~$5	Yes	Yes	Yes

You get the DOL data that myvisajobs paywalls plus Levels-grade comp data in one pipeline for <$5/month at typical job-search volume.

Common pitfalls

LCA data is rich but full of edge cases. Here are the ones that regularly mislead people using it for negotiation:

LCA != offer letter. Companies file LCAs for ranges or multiple candidates. Treat as floor, not ceiling. The filed wage is often the minimum the employer is committing to — the real offer may be 10-30% higher, especially at senior levels.
Case_status filter. Only "Certified" filings are valid; "Denied" and "Withdrawn" are noise. Some public tools aggregate all statuses and skew distributions. Always filter WHERE case_status = 'Certified' or 'Certified-Withdrawn' (the employer got the cert but withdrew it, usually because the candidate chose another offer — still a valid data point).
Location field is messy. "San Francisco, CA" and "SAN FRANCISCO, CALIFORNIA" both appear. Normalize before grouping. Also beware metro-level filings: some employers file with a metro MSA (e.g., "San Jose-Sunnyvale-Santa Clara, CA") that needs a crosswalk to the city you care about.
Remote work complicates location. Post-pandemic, some LCAs list the employer's HQ location while the actual work is remote. Cross-reference with the employer's job postings if the location seems off.
Amendment filings inflate counts. An employer amending an existing H1B (title change, location change, material job-duty change) files a new LCA. These show up as new filings but do not represent a new hire. For "is this employer actively hiring" analysis, watch for duplicate beneficiary fingerprints.
Level of experience confounds title. "Software Engineer" at Company A might mean "L3 new-grad" while at Company B it means "L5 staff-engineer equivalent." Always cross-reference with levels.fyi or Blind mappings before concluding a company pays more/less.
The 2026 H1B wage floor rule changes. In late 2025, DOL proposed raising prevailing wage levels across the board. As of this writing the rule is partially implemented. Filings from Q1 2026 onward may reflect higher floors. Do not compare 2026 filings directly to 2023 without accounting for this.
OPT STEM salaries aren't in LCA data. If you are on F1 OPT, your salary is not in this data. LCAs only cover H1B/H1B1/E-3 workers. For OPT negotiation, rely on levels.fyi and Blind instead.
Prevailing wage levels I-IV. Level I is "entry"; Level IV is "fully competent." The level determines the minimum. An employer filing a senior engineer at Level II is legally questionable — that is a downgrading pattern flag.
Secondary filings (H1B transfers) look like new hires. They are not. A transfer from Company A to Company B generates a new LCA at Company B. For hiring-volume metrics, filter by visa_class = H1B new-employment vs. continuing-employment.

How NexGenData handles this

The h1b-visa-salary-search actor is built to solve these problems rather than expose them:

Pre-normalized employer names. We run every raw DOL employer string through an entity-resolution pipeline that collapses variations. "AMAZON.COM SERVICES LLC", "Amazon Web Services, Inc.", and "Amazon Dev Center U.S., Inc." resolve to a canonical Amazon entity with a stable ID.

Wage-unit normalization. Hourly and monthly filings are converted to annualized equivalents with a standard assumption (2080 hours/year for hourly, 12 months for monthly) and flagged with a wage_source field so you can filter if you prefer.

Location normalization. All location strings are mapped to canonical MSA + state. You can group by MSA without messy string matching.

Status filtering by default. Denied and Withdrawn filings are excluded from default output. A flag lets you include them for specific analyses (e.g., "what employers have a high denial rate").

Fresh data weekly. We pull the quarterly DOL release and interpolate with weekly OFLC datasets. The data is rarely more than 5-7 days stale.

Pay-per-result pricing. A single company's 24-month history is typically under $1. Compare with myvisajobs.com's $99/year subscription — the actor is cheaper at any volume under 100 company queries per year.

Conclusion

The information asymmetry in hiring is entirely on the employer's side by default. A three-hour afternoon with these two actors flips it. You walk into the negotiation with percentiles, not vibes.

Start the toolkit with:

H1B Visa Salary Search — LCA filings and sponsorship history.
Salary Data Search — market compensation with bonus + equity.
GitHub Repo Stats — evaluate the engineering org you are joining.

FAQ

Is this legal to use for negotiation?
Absolutely. LCA data is public information the employer files with the federal government. You are allowed to cite it in negotiation. Most recruiters will not be surprised — some will be impressed you did the work.

What if my target company doesn't sponsor H1B?
Then LCA data will not help you, but salary-data-search still will. Use the market-comp side of the pipeline on its own.

Does H1B salary data reflect total compensation?
No. LCA only covers base salary. Bonus and equity are not disclosed. That is why you must cross-reference with market-comp data for total compensation.

How do I identify my "level" at a target company?
Use levels.fyi's role mapping. Most major tech companies have well-documented leveling guides on levels.fyi, Blind, or Team Blind's anonymous comp sheets. Match your current scope of work and years of experience against their level descriptions, then filter LCAs for the corresponding title.

Can I use this data in a salary dispute with my current employer?
Yes, and it's a useful tool. If your current employer has filed LCAs for peers at your level at a higher wage than you make, that is a reasonable conversation to have in a comp review — though the cultural norms vary by company.

What about non-tech H1B roles?
The actor works for any SOC code. Filings are not tech-specific. Pharmaceutical scientists, financial analysts, civil engineers — all in the data. Adjust your SOC code filter accordingly.

How recent is the data?
Quarterly releases are about 45-60 days behind. Weekly OFLC updates are 7-14 days behind. For negotiation purposes, data within 3 months is considered current.

Is there an equivalent for L1, O1, or TN visas?
LCA data is specific to H1B/H1B1/E-3. L1 visa wages are not published. For O1 and TN, there is no equivalent public filing. These visa types require different research strategies (Glassdoor, levels.fyi, peer networks).