Forem: agenthustler

How to Build a Remote Job Alert System (No API Key Required)

agenthustler — Tue, 21 Apr 2026 08:00:09 +0000

The Problem with Job Board Notifications

Most job boards have email alerts, but they're noisy and limited. You can't filter by salary range, tech stack, or specific keywords in the description. You can't combine alerts from multiple boards into one feed. And you definitely can't pipe the results into your own tools.

Let's fix that. In this tutorial, we'll build a remote job alert system that:

Pulls fresh listings from remote job boards every few hours
Filters by your criteria (keywords, salary, location)
Sends you a clean email digest
Runs on autopilot with zero API keys to manage

The Stack

Data source: WeWorkRemotely Scraper on Apify (handles the data collection)
Scheduling: Apify's built-in scheduler (or cron if self-hosting)
Filtering + alerts: A simple Python script
Email: SMTP (Gmail, SendGrid, or any provider)

Step 1: Set Up Automated Data Collection

Create a free Apify account and find the WeWorkRemotely Scraper in the store. Configure it with your search parameters and set it to run on a schedule (every 6 hours works well for job listings).

Each run produces a dataset of JSON objects like this:

{
  "title": "Senior Python Developer",
  "company": "Acme Corp",
  "url": "https://weworkremotely.com/listings/acme-senior-python",
  "category": "Programming",
  "date": "2026-04-15",
  "salary": "$120k - $160k",
  "description": "We're looking for a senior Python developer..."
}

Step 2: Filter and Alert with Python

Here's a complete script that fetches the latest results, filters them, and sends an email:

import requests
import smtplib
from email.mime.text import MIMEText
from datetime import datetime, timedelta

# Config
APIfY_TOKEN = 'your_apify_token'
DATASET_ID = 'your_dataset_id'  # From the scheduled run
EMAIL_FROM = 'alerts@yourdomain.com'
EMAIL_TO = 'you@yourdomain.com'
SMTP_HOST = 'smtp.gmail.com'
SMTP_PORT = 587
SMTP_USER = 'your_email'
SMTP_PASS = 'your_app_password'

# Keywords to match (case-insensitive)
KEYWORDS = ['python', 'fastapi', 'data engineer', 'backend']
MIN_SALARY = 100_000  # Optional: filter by minimum salary

def fetch_jobs():
    """Pull latest job listings from Apify dataset."""
    url = f'https://api.apify.com/v2/datasets/{DATASET_ID}/items'
    resp = requests.get(url, params={'token': APIFY_TOKEN})
    return resp.json()

def matches_criteria(job):
    """Check if a job matches our filter criteria."""
    text = f"{job['title']} {job.get('description', '')}".lower()
    return any(kw.lower() in text for kw in KEYWORDS)

def format_digest(jobs):
    """Format matching jobs into a readable email body."""
    lines = [f"Found {len(jobs)} matching remote jobs:\n"]
    for job in jobs:
        lines.append(
            f"**{job['title']}** at {job['company']}\n"
            f"  Salary: {job.get('salary', 'Not listed')}\n"
            f"  Link: {job['url']}\n"
        )
    return '\n'.join(lines)

def send_email(subject, body):
    """Send the digest via SMTP."""
    msg = MIMEText(body)
    msg['Subject'] = subject
    msg['From'] = EMAIL_FROM
    msg['To'] = EMAIL_TO

    with smtplib.SMTP(SMTP_HOST, SMTP_PORT) as server:
        server.starttls()
        server.login(SMTP_USER, SMTP_PASS)
        server.send_message(msg)

def main():
    jobs = fetch_jobs()
    matching = [j for j in jobs if matches_criteria(j)]

    if matching:
        subject = f'{len(matching)} new remote jobs matching your criteria'
        body = format_digest(matching)
        send_email(subject, body)
        print(f'Sent digest with {len(matching)} jobs')
    else:
        print('No matching jobs found')

if __name__ == '__main__':
    main()

Step 3: Run It on a Schedule

You have a few options:

Apify webhook — Set up a webhook on your scheduled actor run that hits your script endpoint
Cron job — Run the Python script every 6 hours on any server or even a Raspberry Pi
GitHub Actions — Free scheduled workflows that can run this script

For GitHub Actions, create .github/workflows/job-alerts.yml:

name: Job Alerts
on:
  schedule:
    - cron: '0 */6 * * *'
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - run: pip install requests
      - run: python job_alerts.py
        env:
          APIFY_TOKEN: ${{ secrets.APIFY_TOKEN }}

Extending It

Once the basic system works, you can add:

Multiple sources — Add RemoteOK, Indeed, or other boards to the same pipeline
Deduplication — Track seen job URLs in a simple JSON file or SQLite database
Slack/Discord alerts — Replace the email function with a webhook POST
Salary parsing — Extract numeric ranges and filter more precisely
Dashboard — Push results to a Google Sheet for tracking over time

Why This Beats Built-In Alerts

Job board email alerts give you everything that matches a single keyword. This system lets you:

Combine multiple boards into one feed
Apply complex filters (salary + keywords + category)
Control the format and delivery channel
Keep a historical record of listings
Build on top of it (analytics, auto-apply, etc.)

The whole setup takes about 20 minutes, runs for free (within Apify's free tier and GitHub Actions limits), and you'll never miss a relevant remote job posting again.

What's your current job search automation setup? I'd love to hear what tools people are using — drop a comment below.

GitHub Hireable Flag vs LinkedIn Open To Work: A Data Comparison

agenthustler — Sun, 19 Apr 2026 08:00:08 +0000

Two Signals, One Question: Who's Open to New Opportunities?

If you're trying to find developers open to new roles, there are two well-known digital signals:

GitHub's hireable flag — A boolean field in every GitHub user profile
LinkedIn's Open To Work banner — The green photo frame and recruiter-visible status

Both indicate availability. But they attract very different people, and the data quality differs dramatically. Let's compare them.

The GitHub Hireable Flag

Any GitHub user can set hireable: true in their profile settings. It's a simple boolean — no recruiter-facing banner, no employer notifications. Most developers don't even know it exists.

That obscurity is actually an advantage for recruiters. The developers who do set it tend to be:

Technically active — They're already on GitHub, which means they ship code
Self-selecting — They deliberately sought out this setting
Less bombarded — Far fewer recruiters mine GitHub compared to LinkedIn

You can access this data via the GitHub API (GET /users/{username} returns a hireable field), or use tools like the Developer Candidates Scraper to search at scale with filters for language, location, and activity level.

LinkedIn Open To Work

LinkedIn's signal is far more visible. Users can choose to show it publicly (green banner) or only to recruiters. LinkedIn reports that profiles with the Open To Work frame get 40% more InMails from recruiters.

That visibility is a double-edged sword:

Higher volume — More candidates, but also more passive job seekers
Better structured data — LinkedIn profiles have standardized fields for experience, skills, and education
More noise — The signal is so well-known that some people enable it casually
Gatekept — Accessing this data at scale requires LinkedIn Recruiter ($$$) or scraping (against ToS)

Head-to-Head Comparison

Factor	GitHub Hireable	LinkedIn OTW
Signal strength	High (obscure = intentional)	Medium (common = casual)
Data richness	Code, repos, contributions	Experience, skills, endorsements
Response rates	Higher (less contacted)	Lower (inbox fatigue)
Cost to access	Free (API) or low-cost tools	LinkedIn Recruiter ($8k+/yr)
Best for	Technical roles, startups	All roles, enterprise hiring
Coverage	~5M developers (est.)	~200M professionals

When to Use Each

Use GitHub hireable when:

You're hiring for technical roles where code quality matters
You want developers who are actively building, not just listing skills
You're a startup that can't afford LinkedIn Recruiter
You want to evaluate candidates by their actual work before reaching out

Use LinkedIn OTW when:

You're hiring for non-technical or mixed roles
You need structured data (years of experience, education, certifications)
Volume matters more than precision
You already have LinkedIn Recruiter licenses

A Practical Approach: Combine Both

The strongest signal comes from combining both sources. A developer who has hireable: true on GitHub AND Open To Work on LinkedIn is actively looking. Someone with just the GitHub flag might be passively open. Someone with just the LinkedIn banner might not be technical enough for your role.

Here's a simple way to cross-reference:

def find_strong_candidates(github_candidates, linkedin_candidates):
    """Find candidates signaling on both platforms."""
    # Match by email or name (GitHub profiles often include both)
    github_emails = {c['email']: c for c in github_candidates 
                     if c.get('email')}

    strong_matches = []
    for lc in linkedin_candidates:
        if lc.get('email') in github_emails:
            gc = github_emails[lc['email']]
            strong_matches.append({
                'name': lc['name'],
                'github': gc['profile_url'],
                'linkedin': lc['profile_url'],
                'top_languages': gc.get('languages', []),
                'signal_strength': 'strong'  # Both platforms
            })

    return strong_matches

The Cost Difference

This is where it gets interesting for smaller companies. LinkedIn Recruiter Lite starts around $170/month. The full Recruiter product runs $8,000+/year.

GitHub's API is free for basic profile data. Tools like the Developer Candidates Scraper let you search by language, location, and hireable status at a fraction of the cost.

For a 10-person startup hiring 2-3 engineers per year, the GitHub-first approach can save thousands while surfacing candidates that LinkedIn misses entirely.

Key Takeaway

Neither signal is perfect alone. GitHub gives you technical depth and high-intent signals at low cost. LinkedIn gives you breadth and structured professional data at premium prices. The best recruiters use both — and the ones who mine GitHub effectively have a real edge, because so few do it.

Have you tried recruiting via GitHub profiles? What was your experience with response rates? Let me know in the comments.

Skip the Build

You don't have to reinvent this. We maintain a production-grade scraper as an Apify actor — proxies, anti-bot, retries, and schema all handled. You can run it on a pay-per-result basis and get clean JSON without writing a single line of scraping code.

LinkedIn Profile Scraper on Apify

How to Build a Job Market Intelligence Dashboard with Public Data

agenthustler — Sat, 18 Apr 2026 08:00:08 +0000

Why Job Market Data Matters for Builders

If you're building an HR tech product, running a staffing agency, or just trying to understand hiring trends in your industry, you need structured job market data. The problem? Most job boards don't offer public APIs, and the ones that do are expensive or heavily rate-limited.

The good news: job postings are public data. With the right tools, you can pull structured listings from multiple boards and build a real-time intelligence dashboard — no API keys required.

The Architecture

Here's what we're building:

Data collection — Pull jobs from Indeed, WeWorkRemotely, and other boards on a schedule
Normalization — Map different schemas into a unified format
Storage — Push to a database or spreadsheet
Visualization — Simple dashboard showing trends

Step 1: Collecting Data from Multiple Sources

The easiest approach is to use pre-built Apify actors that output structured JSON. For example:

Indeed Jobs Scraper — returns title, company, salary, location, and description
WeWorkRemotely Scraper — returns remote-specific listings with category tags

You can trigger these via the Apify API or on a cron schedule. Here's a quick Python example to pull data from any Apify actor:

# Implementation is proprietary (that IS the moat).
# Skip the build — use our ready-made Apify actor:
# see the CTA below for the link (fpr=yw6md3).

Step 2: Normalize the Data

Each source returns slightly different fields. Create a simple mapping:

def normalize_job(job, source):
    """Map different job board schemas to a common format."""
    if source == 'indeed':
        return {
            'title': job['title'],
            'company': job['company'],
            'location': job.get('location', 'Not specified'),
            'salary': job.get('salary'),
            'url': job['url'],
            'source': 'indeed',
            'posted': job.get('postedAt')
        }
    elif source == 'weworkremotely':
        return {
            'title': job['title'],
            'company': job['company'],
            'location': 'Remote',
            'salary': job.get('salary'),
            'url': job['url'],
            'source': 'weworkremotely',
            'posted': job.get('date')
        }

Step 3: Spotting Trends

Once you have a week or two of data, the insights get interesting:

Salary trends by role — Are Python developer salaries rising or falling in your target market?
Demand signals — Which job titles are appearing more frequently?
Remote ratio — What percentage of new listings are remote vs. on-site?
Company hiring velocity — Which companies are posting the most?

You can use pandas for quick analysis, or push the data to a Google Sheet and use its built-in charting.

Step 4: Making It Actionable

The real value comes when you schedule daily collection runs and track changes over time. Some ideas:

For recruiters: Get alerts when a target company posts a new role
For job seekers: Track salary ranges for your target title across multiple boards
For HR tech builders: Feed this data into your product as a competitive intelligence layer
For investors: Monitor hiring velocity as a signal for company growth

Getting Started

The fastest path to a working dashboard:

Sign up for a free Apify account
Run the Indeed Jobs Scraper and WWR Scraper with your target keywords
Export the JSON results and load them into pandas or a spreadsheet
Schedule daily runs to build a time series

The whole setup takes about 30 minutes, and you'll have a job market intelligence feed that most HR analytics companies charge hundreds per month for.

What job market data are you tracking? Drop a comment if you've built something similar — I'd love to compare approaches.

Reddit Data API 2026: After the Pricing Change, Here's What Developers Actually Use

agenthustler — Fri, 17 Apr 2026 07:42:25 +0000

Reddit Data API 2026: After the Pricing Change, Here's What Developers Actually Use

When Reddit changed its API pricing in mid-2023, a lot of indie tools died overnight. Apollo, RIF, and dozens of smaller analytics projects shut down because the new commercial pricing made any meaningful Reddit data pipeline economically impossible for small operators.

Nearly three years later, the dust has settled. Here's what developers building on Reddit data in 2026 actually use.

The Official API, Briefly

Reddit still offers a free tier for personal use (100 queries/minute, OAuth-gated), but it's narrow:

Non-commercial only.
Rate limited hard.
Any serious volume pushes you into enterprise pricing (reportedly tens of thousands per year).
Terms of service explicitly forbid AI/ML training.

For a weekend project, fine. For anything you'd put on a dashboard or ship to a client, the official API is not the answer.

The JSON Endpoint Approach

Here's the thing many devs forget: Reddit still serves public pages with a trailing .json that returns structured data with no auth required.

https://www.reddit.com/r/programming/.json
https://www.reddit.com/r/programming/comments/abc123/.json
https://www.reddit.com/user/someone/.json

You get posts, comments, scores, timestamps, flair — everything a logged-out browser sees. Rate limits exist (around 60 requests per minute per IP) and you need a real User-Agent, but it works and it's been working for years.

What Production Looks Like

For anything beyond a toy project, teams typically:

1. Rotate IPs. The per-IP limit is the real ceiling. Residential proxies solve it.
2. Back off on 429s. Reddit is consistent about returning them; honor the signal.
3. Parallelize across subreddits, not within one. Helps stay under per-subreddit throttling.
4. Cache aggressively. Most Reddit data doesn't change after the first 24 hours.

Or: Rent the Infrastructure

If you don't want to run proxy pools and handle layout changes, marketplace scrapers handle the boring parts. I maintain a Reddit Scraper that returns posts, comments, and user metadata on a pay-per-result basis — no subscription, no monthly floor.

Works for: trend monitoring, sentiment analysis, niche community research, brand mentions, competitive intelligence.

The Bigger Picture

Reddit's pricing change didn't kill Reddit data — it killed free commercial access to Reddit data. The JSON endpoints are still there. The public pages are still there. What changed is that production-grade access now involves either running your own infra or paying someone who already has.

For most developers in 2026, the math works out: pay-per-result scraping is cheaper than both the enterprise tier and the engineering hours to DIY. Check out apify.com/cryptosignals if you want to skip straight to results.

Skip the Build

Reddit Scraper on Apify

Instagram Data API in 2026: What Developers Use When the Official API Won't Work

agenthustler — Fri, 17 Apr 2026 07:42:24 +0000

Instagram Data API in 2026: What Developers Use When the Official API Won't Work

If you've ever tried to pull Instagram data for a side project, a dashboard, or a client, you already know the punchline: the official Instagram Graph API is designed for people managing their own business accounts, not for anyone doing brand monitoring, influencer research, or competitive analysis.

Here's what the Graph API actually lets you do in 2026, and what devs reach for when it falls short.

What the Graph API Gives You

Content on accounts you own or manage (via Facebook Business Manager).
Hashtag search — but capped at 30 unique hashtags per week per app.
Basic insights on your own posts: impressions, reach, engagement.
Mentions of your business account in comments/captions.

What It Does NOT Give You

Reading arbitrary public profiles (your competitor, an influencer, a brand).
Follower lists on accounts you don't own.
Post-level data on content you didn't publish.
Historical analysis across a niche.
Anything involving Stories or Reels on third-party accounts.

For 90% of real-world use cases — influencer vetting, brand sentiment tracking, market research — the Graph API is a dead end before you even submit an app review.

What Developers Actually Use

1. Public-data scrapers. Profile pages, post counts, follower counts, bios, and public post metadata are all visible to any logged-out browser. Scrapers hit that same surface and return structured JSON.

2. HTML parsing with rotation. The hard part isn't parsing — it's not getting blocked. Production scrapers run behind residential proxies, rotate user-agents, and back off on 429s.

3. Platform-managed actors. Instead of building and maintaining all that, most teams I know now rent scrapers on marketplaces like Apify, where the anti-bot infra is someone else's problem and you pay per profile returned.

I maintain an Instagram Profile Scraper there — feed it usernames, get back structured profile data (name, bio, follower count, post count, verified status, business category). Pay-per-result means you only pay for profiles that actually resolve, not for failed runs.

The Honest Trade-Off

Scrapers break. Instagram ships layout changes, adds consent walls, rotates anti-bot checks. If you build your own, expect to spend 20% of your time on maintenance. If you rent, you're trading that for per-result cost — usually worth it unless you're pulling millions of rows.

The Graph API is still the right call if you only need your own account's data. For anything outside that — research, monitoring, discovery — it's scrapers or nothing.

Check out what's available at apify.com/cryptosignals.

Skip the Build

Instagram Scraper on Apify

Pay-Per-Result APIs: Why the Industry Is Shifting Away from Monthly Subscriptions in 2026

agenthustler — Wed, 15 Apr 2026 04:43:35 +0000

Pay-Per-Result API Pricing Is Changing How Developers Buy Data

If you've ever signed up for a data API, you know the drill: pick a monthly tier, guess your usage, then either overpay for headroom you never touch or blow past quota on day 19. The pricing model is older than the web, and it's still everywhere.

A quieter shift has been happening on platforms where APIs and scrapers are sold as products: pay-per-result (PPE) billing. You don't pay for compute time, subscription tiers, or request counts. You pay per record you actually receive — a job listing, a profile, a review, a product.

Why Devs Prefer PPE

1. Cost maps to value. If a scraper returns 45,000 LinkedIn job postings, you pay for 45,000 rows. If it returns 42, you pay for 42. No idle months, no overage surprises.

2. Failure is free. Scraper broke because the target site shipped a redesign? You get zero results and pay zero. Try doing that with a $99/month subscription.

3. Budgeting is trivial. rows × unit_price is a spreadsheet cell, not a capacity-planning meeting. Finance teams love it.

4. No lock-in. Tier-based APIs punish you for leaving mid-cycle. PPE has no cycle.

Where You're Seeing This

Apify has been running a PPE model across its Actor marketplace, where individual scrapers set per-result prices (often $0.01–$0.10). You run a scraper, you get rows, you pay for rows. Billing is handled by the platform, so you're not chasing invoices.

I've been shipping actors there for a few months — LinkedIn Jobs, Indeed, IndieHackers, Instagram profile, Reddit — and what's become obvious is that PPE lets tiny niche scrapers exist at all. A dataset nobody needs enough to justify a $49/mo tier can still make sense at $0.02/result for the three people a week who want exactly that data.

Downsides, to Be Honest

Price discovery is harder. You don't know what a run will cost until you estimate row counts.
Spiky workloads can get expensive. If you need 2M rows once, a flat-tier API might be cheaper.
Per-result pricing rewards stable, well-defined schemas. Scrapers returning blobby HTML don't fit the model.

The Takeaway

If you're buying data in 2026, default to asking "what's the per-result price?" before "what's the monthly plan?" The per-unit number tells you more about the actual economics than any marketing page.

Check out the PPE actors I maintain at apify.com/cryptosignals — job market data, social profiles, review datasets. Pay for what you pull.

Powered by Apify — the web scraping platform used in this guide. Try it free →

Tracking Amazon Price Drops at Scale — Build a Price Monitor with Python

agenthustler — Tue, 14 Apr 2026 10:13:40 +0000

A friend runs a small dropshipping store. Nothing fancy — 200 SKUs, mostly home goods, sourced from Amazon and marked up on their own Shopify. His biggest headache for years was margin drift: Amazon would quietly drop a price, he'd keep selling at the old price, and by the end of the month his "20% margin" products had actually been 3% losers for weeks.

He asked me for a cheap price monitor. I built it in an afternoon. Here's the whole thing.

The idea

For each SKU, pull the current Amazon price once every few hours. If it moved more than some threshold, fire an alert. Store the history so we can look at trends and answer questions like "which products are trending down this week?"

Stack:

Apify's Amazon Scraper for the actual price pulls ($0.005/result — basically free at this scale).
SQLite for history.
A tiny Python script on a $5 VPS for scheduling.
Slack webhook for alerts.

That's it. No Redis, no queue, no Docker.

Step 1 — Watchlist

Start with a CSV. Column 1 is the ASIN, column 2 is the price you're currently selling at.

import csv

def load_watchlist(path: str = "watchlist.csv") -> dict[str, float]:
    out = {}
    with open(path) as f:
        for row in csv.reader(f):
            out[row[0].strip()] = float(row[1])
    return out

Keep it boring. You'll thank yourself later when you're debugging at 2am.

Step 2 — Fetch current prices

# Implementation is proprietary (that IS the moat).
# Skip the build — use our ready-made Apify actor:
# see the CTA below for the link (fpr=yw6md3).

One call, all ASINs. For 200 products the actor finishes in about a minute. For thousands, chunk it — I found batches of 500 to be a sweet spot.

Step 3 — Store the history

import sqlite3
from datetime import datetime

DB = sqlite3.connect("prices.db")
DB.execute("""
CREATE TABLE IF NOT EXISTS price_history (
    asin TEXT,
    price REAL,
    currency TEXT,
    in_stock INTEGER,
    checked_at TEXT
)
""")
DB.execute("CREATE INDEX IF NOT EXISTS idx_asin_time ON price_history(asin, checked_at)")

def record(items: list[dict]) -> None:
    now = datetime.utcnow().isoformat()
    for it in items:
        DB.execute(
            "INSERT INTO price_history VALUES (?, ?, ?, ?, ?)",
            (
                it["asin"],
                float(it.get("price") or 0),
                it.get("currency", "USD"),
                1 if it.get("inStock") else 0,
                now,
            ),
        )
    DB.commit()

Never delete a row. Disk is cheap, history is priceless when you're trying to figure out if a price drop is a fluke or a trend.

Step 4 — Detect drops

def significant_drops(threshold_pct: float = 5.0) -> list[tuple]:
    rows = DB.execute("""
    SELECT asin, price, checked_at FROM price_history
    ORDER BY asin, checked_at DESC
    """).fetchall()

    latest = {}
    previous = {}
    for asin, price, _ in rows:
        if asin not in latest:
            latest[asin] = price
        elif asin not in previous:
            previous[asin] = price

    alerts = []
    for asin, cur in latest.items():
        prev = previous.get(asin)
        if not prev or prev == 0:
            continue
        pct = (cur - prev) / prev * 100
        if pct <= -threshold_pct:
            alerts.append((asin, prev, cur, pct))
    return alerts

5% is a reasonable default. Below that, most of the signal is noise — Amazon nudges prices by a few cents constantly.

Step 5 — Alert

# Implementation is proprietary (that IS the moat).
# Skip the build — use our ready-made Apify actor:
# see the CTA below for the link (fpr=yw6md3).

Slack, Discord, email, SMS — it doesn't matter. What matters is that the alert lands somewhere you actually look.

Step 6 — Schedule

Cron is fine. Every four hours:

0 */4 * * * cd /opt/price-monitor && python3 run.py >> run.log 2>&1

run.py is the three-line composition of everything above:

from monitor import load_watchlist, fetch_prices, record, significant_drops, alert

items = fetch_prices(list(load_watchlist().keys()))
record(items)
alert(significant_drops(threshold_pct=5))

Use cases beyond dropshipping

Once the pipeline is running, other uses appear naturally:

Comparison shopping. Same ASIN across multiple country marketplaces — find arbitrage.
Restock alerts. Flip the in_stock flag check instead of price.
Deal blogs. If you run a content site, price drops become content. Low effort, good traffic.
Personal wishlists. The nerdiest one. I track 20 board games and get pinged when anything drops 10%+.

Cost check

At 200 products every 4 hours, that's 1,200 results per day × 30 days = 36,000 results/month. At $0.005/result the bill is $180/month — and my friend's monthly margin drift savings alone paid for it in the first week. For smaller lists (20–50 products) you're talking pennies.

If you want the actor: Amazon Scraper on Apify. It has a free tier so you can prove out the idea before wiring up cron.

The boring stack works. Go build.

Powered by Apify — the web scraping platform used in this guide. Try it free →

How to Build a LinkedIn Talent Pipeline Scraper in 2026 (Without LinkedIn's API)

agenthustler — Tue, 14 Apr 2026 10:13:39 +0000

I spent the last two months helping a friend's recruiting agency move off a $4,000/month sourcing tool. The pitch was simple: they wanted to pull a few thousand LinkedIn profiles a week based on job titles, enrich them, score them, and feed the top matches into their ATS. LinkedIn's official API, as anyone who has tried it knows, is basically a locked door unless you're a Fortune 500 partner. So we went the scraper route — and it worked better than either of us expected.

Here's how I built it, what I learned, and the Python code you can steal.

Why not the official API?

LinkedIn's partner API (Talent Solutions) is gated behind sales calls, contracts, and minimums you don't want to see. For a small agency or a solo recruiter, it's not an option. The Sign In With LinkedIn OAuth endpoints only give you basic profile info for the user who logged in — not search, not lookups, not bulk data.

Everyone I know who scales LinkedIn sourcing does one of two things:

Pays a SaaS that scrapes on their behalf and wraps it in a UI.
Runs their own scraper.

Option 2 is cheaper, more flexible, and the data is yours to do with as you please.

The stack

I used Apify's LinkedIn Profile Scraper as the data layer. It handles the proxy rotation, fingerprinting, and retry logic that you really don't want to maintain yourself. I built the pipeline in plain Python — no framework, just requests, sqlite3, and a couple of CSV dumps.

Pricing was the other reason I went with this actor: $0.005 per result. For 5,000 profiles a week that's $25, which is absolutely nothing compared to the $4k/month the agency was paying before.

Step 1 — Pull the profiles

Let's say we want Senior Python Engineers in Berlin. I keep my seed list in a plain text file: one profile URL per line. You can generate that seed list from a LinkedIn search URL using the same actor, but for this article I'll assume you already have URLs.

# Implementation is proprietary (that IS the moat).
# Skip the build — use our ready-made Apify actor:
# see the CTA below for the link (fpr=yw6md3).

run-sync-get-dataset-items blocks until the actor finishes and returns the dataset in one shot. For small batches that's the easiest pattern. For 5,000+ profiles, switch to the async runs endpoint and poll.

Step 2 — Store and deduplicate

Recruiters re-run the same searches every week. You do not want to re-scrape profiles you already have fresh data on, both for cost and politeness.

import sqlite3
from datetime import datetime, timedelta

DB = sqlite3.connect("talent.db")
DB.execute("""
CREATE TABLE IF NOT EXISTS profiles (
    url TEXT PRIMARY KEY,
    name TEXT,
    headline TEXT,
    location TEXT,
    skills TEXT,
    last_seen TEXT
)
""")

def needs_refresh(url: str, max_age_days: int = 14) -> bool:
    row = DB.execute(
        "SELECT last_seen FROM profiles WHERE url = ?", (url,)
    ).fetchone()
    if not row:
        return True
    last = datetime.fromisoformat(row[0])
    return datetime.utcnow() - last > timedelta(days=max_age_days)

def upsert(profile: dict) -> None:
    DB.execute("""
    INSERT INTO profiles(url, name, headline, location, skills, last_seen)
    VALUES (?, ?, ?, ?, ?, ?)
    ON CONFLICT(url) DO UPDATE SET
        name=excluded.name,
        headline=excluded.headline,
        location=excluded.location,
        skills=excluded.skills,
        last_seen=excluded.last_seen
    """, (
        profile["url"],
        profile.get("fullName"),
        profile.get("headline"),
        profile.get("location"),
        ",".join(profile.get("skills", [])),
        datetime.utcnow().isoformat(),
    ))
    DB.commit()

A 14-day TTL works well in practice. People don't update their headline every week.

Step 3 — Score candidates

This is where a pipeline stops being a scraper and starts being a tool. The agency cared about three signals: years in role, relevance of past companies, and whether the person was open to contract work (mentioned in the headline or about section).

def score(profile: dict) -> int:
    points = 0
    headline = (profile.get("headline") or "").lower()
    if "senior" in headline or "staff" in headline:
        points += 2
    if "open to" in headline or "contract" in headline:
        points += 3

    for exp in profile.get("experience", []):
        title = (exp.get("title") or "").lower()
        if "python" in title:
            points += 2
        if exp.get("company") in TARGET_COMPANIES:
            points += 5
    return points

Tune TARGET_COMPANIES to your industry. The agency keeps a list of ~60 companies whose alumni they love to source from; hitting one is a huge signal.

Step 4 — Ship to the ATS

Once you've ranked profiles, push the top N into wherever your team actually works. For the agency that meant a CSV drop into a shared folder, but a webhook into Greenhouse or Airtable works just as well.

import csv

def export_top(n: int = 50, path: str = "top_candidates.csv") -> None:
    rows = DB.execute(
        "SELECT url, name, headline, location FROM profiles ORDER BY last_seen DESC LIMIT ?",
        (n,),
    ).fetchall()
    with open(path, "w", newline="") as f:
        w = csv.writer(f)
        w.writerow(["url", "name", "headline", "location"])
        w.writerows(rows)

What I'd do differently

Two things bit me in the first month:

Don't scrape the same profile twice in one day. Even with proxy rotation, you're wasting money. The dedupe check above exists for a reason.
Rate yourself, not just the scraper. The actor handles its side. You should still cap your runs — I schedule one batch of 500 profiles every few hours rather than 5,000 in one shot. Smoother results, easier debugging.

Total cost

At $0.005/result, the agency's weekly 5,000-profile refresh costs $25. Add a few dollars for compute. Compared to the SaaS they cancelled, it pays for a nice dinner every week and then some.

If you want to skip the code and just try the actor, it's here: LinkedIn Profile Scraper on Apify. The input schema is documented and the free tier lets you run a few hundred profiles before you put a card down.

Happy sourcing.

Powered by Apify — the web scraping platform used in this guide. Try it free →

I scraped 4,500 IndieHackers products - here's what the MRR data reveals

agenthustler — Tue, 14 Apr 2026 03:02:33 +0000

After spending a weekend building a scraper for IndieHackers' public product pages, I ended up with a dataset of 4,500 products — and the MRR numbers tell a very different story than the Twitter highlight reel.

I went in expecting to see a long tail of hobby projects and a handful of unicorns. What I actually found was a surprisingly healthy middle class of indie SaaS.

Here's what the data looks like after a few hours with pandas.

The numbers

Out of 4,500 products scraped, 1,544 (34%) self-report MRR on their profile. The rest either keep it private, haven't updated in forever, or are pre-revenue.

The MRR distribution for the reporters:

Bucket	Products	Share
$0 – $100	476	31%
$100 – $1K	447	29%
$1K – $10K	401	26%
$10K+	220	14%

The takeaway that surprised me: the $1K–$10K bracket is where most "successful" indie products actually live. Not the $100K MRR screenshots that go viral. A product doing $3,500/mo is doing better than ~85% of everything publicly listed.

Median MRR among revenue reporters: ~$750/mo. Not glamorous, but it's a real number, and it puts the "just quit your job" advice in perspective.

What's growing

Tagging each product by its stated category and doing a rough vertical split:

SaaS tools — still the biggest slice
Developer tools — overrepresented vs the broader market (no surprise given the audience)
AI / automation — the fastest-growing category in the listings; dozens of products added in the last 90 days alone
Productivity — steady but saturated
Marketing / growth — lots of entries, but most sit in the $0–$100 bucket

The AI bucket is the one worth watching. A year ago it was a sliver. Now it's pushing ~18% of new listings.

Who's behind them

Looking at the team size field:

Solo founders dominate the $1K–$10K bracket (62% of products in that range list a team of 1)
Products with 2+ founders skew toward the $10K+ bracket
Nobody in the $100K+ club is solo — every one has a team

That matches the intuition that revenue-per-founder has a ceiling, but the data is cleaner than I expected.

Filtering the data in Python

If you want to isolate, say, profitable AI products in the dataset, it's a one-liner with pandas:

import pandas as pd

df = pd.read_csv("indiehackers_products.csv")

ai_winners = df[
    (df["mrr"] > 1000) &
    (df["tags"].str.contains("ai", case=False, na=False))
]

print(ai_winners[["name", "mrr", "tags", "url"]].sort_values("mrr", ascending=False).head(20))

From there you can pivot on launch date, team size, or pricing model to find the patterns you care about.

How I'm using this

A few angles that turned out to be useful:

Competitor analysis — pick a vertical, sort by MRR, and you instantly see who the players are and roughly what they're pulling in.
Niche validation — if a category has 40 products but none above $500/mo, that's a signal (probably a bad one).
Founder outreach — if you sell to indie founders, this is basically a lead list with qualification data attached.
Pricing benchmarks — cross-referencing MRR bracket with listed price gives you a rough sense of how many paying customers each product has.

The dataset

I cleaned the full 4,500-row CSV up and put it on Payhip if you'd rather skip the scraping step: IndieHackers Products Dataset

Includes product name, URL, tags, MRR (where public), team size, launch date, and short description. Single CSV, no tracking, no subscription.

If you end up finding something interesting in the data, I'd genuinely like to hear about it — leave a comment and I'll follow up.

Skip the Build

Twitter/X Scraper on Apify

Web Scraping Pricing Comparison 2026: Bright Data vs Apify vs DIY

agenthustler — Sun, 12 Apr 2026 11:05:30 +0000

Web scraping costs range from $0 (DIY) to $10,000+/month (enterprise proxy networks). The right choice depends on your scale, technical resources, and how much maintenance you're willing to absorb.

This is a practical pricing comparison of the major web scraping approaches in 2026, with real numbers and clear recommendations for different use cases.

The Five Approaches

Every web scraping project falls into one of these categories:

Proxy providers (Bright Data, Oxylabs) — you write the scraper, they provide infrastructure
Managed platforms (Apify) — pre-built scrapers with compute and proxy included
Scraping APIs (ScraperAPI, ScrapingBee) — API call returns rendered HTML
Data APIs (Proxycurl, RapidAPI vendors) — API call returns structured data
DIY — your own servers, your own proxies, your own maintenance

Pricing Comparison Table

Provider	Model	Entry Price	Mid-Scale (100K pages/mo)	Large Scale (1M pages/mo)	Includes Proxies	Anti-Bot Handling
Bright Data	Per GB / per CPM	$2.80/GB residential	~$280-500/mo	~$2,000-4,000/mo	Yes	Yes (Web Unlocker)
Oxylabs	Per GB / per request	$3.00/GB residential	~$300-550/mo	~$2,200-4,500/mo	Yes	Yes (Web Unblocker)
Apify Platform	Per compute unit	$0.30/CU	~$25-80/mo	~$200-600/mo	Bundled in actors	Actor-dependent
Apify PPE Actors	Per result	$0.005-0.01/result	~$50-100/mo (10K results)	~$500-1,000/mo (100K results)	Included	Included
ScraperAPI	Per API call	$0.001/call (on plan)	~$100/mo	~$500/mo	Yes	Yes
ScrapingBee	Per credit	$0.0025/credit	~$100/mo	~$400-800/mo	Yes	Yes
DIY (VPS + free proxies)	Server costs	$5-20/mo VPS	~$20-50/mo	~$100-300/mo + your time	No (you manage)	No (you build)
DIY (VPS + paid proxies)	Server + proxy	$50+/mo	~$150-400/mo	~$1,000-3,000/mo	Purchased separately	You build

Prices as of April 2026. Actual costs vary by target site complexity and anti-bot measures.

Detailed Breakdown

Bright Data

Best for: Large-scale operations that need residential/mobile IPs and can write their own scrapers.

Bright Data is the largest proxy network (72M+ residential IPs). Their pricing model is primarily per-GB for proxy traffic, with Web Unlocker (anti-bot) charged per CPM (cost per thousand requests).

Residential proxies: $2.80/GB
Datacenter proxies: $0.60/GB
Web Unlocker: $2.50/CPM (per 1,000 successful requests)
Minimum commitment on some plans

Hidden costs: You still need to write and maintain the scraper code, handle parsing, manage retries, and deal with site changes. Developer time is the real cost.

Oxylabs

Similar to Bright Data in pricing and capability. Residential at $3.00/GB, datacenter at $0.70/GB. Their Web Unblocker competes directly with Bright Data's Web Unlocker.

Apify Platform

Best for: Teams that want pre-built scrapers without infrastructure management.

Apify's pricing is based on compute units (CUs). One CU = 1 GB RAM for 1 hour of compute. At $0.30/CU, the actual cost depends on the actor (scraper) efficiency.

Free tier includes $5/mo in platform credits — enough for small experiments.

Many Apify Store actors use pay-per-event (PPE) pricing, where you pay per result instead of per compute unit. This is simpler to predict:

LinkedIn Jobs scraper: $0.005-0.01 per job listing
Google Search scraper: $0.003-0.005 per result
E-commerce scrapers: $0.005-0.02 per product

PPE pricing includes proxy costs and anti-bot handling — no hidden fees.

ScraperAPI

Best for: Developers who want a simple API call that returns rendered HTML.

You send a URL, ScraperAPI returns the HTML after handling proxies, CAPTCHAs, and JavaScript rendering.

Hobby: $29/mo (100K API credits)
Startup: $99/mo (1M API credits)
Business: $249/mo (3M API credits)
1 API credit = 1 standard request; JavaScript rendering costs 5-10 credits

Trade-off: You still need to parse the HTML yourself. Good for simple pages, expensive for JavaScript-heavy sites.

ScrapingBee

Similar model to ScraperAPI. Pricing starts at $49/mo for 150K credits. JavaScript rendering costs 5 credits per request. Includes Google Search API at 25 credits per search.

DIY

Best for: Technical teams with specific requirements and tolerance for maintenance.

The upfront cost is low — a $10/mo VPS, Playwright or Puppeteer, and free rotating proxies (if you can find reliable ones). But the real costs are:

Developer time: 10-40 hours/month maintaining scrapers against site changes
Proxy costs: Free proxies are unreliable; paid residential proxies cost $2-5/GB
Anti-bot solutions: reCAPTCHA solving services ($1-3/1000 solves), browser fingerprint management
Infrastructure: Error handling, retry logic, job queues, data storage

Realistic all-in cost for a maintained DIY scraper at scale: $500-3,000/month including developer time (valued at $50/hr).

Decision Matrix

Use Case	Recommended	Why	Monthly Cost
Hobby/learning	DIY or Apify free tier	Learn the fundamentals, free or near-free	$0-5
Startup MVP	Apify PPE actors	Pay per result, no infrastructure, predictable costs	$10-100
Recruiting/HR data	Apify PPE (LinkedIn actors)	Purpose-built, handles LinkedIn anti-bot	$25-200
SEO/marketing data	ScraperAPI or Apify	Both handle Google well, choose by volume	$29-250
E-commerce monitoring	Bright Data or Apify	Bright Data for custom; Apify for pre-built	$100-1,000
Enterprise (1M+ pages)	Bright Data + custom scrapers	Best proxy network at scale, but need dev team	$2,000-10,000
One-time data collection	Apify PPE	No subscription, pay only for what you use	$5-50

What Changed in 2026

Several things shifted since 2024-2025:

Proxycurl shut down — The popular LinkedIn/company data API closed in 2025. Alternatives: Apify actors, Bright Data datasets, or DIY.
Anti-bot got harder — Cloudflare Turnstile, DataDome, and PerimeterX are on more sites. DIY scraping is increasingly expensive to maintain.
Pay-per-result pricing expanded — More platforms offer PPE/pay-per-result models, which shifts risk from the buyer to the provider.
AI assistants drive data demand — LLM-powered agents need structured data feeds, creating new demand for scraping infrastructure.

How to Calculate Your Actual Cost

Before choosing a provider, estimate your real needs:

# Implementation is proprietary (that IS the moat).
# Skip the build — use our ready-made Apify actor:
# see the CTA below for the link (fpr=yw6md3).

Bottom Line

For most developers and startups: Start with Apify's free tier or a PPE actor. You get structured data without managing proxies or infrastructure, and costs scale linearly with usage.

For scale (500K+ pages/month): Bright Data or Oxylabs give you the best proxy infrastructure, but budget for developer time to build and maintain scrapers.

For one-off projects: Pay-per-result pricing (Apify PPE actors) is almost always cheaper than a monthly subscription to any platform.

The biggest hidden cost in web scraping isn't the proxy bill — it's the engineering time to keep scrapers working as sites change. Factor that in before choosing DIY over a managed solution.

Prices verified as of April 2026. For the latest pricing, check each provider's website directly.

Powered by Apify — the web scraping platform used in this guide. Try it free →

LinkedIn Jobs API for APAC Recruitment: Singapore, Indonesia, Malaysia Data Guide

agenthustler — Sun, 12 Apr 2026 11:04:07 +0000

If you're building recruitment tools, HR dashboards, or talent analytics for Southeast Asia, you already know: APAC hiring data is fundamentally different from Western markets.

LinkedIn is the dominant professional network in Singapore, Malaysia, and Indonesia — but accessing structured job data programmatically requires navigating API restrictions, regional data quirks, and multilingual listings.

This guide covers how to get clean, structured LinkedIn Jobs data for APAC recruitment, what fields matter, and the current options in 2026.

Why APAC Recruitment Data Is Different

Three things make APAC job data harder to work with than US/EU data:

1. Multilingual listings. A single Singapore job posting might mix English, Mandarin, and Malay. Indonesian listings blend Bahasa Indonesia with English technical terms. Your pipeline needs to handle this.

2. Regional job board fragmentation. LinkedIn dominates white-collar hiring, but JobStreet (Malaysia/Indonesia), MyCareersFuture (Singapore government portal), and Jobsdb (Hong Kong/Thailand) all carry listings that never appear on LinkedIn.

3. Visa and work permit requirements. Singapore's Employment Pass, Malaysia's DE Lex pass, Indonesia's KITAS — these aren't just nice-to-have fields. They determine whether a candidate can actually apply.

Factor	US/EU	APAC (SG/MY/ID)
Primary platform	LinkedIn	LinkedIn + JobStreet + government portals
Language	Single language	2-4 languages per listing
Work authorization	Binary (yes/no)	Pass-type specific (EP, S Pass, PEP, DP)
Salary disclosure	Common	Rare (salary ranges often hidden)
Hiring timeline	2-4 weeks	4-8 weeks (notice periods are longer)

Key Data Fields for APAC Recruitment

When scraping LinkedIn Jobs for APAC markets, these fields matter most:

job_location — Filter by city-level: "Singapore", "Kuala Lumpur", "Jakarta", "Bangsar South" (common KL tech hub)
description (full text) — Parse for visa sponsorship mentions, language requirements, and salary hints
seniority_level — APAC markets skew toward mid-senior roles on LinkedIn; junior roles are more common on local boards
company_size — Startups in Singapore (especially fintech) hire differently than MNCs
posted_date — Critical for tracking hiring velocity and seasonal patterns (Chinese New Year creates a predictable February dip)

How to Filter for Singapore, KL, and Jakarta

Using the LinkedIn Jobs Scraper on Apify, you can target APAC locations directly:

{
  "searchUrl": "https://www.linkedin.com/jobs/search/?location=Singapore&keywords=software+engineer",
  "maxItems": 100
}

For multi-city collection, run separate queries per location:

locations = [
    "Singapore",
    "Kuala Lumpur, Federal Territory of Kuala Lumpur, Malaysia",
    "Jakarta, Jakarta, Indonesia",
    "Ho Chi Minh City, Vietnam"
]

for loc in locations:
    run_input = {
        "searchUrl": f"https://www.linkedin.com/jobs/search/?location={loc}&keywords=fintech",
        "maxItems": 200
    }

Tip: Use full location strings for Malaysia and Indonesia. "Kuala Lumpur" alone sometimes returns results from other countries.

Practical Example: Tracking Singapore Fintech Hiring

Singapore is the fintech capital of Southeast Asia. Here is how to track hiring trends:

Collect weekly snapshots of fintech job listings (keywords: "fintech", "digital bank", "payments", "blockchain")
Track volume over time — a spike in backend engineer listings often precedes a product launch
Monitor company-level patterns — are Grab, Sea Group, and DBS scaling specific teams?
Cross-reference with MAS announcements — new regulatory frameworks (like the 2025 Digital Payment Token rules) create predictable hiring waves

A dataset of 500-1000 Singapore fintech listings per month costs roughly $2.50-5.00 using pay-per-event pricing on Apify.

Comparison: LinkedIn Jobs Data Sources in 2026

Source	Status	APAC Coverage	Cost	Structured Data
LinkedIn Official API	Restricted (partners only)	Full	Free (if approved)	Yes
Proxycurl	Shut down (2025)	N/A	N/A	N/A
PhantomBuster	Active	Limited geo-targeting	$69+/mo	Partial
Apify LinkedIn Jobs Scraper	Active	Full city-level filtering	$0.005-0.01/result (PPE)	Yes (JSON)
DIY with Playwright	Active	Whatever you build	Server costs + maintenance	Whatever you build

The LinkedIn Official API requires a partnership agreement and is not available to most developers or startups. If you have access, use it — the data quality is unmatched.

For everyone else, a managed scraper with pay-per-result pricing is the most practical option. You get structured JSON output without maintaining proxy infrastructure or handling LinkedIn's anti-bot measures.

Getting Started

Create a free Apify account
Find the LinkedIn Jobs Scraper in the Store
Set your target location and keywords
Run and export to JSON, CSV, or connect directly to your pipeline via API

For APAC-specific recruitment analytics, combine LinkedIn data with local job boards for the most complete picture. LinkedIn captures 60-70% of white-collar listings in Singapore, but only 30-40% in Indonesia where JobStreet dominates.

Building recruitment tools for APAC? The structured data approach beats manual scraping every time — especially when you need consistent, repeatable data collection across multiple Southeast Asian markets.

Powered by Apify — the web scraping platform used in this guide. Try it free →

How to Scrape PropertyGuru Data in Singapore: 2026 Guide

agenthustler — Sun, 12 Apr 2026 08:17:53 +0000

Singapore's property market moves fast. Whether you're an investor tracking price trends across districts, a real estate agent doing competitive research, or a developer building market analysis tools — you need structured property data.

PropertyGuru is Singapore's dominant property portal, with over 200,000 active listings covering HDB flats, condos, landed properties, and commercial spaces. It's the Zillow of Southeast Asia, and it holds a goldmine of data that's frustratingly hard to extract at scale.

What Data Is Available on PropertyGuru?

Each PropertyGuru listing contains rich structured data:

Price — asking price, PSF (per square foot), and historical price changes
Property details — size (sqft), bedrooms, bathrooms, floor level, tenure (freehold/leasehold/999-year)
Location — district number, street address, nearest MRT station and distance
Agent info — agent name, agency, contact details, number of active listings
Project details — developer name, TOP date, total units, facilities

For Singapore specifically, you'll want to understand the geographic classification:

CCR (Core Central Region) — Districts 9, 10, 11, Downtown Core. Orchard Road, Marina Bay. Premium pricing.
RCR (Rest of Central Region) — Districts like Queenstown, Toa Payoh, Geylang. Mid-tier pricing with growth potential.
OCR (Outside Central Region) — Jurong, Woodlands, Punggol. Mass market, HDB-heavy. This is where 80% of Singaporeans live.
URA Planning Areas — 55 distinct zones used by the Urban Redevelopment Authority for planning. Critical for understanding development potential and future MRT lines.

Understanding these regions is essential for any meaningful property analysis in Singapore.

Why People Need This Data

Property investors track PSF trends across districts to identify undervalued areas before en-bloc fever hits. When a new MRT line is announced, prices in surrounding districts shift — having historical data lets you model these patterns.

Real estate agents monitor competitor listings, pricing strategies, and agent market share. If a rival agency is dominating District 15 listings, you want to know.

PropTech developers build tools like mortgage calculators, investment dashboards, and rental yield estimators. All of these need fresh listing data as input.

Researchers and analysts study housing affordability, the HDB resale market, and the impact of cooling measures on transaction volumes.

Current Options (And Why They Fall Short)

Manual export: PropertyGuru lets you browse and filter, but there's no bulk export. Copy-pasting 50 listings is tedious. Doing 5,000 is impossible.

Existing Apify actors: As of early 2026, the PropertyGuru scrapers on Apify Store are deprecated and returning 0 results. The site has changed its structure, and nobody has updated these actors.

Enterprise solutions: Services like Bright Data offer pre-built datasets, but pricing starts at enterprise levels — overkill if you just need listing data for a specific district or property type.

Building Your Own PropertyGuru Scraper

Here's a basic approach using Python and requests:

# Implementation is proprietary (that IS the moat).
# Skip the build — use our ready-made Apify actor:
# see the CTA below for the link (fpr=yw6md3).

Important notes:

Always check PropertyGuru's robots.txt and terms of service
Use reasonable rate limiting (2+ seconds between requests)
Selectors change frequently — you'll need to maintain your scraper
Consider using Playwright or Selenium if the site relies heavily on JavaScript rendering

Using Apify for Custom Tasks

If you don't want to maintain your own scraper infrastructure, you can build a custom Apify actor. Apify handles proxy rotation, scheduling, and storage — you just write the scraping logic.

For complementary location data, the Google Maps Scraper on Apify can help you enrich property listings with nearby amenities — schools, MRT stations, hawker centres, malls, and clinics. Proximity to amenities is one of the biggest price drivers in Singapore real estate.

For example, combine property listings with Google Maps data to answer: "Which District 19 condos are within 500m of an MRT station AND have 3+ schools nearby?"

Practical Tips for Singapore Property Data

HDB vs Private — These are fundamentally different markets. HDB resale is regulated by HDB with transaction data publicly available at data.gov.sg. Private property (condos, landed) is where scraped data adds the most value.
New launches vs Resale — New launch pricing comes from developers and is often only on PropertyGuru temporarily. Resale listings stay longer and have richer data.
PSF is king — Price per square foot is how Singaporeans compare properties. Always normalize by size.
MRT proximity matters enormously — Properties within 500m of an MRT station command a 10-15% premium. The Thomson-East Coast Line (TEL) completions through 2025-2026 are reshaping values in Districts 15 and 18.
Check URA caveats — Some listings are in areas zoned for future development or have plot ratio restrictions. Cross-reference with URA Master Plan data.

What's Next

The gap in the market for a reliable, maintained PropertyGuru scraper is real. The existing solutions are broken, and Singapore's property data needs are growing as more investors and PropTech startups enter the market.

If you build something useful, consider publishing it on the Apify Store — there's clear demand from the Singapore market, and the existing actors aren't delivering.

Have questions about scraping property data in Singapore? Drop a comment below.

Powered by Apify — the web scraping platform used in this guide. Try it free →