Forem: Sami

Weibo's Hot Search Is the Best Real-Time Feed of Chinese Public Sentiment in 2026

Sami — Fri, 08 May 2026 17:33:19 +0000

Weibo's "hot search" (热搜) is the closest thing China has to a real-time barometer of public attention. It updates every few minutes, ranks topics by an opaque heat score, and is where every news cycle, celebrity scandal, and viral product launch lands first. For brands, agencies, and researchers covering China, this feed is gold — and unlike most of Weibo, it's accessible without a single cookie.

This post is for anyone building a brand-monitoring, sentiment-tracking, or trend-discovery pipeline aimed at China.

Why hot search matters

Weibo (微博) is China's microblogging giant — 580M+ monthly active users. The hot search ranking is curated by Weibo's own engagement signals: a topic earns a spot when search volume, post creation, and engagement spike together within a short window.

That makes hot search a leading indicator for:

PR crises: a brand mention reaches the top 50 within minutes of a viral video
Product launches: launches by Apple, Tesla, Xiaomi, etc. typically hit the top 20 within an hour
Cultural shifts: holiday spikes, generational slang, viral memes
Geopolitics: state-affiliated topics surface predictably; their ranking velocity tells a story

If you're tracking China for any of these use cases, polling hot search every 5–15 minutes gives you sub-news-cycle response time.

What you actually get

Each hot search row exposes:

rank (1–50)
title (the search term itself, in Chinese)
hotValue — an integer that approximates topical heat
category (科技 = tech, 娱乐 = entertainment, 时尚 = fashion, etc.)
labelName — content-moderation labels: 热 (hot), 新 (new), 沸 (boiling), 爆 (exploding)
isHot flag
url to the search results page on weibo.com

Sample row:

{
    "rank": 1,
    "title": "人工智能最新突破",
    "category": "科技",
    "hotValue": 2847562,
    "labelName": "热",
    "isHot": true,
    "url": "https://s.weibo.com/weibo?q=%23..."
}

A minimal Python pipeline

import time
from datetime import datetime
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

def snapshot_hot_search():
    run = client.actor("zhorex/weibo-scraper").call(run_input={
        "mode": "hot_search",
        "maxResults": 50,
    })
    return list(client.dataset(run["defaultDatasetId"]).iterate_items())

# Poll every 10 minutes and dedupe by title
seen = {}
while True:
    snap = snapshot_hot_search()
    ts = datetime.utcnow().isoformat()
    for row in snap:
        title = row["title"]
        if title not in seen or seen[title]["rank"] != row["rank"]:
            seen[title] = {"rank": row["rank"], "first_seen": ts}
            print(f"[{ts}] rank={row['rank']:>2}  {title}  heat={row['hotValue']}")
    time.sleep(600)

A small loop and you've built a brand-mention monitor.

Common patterns I see customers run

1. Brand watch. Match new hot-search titles against a list of brand keywords. Trigger alerts when a brand name enters top 50.

2. Velocity tracking. Compute the rank-change velocity per topic. Topics that jump from rank 40 → 5 in under 30 minutes are early-warning signals for going viral.

3. Category drift. Track which categories dominate hot search hour-by-hour. Useful for media planning and ad targeting timing.

4. Cross-platform correlation. Pair Weibo hot search with Bilibili trending and RedNote search to detect cross-platform memes early. The platforms are surprisingly correlated 1–6 hours apart.

Going deeper: posts and comments

Hot search gives you topics. To go deeper into actual conversation, pivot from a hot title to its underlying posts:

# After identifying a hot topic, search posts about it
posts_run = client.actor("zhorex/weibo-scraper").call(run_input={
    "mode": "search",
    "searchQuery": "人工智能最新突破",
    "maxResults": 100,
})

That returns post-level data: text, author, like/repost/comment counts, embedded images, and post URLs. Pair with mode: post_comments to harvest reactions.

Why a hosted scraper, not raw scraping

Weibo's public web endpoints work without login for most read paths, but they require a visitor session token (Sina Visitor System) and exponential backoff on throttling responses. A naive requests script will either get throttled within 100 calls or pull empty arrays without realizing.

The Weibo Scraper on Apify handles session bootstrap, throttling, retries, and consistent schema across modes (hot_search, post_comments, search, user_posts). Pure HTTP — no browser, no proxy required.

Pricing is pay-per-event: $0.005 per item. 1,000 items = $5. The free Apify tier covers 1,000 items/month.

FAQ

Is hot search censored? Some topics are rate-limited or removed by Weibo's moderation. The labelName field hints at moderation state. You'll see topics appear and disappear.

Can I get historical hot search? Not via Weibo directly — they don't expose archives. You build your own archive by snapshotting at intervals.

What about session tokens? They expire periodically. Hosted scrapers refresh them automatically; if you DIY, plan for re-auth.

Is scraping Weibo legal? This accesses publicly visible data. No authentication is bypassed. Always check your local laws and Weibo's ToS.

Building a Chinese intelligence stack?

I maintain the full suite for production pipelines:

Weibo Scraper — (this one)
Bilibili Scraper — China's YouTube, 300M MAU
RedNote (Xiaohongshu) Scraper — lifestyle social
RedNote Shop Scraper — Xiaohongshu e-commerce

Running 50K+ items per month? I offer custom output schemas, dedicated proxy pools, SLA, and volume pricing. DM me on Apify or open an Issue titled "Enterprise inquiry".

Found a bug? Open an Issue and I usually ship fixes within 48 hours.

A 30-second review on the Apify Store helps other users find this tool. ⭐

Building a Xiaohongshu (RedNote) E-commerce Scraper for RedShop Product Data

Sami — Wed, 06 May 2026 01:56:00 +0000

When Xiaohongshu (RedNote / Little Red Book / 小红书) launched RedShop — its US-facing e-commerce platform — in April 2026, I noticed every existing scraper on Apify only covered the social side: posts, profiles, comments, videos. None of them touched product listings, vendor catalogs, or pricing data.

So I built one.

Why a dedicated shop scraper?

Xiaohongshu is unusual among Chinese platforms because product listings live in a separate URL space from social posts. The all-in-one social scrapers handle the /explore/ posts surface. RedShop products live behind /goods-detail/ with completely different structure.

Trying to extract product data from a "social" scraper means hacky workarounds. A dedicated commerce-focused tool gives you:

Structured product fields (price, sold count, SKU variants, vendor metadata)
Native support for vendor/store browsing
Cross-border vs domestic flagging
Cleaner pricing model: charge per product, not per "result"

What it extracts

For each product:

itemId, title, productUrl
salePrice, originalPrice, discountPct, currency (CNY for domestic, USD for cross-border)
soldCount, wantCount (popularity signals)
cover, images
vendor (sellerId, name, rating)
category path
skus (variants with prices and stock)
crossBorder flag and shippingOrigin

Three modes

Mode	What it does
`product_search`	Search products by keyword, sort by price/sales, filter by price range
`vendor_products`	Full catalog from a specific seller
`product_detail`	Deep dive on specific product URLs (full SKU breakdown)

Real-world use cases

DTC brands: monitor your own listings and competitor pricing on China's #1 social commerce platform
Dropshippers and resellers: discover trending Chinese products before they hit Amazon or Etsy
Cross-border arbitrage: identify SKUs popular in China that haven't reached Western markets yet
Investment analysts: track e-commerce activity for Chinese consumer brands
Sourcing agents: scout Chinese products at scale for clients in cosmetics, fashion, or home goods

Combined with the RedNote All-in-One Scraper (social side), you can map products to the influencers tagging them — extremely valuable for influencer-product correlation studies.

How to use it

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("zhorex/rednote-shop-scraper").call(run_input={
    "mode": "product_search",
    "searchQuery": "skincare",
    "maxResults": 100,
    "sortBy": "sales",
    "minPrice": 50,
    "maxPrice": 500,
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"{item['title']} — ¥{item['salePrice']} (sold {item['soldCount']})")

Output sample

{
    "itemId": "642a1b3c0000000023019f7e",
    "title": "Skincare Set - Hydrating Toner + Serum + Moisturizer",
    "salePrice": 199.00,
    "originalPrice": 299.00,
    "discountPct": 33.4,
    "currency": "CNY",
    "soldCount": 12500,
    "wantCount": 5400,
    "vendor": {"sellerId": "...", "name": "BeautyBrand Official", "rating": 4.8},
    "category": "Beauty / Skincare / Sets",
    "skus": [{"spec": "Normal Skin", "price": 199.00, "stock": 1200}],
    "crossBorder": false,
    "shippingOrigin": "China"
}

Pricing

Pay-per-event:

$0.0075 per product scraped
$0.025 per vendor info record

Typical costs:

Search 100 products: ~$0.75
Full vendor catalog (200 products): ~$1.53
5 competitor vendors with 100 products each: ~$3.88

FAQ

Does it work for cross-border products?
Yes — products are explicitly flagged in the output (crossBorder: true/false) so you can filter domestic vs international listings.

Can I track price changes over time?
Schedule the actor to run daily/weekly via Apify Schedules. The dataset versioning gives you a price history for any product or vendor.

Does it need a proxy?
Residential proxies are recommended for reliable results. The default config uses Apify's residential pool.

Is there an official Xiaohongshu shop API?
No — Xiaohongshu doesn't offer a commerce API for international developers. This actor is the practical alternative.

Try it

RedNote Shop Scraper on Apify — works with Apify's free plan ($5/month credits cover hundreds of products at no cost).

If you build something useful with it, drop a comment — always interested in seeing how people use commerce data.

Google Ads can spend up to 2x your daily budget. I built a Chrome extension that catches it before it happens.

Sami — Thu, 30 Apr 2026 15:17:28 +0000

If you've ever opened Google Ads and noticed your campaign spent way more than the daily budget you set, you're not imagining it. Google's documentation explicitly says they may spend up to twice your daily budget on any given day, evening it out across the month. That's not a bug — it's how their pacing engine has always worked.

What changed in March 2026: Google now aggressively targets 100% of your monthly limit — which is 30.4× your daily budget. Even with ad scheduling. So if your campaigns only run 22 days a month (weekdays only, for example), Google can push up to 38% more spend per active day than you'd expect from your daily budget setting.

Most PPC managers don't notice until the damage is done. The Campaigns tab in Google Ads doesn't tell you whether you're on pace or headed for overspend. You'd need a spreadsheet, a calendar, and a calculator open in another window — or a SaaS tool that costs $49 to $749 per month.

I got tired of the spreadsheet route. So I built a Chrome extension that does it inside Google Ads, in real time, for free up to 3 campaigns. Walking through the build because the technical approach is interesting and the pricing math vs SaaS tools is genuinely lopsided.

What budget pacing actually requires

The math is simple. For each campaign:

expected_spend_today = daily_budget × (days_elapsed_in_month / total_days_in_month)
pacing_ratio = actual_spend_today / expected_spend_today

# pacing_ratio < 1.10 → on pace
# pacing_ratio 1.10–1.20 → slight overspend
# pacing_ratio > 1.20 → overspend risk

That's the whole core logic. SaaS tools wrap this in dashboards, alerts, multi-account aggregation, and reporting. But the underlying calculation is six lines of code.

The reason SaaS tools charge $49+/month isn't the math — it's the data plumbing. They connect to the Google Ads API (OAuth, refresh tokens, quota management), run server-side jobs to pull your accounts on a schedule, store results in a database, render charts. Real infrastructure cost.

But here's the thing: your campaign data is already visible on your Google Ads screen. Names, budgets, costs, statuses — the information is sitting in the DOM right there. If you're already looking at Google Ads, why does anyone need to call an API to tell you what you're already looking at?

The Chrome extension approach

I built AdPacer as a Manifest V3 Chrome extension that reads the campaign data from the Google Ads page DOM and overlays three pacing indicators directly on the interface you're already using. Architecture:

Content script runs on ads.google.com URLs
MutationObserver detects when the campaigns table renders or updates (Google Ads is a heavy SPA so this matters)
DOM parsing extracts campaign name, daily budget, current spend per row
Pacing math runs locally on the extracted values
DOM injection adds the colored pacing bars and projected-spend badges next to each campaign row
Notifications API for the periodic overspend checks

Zero API calls. Zero authentication flows. Zero backend. Zero data leaves the user's browser. Everything runs in the page's content-script context.

The privacy implication is meaningful: AdPacer cannot exfiltrate your Google Ads data even if it wanted to. There's no network request to anywhere. SaaS tools, however privacy-conscious their privacy policies are, send your campaign data to their servers as a fundamental part of how they work.

What you actually see in Google Ads after install

Three additions to the standard Campaigns tab:

1. Pacing bars — a color-coded bar next to each campaign:

Green: on pace, within 10% of expected spend
Yellow: ahead of pace, 10–20% over expected
Red: overspend risk, 20%+ over expected

2. Projected end-of-month spend — a badge showing what your monthly spend will be if you continue at the current daily run rate. Updates as the page data updates. No spreadsheet required.

3. Browser notifications — when any campaign crosses your threshold (configurable from 10% to 25%). Checks automatically every 30 minutes. Catch problems early instead of at month-end reconciliation.

That's it. Install, open Google Ads, see your pacing.

Pricing — and why this is structured the way it is

I deliberately wanted to make this accessible to freelancers and small teams, not enterprise-priced.

Tier	Price	Limit
Free	$0	Up to 3 campaigns. All core features. No credit card, no trial expiration.
Pro	$14/mo	Unlimited campaigns, custom thresholds, priority support.
Agency	$29/mo	Multi-account support, PDF pacing reports, team sharing.

The free tier covers most freelancers managing 1-3 client accounts at a time, or small e-commerce teams running a couple of brand/generic/shopping campaigns. Pro is for in-house PPC managers running 5-50 campaigns. Agency is for teams managing multiple clients.

For comparison with the SaaS landscape:

Tool	Lowest tier	Notes
TrueClicks	$49/mo	Broader PPC management
Optmyzr	$129/mo	Optimization suite
WordStream	$299/mo+	Enterprise tier
AdPacer	$0–$14/mo	Pacing only

If you need full PPC management — bid optimization, A/B testing, audience suggestions, the whole stack — the SaaS tools are doing a lot more than pacing. But if all you actually need is "tell me when a campaign is going to overspend," paying $49-299/month for that single feature is overkill.

Who I built this for

PPC managers running Google Ads daily who want instant budget visibility without context-switching to another tool. Freelancers managing 1-5 client accounts where SaaS pricing eats too much of the margin. Agency teams who need quick pacing checks across multiple campaigns. E-commerce advertisers watching ROAS and budget efficiency in real time.

If you're an enterprise team running 200+ campaigns with complex bid strategies, this isn't for you — you probably already have an Optmyzr-class tool. If you're somewhere between "spreadsheet" and "expensive SaaS," this fills the gap.

What it doesn't do (yet)

Being honest about scope:

No Microsoft Ads / Bing Ads support yet (Google Ads only)
No Meta / TikTok Ads (different DOMs, different challenges, would be a separate extension)
No historical pacing trends beyond current month
No bid suggestions or campaign optimization (that's a different problem space)

Pacing is a single, focused use case. The extension does that one thing well rather than trying to be a half-decent everything tool.

Install link

AdPacer on the Chrome Web Store

Free tier covers up to 3 campaigns with no credit card and no trial expiration — install and see if it solves your problem before paying anything.

If you're a PPC manager and the spending pattern Google introduced in March 2026 has been causing you headaches, this is the lowest-friction way to catch overspend before it happens. If you're a developer reading this for the technical approach: yes, the entire thing runs client-side via DOM parsing — no API key, no backend, no data leaves the browser.

Happy to answer questions about either side.

How to scrape Weibo (微博) data with Python in 2026 — the Sina Visitor System and how to handle it

Sami — Thu, 30 Apr 2026 14:58:19 +0000

Weibo is China's Twitter — the platform where Chinese public opinion forms, brand crises break first, and government statements land. 580M+ monthly active users, mostly mainstream demographics. If you're doing China market intelligence, brand monitoring, or PR analytics, Weibo is one of the platforms you can't skip.

The challenge: Weibo's developer API requires a Chinese business license, has severe rate limits, and exposes very limited data. For Western teams, web scraping is the practical option. The interesting twist is Weibo's Sina Visitor System — an auth flow that makes anonymous access possible for some endpoints but not others. Understanding which is which matters for what you can actually scrape.

This article covers the technical landscape (with real Python code) and points to a hosted scraper if you'd rather skip the maintenance.

What Weibo serves

A Weibo post is structured similarly to a tweet but with longer character limits and more structured engagement signals:

Post text (140 to 2,000 characters depending on user level)
Repost chain — Weibo's quote-tweet equivalent, central to virality tracking
Engagement metrics — attitudes_count (likes), comments_count, reposts_count
Hashtags and mentions
Geolocation if disclosed by user
Author profile — follower count, verification status, verified reason (e.g., "新浪科技 official Weibo")
Media — images, videos

A Weibo user profile gives you:

User ID (numeric)
Screen name (display name)
Description / bio
Followers / friends counts
Statuses count (total posts)
Verification status with reason text — this is gold for identifying official accounts vs personal vs corporate

For monitoring use cases, the metric that matters most depends on your goal:

Crisis monitoring: track comments_count and repost velocity. A spike in either signals viral attention.
Brand presence: track post frequency from verified accounts in your category.
KOL identification: filter by verified=true + follower count above a threshold.

The Sina Visitor System

This is the key technical concept for scraping Weibo without a Chinese business license.

When you visit Weibo without logging in, Sina automatically issues you a "visitor cookie" via what they call the Sina Visitor System (SVS). This cookie lets you access limited public data — specifically:

Hot search / trending topics: full access
Post comments: full access for any public post
Post viewing: limited

For these endpoints, scraping is straightforward — get a visitor cookie, hit the AJAX endpoint, parse JSON.

What the visitor cookie does NOT give you:

Search by keyword (returns hot timeline as a fallback instead of true search results)
User posts beyond profile basics (you get the profile, not the user's post history)

For those, you need a real logged-in cookie — specifically the SUB cookie value from a logged-in browser session. We'll get to that.

Approach 1: Build it yourself

The Sina Visitor System flow looks roughly like this:

import httpx

# Step 1: Hit the visitor system to get a tid (temporary ID)
visitor_url = "https://passport.weibo.com/visitor/genvisitor"
resp = httpx.post(visitor_url, data={
    "cb": "gen_callback",
    "fp": '{"os":"1","browser":"Chrome","fonts":"undefined","screenInfo":"1920*1080*24","plugins":""}',
})
# The response is a JSONP-wrapped JSON. Strip the wrapper, parse, extract tid.

# Step 2: Use tid to get the SUB visitor cookie
incarnate_url = "https://passport.weibo.com/visitor/visitor"
resp = httpx.get(incarnate_url, params={
    "a": "incarnate",
    "t": tid,
    "w": "2",
    "c": "100",
})
# Response sets cookies. Extract SUB and SUBP from response.cookies.

# Step 3: Use those cookies to call AJAX endpoints
hot_search_url = "https://weibo.com/ajax/side/hotSearch"
resp = httpx.get(hot_search_url, cookies={"SUB": sub, "SUBP": subp})
data = resp.json()
# data["data"]["realtime"] is the hot search list

That's the rough shape. In practice you'll handle:

Rate limit responses (HTTP 418, 429) with exponential backoff
Cookie expiration (visitor cookies last hours, not days)
AJAX endpoint changes (Weibo periodically reshuffles paths)
Anti-scraping fingerprint checks (less aggressive than RedNote, but still present)

For the keyword-search and user-posts endpoints, you'll need a real SUB cookie from a logged-in account:

# Get SUB from your browser DevTools → Application → Cookies → weibo.com
# Look for the cookie named "SUB"
sub_cookie = "SUB=_2A25Fxxxxxx..."

resp = httpx.get(
    "https://weibo.com/ajax/side/searchAll",
    params={"q": "人工智能"},
    cookies={"SUB": sub_cookie.split("=", 1)[1]},
)

Cookies typically last several days before expiring, depending on Weibo's session policies.

DIY cost breakdown

Cost	Estimate
Initial setup (visitor system, hot search, comments)	4-8 hours
User session cookie management	1-2 hours/week
Maintenance when Weibo changes endpoints	2-4 hours, every 2-3 months
No proxy needed for most endpoints	$0

Weibo is genuinely the easiest of the major Chinese platforms to scrape if you stay within visitor-system endpoints. RedNote and Bilibili both have more complex auth.

Approach 2: Use a hosted scraper

If you don't want to maintain visitor-system handling and cookie management, the zhorex/weibo-scraper Apify actor handles it.

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_API_TOKEN")

# Hot search (no cookie needed)
run = client.actor("zhorex/weibo-scraper").call(run_input={
    "mode": "hot_search",
    "maxResults": 50,
})

for topic in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"#{topic['rank']}: {topic['title']} (heat: {topic['hotValue']:,})")

Output format:

{
  "rank": 1,
  "title": "人工智能最新突破",
  "category": "科技",
  "hotValue": 2847562,
  "labelName": "热",
  "isHot": true,
  "url": "https://s.weibo.com/weibo?q=...",
  "scrapedAt": "2026-04-25T12:00:00Z"
}

For brand monitoring, search mode is what you want — though note the search-vs-cookie tradeoff:

# Without cookie: returns hot timeline as fallback
run = client.actor("zhorex/weibo-scraper").call(run_input={
    "mode": "search",
    "searchQuery": "CeraVe",
    "maxResults": 50,
})

# With cookie: returns true keyword-matched results
run = client.actor("zhorex/weibo-scraper").call(run_input={
    "mode": "search",
    "searchQuery": "CeraVe",
    "maxResults": 50,
    "cookieString": "SUB=your_logged_in_cookie",
})

The hosted actor handles the visitor system, exponential backoff, and rate limit recovery internally. Pricing: $5 per 1,000 results.

Honest stats on the actor right now: 4 paying users, 11 free-tier users, 92.5% success rate, 3,768 result extractions to date. Average issue response time when something breaks: under a few hours.

When DIY vs hosted

DIY makes sense when:

You're processing > 1M posts/month (per-result cost adds up)
You have ops capacity to refresh SUB cookies regularly
You need to scrape behind login at scale
You have specific endpoints not covered by hosted scrapers

Hosted makes sense when:

You don't have a dedicated scraper engineer
Volume is moderate (< 500k posts/month)
You want the visitor-system handling to be someone else's problem
You're prototyping and want to validate the use case before committing

What you do with the data downstream

Sentiment analysis on Chinese text is the obvious next layer. Off-the-shelf Chinese BERT models work reasonably for Weibo's discourse style — Weibo posts tend to be more formal than RedNote slang, so general Chinese sentiment models accuracy is higher (typical 75-85% on neutral/positive/negative classification).

For brand crisis detection, the signal you usually want is *velocity

Coin-per-view: the Bilibili metric that beats subscriber count for creator vetting

Sami — Wed, 29 Apr 2026 17:48:32 +0000

If you've ever sponsored a YouTube creator and been disappointed by the ROI, you've already lived through what subscriber count actually measures: not engagement, not influence, not purchase intent. Just historical clicks on a follow button. Many of those followers stopped opening videos two years ago. Some are inactive accounts. Some followed for a single piece of content that has nothing to do with your brand.

This is universally true on creator platforms, but it's especially true on Bilibili — China's YouTube. With 300M+ monthly active users skewed Gen Z and millennials, Bilibili is where Chinese creator marketing happens. And Bilibili exposes three engagement signals that YouTube doesn't, which together let you cut through the noise of follower counts and identify creators whose audiences actually engage.

The single most useful one is coin-per-view ratio. This post explains what it is, why it matters, what threshold to use, and how to compute it for any Chinese creator in a few lines of code.

Why follower count is a lying signal

Three reasons follower counts mislead in creator marketing:

1. Followers are a lagging indicator. Someone followed a creator in 2023 because they liked one video. That doesn't tell you whether they still watch in 2026, whether they engage, or whether they trust the creator's recommendations enough to buy.

2. Followers are gameable. Not everyone games them, but enough creators do that you can't trust raw counts without other signals. Bot followers, follow-for-follow campaigns, paid follower services. China specifically has a robust market for these.

3. The follower-to-engagement ratio varies wildly. A creator with 100k followers and 1M average views per video has fundamentally different audience economics than another creator with 100k followers and 5k average views per video. Both have the same "follower count" — the engagement quality is the actual signal.

This is why every serious creator marketing tool talks about "engagement rate" — which on YouTube is usually computed as (likes + comments) / views. It's better than raw follower count, but on Bilibili you can do meaningfully better.

The three Bilibili-native metrics

Bilibili was designed by anime fans for anime fans, and the engagement system reflects values around quality and creator support that YouTube's flat "like" button never captured. Three metrics that come back from any Bilibili video scrape:

Danmaku (弹幕) — real-time scrolling comments overlaid on the video as users watch. Think livestream chat, but for pre-recorded video. The danmaku count tells you how many people were engaged enough mid-watch to type something. It's a leading indicator of viewing time and attention.

Favorites (收藏) — equivalent to "save for later" or YouTube's bookmark. Strong long-term value signal: high favorites relative to views means people return to this video. Tutorials, references, and definitive content score high here.

Coins (投币) — Bilibili's tipping system. This is the interesting one. Each user gets a small daily allocation of coins (typically 5 per day for active users), and they can "throw" them at videos they want to support. Because coins are scarce by design — you only have a few to spend, ever — coin counts are a strong genuine-appreciation signal.

A user gives a coin to a video they love. They give a coin to a creator they want to keep making content. They don't give a coin to a video they passively watched and forgot. The cost is real (relative to the user's daily allocation), so the signal is real.

Coin-per-view ratio: the single best signal

If I had to pick one metric to evaluate a Bilibili creator, it would be median coin-per-view ratio across their last 20-30 videos.

The math is simple:

coin_per_view = coin_count / view_count * 100  # express as percentage

What I've found from looking at hundreds of Bilibili creators across categories:

Coin/View %	Audience quality
< 0.5%	Passive viewers. Casual scrolling traffic, not engaged.
0.5% – 1%	Average. Normal Bilibili content, decent audience.
1% – 2%	Strong. Genuinely engaged audience. Worth sponsoring.
> 2%	Exceptional. Users actively spending limited resources on this content.

Above 2% is rare. It typically indicates either: (a) genuinely high-quality educational/tutorial content that people return to, (b) a creator with a deeply loyal niche audience, or (c) content that struck a strong emotional/cultural nerve.

For creator vetting, my heuristic is: if median coin-per-view is below 1%, the audience is more passive than the follower count suggests; sponsorship ROI will probably disappoint.

How to compute this for any creator

The data you need: a creator's recent videos with their view and coin counts. Bilibili exposes this through their public API — no auth required. You can use the open-source bilibili-api Python library, or call their /x/space/wbi/arc/search endpoint directly.

If you'd rather skip the API integration entirely, I built a hosted scraper on Apify Store: zhorex/bilibili-scraper. $5 per 1,000 results, free tier covers ~1,000 results.

from apify_client import ApifyClient
from statistics import median

client = ApifyClient("YOUR_APIFY_API_TOKEN")

# Get a creator's last 30 videos
# user_id (mid) is the number in their profile URL: space.bilibili.com/{mid}
run = client.actor("zhorex/bilibili-scraper").call(run_input={
    "mode": "user_videos",
    "userIds": ["546195"],   # 老番茄 (a well-known Bilibili gamer)
    "maxResults": 30,
})

videos = []
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item.get("type") == "video":
        videos.append(item)

# Compute coin-per-view ratio per video
ratios = []
for v in videos:
    views = v.get("viewCount", 0)
    if views < 1000:  # skip videos with too few views to be meaningful
        continue
    coins = v.get("coinCount", 0)
    ratios.append(coins / views * 100)

print(f"Videos analyzed: {len(ratios)}")
print(f"Median coin-per-view: {median(ratios):.2f}%")
print(f"Best video coin-per-view: {max(ratios):.2f}%")

# Categorize
median_ratio = median(ratios)
if median_ratio >= 2.0:
    label = "EXCEPTIONAL"
elif median_ratio >= 1.0:
    label = "STRONG"
elif median_ratio >= 0.5:
    label = "AVERAGE"
else:
    label = "PASSIVE"

print(f"Audience quality: {label}")

Run this against any Bilibili creator's user ID and you have a concrete answer about audience engagement quality.

A workflow for vetting creators at scale

If you're building a creator marketing program for the Chinese market, the workflow that works for the teams I've seen using this:

Gather candidates. From competitor sponsorship lists, from category trending, or from agency recommendations. Aim for 30-50 candidates per round.
Pull their recent video portfolios. Use user_videos mode to get the last 20-30 videos per creator.
Compute aggregate metrics. For each creator: median coin-per-view, median favorite-per-view, median danmaku-per-view, view consistency (standard deviation).
Filter on quality threshold. Drop anyone with median coin-per-view below 1%. This usually cuts the candidate list by 40-60%.
Manual review of the survivors. Watch a sample of their videos. Check for content fit. Evaluate sponsorship history (do their sponsored posts feel native or forced?).
Negotiate from the qualified shortlist.

Total cost using a hosted scraper: ~$5-10 in scraping for a 50-creator vetting round. Compared to agency rates for the same work ($500-2000 per round), the math is obvious once you do it.

Cross-platform creator vetting

Bilibili is not the whole story. If you're vetting creators for a comprehensive China presence:

Bilibili for video content (gaming, tech, anime, education)
RedNote (Xiaohongshu) for product-discovery content (beauty, fashion, lifestyle, food)
Weibo for public discourse and broad reach campaigns

Each platform has different engagement signals. Bilibili has coins; RedNote has saves (similarly scarce intent-to-buy signal); Weibo has reposts and verified-account hierarchy. A creator strong on one isn't necessarily strong on others.

I maintain scrapers for all three on Apify Store under the zhorex profile, with consistent output schemas across the suite. Same pricing model ($5/1000 results), same Apify infrastructure. If you're doing cross-platform creator analytics, the consistency saves integration time.

When this approach fails

Two cases where coin-per-view ratio is a misleading signal:

1. Brand-new creators with very few videos. If a creator has uploaded 3 videos and one went viral with high coins, the ratio looks artificial. Wait until you have 15-20 videos to compute median.

2. Live-stream-focused creators. Bilibili lets creators upload archived live streams. Coin economics are different in livestream context (gifts replace coins). For livestream-heavy creators, you need different analysis.

For everyone else, coin-per-view ratio is the single best signal I've found for vetting Bilibili creator quality at scale.

What this won't tell you

Whether the audience is geographically right for your campaign (need follower demographics, which require auth)
Whether the creator has done sponsorships before that flopped (need to scrape their content for promo patterns)
Whether their audience overlaps with your target customer profile (need cross-reference with other platforms)

Treat coin-per-view as the engagement-quality filter. Everything else still requires manual review or additional data sources.

If you're working on creator marketing for the Chinese market and want to compare notes on what works — drop a comment. I write about Chinese platform analytics (Bilibili, RedNote, Weibo) and the build-vs-buy tradeoffs around them.

Hosted Bilibili scraper: apify.com/zhorex/bilibili-scraper
Other Chinese platform scrapers in the same suite: RedNote for product-discovery content, Weibo for public discourse.

Bilibili API Alternative: Scrape Videos & Creators in 30 Lines of Python (No Browser, No Proxy)

Sami — Wed, 29 Apr 2026 17:41:42 +0000

If you've ever tried to scrape Chinese-platform data at scale, you know the landscape: TLS fingerprinting walls, residential-proxy bills, integration headaches that take weeks before you even see one row of data.

Bilibili is the outlier. No API key. No browser. No proxy. Pure HTTP. Runs in 256 MB of RAM. Genuinely the easiest Chinese platform to scrape in 2026.

If you're tracking a gaming brand launch, doing creator research, building a market-trends dataset, or feeding a recommender system, Bilibili is where you should start. This post walks through how to scrape it from scratch in Python, what data you actually get, and when it makes sense to switch from DIY to a hosted scraper.

Why Bilibili is unusually scrape-friendly

Bilibili (哔哩哔哩) is China's YouTube — 300M+ monthly active users, skewed Gen Z and millennials, dominant in anime, gaming, tech, and educational content. From a scraping perspective, three things make it different from RedNote and Weibo:

1. Public JSON endpoints. Bilibili exposes JSON for video metadata, popular/trending, user info, and comments. Most don't require auth for public content.

2. Stable structure. The endpoint shapes change rarely — months between meaningful updates. You can ship a scraper and it'll keep working.

3. Globally accessible. No geo-fencing on most public content. You can hit it from a US datacenter without proxies for moderate volumes.

The trade-off: comments throttle aggressively from cloud IPs (only top ~3 comments come back). Search has its own constraints (more on that below). For everything else, Bilibili is a friendly target.

What you can actually extract

Per video:

BVID, title, description, URL, duration
View count, like count, danmaku (live scrolling comments) count, coin (tip) count, favorite count, share count, reply count
Author MID, name, publish date, category, tags

Per user/creator:

MID, name, signature/bio, level, follower count
Total archive (video) count
Profile URL

Per comment:

Comment ID, text, like count, reply count, created timestamp
Author MID, name, avatar, level

Bilibili-specific signal worth highlighting: coin-per-view ratio. When users "throw coins" at a video, they spend a finite daily allowance — that's a stronger quality signal than likes (which are free). Useful for creator vetting.

A minimal Python scraper (≈30 lines)

This pulls Bilibili's trending/popular videos as clean structured rows. No external dependencies beyond httpx. No request signing. Works on the first run.

import httpx
import json

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
    "Accept": "application/json",
    "Referer": "https://www.bilibili.com/",
}

def fetch_popular_bilibili(max_results: int = 30):
    url = "https://api.bilibili.com/x/web-interface/popular"
    results = []
    page = 1
    while len(results) < max_results:
        r = httpx.get(url, headers=HEADERS, params={"ps": 20, "pn": page})
        data = r.json().get("data") or {}
        items = data.get("list") or []
        if not items:
            break
        for it in items:
            owner = it.get("owner") or {}
            stat = it.get("stat") or {}
            results.append({
                "bvid": it.get("bvid"),
                "title": it.get("title"),
                "author": owner.get("name"),
                "viewCount": stat.get("view"),
                "likeCount": stat.get("like"),
                "danmakuCount": stat.get("danmaku"),
                "duration": it.get("duration"),
                "url": f"https://www.bilibili.com/video/{it.get('bvid')}",
            })
        page += 1
    return results[:max_results]

if __name__ == "__main__":
    rows = fetch_popular_bilibili(max_results=20)
    print(json.dumps(rows, indent=2, ensure_ascii=False))

That's the whole scraper. Plain HTTP. No browser. No proxy. Returns Bilibili's currently-trending videos with the engagement metrics that matter.

Sample output

{
    "bvid": "BV1YXDfBUETP",
    "title": "AI教程入门:从零开始",
    "author": "技术老师",
    "viewCount": 1570113,
    "likeCount": 182455,
    "danmakuCount": 7466,
    "duration": 767,
    "url": "https://www.bilibili.com/video/BV1YXDfBUETP"
}

When DIY breaks down

The 30-line popular fetcher is great for prototyping and tracking trends. It falls over when you need:

Search by keyword. Bilibili's search endpoints have additional request requirements that change over time. Reimplementing them yourself means signing logic that you'll need to chase as Bilibili updates it.
Comments at depth. Cloud IPs get throttled to ~3 comments per video. You need residential IPs or session continuity to paginate full threads.
Multi-mode pipelines. Search + video detail + user-videos + comments + popular all wired together with retries, rate-limit handling, and consistent output schemas — that's a weekend's work, not 30 lines.
Long-running jobs. Cookie refresh, exponential backoff on rate-limit responses, image URL normalization, edge cases on user pages with paid memberships.
Output stability. Bilibili sometimes changes a single field name silently. Your downstream breaks.

When that's where you are, point a hosted Actor at it.

Hosted alternative

I maintain the Bilibili Scraper on Apify — five modes (search, video_detail, video_comments, user_videos, popular) in one Actor. Pure HTTP under the hood, same as above, but with all the rate-limiting + retry + schema-stability work already done — including search, which the DIY example skips.

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("zhorex/bilibili-scraper").call(run_input={
    "mode": "search",
    "searchQuery": "人工智能教程",
    "maxResults": 100,
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["title"], item["danmakuCount"])

Pay-per-result: $0.005 per item. First 1,000 items are on Apify's free tier ($5/month credit).

Common questions

Will Bilibili add anti-bot eventually? Some endpoints already have additional request requirements; popular/trending and basic video metadata have stayed open for years. The hosted Actor tracks any changes.

What about geo-restricted content? Some licensed videos (anime, copyrighted music) are mainland-only. Public metadata for those still works; the streams don't.

Is this legal? This Actor only accesses Bilibili's public HTTP endpoints — the same data any browser visitor sees without logging in. No authentication is bypassed. Always consult your local laws.

Building Chinese intelligence at scale?

If you're combining Bilibili with other Chinese platforms (Weibo, RedNote), I maintain the full suite under the same code style and schema conventions:

Bilibili Scraper — (this one)
Weibo Scraper — microblogging, 580M MAU
RedNote (Xiaohongshu) Scraper — lifestyle social
RedNote Shop Scraper — Xiaohongshu e-commerce

Running 1,000+ items per week? I offer custom output schemas, dedicated proxy pools, SLA support, and volume discounts above 50K items/month. DM me on Apify or open an Issue with subject "Enterprise inquiry".

Found a bug? Open an Issue on the Actor page — I usually ship fixes within 48 hours.

If this saved you time, a 30-second review on the Apify Store helps a lot. ⭐

How to scrape RedNote (Xiaohongshu) with Python in 2026 — the auth/signing problem and how to handle it

Sami — Sat, 25 Apr 2026 01:13:23 +0000

RedNote (Xiaohongshu, 小红书, sometimes "Little Red Book" or just XHS) is the platform a lot of Western teams realized they needed to monitor in 2024-2025, when the TikTok regulatory mess in the US sent millions of users — and brand attention — toward Chinese platforms. It's now China's #1 lifestyle and product-discovery network, with 300M+ monthly active users and a search-driven discovery model that makes it different from every other Chinese social platform.

The problem: there's no official public API. Western teams who try to monitor it usually end up either (a) paying enterprise vendors $20-50k/year for limited China coverage, or (b) trying to scrape it themselves and discovering that RedNote has one of the more aggressive anti-scraping stacks in Chinese social.

This article walks through the actual technical challenges and shows you both DIY and hosted approaches with real Python code. I've shipped a hosted RedNote scraper on Apify that I'll mention later — but the goal here is for you to understand the problem space well enough to make an informed build-vs-buy decision, not to sell you anything.

What RedNote actually serves

Before we go technical: what data does RedNote expose, and what's actually useful?

A RedNote post is structured roughly like this:

Title (often very short, sometimes empty)
Body text — long-form description with product mentions, hashtags, location tags
Image carousel — 1-9 images. Critical: a non-trivial portion of product info lives in image text overlays, not in the body
Engagement metrics — likes, saves, comments, shares
Author profile — username, avatar, follower/following counts, bio, verification, location
Tags / categories — hashtags and platform-assigned categories

For most monitoring use cases, the metric that matters more than likes is saves. Saves on RedNote are the closest equivalent to "I want to buy this later" — they correlate with purchase intent. Likes on RedNote are casual engagement, similar to Twitter likes.

Profile data is structured similarly:

User ID, RedNote ID (red ID), nickname, avatar
Bio / description
Follower / following counts
Location, gender, profile tags
Total likes received across all posts
Verification status

The technical challenges (why this is harder than scraping Twitter)

If you've scraped Western social platforms, your default toolkit is probably httpx or requests plus maybe a residential proxy. RedNote is going to break each of those defaults.

Challenge 1: TLS fingerprinting

RedNote uses TLS fingerprinting (specifically JA3/JA4) to identify and block requests that don't come from real browsers. The requests library has a Python-specific TLS fingerprint that RedNote's bot-detection layer recognizes immediately.

The standard fix is to use curl_cffi, which lets you spoof a Chrome or Safari TLS fingerprint:

from curl_cffi import requests as curl_requests

# Spoof Chrome 120's TLS fingerprint
response = curl_requests.get(
    "https://www.xiaohongshu.com/explore",
    impersonate="chrome120"
)

This alone gets you past the first layer of detection.

Challenge 2: Request signing

RedNote signs every API request with a value called x-s (sometimes seen as xs) plus other parameters like x-t and x-s-common. These are computed client-side from a JavaScript function in their web app.

The signing function changes roughly monthly. When it changes, every scraper using the old signing logic breaks until someone reverse-engineers the new function.

Here's roughly what you need to do:

# Pseudo-code — actual signing logic is more complex

import time
import json
import hashlib

def generate_signing_headers(url_path: str, body: dict) -> dict:
    """
    The actual logic is reverse-engineered from RedNote's web client JS.
    This requires reading their obfuscated bundle and reproducing it in Python.
    """
    timestamp = str(int(time.time() * 1000))
    body_str = json.dumps(body, separators=(',', ':')) if body else ""

    # Real implementation involves:
    # - Specific input concatenation order
    # - Custom hashing scheme (not standard HMAC)
    # - Several "magic constants" that change when RedNote rotates
    # - Sometimes a captcha-derived token

    raw = f"url={url_path}&data={body_str}&t={timestamp}"
    x_s = hashlib.md5(raw.encode()).hexdigest()  # This is NOT the actual algorithm

    return {
        "x-s": x_s,
        "x-t": timestamp,
        # x-s-common is computed separately
    }

The actual signing algorithm is more complex than what I've shown. There are open-source libraries that have reverse-engineered it (xhs-api and similar on GitHub) — they get you most of the way there, but expect to patch them when RedNote rotates.

Challenge 3: IP-level rate limiting and datacenter blocking

RedNote blocks requests from datacenter IPs (AWS, GCP, Azure, DigitalOcean, etc.) within minutes. You need residential proxies, ideally with Chinese geolocation or at least Asia-Pacific.

Even with residential IPs, there's a per-IP rate limit. Realistic throughput is around 10-20 requests per minute per IP before you start getting 412/418 errors and eventually IP bans.

Challenge 4: SPA / dynamic rendering for some endpoints

Search and the explore feed are loaded via AJAX after initial page load, but a few endpoints (some user pages, certain post types) only render their data in the Vue.js application state. You either need to extract data from the inline <script> tag (look for window.__INITIAL_STATE__) or render with Playwright.

Challenge 5: Login walls on certain features

True keyword-filtered search requires login. Without login, you get the explore feed (trending/recommended for your keyword), which is useful but not the same. This is a structural product limitation, not a scraping limitation — you can scrape it the same way logged-in users see it, you just need to either provide cookies or accept the explore-feed fallback.

Approach 1: Build it yourself

If you have ops capacity to maintain it (someone who can read JavaScript and reverse-engineer signing functions monthly), DIY is feasible. Here's a minimal example using curl_cffi plus an open-source signing library.

First install:

pip install curl_cffi xhs

Note: xhs is one of several open-source libraries on GitHub that wrap RedNote's API. Check their commit history before depending on one — the abandoned ones break monthly.

from xhs import XhsClient
from xhs.exceptions import DataFetchError

# You need to provide your own cookies and signing function URL
# The 'sign' function comes from the library's reverse-engineered JS
def sign(uri, data=None, a1="", web_session=""):
    # Implementation provided by the library
    # When RedNote rotates, you'll need to update this
    pass

client = XhsClient(
    cookie="abRequestId=...; webBuild=...; xsecappid=xhs-pc-web; ...",
    sign=sign
)

try:
    # Get a user's posts
    user_id = "5cfbc3f10000000018023ebb"
    posts = client.get_user_notes(user_id)

    for post in posts.get("notes", []):
        print(f"Title: {post.get('display_title')}")
        print(f"Likes: {post.get('interact_info', {}).get('liked_count')}")
        print("---")

except DataFetchError as e:
    print(f"RedNote rejected the request: {e}")
    # Common causes:
    # - Signing function out of date (update from upstream)
    # - Cookie expired (re-login)
    # - IP throttled (rotate proxy)

The cookie you need comes from logging into RedNote in a browser and copying the relevant cookies from DevTools. The cookies expire — typically a few days — so you'll need to refresh them periodically.

Here's the honest cost breakdown for DIY:

Cost component	Estimate
Initial setup (researching libraries, getting first scrape working)	8-16 hours
Residential proxy (Bright Data, Oxylabs, etc.)	$50-200/month for moderate volume
Per-incident maintenance when RedNote rotates	4-8 hours, 1-2x/month
Ongoing: cookie refresh, error handling	1-2 hours/week

If you have a developer whose time is worth $50-100/hour, DIY is around $400-1000/month all-in for moderate scraping volumes.

Approach 2: Use a hosted scraper

The build-vs-buy math changes if you don't have someone on the team who can read reverse-engineered JavaScript and patch signing logic. Hosted Apify Actors handle that for you.

Several developers (including me) maintain RedNote scrapers on Apify. Mine is:

apify.com/zhorex/rednote-xiaohongshu-scraper
$5 per 1,000 results
14 paying users currently, 38 on free tier
Average issue response time: 1.6 hours
88.8% success rate (the gap is mostly RedNote-side transient errors)

Using it from Python:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_API_TOKEN")

# Search
run = client.actor("zhorex/rednote-xiaohongshu-scraper").call(run_input={
    "mode": "search",
    "searchQuery": "skincare routine",
    "maxResults": 50,
    "filterByMinLikes": 100  # Only return posts with 100+ likes
})

# Iterate over results
for post in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"Title: {post['title']}")
    print(f"Likes: {post['likes']}")
    print(f"Author: {post['author']['nickname']}")
    print(f"URL: {post['postUrl']}")
    print("---")

The output JSON is flat:

{
  "mode": "search",
  "postId": "69d269310000000023017e07",
  "postUrl": "https://www.xiaohongshu.com/explore/69d269310000000023017e07",
  "type": "normal",
  "title": "Morning skincare routine for dry skin",
  "images": ["https://sns-webpic-qc.xhscdn.com/..."],
  "likes": 15234,
  "author": {
    "userId": "575d32285e87e733f0162c0a",
    "nickname": "BeautyQueen",
    "avatar": "https://sns-avatar-qc.xhscdn.com/..."
  },
  "scrapedAt": "2026-04-25T21:14:30Z"
}

This is one option among several on Apify Store. EasyApi has the most users by volume; OrbitData Labs has a different all-in-one approach. Pricing is roughly the same across them ($5/1000 ± $1). Differences are in:

Output schema (some return RedNote's raw nested API response, some flatten it)
Update frequency (some are abandoned and break for weeks at a time)
Mode coverage (some only do search; others handle profiles, comments, videos, etc.)
Issue response time

If you're evaluating, run a free-tier test on 2-3 of them with the same input and compare what you get back. The free tier costs you nothing.

When does each approach make sense?

DIY (build it yourself):

You have a Chinese-language ops team and can monitor breakages
You're processing > 1M posts/month (the per-result cost of hosted starts to add up)
You need to scrape behind login (which means you need cookies from logged-in accounts you control)
You have specific data needs that no hosted scraper covers

Hosted Apify actor:

You don't have a dedicated scraper engineer
Volume is variable or moderate (< 500k posts/month)
You want to outsource the cat-and-mouse with RedNote's anti-bot updates
You're prototyping and want to validate the approach before committing

The middle ground that often makes sense: use a hosted actor for production data flow, build a thin DIY layer for any specific endpoints the hosted version doesn't cover. The hosted scraper handles the maintenance burden on the parts that break most often (search, profile, posts), and you keep custom DIY logic for the edges.

What you do with the data downstream

Scraping is one third of the problem. The other two:

Sentiment analysis on Chinese text. Off-the-shelf Chinese BERT models (like bert-base-chinese from Huggingface) are a starting point but accuracy varies wildly by domain. RedNote slang, in particular, doesn't appear in the training data of general Chinese sentiment models — fine-tuning on RedNote-specific labeled samples gets you significant accuracy lift if accuracy matters.

Image text extraction. A non-trivial portion of product mentions on RedNote live in image text overlays (Chinese users frequently put product names visible in images, not in the post body). PaddleOCR is the open-source standard for Chinese OCR. Slow (~30 seconds per image) but reliable. Adds significant cost to processing pipelines but you'll miss a measurable percentage of product mentions without it.

Both of these are downstream of scraping — solve scraping first, then layer.

FAQ

Is scraping RedNote legal?

Public-data scraping legality varies by jurisdiction. RedNote's ToS prohibits automated access (as do most platform ToS). The Apify approach (and most public-scraping infrastructure) treats public web pages as accessible, the same way Google's crawler would. You should consult legal counsel for your specific use case. Not legal advice.

How fast can I scrape RedNote?

Realistic sustained throughput per IP is around 10-20 requests per minute before triggering rate limits. With residential proxy rotation and proper backoff, you can scale this horizontally — Apify's actor handles this internally. For DIY, plan for ~1 result per second per IP as a conservative number.

Do I need a Chinese IP?

Not strictly required, but residential IPs (Asian residential preferred) have notably higher success rates than US/European residential. Datacenter IPs are blocked outright.

What's xsec_token and why does it matter?

When users share posts via the RedNote app or copy URLs, those URLs include an xsec_token query parameter that authenticates the link request. Some scrapers don't handle URLs with xsec_token correctly and return errors. If you're scraping URLs collected from real users, make sure your tooling supports this.

Can I scrape video files from RedNote posts?

Yes. Video URLs are returned in the post metadata. Direct download from those URLs works without authentication for public videos.

How often does the request signing change?

Roughly monthly, sometimes more frequently around major RedNote app updates. If you're doing DIY, plan to dedicate 4-8 hours per rotation to update your signing function, or rely on an actively maintained library that pushes updates fast.

What's the difference between RedNote and Xiaohongshu?

They're the same platform. "Xiaohongshu" (小红书) is the Chinese name and means "Little Red Book". "RedNote" is the English brand they pushed during the 2024-2025 TikTok migration to be more accessible to global users. Same app, same data, same API endpoints.

If you're working on China market intelligence, brand monitoring, or competitive research and want to compare notes — drop me a comment. I write about Chinese platform scraping (Weibo, Bilibili, RedNote) and the build-vs-buy trade-offs around them.

The actor I mentioned: apify.com/zhorex/rednote-xiaohongshu-scraper. Free tier covers ~1,000 results, which is enough to validate against your specific use case before committing.

Monitoring the Chinese Social Media Ecosystem: RedNote, Weibo & Bilibili Data Pipeline

Sami — Thu, 23 Apr 2026 14:56:48 +0000

If you are doing market research, brand monitoring, competitive intelligence, or academic research on China, no single platform tells the whole story. Chinese internet users spread across specialized platforms the way Western users split between Twitter, Instagram, YouTube, and Reddit — but the Chinese platforms are larger, more fragmented, and harder to access from outside the country.

This article maps the three most important Chinese social platforms for data collection, explains what each one covers, and shows how to build a unified monitoring pipeline using three Apify Actors that share a common architecture: no browser, no proxy, no API keys.

The Three Pillars of Chinese Social Media Intelligence

1. RedNote / Xiaohongshu (小红书) — Social Commerce & Lifestyle

What it is: China's Instagram meets Pinterest. A social commerce platform where users share product reviews, lifestyle content, travel diaries, and beauty routines. Known for its influence on Chinese consumer purchasing decisions.

User base: 200M+ monthly active users

Why it matters for research:

Product sentiment and unfiltered consumer reviews
Trend detection in beauty, fashion, food, travel, and lifestyle
Influencer (KOL/KOC) discovery for Chinese market campaigns
Brand perception monitoring among Chinese consumers

Actor: zhorex/rednote-scraper

Modes: Search posts, user profiles, post comments, trending topics

2. Weibo (微博) — Microblogging & Public Opinion

What it is: China's Twitter. The platform where Chinese public opinion forms in real-time. Breaking news, celebrity drama, government communications, and brand PR all happen on Weibo.

User base: 580M+ monthly active users

Why it matters for research:

Real-time trending topics with heat scores and categories
Public opinion tracking on policy, brands, and international events
Celebrity and KOL influence measurement
Crisis monitoring and brand sentiment during PR events

Actor: zhorex/weibo-scraper

Modes: Hot search/trending, post comments, keyword search, user posts

Auth note: Trending topics and post comments work without login. Search and user posts require a Weibo SUB cookie (easily obtained from a browser session).

3. Bilibili (哔哩哔哩) — Video & Creator Analytics

What it is: China's YouTube, with a strong focus on anime, gaming, tech education, and Gen Z culture. Known for its unique danmaku (弹幕) system — scrolling comments that overlay the video in real-time.

User base: 300M+ monthly active users

Why it matters for research:

Creator analytics with Bilibili-specific metrics (danmaku, coins, favorites)
Gaming and anime industry monitoring
Chinese Gen Z content trends and preferences
Tech and education content analysis

Actor: zhorex/bilibili-scraper

Modes: Search videos, video details, video comments, user/creator videos, popular/trending

Platform Comparison

Dimension	RedNote	Weibo	Bilibili
Platform type	Social commerce	Microblogging	Video
Primary audience	Women 18-35, consumers	All demographics	Gen Z, gamers, students
Content format	Photos + short text	Short posts (tweets)	Long-form video
Best for	Product sentiment, lifestyle trends	Breaking news, public opinion	Creator analytics, youth culture
Unique data	Purchase intent, product reviews	Hot search rankings, heat scores	Danmaku counts, coin tipping
Auth required	No	Partial (trending: no, search: yes)	No
MAU	200M+	580M+	300M+
Actor	`rednote-scraper`	`weibo-scraper`	`bilibili-scraper`

Building a Unified Pipeline

All three actors share common design principles:

Pure HTTP — no browser needed, minimal compute
256MB RAM — cheap to run
Pay-per-event — $5 per 1,000 items for all three
JSON/CSV/Excel output — same export formats
Apify integrations — Google Sheets, S3, webhooks, Zapier, Make, n8n

Example: Daily China Brand Monitor

Here is a practical pipeline architecture using all three actors:

Step 1: Schedule daily runs for each actor via Apify Schedules.

RedNote — search for your brand name in Chinese:

{
  "mode": "search",
  "searchQuery": "你的品牌名",
  "maxResults": 50
}

Weibo — monitor trending mentions:

{
  "mode": "hot_search",
  "maxResults": 50
}

Bilibili — track video mentions:

{
  "mode": "search",
  "searchQuery": "你的品牌名",
  "maxResults": 50
}

Step 2: Use Apify webhooks to push results to your data warehouse (S3 → Snowflake/BigQuery) or directly to Google Sheets for a lightweight dashboard.

Step 3: Set up alerting — use Zapier or Make to send Slack notifications when specific keywords appear in trending topics or when engagement spikes on brand-related content.

Use Case Examples

Brand Entering the Chinese Market

A Western consumer brand preparing to launch in China runs all three actors to understand:

RedNote: What do Chinese consumers say about the product category? What do they value? Which competitor products are reviewed positively?
Weibo: Is the brand already discussed in Chinese media? What is the general sentiment?
Bilibili: Are there video reviews or unboxing content for the brand or competitors?

Academic Research on Chinese Digital Culture

A researcher studying Chinese Gen Z media consumption uses:

Bilibili: Trending video content, danmaku engagement patterns, creator growth trajectories
Weibo: Public discourse topics, hashtag trends over time
RedNote: Lifestyle and consumption trends, product preference signals

PR Crisis Monitoring

A multinational company monitors all three platforms for brand mentions after a negative news cycle:

Weibo: Real-time trending topics and comment sentiment (the fastest-moving platform)
RedNote: Consumer reaction and product boycott signals
Bilibili: Video commentary and creator opinion pieces

Technical Architecture

All three actors share:

Feature	Detail
Runtime	Pure HTTP (no Playwright/Puppeteer)
Memory	256MB RAM
Proxy	Not required
API keys	Not required
Output	JSON, CSV, XLSX, JSONL
Rate limiting	Built-in backoff and retry
Language	Content in original Simplified Chinese
Translation	Pipe through Google Translate, DeepL, or Claude for English

Pricing

All three actors use the same pricing model:

Volume	Cost per actor
1,000 items	$5
10,000 items	$50
100,000 items	$500

Running all three actors daily with 50 items each = 150 items/day × 30 days = 4,500 items/month ≈ $22.50/month for comprehensive daily China monitoring across three platforms.

Apify's free plan includes $5 of monthly credits — enough to test all three actors.

FAQ

Do I need a VPN to scrape Chinese platforms?
No. All three actors use public HTTP endpoints that are globally accessible. No VPN or proxy is required.

Is the data in Chinese?
Yes. All content is returned in the original Simplified Chinese. For English translations, pipe the output through a translation service.

Can I combine data from all three actors?
Yes. All three output JSON with similar schemas. Use Apify's dataset API or export to a shared warehouse for unified analysis.

Is scraping Chinese social media legal?
These actors only access publicly available data through public web endpoints. No authentication is bypassed and no private data is accessed. Always review your local laws and each platform's terms of service.

Get Started

RedNote: apify.com/zhorex/rednote-scraper
Weibo: apify.com/zhorex/weibo-scraper
Bilibili: apify.com/zhorex/bilibili-scraper

Start with one platform, validate the data quality, then expand to all three for comprehensive Chinese social media intelligence.

Built by Zhorex — the only developer on Apify specializing in Chinese platform intelligence.

Automating Perplexity AI Searches for Content Research and Brand Monitoring

Sami — Thu, 23 Apr 2026 14:55:14 +0000

Perplexity AI has become one of the most popular AI-powered search engines, giving users synthesized answers with cited sources instead of a list of blue links. For marketers, content strategists, and SEO professionals, a new discipline has emerged: Answer Engine Optimization (AEO) — the practice of getting your brand mentioned and cited in AI-generated search results.

The challenge is that monitoring how Perplexity answers queries about your brand, competitors, or industry requires manually searching one query at a time. If you want to track visibility across dozens or hundreds of queries on a regular schedule, you need automation.

This article walks through how to automate Perplexity AI searches at scale using the zhorex/perplexity-ai-scraper Actor on Apify — no Perplexity API key required.

TL;DR

What: Extract AI-generated answers, cited sources, and related questions from Perplexity AI
How: zhorex/perplexity-ai-scraper on Apify
Cost: $0.02 per query ($20 per 1,000 queries)
No API key: Scrapes the public web interface directly via headless browser
Two modes: Full search results, or brand mention monitoring

Why Automate Perplexity Searches?

Answer Engine Optimization (AEO)

As more users shift from Google to AI search engines like Perplexity, ChatGPT, and Google AI Overviews, monitoring your visibility in AI-generated answers is becoming critical. AEO focuses on:

Is your brand mentioned when users ask about your product category?
What position does your brand appear in the AI answer?
Which competitors are mentioned alongside you?
What sources does Perplexity cite — and is your content among them?

Content Strategy

Perplexity curates answers from multiple sources and cites them. By analyzing which URLs get cited for your target keywords, you can identify content gaps: if your competitors are cited but you are not, that signals where to invest in content.

Competitive Intelligence

Track how Perplexity positions your competitors for your target keywords. Which brands does it recommend? How does it describe them? This gives you a direct view into how AI search engines perceive your market.

Two Modes of Operation

Mode 1: Search — Extract Full AI Answers

Submit any query and get the full AI-generated answer with all cited sources.

Input:

{
  "mode": "search",
  "queries": [
    "best CRM software for small business 2026",
    "top project management tools for startups"
  ]
}

Output:

{
  "query": "best CRM software for small business 2026",
  "answer": "For 2026, the best CRM overall is...",
  "sources": [
    {
      "position": 1,
      "title": "The Best CRM Software for 2026",
      "url": "https://www.pcmag.com/picks/the-best-crm-software",
      "domain": "pcmag.com",
      "snippet": "Customers are vital to any business..."
    }
  ],
  "relatedQuestions": [
    "Zoho CRM vs Salesforce which is better for small businesses",
    "Best free or low-cost CRM alternatives for startups 2026"
  ],
  "totalSources": 10,
  "answerLength": 1542,
  "searchUrl": "https://www.perplexity.ai/search?q=...",
  "scrapedAt": "2026-04-09T15:00:00+00:00"
}

Mode 2: Brand Monitor — Track Brand Mentions in AI Answers

Automatically detect whether your brand appears in Perplexity's answer for a set of queries, and see which competitors are mentioned.

Input:

{
  "mode": "brand_monitor",
  "brandName": "HubSpot",
  "queries": [
    "best CRM software 2026",
    "top email marketing platforms for ecommerce"
  ]
}

Output:

{
  "brand": "HubSpot",
  "query": "best CRM software 2026",
  "mentioned": true,
  "mentionContext": "HubSpot is the easiest all-around option for teams that want a simple start...",
  "position": "Mentioned early in the answer (top section)",
  "competitorsMentioned": ["Zoho CRM", "Salesforce", "Pipedrive"],
  "sourcesCount": 10,
  "scrapedAt": "2026-04-09T15:00:00+00:00"
}

Practical Use Cases

1. Weekly AEO Dashboard

Schedule the actor to run weekly with your core keyword list (e.g., "best [your category] software 2026"). Track over time whether your brand is being recommended, and whether your visibility is improving after content investments.

2. Competitor Monitoring

Run brand_monitor mode with your top competitor names. See which queries they appear for that you do not — those are opportunities.

3. Content Gap Analysis

For your target queries, look at which sources Perplexity cites. If your content is not cited, study what the cited sources cover that you do not. Use this to prioritize content creation.

4. Market Research at Scale

Need AI-curated answers to dozens of research questions? Use search mode with a list of queries and get structured, sourced answers in bulk — with full attribution for verification.

Python and JavaScript Integration

Python:

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")
run = client.actor("zhorex/perplexity-ai-scraper").call(run_input={
    "mode": "brand_monitor",
    "brandName": "YourBrand",
    "queries": ["best project management tools 2026"]
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["mentioned"], item["competitorsMentioned"])

JavaScript:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });
const run = await client.actor('zhorex/perplexity-ai-scraper').call({
    mode: 'search',
    queries: ['best CRM software 2026'],
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items[0].answer);

How It Works Under the Hood

The actor opens Perplexity.ai in a headless Chromium browser (Playwright), navigates to the search URL for each query, waits for the AI to finish generating its streamed answer, then extracts the full answer text, cited sources with URLs, and related questions. Results are pushed to a structured Apify dataset.

Performance:

~15-30 seconds per query (AI answer generation takes time)
1024 MB RAM recommended (Playwright + Chromium)
Sequential queries with 5s delay to avoid rate limiting

Perplexity API vs. This Actor

Aspect	Perplexity Sonar API	This Actor
API key required	Yes (paid subscription)	No
Cost	Sonar Pro pricing	$0.02 per query
Speed	Fast (direct API)	~15-30s per query
Sources/citations	Included	Included
Brand monitoring	Build it yourself	Built-in mode
Related questions	Not included	Included

The trade-off is speed: the API is faster, but this actor requires no API key or subscription.

Pricing

$0.02 per query ($20 per 1,000 queries). You can test with Apify's free tier which gives you $5 of monthly usage — enough for 250 queries.

FAQ

How is this different from the Perplexity API?
The official Perplexity API (Sonar) gives programmatic access but requires a paid API key. This Actor scrapes the free public web interface — same answers, same sources, no API key required. The trade-off is speed (~15-30s per query).

What is AEO?
Answer Engine Optimization is the practice of getting your brand mentioned in AI search results. As more users shift to AI search engines, monitoring your visibility in AI-generated answers is becoming as important as traditional SEO.

Can I schedule regular monitoring runs?
Yes. Use Apify Schedules to run the actor daily, weekly, or at any interval. Combine with webhooks to send Slack or email alerts when your brand visibility changes.

Are the answers always the same?
No. AI answers are non-deterministic — the same query can produce slightly different answers on different runs. This is a characteristic of AI search, not a limitation of the actor.

Get Started

https://apify.com/zhorex/perplexity-ai-scraper

Try a single query in search mode to see the output quality, then build out your keyword list for systematic monitoring.

Scraping Bilibili Videos and Creators for Market Research in 2026

Sami — Thu, 23 Apr 2026 14:53:41 +0000

Bilibili (哔哩哔哩) is China's premier video platform with 300M+ monthly active users. Think YouTube, but with a younger demographic (Gen Z and millennials), a thriving anime and gaming community, and a unique feature called danmaku (弹幕) — real-time scrolling comments that overlay the video as it plays.

For anyone doing market research on Chinese youth culture, gaming audiences, tech content, or creator economics, Bilibili is one of the richest data sources available. The problem is that there is no official public Bilibili API for international developers. Bilibili's internal APIs are undocumented, require Chinese phone verification, and change frequently.

This article shows how to extract Bilibili videos, comments, creator profiles, and trending content using the zhorex/bilibili-scraper Actor on Apify — no API key, no browser, no proxy required.

TL;DR

What: Extract videos, comments, creator profiles, and trending content from Bilibili
How: zhorex/bilibili-scraper on Apify — pure HTTP, 256MB RAM
Cost: $5 per 1,000 items scraped
Auth: None required — all endpoints are public
Unique data: Danmaku counts, coin counts (投币), favorites, plus standard engagement metrics

What Makes Bilibili Data Unique

Unlike YouTube or TikTok, Bilibili has platform-specific metrics that this actor captures:

Danmaku count (弹幕) — live scrolling comments overlaid on the video. High danmaku signals active community engagement, not just passive viewing
Coin count (投币) — Bilibili's tipping system where users "throw coins" at creators. A direct signal of audience appreciation
Favorite count (收藏) — equivalent to "save" on other platforms
Standard metrics: views, likes, shares, replies

These metrics together give a much richer picture of content performance than views alone.

Five Scraping Modes

1. Search Videos

Search by keyword (Chinese or English). Supports sort and filter options.

{
  "mode": "search",
  "searchQuery": "人工智能教程",
  "sortOrder": "pubdate",
  "maxResults": 50
}

Sort options: totalrank (relevance), click (most views), pubdate (newest), dm (most danmaku), stow (most favorites), scores (most comments).

Duration filters: short (<10min), `medium` (10-30min), `long` (30-60min), `verylong` (>60min).

2. Video Details

Full video info with all engagement metrics and tags.

{
  "mode": "video_detail",
  "videoUrls": [
    "https://www.bilibili.com/video/BV1GJ411x7h7",
    "BV1xx411c7mD"
  ]
}

3. Video Comments

Extract comments with author info and likes.

{
  "mode": "video_comments",
  "videoUrls": ["https://www.bilibili.com/video/BV1GJ411x7h7"],
  "maxComments": 50,
  "sortComments": "hot"
}

Sort options: hot (top/most-liked), time (newest), likes (by like count).

Note: Bilibili throttles comment pagination from datacenter IPs, returning only top/pinned comments. Full comment pagination requires residential IPs or authenticated sessions. Other modes are unaffected.

4. Creator/User Videos

Get user profile info plus their recent videos.

{
  "mode": "user_videos",
  "userIds": ["546195", "1340190821"],
  "maxResults": 30
}

Find a user's mid in their profile URL: space.bilibili.com/{mid}. Multiple users are processed in parallel (up to 3 concurrent).

5. Popular/Trending Videos

Trending videos, filterable by Bilibili category.

{
  "mode": "popular",
  "category": "game",
  "maxResults": 20
}

Available categories: Animation, Music, Dance, Gaming, Knowledge, Tech, Sports, Cars, Life, Food, Animal, Fashion, Entertainment.

Output Example

Video:

{
  "type": "video",
  "bvid": "BV1YXDfBUETP",
  "title": "Example Video Title",
  "url": "https://www.bilibili.com/video/BV1YXDfBUETP",
  "duration": 167,
  "durationFormatted": "2:47",
  "viewCount": 1570113,
  "likeCount": 182455,
  "coinCount": 110535,
  "favoriteCount": 63471,
  "shareCount": 45918,
  "danmakuCount": 7466,
  "replyCount": 17276,
  "authorName": "Creator Name",
  "authorMid": 1340190821,
  "publishDate": "2026-04-08T12:00:00+00:00",
  "category": "Gaming",
  "tags": ["anime", "action", "review"],
  "scrapedAt": "2026-04-10T10:00:00+00:00"
}

User Profile:

{
  "type": "user",
  "mid": 546195,
  "name": "老番茄",
  "fans": 20189060,
  "archiveCount": 652,
  "verified": false,
  "profileUrl": "https://space.bilibili.com/546195",
  "scrapedAt": "2026-04-10T10:00:00+00:00"
}

Use Cases for Market Research

Gaming/anime brand monitoring — Track game launches and anime reactions on China's largest anime community
Content trend analysis — Identify trending topics in Chinese youth culture and Gen Z interests
Creator evaluation — Analyze Bilibili KOLs (Key Opinion Leaders) for partnerships and sponsorships using follower counts, engagement ratios, and content frequency
Ad placement research — Understand which categories and content types perform best by danmaku density, coin rates, and view-to-engagement ratios
Academic research — Study Chinese digital culture, danmaku behavior, and content consumption patterns
Product launch monitoring — Track brand mentions and competitor content in China

Python and JavaScript Integration

Python:

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")
run = client.actor("zhorex/bilibili-scraper").call(run_input={
    "mode": "search",
    "searchQuery": "人工智能教程",
    "maxResults": 50
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["title"], item["viewCount"], item["danmakuCount"])

JavaScript:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });
const run = await client.actor('zhorex/bilibili-scraper').call({
    mode: 'search',
    searchQuery: '人工智能教程',
    maxResults: 50,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => console.log(item.title, item.viewCount));

Technical Details

No browser — pure HTTP requests to Bilibili's public APIs
No proxy — Bilibili is accessible globally (some licensed content may be geo-restricted)
No API key — all endpoints are public
256MB RAM — lightweight and efficient
Concurrent fetching — video_detail: up to 5 in parallel; user_videos and video_comments: up to 3 in parallel

Pricing

$0.005 per item scraped ($5 per 1,000 results). Start with Apify's free plan which includes $5 of monthly credits.

Part of the Chinese Digital Intelligence Suite

Bilibili covers video and creator analytics. For full China market coverage, combine with:

zhorex/weibo-scraper — Weibo microblogging, trending topics, public opinion (580M+ MAU)
zhorex/rednote-scraper — RedNote/Xiaohongshu social commerce, lifestyle content (200M+ MAU)

All actors: no browser, no proxy, no API keys. Built by Zhorex.

Get Started

https://apify.com/zhorex/bilibili-scraper

Try the popular mode first (no auth needed) to see what is trending on Chinese video right now.

How to Scrape Weibo Without Login in 2026: The Complete Guide

Sami — Thu, 23 Apr 2026 14:51:50 +0000

Weibo (微博) is China's dominant microblogging platform — think Twitter meets Instagram, with 580M+ monthly active users. It is where Chinese public opinion forms, brands communicate, celebrities post, and news breaks. Government officials, industry leaders, and major brands all maintain active Weibo accounts.

For anyone doing China market research, PR monitoring, influencer analysis, or geopolitical tracking, Weibo data is essential. The problem is that there is no official public Weibo API available for international developers. Weibo's developer platform requires a Chinese business license, imposes strict rate limits, and returns limited data.

This guide walks through how to extract Weibo posts, trending topics, comments, and creator profiles using the zhorex/weibo-scraper Actor on Apify — no API key, no browser, no VPN required.

TL;DR

What: Extract posts, trending topics, comments, and user profiles from Weibo
How: zhorex/weibo-scraper on Apify — pure HTTP, no browser needed
Cost: $5 per 1,000 items scraped
Auth: Trending topics and post comments work without any login. Search and user posts require a Weibo SUB cookie
No VPN needed: All endpoints are globally accessible

Why Weibo Data Matters

Who	Why
PR & Communications	Track brand mentions in real-time on China's public square
Market Research	Monitor what is trending among Chinese consumers
Influencer Marketing	Find and evaluate KOLs by followers, engagement, verification
Competitive Intelligence	Track Chinese competitor announcements and campaigns
Geopolitical Analysis	Monitor public discourse on policy and international topics
Journalism	Access Chinese public opinion data for reporting
Academic Research	Study Chinese social media behavior and trends

Four Scraping Modes

The actor supports four distinct modes, each targeting a different type of Weibo data:

1. Hot Search / Trending Topics (no login needed)

Get the real-time pulse of the Chinese internet. Returns trending topics with heat scores and categories.

{
  "mode": "hot_search",
  "maxResults": 50
}

Output:

{
  "rank": 1,
  "title": "人工智能最新突破",
  "category": "科技",
  "hotValue": 2847562,
  "labelName": "热",
  "isHot": true,
  "url": "https://s.weibo.com/weibo?q=...",
  "scrapedAt": "2026-04-10T12:00:00Z"
}

2. Post Comments (no login needed)

Extract comments from specific posts. Provide post IDs (mid) or detail URLs.

{
  "mode": "post_comments",
  "postIds": ["5285773987283226"],
  "maxComments": 50
}

3. Search Posts (login optional)

Search by keyword. Without cookies, returns hot timeline posts as a fallback. With cookies, searches the full index.

{
  "mode": "search",
  "searchQuery": "人工智能",
  "maxResults": 50,
  "cookieString": "SUB=your_sub_cookie_value"
}

4. User Posts (login needed for posts)

Get profile info (always works without login) plus posts (requires cookies). Provide numeric user IDs or profile URLs.

{
  "mode": "user_posts",
  "userIds": ["1642634100"],
  "maxResults": 50,
  "cookieString": "SUB=your_sub_cookie_value"
}

How to Get Cookies

For search and user posts modes, you need a Weibo SUB cookie:

Open weibo.com in your browser and log in
Open DevTools (F12) → Application → Cookies → weibo.com
Copy the value of the SUB cookie
Paste it in the cookieString field as: SUB=your_value_here

The cookie typically lasts several days before expiring.

Python and JavaScript Examples

Python:

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")
run = client.actor("zhorex/weibo-scraper").call(run_input={
    "mode": "hot_search",
    "maxResults": 50
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

JavaScript:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });
const run = await client.actor('zhorex/weibo-scraper').call({
    mode: 'hot_search',
    maxResults: 50,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => console.log(item));

Technical Details

No browser needed — pure HTTP requests using httpx, runs in 256MB RAM
No VPN needed — Weibo endpoints are globally accessible
Automatic session — visitor cookies obtained automatically via the Sina Visitor System
Rate-limit handling — exponential backoff on 418/429 errors
Chinese text preserved — all content returned as-is in original Simplified Chinese

Pricing

Volume	Cost
1,000 items	$5
10,000 items	$50
100,000 items	$500

Each scraped item (post, comment, trending topic, or profile) counts as one result. You can start with Apify's free plan, which includes $5 of monthly credits — enough for 1,000 data points.

Part of the Chinese Digital Intelligence Suite

Weibo covers microblogging and public opinion, but for comprehensive China market intelligence you need the full picture:

Platform	Users	What it covers	Actor
Weibo	580M+ MAU	Microblogging, trending, celebrity content	`zhorex/weibo-scraper`
RedNote (Xiaohongshu)	200M+ MAU	Social commerce, lifestyle, product reviews	`zhorex/rednote-scraper`
Bilibili	300M+ MAU	Video content, danmaku, creator analytics	`zhorex/bilibili-scraper`

All three actors: no browser, no proxy, no API keys. Built by Zhorex — the only developer on Apify specializing in Chinese platform intelligence.

FAQ

Is there a Weibo API?
There is no official public Weibo API available for international developers. Weibo's developer platform requires a Chinese business license and imposes strict rate limits. This scraper is the best alternative.

Do I need a VPN?
No. The Weibo endpoints used by this actor are globally accessible without a VPN or proxy.

Is the content in Chinese?
Yes. Weibo is a Chinese-language platform — all content is returned in the original Simplified Chinese. If you need English translations, pipe the output through a translation API (Google Translate, DeepL, or Claude).

Is scraping Weibo legal?
This scraper only accesses publicly available data through Weibo's public web endpoints. It does not bypass authentication or access private/locked accounts. Always review your local laws and Weibo's terms of service.

Get Started

The Actor page, full input schema, and a free trial run are at:

https://apify.com/zhorex/weibo-scraper

Start with trending topics (no login needed) to see the data quality, then expand to search and user profiles as needed.

Scraping G2 Reviews Without Kasada Headaches: A SaaS Competitive Intelligence Pipeline With 29 Fields Per Review

Sami — Fri, 17 Apr 2026 17:51:52 +0000

G2 holds roughly two million verified software reviews across tens of thousands of SaaS categories. For anyone doing competitive intelligence, sales prospecting, or product research in the B2B software space, it is one of the single highest-signal public datasets on the internet. The problem is that scraping it has become almost comically painful in 2026.

If you have tried hitting g2.com with requests lately, you already know the story. Cloudflare turnstile, then a Kasada challenge, then a TLS fingerprint check, then a behavioral JavaScript puzzle, then an invisible CAPTCHA. Even well-configured headless browsers with residential proxies get flagged within the first dozen pages. The G2 review data is public, but getting to it at scale has turned into a cat-and-mouse game that burns engineering hours and proxy budget in equal measure.

This article walks through a cleaner path: using the zhorex/g2-reviews-scraper Actor on Apify to pull structured reviews, 29 fields deep, without running a single browser or rotating a single proxy.

Why G2 Data Is Worth the Effort

Before getting into the scraper itself, it helps to articulate what the data is actually good for. G2 reviews are structured, long-form, and written by verified business users. That combination makes them uniquely useful for:

Sales intelligence. A review that complains about vendor X's integration with Salesforce is a warm lead for vendor Y whose Salesforce integration is its headline feature.
Win/loss analysis. Reviews of competitors often name the alternatives the reviewer evaluated. This is free narrative market research.
Feature gap detection. Aggregating thousands of "what do you dislike" fields across a category surfaces the roadmap items customers actually care about.
Churn signals. Negative sub-ratings on "support" or "ease of setup" for a specific competitor, trended over quarters, predict defection windows.

The value is there. The delivery mechanism is the bottleneck.

The Problem With the Official Path

G2 does offer a paid API, but it is gated behind an enterprise contract, requires a seat license on the G2 side, and typically restricts you to data about your own product and a handful of named competitors. Pulling the full set of reviews for a category like "CRM Software" or "Marketing Automation" across all vendors is not on the menu.

The community workarounds are worse. DIY scrapers hit Kasada within minutes. Proxy bills for rotating residential IPs run $500 to $2,000 a month at modest volumes. Every time G2 rolls a new JS challenge, your Playwright script breaks at 3 AM and you spend a Saturday fingerprinting headers.

"We burned six weeks and about $4,000 in proxies before admitting our in-house G2 scraper was never going to be stable."
— Growth engineer at a mid-market PLG startup

What the Actor Does

The g2-reviews-scraper Actor bypasses all of that by calling G2's public review feed directly. No browser, no proxy rotation, no Kasada bypass hacks. You give it a product URL or slug and it returns structured JSON.

Feature set:

Scrape reviews for any G2 product URL or slug
29 fields per review including sub-ratings, reviewer job title, company size, industry, and verification status
Pagination handled internally, up to the full review history of a product
Filters for star rating, date range, and review language
JSON, CSV, Excel, or JSONL output
Runs on Apify infrastructure, so no local Node/Python setup required for the scraping itself

Comparison Table

Capability	G2 Official API	DIY scraper + residential proxies	`zhorex/g2-reviews-scraper`
Access to competitor reviews	No (own product only)	Yes, if you can keep it running	Yes
Kasada / JS challenge handling	N/A	Your problem, breaks weekly	Handled, no browser needed
Setup time	Weeks (contract + provisioning)	Days to weeks	Minutes
Cost at 10k reviews/month	Custom enterprise quote	~$300-600 proxies + eng time	$50
Sub-ratings included	Yes	Usually not	Yes (29 fields)
Export formats	JSON via API	Whatever you build	JSON, CSV, XLSX, JSONL
Maintenance burden	Low	High	None

Input Example

A realistic starter config for pulling reviews across three CRM competitors:

{
  "productUrls": [
    "https://www.g2.com/products/salesforce-sales-cloud/reviews",
    "https://www.g2.com/products/hubspot-sales-hub/reviews",
    "https://www.g2.com/products/pipedrive/reviews"
  ],
  "maxReviewsPerProduct": 500,
  "minRating": 1,
  "maxRating": 5,
  "dateFrom": "2024-01-01",
  "language": "en",
  "includeSubRatings": true,
  "includeReviewerProfile": true
}

Output Example

Here is one review item, trimmed to the fields most people care about. The full object has all 29 fields.

{
  "reviewId": "g2-8421930",
  "productSlug": "hubspot-sales-hub",
  "productName": "HubSpot Sales Hub",
  "reviewTitle": "Great for SMB, feels cramped above 50 reps",
  "starRating": 3.5,
  "subRatings": {
    "easeOfUse": 4.5,
    "qualityOfSupport": 4.0,
    "easeOfSetup": 4.0,
    "meetsRequirements": 3.5
  },
  "reviewLikes": "Pipeline view is clean, sequences are easy to build, and the free tier got us started without procurement.",
  "reviewDislikes": "Once we hit 60 reps the reporting module struggled. Forecasting is weaker than Salesforce and custom objects are limited.",
  "recommendations": "Fine for teams under 50. Above that, evaluate Salesforce or Dynamics.",
  "reviewer": {
    "displayName": "Verified User in Software",
    "jobTitle": "RevOps Manager",
    "industry": "Computer Software",
    "companySize": "51-200 employees",
    "isVerified": true
  },
  "publishedAt": "2025-11-08T14:22:10Z",
  "language": "en",
  "helpfulCount": 12,
  "source": "organic",
  "reviewUrl": "https://www.g2.com/products/hubspot-sales-hub/reviews/hubspot-sales-hub-review-8421930"
}

Fields include reviewId, productSlug, productName, starRating, five sub-ratings, reviewLikes, reviewDislikes, recommendations, problemsSolved, benefitsRealized, reviewer display name, job title, industry, company size, region, validation status, publishedAt, updatedAt, language, helpfulCount, source, incentive (if the review was incentivized), and the canonical reviewUrl.

Four Real Use Cases

1. SaaS Sales Displacement Plays

A sales team selling a CRM builds a nightly job that pulls all 1- and 2-star reviews for three major competitors. Each review is piped through an LLM that extracts the specific complaint and the reviewer's company. The result is a prioritized outbound list where every lead comes with a documented pain point in their own words. Open rates on personalized sequences built from real G2 complaints routinely run 2-3x generic cold outbound.

2. Category-Level Feature Gap Analysis

A product manager at a marketing automation vendor scrapes every review in the "Marketing Automation" category filed in the last 12 months, roughly 40,000 reviews across 30 products. She clusters the "dislikes" text with embeddings and counts cluster frequency per vendor. The result is a heatmap showing which features are consistent weak spots across the category (great roadmap input) and which are only weak for specific competitors (great competitive collateral).

3. Churn and Renewal Risk Signals

A customer success team at an observability vendor subscribes to a rolling scrape of their own product's reviews plus the top five competitors. Any new 1- or 2-star review mentioning an integration or feature their product covers gets routed to a Slack channel. It acts as an early-warning system for account risk and a real-time queue of switch-ready prospects.

4. Private Equity Due Diligence

A PE analyst evaluating a SaaS acquisition scrapes 5 years of G2 history for the target and three comparable vendors. The trend of monthly average star rating, sub-rating deltas, and review volume growth becomes part of the investment memo. This is one of the few ways to reality-check the seller's narrative about product quality and market position.

Pricing

The Actor is priced at $0.005 per review scraped, billed through Apify. Platform compute usage is negligible because there is no browser.

Worked examples:

1,000 reviews: $5
10,000 reviews (a large product's full history): $50
100,000 reviews (a full category sweep): $500
1,000,000 reviews (multi-category enterprise pull): $5,000

Compare that to a DIY build where a single month of residential proxies for the same volume runs $1,500 to $3,000, plus engineering time to keep the Kasada bypass alive. For most teams the break-even point is well under a week.

FAQ

Is scraping G2 reviews legal?
G2 reviews are publicly accessible and the Actor only collects data that any logged-out visitor can see. Public data scraping has been repeatedly upheld in US courts (hiQ v. LinkedIn is the landmark). That said, how you use and redistribute the data is on you. Do not republish full review text as your own content, and respect GDPR if you process reviewer metadata for EU subjects.

Do I need proxies or a Kasada solver?
No. The Actor uses G2's public review feed directly and does not trip Kasada. You do not need to supply proxies, browser fingerprints, or CAPTCHA solver tokens.

How fresh is the data?
Reviews are scraped live at run time. If a review was published five minutes before your run, it will be included. For ongoing monitoring, schedule the Actor hourly or daily via Apify Schedules.

What is the rate limit?
Practically speaking, you are limited by Apify concurrency and the Actor's internal pacing, not by G2 blocking. Expect roughly 500-1,000 reviews per minute per run. Runs can be parallelized across products.

Can I get all 29 fields or is that a premium tier?
All 29 fields are included at the flat $0.005 per review rate. There is no feature-gated premium tier.

How do I export to my warehouse?
Apify exposes datasets as JSON, CSV, XLSX, JSONL, RSS, and HTML table, and has native integrations for S3, Google Drive, and webhooks. A common pattern is JSONL to S3, then COPY into Snowflake or BigQuery.

Pair It With Capterra for Full B2B Coverage

G2 skews toward mid-market and enterprise SaaS. Capterra, owned by Gartner, leans more toward SMB and has broader coverage of vertical software (construction, healthcare, legal). For any serious competitive intel project, you want both. The companion Actor zhorex/capterra-reviews-scraper uses the same schema philosophy and pairs cleanly with this one in a single pipeline. If you are also tracking sentiment on Chinese-language software or consumer platforms, zhorex/weibo-scraper covers the APAC side.

Get Started

The Actor page, full input schema, and a free trial run live at:

https://apify.com/zhorex/g2-reviews-scraper

Drop in a product URL, run it, and you will have a clean JSON dataset in the Apify console in a couple of minutes. No proxy contract, no Kasada cat-and-mouse, no maintenance bill at 3 AM.