Forem: SIÁN Agency

Rate Limits Are a Feature, Not a Bug

SIÁN Agency — Thu, 07 May 2026 05:39:33 +0000

Most scraper "incidents" I'm pulled into start the same way: someone shows me a graph of 429 responses and asks how to make them go away. The honest answer — that nobody likes — is that the 429s are the well-behaved part of the system. The rest is what's broken.

I'm going to argue that rate limits are not your enemy. They're a contract. And scrapers that treat them like a contract — instead of an obstacle — are the only ones I trust to run unsupervised for more than a quarter.

The teardown

Three things teams typically do when they hit rate limits, in order of how bad they are:

Add proxies. "If they limit me, I'll just be more people." This works for about six weeks. Then the target site fingerprints your residential proxy pool and you're back to where you started, with a higher monthly bill.
Decrease delays. "If we go faster, we'll finish before they notice." Faster only matters if the request budget exists. Going faster against a hard limit just stacks failures earlier.
Retry harder. Add exponential backoff with a 30-minute cap. Now your "1-hour scraper" is a 4-hour scraper that completes when the throttle window expires.

All three are forms of the same denial: refusing to accept that the source site is telling you the rate at which they're willing to serve you data. They are. You should listen.

What rate limits actually are

A rate limit is the source-site engineer's way of saying: here is the contract under which my system stays healthy. They published the rate (often: in headers) because they've measured what their infrastructure can serve before things degrade. When you exceed it, you don't just hurt yourself — you contribute to the conditions that get scrapers blocked entirely.

There are three signals you should be reading from every response, not just the body:

Retry-After header. This is the source telling you, in seconds, when it'll talk to you again. Respect it literally.
X-RateLimit-Remaining (or equivalent). Some sites publish their budget. Use it. Slow down before you hit zero.
Status code distribution over time. If your 200 rate is dropping while 429 rises, you're approaching a soft limit you can't see. Back off proactively.

If you're not reading those, your scraper is operating blind against an opponent who is leaving the lights on for you.

The replacement pattern

Here's the rate-aware request loop I drop into every actor:

import asyncio, time
from collections import deque

class RateBudget:
    """Token bucket — refills at `rate` per second, max `burst` tokens."""
    def __init__(self, rate: float, burst: int):
        self.rate = rate
        self.burst = burst
        self.tokens = burst
        self.last_refill = time.monotonic()

    async def take(self):
        while self.tokens < 1:
            await asyncio.sleep((1 - self.tokens) / self.rate)
            now = time.monotonic()
            self.tokens = min(self.burst,
                              self.tokens + (now - self.last_refill) * self.rate)
            self.last_refill = now
        self.tokens -= 1

async def fetch(url, budget, session):
    await budget.take()
    resp = await session.get(url)
    if resp.status == 429:
        retry_after = int(resp.headers.get('Retry-After', '60'))
        await asyncio.sleep(retry_after)
        return await fetch(url, budget, session)
    return resp

Three things this does that "decrease the delay" doesn't:

Token bucket means the rate is global, not per-request. Concurrency works without exceeding the contract.
Retry-After is honoured literally. No exponential backoff guessing — the source already told you.
No proxy rotation. You don't need to be more people. You need to be one well-behaved person.

Result

On the two scrapers I migrated to this pattern this quarter:

Idealista. 429 rate dropped from 8% to 0.4%. Total run time went up by 11% (from 47min to 52min average) — because we stopped hammering. Per-run cost went down 38% — because we stopped paying for retries that were never going to succeed.
Sephora. 429 rate from 15% to <1%. Run time about the same. Block rate (full IP block requiring rotation) went from "monthly" to "zero in the last 90 days." This one's the real win — we used to burn a residential proxy pool subscription. Now we don't need it.

The pattern that emerges every time: respecting the rate makes you slower per-request, but more reliable per-run, and significantly cheaper per-result. The unit economics of a polite scraper beat the unit economics of an aggressive one. By a lot.

When it's wrong

This is wrong if the source site doesn't publish a contract — no Retry-After, no rate header, just blanket blocks. There you genuinely are guessing. But the guess should still bias toward "much slower than you think you need to be," not toward "more proxies." A token bucket at 1 req/sec is a fine starting point for an unknown site; you can ratchet up while watching error rates.

This is also wrong if you have explicit business permission to scrape at higher rates — a partnership, an API key, a contract. Those are different relationships. The advice here is for scrapers running against the public web, where 429 is the only contract you have.

Closing

Stop thinking of rate limits as the cost of doing business. Start thinking of them as a free service the target site is providing you: telling you exactly how to stay welcome. Most blocked scrapers I see were blocked not because they "got caught" — they were blocked because they ignored repeated, clearly-articulated signals that they were being rude.

We packaged the token bucket + Retry-After honour into a small middleware that sits in front of every actor we ship — visible across our Apify portfolio. About 30 lines of code. It's the most boring reliability win I've shipped this year, and the most consistent.

Which response header is your scraper currently ignoring? Drop it in the comments — I'll show you what to do with it.

Written by **Jonas Keller, Senior Automation Architect at SIÁN Agency. Find more from Jonas on dev.to. For custom scraping or automation work, hire SIÁN Agency.

Instagram Reel Transcripts in 5 Lines — and Word-Level Timestamps Are Free

SIÁN Agency — Sat, 02 May 2026 04:54:03 +0000

If you've ever priced Instagram transcription at scale, you already know the trap: per-video pricing on the SaaS tier, plus an upcharge for word-level timestamps. Run the math on 500 reels and you'll close the tab.

I'm not going to talk you out of building your own pipeline. I'm just going to show you the five lines I run when I don't want to.

The trap: per-URL pricing on transcript metadata

Most Instagram transcription APIs in 2026 charge:

A base rate per processed video.
Sometimes a separate rate per minute of audio.
An additional fee to expose word-level timestamps (the thing you actually need if you're building captions, search, or any kind of clip editor).

That works for a single creator's library. It does not work for an agency processing client A's 200 reels, then client B's 1,000.

The five-line replacement

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("sian.agency/instagram-ai-transcript-unlimited").call(run_input={
    "bulkUrls": ["https://www.instagram.com/reel/DG06PnPT9aT/"],
    "wordLevelTimestamps": True,
})
print(next(client.dataset(run["defaultDatasetId"]).iterate_items())["transcript"])

Three input fields you actually need:

instagramUrl (string) — single reel or video post. Pattern enforced; /reels/ auto-corrects to /reel/.
bulkUrls (array) — paste 1, paste 1,000. Bulk edit, .txt upload, manual list. Same input shape regardless of volume.
wordLevelTimestamps (boolean, default true) — get a per-word timestamp on every transcript. Free. You don't pay extra for it.

That third one is the point of this post. It's on by default. Most tools hide it behind a paywall. This one doesn't.

What you can't transcribe

Be honest about the constraints up front:

Image carousels — no audio, nothing to transcribe.
Music-only videos — no spoken audio, the transcript will be empty.
Private profiles — Instagram blocks scraping public-side, so the actor only handles public reels and posts.

If you're building a "scrape any Instagram URL" feature, you'll hit those edges. The actor returns a clear error per URL — handle it client-side and skip silently.

Why "unlimited" is a real claim, not marketing

The actor doesn't charge per validated URL. It charges for compute time per run. If you're processing 1,000 reels in one batch, that's one run. The pricing model rewards batching, which is what you want anyway — bulk is faster than serial because the runtime queue stays warm.

I migrated an agency client's Instagram audit workflow last week. Old setup: a per-video API at $0.05 + $0.02 word-timestamp upcharge — $35 for 500 reels per audit. New setup: one bulk run, predictable monthly compute. Roughly 1/4 the cost at their volume, and the dataset shape is identical.

What to do next

If you want to see what 30+ data points + word-level transcripts look like for your own client list, run it once: Instagram AI Transcript Unlimited.

Single-URL test costs less than a coffee. Bulk run is unlimited.

Tell me where this breaks. If you've found a public reel format the URL pattern misses, drop it in the comments. I'll get the maintainer to ship a fix in the next build.

Written by **Nova Chen, Automation Dev Advocate at SIÁN Agency. Find more from Nova on dev.to. For custom scraping or automation work, hire SIÁN Agency.

I Stopped Writing TikTok Scrapers. Five Lines of Python Replaced Them.

SIÁN Agency — Mon, 27 Apr 2026 13:34:57 +0000

If your TikTok scraper still uses Playwright + custom selectors, this post will annoy you. Good. Read it anyway.

I burned three weekends last quarter on a "minimal" TikTok scraper. Selector-first, headless, the works. Worked beautifully for nine days. Then TikTok shipped a layout change at 2am UTC and my fixtures became fiction.

The honest answer most devs avoid: for known platforms with stable APIs around them, you should not be writing the scraper. You should be calling someone's actor.

Stop owning the layer that breaks

Three things break a TikTok scraper, and none of them are about your code:

Layout drift. Selectors are a liability the second TikTok touches the DOM.
Auth + rate-limit games. Cloudflare, fingerprinting, the whole party.
Audio extraction + transcription. Even if you got the video, now you need Whisper, ffmpeg, a queue, and a dead body to bury when it OOMs.

You're not getting paid to maintain that. You're getting paid to ship the thing on top of it.

What replaced 800 lines of Python for me

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("sian.agency/best-tiktok-ai-transcript-extractor").call(
    run_input={"bulkUrls": ["https://www.tiktok.com/@user/video/7565659068153531669"]}
)
print(list(client.dataset(run["defaultDatasetId"]).iterate_items()))

That's the whole thing. Five lines. The actor's input schema has exactly two fields you need to know about:

tiktokUrl (string) — single video. Pass any URL format. Short links from vm.tiktok.com get resolved. Mobile share URLs work.
bulkUrls (array) — paste 5, 50, or 500. Bulk edit, file upload, line-separated, comma-separated. It doesn't care.

That's the entire input surface. Two keys. No proxy config, no captcha settings, no "headless or headful" debate.

What you get back

Per video, you get the AI transcript (99%+ accuracy claimed by the actor — empirically I see ~98% on English, lower on heavy slang) plus 45 metadata fields: views, likes, shares, creator stats, hashtags, music ID, location, content categories. The transcript ships with detected language and segment timing, so you can search inside videos like text.

I rewrote a competitor-monitoring pipeline last month using this. Old stack: Playwright cluster + Whisper container + Redis + a cron + a Slack channel where I apologized weekly. New stack: a 60-line Python script and the actor. Same dataset, less surface area, no apologies.

The objection I keep getting

"Why pay per run when I can self-host?"

Because your time isn't free, and you don't actually self-host — you self-rebuild every two weeks when something shifts. The actor charges per validated result. You only pay for the runs that gave you usable data. That's a different cost model than "compute hours your worker spent crashing."

If your volume is genuinely huge, sure, build it. But "huge" is an engineering decision, not a default.

Try it on your own URL

The free tier handles 5 videos per run, 8s delay between them. If you want to see the dataset shape for your own use case, drop a TikTok URL in and watch it run: TikTok AI Transcript Extractor on Apify.

Bulk mode is paid — unlimited per run, no delays, no per-video charges. Use it when you're past the experiment phase.

Disagree? Drop the snippet you're using to scrape TikTok in the comments. I'll tell you which line is going to break first. Be specific — "I use Puppeteer" is not a snippet.

Written by **Nova Chen, Automation Dev Advocate at SIÁN Agency. Find more from Nova on dev.to. For custom scraping or automation work, hire SIÁN Agency.