Forem: Sergey Shmakov

How RunPod FlashBoot Actually Works (4-Request Test)

Sergey Shmakov — Tue, 26 May 2026 00:00:00 +0000

Last Updated: 2026-05-27

If you’re shipping vLLM or any heavy ML model on RunPod Serverless, you’ve probably looked at FlashBoot, ticked the checkbox, and then watched your cold starts still take 60-120 seconds. RunPod’s marketing says “1-second cold starts.” Their docs describe FlashBoot as “pre-loading container images.” Neither of those matches what most ML workloads actually see.

I ran four cold-start tests on a deployed RunPod endpoint serving a vLLM-backed PDF parser. The wall-clock numbers ranged from 7 seconds to 7 minutes. The point of this post is to explain why — what FlashBoot actually does at the systems level, when it kicks in, and how to set up your worker so it kicks in more often.

What does FlashBoot actually do?

FlashBoot is a CRIU-style process snapshot mechanism. When a worker scales to zero, RunPod captures the full process state (Python interpreter, CUDA VRAM, subprocess tree) into a snapshot on the host’s local storage. When the worker scales back up on the same host, RunPod restores from that snapshot. The restored process resumes mid-stride: model still in VRAM, vLLM engine subprocess still alive, IPC pipes still connected.

The key qualifier that RunPod’s docs don’t mention: snapshots are per (host, image SHA), not per endpoint. If the next scale-from-zero lands on a different host, there’s no snapshot to restore from. The worker boots fresh and pays the full warmup cost. Once.

The TL;DR for an ML workload: set up an eager warmup at worker boot, then let FlashBoot do its thing. Each new host pays the warmup tax once. Subsequent scale-from-zeroes on that same host get the snapshot restore and finish a typical request in single-digit seconds.

Why do “cold” starts sometimes take 7 seconds and sometimes 110?

Because they’re hitting different parts of the per-host model. Four consecutive requests against the same endpoint, single-page parse on each, with a deliberate scale-to-zero between every one:

Request	Wall-clock	Host	Snapshot?	What the worker did
1	456 s	A (post-rebuild)	none	Image pull + fitness checks + warmup (101 s) + parse (5.6 s)
2	7.6 s	A (same as R1)	yes	Snapshot restore + parse (4.7 s)
3	122 s	B (different host)	none	Fitness checks + warmup (101.5 s) + parse (5.6 s)
4	7.4 s	B (same as R3)	yes	Snapshot restore + parse (4.6 s)

First hit on a fresh host pays ~110 s for the warmup. Every subsequent restore on that same host is ~7-8 s. A new host, when RunPod’s scheduler picks one, starts the cycle over.

The 456 s on Request 1 included a one-time image pull (the worker image is ~27 GB; this was the first time that physical host had ever seen it). Strip that off and you get ~110 s of actual boot work, which matches Request 3 exactly.

How can you tell if a request hit a snapshot restore?

By what’s missing from the worker logs. A FlashBoot-restored worker skips its boot sequence entirely — no fitness checks, no Python import logs, no vLLM engine initialization, no model load. The first log line is Jobs in queue: 1, immediately followed by your handler’s “starting job” entry.

Compare a fresh boot to a snapshot restore for the same request shape:

Fresh boot (Request 3):

04:45:45 Running 7 fitness check(s)...
04:45:46 All fitness checks passed. (1285.99ms)
04:45:46 [mineru-warmup] starting (backend=vlm-auto-engine ...)
04:45:51 Using vllm-async-engine as the inference engine for VLM.
04:46:23 Initializing a V1 LLM engine (v0.11.2) ...
04:46:47 Model loading took 2.1601 GiB memory and 18.41 seconds
04:47:14 torch.compile takes 22.81 s in total
04:47:17 init engine (profile, create kv cache, warmup model) took 30.66 seconds
04:47:18 get vllm-async-engine predictor cost: 87.26s
04:47:28 [mineru-warmup] done in 101.5s
04:47:28 Jobs in queue: 1
04:47:28 Started.
04:47:28 "starting job" {...}
04:47:34 "done" {...elapsed_seconds: 5.58...}

Snapshot restore (Request 4):

04:51:25 Jobs in queue: 1
04:51:25 Started.
04:51:25 "starting job" {...}
04:51:26 Using vllm-async-engine ... (instant — engine handle restored from snapshot)
04:51:30 "done" {...elapsed_seconds: 4.58...}

No boot sequence. Three timestamps. The vLLM engine subprocess PID from the previous boot is reused — same EngineCore_DP0 pid=NNN from the snapshot. If you grep your own worker logs for the gap between Jobs in queue: 1 and the previous activity, you’ll see whether RunPod did a fresh boot or a snapshot restore.

What does the FlashBoot snapshot preserve?

Everything that lived in the worker process at snapshot time, mediated by CRIU semantics:

Python interpreter state. Module imports stay loaded. Globals (job counters, contextvars, signal handlers) keep their values. The MinerU engine registry returns the same handles it returned before the snapshot.
GPU VRAM. Model weights (~2.16 GiB for our VLM), vLLM’s KV cache (~8.17 GiB on a 24 GB card), and captured CUDA graphs (~0.3 GiB) all survive. The first request after restore parses with the same allocations it had before.
The subprocess tree. vLLM runs its engine in a child process for memory isolation. That subprocess gets captured along with the parent and restored with its IPC pipes intact. The engine PID persists.
torch.compile cache. The JIT-compiled Dynamo / Inductor output stays valid across restore. No 22-second recompile.

What doesn’t survive: snapshot lifetime is limited. RunPod doesn’t publish the eviction policy, but obvious triggers include image rebuild (new SHA invalidates the snapshot), and presumably long enough idle on a busy host that the snapshot storage gets pushed out.

What broke before this worked? The asyncio gotcha

The “eager warmup at boot” idea is obvious in principle: run one throwaway parse during worker startup so the model is loaded and warm before any user request arrives. The implementation has one trap.

vLLM’s AsyncLLMEngine binds its IPC primitives (transports, queues) to the asyncio event loop that initialized it. If you call asyncio.run(warmup()) followed by runpod.serverless.start(), your warmup creates loop A, runs the parse, then tears loop A down when asyncio.run returns. Then runpod.serverless.start() creates loop B for serving. When the first user request tries to talk to the vLLM engine through loop B, the engine handle is bound to the now-dead loop A. Result:

"error_type": "EngineDeadError",
"error_message": "EngineCore encountered an issue. See stack trace (above) for the root cause."

The engine subprocess itself is still alive. It’s only the parent process’s IPC reference that’s broken.

The fix is to keep the warmup and the serve loop on the same asyncio event loop. RunPod’s runpod.serverless.start() internally calls asyncio.run(JobScaler.run()), but JobScaler (in runpod.serverless.modules.rp_scale) is constructible directly and its run() is an awaitable coroutine. So you can compose:


import asyncio

from runpod.serverless.modules import rp_ping, rp_scale

from runpod.serverless.modules.rp_fitness import run_fitness_checks

config = {"handler": handler, "concurrency_modifier": _concurrency_modifier, "rp_args": {}}

async def _bootstrap():

    await run_fitness_checks()

    await warmup_async() # <- engine binds to THIS loop

    rp_ping.Heartbeat().start_ping()

    await rp_scale.JobScaler(config).run() # <- and stays here

asyncio.run(_bootstrap())

Now both phases share one event loop. The engine handle stays valid across the warmup → serve transition. FlashBoot captures a snapshot of a process where the loop, the engine, and the IPC are all alive together. On restore, they come back together too.

This does reach into runpod-python’s internals (the runpod.serverless.modules.* submodules aren’t part of the documented public API). Cheap to guard against drift: a unit test that asserts JobScaler exists with the expected constructor and an awaitable run() method. If RunPod refactors, CI catches it before production does.

When does the warmup pay off and when doesn’t it?

Per host, not per endpoint. The math depends on your traffic pattern.

Scenario	Likely outcome
`workers_min ≥ 1` (always-on worker)	Worker stays on its host. Every request is on a fully warm worker (~5 s parse). No cold starts at all.
High-frequency endpoint, workers scale up and down fast	Same hosts get re-selected. Most cold starts are happy-path restores (~7 s).
Quiet endpoint, infrequent requests, long idle gaps	RunPod’s scheduler may pick a different host. Some cold starts will be on new hosts (~110 s).
First request after a rebuild	Always cold path. Every endpoint’s first request after a fresh image pays ~5-7 min (image pull) + ~110 s (warmup). One-time per worker host.
`MINERU_SKIP_WARMUP=1` (warmup off)	Every cold start is ~110-130 s. No per-host amortization. Don’t do this in production.

The case that stings is “quiet endpoint with sporadic traffic” — a few requests an hour, 10-minute idle gaps, RunPod bouncing between hosts. Without warmup, every cold start would be ~110-130 s. With warmup, you get a mix: some 7-second restores, some 110-second fresh boots. The mix tilts toward fast as the endpoint warms up across more hosts and RunPod’s scheduler starts re-selecting them.

If your traffic is sustained enough that you can pin a worker (workers_min=1), you skip the entire question. You’re paying for the GPU 24/7 but never paying a cold start. For workloads with even modest cost sensitivity, the warmup + FlashBoot path is the better trade.

What this means if you’re shipping vLLM on RunPod

Three takeaways from the live measurements:

Always set up an eager warmup at worker boot. Loading the model on first request is silently worse than it sounds — you don’t just pay 110 s once per cold start, you pay it every time a host doesn’t have a snapshot, AND you forfeit the per-host amortization that makes the second-hit-on-a-host cheap. Without warmup, FlashBoot has nothing to snapshot.
Compose warmup and the serving loop under one asyncio.run(). If you asyncio.run() the warmup separately, the engine dies at the loop boundary. The fix is straightforward but the failure mode is opaque (EngineDeadError 75 ms into the first request) — easy to misdiagnose as a vLLM bug.
Don’t market your cold start as “X seconds” without acknowledging the per-host mix. A snapshot-restore cold start is genuinely 7-8 seconds. A new-host cold start is ~110 s. Both are big improvements over the no-warmup baseline (~110-130 s per request, every request). But your users will see the mix, and a too-clean claim makes the bad days look broken.

The whole investigation was on a 24 GB A5000 / RTX 4090 class GPU running MinerU’s 1.2B VLM via vLLM 0.11.2. The numbers will shift on larger models (more VRAM to snapshot, longer model load on cold path) but the mechanism applies the same way. If your cold start dominates wall-clock latency on a serverless GPU workload, set up boot-time warmup, watch the worker logs for the snapshot pattern, and tune your workers_min accordingly.

FAQ

Does FlashBoot snapshot the vLLM engine subprocess?

Yes. The vLLM engine runs as a child process for memory isolation, and FlashBoot’s CRIU-style mechanism captures the full process tree including subprocesses. The engine’s PID persists across snapshot/restore, and its IPC pipes back to the parent stay connected.

Why does my cold start take 60-120 seconds even with FlashBoot enabled?

Most likely your model is being loaded lazily on first request rather than at worker boot. FlashBoot only snapshots state that already exists in the worker process when it scales to zero. If your model loads on first request, the snapshot captures a worker without the model, and every cold start has to load the model again. Move the model load to worker boot (before runpod.serverless.start()) and FlashBoot will start carrying the warm state forward.

What’s the difference between FlashBoot and a network volume?

A network volume is shared file storage attached to your worker (e.g., for model weights you don’t want to bake into the Docker image). FlashBoot is process-state preservation — it captures the running Python process, including data already loaded from disk into VRAM. They solve different problems and can be used together: a network volume avoids re-downloading model files on image pull; FlashBoot avoids re-loading them into VRAM on cold start.

Does FlashBoot work for non-GPU workloads?

The mechanism (process snapshot via CRIU or equivalent) doesn’t depend on GPU memory specifically. CPU-bound workloads with significant cold-start cost (heavy library imports, large in-memory indices, JIT compilation) should benefit similarly. The framing in this post happens to use a GPU workload because that’s where the cold-start tax is most painful.

How do I know if my worker is hitting a snapshot restore vs a fresh boot?

Check the worker logs in the RunPod dashboard. A fresh boot shows fitness checks, framework init logs, and any warmup output. A snapshot restore is silent until the first Jobs in queue: 1 line, then jumps straight to your handler’s request-processing logs. The presence or absence of the boot sequence is the cleanest signal.

Is FlashBoot the same as RunPod’s “Active Workers” tier?

No. Active Workers are a billing tier where you pre-commit to a number of workers that are always on, billed at a discount in exchange for the 24/7 commitment. FlashBoot is a free runtime optimization that applies to flex (scale-to-zero) workers. The two can be combined: an Active Worker on the same endpoint can also benefit from FlashBoot when it cycles, though for a worker that never goes idle there’s nothing to snapshot.

Will FlashBoot survive a Docker image rebuild?

No. Each image gets its own SHA, and FlashBoot snapshots are scoped to (host, image SHA). When you push a new image, all existing snapshots are invalid. The first request after a rebuild on any host pays the full cold-start cost (image pull + warmup). Once each host has served the new image once, subsequent restores work normally.

What’s next

The runpod-mineru repo wraps all of this into one Docker image: MinerU 3.2.x + the MinerU2.5-Pro-2605-1.2B VLM, the JobScaler-bypass composition for warmup, structured logging, and the rest. Open source (GitHub), MIT-licensed, deploys from the RunPod Hub in two clicks.

If you want the deeper breakdown of which phases of a cold start cost what, the troubleshooting guide has the per-phase timing table from the same test runs. The scaling guide covers when to pair FlashBoot with workers_min ≥ 1 for fully predictable latency.

Disclosure: RunPod links in this post use a referral code that credits me at no cost to you. The post would read the same without it.

RunPod 20 MB Response Cap: Fix NoneType with Cloudflare R2

Sergey Shmakov — Wed, 20 May 2026 00:00:00 +0000

Last Updated: 2026-05-27

If your RunPod serverless worker logs say done but your client raises unexpected handler return type: <class 'NoneType'>, you’ve hit RunPod’s bidirectional 20 MB payload cap on /runsync. The handler succeeded. The gateway dropped the response on the way back because the payload was too large.

The fix is two steps. Set return: "s3" on the job, and configure four env vars on the endpoint pointing at a Cloudflare R2 bucket. The worker uploads the result to R2 and returns a small presigned URL. Your client downloads from R2 directly. No gateway cap in the path.

I hit this on an 82-page Cyrillic fiscal report (30 MB input, ~25 MB output with embedded images) running my open-source mineru-runpod template. Two retries via return: "inline" and return: "tarball_b64" failed the same way. R2 mode worked first try. The rest of this post is the symptom, the env-var recipe, the cost comparison vs S3, and a few gotchas worth knowing.

Why does my RunPod worker return NoneType after a successful parse?

The worker handler completed and returned a valid dict. RunPod’s runtime then tried to POST that result back to RunPod’s API via /job-done, and the API returned HTTP 400 because the payload exceeded ~20 MB. The result was discarded. The SDK saw no output, returned None to the client, and the client wrapper raised the NoneType error.

The worker logs make the chain explicit:

[mineru-worker] done: elapsed=91.77s phase_ms={'fetch_input': 972, 'mineru_parse': 90789, 'package': 66}
{"requestId": "sync-fdcd03cd-...", "message": "Failed to return job results. | 400, message='Bad Request',

 url='https://api.runpod.ai/v2/<endpoint>/job-done/<worker>/sync-fdcd03cd-...?gpu=NVIDIA+RTX+A5000&isStream=false'"}

The first line shows the handler finished cleanly: 82 pages parsed in 91.8 s on the worker (this test ran on A5000; on the current 4090 default the warm parse is 2–3× faster). The second line shows the gateway rejecting the result. The handler already returned and never knows the rejection happened. The SDK sees the discarded result and returns None to your code.

If you see this NoneType error on a small doc, the diagnosis is different (worker OOM, crash, timeout). On a multi-page parse that the worker logs as done, the answer is almost always the 20 MB cap.

What is RunPod’s /runsync response payload limit?

RunPod’s /runsync gateway caps payloads at roughly 20 MB in both directions. The request cap affects file_b64 inline uploads. The response cap affects what the worker can return. Both are independent of execution time and memory budget. A fast, successful parse can hit the response cap simply by producing a large output.

Direction	Limit	What triggers it
Request → gateway → worker	~20 MB	`file_b64` inline transport for large PDFs
Worker → gateway → client	~20 MB	Multi-page parse outputs with embedded images

The request cap is in RunPod’s docs and widely discussed. The response cap is mentioned only in passing. I found three open issues on the runpod-workers repos where other users hit the same symptom and didn’t realise what it was, so this post is partly to make that searchable.

Practical threshold for mineru-runpod: pure-text PDFs are fine for longer. Image-heavy PDFs with embedded raster output hit the response cap around 50–80 pages on inline or tarball_b64 transport.

Does `return: "tarball_b64"` get around the 20 MB cap?

No. return: "tarball_b64" gzips the output into a single .tar.gz before base64-encoding it. Gzip compresses the JSON and Markdown text well, but the page images inside the tarball are already raster bytes (PNG, JPEG) and barely compress further. Multi-page parses with embedded images keep the tarball over 20 MB.

I confirmed this on the same 82-page PDF. Same 400 from /job-done. Same NoneType in the client. Both inline and tarball_b64 route through the gateway response, so both inherit the cap. Only return: "s3" avoids it because the worker uploads out-of-band.

How do I configure Cloudflare R2 to bypass the RunPod response cap?

Set return: "s3" in the job input, then add four env vars on the RunPod endpoint pointing at a Cloudflare R2 bucket. The worker uploads the gzipped tarball directly to R2 and returns a small presigned URL (~1 h TTL). Your client downloads from R2.

The job input changes one field:

{

  "input": {

    "file_url": "https://example.com/big.pdf",

    "return": "s3"

  }

}

The four env vars go on the endpoint (not the template — they’re secrets):

Env var	Cloudflare R2 value
`BUCKET_ENDPOINT_URL`	`https://<account-id>.r2.cloudflarestorage.com`
`BUCKET_NAME`	your bucket name
`BUCKET_ACCESS_KEY_ID`	R2 API token access key
`BUCKET_SECRET_ACCESS_KEY`	R2 API token secret
`BUCKET_REGION` (optional)	`auto`

You generate the access key pair in the Cloudflare dashboard: R2 → Manage R2 API Tokens → Create API Token → Object Read & Write scoped to the bucket. The worker auto-restarts when you save endpoint env vars in RunPod. Test with one small doc before sending production traffic.

Why pick Cloudflare R2 over AWS S3 for RunPod output storage?

R2 has zero egress fees , a 10 GB free storage tier, 1M Class A ops and 10M Class B ops per month free, and is fully S3-compatible. AWS S3 charges egress at roughly $0.085/GB plus storage at $0.023/GB/month. For a RunPod pipeline doing dozens of GB of I/O per month, R2’s bill stays near zero while S3 lands in the $5–$15 range.

A back-of-envelope month for the workload I tested:

1,000 multi-page parses, average output 8 MB → 8 GB stored then deleted
1,000 worker→bucket uploads + 1,000 client→bucket downloads = 2,000 ops
Storage: free (under 10 GB). Egress: free (R2 doesn’t bill egress). Ops: free (well under 1M Class A).

Same workload on S3: ~$0.18 storage + ~$0.68 egress + per-request fees, maybe $1–$3 total. Cheap but R2’s $0 is cheaper.

S3 still makes sense if you’re already deep in AWS, if you need IAM-controlled access patterns, or if RunPod workers and your AWS region are colocated tightly enough that egress doesn’t apply. For everyone else and especially for solo / indie deploys, R2 is the right default. See R2 pricing for current rates.

What does the parse flow look like end-to-end with `return: "s3"`?

The worker fetches the input PDF, runs MinerU, gzips the outputs into a tarball, uploads to R2 via the configured BUCKET_* env vars, and returns a small JSON response with tarball_url, tarball_url_expires_in (3600 s), and bucket_key. Your client follows the URL and extracts the tarball locally. No payload ever crosses RunPod’s 20 MB-capped response path.

Concrete numbers from the 82-page test (on A5000; current default is 4090):


result = client.parse_document(

    file_url="https://pub-....r2.dev/report.pdf",

    backend="vlm-auto-engine",

    return_format="s3",

)
# result["tarball_url"] -> presigned R2 URL, valid ~1 h
# result["tarball_url_expires_in"] -> 3600
# result["bucket_key"] -> "report-<hash>.tar.gz"

client.save_s3_tarball(result, "./out/")

# downloads + extracts -> out/report.md, out/report_content_list_v2.json, out/images/, ...

End-to-end wall-clock: 211 s for an 82-page doc on a cold worker. Breakdown: ~112 s before MinerU started parsing (worker boot + warmup), ~92 s warm parsing (1.1 s/page on A5000), ~11 s gzip and upload to R2 (the package phase). The extracted output: 313 KB Markdown plus structured JSON plus per-page images. Roughly 3.5 minutes for a document that previously couldn’t return its output at all.

The cold-start portion is a separate concern from the response cap. The FlashBoot mechanism investigation covers why the ~112 s exists, how the boot-time warmup interacts with RunPod’s snapshot system, and when subsequent cold starts are much faster.

What should I watch out for with the R2 bridge?

Four things the docs don’t say loudly. The presigned URL TTL is 60 minutes. R2 doesn’t auto-clean uploaded objects. One bucket can serve input and output. The 20 MB cap applies to /run (async) too, not just /runsync.

Presigned URL TTL is 60 minutes. If your client is slow to download (e.g. a job-queue worker that picks up results minutes later), bump _S3_PRESIGN_TTL_SECONDS in the handler. Don’t rely on the default in long-tail flows.
R2 doesn’t auto-clean uploaded objects. Add an R2 lifecycle rule (e.g. delete after 7 days) so your output bucket doesn’t grow forever.
One R2 bucket can serve input and output. Upload your PDFs to R2 ahead of time, pass file_url pointing at the R2 public dev URL, and the worker writes outputs to the same bucket at the root. Add BUCKET_PREFIX env var if you want outputs in a subdirectory.
The 20 MB cap applies to /run (async) too. Same gateway, same limit. Switching to async polling doesn’t help.

FAQ

How do I get the R2 access key for `BUCKET_ACCESS_KEY_ID` and `BUCKET_SECRET_ACCESS_KEY`?

In the Cloudflare dashboard: R2 → Manage R2 API Tokens → Create API Token. Set permissions to “Object Read & Write” scoped to the specific bucket. Cloudflare shows the access key ID and secret access key once; copy both into your RunPod endpoint env vars immediately. The secret isn’t retrievable later.

Does the presigned URL expire?

Yes. The default TTL is 3600 seconds (one hour). If your downstream client picks up the response asynchronously (job queue, cron, etc.), download promptly or bump _S3_PRESIGN_TTL_SECONDS in the worker handler before redeploying.

Can I reuse the same R2 bucket for input and output?

Yes. The worker doesn’t care about the bucket layout. Upload your input PDFs to bucket/inputs/ and the worker writes outputs to bucket/<basename>-<hash>.tar.gz at the root. Add BUCKET_PREFIX env var if you want outputs pushed into a subdirectory.

What if I can’t set up R2? Is there a fallback?

Page chunking. Split the parse with start_page and end_page into segments small enough that each output tarball stays under 20 MB, then concatenate the .md files client-side. Slower (you may pay multiple cold starts if the worker scales to zero between calls) and you handle joining yourself, but no infra changes needed.

Is the 20 MB cap on `/run` too, or only `/runsync`?

Both. RunPod’s /run (async) and /runsync (synchronous) share the same gateway and the same payload limits. Switching to async doesn’t help the response-size problem. The cap is at the gateway layer, not the polling protocol.

Does using `return: "s3"` add to cold-start time?

No. The S3 upload happens at the end of the parse, not the beginning. The handler’s package phase grew from ~95 ms (in-memory tarball) to ~11 s (gzip + upload to R2) on an 82-page job, but cold-start is unchanged. The S3 mode adds a small constant to warm-job latency, not a multiplier.

How big can the R2-uploaded tarball be?

Effectively unlimited for mineru-runpod workloads. R2 supports multipart uploads up to 5 TB per object. You’ll hit the worker’s executionTimeoutMs long before you hit R2’s per-object limit.

Does R2 work for input PDFs too, or only output?

Both. The worker accepts file_url pointing at an R2 public dev URL (or a presigned R2 GET URL for private buckets) and fetches the input from R2. This avoids the inbound 20 MB cap on file_b64 for large PDFs. You can run an R2-in / R2-out setup with one bucket and avoid every payload-size limit RunPod has.

Where to next

If you’ve shipped a multi-page PDF pipeline on RunPod and you’re not using return: "s3", you’ll hit the gateway cap eventually. Set it up before you need it. The cost is ten minutes of env-var configuration and possibly zero dollars per month at indie volumes.

If you’re new to the template, the getting-started guide walks through the full deploy in about ten minutes. For the cold-start side of the picture (separate from the response cap covered here), see the FlashBoot mechanism investigation. For GPU sizing, Choosing a GPU covers when the default ADA_24 (RTX 4090) is enough and when to opt up.

If this saved you time, the easiest way to say thanks is signing up for RunPod through this link. Star the repo on GitHub for updates.

Disclosure: RunPod links in this post use a referral code that credits me at no cost to you. The post would read the same without it.

Serverless MinerU on RunPod: honest cost math (2026)

Sergey Shmakov — Tue, 19 May 2026 00:00:00 +0000

Last Updated: 2026-05-27

If you’re building a RAG pipeline, a document indexer, or any product that ingests PDFs at scale, you’ve probably hit the same wall I did. Hosted OCR APIs charge pennies per page that compound into thousands per million. CPU parsers are too slow for production volume. A permanent GPU pod is wasteful when traffic comes in bursts.

MinerU 2.5 is genuinely state-of-the-art for PDF → Markdown / structured JSON. Apache 2.0 license. The MinerU2.5-Pro-2605-1.2B model fits comfortably on a 24 GB GPU. RunPod Serverless scales to zero when nothing is calling. Wiring those two together is the obvious move.

Real numbers from my open-source mineru-runpod template, measured on a 24 GB RTX 4090 in May 2026: ~$0.001 per page for warm parses, plus a ~$0.03 fixed tax per cold start. The all-in per-page cost depends on how much work you do before the worker scales back to zero. Here’s the deploy, the response shape, and the workload patterns this template is the right fit for.

What does it actually cost to run MinerU on RunPod Serverless?

About $0.001 per page on an RTX 4090 once the worker is warm. Each scale-from-zero adds a ~$0.03 fixed tax : roughly 110 seconds of GPU billing for vLLM engine init plus model load. Per-page math depends entirely on amortization. Sparse traffic with one short request per cold start lands closer to $0.005–$0.01 per page.

Real workload-shape math using ADA_24 (RTX 4090, ~$1.10/hr Flex):

Workload shape	Per-page cost
1,000 pages amortized across one cold start	~$0.001
100 pages amortized across one cold start	~$0.0013
10 pages then idle out	~$0.004
One short doc per scale-from-zero (worst case)	~$0.007

Compared to alternatives:

Tool / setup	Per-page cost	Notes
Hosted OCR APIs (typical)	$0.001 – $0.01	vendor lock-in, rate limits, documents leave your stack
Permanent GPU pod (24 h on A5000)	$0.001 – $0.003	24 h of bills whether you use it or not
mineru-runpod, amortized	~$0.001 – $0.004	scales to zero; cold-start tax is real
Marker / Nougat on CPU	$0 cash, $$$ time	~30 s/page sequential (Marker docs)

The trick is RunPod’s per-second billing. No worker running, no bill. The catch is every scale-from-zero pays a real fixed cost.

How do I deploy MinerU to RunPod Serverless in ten minutes?

Fork the repo, point RunPod’s GitHub auto-build at your fork, create a Serverless Endpoint with ADA_24 (RTX 4090) and FlashBoot enabled, send a request via the included Python client. Total wall-clock from RunPod sign-up to first parse: roughly ten minutes, dominated by the image build (~5–10 min) plus the first cold start (~110 s).

1. Get a RunPod account

2. Fork the repo


gh repo fork sergeyshmakov/mineru-runpod --clone

cd mineru-runpod

The repo stays small. Dockerfile, handler.py, a worker/ package, a Python client (mineru_client), three GitHub Actions workflows, Hub metadata under .runpod/. MIT licensed, ~30 files.

3. Wire RunPod’s GitHub auto-build

In the RunPod dashboard:

Serverless → Templates → New → Import Git Repository
Point at your fork. Branch main, Dockerfile path Dockerfile.
RunPod clones, builds the image, stores it in its own registry, and gives you a template_id. The build runs ~5–10 minutes. Watch the log if you want.

4. Create the endpoint

Dashboard path:

Serverless → Endpoints → New
Template: the one you just created
GPU pool: ADA_24 (RTX 4090, 24 GB)
Workers min: 0, max: 3
Idle timeout: 10 seconds
FlashBoot: on
Save, grab the endpoint id

Or as code (reproducible across redeploys):


pip install -e .[deploy]

python deploy.py --template-id <tid>

deploy.py exposes every endpoint setting as a CLI flag.

5. Parse your first PDF


from mineru_client import MineruClient

client = MineruClient(

    endpoint_id="<your-endpoint-id>",

    api_key="<your-runpod-api-key>",

)

result = client.parse_document(

    file_url="https://example.com/report.pdf",

    end_page=4, # smoke test on first 5 pages

)

client.save_tarball(result, "./out/doc")

# → ./out/doc/<basename>.md
# → ./out/doc/<basename>_content_list_v2.json
# → ./out/doc/<basename>_middle.json
# → ./out/doc/images/*.png

First parse pays a cold start. Subsequent parses on the same warm worker run at ~1–6 s/page on the 4090, content density dependent. After 10 s of idle, the worker scales to zero.

What does the MinerU response actually contain?

Three structured outputs plus extracted images. <basename>.md is Markdown with LaTeX equations, HTML tables, and image references. <basename>_content_list_v2.json is a flat list of typed entries (text, equation, table, image, code) each tagged with page_idx. <basename>_middle.json carries the full layout with bounding boxes and reading order. Pick the transport via return: tarball_b64, inline, or s3.

For a document indexer or RAG pipeline, content_list_v2.json is the file you’ll spend the most time with. Group entries by level: "title" boundaries for section-based chunking. Embed each chunk and store page_idx for citation back to the source.

The Markdown is for human-readable display. middle.json has bounding boxes per span when you need page coordinates for hover-to-source UI.

Transport options on the request: tarball_b64 (default) for outputs under ~20 MB, inline if you want the markdown directly in the JSON response, s3 for anything that would exceed RunPod’s response cap. See the R2 bridge post for the s3 setup.

When does mineru-runpod fit your workload, and when doesn’t it?

Good fit: batch ingest jobs, bursty traffic (50 docs in a minute, then quiet), background pipelines, OCR-API replacement. Poor fit: interactive single-document apps (cold starts make users think it’s broken), sparse traffic (one job per cold start dominates the bill), strict latency SLOs without provisioning workers_min ≥ 1.

I run this template in production for a document indexer. Six months of operation, here’s the honest fit picture:

Good fit:

Batch ingest. Drop 500 PDFs into a queue. One cold start amortizes across the whole batch at ~$0.001 per page.
Bursty traffic. A user uploads 50 documents in a minute. One cold start, 49 warm parses.
Background pipelines. Nightly cron processes yesterday’s intake. Cold start cost is rounding error against a multi-hour batch.
OCR-API replacement. Comparable per-page cost without shipping documents to a third party.

Poor fit:

Interactive single-document parsing. Your user uploads one PDF and waits two minutes for the cold start. They’ll think it’s broken.
Sparse traffic (one job every 20–60 min). Almost every request is a cold start. The ~$0.03 cold-start tax dominates. Rent a permanent low-tier GPU pod and skip serverless instead.
Strict latency SLOs. Cold-start latency is partly outside your control. Provisioning workers_min ≥ 1 eliminates cold starts but you pay for the warm worker around the clock.

The repo’s defaults (workers_min=0, idle_timeout=10s) are tuned for batch-with-bursts. The dashboard’s scaling settings are where you tune for other patterns.

What’s the real cold-start cost on RunPod Serverless?

Roughly 110 seconds before MinerU starts parsing your first request after a scale-from-zero. The composition: ~3 s fitness checks, ~20 s vLLM engine config, ~20 s model load, ~25 s torch.compile, ~5 s CUDA graph capture, ~5 s of actual parse. Billed at ~$1.10/hr on the 4090 default, that’s roughly $0.03 per cold start.

The per-phase breakdown is documented in the troubleshooting guide if you want to see where the time goes. The boot-time warmup in this template loads MinerU’s model and JIT-compiles vLLM kernels before the worker accepts requests. When RunPod’s FlashBoot snapshot is available on a subsequent scale-from-zero, the wall-clock drops to ~7–8 seconds because the snapshot captured a warm process. When the snapshot isn’t available (new host, image rebuild), warmup re-runs and you pay the full ~110 s again.

The FlashBoot mechanism investigation covers when the fast path applies, with measured numbers across multiple consecutive cold starts.

What should I watch out for before going to production?

Three production gotchas the marketing won’t mention. The 20 MB response cap silently drops large outputs (symptom: NoneType after a successful parse — covered by the R2 bridge). execution_timeout defaults to 900 s and won’t cover full books. file_b64 inline payloads cap around 10 MB on the way in. None of these crash the worker; they manifest as confusing client-side errors.

20 MB response cap. RunPod’s /runsync gateway drops responses over ~20 MB. Multi-page parses with embedded images hit this around 50–80 pages. Worker logs done; client gets NoneType. Fix: return: "s3" + Cloudflare R2, walked through in the R2 bridge post.
Long-job timeout. Repo defaults execution_timeout=900s (good for ~150–300 pages on 4090). A 5,000-page book is 80–500 minutes depending on content density. Bump execution_timeout for long jobs; the endpoint upper limit is 24 hours.
Inline payload cap on the way in. file_b64 requests cap around 10 MB. For bigger files, pass file_url and let the worker fetch from your storage. R2 public dev URLs work well.
Cold-start economics. “Pennies per page” depends on amortization. Track average pages per cold start in your logs. If it’s under 30, bump idle_timeout or run workers_min=1.

Where to next

The repo ships with:

Typed Python client (MineruClient)
deploy.py / destroy.py for endpoint lifecycle automation
Reference adapter pattern for wrapping MinerU output into domain models
96 unit tests, CI on every PR
Commitlint + semantic-release for automated CHANGELOG / GitHub Releases

For the deeper context that didn’t fit:

How RunPod FlashBoot actually works — four-request investigation into the cold-start mechanism and the per-host snapshot caveat.
The R2 bridge for the 20 MB response cap — fix for NoneType on multi-page outputs.
Choosing a GPU — when 24 GB is enough, when to opt up to 48 GB.

If this saved you time, the easiest way to say thanks is signing up for RunPod through this link. Star the repo on GitHub for updates.

FAQ

How does mineru-runpod compare to hosted PDF APIs?

Per-page cost is in the same ballpark ($0.001–$0.004) when amortizing cold starts across reasonable batches. The differences are control and lock-in. You deploy your own RunPod endpoint, pick your GPU and concurrency, run whichever MinerU version you want, and never send documents to a third party. The trade-off is operating a serverless template instead of consuming a managed API.

Can MinerU 2.5 handle non-English PDFs?

Yes. The vlm-auto-engine default backend handles English and Chinese well per the model card. For other scripts (Cyrillic, Arabic, Devanagari, Japanese, Korean), the pipeline backend uses PaddleOCR with script-family models, covering 109 languages. Empirically the Pro VLM also handles Cyrillic correctly even though lang is ignored on the VLM path. Switch backends per-request via the backend field.

What’s the difference between `vlm-auto-engine`, `pipeline`, and `hybrid-auto-engine`?

vlm-auto-engine uses MinerU’s 1.2B VLM via vLLM. Fastest on English / Chinese, ~1–6 s/page warm. pipeline uses PaddleOCR plus dedicated layout / formula / table models. Slower (~3–5 s/page) but more memory-predictable (4 GB minimum VRAM) and covers 109 languages. hybrid-auto-engine routes each page through either backend based on content. Highest quality on mixed-content docs; needs 48 GB on dense layouts.

Does the per-page cost include the cold-start tax?

No. The ~$0.001 per page is warm-worker math. Each scale-from-zero adds a roughly $0.03 fixed cost on the 4090 default. Your effective per-page cost is (0.001 × pages) + (0.03 × cold_starts) / pages. For 100 pages across one cold start, that’s $0.0013 per page. For 10 pages, it’s $0.004.

Can I use mineru-runpod with my own MinerU model?

Yes. Fork the repo and update the Dockerfile’s huggingface_hub.snapshot_download call to point at your model. Rebuild and redeploy. The handler is model-agnostic; MinerU’s aio_do_parse resolves whatever model is in HF_HOME at runtime.

What GPU does the template default to?

ADA_24 (RTX 4090, 24 GB). Switched from AMPERE_24 (A5000) on 2026-05-26 after measuring per-page cost. The 4090 is 2–4× faster per page than the A5000 and cheaper per page despite the higher hourly rate. See Choosing a GPU for the full math and when to opt up to 48 GB.

How do I keep my RunPod endpoint warm to avoid cold starts?

Set workers_min=1 on the endpoint. You pay for the always-on worker around the clock (~$0.000306/s on the 4090 default, so ~$26/day or ~$800/month). Worth it if your traffic is steady enough that the warm worker stays busy, or if your latency SLO can’t tolerate the cold-start window. For bursty traffic, workers_min=0 with FlashBoot enabled is usually cheaper.

Disclosure: RunPod links in this post use a referral code that credits me at no cost to you. The post would read the same without it.

Forem: Sergey Shmakov

How RunPod FlashBoot Actually Works (4-Request Test)

What does FlashBoot actually do?

Why do “cold” starts sometimes take 7 seconds and sometimes 110?

How can you tell if a request hit a snapshot restore?

What does the FlashBoot snapshot preserve?

What broke before this worked? The asyncio gotcha

When does the warmup pay off and when doesn’t it?

What this means if you’re shipping vLLM on RunPod

FAQ

Does FlashBoot snapshot the vLLM engine subprocess?

Why does my cold start take 60-120 seconds even with FlashBoot enabled?

What’s the difference between FlashBoot and a network volume?

Does FlashBoot work for non-GPU workloads?

How do I know if my worker is hitting a snapshot restore vs a fresh boot?

Is FlashBoot the same as RunPod’s “Active Workers” tier?

Will FlashBoot survive a Docker image rebuild?

What’s next

RunPod 20 MB Response Cap: Fix NoneType with Cloudflare R2

Why does my RunPod worker return NoneType after a successful parse?

What is RunPod’s /runsync response payload limit?

Does return: "tarball_b64" get around the 20 MB cap?

How do I configure Cloudflare R2 to bypass the RunPod response cap?

Why pick Cloudflare R2 over AWS S3 for RunPod output storage?

What does the parse flow look like end-to-end with return: "s3"?

What should I watch out for with the R2 bridge?

FAQ

How do I get the R2 access key for BUCKET_ACCESS_KEY_ID and BUCKET_SECRET_ACCESS_KEY?

Does the presigned URL expire?

Can I reuse the same R2 bucket for input and output?

What if I can’t set up R2? Is there a fallback?

Is the 20 MB cap on /run too, or only /runsync?

Does using return: "s3" add to cold-start time?

How big can the R2-uploaded tarball be?

Does R2 work for input PDFs too, or only output?

Where to next

Serverless MinerU on RunPod: honest cost math (2026)

What does it actually cost to run MinerU on RunPod Serverless?

How do I deploy MinerU to RunPod Serverless in ten minutes?

1. Get a RunPod account

2. Fork the repo

3. Wire RunPod’s GitHub auto-build

4. Create the endpoint

5. Parse your first PDF

What does the MinerU response actually contain?

When does mineru-runpod fit your workload, and when doesn’t it?

What’s the real cold-start cost on RunPod Serverless?

What should I watch out for before going to production?

Where to next

FAQ

How does mineru-runpod compare to hosted PDF APIs?

Can MinerU 2.5 handle non-English PDFs?

What’s the difference between vlm-auto-engine, pipeline, and hybrid-auto-engine?

Does the per-page cost include the cold-start tax?

Can I use mineru-runpod with my own MinerU model?

What GPU does the template default to?

How do I keep my RunPod endpoint warm to avoid cold starts?

Does `return: "tarball_b64"` get around the 20 MB cap?

What does the parse flow look like end-to-end with `return: "s3"`?

How do I get the R2 access key for `BUCKET_ACCESS_KEY_ID` and `BUCKET_SECRET_ACCESS_KEY`?

Is the 20 MB cap on `/run` too, or only `/runsync`?

Does using `return: "s3"` add to cold-start time?

What’s the difference between `vlm-auto-engine`, `pipeline`, and `hybrid-auto-engine`?