Why your Vercel bill is higher than expected (hint: AI bots)

PlainTOS App — Tue, 12 May 2026 18:45:24 +0000

I run a small side project on Vercel. Last month
my bandwidth bill was 3x higher than usual. No
viral traffic, no new features. Just bots.

Here is what I found and how to fix it.

The "Hidden" Bandwidth Tax

In 2026, the internet is no longer just for humans.
AI crawlers — GPTBot, ClaudeBot, Bytespider, and
PerplexityBot — are aggressively indexing the web
to train new models and power real-time search
engines.

While traditional search engines like Google are
relatively polite, AI bots are often greedy. They
don't just index your home page — they hit your
dynamic routes, parse your JSON, and re-crawl your
assets multiple times a day.

Why this kills you on Vercel

Vercel is incredible for performance, but bandwidth
egress (the data sent from your server to the
visitor) is where costs scale.

A polite bot: Hits your page once a week = negligible cost
An AI swarm: Hits your high-data routes 20 times an hour = a $50 bill by end of week

How to see the "ghost" traffic

Most developers just look at total request counts.
You need to look at the User-Agent strings in your
logs.

If you see Bytespider (the TikTok bot) or
GPTBot appearing thousands of times, you are
effectively paying Vercel to serve data that trains
someone else's AI model.

Export your Vercel logs from your dashboard
(Project → Logs → Export) and open them in a text
editor. Search for these strings:

GPTBot — OpenAI training crawler
ClaudeBot — Anthropic training crawler
Bytespider — ByteDance, known to ignore robots.txt
CCBot — Common Crawl, used by many AI companies
PerplexityBot — Perplexity search crawler

If you see any of these hitting hundreds or
thousands of times, that is your bill spike.

How to stop the leak

Option 1 — robots.txt

The first line of defense. Add this to your
public/robots.txt:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /

Warning: Many aggressive AI bots in 2026 ignore
robots.txt entirely. Bytespider is a known offender.

Option 2 — Next.js middleware

Add this to middleware.ts in your project root:

import { NextResponse } from 'next/server'

const BLOCKED_BOTS = [
  'GPTBot', 'ClaudeBot', 'Bytespider', 
  'CCBot', 'PerplexityBot'
]

export function middleware(req) {
  const ua = req.headers.get('user-agent') || ''
  if (BLOCKED_BOTS.some(bot => 
    ua.toLowerCase().includes(bot.toLowerCase())
  )) {
    return new NextResponse('Forbidden', { status: 403 })
  }
  return NextResponse.next()
}

export const config = {
  matcher: ['/((?!_next/static|_next/image|favicon.ico).*)'],
}

This blocks known AI bots at the edge before they
consume your bandwidth.

Option 3 — Cloudflare WAF

If you use Cloudflare in front of Vercel, create a
custom WAF rule:
(http.user_agent contains "GPTBot") or
(http.user_agent contains "ClaudeBot") or
(http.user_agent contains "Bytespider") or
(http.user_agent contains "CCBot")

Set action to Block. This is the most reliable
method since it stops bots before they even reach
Vercel.

The faster way

Manually hunting through log files is tedious.
I built a free browser-based tool that does it
automatically.

Drop your Vercel log export into botcost.dev and
it shows you exactly which bots are hitting you,
how much bandwidth each one consumed, and generates
the exact WAF rule or middleware code to block them.

Everything runs locally in your browser. Your log
file never leaves your device. Free, no account
required.

botcost.dev

Have you seen unusual bot traffic in your Vercel
logs? Drop the bot names in the comments — curious
what people are seeing in the wild.

I built a free tool that shows what AI bots are really costing your server

PlainTOS App — Mon, 11 May 2026 15:51:19 +0000

The problem

If you run a website, AI crawlers are hitting it
constantly. GPTBot, ClaudeBot, Bytespider, CCBot —
they scrape your content 24/7 for training data and
search indexing. You pay the bandwidth bill. They
get your content for free.

The frustrating part is most developers have no idea
how much of their server bill is pure bot waste until
they actually look at their logs.

What I built

BotCost.dev is a free browser-based log analyzer.
You drag and drop your server log file and it shows
you exactly:

Which AI bots are hitting your site
How many requests each one made
How much bandwidth each one consumed
Estimated monthly cost in dollars
A ready-to-paste Cloudflare WAF rule or Nginx config to block them

The privacy part

Everything runs in your browser. Your log file never
leaves your device. We have no server that receives
it. You can verify this yourself by opening your
browser Network tab while running an analysis — you
will see zero upload requests.

Supports Nginx, Apache, Cloudflare CSV export, and
Vercel JSON logs.

Try it

botcost.dev — free, no account required.

Would love feedback from anyone who tries it on real
logs. Curious what bot percentages people are seeing
in the wild.

Forem: PlainTOS App