A $200,000 AWS Bill Nobody Noticed — Fixed in 5 Minutes

Bharath Nadar — Wed, 11 Mar 2026 14:47:30 +0000

We had a $200,000 problem hiding in plain sight.

Not a breach. Not a runaway workload. A single config field set years ago that nobody had revisited as the company scaled.

The Numbers That Didn't Add Up

Late 2025. Reviewing AWS Cost Explorer. CloudTrail — a line item that used to sit quietly around $1,000 a month — had crept to $10,000 in one prod account and $30,000 at the org level.

It never triggered an alert because it didn't spike. It crept.

Around the same time, other costs had come down — reserved instance discounts, some workloads rightsized. The overall bill looked reasonable. CloudTrail was hiding in the noise.

By the time I caught it: roughly $200,000 spent on audit logs nobody had asked for.

What Was Causing It

Breaking down CloudTrail by usage type in Cost Explorer showed the bill was driven entirely by data event recording — 8.98 billion S3 events in us-east-1 alone in December, at $0.10 per 100,000 events.

The org trail config explained why:

{
  "ReadWriteType": "All",
  "DataResources": [
    { "Type": "AWS::S3::Object", "Values": ["arn:aws:s3:::"] }
  ]
}

Every S3 API call. Every read, every write. Every bucket. Every account in the org.

S3 reads account for 85–95% of all S3 data events in a production environment. ML pipelines, CDN origins, build artifact stores — all logging every single fetch. Billions of them. Every month.

Does Compliance Actually Require This?

Before touching anything, I checked.

Framework	S3 Read Logging Required?
SOC 2 Type II	No
ISO 27001	No
GDPR	No
CIS AWS Benchmark	No

Nobody needs to know a Lambda function fetched a static asset at 3am. They do need to know if someone deleted a production backup.

The Fix

Phase 1 — Switch to WriteOnly

aws cloudtrail put-event-selectors \
  --trail-name <your-org-trail-arn> \
  --event-selectors '[{"ReadWriteType":"WriteOnly","IncludeManagementEvents":true,"DataResources":[{"Type":"AWS::S3::Object","Values":["arn:aws:s3:::"]}]}]' \
  --region us-east-1

"All" → "WriteOnly".

One word. ~$300,000 in annual savings.

Phase 2 — Scope to Sensitive Buckets Only

Switching to WriteOnly eliminates ~90% of the cost. The next step is replacing the wildcard with only buckets that genuinely need auditing — PII, credentials, or compliance data.

{
  "ReadWriteType": "WriteOnly",
  "DataResources": [
    {
      "Type": "AWS::S3::Object",
      "Values": [
        "arn:aws:s3:::your-pii-bucket/",
        "arn:aws:s3:::your-secrets-bucket/"
      ]
    }
  ]
}

This brings data event costs close to zero while keeping full audit coverage where it actually matters.

At org scale, tag sensitive buckets (security-scan: enabled) and use that tag to drive both CloudTrail scoping and security scanner configuration.

The Real Lesson

A security tool was onboarded and as part of its setup, full S3 data event logging was recommended — a legitimate recommendation for visibility. The team followed it. Nobody flagged the cost implication at scale.

When a security team introduces a new tool, the wider team deserves to know:

What config changes does it recommend — and what do those cost at your scale?
What AWS permissions does it need?
Who owns the monthly bill for this tool's footprint?

Security and FinOps need to be in the same conversation when a new tool lands.

Not because security is the enemy of cost — but because the most expensive line on your bill might be a setting a well-intentioned engineer enabled six months ago and never revisited.

The config that cost $1,000 a month in 2024 cost $360,000 a year in 2025. The infrastructure scaled. The config didn't.

Forem: Bharath Nadar