That Time We Found a Service Account Token in my Log Files

Vincent von Büren — Thu, 04 Sep 2025 07:59:04 +0000

⚠️ Disclaimer
This article assumes you're already somewhat familiar with Kubernetes concepts (Pods, ServiceAccounts) and the basics of JSON Web Tokens (JWTs).

It was a Tuesday.

Nothing special - just your average day as a platform engineer. My team's notifications were mercifully quiet, and I thought, "Perfect, I can finally clean up that old Helm chart that's been bothering me."

I opened the repo of the underlying image written in Go to double-check the config before merging.

Before I even got far, a colleague — Martin Odermatt — pinged me:

“You might want to take a look at this…”

He had spotted something concerning in the code:

log.Println("SA Token:", token)

Wait. What?

A debug statement. Still in production code. Logging an actual Kubernetes ServiceAccount token. Not cool...

I paused. My heart rate didn’t. Curious but mostly horrified, I took the token Martin had found and decoded the payload in my shell:

{
"iss": "https://kubernetes.default.svc.cluster.local",
"kubernetes.io/serviceaccount/namespace": "payments",
"kubernetes.io/serviceaccount/secret.name": "payments-token-6gh49",
"kubernetes.io/serviceaccount/service-account.name": "payments-sa",
"kubernetes.io/serviceaccount/service-account.uid": "f9a2c144-11b3-4eb0-9f30-3c2a5063e2e7",
"aud": "https://kubernetes.default.svc.cluster.local",
"sub": "system:serviceaccount:payments:payments-sa",
"exp": 1788201600,   // Sat, 01 Aug 2026 00:00:00 GMT
"iat": 1756665600    // Fri, 01 Aug 2025 00:00:00 GMT
}

Default audience claim. A 1-year expiry.

As we dug deeper, Martin Odermatt pointed out the underlying issue: this was a legacy ServiceAccount token, and we should be using projected tokens instead.

This "bad boy" wasn't just a dev leftover - it was a high-privilege token with zero constraints floating around in plaintext logs!

What This Article Covers

In this post, I'll guide you through:

The inner workings of Vault authentication with JWT and Kubernetes methods
What Kubernetes ServiceAccounts and their tokens are, and how they’re (mis)used
How projected ServiceAccount tokens fix many of the hidden dangers of older token behavior
Why you should start adopting token projection and Vault integration today

We'll cover real-world use cases, implementation tips, and common pitfalls - so you don't end up like I did, staring at a:

log.Println("SA token:", token)

...and wondering how close you just came to a security incident.

Why This Matters

To really understand why that log statement gave me chills, we need to unpack a few core concepts:

What is a JWT?
How do Kubernetes ServiceAccounts and their tokens work?
And what role do these tokens play in authenticating to systems like Vault?

Let's start with the fundamentals.

What Is a JWT?

If you've been around authentication systems long enough, you've probably seen one of these beasts:

eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9...

This is a JSON Web Token (short: JWT).

It's a compact, URL-safe format for representing claims between two parties. They're used everywhere: web apps, APIs, and yes — inside your Kubernetes cluster.

A JWT consists of three parts:

Header – declares the algorithm used to sign the token (e.g. RS256)
Payload – contains the claims (who you are, what you're allowed to do, etc.)
Signature – a cryptographic seal that verifies the payload hasn't been tampered with

Claims are the heart of a JWT — key-value pairs that describe who the token refers to and what it can be used for.

They can be:

Standard claims defined by the spec (e.g., iss, sub, exp, aud)
Custom claims added by the issuer for domain-specific needs

Closer Look at `aud`

The audience (aud) claim tells who the token is meant for. Think of it as the intended recipient.

Example: Imagine a Coldplay concert ticket. It says valid for Stadium X on 01-09-2025. You can't take the same ticket and use it at Stadium Y — they'll reject it (...trust me, I tried).

A JWT works the same way:

If the token has "aud": "https://kubernetes.default.svc", then only the Kubernetes API server should accept it.
If some other service receives that token, the aud won't match and the token must be rejected.

Without this check, a token could be misused anywhere that trusts the signing key. With aud, it's scoped to the right system.

Kubernetes and ServiceAccounts

Kubernetes is an open-source platform that orchestrates containers at scale. At its heart is the Pod — the smallest deployable unit.

But every pod needs an identity. That's where ServiceAccounts come in.

ServiceAccounts 101

Every Pod references a ServiceAccount (default if none is set), but a token is only mounted if enabled
Kubernetes mounts the identity at:

/var/run/secrets/kubernetes.io/serviceaccount/token

That token is a JWT, signed by the Kubernetes control plane
It lets the pod authenticate with the API server — and sometimes even external systems like Vault

The Catch

Until recently, these tokens came with dangerous defaults:

Long-lived (often valid for a year)
Previous to Kubernetes v1.24, there was no default audience set (https://kubernetes.default.svc)
Automatically mounted into every pod, even if unused

Enter Vault: The Gatekeeper of Secrets

HashiCorp Vault is your cluster’s paranoid librarian:
it stores API keys, certs, passwords — and only hands them out when it's sure you should have them.

How? Authentication methods.

Vault Authentication Methods

Username & password
AppRole
LDAP
Kubernetes
JWT

Let's zoom into the last two.

Kubernetes Auth Method

Pod sends its mounted ServiceAccount token to Vault
Vault validates it against the Kubernetes API
If valid, Vault maps it to a policy

This is simple and works well when Vault runs inside the cluster.

JWT Auth Method

Vault verifies the JWT itself (signature, claims, expiration)
No need for Kubernetes API access
More portable

Rule of thumb:

Use Kubernetes if Vault runs inside your cluster and simplicity matters
Use JWT if you want portability, stronger boundaries, and flexibility

Projected Tokens: Because It's 2025

Old tokens were static and long-lived — exactly what we were looking at here. As Martin pointed out during the investigation, projected tokens are designed to fix this entire class of problems.

Instead of mounting a one-year token into every pod, Kubernetes can now generate short-lived, audience-bound tokens on demand.

What You Get

Short TTL (e.g. 10 minutes)
Audience restrictions (aud: vault)
Automatic rotation by kubelet
No automatic mounting into pods

Example Pod with Projected Token

apiVersion: v1
kind: Pod
metadata:
  name: projected-token-test-pod
  namespace: demo
spec:
  serviceAccountName: projected-auth-sa
  containers:
    - name: projected-auth-test
      image: demo/vault-curl:latest
      command: ["sleep", "3600"]
      volumeMounts:
        - name: token
          mountPath: /var/run/secrets/projected
          readOnly: true
  volumes:
    - name: token
      projected:
        sources:
          - serviceAccountToken:
              path: token
              expirationSeconds: 600
              audience: vault

Why Vault Loves This

Vault's JWT auth method is tailor-made for projected tokens:

It parses and verifies the JWT signature (via a configured PEM key or JWKS endpoint)
Validates all claims (aud, sub, exp, iss) locally
Issues secrets only if every check passes

Minimal dependencies. Strong claim validation. Secure, verifiable checks.

Back to the Log

Imagine you stumble upon this in a Go app:

log.Println("Auth Token:", token)

Old world: a one-year, cluster-wide token with no audience. A time bomb.
New world: a 10-minute token, scoped to Vault, rotating automatically.

It's still bad to log tokens — but at least it's not catastrophic.

Try It Yourself: Vault + K8s AuthN Lab

I've built a hands-on demo repo where you can test this locally with KIND (Kubernetes in Docker) and Vault Helm charts:

👉 GitHub: VincentvonBueren/erfa-projected-sa-token

What's Inside

KIND cluster with Vault
Both Kubernetes and JWT auth methods enabled
Vault policies and roles
Four demo pods:
- Kubernetes auth method
- JWT with static token
- JWT with projected token
- JWT with wrong audience (failure demo)

Final Drop 🎤

If your pods still run with default, long-lived tokens:
you’re one debug log away from giving away the keys to your cluster.

Projected tokens aren't optional. They're essential.
Adopt them today — and stop shipping security disasters.

Acknowledgment

The discovery of the exposed ServiceAccount token — and the push towards using projected tokens — came from my dear fellow engineer Martin Odermatt, whose input significantly shaped this investigation and motivated me to tell this story.

Forem: Vincent von Büren