Forem: John Loper

EnvSecOps: The Obviously Painful Path Forward

John Loper — Wed, 12 Nov 2025 23:17:39 +0000

Turning “Check the Bag” into a Working System

The idea was simple: verify the bag, not the badge.
The hard part is turning that into a system someone can actually build.

This is the actionable roadmap — not a manifesto, not another “zero-trust” slide.
It’s the pragmatic route to the world described in Check the Bag: a unified model where credentials are issued only if the environment proves itself clean.

Task 1 — Identity & Attestation

Goal: unify workload and environment identity under one auditable standard.

Evaluate SPIFFE/SPIRE as the Identity Control Plane for all non-human workloads.
SPIRE becomes the source of truth for what runs where, and under what evidence.
Attestation plugins can integrate OS, hardware, and custom attributes (like the Nix or devcontainer signature from Task 2).
Every service, job, or build agent receives its identity through SPIRE issuance.

Outcome: a cryptographically provable binding between a workload and its attested environment.

Task 2 — Reproducible Environments

Goal: make “the bag” reproducible and attestable.

Stand up a PoC comparing Nix and DevContainers for reproducible dev/runtime environments.
The output must be deterministic: a Nix closure hash or a signed devcontainer manifest.
Feed that hash into SPIRE’s attestation selector — so credentials can only be issued if the environment matches a known, reproducible build.
Document developer workflow friction early; this is where usability sinks most pilots.

Outcome: every environment can prove exactly what it is — not just who’s using it.

Task 3 — Policy

Goal: codify trust decisions in one place.

Adopt Open Policy Agent (OPA) as the global Policy Decision Point (PDP).
All attestation, identity, and credential issuance checks route through OPA.
OPA policies evaluate:
- whether an environment is in policy,
- whether its attestation is valid,
- and what privileges the resulting identity should get.
Extend existing CI/CD gates with conftest for developer feedback loops.

Outcome: the organization’s trust logic is versioned, testable, and enforced everywhere.

Task 4 — Provenance

Goal: link build outputs to their attested inputs.

Require Sigstore signing for all build artifacts.
Integrate Sigstore’s Rekor log and Fulcio CA for transparent verification.
Use SLSA v1.0 as the minimum compliance level; pipelines must produce signed provenance linking:
- the SPIRE-attested environment (from Task 1–2),
- the OPA-approved policy decision (Task 3),
- and the final artifact digest.
Store provenance as evidence, not paperwork.

Outcome: an end-to-end chain of custody — from developer environment to running workload.

Putting It Together

Pillar	Technology	Function
Identity	SPIFFE/SPIRE	Verifiable workload identity
Environment	Nix / DevContainers	Deterministic, attested environments
Policy	OPA / Conftest	Centralized, auditable trust logic
Provenance	Sigstore / SLSA	Signed, traceable artifact lineage

This stack isn’t speculative — it’s all open, CNCF-anchored, and proven at scale.
What’s missing isn’t tech; it’s discipline and integration glue.

Why This Is the Path Forward

Because the opposite has already failed.

Secrets rotate but drift remains.
Tokens expire but trust persists unearned.
“Zero-trust” dashboards glow while production runs on :latest.

This model replaces vibes with verification.
It’s painful, yes — but every painful system you build now prevents a post-incident rebuild later.

Why This Stack, Why Now

Five years ago, you couldn't build this. The pieces didn't exist:

SPIFFE/SPIRE was alpha code
SLSA was a Google internal concept
Sigstore didn't exist
DevContainers were a VS Code experiment

Today, it's all CNCF-graduated, production-proven, and free.

The industry also just learned the hard cost of NOT doing this:

CircleCI (2023): Session token theft → full production access
LastPass (2022): DevOps engineer's compromised laptop → vault breach
Okta (2024): Support engineer's personal device → customer data access

The pattern is identical: trusted identity + untrusted environment = incident.

The tech to fix this finally exists. The question is whether your org will adopt it before or after your breach.

RC = 0

You don’t need another framework.
You need fewer excuses.

Identity, Environment, Policy, Provenance — that’s the whole playbook.
Pin them, wire them, and stop pretending automation is trust.

Pin It or Bin It

John Loper — Wed, 12 Nov 2025 22:28:25 +0000

How to Stop Shipping Garbage Before It Ships You

If your build still pulls ubuntu:latest, you’re not “cloud-native.” You’re running a randomizer in production.

Why Pinning Matters

Every tag that isn’t a digest is a liability.
:latest is not convenience; it’s negligence.

What you write	What happens next
`FROM node:latest`	You rebuild on Tuesday and ship a new kernel vulnerability.
`pip install flask`	You pull in a supply-chain bomb before lunch.
`terraform init`	You silently upgrade the AWS provider, then wonder why IAM exploded.

If you don’t pin it, you can’t attest it.
If you can’t attest it, you can’t trust it.

The Tools of the Trade

OPA (Open Policy Agent)

A fast, embeddable policy engine.
It doesn’t care what file you feed it — only that you speak JSON.

Conftest

A CLI built on top of OPA.
It handles the boring bits:

parses Dockerfiles, YAML, Terraform, JSON
feeds them into OPA
exits 1 on fail, 0 on pass That’s all you need for CI or pre-commit hooks.

Install it:

# macOS / Linux
brew install conftest@0.56.0
# or
curl -L https://github.com/open-policy-agent/conftest/releases/latest/download/conftest_$(uname -s)_$(uname -m).tar.gz \
  | tar -xz -C /usr/local/bin

Run it:

conftest test Dockerfile --parser dockerfile

It will parse your Dockerfile into JSON, run your Rego rules, and tell you exactly why your laziness failed policy.

Crane

Part of Google’s go-containerregistry toolkit.
It’s the BOFH’s Swiss army knife for container registries.

Use it to fetch digests, manifests, and metadata — fast, no daemon, no Docker socket.

Install it:

brew install go-containerregistry
# or
go install github.com/google/go-containerregistry/cmd/crane@0.20.02

Pin any image:

img=python:3.12
echo "$img@$(crane digest "$img")"

Result:

python:3.12@sha256:deadbeefbadc0ffee123456789...

Now you know exactly which bits are running in prod.
No excuses, no surprises.

🪓 Policy as Code: The OPA Way

Create policies/dockerfile/pinning.rego:

package policies.dockerfile.pinning

deny[msg] {
  some s
  f := s.Commands[_]
  f.Name == "from"
  not contains(f.Value, "@sha256:")
  msg := sprintf("FROM %q missing digest", [f.Value])
}

deny[msg] {
  f := input.Stages[_].Commands[_]
  f.Name == "user"
  trim(f.Value) == "root"
  msg := "Dockerfile must set non-root USER"
}

Test it manually:

conftest test Dockerfile --parser dockerfile --output table

Expected output:

FAIL - Dockerfile - FROM "ubuntu:latest" missing digest

🧷 Make It Automatic: Pre-commit Hook

.git/hooks/pre-commit:

#!/usr/bin/env sh
set -eu
conftest pull ghcr.io/your-org/precommit:1 >/dev/null || true
files=$(git diff --cached --name-only | grep -Ei '(Dockerfile|\.ya?ml)$' || true)
[ -z "$files" ] && exit 0
conftest test --parser=dockerfile --output table $files

Try to commit a floating Dockerfile and watch it fail:

FAIL - Dockerfile - FROM "nginx:latest" missing digest
✖ OPA pre-commit checks failed.

Success: laziness stopped at the gate.

The Philosophy

Small tools. crane for digests. conftest for checks. jq and yq for glue.
Zero GUIs. If it doesn’t fit in a pipeline, it doesn’t belong.
Fast failure. The only “UX” a BOFH cares about is exit 1.
Determinism > decoration. Pretty dashboards are for people who don’t read logs.

TL;DR

Pin everything — image@sha256:digest.
Write Rego policies once; run everywhere.
Use Conftest in pre-commit and CI.
Use Crane to fetch digests and enforce reproducibility.
Automate the yelling so you can focus on real problems.

Pin It or Bin It (for the brewsters)

John Loper — Wed, 12 Nov 2025 14:05:29 +0000

Opening jab

You can’t secure what you can’t reproduce.
“Latest” isn’t a strategy — it’s a confession that you don’t control your own environment.

The Real Problem

Most “secure” pipelines fall apart before they start.
Developers run different OPA versions.
CI installs whatever Homebrew happens to serve that day.
And your attestation chain begins with curl https://sh.somestartup.io | bash.

If your tooling isn’t pinned, your policies are theater.

The Only Acceptable Rule

Every tool in the critical path — compiler, linter, scanner, policy engine, image builder — must be pinned by version and checksum.

You don’t need a platform for this. You need discipline.

Homebrew, Done Right

Homebrew is fine if you cage it properly.
Here’s how adults install their tools:

# 1. install specific versions
brew install open-policy-agent/tap/opa@1.1.1
brew install instrumenta/instrumenta/conftest@0.56.0
brew install go-containerregistry@0.20.2

# 2. freeze them in place
brew pin opa
brew pin conftest
brew pin go-containerregistry

If your version isn’t directly available, extract it yourself:

brew extract --version=0.56.0 conftest instrumenta/instrumenta
brew install instrumenta/instrumenta/conftest@0.56.0
brew pin conftest

Now you can rebuild your environment on any box, any day, and get the same bits.

No curl-to-bash. No “latest.”
If it’s not signed, hashed, or attested, it doesn’t enter $PATH.

Lock Your Provenance

Your toolchain is part of your SBOM.
Export it like any other dependency:

brew bundle dump --file=./Brewfile.lock.json
git add Brewfile.lock.json

This file is now the attestable manifest for your developer environment.
Your CI and devcontainers can reproduce it 1:1.

Why This Matters

Pinning is provenance.
It’s the difference between “I built this” and “something built this once, maybe.”
Without it, your SBOM is fiction and your attestations are decorative art.

This is the first layer of EnvSecOps:

Trust begins with reproducibility.

TL;DR

Tool	Action	Why
`brew install …@version`	explicit version	deterministic installs
`brew pin`	freeze it	prevent drift
`brew bundle dump`	export	attest environment
`sha256sum -c`	verify	don’t trust strangers
`jq`, `yq`, `crane`, `conftest`, `opa`	small tools	big security

The Exit Code

If your pipeline depends on brew upgrade, it’s not a pipeline — it’s a coin toss.
Version-lock it, checksum it, and move on.

EnvSecOps - What It Actually Is (And Why DevSecOps Won't Cut It)

John Loper — Sat, 08 Nov 2025 15:46:21 +0000

Meme recap: Old security checks IDs. EnvSecOps checks IDs and the bag—the actual execution environment.

This post pins down what EnvSecOps is, where it sits next to DevOps/DevSecOps, and why it closes the door those frameworks leave open.

TL;DR (30 seconds)

DevOps: ship faster (automation, reliability).
DevSecOps: ship safer code (shift-left testing, compliance).
EnvSecOps: only issue credentials when the environment proves it’s policy-approved right now—and keep re-verifying on renewal. If the bag mutates, the token dies.

EnvSecOps secures where work is performed (dev, CI/CD, ops terminals) instead of assuming those machines are clean.

Assumptions (explicit)

Platform examples: Kubernetes (pods, service accounts, namespaces) and cloud IAM.
Identity: OIDC for humans/CI; SPIFFE/SPIRE for workloads.
Evidence: Signed attestations (image/tooling/config digests; provenance).
Policy: OPA/Cedar to express “only approved tooling,” short TTLs, and runtime disqualifiers.
Runtime signals: eBPF-driven tools (Falco/Tetragon) to catch both on-disk and in-memory tricks.

The pattern is product-agnostic. Swap components as needed; the gate is the invariant.

What DevOps Solved (and didn’t)

DevOps removed handoffs and manual toil—CD pipelines, IaC, fast recovery. Security lived mostly in operations: hardening, monitoring, incident response. Useful, but post-issuance.

What DevSecOps Added (and where it stops)

DevSecOps pushes security left—SAST/DAST, dependency and image scanning, policy checks in CI. It answers “Is this code/artifact safe?” It assumes the runner/terminal doing the work is trustworthy.

That assumption is the hole: if the environment running the scanner/deployer is compromised, DevSecOps happily validates the attacker’s workflow too.

EnvSecOps, precisely

Definition: A credential exists only if the requesting environment can cryptographically prove it contains only policy-approved tooling and shows no recent runtime compromise. Verification happens at issuance and every renewal.

Flow:
Evidence → Policy → Token → Go (repeat on a short heartbeat)

Evidence: signed statements about image/tooling/config/provenance + recent runtime signals
Policy: allowlists + disqualifiers (writes under /usr, memfd exec, RW mounts, caps gain, etc.)
Token: short-lived, tagged with env_hash and policy_id; logs to append-only storage

If either evidence or runtime fails → no token (or renewal denied) with a human-readable reason.

The production-ops blind spot (the last mile)

DevSecOps secures artifacts. EnvSecOps secures the consoles that touch prod.
SREs, DBAs, compliance analysts—anyone running privileged actions—get access only from attested, policy-approved environments. Compromised laptops and ad-hoc jump boxes stop at the door.

Audit that finally proves where actions came from

Every issued token carries env_hash and policy_id. Cloud audit now answers:
who acted, what they did, and from which attested environment—provably. That’s post-incident clarity and real compliance, not ceremony.

Where they overlap (and don’t)

DevOps: deploy velocity, reliability.
DevSecOps: artifact safety, shift-left assurance.
EnvSecOps: environment assurance and credential lifecycle across dev, CI, and ops.

You need all three. EnvSecOps doesn’t replace DevSecOps; it binds DevOps/DevSecOps activity to verified environments.

Why “check the bag” matters more than “check the ID”

Stolen or misused creds authenticate fine. ID passes.
Attacker’s environment can’t produce a valid attestation or passes runtime checks. Bag fails.
EnvSecOps makes that failure the gate: no proof, no token.

Works the same for pushing code, deploying, restarting services, or exporting audit logs.

What you always enforce (model-agnostic hygiene)

Short lifetimes (5–15m): renew at TTL/2; deny on stale or drifted env.
Runtime signals in the decision loop (not just dashboards).
Least privilege via issuance scope, not just roles.
Read-only roots, minimal caps, signed artifacts—boring on purpose.

When you need EnvSecOps

Supply chain programs (SLSA, provenance requirements): prove environments for dev, build, deploy, and ops.
Post-breach realism: stolen tokens and compromised laptops are a given; issuance must be gated.
Distributed teams/contractors: network controls aren’t environment controls.
Platform engineering: self-service access that’s cryptographically enforced, not ticketed.

Getting started (one-door pilot)

Pick one door (a meaningful role with limited blast radius).
Freeze the bag (pinned image, approved tooling/config digests; sign + log).
Gate issuance (require fresh proof + no disqualifying runtime events).
Short TTLs, renew at TTL/2; deny on mutation with clear reasons.
Tag tokens (env_hash, policy_id) and log decisions append-only.
Publish: the policy snippet and one denial screenshot.

TL;DR

DevOps ships fast.
DevSecOps ships safer code.
EnvSecOps ensures the work of shipping and operating happens only from attested, policy-compliant environments.

No attested, policy-approved bag → no token.
That’s how you stop the 2 AM “SSH from a mystery laptop” from being business as usual.

EnvSecOps: Nix, Recognized: The Strongest Bag Wins

John Loper — Sat, 08 Nov 2025 14:28:42 +0000

Nix's functional purity makes deterministic artifacts a first class citizen. Shout out to the community for keeping it real. Its a primary building block we can build upon to keep our environments clean as a whistle.

Meme recap: Old security checks IDs. EnvSecOps checks IDs and the bag.
This post: Why Nix is a first-class way to define, prove, and police the bag.

Why Nix fits EnvSecOps

Reproducibility: Inputs are pinned (flake.lock), outputs are content-addressed.
Minimal drift surface: The /nix/store is read-only; mutation stands out.
Smaller policy: Instead of enumerating a thousand files, you allowlist a closure root (and optional successor roots).

The bag stops being “a pile of files” and becomes “this exact closure hash and nothing else.”

Policy: allow the closure, not vibes

Renewal: prove the running bag AT EACH RENEWAL

Migration playbook (fast & boring)

Build from Nix (for the service you’ll gate first).
Emit labels/artifacts at build time
CI signs an attestation
Publish a policy data bundle
Gate one role on that bundle
Renew at TTL/2; deny on mismatch; measure and ratchet down TTLs as confidence grows.

Where Nix shines in this setup (reduces policy toil)

From file allowlists to root allowlists. One item per release instead of a thousand.

Successor maps tame digest churn. Hotfixes don’t flap renewals.

Lockfile diffs become change requests.

Caveats (be honest)

Purity isn’t magic: Ensure sandboxed builds; eliminate impure sources (time, network) or you’ll get “same intent, new hash” churn.

Runtime still matters: Even with a read-only store, enforce RO mounts and minimal caps; detect in-memory tricks at renewal time.

TL;DR

Nix turns “check the bag” into check the closure.
Attest a root, allowlist it, and refuse to mint tokens for anything else.

No attested, policy-approved closure → no token.

EnvSecOps: The Painfully Obvious Path Forward

John Loper — Thu, 06 Nov 2025 22:19:49 +0000

It's not radical.
It's obvious.
We're missing a big piece of security right at the entrance. We've been hanging out season passes (static creds) to kids with backpacks full of who knows what.
We can check those bags. We have the technology.

Check the Bag: The Meme You Cant Unsee

Old security: “Check IDs.”
New security: “Check IDs and the bag.”

The meme is the message. We don’t just ask who is calling our systems—we open the bag and prove what’s inside. And that bag must contain only policy-approved tooling, cryptographically attested, no surprises, no freebies.

Call it EnvSecOps if you want a name. Practically, it’s the painfully obvious next step:
No attested, policy-approved bag → no token.

What “the Bag” Actually Means

The bag = the execution environment: image, toolchain, configs, helper scripts.
Policy-approved = explicit allowlist: pinned digests, permitted binaries, known configs.
Attested = cryptographically proven: signed statements that the environment only contains those allowed artifacts.

If anything unapproved is in the bag? Deny.
If the bag can’t prove itself? Deny.
No vibes. No :latest. No “temporary” helper scripts.

You still keep IAM, SCPs, runtime sensors. The shift is earlier:
Tokens exist only after the bag proves it contains nothing except what policy allows.

Why This Is Obvious (And Superior)

Prevention beats monitoring. If a tool isn’t approved, it never rides inside a token.
Deterministic access. Evidence → policy → token → go. No ticket tennis.
Audit that speaks English. Every token says: “Issued because this bag (hash X) matched that policy (ID Y) at this time.”

Here’s a sharper, more ruthless version — same facts, same events, same message, just delivered with more bite and clarity.

Why This Matters Right Now

Look at the pattern in recent breaches:

CircleCI (2023): Malware on laptop → stolen session → production access
LastPass (2022): Compromised home PC → vault extraction
Okta (2024): Personal device → customer tenant access

Same root cause every time:

We obsess over who someone is while never verifying what they’re running from.
It’s the equivalent of checking IDs at the door while ignoring the backpack stuffed with explosives.

Real scenario (CircleCI, 2023):

Malware on an engineer’s laptop grabs a session cookie
Cookie is valid — bypasses MFA, identity checks, everything
Attacker walks straight into the CI system
Extracts production secrets, compromises customer environments
Postmortem: lots of shocked faces, very few surprises

And now organizations are bolting LLM agents directly onto production systems, with tool-calling turned on, running from environments that could charitably be described as “unmonitored optimism.”

Agents don’t magically become secure because you added a guardrail prompt.
They inherit whatever chaos is in their runtime environment — usually hope, drift, and the ghosts of abandoned dependencies.

The irony?
The tech to fix this has existed for years — SPIFFE for workload identity, Sigstore for signing and verification, OPA for real-time policy enforcement.
CNCF graduated, production-ready, already deployed at hyperscale.

The only open question is whether you adopt this before your breach… or after you’re writing the apology blog post.

The Absolute Minimum

Pin one tool - Replace FROM ubuntu:latest with FROM ubuntu@sha256:abc123...
Verify it - Add cosign verify before using it
Gate on it - Don't issue credentials unless verification passes

That's it. That's the nucleus.

Everything else (SPIRE, OPA, SLSA) is scaling that pattern.

Field Rules (Pin These)

Minutes, not hours. Long-lived = stolen-lived.
Only approved tooling. If it’s not on the list, it’s not in the bag.
No drift. Changing the bag means re-attesting or getting denied.
Humans ≈ workloads. MFA is a signal, not a hall pass.
Policy before platform. Declare issuance conditions; automate enforcement.
Boring by design. Deterministic gates beat heroics and dashboards.

Starter Kit

For attestation: Sigstore (Cosign + Rekor)

Why: Free, open, CNCF-graduated, works today
Alternative: in-toto if you need custom attestation predicates

For identity: SPIFFE/SPIRE for workloads, OIDC for humans

Why: Industry standard, integrates with everything
Alternative: Cloud IAM if you're single-cloud only

For policy: OPA

Why: Flexible, testable, huge community
Alternative: Cedar if you're deep in AWS

The pattern is the product. Tools are replaceable; the gate is not.

The Line to Use in Every Review

“Once prod creds are issued to an environment, that environment is prod.”
So check the bag—and prove it only contains what policy allows—before issuing the wristband.

Call to (Obvious) Action

Replace one :latest with a pinned, approved base image and sign it.

# 1. Get the digest
docker pull ubuntu:22.04
docker inspect ubuntu:22.04 --format='{{.RepoDigests}}'
# Output: ubuntu@sha256:abc123...

# 2. Update your Dockerfile
FROM ubuntu@sha256:abc123...

# 3. Sign it
docker build -t myapp:v1 .
cosign sign --yes myapp:v1

# 4. Verify before deploy
cosign verify myapp:v1 --certificate-identity=...

If your security only checks IDs, you’re letting backpacks full of trouble stroll past the rope.
Check the bag. Prove it only holds policy-approved tooling. Then, and only then, hand out the wristband.