Why we ship untested prompts (and the supply-chain pattern that fixes it)

rp1run — Wed, 29 Apr 2026 15:00:00 +0000

I'd never approve a PR that bypassed CI.

But I've watched dozens of teams — including ones I've worked on — deploy prompt changes with zero of the verification we'd insist on for a code change. Edit a string in a config file. Push. Hope.

A prompt change is a logic change. It alters how the system behaves under uncertainty, what it returns under load, and how it handles edge cases nobody enumerated. The fact that it's text and not Python doesn't change what it does.

The gap between how we deploy code and how we deploy prompts is going to bite hard as agentic systems scale. And the answer might already exist — in the tooling the supply-chain security world has been building for the last five years.

The supply-chain parallel

Sigstore, SLSA, in-toto. These tools solved a related problem for binaries: how do you cryptographically prove that the artifact in production is the one that passed your checks?

The primitives:

Content-addressable hashing. Identify the artifact by the hash of its content. Two artifacts with the same hash are identical, byte-for-byte.
Signed attestations. A cryptographic statement: "this hash passed this evaluation, witnessed by this entity."
Verification gates. Deployment refuses any artifact without a valid attestation.

Applied to prompts:

Hash the prompt text. prompt[sha256:abc123...] is now uniquely identifiable.
Run your eval suite against that exact hash.
Generate a signed attestation: "prompt[abc123] passed eval suite v2 on date X."
Production deployment verifies the attestation before promoting.

Now "what prompt is in production?" has an answer that doesn't depend on git archaeology or trusting a config dashboard.

What this doesn't solve

This is the part most discussions of prompt evaluation skip over.

Eval reproducibility is non-trivial when the underlying model version drifts. An attestation from last month against gpt-4o-2024-08-06 doesn't tell you anything about behaviour against gpt-4o-2024-11-20. Either you pin model versions in the attestation (and accept the operational cost of staying on old models), or you re-attest on every model version change (and accept the eval cost). There's no free lunch.

There's also the question of whether "passing evals" is actually the right gate. Code passes tests but can still ship bugs. Prompt evals are coarser — they sample behaviour, they don't prove correctness.

The bigger question

Are prompts code or configuration?

Most teams haven't picked, which is why they fall into the worst of both: edited freely like config, executing logic like code. Picking one would mean deciding whether prompts go through a CI pipeline (code-treated) or a configuration management system with rollback (config-treated). Either is better than the current default of "text in a file, deployed by whoever has commit access."

Prem Pillai (@cloud-on-prem) wrote a longer treatment of the architecture and gaps as rp1 blog post.

If you're working on prompt evaluation, deployment pipelines for agentic systems, or just struggling with the operational chaos of prompt management at scale — we have a Discord where engineers are talking through these patterns.

AI built your codebase in 2 months. Who's going to maintain it?

rp1run — Wed, 22 Apr 2026 08:47:44 +0000

Cloudflare shipped EmDash in April 2026 — an open-source CMS written in TypeScript, built in ~2 months by AI coding agents. It's a genuinely impressive achievement and a real signal of where the industry is going.

But it also surfaces a question that the AI coding conversation has been avoiding: what happens after the AI ships the first version?

The "plans that read well don't build well" problem

There's a failure mode I keep seeing in AI-assisted codebases. The initial build is fast. The prose in the plan reads authoritatively. The code compiles and the tests pass. Three weeks later, the second engineer tries to extend it, and nothing quite fits — because the agent's narrative was persuasive without being correct about the underlying constraints.

This isn't a model problem. Frontier models will keep getting better at writing plausible code. It's a workflow problem. The missing layer is the one that turns ephemeral agent sessions into durable, reviewable architectural decisions.

What the missing layer looks like

We have been building rp1 with this exact gap in mind. Three ideas, each directly addressing a specific failure mode.

1. Constitutional prompting

Most "prompt engineering" is additive — you stack instructions on top of a model and hope. Constitutional prompting is subtractive: workflows encode the patterns an expert would follow as constraints. /build isn't a prompt, it's a pipeline:

Generate a blueprint from requirements
Form a hypothesis about the existing codebase
Validate the hypothesis against actual code before writing anything
Implement against the validated plan
Run verification

The hypothesis validation step is the one that catches the "plan reads well but is wrong about your ListView" class of bug.

2. Knowledge-aware agents

Most AI coding sessions start blank. You re-explain your architecture everytime. rp1's /knowledge-build runs once and maps your codebase into a persistent knowledge base that every subsequent command inherits.

The practical effect: you stop getting generic advice that ignores your patterns. Every /build starts with full awareness of the actual system, not an imagined one.

3. Durable artefacts

Every rp1 workflow produces inspectable design documents — requirements, design, hypothesis, verification, reports — attached to the project, not trapped in chat scrollback.

This is the onboarding primitive. When the second engineer joins an
AI-built codebase, they can read what was decided and why instead of re-prompting their way to an understanding.

Try it

rp1 is open source and works across Claude Code, OpenCode, Codex, and GitHub Copilot CLI. Same workflows, different harnesses.

The full write-up on how this plays out specifically for EmDash-style codebases is on our blog: rp1 on EmDash — the workflow layer that makes AI-built codebases navigable.

If you're maintaining a codebase an agent wrote, Prem and I would genuinely like to hear what's broken. That's the feedback that's shaped everything we've built so far.

rp1 is built by Prem Pillai (@cloud_on_prem) and Mahesh Shivamallappa (@maheshs786).

Forem: rp1run