Forem: willamhou

How to Add Tamper-Evident Audit Trails to Your OpenClaw Assistant

willamhou — Tue, 12 May 2026 06:17:29 +0000

Your OpenClaw assistant just deleted a file, sent an email, or ran a shell command on your machine. Can you prove what it did? When? Authorized by whom?

Standard log files don't answer that. They can be edited. They can be rotated. They can be deleted. After an incident, "the agent did X" is your word against the runtime that produced the log.

This post walks through adding cryptographic audit trails to OpenClaw using @signet-auth/openclaw-plugin. Every tool call gets:

An Ed25519 signature over the canonical action payload (RFC 8785 JCS → SHA-256 → Ed25519)
A hash-chained entry in ~/.signet/audit/*.jsonl — deletion or reordering breaks the chain
Optional policy enforcement (deny dangerous tools before they run)
Optional encryption of tool params at rest

Total setup time: under a minute.

What you need

OpenClaw (>=2026.3.24-beta.2) — the gateway you already run
The signet CLI on $PATH. Install via cargo install signet-cli, or grab a release binary
Two minutes

That's it.

Step 1: Create a signing identity

The plugin signs with a local Ed25519 key. Generate one:

signet identity generate --name openclaw-agent --owner you@example.com

This writes the keypair to ~/.signet/identities/openclaw-agent/. The private key stays on disk; the public key is what verifiers (auditors, you, anyone) use to check signatures later.

If you want the key passphrase-protected, add --passphrase. We'll set the passphrase env var below.

Step 2: Install the plugin

openclaw plugins install @signet-auth/openclaw-plugin

OpenClaw checks ClawHub first, falls back to npm.

Step 3: Configure

Add the plugin entry to ~/.openclaw/config.json:

{
  "plugins": {
    "entries": {
      "signet": {
        "config": {
          "keyName": "openclaw-agent",
          "target": "openclaw://gateway/local"
        }
      }
    }
  }
}

Two fields are enough to get started. Everything else has sensible defaults.

If your key is passphrase-protected, export the passphrase before launching the gateway:

export SIGNET_PASSPHRASE='...'
openclaw start

Step 4: Use OpenClaw normally

Run any task that exercises tools — file operations, web search, shell commands, anything. Every tool call now produces a signed receipt before execution. Tool errors are logged. Nothing in your workflow changes.

While OpenClaw runs, watch the gateway log for lines like:

[signet] signed file_read (call=tc_abc123 session=ses_xyz789) → rec_a1b2c3d4...
[signet] signed shell_exec (call=tc_def456 session=ses_xyz789) → rec_e5f6a7b8...

Each rec_... is a receipt id derived from the signature itself.

Step 5: Verify the audit trail

After OpenClaw has done some work, verify the chain:

signet audit --verify

Output:

Hash chain integrity:    valid (records=42)
Signature verification:  42 / 42 valid

That second line is the cryptographic guarantee: 42 receipts, every one of them signed by the key whose public half lives in ~/.signet/identities/openclaw-agent/openclaw-agent.pub.json. Modify any field of any receipt and the verification fails. Delete a record and the chain breaks.

To browse what your assistant actually did:

signet explore

Press a tool name to see the full action payload, params, timestamp, and the run id that ties together the before_tool_call and after_tool_call events.

What a receipt looks like

Each line of ~/.signet/audit/audit.jsonl is one record. The signed receipt inside looks like this:

{
  "v": 1,
  "id": "rec_a1b2c3d4e5f6a7b8...",
  "action": {
    "tool": "shell_exec",
    "params": { "command": "git status" },
    "params_hash": "sha256:...",
    "target": "openclaw://gateway/local",
    "transport": "stdio"
  },
  "signer": {
    "name": "openclaw-agent",
    "pubkey": "ed25519:...",
    "owner": "you@example.com"
  },
  "ts": "2026-04-26T14:30:00.000Z",
  "nonce": "rnd_...",
  "sig": "ed25519:..."
}

The signature covers the entire payload. Modify action.params.command from "git status" to "rm -rf /" and the signature stops verifying.

Adding policy enforcement

Audit-after-the-fact is good. Blocking dangerous calls before they run is better.

Create a policy at ~/.signet/policies/openclaw.yaml:

version: 1
name: openclaw-safe
default_action: allow

rules:
  - id: deny-rm-rf
    match:
      tool: shell_exec
      params:
        command:
          contains: "rm -rf"
    action: deny
    reason: "destructive command — never run without human approval"

  - id: rate-limit-network
    match:
      tool:
        one_of: [http_request, fetch_url]
    action: rate_limit
    rate_limit:
      window_secs: 60
      max_calls: 10

Wire it into the plugin config:

{
  "plugins": {
    "entries": {
      "signet": {
        "config": {
          "keyName": "openclaw-agent",
          "target": "openclaw://gateway/local",
          "policy": "~/.signet/policies/openclaw.yaml"
        }
      }
    }
  }
}

Restart OpenClaw. When the policy denies a call, you'll see:

[signet] policy denied shell_exec: destructive command — never run without human approval

OpenClaw skips the tool call entirely. The denial itself isn't signed — that's by design. Only allowed actions produce receipts. The denial is logged at warn level so it's still observable.

What this gives you

Without Signet	With Signet
"OpenClaw ran shell_exec" — log entry, editable	Ed25519 signature proving exactly what command, when, by which key
Ordering can be falsified	Hash chain breaks if any entry is removed or reordered
Trust your local logs	Verify offline with just the public key
No regulatory mapping	Maps to EU AI Act Article 12 "automatic event logging" requirement (effective August 2026)

Verifying as an external auditor

If someone else needs to verify your audit log — a security team, a regulator, you on a different machine — they only need:

The audit log file (~/.signet/audit/audit.jsonl)
The agent's public key (~/.signet/identities/openclaw-agent/openclaw-agent.pub.json)

No private key. No access to the OpenClaw runtime. They run:

signet audit --verify --keys-dir ./received-keys

If the chain is intact and every signature checks out, the audit log is authentic. If anyone tampered with anything, verification fails at the modified record.

This is the property that "we keep good logs" can never give you.

What's next

The plugin is open source (Apache-2.0 OR MIT). If you want:

Bilateral co-signing: server-side keys signing alongside the agent for two-party non-repudiation. Already in Signet core; can be wired into OpenClaw via a follow-up plugin.
Trust bundles: pin a published bundle of trusted public keys so verifiers don't need to track keys out-of-band.
Encrypted params: set encryptParams: true in the plugin config to wrap action.params in an XChaCha20-Poly1305 envelope keyed off the signing key. The signature chain stays verifiable; only key holders see the params.

Repo: https://github.com/Prismer-AI/signet
Plugin: https://github.com/Prismer-AI/signet/tree/main/packages/signet-openclaw-plugin
Issues: https://github.com/Prismer-AI/signet/issues

If you build something with this, I'd love to know. The OpenClaw maintainers in particular — feedback on the hook integration and security audit collector behavior would be valuable.

5 things missing from your AI agent audit logs (and how we fixed them in Signet v0.10)

willamhou — Thu, 07 May 2026 09:39:02 +0000

TL;DR — If your AI agent audit log only signs the intent (tool name + args), you're shipping demo-ware. Real audit needs 5 things most projects skip: outcome binding inside the signature scope, durable nonce stores, persistent server identity, portable forensic bundles, and encrypted-but-verifiable payloads. v0.10 of Signet ships all five today.

The gap between "demo passes" and "compliance team approves"

I've spent the last few months building Signet — an Ed25519-signed cryptographic audit SDK for AI agents. Along the way I noticed a pattern across every "let's add cryptographic audit to AI agents" project I looked at:

Generate a keypair.
Agent calls a tool.
Sign {tool, params} with Ed25519.
Stuff the signature into the audit log.
npm test passes. Demo done.

That's a working prototype. It is not what a compliance team can sign off on.

Below are the five things v0.9 of Signet did not have, that v0.10 (shipped today) does. If you're building anything in this space, I think you have to answer these before claiming production-ready.

1. Outcome binding — sign what happened, not just what was requested

Most agent receipts sign the request: {tool: "delete_user", params: {id: 42}}. The receipt becomes a record that the agent intended to call that tool.

But here's the audit question that actually matters: what did the server actually do? Did it succeed? Did it reject the call (policy violation)? Did it crash mid-flight? An intent-only receipt can't tell you.

In v0.10, Signet's bilateral receipt (v3) embeds an Outcome field with one of four states — verified, rejected, executed, failed — inside the signature scope. If a server claims it logged "executed" but the agent observed "failed", the receipt's signature is invalid.

// Rust: signing a bilateral receipt with outcome
let receipt = sign_bilateral_with_outcome(
    &server_key,
    &agent_receipt,
    &response,
    Outcome {
        status: OutcomeStatus::Executed,
        reason: None,
        error: None,
    },
)?;

// The status field is now under the signature.
// Tampering with it after the fact breaks verification.

# Python — same shape
agent.sign_bilateral_with_outcome(
    agent_receipt=receipt,
    response={...},
    outcome={"status": "executed"},
)

This is small in code but big in semantics. The receipt becomes a record of what actually happened, not just what was requested.

2. Durable nonce store — replay protection that survives a restart

Replay attacks against signed receipts are well-known. The standard mitigation: track nonces, reject any receipt whose nonce you've seen before.

The catch: most implementations use an in-memory hash set. Process restarts? Set is empty. Replay protection: gone for the duration of the warm-up window.

v0.10 ships FileNonceChecker — a JSON-file-backed nonce store, single-host pilot grade. It survives process restarts:

let checker = FileNonceChecker::new(Path::new("/var/lib/signet/nonces.json"))?;
let opts = BilateralVerifyOptions::default()  // now includes replay check by default
    .with_nonce_checker(Box::new(checker));

verify_bilateral(&receipt, &server_pubkey, &opts)?;

# Python equivalent
agent.verify_bilateral(
    receipt=receipt,
    server_pubkey=pubkey,
    nonce_store="/var/lib/signet/nonces.json",
)

// TypeScript — packages/signet-mcp-server FileNonceCache
import { FileNonceCache } from '@signet-auth/mcp-server';
const cache = new FileNonceCache('/var/lib/signet/nonces.json');
verifyRequest(req, { nonceCache: cache });

Single-host only — distributed nonce stores (Redis-backed, etc.) are post-1.0. But "single-host pilot" is what most teams need first, and v0.9 didn't even have that.

Behavioral break to note: BilateralVerifyOptions::default() now enables in-memory replay protection by default. Use BilateralVerifyOptions::insecure_no_replay_check() for forensic replay flows where nonce reuse is expected.

3. Persistent server identity — your trust bundle finally has something to anchor

Bilateral signing means both the agent and the server sign the receipt. The server's public key needs to be stable — otherwise nothing can pin it. Trust bundles, allowlists, audit replay — all break the moment the server pubkey rotates.

In v0.9, the server keypair was ephemeral: generated on signet proxy startup. New process, new pubkey. Compliance teams loved that.

In v0.10:

# Generate identities — agent and server must be different keys
signet identity generate --name agent-prod
signet identity generate --name openclaw-gateway

# Use them persistently — same pubkeys across every restart
signet proxy \
  --target ./my_mcp_server \
  --key agent-prod \
  --server-key openclaw-gateway

# Compose the trust bundle by hand — CLI bundle creation is on the roadmap;
# `signet trust` today exposes inspect / list / disable / revoke / rotate for
# editing existing bundles. Full schema lives in docs/guides/team-deployment.md.
cat > trust.json <<JSON
{
  "version": 1,
  "bundle_id": "pilot-2026-Q2",
  "org": "your-org",
  "env": "pilot",
  "generated_at": "2026-05-11T00:00:00Z",
  "agents": [{
    "id": "agent-prod-2026-05",
    "name": "agent-prod",
    "owner": "you",
    "pubkey": "$(signet identity export --name agent-prod)",
    "status": "active",
    "created_at": "2026-05-11T00:00:00Z"
  }],
  "servers": [{
    "id": "openclaw-gateway-2026-05",
    "name": "openclaw-gateway",
    "owner": "you",
    "pubkey": "$(signet identity export --name openclaw-gateway)",
    "status": "active",
    "created_at": "2026-05-11T00:00:00Z"
  }],
  "roots": []
}
JSON

# Sanity-check the bundle parses
signet trust inspect ./trust.json

The CLI also refuses to use the same key for both agent and server roles — a common foot-gun that silently invalidates the entire bilateral protocol.

4. Forensic bundle / restore — the artifact compliance teams actually want

A compliance team's question is rarely "is this audit log valid right now on this machine?" It's usually:

"Six months from now, on a different machine, with no access to your keystore, can you prove that this set of audit records was signed by [these specific keys] and that nothing was tampered with?"

That requires the audit data + the keys + the hash chain proof + version metadata, packaged together, self-verifying.

v0.10 ships:

# Producer side — package up an evidence bundle
signet audit --bundle ./evidence-2026-Q2 \
  --include-trust-bundle ./trust.json

# Output:
#   evidence-2026-Q2/
#   ├── records.jsonl       # the audit records
#   ├── manifest.json       # version + chain root + signer pubkeys
#   ├── hash-summary.txt    # human-readable chain summary
#   └── trust-bundle.json   # (optional) signer key set

# Verifier side — re-verify on ANY machine, no keystore required
signet audit --restore ./evidence-2026-Q2
# Verifies:
#   - every receipt's signature
#   - hash chain integrity
#   - timestamps in expected window
#   - trust bundle attestation chain

The --restore flow is replay-tolerant by design: it uses BilateralVerifyOptions::forensic() so re-verifying the same receipt twice doesn't fail nonce-replay checks. Forensic verification is a different mode from live verification, and v0.10 makes that explicit instead of confusing the two.

5. Encrypted audit envelope — verifiable AND confidential

Audit logs that are cryptographically verifiable are great for forensics. Audit logs that contain customer data in plaintext are great for incident response. These two goods are usually in tension.

v0.10 wraps the params field of audit records in an XChaCha20-Poly1305 encrypted envelope:

# When signing, opt the audit record into encrypted params
signet sign \
  --tool send_email \
  --params '{"to": "alice@example.com"}' \
  --key agent-prod \
  --encrypt-params

# Forensic decrypt during export — uses the matching local identity from
# the keystore (no separate "encryption-only" key flag today)
signet audit --export ./audit-decoded.jsonl --decrypt-params

The signature still covers the encrypted ciphertext — meaning verifiability is preserved without ever exposing the plaintext at rest. Anyone holding the audit log without the keystore sees only ciphertext; the --decrypt-params export materialises plaintext into the export file, scoped to that one operation.

v0.10 limitation: --encrypt-params reuses the signing agent's identity to derive the envelope key, so today "decrypt access" and "sign access" share one key. A dedicated encryption-only identity is on the roadmap; if you need that separation today, please open an issue.

This is the thing your security team has been asking for since you turned audit on.

Putting it together — a pilot deployment

Here's roughly what a single-host Signet pilot looks like with all 5 in play:

# 1. Generate identities (agent and server must be different keys)
signet identity generate --name agent-prod
signet identity generate --name openclaw-gateway   # stable server key

# 2. Compose the trust bundle JSON by hand
#    (paste pubkeys from `signet identity export --name <NAME>` — see Section 3)
$EDITOR ./trust.json
signet trust inspect ./trust.json   # verify it parses

# 3. Run the proxy with persistent agent + server identities + a policy
signet proxy \
  --target ./my_mcp_server \
  --key agent-prod \
  --server-key openclaw-gateway \
  --policy /etc/signet/policy.yaml

# 4. On the verifier side (same host or compliance host) — durable nonce store
#    is a `signet verify` flag, not a proxy flag in v0.10
signet verify <receipt-path> --nonce-store /var/lib/signet/nonces.json

# 5. Periodically bundle audit evidence for off-host handoff
signet audit --bundle ./evidence-2026-05-14 --include-trust-bundle ./trust.json

# 6. Compliance team can verify on a different machine
scp -r ./evidence-2026-05-14 compliance-host:
ssh compliance-host
signet audit --restore ./evidence-2026-05-14
# ✓ all signatures valid
# ✓ hash chain intact
# ✓ trust bundle attestation chain valid

Full operator runbook: docs/guides/team-deployment.md.

What's not in v0.10 (and where the work is going)

Honest disclaimers:

Single-host only. Distributed nonce stores (Redis, FoundationDB-backed) are post-1.0.
No managed key rotation. You can rotate manually with new identities and bundle merges, but there's no operator-friendly flow yet.
No multi-tenant isolation. Each pilot is a single-tenant deployment.
BilateralVerifyOptions::default() is a behavioral break. It now defaults to in-memory replay protection. If you were calling verify_bilateral() repeatedly on the same receipt for forensic replay, you need insecure_no_replay_check().
OpenClaw plugin is fail-closed by default. That means a misconfigured Signet will block all tool calls, not silently allow them. This is the right default but trips people up — see the pilot runbook for the readiness signal flow.

Try it

# Python
pip install signet-auth==0.10.0

# Rust
cargo install signet-cli

# OpenClaw gateway plugin
openclaw plugins install @signet-auth/openclaw-plugin

Repo: github.com/Prismer-AI/signet
v0.10 release notes: v0.10.0 release
Compliance mapping (SOC 2 / ISO 27001 / EU AI Act / NIST AI RMF): COMPLIANCE.md

If your AI agent audit log is missing any of these five things, I'd love to hear which one bites first in your environment. The fastest way to shape v0.11 is to tell me what doesn't work for you.

NIST NCCoE AI Agent Identity & Authorization: What Developers Need to Build

willamhou — Sat, 02 May 2026 01:00:00 +0000

Your agent can send an email, place an order, or merge a PR. If an auditor asks "prove it," what artifact do you hand them?

Plaintext logs aren't an answer. They're editable, deletable, and reorderable by anyone who controls the runtime. NIST has been quiet about this gap until recently — but in early 2026 they started lining up the answer.

On February 5, 2026, NIST NCCoE published a concept paper on AI agent identity and authorization surfacing four control areas any production agent deployment must address. Twelve days later, February 17, 2026, NIST CAISI launched the AI Agent Standards Initiative — more deliverables coming, exact timelines still emerging.

The concept paper is scoping work, not a prescriptive standard yet. But the four control areas are settled, and if you're building AI agents today, they tell you what you'll need to have working before NIST's normative output lands.

This post walks through each area, what it actually requires, and where the implementation gaps are today. Python code throughout.

The four control areas

The NCCoE concept paper surfaces four areas (the paper itself doesn't call them "pillars" — that's my framing for this post):

Identification — How are AI agents identified? Persistent vs task-specific identities, metadata for action scoping.
Authentication & Authorization — OAuth 2.0 extensions, ABAC, policy-based access control for agents as distinct principals. Delegation is discussed under authorization, not as a standalone area.
Access Delegation (sub-area of authorization) — Linking user identities to agents while preventing privilege escalation through delegation chains.
Auditing & Non-repudiation — "Mechanisms by which specific AI agent actions are attributed to their non-human entity for audit and forensic purposes."

The fourth area is where most production deployments are weakest today. Most agents generate logs. Few generate evidence.

Pillar 1: Identification

What NCCoE asks: A mechanism for issuing and resolving agent identities. Either persistent (the agent is a long-lived entity) or task-specific (a new identity per task).

What "good" looks like: Public-key cryptographic identity. Each agent has a keypair. The public key is the verifiable identifier. No central registry required.

Minimal working code:

from signet_auth import SigningAgent

# Persistent identity: create once, reuse across tasks
agent = SigningAgent.create("procurement-bot-01", owner="acme-corp")
print(f"Agent ID: {agent.public_key}")
# X9kF2mN8pQ3...   (raw base64 Ed25519 public key, 44 chars)

The Ed25519 public key is the agent identifier. A verifier who receives any artifact signed by this key knows it came from this agent. No registry lookup, no external service.

Gap to watch: NCCoE mentions SPIFFE/SPIRE for workload identity. If you're in a Kubernetes environment, the integration point is real — SPIRE issues short-lived SVIDs that can back your agent identity instead of long-lived local keys.

Pillar 2: Authorization

What NCCoE asks: OAuth 2.0 extensions, ABAC, or policy-based access control. The agent is a distinct principal, not a user. Decisions about what the agent can do happen at policy evaluation time.

What "good" looks like: The policy decision is evaluated before the action executes, and the decision is captured cryptographically so an auditor can later verify that the policy ran.

Minimal working code:

import json
from signet_auth import (
    SigningAgent,
    Receipt,
    parse_policy_yaml,
    sign_with_policy,
    load_signing_key,
    default_signet_dir,
)

# Define policy in YAML. Rules use `id:` and numeric comparisons use operator
# objects (e.g. {gt: 1000}), not string expressions.
policy_yaml = """
version: 1
name: procurement-safe
default_action: deny
rules:
  - id: allow-search
    match:
      tool: web_search
    action: allow
  - id: require-approval-over-threshold
    match:
      tool: place_order
      params:
        amount: {gt: 1000}
    action: require_approval
"""

# Canonical JSON policy is what gets hashed into the attestation.
policy_json = parse_policy_yaml(policy_yaml)

agent = SigningAgent("procurement-bot-01")
action_json = json.dumps({
    "tool": "web_search",
    "params": {"query": "laptop prices"},
    "params_hash": "",
    "target": "",
    "transport": "stdio",
})

# The low-level binding takes string args and returns (receipt_json, eval_json).
# It reads the signing key off disk; pass the raw key bytes yourself so the
# signing agent's in-memory key handle is not re-exported.
secret_key = load_signing_key(default_signet_dir(), "procurement-bot-01")
receipt_json, eval_json = sign_with_policy(
    secret_key,
    action_json,
    agent.name,
    agent.owner or "",
    policy_json,
)

receipt = Receipt.from_json(receipt_json)
# receipt.policy contains the attestation: which policy hash, which rule id,
# which decision. All inside the Ed25519 signature scope.

The CLI-equivalent one-liner is signet sign --key procurement-bot-01 --tool web_search --params '{"query":"laptop prices"}' --policy policy.yaml.

What this gives you: The policy version (hashed), the matched rule, and the decision are co-signed with the action. A verifier can confirm "the policy evaluated allow-search for this exact call" without trusting the runtime that produced it.

Gap to watch: NCCoE emphasizes that enforcement and attestation are separate concerns. The policy must actually run before the action, not after. If your implementation signs the policy decision post-hoc, it's not enforcement — it's reconstruction.

Pillar 3: Access Delegation

What NCCoE asks: Mechanisms for linking user identities to agents while preventing privilege escalation. Actions must trace back to the human authority that delegated them.

What "good" looks like: A cryptographically signed delegation chain. The root is a human (or org). Each delegation narrows scope, never widens. Every delegation has an expiration. Verification is offline.

Minimal working code:

from signet_auth import SigningAgent

# Assumes both keys exist already (signet identity create alice-human, etc.).
# Use SigningAgent.create(name, owner=...) on first run to mint them.
alice = SigningAgent("alice-human")
bot = SigningAgent("procurement-bot-01")

# Alice delegates scoped authority to bot, expiring in 1 hour.
# Scope fields are passed as keyword args; permissions can only narrow from here.
token_json = alice.delegate(
    bot.public_key,
    "procurement-bot-01",
    tools=["web_search", "place_order"],
    targets=["mcp://procurement-api"],
    max_depth=0,           # cannot re-delegate
    expires="2026-06-30T23:59:59Z",
)

# Bot signs an action carrying the delegation chain as proof.
# chain_json is a JSON array string of delegation tokens.
receipt_json = bot.sign_authorized(
    "place_order",
    params={"sku": "LAPTOP-01", "amount": 850},
    target="mcp://procurement-api",
    chain_json=f"[{token_json}]",
)

# The v4 receipt carries:
# - authorization.chain_hash: SHA-256 of the delegation chain
# - authorization.root_pubkey: Alice's public key
# - All inside the signature scope

Verification is offline. A third party with Alice's public key can verify the chain without contacting Alice:

scope_json = SigningAgent.verify_authorized(
    receipt_json,
    trusted_roots=[alice.public_key],
    clock_skew_secs=60,
)
# Returns the effective scope as a JSON string, or raises if:
# - Signature invalid
# - Chain scope narrowing violated
# - Delegation expired
# - Root not in trusted_roots

Gap to watch: NCCoE calls out the privilege escalation risk explicitly. Without "permissions only narrow, never widen," a compromised intermediate agent could issue itself broader permissions. The scope narrowing check must be enforced at verification time, not just at delegation time.

Pillar 4: Logging and Transparency

This is where most deployments fail the auditor test. NCCoE asks:

"Mechanisms by which specific AI agent actions are attributed to their non-human entity for audit and forensic purposes."

Most current implementations answer this with: "We write logs." That is not what NCCoE is asking for.

What "good" looks like:

Non-repudiation: The agent cannot later deny it took an action. Ed25519 signatures.
Tamper-evident: Modifying a log entry is detectable. Signatures break.
Tamper-evident ordering: Deleting or reordering entries is detectable. SHA-256 hash chains.
Independently verifiable: An auditor doesn't need access to the original runtime. Offline verification with the public key.

Most audit log implementations satisfy 0 of 4. The concept paper's language is specific:

"Mechanisms by which agent actions can be logged in a tamper-proof manner."

Minimal working code:

from signet_auth import SigningAgent, audit_verify_chain, default_signet_dir

# Assumes the key already exists; .create(...) on first run.
agent = SigningAgent("procurement-bot-01")

# Every action signed and appended to hash-chained audit log
agent.sign("web_search", params={"query": "laptop"}, audit=True)
agent.sign("place_order", params={"sku": "LAPTOP-01", "amount": 850}, audit=True)

# An auditor — who never ran this code — verifies the chain:
signet_dir = default_signet_dir()
status = audit_verify_chain(signet_dir)
print(f"Chain intact: {status.valid}")
print(f"Total records: {status.total_records}")

If any entry is modified: the Ed25519 signature fails. If any entry is deleted: the SHA-256 hash chain breaks. If any entry is reordered: the hash chain breaks. All detectable independently of the runtime that produced them.

How the four areas compose

The four areas aren't independent — they compose into a single verifiable artifact. Real Signet receipt shape (simplified):

Receipt {
    v: 4,
    id: "rec_...",
    action: {
        tool: "place_order",
        params: { ... },
        params_hash: "sha256:...",
        target: "mcp://procurement-api",
        transport: "stdio",
    },
    signer: {                         // Pillar 1 (Identification)
        pubkey: "...",                // raw base64 Ed25519
        name: "procurement-bot-01",
        owner: "acme-corp",
    },
    policy: {                         // Pillar 2 (Authorization)
        policy_hash: "sha256:...",
        policy_name: "procurement-safe",
        decision: "allow",
        matched_rules: ["allow-search"],
    },
    authorization: {                  // Pillar 3 (Delegation)
        chain_hash: "sha256:...",
        root_pubkey: "...",           // Alice's public key, raw base64
    },
    ts: "2026-04-30T12:00:00Z",
    nonce: "rnd_...",
    sig: "ed25519:...",               // binding the whole thing (Pillar 4)
}

Every field inside the signature scope is tamper-evident. A verifier with the root public key can confirm, offline:

Identity: this specific agent produced this receipt
Authorization: this agent was delegated specific scope by this root
Policy: this policy was evaluated and returned this decision
Action: this tool was called with these parameters at this time
Chain integrity: this receipt is part of an unbroken sequence

That's what NCCoE is asking for. Not logs. Not telemetry. Cryptographic receipts.

Where to start

The four areas are additive:

Start with Auditing & Non-repudiation (signed receipts). This is the foundation — without it, the other three don't produce verifiable evidence.
Add Identification. Name the agent with a public-key ID.
Add Delegation if your agents act on behalf of humans or other agents.
Add Authorization (policy) if you have deny rules that must be provably enforced.

Most teams start by retrofitting signed receipts onto an existing agent framework via callbacks. The examples above use Signet, which handles all four areas. The specific tool matters less than the pattern. Whatever you build, the verifier should be able to answer four questions without calling back to your infrastructure:

Who did this? (signer pubkey)
Were they allowed? (authorization chain root)
Did the policy approve it? (policy hash + decision)
Is the audit trail intact? (hash chain)

If your current implementation can't answer all four from a receipt file alone, that's the gap the NCCoE concept paper will push you to close.

What's next

NIST has signaled that more deliverables are coming under the AI Agent Standards Initiative — an Interoperability Profile is on the roadmap, but the published timeline and exact contents are still emerging. The direction is clear (cryptographic identity, signed delegation, tamper-evident audit) even if the final profile is not.

The IETF draft draft-farley-acta-signed-receipts is currently the most advanced concrete receipt specification — check Datatracker for the latest revision before citing a specific version.

If you want to follow the standards track:

NIST CAISI AI Agent Standards Initiative (main page)
NCCoE Concept Paper (the four control areas)
Express interest in working with CAISI
IETF draft-farley-acta-signed-receipts

The window between now and whenever the normative profile lands is when the formats get locked in. What you ship in the next quarter will probably dictate whether you're ahead of or behind the NIST curve.

Signet (Apache-2.0 OR MIT) is one open-source implementation in this space — Rust core, Python and TypeScript bindings — used as the working code in this post. pip install signet-auth.

How to Add Tamper-Evident Audit Trails to Your CrewAI Agents

willamhou — Wed, 29 Apr 2026 08:39:24 +0000

Your CrewAI crew kicks off a task. Agents delegate to each other, call tools, return results. But can you prove what each agent actually did?

CrewAI's built-in logs capture what happened. Cryptographic receipts prove it. The difference matters when an auditor, a customer, or a regulator asks "show me exactly what the agent did and prove it wasn't altered after the fact."

This tutorial adds Ed25519-signed, hash-chained audit trails to a CrewAI crew in under 5 minutes. Signet itself needs no external service, no signing API, no infrastructure — receipts verify offline with a public key. (Your CrewAI agent and any tools it uses still need their own keys; that part is unchanged.)

What you'll build

A CrewAI crew where every tool call produces a signed receipt containing:

What: which tool was called, with what parameters
Who: the agent's Ed25519 public key
When: timestamp
Proof: Ed25519 signature over JCS-canonicalized (RFC 8785) payload
Chain: SHA-256 hash linking to the previous receipt (tamper-evident ordering)

If anyone modifies a receipt after the fact, the signature breaks. If anyone deletes or reorders receipts, the hash chain breaks.

Install

pip install signet-auth[crewai] crewai-tools

Requires a CrewAI release that exposes the crewai.hooks global tool-hook API (tested with the current PyPI release). crewai-tools ships the SerperDevTool used below; it is not bundled into the [crewai] extra.

SerperDevTool calls Serper's web-search API and the CrewAI agent itself calls an LLM provider, so this exact demo also wants:

export OPENAI_API_KEY="sk-..."
export SERPER_API_KEY="..."

If you'd rather run zero-key, swap SerperDevTool() for any local tool — Signet signs whatever flows through crewai.hooks regardless of what the tool does.

Step 1: Create a signing identity

from signet_auth import SigningAgent, KeyNotFoundError

# Load an existing Ed25519 identity, or create one on first run.
# Keys live at ~/.signet/keys/ — the private key stays local.
try:
    agent = SigningAgent("my-crewai-agent")
except KeyNotFoundError:
    agent = SigningAgent.create("my-crewai-agent", owner="acme-corp")

print(f"Public key: {agent.public_key}")

No key server, no certificate authority. The private key stays on disk, the public key is what verifiers use.

Step 2: Install the signing hooks

CrewAI exposes global tool hooks. Signet plugs into them with one call:

from signet_auth.crewai import install_hooks

install_hooks(agent)

That's it. Every tool call across every agent in every crew is now signed automatically.

The hooks cover the full lifecycle: before_tool_call (what was called, signed before the tool runs) and after_tool_call (what was returned, hashed and signed). Recoverable signing errors (SignetError — bad payload, audit-log IO trouble) are caught and logged as a warning. Programmer errors that imply your code is in a bad state (e.g. calling agent.close() and then continuing to use the agent) still raise — by design, so you find them at dev time rather than silently dropping receipts in production.

Step 3: Run your crew normally

from crewai import Agent, Crew, Task
from crewai_tools import SerperDevTool

researcher = Agent(
    role="Research Analyst",
    goal="Find information on a given topic",
    backstory="Experienced analyst with attention to detail",
    tools=[SerperDevTool()],
)

task = Task(
    description="Research the weather in Tokyo",
    expected_output="A summary of today's weather",
    agent=researcher,
)

crew = Crew(agents=[researcher], tasks=[task])
result = crew.kickoff()

No changes to your agents, tools, or crew. The signing happens transparently through the hooks.

Step 4: Inspect the receipts

import json
from signet_auth.crewai import get_receipts

for i, receipt in enumerate(get_receipts()):
    data = json.loads(receipt.to_json())
    print(f"\nReceipt #{i+1}")
    print(f"  Tool:       {data['action']['tool']}")
    print(f"  Params hash: {data['action']['params_hash']}")
    print(f"  Signature:   {data['sig'][:40]}...")
    print(f"  Timestamp:   {data['ts']}")

Output:

Receipt #1
  Tool:       serper_dev
  Params hash: sha256:a1b2c3...
  Signature:   ed25519:Mz4xNTk2NjQ0NDgw...
  Timestamp:   2026-04-20T14:30:00Z

Receipt #2
  Tool:       _tool_end
  Params hash: sha256:d4e5f6...
  Signature:   ed25519:Nk5yODk3MjE1Njg4...
  Timestamp:   2026-04-20T14:30:02Z

Notice the start/end pair: the first receipt captures the tool call, the second captures a SHA-256 hash of the output. Together they prove what was called and the hash of what it returned. (The full output stays in your application; the receipt only signs the hash, which is enough to detect any after-the-fact tampering of an output you've stored elsewhere.)

Step 5: Verify a receipt

Anyone with the public key can verify, offline:

from signet_auth import verify

receipts = get_receipts()
receipt = receipts[0]

is_valid = verify(receipt, agent.public_key)
print(f"Valid: {is_valid}")  # True

Tamper with any field and verification fails:

data = json.loads(receipt.to_json())
data["action"]["tool"] = "evil_tool"  # tamper

from signet_auth import Receipt
tampered = Receipt.from_json(json.dumps(data))
print(verify(tampered, agent.public_key))  # False

Step 6: Verify the audit chain

The audit log is a hash-chained JSONL file. Each entry's hash covers the previous entry, so deleting or reordering receipts breaks the chain:

from signet_auth import audit_verify_chain, default_signet_dir

signet_dir = default_signet_dir()
chain_status = audit_verify_chain(signet_dir)
print(f"Chain intact: {chain_status.valid}")
print(f"Entries: {chain_status.total_records}")

Step 7: Clean up (optional)

When you're done signing, uninstall the hooks:

from signet_auth.crewai import uninstall_hooks

uninstall_hooks()

Why this matters for CrewAI specifically

CrewAI's strength is agent-to-agent delegation. A researcher agent delegates to a writer agent, who calls tools, who returns results that feed back into the chain. When something goes wrong, "which agent did what" becomes a real question.

Signed receipts answer that question independently of CrewAI's own logs. CrewAI's ToolCallHookContext does expose agent to the hook today, but Signet's current install_hooks() binding ties one signing identity to the global hook registration; per-call routing to a per-agent key is on the roadmap. For now, if you need separate keys per agent, the practical pattern is: install with one agent's key, run that agent's task, call uninstall_hooks(), then re-install with the next agent's key.

Either way, the audit trail proves not just what happened but who signed it.

What this gives you

Without Signet	With Signet
"The research agent called serper_dev" (log entry)	Ed25519 signature proving it, verifiable by anyone with the public key
Logs can be edited after the fact	Signature breaks if any field is modified
No ordering proof	Hash chain breaks if receipts are deleted or reordered
Trust CrewAI's logs	Verify independently, offline

When you need this

Regulated industries: EU AI Act Article 12 requires "automatic recording of events" for high-risk systems. Tamper-evident signed receipts can support that traceability + log-integrity requirement (the legal sufficiency check is your auditor's call, not Signet's — but they need something better than rotatable plaintext logs to point at).
Enterprise deployments: When the question is "can you prove what the agent did?", signed receipts are the answer.
Agent-to-agent: CrewAI's core pattern — when one agent verifies another's work, signatures make it cryptographic, not just log-based.
Incident response: After something goes wrong in a multi-agent crew, tamper-evident receipts let you reconstruct exactly what happened without trusting anyone's claim.

Next steps

Bilateral co-signing: Have both the agent and the tool server sign each interaction independently. Neither party can fabricate receipts. See signet proxy for MCP integration.
Policy attestation: Evaluate YAML policy rules and include the decision (allow/deny/require_approval) inside the signed receipt.
Delegation chains: Prove that Agent A was authorized by Human B to perform a specific action with scoped constraints. Useful when CrewAI agents are acting on behalf of specific users.

All of these are in signet-auth today.

pip install signet-auth[crewai]

GitHub: Prismer-AI/signet

Signet is open source (Apache-2.0 OR MIT). Rust core with Python and TypeScript bindings. The signing layer needs no external service or signing API — receipts verify offline with the public key.

Auto-updating Kubernetes workloads: an annotation-driven rollout, with circuit breaker

willamhou — Mon, 27 Apr 2026 04:20:33 +0000

You have ten agent pods on a cluster, each running a different runtime image. Every Tuesday somebody publishes a new version of one of them. Are you going to kubectl set image ten things by hand? Are you sure you'll know if v1.4.2 was the one that wedged the pods?

This post is about the auto-update controller in k8s4claw, a Kubernetes operator for AI agent runtimes. It polls OCI registries on cron, picks the highest semver tag that matches your constraint, flips a single annotation, and lets the main reconciler do the rollout. If the rollout doesn't go ready inside a timeout, it rolls back. If it rolls back too many times, it stops trying and asks for a human.

The whole controller is one Go file (autoupdate_controller.go), about 470 lines. This is the design walkthrough — not the API reference, not the README.

The shape of the problem

A Claw resource looks like this when auto-update is on:

spec:
  runtime: openclaw
  autoUpdate:
    enabled: true
    schedule: "0 3 * * *"           # daily at 3 AM
    versionConstraint: ">=1.0.0,<2"
    healthTimeout: "10m"
    maxRollbacks: 3

Five fields, and the controller has to:

Wake up on schedule (cron expression, not "every N seconds").
Ask the registry what tags exist for ghcr.io/prismer-ai/k8s4claw-openclaw.
Filter to semver tags inside the constraint.
Pick the highest one that's strictly greater than what's running, skipping any version we've already tried and rolled back.
Apply it — but not by patching the StatefulSet directly.
Watch readiness for healthTimeout (10 min default).
If both sts.Status.UpdatedReplicas and sts.Status.ReadyReplicas reach the desired count: record success, reset rollback counter.
If it times out: clear the target-image annotation so the main reconciler reverts to the runtime adapter's default image, mark this version as failed, increment rollback counter.
After maxRollbacks consecutive failures: open the circuit and stop trying. Subsequent version checks then emit a "version available, circuit open" event/condition instead of applying the new image.

The non-obvious bits are where the state lives and how the rollout actually happens. Both turn out to use the same trick.

Mechanism 1 — annotations drive the in-flight rollout

The auto-update controller never holds in-memory state across reconciles. State lives in two places on the Claw resource:

Annotations drive the in-flight update — what image we want, what phase we're in, when we started.
status.autoUpdate holds the durable bookkeeping — current version, available version, rollback count, circuit-breaker flag, failed-version list, version history.

The three annotations:

const (
    annotationTargetImage = "claw.prismer.ai/target-image"
    annotationUpdatePhase = "claw.prismer.ai/update-phase"
    annotationUpdateStart = "claw.prismer.ai/update-started"
)

target-image — the full image reference we want running (ghcr.io/.../openclaw:1.2.0). Stays set after a successful update.
update-phase — currently only HealthCheck or absent. Absent = idle. Anything else falls through to the idle path.
update-started — RFC3339 timestamp of when we set the phase annotation. Used by the health-check timer.

Reconcile is a two-way fork on the phase:

phase := claw.Annotations[annotationUpdatePhase]
if phase == "HealthCheck" {
    return r.reconcileHealthCheck(ctx, &claw)
}
// otherwise: idle — check if a version poll is due

This means the controller is stateless and idempotent. If the operator pod restarts mid-update, the next reconcile reads the annotation back from etcd and picks up exactly where the old one left off. There's no map[types.NamespacedName]updateState to rehydrate, no leader-election dance for in-flight work. Kubernetes is the database. The controller is a function over its current state.

The other thing this gets you: kubectl describe claw foo shows the in-flight update verbatim. No tracing, no controller logs to grep. The state is on the resource.

Mechanism 2 — the rollout is one annotation

Here's the thing that surprised me when I wrote this controller. The auto-update logic does not patch the StatefulSet. It does not touch pods. It does this:

targetImage := baseImage + ":" + newVersion
claw.Annotations[annotationTargetImage] = targetImage
claw.Annotations[annotationUpdatePhase] = "HealthCheck"
claw.Annotations[annotationUpdateStart] = now.Format(time.RFC3339)
r.Update(ctx, &claw)

That's it. That's the whole "apply a new version" code path.

The rollout actually happens because the main ClawReconciler watches the same Claw resource and rebuilds the pod template every reconcile. It checks the annotation when it does:

// claw_controller.go
podTemplate := adapter.PodTemplate(claw)

// Auto-update: override runtime image if target-image annotation is set.
if targetImage := claw.Annotations["claw.prismer.ai/target-image"]; targetImage != "" {
    for i := range podTemplate.Spec.Containers {
        if podTemplate.Spec.Containers[i].Name == "runtime" {
            podTemplate.Spec.Containers[i].Image = targetImage
            break
        }
    }
}

So the auto-update controller is purely a signal source. It says "I want this image to be running." The main reconciler is responsible for translating that into a StatefulSet update, which then translates into a rolling pod replacement, which the auto-update controller observes via sts.Status.UpdatedReplicas and sts.Status.ReadyReplicas (both required — see Mechanism 4).

This separation matters because:

Rollback is mostly just deleting annotations. When we roll back, we delete(claw.Annotations, annotationTargetImage) and the main reconciler reverts to the adapter's default image on the next pass. No special "rollback path" in the StatefulSet logic. (The update-phase and update-started annotations also get cleared.)
Manual image overrides keep working. If somebody set target-image by hand for a hotfix, the main reconciler honors it for the pod template. The auto-update controller compares against status.CurrentVersion (not the annotation) when deciding whether to propose a new version, so a manual override doesn't accidentally redirect what the controller thinks "current" means.
The auto-update controller can be removed entirely without breaking anything. Stale annotation, sure, but the cluster doesn't fall over.

If you're writing a new controller and you find yourself directly mutating sub-resources, ask whether you could mutate annotations on the parent CR instead and let the existing reconciler do the work. It's almost always cleaner.

Mechanism 3 — semver resolution

The version-picking logic is in internal/registry/resolver.go:

func ResolveBestVersion(tags []string, constraint, current string, failedVersions []string) (string, bool) {
    c, err := semver.NewConstraint(constraint)
    if err != nil {
        return "", false
    }

    var currentVer *semver.Version
    if current != "" {
        currentVer, _ = semver.NewVersion(current)
    }

    failedSet := make(map[string]bool, len(failedVersions))
    for _, f := range failedVersions {
        failedSet[f] = true
    }

    var best *semver.Version
    for _, tag := range tags {
        v, err := semver.NewVersion(tag)
        if err != nil {
            continue // skip non-semver tags like "latest", "sha-abc"
        }
        if !c.Check(v) {
            continue
        }
        if failedSet[v.Original()] {
            continue
        }
        if currentVer != nil && !v.GreaterThan(currentVer) {
            continue
        }
        if best == nil || v.GreaterThan(best) {
            best = v
        }
    }

    if best == nil {
        return "", false
    }
    return best.Original(), true
}

Three subtleties worth flagging:

Non-semver tags are silently dropped. latest, sha-abc1234, nightly — they all fail semver.NewVersion() and get skipped. This is the right default for an auto-updater: anything you can't compare to a version constraint is something you don't want to roll into automatically.
failedVersions is checked after the constraint check, by exact original tag string. A version that has been rolled back gets recorded in Status.AutoUpdate.FailedVersions and is excluded from future auto-selection. The match is on v.Original(), so "1.2.0" and "v1.2.0" would be treated as different strings — the constraint check is semver-aware, but the failed-version filter is not. To retry a failed version automatically you have to clear it from status manually; you can also force a manual rollout via the annotations (see the circuit-breaker section). This is conservative on purpose — the assumption is that if v1.2.0 wedged your pods once, the next 3 AM cron run isn't going to fix that.
!v.GreaterThan(currentVer) excludes equal. Reinstalling the same version on every cron tick would be a noisy mistake.

The auto-update controller also has an early bail-out for digest-pinned images:

currentImage := claw.Annotations[annotationTargetImage]
if currentImage != "" && registry.IsDigestPinned(currentImage) {
    logger.Info("skipping auto-update: image is digest-pinned", "image", currentImage)
    return r.requeueAtNextCron(spec), nil
}

It checks the target-image annotation, not the actual running image. IsDigestPinned is just strings.Contains(image, "@sha256:"). If you set target-image to a digest-pinned reference (manually or via a previous override), the controller stops touching that Claw on its cron schedule. If the annotation is absent, the check is skipped and version polling proceeds normally.

Mechanism 4 — health verification

Once the annotation is set and the main reconciler has rolled the StatefulSet, the auto-update controller requeues every 15 seconds and watches readiness:

desiredReplicas := int32(1)
if sts.Spec.Replicas != nil {
    desiredReplicas = *sts.Spec.Replicas
}
if sts.Status.UpdatedReplicas >= desiredReplicas &&
   sts.Status.ReadyReplicas >= desiredReplicas {
    // Health check passed.
}

Two conditions, both required:

UpdatedReplicas — pods running the new template, not the old one. Without this check, you'd "succeed" the moment the old pods are still ready before the rollout has even started.
ReadyReplicas — pods passing their readiness probes.

If both clear within healthTimeout (10 min default), we record success: reset rollback counter, reset circuit breaker, append a Healthy entry to version history, and clear the update-phase and update-started annotations. Note we deliberately keep target-image — it's still the signal the main reconciler uses to override the runtime container image, and clearing it would silently revert the running pods to the adapter default on the next reconcile.

If the timer expires first:

if r.clock().Since(startedAt) > healthTimeout {
    return r.rollback(ctx, claw, "health check timed out")
}

We also roll back if the StatefulSet itself disappears past the timeout (the resource was deleted while we were watching), or if the start-time annotation is somehow malformed (you have to handle that — annotations are just strings).

15 seconds is a polling interval, not a deadline. The actual deadline is healthTimeout, parsed from the spec. If you're upgrading a heavyweight runtime that takes 8 minutes to warm up, set healthTimeout: 15m and the controller will wait that long.

Mechanism 5 — circuit breaker

Rolling back once is a hiccup. Rolling back three times in a row is a system telling you to stop.

maxRollbacks := defaultMaxRollbacks  // 3
if spec.MaxRollbacks > 0 {
    maxRollbacks = spec.MaxRollbacks
}
if status.RollbackCount >= maxRollbacks {
    status.CircuitOpen = true
    SetAutoUpdateCircuit(claw.Namespace, claw.Name, true)
    r.Recorder.Event(claw, corev1.EventTypeWarning, EventAutoUpdateCircuitOpen,
        fmt.Sprintf("Circuit breaker opened after %d rollbacks", status.RollbackCount))
}

When the circuit is open, the main Reconcile path detects new versions and emits an event saying "version X is available, but we're not applying it." The user sees this on kubectl describe claw foo and can decide whether to investigate or override.

The recovery story is deliberately blunt: the controller does not auto-recover the circuit. There's no "wait 24 hours and try again" timer, no exponential backoff, no separate trial deployment. The gating check is if status.CircuitOpen — it doesn't look at RollbackCount. So the recovery paths are:

A human patches status.autoUpdate.circuitOpen to false (and usually rollbackCount to 0 for a clean slate). The next cron tick will resume normal version polling.
A human forces an update path some other way — for example, setting all three annotations (target-image to a known-good image, update-phase to HealthCheck, update-started to a fresh RFC3339 timestamp) by hand. The phase check happens before the circuit check, so the next reconcile enters reconcileHealthCheck directly and, on a successful rollout, resets RollbackCount and CircuitOpen. (FailedVersions is left intact, so the controller still won't auto-pick the versions that failed before.) Skipping the timestamp or pointing target-image at something that won't go ready will just cause an immediate rollback, so the manual path needs all three pieces.

The argument for this design: three consecutive bad versions probably means something is wrong outside the controller's view (broken upstream image, broken probe, broken cluster networking). Auto-recovery would just rediscover the broken state on a fresh schedule and burn through more rollouts. We'd rather page somebody.

If you wanted to add a "soak then retry" mode, the natural place is to have the recovery logic clear CircuitOpen after, say, the third consecutive successful version-poll-with-no-update — i.e., a stable period where there's nothing new to try. That's a reasonable PR.

Mechanism 6 — version history (with a cap)

Every successful update and every rollback appends an entry to Status.AutoUpdate.VersionHistory:

status.VersionHistory = append(status.VersionHistory, clawv1alpha1.VersionHistoryEntry{
    Version:   version,
    AppliedAt: metav1.Now(),
    Status:    clawv1alpha1.VersionHistoryHealthy,  // or VersionHistoryRolledBack
})
trimVersionHistory(status)

trimVersionHistory exists because etcd objects have size limits, and a Claw that's been updating daily for two years can otherwise accumulate 700+ history entries:

const maxVersionHistory = 50

func trimVersionHistory(status *clawv1alpha1.AutoUpdateStatus) {
    if len(status.VersionHistory) > maxVersionHistory {
        status.VersionHistory = status.VersionHistory[len(status.VersionHistory)-maxVersionHistory:]
    }
}

50 entries is enough to debug the last few months of activity. If you need long-term audit, scrape the controller's events into your observability stack. Status fields are not an audit log.

The Update vs Status.Update dance

Annotations live on the resource (under metadata). Status fields live under .status. In Kubernetes, these are written through different subresources:

r.Update(ctx, claw) — writes metadata and spec. Bumps resourceVersion.
r.Status().Update(ctx, claw) — writes .status. Also bumps resourceVersion.

When a single reconcile needs to write both — like the "start an update" path, which sets three annotations and writes status fields — the in-memory claw object goes stale between the two calls. The controller does an explicit re-fetch in between:

// Update annotations first, then re-fetch and merge status.
if err := r.Update(ctx, &claw); err != nil {
    return ctrl.Result{}, fmt.Errorf("failed to set target-image annotation: %w", err)
}
// Re-fetch to get updated resourceVersion before status update.
if err := r.Get(ctx, req.NamespacedName, &claw); err != nil {
    return ctrl.Result{}, fmt.Errorf("failed to re-fetch after annotation update: %w", err)
}
mergeAutoUpdateStatus(&claw, status)
for _, c := range pendingConditions {
    apimeta.SetStatusCondition(&claw.Status.Conditions, c)
}
if err := r.Status().Update(ctx, &claw); err != nil {
    return ctrl.Result{}, fmt.Errorf("failed to update status: %w", err)
}

The re-fetch picks up the new resourceVersion so Status().Update doesn't conflict with the write we just did. Without it you'll see 409 errors under any non-trivial reconcile rate.

mergeAutoUpdateStatus is the other half. It copies our locally-tracked status fields one at a time into the freshly-fetched object instead of swinging claw.Status.AutoUpdate to a different pointer. Field-by-field copy is conservative: if a future field is added to AutoUpdateStatus and we forget to track it locally, a wholesale pointer replacement would silently zero it. The merge style makes the controller's status writes additive within the auto-update sub-object.

func mergeAutoUpdateStatus(claw *clawv1alpha1.Claw, local *clawv1alpha1.AutoUpdateStatus) {
    if claw.Status.AutoUpdate == nil {
        claw.Status.AutoUpdate = &clawv1alpha1.AutoUpdateStatus{}
    }
    s := claw.Status.AutoUpdate
    s.CurrentVersion = local.CurrentVersion
    s.AvailableVersion = local.AvailableVersion
    // ... field-by-field copy ...
}

If your controller writes both annotations and status, you need this dance. If it only writes one, you don't.

Testability: Clock and TagLister

Two interfaces, both for tests:

type TagLister interface {
    ListTags(ctx context.Context, image string) ([]string, error)
}

type Clock interface {
    Now() time.Time
    Since(t time.Time) time.Duration
}

TagLister lets unit tests inject []string{"1.0.0", "1.1.0", "2.0.0-rc1"} instead of hitting GHCR. Clock lets them advance time without time.Sleep. Both have one-line production implementations and one-line fake implementations.

These get wired in the manager setup:

// cmd/operator/main.go
registryClient := clawregistry.NewRegistryClient()
&controller.AutoUpdateReconciler{
    Client:    mgr.GetClient(),
    Scheme:    mgr.GetScheme(),
    Recorder:  mgr.GetEventRecorderFor("autoupdate-controller"),
    TagLister: registryClient,
    // Clock is left nil; clock() falls back to realClock{}.
}

In the reconcile-path tests, both fields get fakes:

cl := fake.NewClientBuilder().
    WithScheme(scheme).
    WithObjects(claw).
    WithStatusSubresource(claw).
    Build()
r := &AutoUpdateReconciler{
    Client:    cl,
    Scheme:    scheme,
    Recorder:  record.NewFakeRecorder(10),
    TagLister: &testTagLister{tags: []string{"1.0.0", "1.1.0"}},
    Clock:     &testClock{now: time.Now()},
}

The autoupdate unit tests use controller-runtime/pkg/client/fake — no envtest API server, no kube-apiserver process, just an in-memory client backed by typed scheme. They create a Claw, run a single Reconcile pass with a controlled clock, and assert on annotations and Status.AutoUpdate. No real registry calls, no real timers, no flake. Total run time is sub-second per test.

If you find yourself reaching for time.Now() or hitting an external API directly inside a reconciler, stop and define the interface first. Future-you writing tests will thank present-you.

What we didn't do (on purpose)

Pre-flight image probe. We don't pull the new image and try docker run it on a node before flipping the StatefulSet. That would be a much heavier dependency (DaemonSet? privileged container?) and the StatefulSet rollout is itself a kind of probe — the readiness check just runs in production.
Canary deploys. Roll one pod, observe, then the rest. For most agent workloads we have, replicas is 1 and there's nothing to canary against. For higher-replica deployments, this is a worthwhile follow-up — the existing state machine could grow a Canary phase between idle and HealthCheck.
Webhook-driven updates. Push from registry instead of poll. Simpler operationally but creates an inbound dependency from the registry to the cluster, which is not a thing most clusters want. Cron-poll wins on operational simplicity.
Cross-namespace coordination. If you have ten Claws on the same image and a bad version drops, they will all roll back independently. We considered tying them together via a shared ClawImageGroup resource and decided the complexity wasn't worth it. The circuit breaker + failed-versions list is good enough: each Claw learns from its own pain.
Image signature verification. Sigstore / cosign integration would slot in at IsDigestPinned's level — verify, then set target-image. We didn't ship it because the projects we serve aren't there yet, but it's an obvious next step for security-sensitive deployments.

Testing

Unit tests are split across three files:

internal/controller/autoupdate_reconcile_test.go — the largest reconcile-path set. Covers initiating an update, skipping digest-pinned images, health-check success, rollback on timeout, circuit-breaker opening after consecutive rollbacks, StatefulSet-not-found behavior, invalid update-started triggering an immediate rollback, custom healthTimeout override, and the schedule-not-due requeue path.
internal/controller/autoupdate_controller_test.go — a mix: helper-function coverage (extractVersionFromImage, trimVersionHistory, containsString, cron-due math, the realClock fallback inside clock()) plus a smaller batch of reconcile tests for the disabled/no-new-version/not-found/circuit-already-open paths.
internal/controller/autoupdate/autoupdate_controller_test.go — an older parallel suite kept alive against the same controller code.

The reconcile-path tests pre-load a Claw (and optionally a StatefulSet with the desired readiness state), run a single Reconcile pass, and assert on annotations or Status.AutoUpdate. Most tests are under 50 lines. The fake clock and fake tag lister make timing deterministic, which is the main reason the tests aren't flaky.

What this bought us

A ~470-line controller that does cron-driven, semver-filtered, health-verified, automatically-rolling-back image updates for a CRD, with a circuit breaker and version history. All in-flight state lives on the Claw resource (annotations for phase, .status for durable bookkeeping), so the controller has no in-memory state to lose across restarts. Supported runtime types are mapped to their base OCI images via a small ImageForRuntime(string) string helper — adding a new runtime there is one switch-case, not a controller change. Runtimes without an entry are silently skipped by auto-update (we currently have a couple of those — hermesrs and k8sops — that don't track a public OCI release cadence). The rest of the controller works in plain semver tags.

The thing I'd point a junior K8s-controller author at, in this code, is the annotation-driven separation: the controller doesn't do the rollout, it asks for the rollout. Once you internalize that, a lot of K8s controllers get smaller.

What to look at next

The k8s4claw repo if you want the full operator
autoupdate_controller.go for the controller in one file
registry/resolver.go for the version picker
The IPC bus deep dive — Post 2 of this series, on how channel sidecars talk to the runtime

Open source, Apache-2.0. If you've built an auto-updater that handles canary deploys or signature verification, I'd genuinely like to read your code. Drop a link in the comments.

Building an IPC bus for Kubernetes sidecars: WAL, DLQ, and ring-buffer backpressure

willamhou — Thu, 23 Apr 2026 02:18:14 +0000

If you put two sidecars in a pod and ask them to talk to each other over HTTP, sooner or later one of them crashes mid-request and you lose a message. If you do it enough times, you reinvent a message bus.

This post is about the small in-pod message bus we ended up writing for k8s4claw, a Kubernetes operator for AI agent runtimes. The bus sits between channel sidecars (Slack, Discord, Webhook) and the agent runtime container. It has four wire protocols, a write-ahead log, a BoltDB-backed dead letter queue, and a ring buffer with backpressure. All of it is open source (internal/ipcbus/), around 2k lines of Go.

This post is the design doc you actually want to read, not the one we had to write.

The shape of the problem

A Claw pod looks like this when it has a Slack channel attached:

┌──────────────────────────────────────────────┐
│  Pod                                         │
│                                              │
│  [channel-slack] ──UDS──► [ipc-bus] ──►┐     │
│                                        ▼     │
│                                  [runtime]   │
│                                              │
└──────────────────────────────────────────────┘

Three containers. The channel sidecar reads from Slack. The runtime is the actual AI agent. The IPC bus is a native sidecar (init container with restartPolicy: Always) that routes messages between them.

The naive version of this is: let the two containers talk HTTP directly. The reality is that at least four things are going to go wrong:

The runtime will be overloaded when a Slack event arrives and we need somewhere to buffer it.
The runtime will crash mid-response and we need to redeliver.
A slow downstream (say, a user's laptop on 3G) will fall behind and we need to push back instead of dropping.
Two different runtimes we support speak four different wire protocols. HTTP isn't enough.

So we wrote a bus. Let me walk through the four mechanisms that earn their keep.

Mechanism 1 — length-prefix framing

This isn't glamorous, but it's the first thing you get wrong in a message bus.

Every Message is a JSON blob on the wire:

type Message struct {
    ID            string          `json:"id"`
    Type          MessageType     `json:"type"`
    Channel       string          `json:"channel,omitempty"`
    CorrelationID string          `json:"correlationId,omitempty"`
    ReplyTo       string          `json:"replyTo,omitempty"`
    Timestamp     time.Time       `json:"timestamp"`
    Payload       json.RawMessage `json:"payload,omitempty"`
}

On the wire it looks like [4-byte big-endian length][JSON bytes]:

const (
    MaxMessageSize  = 16 * 1024 * 1024
    FrameHeaderSize = 4
)

func WriteMessage(w io.Writer, msg *Message) error {
    data, err := json.Marshal(msg)
    if err != nil {
        return fmt.Errorf("failed to marshal message: %w", err)
    }
    if len(data) > MaxMessageSize {
        return fmt.Errorf("message size %d exceeds maximum %d",
            len(data), MaxMessageSize)
    }

    frame := make([]byte, FrameHeaderSize+len(data))
    binary.BigEndian.PutUint32(frame, uint32(len(data)))
    copy(frame[FrameHeaderSize:], data)
    _, err = w.Write(frame)
    return err
}

Why length-prefix instead of newline-delimited JSON? Because JSON payloads can contain newlines inside strings and you'd have to escape them on the wire. Length-prefix framing just works: a reader reads 4 bytes, gets the length, reads that many bytes, deserializes. No lookahead, no escape tables.

The 16 MB cap is there to fail loudly rather than run out of memory on a malformed header. In practice our real messages are well under 64 KB.

Mechanism 2 — four bridge protocols behind one interface

Different runtimes speak different things:

Runtime	Protocol	Why
OpenClaw	WebSocket	Full-duplex, JSON-native, easy from Node.js
NanoClaw	UDS	Lowest overhead for same-pod communication
ZeroClaw	SSE	Already has an HTTP API, SSE for server-push
PicoClaw	TCP	Minimal client, hand-rolled in 50 lines

The bus abstracts them behind one interface:

type RuntimeBridge interface {
    Connect(ctx context.Context) error
    Send(ctx context.Context, msg *Message) error
    Receive(ctx context.Context) (<-chan *Message, error)
    Close() error
}

Four methods. Adding a new protocol is one file (example: TCP bridge):

type TCPBridge struct{ streamBridge }

func (b *TCPBridge) Connect(ctx context.Context) error {
    conn, err := (&net.Dialer{}).DialContext(ctx, "tcp", b.addr)
    if err != nil {
        return err
    }
    b.conn = conn
    return nil
}

streamBridge is a shared base that implements Send/Receive/Close on top of any net.Conn. It handles context.Context deadlines properly:

func (b *streamBridge) Send(ctx context.Context, msg *Message) error {
    b.mu.Lock()
    defer b.mu.Unlock()

    if b.conn == nil {
        return fmt.Errorf("not connected")
    }

    // Respect context deadline for the write.
    if deadline, ok := ctx.Deadline(); ok {
        _ = b.conn.SetWriteDeadline(deadline)
        defer func() { _ = b.conn.SetWriteDeadline(time.Time{}) }()
    }

    return WriteMessage(b.conn, msg)
}

The subtle bit is Receive. ReadMessage blocks on the socket. If the caller cancels the context, we want the read to unblock. So Receive spawns a second goroutine whose only job is to watch the context and call Close on the conn, which makes the blocked ReadMessage return with an error.

go func() {
    select {
    case <-ctx.Done():
        _ = b.conn.Close()
    case <-b.closed:
    }
}()

The SSE bridge is the odd one out because SSE is unidirectional (server → client, event-stream format) and we need bidirectional. So it uses an HTTP POST for send and an SSE GET /events for receive, with exponential-backoff reconnect on the stream:

backoff := time.Second
for {
    // ... connect and read events ...
    time.Sleep(backoff)
    backoff *= 2
    if backoff > 30*time.Second {
        backoff = 30 * time.Second
    }
}

Mechanism 3 — Write-Ahead Log (WAL)

This is the one that earns the bus the right to exist.

When a message comes in from a channel sidecar, the router does three things:

Append a WAL entry to disk (emptyDir-backed) with state pending.
Call bridge.Send(ctx, msg) to hand it off to the runtime bridge.
Mark the WAL entry complete as soon as Send returns success. If Send fails, call scheduleRetry.

We delivery-mark on transport success (the bridge accepted the bytes), not on runtime ack. We considered a runtime-ack round-trip and decided against it: it doubles round-trips, forces every runtime to implement ack semantics, and our Message.ID is already idempotency-safe so downstream retries aren't harmful. If a message leaves bridge.Send OK but the runtime crashes before processing it, we lose that one message. Tradeoff: acceptable for a chat agent, not acceptable for a payment system. Different design calls, different bus.

scheduleRetry increments Attempts on the WAL entry. After maxRetryAttempts = 5, the entry is marked dlq and a copy is parked in the DLQ.

The WAL is a JSON-lines file. Each line is a WALEntry:

type WALEntry struct {
    ID       string   `json:"id"`
    Channel  string   `json:"channel"`
    State    WALState `json:"state"`       // pending | complete | dlq
    Attempts int      `json:"attempts"`
    TS       string   `json:"ts"`
    Msg      *Message `json:"msg,omitempty"`
}

JSON-lines is nice because you can cat wal.log | jq during an incident and see exactly what the bus was doing. It's also append-only, which means writes are O(1) and you never corrupt the middle of the file on a crash — at worst you have a half-written last line, which the recovery code handles.

The interesting operation is compaction. The file grows without bound otherwise. Compaction rewrites the file keeping only pending entries:

func (w *WAL) Compact() error {
    // ... write all pending entries to wal.log.tmp ...
    // atomic rename
    return os.Rename(tmpPath, w.path())
}

func (w *WAL) NeedsCompaction() bool {
    info, _ := w.file.Stat()
    return info.Size() > compactionThreshold  // 10 MB
}

We don't compact on every Complete call — that would tank throughput. The cmd/ipcbus binary runs a 60-second ticker that checks NeedsCompaction() and rewrites the file when it grows past 10 MB. That's a coarse heuristic — it will compact even if most entries are still pending, wasting some I/O — but it's simple and steady-state overhead is near zero. A smarter policy (also consider the pending ratio, pre-commit) would be a reasonable first PR.

The WAL does not fsync on every append. We batch. If a node hard-kills, we can lose the last few hundred milliseconds of messages. That's an acceptable tradeoff for a system where the upstream Slack delivery is already best-effort. If you care more about durability, Flush() is exposed and you can call it from your own code, but we chose not to make it automatic.

Mechanism 4 — Dead Letter Queue (DLQ)

After 5 delivery attempts, a message is "dead." We don't silently drop it; we move it to the DLQ:

func NewDLQ(path string, maxSize int, ttl time.Duration) (*DLQ, error) {
    db, err := bolt.Open(path, 0600, &bolt.Options{Timeout: 1 * time.Second})
    // ...
}

BoltDB is embedded KV storage with B+tree on-disk layout. It's fast, transactional, and single-file. Perfect for a sidecar that needs a few megabytes of dead messages, queryable by ID and age.

Two eviction policies:

maxSize — a hard cap on entry count. When we're full, we evict the oldest.
ttl — entries older than the TTL are purged. NewDLQ(path, maxSize, ttl) takes both as constructor args; the cmd/ipcbus binary passes maxSize=10000, ttl=24h and runs an hourly PurgeExpired ticker. Library callers can pick their own.

This matters because the DLQ is the debugging surface for the bus. Something went wrong? kubectl exec into the sidecar, open the BoltDB file, and look at the last N entries. We've caught a couple of real bugs this way that would have been invisible with "drop on failure."

func (d *DLQ) PurgeExpired() (int, error)
func (d *DLQ) Size() int
func (d *DLQ) List() ([]*DLQEntry, error)

Deliberately no replay-from-DLQ. If something's dead, it's dead. We want human attention, not automatic retry that hides a real problem.

Mechanism 5 — ring buffer with backpressure

The remaining problem: what if a channel sidecar is producing faster than the runtime can consume?

Naive answer: unbounded queue. Result: OOM-killed pod.

Real answer: bounded ring buffer with high/low watermarks.

func NewRingBuffer(size int, highWatermark, lowWatermark float64) *RingBuffer {
    // ... defaults to high=0.8, low=0.3 ...
}

When the buffer fills past 80%, the bus emits a slow_down control message upstream. The channel sidecar sees it and stops pulling from Slack. When the buffer drains below 30%, the bus emits resume and the sidecar starts pulling again.

Why two watermarks? Because if you use one, you thrash. Right at the threshold, every push flips state. Two watermarks with a gap gives you hysteresis. Classic control-theory stuff, very little Go stuff.

The slow_down / resume messages ride the same wire format as everything else:

switch m.Type {
case TypeAck, TypeNack, TypeSlowDown, TypeResume,
     TypeShutdown, TypeRegister, TypeHeartbeat:
    return true
}

Treating control traffic as just another MessageType means channel sidecars don't need a separate control channel. One TCP/UDS/WS connection carries both payloads and backpressure signals. Simpler, fewer failure modes.

Shutdown

Graceful shutdown is its own hazard. On SIGTERM the cmd/ipcbus binary runs a local shutdown() helper that does the bare minimum:

func shutdown(logger, router, wal, bridge, cancel) {
    router.SendShutdown()      // tell sidecars we're going away
    time.Sleep(5 * time.Second) // fixed grace window
    wal.Flush()                 // flush WAL to disk
    bridge.Close()              // close the runtime bridge
    cancel()                    // stop the UDS server + background tickets
}

That's it. No polling, no early exit if sidecars disconnect, no DLQ close (process-exit flushes BoltDB's mmap and that's enough). Whatever is still pending in the WAL when we exit gets replayed on next startup — that's the whole point of the WAL.

There's also a fancier ShutdownOrchestrator in internal/ipcbus/shutdown.go that takes a drainTimeout parameter and polls router.ConnectedCount() every 100 ms to exit early, but the current binary doesn't wire it up. Good first PR: swap the local helper out for the orchestrator so the sleep becomes a real wait-for-drain.

What we didn't do (on purpose)

Multi-pod clustering. The bus is deliberately in-pod. If you want cross-pod messaging, use a real broker (NATS, Redis streams). Scoping this to one pod kept us sane.
Ordering guarantees across channels. Within one channel, messages are ordered. Across channels, no promise. Most agent workloads don't care.
Exactly-once. At-least-once with idempotent consumers is simpler and good enough. The runtime is expected to deduplicate on Message.ID.
Protobuf on the wire. JSON is ~2× larger but 10× easier to debug. Given our throughput (tens of messages per second per pod, not millions), JSON is the right call.

Testing

We aimed for >80% statement coverage on the ipcbus package, approximately. The non-obvious piece: most of the reliability features are hard to unit-test with mocks because they're about failure modes. So we have a lot of tests that spin up real local listeners (net.Listen("tcp", "127.0.0.1:0"), net.Listen("unix", t.TempDir()+"/sock"), httptest.NewServer(...)) and exercise the bridges end-to-end.

For example, the SSE bridge test spins up an httptest server that handles both GET /events (as an SSE stream) and POST /messages, and checks that connecting, sending, and receiving all work:

func TestSSEBridge_SendReceive(t *testing.T) {
    srv, ready := sseEchoServer(t)
    defer srv.Close()

    bridge := NewSSEBridge(srv.URL)
    // ... connect, wait for SSE stream to establish, send, receive ...
}

About 70 tests total, -race clean. Good enough for a sidecar.

What this bought us

A uniform contract for channel sidecars. You write one Slack sidecar, it works with every runtime. You write one Discord sidecar, same thing. Runtime authors pick a protocol that fits their stack; they don't think about durability, retries, or backpressure — the bus handles it.

The runtime adapter for a new protocol is ~50 lines. The channel sidecar SDK (sdk/channel/) hides the framing entirely; you call client.Send(ctx, json.RawMessage(...)) and move on.

The whole ipcbus package is ~2k lines of Go. If you want to read one file to get the flavor, router.go is where all five mechanisms meet.

What to look at next

The k8s4claw repo if you want to use it
internal/ipcbus/ if you want to read the code
The intro post if you want context on how this fits into the operator

Open source, Apache-2.0. Questions and PRs welcome. If you've built something similar and went in a different direction, I'd love to hear why in the comments.

When Rust's Exhaustive Match Helps (And When It Doesn't): Notes from a Bare-Metal Hypervisor

willamhou — Wed, 22 Apr 2026 03:41:04 +0000

Disclaimer: This is about an experimental hypervisor project that only runs on QEMU virt — no real-hardware validation yet. The lessons apply to "Rust's tooling edges in systems programming," not production guidance.

10 weeks into writing an ARM64 bare-metal hypervisor, I assumed Rust's exhaustive match would be the safety net when I extended my state machine. Two observations, from one week of commits: exhaustive match didn't help my state machine at all, but caught 6 errors the one time I extended my Device enum. This post is about why — and why the distinction is about cardinality, not typestate vs tag enums.

I'm writing an ARM64 bare-metal hypervisor. Part of it is a thing called a Secure Partition (SP) — a lightweight VM managed by the SPMC. Each SP has a lifecycle: Reset → Idle → Running → Blocked → Preempted. 5 states, 7 legal transitions.

Two weeks ago I added a new transition: Blocked → Preempted, for chain preemption between SPs. By the textbook, this is exactly the scenario where Rust's enum + match should shine: add a state/transition, the compiler finds every site that needs updating.

The compiler said nothing.

This post is about why I didn't use the "enum-with-fields" pattern you see in tutorials, why match exhaustiveness didn't help on this state machine, and where it actually did help.

The Real Code

No toy examples. Here's the actual SpState from the repo:

// src/sp_context.rs
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
#[repr(u8)]
pub enum SpState {
    Reset = 0,
    Idle = 1,
    Running = 2,
    Blocked = 3,
    Preempted = 4,
}

Classic tag-only enum — #[repr(u8)], every variant is one byte, no payload. Why not the textbook Running { entry_pc: u64 } / Preempted { saved_ctx: VcpuContext }?

Because the state lives in an AtomicU8.

The SPMC runs on multiple physical CPUs. Different CPUs inside TF-A's SPMD (Secure Partition Manager Dispatcher) can route requests to the same SP at once. Two CPUs racing to do Idle → Running — one must lose, or both will ERET into the same SP and clobber register context.

CAS drives the race:

pub fn try_transition(&self, expected: SpState, new_state: SpState) -> Result<(), SpState> {
    match self.state.compare_exchange(
        expected as u8,      // success: AcqRel publishes our context-save
        new_state as u8,     // failure: Acquire syncs the observed loser
        Ordering::AcqRel,
        Ordering::Acquire,
    ) {
        Ok(_) => Ok(()),
        Err(actual) => Err(SpState::try_from(actual).expect("corrupt SP state value")),
    }
}

The constraint isn't memory layout — #[repr(u8, C)] on a fields-carrying enum does give stable layout. The real constraint is size: AtomicU8 wraps one byte, and any enum with a u64 payload is at least 8 bytes wide. Atomic u64 CAS is fine on aarch64, but that means every state change either serializes through a fat struct CAS or falls back to a lock. I wanted single-byte CAS in the fast path, so the payload lives elsewhere (in a separate VcpuContext guarded by the state transition itself).

Side note on expect("corrupt SP state value"): it really does panic. In this project the panic handler halts the offending CPU and dumps state via UART — because if the AtomicU8 ever holds a value outside 0..=4, memory corruption has already happened and limping along is worse than stopping. That's a conscious choice for this binary, not a general bare-metal guideline.

Why Exhaustive Match Didn't Help

The legal-transition check lives in one function:

// src/sp_context.rs
pub fn transition_to(&mut self, new_state: SpState) -> Result<(), &'static str> {
    let current = self.state();
    let valid = match (current, new_state) {
        (SpState::Reset, SpState::Idle) => true,
        (SpState::Idle, SpState::Running) => true,
        (SpState::Running, SpState::Idle) => true,
        (SpState::Running, SpState::Blocked) => true,
        (SpState::Blocked, SpState::Running) => true,
        (SpState::Blocked, SpState::Preempted) => true,  // ← the newly added line
        (SpState::Running, SpState::Preempted) => true,
        (SpState::Preempted, SpState::Running) => true,
        _ => false,
    };
    // ...
}

Note the final _ => false. This is not an exhaustive match — the wildcard swallows every unlisted combination as "illegal."

The commit that added Blocked → Preempted was literally 1 line. The compiler reported nothing, because to the compiler, all 25 (from, to) combinations are covered (7 explicit + _ fallback).

I could have replaced _ => false with all 18 illegal combinations enumerated. I started to — "exhaustive is more Rust-y". Then I gave up halfway:

// This way...
(SpState::Reset, SpState::Reset) => false,
(SpState::Reset, SpState::Running) => false,
(SpState::Reset, SpState::Blocked) => false,
// ... 15 more lines of this

No new information, and every future state addition means maintaining an N² table. _ => false is the documentation here: what's listed is legal; everything else isn't.

Verdict: For simple C-style enum + state-transition pairs, match exhaustiveness doesn't save you. Bugs at this layer can only be caught by unit tests (my test_sp_context.rs has 58 assertions covering every legal transition plus key illegal ones).

Where It Actually Saved Me

The place where match exhaustiveness actually saved me was device dispatch.

My hypervisor uses a Device enum to enumerate all virtual devices. Every time the guest touches MMIO, a match dispatches to the right implementation:

// src/devices/mod.rs
pub enum Device {
    Uart(pl011::VirtualUart),
    Gicd(gic::VirtualGicd),
    Gicr(gic::VirtualGicr),
    VirtioBlk(virtio::mmio::VirtioMmioTransport<virtio::blk::VirtioBlk>),
    VirtioNet(virtio::mmio::VirtioMmioTransport<virtio::net::VirtioNet>),
    Pl031(pl031::VirtualPl031),
}

This is a fields-carrying enum — each variant holds the state struct for its device. No _ fallback on matches against it, because every variant has its own handler:

impl MmioDevice for Device {
    fn read(&mut self, offset: u64, size: u8) -> Option<u64> {
        match self {
            Device::Uart(d) => d.read(offset, size),
            Device::Gicd(d) => d.read(offset, size),
            Device::Gicr(d) => d.read(offset, size),
            Device::VirtioBlk(d) => d.read(offset, size),
            Device::VirtioNet(d) => d.read(offset, size),
            Device::Pl031(d) => d.read(offset, size),
        }
    }
    // write, contains, is_ready, ...
}

When I added Pl031 (PL031 RTC) for Android boot, I only touched the enum definition. The compiler immediately fired 6 errors — every site that matches against Device was missing the Pl031 arm:

error[E0004]: non-exhaustive patterns: `&Device::Pl031(_)` not covered
  --> src/devices/mod.rs:51:15
error[E0004]: non-exhaustive patterns: `&mut Device::Pl031(_)` not covered
  --> src/devices/mod.rs:62:15
error[E0004]: non-exhaustive patterns: `&Device::Pl031(_)` not covered
  --> src/devices/mod.rs:73:15
// ... 6 total

Two of those were helper methods I'd written when adding VirtioNet and completely forgotten about. Had I used C switch without -Wswitch-enum (which Linux kernel and TF-A both enable by default), those two sites would silently fall into default and return "unknown device." The guest would do any MMIO to the RTC, fail to find a device, and hang mid-boot with an error pointing somewhere completely unrelated.

C with -Wswitch-enum + -Werror gives you the same check — the relevant difference is that Rust makes it a precondition for compiling instead of a build-system setting you can drop. Worth more in a solo project, less in a shop with a strict style guide.

Either way, the compiler caught this bug instead of the guest doing so at boot time.

When Exhaustive Match Actually Pays Off

Reviewing this state-machine extension + Device extension, here's my distilled rule:

Exhaustive match saves you: fields-carrying enum + every variant has independent handler logic.

Device::{Uart, Gicd, ..., Pl031} — each device's read/write is totally different
MmioAccess::{Read { reg, size }, Write { reg, size, val }} — read vs write semantics differ
ExitReason::{HvcCall, SmcCall, DataAbort, WfiWfe, ...} — each exception class has its own handler

Common trait: adding a variant potentially leaves gaps across the entire codebase, and each gap's correct implementation is non-trivial (not just "error vs OK" binary output).

Exhaustive match doesn't help: simple tag enum + cartesian-product check.

State machine (from, to) transition table — N² explosion, _ => false is more readable
Permission matrix (user_role, action) — same
Input sanity check match(input) { valid_range => ..., _ => reject } — tautological

These scenarios are "enumerate a small set of legal cases, reject everything else." _ => fallback loses no information — it's more readable.

A Few Takeaways

1. #[repr(u8)] is everyday life in hypervisor/kernel/driver code. Don't apologize for the atomic trade-off.

Every time a "Rust state machine" tweet appears, someone in the replies recommends typestate. Typestate is genuinely powerful when transitions happen through owning APIs (File::open → Handle<Open>), but it doesn't compose with shared mutable state across CPUs — the entire point of AtomicU8 is that multiple cores hold a reference to one byte. Typestate requires owning self by value to consume the old state; a multi-CPU SPMC can't do that on the fast path. Not a rejection of typestate, just the wrong tool for this edge.

2. _ => fallback isn't a sin, but ask yourself every time.

"If I add a new variant in the future, should this site force me to update it?"

Yes → drop the _, enumerate every variant
No (illegal state-machine pair, MMIO unknown-offset) → _ => default is documentation

3. State-machine correctness is never a gift from Rust. It's a gift from tests + documentation + code review.

My test_sp_context.rs has dedicated tests for every legal transition, a bunch of illegal ones, and CAS races. Rust didn't generate those; I wrote them. Rust saved me from some defensive code (no "sixth value" of SpState — try_from_u8 rejects it), but whether the legal-transition table is correct, Rust has no opinion.

4. What really saves you is "fields-carrying enum + each variant has its own handler."

That's Rust's signature strength. Find the places in your codebase that fit this pattern and get them right — it pays more than agonizing over whether the state machine should be typestate-ified.

Closing

My hypervisor isn't a "zero-unwrap" project. The repo has about 6 unwrap() calls (concentrated in test fixtures and boot-time paths that can't reasonably panic) and 45 _ => default fallback arms (mostly in MMIO register decode for unknown offsets).

Every unwrap() and _ => was a decision at the time, not laziness. Engineering beats slogans.

Rust gives you a good weapon. It doesn't think for you. Whether the state-transition table is legal is in your head, not the compiler's.

Code: github.com/willamhou/hypervisor

Blog: willamhou.github.io/hypervisor

This is part 5 of the ARM64 Hypervisor development series. The Chinese version is the canonical source — see part5-enum-state-machine.md.

k8s4claw: A Kubernetes Operator for Managing AI Agent Runtimes

willamhou — Tue, 21 Apr 2026 05:08:42 +0000

Every AI agent framework has its own deployment story. Claude-based assistants run one way, OpenAI agents another, security-focused runtimes yet another. If you run more than one on Kubernetes, you end up writing the same boilerplate over and over: secret management, persistent storage, graceful updates, inter-service messaging, observability.

k8s4claw is an open-source Kubernetes operator that wraps all of this behind a single CRD. You describe what the agent is, it handles how it runs.

apiVersion: claw.prismer.ai/v1alpha1
kind: Claw
metadata:
  name: research-agent
spec:
  runtime: openclaw
  config:
    model: "claude-sonnet-4"
  credentials:
    secretRef:
      name: llm-api-keys

The operator reconciles this into a StatefulSet, headless Service, ConfigMap, ServiceAccount, PodDisruptionBudget, and optionally NetworkPolicy and Ingress. When you add a channel (Slack, Discord, Webhook), it also wires up sidecars and a local message bus.

This post walks through the architecture, shows how to get it running locally, and explains the design decisions behind the IPC bus, the auto-update controller, and the runtime adapter system.

The Problem

We had several agent runtimes in flight at once — different languages, different process models, different resource profiles:

Runtime	Language	Use Case
OpenClaw	TypeScript/Node.js	Full-featured AI assistant
NanoClaw	TypeScript/Node.js	Lightweight personal assistant
ZeroClaw	Rust	High-performance agent
PicoClaw	Go	Ultra-minimal serverless
IronClaw	Rust + WASM	Security-focused agent
HermesClaw	Python	Conversational with tool use
K8sOps	Go	Cluster self-healing (claw4k8s)

Each had its own Helm chart, sidecar layout, and update strategy. Adding a Slack channel meant editing several files. Rotating credentials meant touching every deployment. Rolling back a bad update was a manual process.

We wanted one control plane for all of them.

Architecture

graph TB
    subgraph "Kubernetes Cluster"
        OP[k8s4claw Operator]

        subgraph "Claw Pod (with channels)"
            INIT["claw-init"]
            RT["Runtime Container"]
            IPC["IPC Bus Sidecar"]
            CH["Channel Sidecar"]
        end

        STS[StatefulSet]
        SVC[Service]
        CM[ConfigMap]
        PVC[(PVCs)]

        OP -->|manages| STS
        OP -->|manages| SVC
        OP -->|manages| CM
        STS -.->|runs| RT
        STS -.->|runs| IPC
        STS -.->|runs| CH

        CH <-->|UDS| IPC
        IPC <-->|Bridge| RT
    end

    EXT["Slack / Discord / Webhook"]
    CH <-->|API| EXT

The operator watches Claw custom resources and reconciles a full stack of Kubernetes objects. A minimal agent (no channels, no persistence) gets just the runtime container plus claw-init. If you declare any channels in spec.channels, the operator also injects:

claw-init — an init container that merges default runtime config with any user overrides before the runtime starts.
Runtime container — the actual AI agent binary.
IPC Bus sidecar (only when channels are present) — a WAL-backed message router that sits between the runtime and the channel sidecars.
Channel sidecar(s) — one per referenced ClawChannel (Slack, Discord, Webhook today).

There is a second CRD, ClawChannel, that describes how to connect to an external system. Channels are defined once and referenced by many Claws.

Quick Start

Prerequisites

Kubernetes 1.28+ (or kind for local development)
Go 1.25+
controller-gen (go install sigs.k8s.io/controller-tools/cmd/controller-gen@latest) — needed by make install

Install and run

git clone https://github.com/Prismer-AI/k8s4claw.git
cd k8s4claw

# Install CRDs into the active cluster
make install

# Run the operator locally against your current kubeconfig.
# --disable-webhooks lets you skip cert-manager setup during local dev.
# In-cluster deployments should leave webhooks enabled.
go run ./cmd/operator/ --disable-webhooks

Create your first agent

kubectl create secret generic llm-api-keys \
  --from-literal=ANTHROPIC_API_KEY=sk-ant-xxx

cat <<EOF | kubectl apply -f -
apiVersion: claw.prismer.ai/v1alpha1
kind: Claw
metadata:
  name: my-agent
spec:
  runtime: openclaw
  config:
    model: "claude-sonnet-4"
  credentials:
    secretRef:
      name: llm-api-keys
  persistence:
    session:
      enabled: true
      size: 2Gi
      mountPath: /data/session
    workspace:
      enabled: true
      size: 10Gi
      mountPath: /workspace
EOF

kubectl get claw my-agent -w

Connect Slack

apiVersion: claw.prismer.ai/v1alpha1
kind: ClawChannel
metadata:
  name: team-slack
spec:
  type: slack
  mode: bidirectional
  credentials:
    secretRef:
      name: slack-bot-token
  config:
    appId: "A0123456789"

Reference it from your Claw:

spec:
  channels:
    - name: team-slack
      mode: bidirectional

On the next reconcile the operator injects a Slack sidecar, spins up the IPC bus sidecar, and wires them together. The runtime container does not need to know anything about Slack — it just talks to the bus.

Deep Dive: The IPC Bus

The IPC bus is the most interesting piece of k8s4claw. It is a Kubernetes native sidecar (an init container with restartPolicy: Always) that routes JSON messages between channel sidecars and the agent runtime.

Channel Sidecar ──UDS──► IPC Bus ──Bridge──► Runtime Container
                         │ WAL  │
                         │ DLQ  │
                         │ Ring │
                         └──────┘

Why not just HTTP?

We tried. The problem is reliability. When a Slack event arrives while the runtime is overloaded, you need somewhere to buffer it. If the runtime crashes mid-response, you need to redeliver. When a channel sidecar falls behind, you need backpressure instead of dropped messages.

Three mechanisms do the work:

1. Write-Ahead Log (WAL) — Every inbound message is appended to a WAL on emptyDir before delivery. On restart, unacknowledged messages are replayed. Periodic compaction keeps the file bounded.

2. Dead Letter Queue (DLQ) — Messages that exceed the retry limit land in a BoltDB-backed DLQ instead of being dropped silently. You can inspect them later.

3. Ring buffer with backpressure — A fixed-size circular buffer with configurable high/low watermarks. Crossing the high watermark sends slow_down upstream; draining to the low watermark sends resume.

Bridge protocols

Different runtimes speak different wire protocols. The bus abstracts this behind a RuntimeBridge interface:

Runtime	Bridge	Protocol
OpenClaw	WebSocket	Full-duplex JSON over WS
NanoClaw	UDS	Length-prefix framed
ZeroClaw	SSE	HTTP POST + Server-Sent Events
PicoClaw	TCP	Length-prefix framed

Here is the actual interface (internal/ipcbus/bridge.go):

type RuntimeBridge interface {
    Connect(ctx context.Context) error
    Send(ctx context.Context, msg *Message) error
    Receive(ctx context.Context) (<-chan *Message, error)
    Close() error
}

Adding a new transport means implementing these four methods.

Deep Dive: Auto-Update Controller

The auto-update controller polls OCI registries on a cron schedule, filters new tags by a semver constraint, and performs health-verified rollouts with automatic rollback.

spec:
  autoUpdate:
    enabled: true
    versionConstraint: "^1.x"
    schedule: "0 3 * * *"
    healthTimeout: "10m"
    maxRollbacks: 3

How it works

Poll — on each cron tick, list tags from the registry and filter by the semver constraint.
Initiate — annotate the Claw with the target image and transition into the HealthCheck phase.
Health check — watch the StatefulSet readiness until all replicas are ready or the timeout fires.
Success — update status, clear the annotation, schedule the next cron tick.
Timeout — roll back to the previous image.
Circuit breaker — after N consecutive rollbacks, stop trying and emit an event plus a Prometheus metric.

The state machine lives in annotations and status conditions, so it survives operator restarts:

phase := claw.Annotations["claw.prismer.ai/update-phase"]
if phase == "HealthCheck" {
    return r.reconcileHealthCheck(ctx, &claw)
}

Version history

Every attempt is recorded:

status:
  autoUpdate:
    currentVersion: "1.2.0"
    versionHistory:
      - version: "1.2.0"
        appliedAt: "2026-03-28T03:00:00Z"
        status: Healthy
      - version: "1.1.5"
        appliedAt: "2026-03-21T03:00:00Z"
        status: RolledBack
    failedVersions: ["1.1.5"]
    circuitOpen: false

The Runtime Adapter Pattern

Each runtime is a Go struct implementing RuntimeAdapter:

type RuntimeAdapter interface {
    // Pod shape
    PodTemplate(claw *v1alpha1.Claw) *corev1.PodTemplateSpec
    HealthProbe(claw *v1alpha1.Claw) *corev1.Probe
    ReadinessProbe(claw *v1alpha1.Claw) *corev1.Probe
    DefaultConfig() *RuntimeConfig
    GracefulShutdownSeconds() int32

    // Spec validation
    Validate(ctx context.Context, spec *v1alpha1.ClawSpec) field.ErrorList
    ValidateUpdate(ctx context.Context, oldSpec, newSpec *v1alpha1.ClawSpec) field.ErrorList
}

A new adapter typically lives in a single file of ~100 lines. The shared BuildPodTemplate helper handles init containers, volume mounts, security context, and environment variables, so the adapter only declares what is actually different:

type MyRuntimeAdapter struct{}

func (a *MyRuntimeAdapter) PodTemplate(claw *v1alpha1.Claw) *corev1.PodTemplateSpec {
    return BuildPodTemplate(claw, &RuntimeSpec{
        Image:     "my-registry/my-runtime:latest",
        Ports:     []corev1.ContainerPort{{Name: "gateway", ContainerPort: 8080}},
        Resources: resources("100m", "256Mi", "500m", "512Mi"),
        // ...
    })
}
// plus HealthProbe, ReadinessProbe, DefaultConfig, GracefulShutdownSeconds,
// Validate, ValidateUpdate

Validation is per-runtime on purpose. OpenClaw and IronClaw require credentials because they call LLM APIs. ZeroClaw and PicoClaw permit credential-less operation. HermesClaw rejects spec.channels because it brings its own gateway. NanoClaw currently has no update-time persistence checks. The point is each adapter owns its own rules.

Go SDK

For programmatic access there is a Go SDK (sdk/):

import (
    "context"

    "github.com/Prismer-AI/k8s4claw/sdk"
)

client, err := sdk.NewClient() // uses the ambient kubeconfig by default
if err != nil {
    return err
}

claw, err := client.Create(ctx, &sdk.ClawSpec{
    Runtime: sdk.OpenClaw,
    Config: &sdk.RuntimeConfig{
        Environment: map[string]string{"MODEL": "claude-sonnet-4"},
    },
})
if err != nil {
    return err
}

// Block until the Claw reaches phase "Running" or ctx expires.
if err := client.WaitForReady(ctx, claw); err != nil {
    return err
}

There is also a channel SDK for writing custom sidecars:

import (
    "context"
    "encoding/json"

    "github.com/Prismer-AI/k8s4claw/sdk/channel"
)

client, err := channel.Connect(ctx,
    channel.WithChannelName("my-channel"), // or set CHANNEL_NAME env
    channel.WithSocketPath("/var/run/claw/bus.sock"),
    channel.WithBufferSize(100),
)
if err != nil {
    return err
}
defer client.Close()

// Send a message to the runtime.
if err := client.Send(ctx, json.RawMessage(`{"text":"Hello"}`)); err != nil {
    return err
}

// Receive returns a channel of *InboundMessage.
inbox, err := client.Receive(ctx)
if err != nil {
    return err
}
for msg := range inbox {
    // handle msg
    _ = msg
}

Testing Strategy

The repo has reasonable test coverage on the core packages. A recent local run looked roughly like this:

Package	Coverage (approx.)
`internal/webhook`	~97%
`internal/runtime`	~94%
`internal/registry`	~86%
`sdk`	~83%
`internal/controller`	~81%
`sdk/channel`	~81%
`internal/ipcbus`	~80%

Numbers move PR by PR. CI publishes a coverage report as an artifact and gates on a total-coverage threshold; there is no per-package floor enforced today. Treat the table as a snapshot, not a contract.

The testing pyramid:

Unit tests — pure functions, table-driven, t.Parallel() everywhere.
Fake-client tests — fake.NewClientBuilder() for controller logic without a real cluster.
envtest integration tests — real etcd + API server for webhook validation and reconcile loops.

The auto-update controller uses dependency injection via Clock and TagLister interfaces so time-dependent and registry-dependent code is fully testable with no network calls.

What's Not Done Yet

Worth being honest about:

custom runtime type is present in the CRD enum but no adapter is registered. If you want a runtime that is not in the built-in list today, you fork and add an adapter.
HermesClaw does not yet integrate with the k8s4claw channel sidecars — it uses its own gateway.
Local operator runs need --disable-webhooks unless you've set up cert-manager or your own TLS. In-cluster deployments via the Helm chart handle this for you.
CRD surface is larger than just Claw — ClawChannel, ClawSelfConfig, and related types are part of the contract. "Single CRD" is a simplification; "small, focused set of CRDs" is closer to the truth.

What's Next

k8s4claw is open source under Apache-2.0. The current open contribution target is Issue #4: add snapshot and PDB envtest coverage. If you want to propose something else, open a new issue and we'll triage it.

GitHub: github.com/Prismer-AI/k8s4claw

If you run AI agents on Kubernetes and you're tired of maintaining the plumbing around them, give it a try. Star the repo if it helps, and open an issue if something is off — both signals are useful.

How to Add Tamper-Evident Audit Trails to Your LangChain Agent

willamhou — Mon, 20 Apr 2026 02:16:33 +0000

Your LangChain agent calls tools. It searches the web, reads files, queries databases, calls APIs. But can you prove what it did?

Logs capture what happened. Cryptographic receipts prove it. The difference matters when an auditor, a customer, or a regulator asks "show me exactly what the agent did and prove it wasn't altered after the fact."

This tutorial adds Ed25519-signed, hash-chained audit trails to a LangChain agent in under 5 minutes. No external service, no API keys, no infrastructure. Everything verifies offline with a public key.

What you'll build

A LangChain agent where every tool call produces a signed receipt containing:

What: which tool was called, with what parameters
Who: the agent's Ed25519 public key
When: timestamp
Proof: Ed25519 signature over JCS-canonicalized (RFC 8785) payload
Chain: SHA-256 hash linking to the previous receipt (tamper-evident ordering)

If anyone modifies a receipt after the fact, the signature breaks. If anyone deletes or reorders receipts, the hash chain breaks.

Install

pip install signet-auth[langchain] langchain-openai langchain-community

Step 1: Create a signing identity

from signet_auth import SigningAgent

# Creates an Ed25519 keypair, stored locally in ~/.signet/
# If the key already exists, just load it:
try:
    agent = SigningAgent("my-langchain-agent")
except Exception:
    agent = SigningAgent.create("my-langchain-agent", owner="acme-corp")

print(f"Public key: {agent.public_key}")

That's it. No key server, no certificate authority. The private key stays on disk, the public key is what verifiers use.

Step 2: Add the signing callback

Signet ships a LangChain callback handler that signs every tool call automatically. Two lines:

from signet_auth.langchain import SignetCallbackHandler

signer = SignetCallbackHandler(agent)

This handler signs the full tool lifecycle: on_tool_start (what was called), on_tool_end (what it returned, hashed), and on_tool_error (what went wrong). If signing fails, the handler logs a warning and lets the agent continue. It never crashes your chain.

Step 3: Wire it into your agent

from langchain import hub
from langchain_community.tools import DuckDuckGoSearchRun
from langchain.agents import AgentExecutor, create_react_agent
from langchain_openai import ChatOpenAI

# Standard LangChain setup
llm = ChatOpenAI(model="gpt-4o-mini")
tools = [DuckDuckGoSearchRun()]
prompt = hub.pull("hwchase17/react")

# Create and run agent with signing callback
agent_executor = AgentExecutor(
    agent=create_react_agent(llm, tools, prompt),
    tools=tools,
    callbacks=[signer],
)

result = agent_executor.invoke({"input": "What is the weather in Tokyo?"})

Every tool call now produces a signed receipt. No code changes to the tools themselves.

Step 4: Inspect the receipts

import json

for i, receipt in enumerate(signer.receipts):
    data = json.loads(receipt.to_json())
    print(f"\nReceipt #{i+1}")
    print(f"  Tool:       {data['action']['tool']}")
    print(f"  Params hash: {data['action']['params_hash']}")
    print(f"  Signature:   {data['sig'][:40]}...")
    print(f"  Timestamp:   {data['ts']}")

Output:

Receipt #1
  Tool:       duckduckgo_search
  Params hash: sha256:a1b2c3...
  Signature:   ed25519:Mz4xNTk2NjQ0NDgw...
  Timestamp:   2026-04-19T10:30:00Z

Receipt #2
  Tool:       _tool_end
  Params hash: sha256:d4e5f6...
  Signature:   ed25519:Nk5yODk3MjE1Njg4...
  Timestamp:   2026-04-19T10:30:01Z

Notice the start/end pair: the first receipt captures the tool call, the second captures a hash of the output. Together they prove what was called and what it returned.

Step 5: Verify a receipt

Anyone with the public key can verify, offline:

from signet_auth import verify

receipt = signer.receipts[0]

is_valid = verify(receipt, agent.public_key)
print(f"Valid: {is_valid}")  # True

Tamper with any field and verification fails:

data = json.loads(receipt.to_json())
data["action"]["tool"] = "evil_tool"  # tamper

from signet_auth import Receipt
tampered = Receipt.from_json(json.dumps(data))
print(verify(tampered, agent.public_key))  # False

Step 6: Verify the audit chain

The audit log is a hash-chained JSONL file. Each entry's hash covers the previous entry, so deleting or reordering receipts breaks the chain:

from signet_auth import audit_verify_chain, default_signet_dir

signet_dir = default_signet_dir()
chain_status = audit_verify_chain(signet_dir)
print(f"Chain intact: {chain_status.valid}")
print(f"Entries: {chain_status.length}")

What this gives you

Without Signet	With Signet
"The agent called web_search" (log entry)	Ed25519 signature proving it, verifiable by anyone with the public key
Logs can be edited after the fact	Signature breaks if any field is modified
No ordering proof	Hash chain breaks if receipts are deleted or reordered
Trust the operator's logs	Verify independently, offline

When you need this

Regulated industries: EU AI Act Article 12 requires "automatic recording" of AI system activities. Signed receipts satisfy this with cryptographic proof, not just logs.
Enterprise deployments: When the question is "can you prove what the agent did?", signed receipts are the answer.
Agent-to-agent: When Agent B needs to verify what Agent A actually did before acting on its output.
Incident response: After something goes wrong, tamper-evident receipts let you reconstruct exactly what happened without trusting anyone's claim.

Next steps

Bilateral co-signing: Have both the agent and the tool server sign each interaction independently. Neither party can fabricate receipts. See signet proxy for MCP integration.
Policy attestation: Evaluate YAML policy rules and include the decision (allow/deny/require_approval) inside the signed receipt.
Delegation chains: Prove that Agent A was authorized by Human B to perform a specific action with scoped constraints.

All of these are in signet-auth today.

pip install signet-auth

GitHub: Prismer-AI/signet

Signet is open source (Apache 2.0). Rust core with Python and TypeScript bindings. No external service, no API keys.

Claude Managed Agents Has Built-in Tracing. Here's What It Can't Do.

willamhou — Tue, 14 Apr 2026 09:52:23 +0000

Claude Managed Agents Has Built-in Tracing. Here's What It Can't Do.

Anthropic shipped Claude Managed Agents last week. The pitch: production-grade agents with sandboxing, scoped permissions, and session tracing — built in, no setup required.

The tracing feature specifically: "Session tracing, integration analytics, and troubleshooting guidance are built directly into the Claude Console, so you can inspect every tool call, decision, and failure mode."

This is genuinely useful. If you're debugging a multi-step agent workflow, having every tool call logged in a console is miles better than parsing stderr.

But there's a distinction worth making — one that matters in exactly the situations where it matters most.

"Anthropic Recorded It" vs. "You Can Prove It"

Claude Managed Agents is cloud-hosted. The tracing data lives in Claude Console, on Anthropic's infrastructure.

That means the audit trail is: Anthropic says this happened.

For most debugging use cases, that's fine. You trust Anthropic. They trust you. The logs are accurate. Nobody is lying.

But consider the situations where audit trails actually get pulled:

Your agent made an unauthorized transfer. The question isn't "what does the console say" — it's "can you prove, to a third party, that the agent executed this action with these parameters at this time, and that this record hasn't been modified?"

A compliance audit. SOC 2, HIPAA, GDPR. The auditor asks for evidence of agent actions on sensitive data. "Here are logs from Anthropic's console" is not the same as "here is a cryptographically signed chain of records that I hold and you can independently verify."

An incident investigation. After a breach, forensic investigators need evidence that is tamper-evident and independently verifiable. If the evidence lives on the infrastructure that may have been compromised — or that a vendor controls — its integrity cannot be assumed.

The distinction isn't about trust in Anthropic. It's about the difference between a record and evidence.

What Cryptographic Signing Adds

A signed audit trail works differently.

Each tool call generates a receipt: the action, the parameters, the timestamp, the agent identity — all hashed and signed with the agent's private Ed25519 key. Receipts chain together: each receipt includes the hash of the previous one. Modifying any record breaks the chain. Deleting a record is detectable.

The key difference: you hold the proof, not a vendor.

from signet_auth import SigningAgent

agent = SigningAgent.create("procurement-bot", owner="ops-team")
receipt = agent.sign("marketplace_purchase",
    params={"item": "GPU-A100", "quantity": 2, "price": 15000})

# This receipt is a cryptographic artifact.
# You hold it. Anthropic doesn't.
# Any third party can verify it without contacting anyone.
assert agent.verify(receipt)

When an auditor asks "prove this agent executed this action with these parameters," you hand them the receipt and the public key. They verify it offline. No Anthropic console access required. No vendor dependency in the evidence chain.

The Three Gaps

1. Vendor-held vs. self-held evidence

Managed Agents tracing: logs live in Claude Console. Anthropic controls the data.

Signed receipts: cryptographic artifacts you hold locally. No third party in the verification chain.

2. Log integrity vs. cryptographic integrity

Managed Agents: session logs. Accurate under normal conditions. But a log file — even a well-managed one — can be modified. There's no mechanism in a standard log that makes tampering detectable after the fact.

Signed receipts: hash-chained. Tamper with any entry and the chain breaks. Detect deletions. Detect reordering. The integrity guarantee is mathematical, not administrative.

3. Single-party vs. bilateral proof

Managed Agents: Anthropic logs what happened on their infrastructure.

Bilateral signing (Signet v0.4+): the agent signs the request, the server independently signs the response. One tamper-evident record, two signatures, two trust domains. Rewriting the chain requires compromising both keys on separate machines.

What Managed Agents Does Well

To be clear about what this is not: this is not a criticism of Managed Agents as a product.

For developers building Claude-based agents who need to go to production quickly, Managed Agents is a compelling offer. Sandboxing, authentication, session persistence, scoped permissions, multi-agent coordination — real infrastructure problems, solved. The tracing in Console is useful for development and operational debugging.

The gaps above only matter in specific contexts:

Regulated industries (finance, healthcare, legal) where audit evidence must be third-party verifiable
Incident response and forensics where evidence integrity must be demonstrable
Enterprise compliance where "trust the vendor" isn't an accepted audit answer
Cross-vendor or multi-agent workflows where a single vendor doesn't control the full chain

For consumer applications, hobby projects, or internal tools where you trust Anthropic and compliance requirements are light: Managed Agents tracing is probably sufficient.

The Complementary Stack

Managed Agents and signed audit trails aren't competitors. They operate at different layers.

Managed Agents handles: infrastructure, sandboxing, session management, permission scoping, operational tracing.

Signed receipts handle: cryptographic proof of what happened, independently verifiable by any third party, held by you, not a vendor.

Signet works with Managed Agents. Claude Managed Agents uses MCP to connect to external tools — Signet's @signet-auth/mcp intercepts at the MCP transport layer and signs every tool call before it executes. The two layers stack.

Claude Managed Agents
  └── MCP tool calls
        └── Signet SigningTransport  ← signs here
              └── your tool server

The Console shows you what happened. The signed receipts prove it.

The Bottom Line

Claude Managed Agents ships a real, useful tracing feature. If you're using it, your debugging workflow just got better.

But "Anthropic recorded it" and "you can prove it" are different claims. In the situations where audit trails matter most — compliance, incident response, regulated industries — the difference is significant.

Signing is the layer that converts logs into evidence.

Signet adds Ed25519 signing and tamper-evident audit chains to AI agent tool calls. Works with Claude Managed Agents, LangChain, CrewAI, AutoGen, and 7 other frameworks. Apache-2.0 + MIT.

Now on the official Claude Code plugin marketplace: /plugin install signet@claude-plugins-official

AI Agents Can Move Money But Can't Produce Receipts

willamhou — Tue, 14 Apr 2026 03:12:41 +0000

AI Agents Can Move Money But Can't Produce Receipts

In March 2026, security researchers disclosed ZombieClaw — a botnet recruiting compromised AI agent instances. Over 30,000 instances were found exposed with default configurations. Reported losses reached up to $16 million in cryptocurrency. Hundreds of malicious skills were distributed through ClawHub (341 initially identified by Koi, with more found by VirusTotal).

Kaspersky found 512 vulnerabilities, eight critical. Bitdefender, VirusTotal, Sophos, and Oasis Security all published analyses.

But here's what nobody is talking about: after the attack, there is no cryptographic proof of what any compromised agent actually did.

No signed records. No tamper-evident logs. No way to distinguish "the agent executed transfer_eth() because the user asked" from "the agent executed transfer_eth() because a prompt injection rewrote its instructions."

The text logs exist, sure. But text logs can be edited, deleted, or fabricated. When $16M is missing, "trust the logs" is not a forensic standard.

The Forensics Problem

When a traditional server gets compromised, incident response teams have tools: immutable audit logs, signed system events, chain-of-custody protocols. When an AI agent gets compromised, you have:

Conversation history — stored by the agent itself. The compromised agent can edit its own history.
Tool call logs — if they exist at all, they're unsigned text files. An attacker who controls the agent controls the logs.
"The agent did it" — not enough for insurance claims, compliance reports, or criminal prosecution.

ZombieClaw exploited this gap perfectly. The attackers didn't just steal money — they operated in an environment where there is no verifiable evidence of what happened.

Why This Matters Beyond ZombieClaw

The AI agent security conversation focuses on prevention: sandboxing, permission systems, policy engines, skill auditing. These are important. But prevention has a 100% failure rate over time. Every system eventually gets breached.

What happens after?

Without cryptographic proof of agent actions, you can't answer:

Which agent initiated the transaction?
Were the parameters what the user actually approved?
When exactly did the compromise begin?
Was this agent's audit log tampered with after the fact?

SOC 2, HIPAA, and GDPR all require audit trails for actions on sensitive data. "The AI agent did it and we have no verifiable records" creates real gaps in compliance posture.

What a Signed Audit Trail Would Have Changed

If every tool call had been cryptographically signed at execution time, the ZombieClaw investigation would look different:

Before compromise: Signed receipts establish a baseline. Each agent has an Ed25519 identity. Every tool call is signed with the agent's key, timestamped, and chained into a tamper-evident log. The hash chain means you can't delete or reorder entries without breaking the chain.

During compromise: The attacker takes control of the agent. If the attacker uses the agent's existing key, every malicious action is still signed — you have a record of what was executed and when. If the attacker generates a new key, the signing identity changes — the anomaly is visible in the chain.

After compromise: Forensics teams can verify the entire chain offline. They can see which actions were signed by the legitimate agent key vs. an unknown key. They can narrow down when the signing identity changed. They can verify that the log hasn't been modified after the fact.

None of this is possible with unsigned text logs.

What This Doesn't Solve

Signing is not prevention. A signed receipt that says "agent transferred 50 ETH to attacker's wallet" doesn't stop the transfer — it proves it happened.

A signed audit trail doesn't solve:

Malicious skills — A signed record of a malicious skill executing is evidence, not a defense.
Prompt injection — The agent was tricked, not unauthorized. The signature is valid because the agent really did execute the call.
Key compromise — If the attacker steals the signing key, they can sign anything. Bilateral co-signing (where the server independently signs the receipt) mitigates this by requiring two keys from two trust domains.
User intent — A signed receipt proves the agent executed the call, not that the user wanted it.
Full host compromise — If the attacker owns the entire machine, they control the key and the log. Off-host anchoring (publishing chain hashes externally) is the mitigation, but it's not free.

Signing is the forensics layer. You still need sandboxing, permission systems, and skill auditing for prevention. But when prevention fails — and it will — you need evidence.

The Gap in Current Tools

As of April 2026, most major AI agent frameworks have no cryptographic signing on tool call records:

Category	Examples	Typical audit mechanism	Signed?
General-purpose agents	OpenClaw, Hermes Agent	Conversation logs, SQLite	No
Agent OS	OpenFang	SHA-256 hash chain	Hash only, no signatures
Orchestration frameworks	LangChain, CrewAI	Callbacks, event logs	No

OpenFang is the closest — they have a hash chain, which detects casual tampering. But without signatures, an attacker with database access can rewrite the entire chain and it still validates.

What Can You Do Today

If you're running AI agents in production:

Sign every tool call. Give each agent an Ed25519 identity and sign every action. Signet does this as a library — pip install signet-auth or npm install @signet-auth/core.
Chain signed receipts. Individual signatures are good. A hash-chained log of signed receipts is better — deletion and reordering become detectable.
Use bilateral signing when possible. Agent signs the request, server signs the response. Now rewriting the chain requires compromising both keys on different machines.
Export chain hashes off-host. Periodically publish the tip hash to an external system (git commit, append-only cloud storage, even a tweet). This anchors the chain against full-host compromise.
Treat audit integrity as a security requirement, not a feature. If your agent can move money, it needs signed receipts. Period.

The Uncomfortable Truth

AI agents can move money, execute code, and access credentials. Most still can't produce a receipt.

The next ZombieClaw is coming. The question is whether you'll have evidence when it happens.

Signet adds Ed25519 signing and tamper-evident audit logs to AI agent tool calls. Open source, Apache-2.0 + MIT.

Your MCP Server Has No Audit Trail — A Security Checklist

willamhou — Mon, 13 Apr 2026 06:24:46 +0000

Your MCP Server Has No Audit Trail — A Security Checklist

Last month, an AI agent mass-deleted a production environment. The team spent 3 days piecing together what happened — stderr logs, partial timestamps, no proof of which agent or what parameters. No audit trail.

This isn't rare. Amazon Kiro deleted a prod environment. Replit's agent dropped a live database. Supabase MCP leaked tokens via prompt injection. In every case: zero cryptographic evidence of what happened.

MCP is becoming the standard for agent-tool communication. Claude Code, Cursor, Windsurf, and dozens of tools use it. But the MCP spec ships with:

❌ No request signing
❌ No audit log
❌ No caller identity verification
❌ No replay protection
❌ No parameter integrity checks

Your MCP server accepts any request from any process, trusts it completely, and keeps no verifiable record. Here's a practical checklist to fix that.

The Threat Model

Before the checklist, understand what you're defending against:

Attack	How it works	Impact
Parameter tampering	Agent sends `create_issue("fix bug")`, something in the pipeline changes it to `delete_repo("production")`	Data loss
Replay	Legitimate `deploy_to_prod` captured and replayed 50 times	Repeated side effects
Impersonation	Rogue process sends requests claiming to be your trusted agent	Unauthorized actions
Cross-server forwarding	Request intended for staging gets forwarded to production	Wrong environment
Log tampering	Text logs edited after an incident to cover tracks	No incident response
Compliance gap	SOC 2 / HIPAA / GDPR require audit trails; "the AI did it" is not sufficient	Regulatory risk

Checklist

✅ 1. Use TLS for HTTP transports

If your MCP server uses HTTP (SSE or Streamable HTTP), always terminate TLS. This protects data in transit but does not protect against:

Compromised clients sending bad requests
Replay attacks (TLS protects the pipe, not the message)
Log tampering after the fact

For stdio transports (most local MCP servers), TLS doesn't apply — the attack surface is different (any local process can connect).

# nginx example
location /mcp {
    proxy_pass http://localhost:3001;
    proxy_set_header X-Forwarded-For $remote_addr;
}

Covers: Data in transit.

Doesn't cover: Request integrity, identity, audit.

✅ 2. Validate inputs at the boundary

Every tool handler should validate its arguments. MCP passes arbitrary JSON — treat it like user input.

server.setRequestHandler(CallToolRequestSchema, async (request) => {
  const { name, arguments: args } = request.params;

  if (name === "create_issue") {
    if (typeof args?.title !== "string" || args.title.length > 200) {
      return { content: [{ type: "text", text: "Invalid title" }], isError: true };
    }
    // proceed...
  }
});

Use Zod or similar for runtime validation. Never trust args blindly.

Covers: Malformed input, injection.

Doesn't cover: Who sent it, whether it's a replay, audit trail.

✅ 3. Add authentication (API keys or mTLS)

For HTTP transports, require an API key or use mutual TLS:

// Simple API key check
server.setRequestHandler(CallToolRequestSchema, async (request, extra) => {
  const apiKey = extra.requestHeaders?.["x-api-key"];
  if (apiKey !== process.env.MCP_API_KEY) {
    return { content: [{ type: "text", text: "Unauthorized" }], isError: true };
  }
  // proceed...
});

For stdio, authentication is harder — any local process with access to the pipe can send requests. This is where cryptographic signing becomes necessary.

Covers: Unauthorized callers (HTTP only).

Doesn't cover: Parameter integrity, replay, stdio auth, audit trail.

✅ 4. Sign every request with cryptographic receipts

This is the gap most MCP servers don't address. Signing binds a request to a specific agent identity and makes tampering detectable.

Signet adds Ed25519 signing to MCP. A signed receipt:

{
  "v": 1,
  "action": {
    "tool": "create_issue",
    "params_hash": "sha256:b878192...",
    "target": "mcp://github.local"
  },
  "signer": {
    "pubkey": "ed25519:0CRkURt/tc6r...",
    "name": "deploy-bot"
  },
  "ts": "2026-04-09T10:30:00.000Z",
  "nonce": "rnd_dcd4e13579...",
  "sig": "ed25519:6KUohbnS..."
}

Tamper with any field → signature fails. Replay → nonce rejected.

Client side — sign every tool call:

import { SigningTransport } from "@signet-auth/mcp";

const inner = new StdioClientTransport({ command: "my-mcp-server" });
const transport = new SigningTransport(inner, secretKey, "my-agent");
// Every tools/call now carries a signed receipt in params._meta._signet

The receipt is injected into _meta._signet. MCP servers ignore unknown fields by spec — zero server changes needed to start signing. Works with stdio and HTTP.

Server side — verify incoming signatures:

import { verifyRequest, NonceCache } from "@signet-auth/mcp-server";

const nonceCache = new NonceCache();

server.setRequestHandler(CallToolRequestSchema, async (request) => {
  const result = verifyRequest(request, {
    trustedKeys: ["ed25519:..."],       // allowed agent keys
    expectedTarget: "mcp://my-server",  // anti-forwarding
    maxAge: 300,                        // 5-min freshness window
    nonceCache,                         // replay protection
  });

  if (!result.ok) {
    return { content: [{ type: "text", text: result.error }], isError: true };
  }

  // Verified: signature valid, signer trusted, fresh, correct target
  // proceed with tool execution...
});

In ~50 microseconds, this checks: signature validity, signer trust, freshness, target binding, tool/params integrity, and nonce uniqueness.

Python (works with LangChain, CrewAI, AutoGen, or standalone):

from signet_auth import SigningAgent

agent = SigningAgent.create("my-agent", owner="devops-team")
receipt = agent.sign("create_issue", params={"title": "fix bug"})
assert agent.verify(receipt)

Covers: Identity, parameter integrity, replay, freshness, target binding.

Doesn't cover: Preventing the action (signing is attestation, not policy).

✅ 5. Keep a tamper-evident audit log

Signing individual requests is good. Chaining them into a tamper-evident log is better. If someone deletes or reorders records, the chain breaks.

Signet does this automatically — every signed receipt is appended to a SHA-256 hash-chained JSONL log at ~/.signet/audit/:

record_1: { receipt, prev_hash: "sha256:0000...", record_hash: "sha256:abc1..." }
record_2: { receipt, prev_hash: "sha256:abc1...", record_hash: "sha256:def2..." }
record_3: { receipt, prev_hash: "sha256:def2...", record_hash: "sha256:ghi3..." }

Query and verify from the CLI:

signet audit --since 24h              # what happened today
signet audit --tool github --since 7d # github calls this week
signet audit --verify                 # verify all signatures
signet verify --chain                 # check hash chain integrity

Or from Python:

for record in agent.audit_query(since="24h"):
    print(f"{record.receipt.ts}  {record.receipt.action.tool}")

chain = agent.audit_verify_chain()
assert chain.valid

Covers: Tamper detection, incident forensics, compliance audit.

Doesn't cover: Tamper proof (someone with disk access can delete the entire log; off-host anchoring is on the roadmap).

✅ 6. Implement rate limiting and timeouts

Even with signing, a compromised agent can flood your server. Add rate limits:

const callCounts = new Map<string, number>();

server.setRequestHandler(CallToolRequestSchema, async (request) => {
  const signer = request.params.arguments?._meta?._signet?.signer?.name ?? "unknown";
  const count = (callCounts.get(signer) ?? 0) + 1;
  callCounts.set(signer, count);

  if (count > 100) {  // per-agent limit
    return { content: [{ type: "text", text: "Rate limit exceeded" }], isError: true };
  }

  // proceed...
});

And always set timeouts on tool execution:

const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 30_000);
try {
  const result = await executeTool(args, { signal: controller.signal });
} finally {
  clearTimeout(timeout);
}

✅ 7. Principle of least privilege

Don't give your MCP server access to everything. Run it with minimal permissions:

Separate API keys per tool (read-only key for list_issues, write key for create_issue)
Filesystem access scoped to specific directories
Database user with only the required grants
Network egress limited to required endpoints

This is independent of MCP — it's basic defense-in-depth.

Summary

#	Practice	Protects against	Difficulty
1	TLS	Eavesdropping	Easy
2	Input validation	Injection, malformed data	Easy
3	Authentication	Unauthorized callers	Medium
4	Request signing	Tampering, replay, impersonation	3 lines
5	Audit log	Incident response, compliance	Automatic with signing
6	Rate limiting	Denial of service	Easy
7	Least privilege	Blast radius	Medium

Most MCP servers today implement 1-3 at best. Steps 4 and 5 — signing and audit — are the gap. They're also the hardest to bolt on after the fact, which is why starting with a library that handles both is worth the npm install.

Get Started

npm install @signet-auth/core @signet-auth/mcp
# or
pip install signet-auth

GitHub: github.com/Prismer-AI/signet

Apache-2.0 + MIT dual licensed. Open source, no SaaS, no phone-home.

If your AI agent can delete a database, you should be able to prove it did.