Forem: willamhou

Building an IPC bus for Kubernetes sidecars: WAL, DLQ, and ring-buffer backpressure

willamhou — Thu, 23 Apr 2026 02:18:14 +0000

If you put two sidecars in a pod and ask them to talk to each other over HTTP, sooner or later one of them crashes mid-request and you lose a message. If you do it enough times, you reinvent a message bus.

This post is about the small in-pod message bus we ended up writing for k8s4claw, a Kubernetes operator for AI agent runtimes. The bus sits between channel sidecars (Slack, Discord, Webhook) and the agent runtime container. It has four wire protocols, a write-ahead log, a BoltDB-backed dead letter queue, and a ring buffer with backpressure. All of it is open source (internal/ipcbus/), around 2k lines of Go.

This post is the design doc you actually want to read, not the one we had to write.

The shape of the problem

A Claw pod looks like this when it has a Slack channel attached:

┌──────────────────────────────────────────────┐
│  Pod                                         │
│                                              │
│  [channel-slack] ──UDS──► [ipc-bus] ──►┐     │
│                                        ▼     │
│                                  [runtime]   │
│                                              │
└──────────────────────────────────────────────┘

Three containers. The channel sidecar reads from Slack. The runtime is the actual AI agent. The IPC bus is a native sidecar (init container with restartPolicy: Always) that routes messages between them.

The naive version of this is: let the two containers talk HTTP directly. The reality is that at least four things are going to go wrong:

The runtime will be overloaded when a Slack event arrives and we need somewhere to buffer it.
The runtime will crash mid-response and we need to redeliver.
A slow downstream (say, a user's laptop on 3G) will fall behind and we need to push back instead of dropping.
Two different runtimes we support speak four different wire protocols. HTTP isn't enough.

So we wrote a bus. Let me walk through the four mechanisms that earn their keep.

Mechanism 1 — length-prefix framing

This isn't glamorous, but it's the first thing you get wrong in a message bus.

Every Message is a JSON blob on the wire:

type Message struct {
    ID            string          `json:"id"`
    Type          MessageType     `json:"type"`
    Channel       string          `json:"channel,omitempty"`
    CorrelationID string          `json:"correlationId,omitempty"`
    ReplyTo       string          `json:"replyTo,omitempty"`
    Timestamp     time.Time       `json:"timestamp"`
    Payload       json.RawMessage `json:"payload,omitempty"`
}

On the wire it looks like [4-byte big-endian length][JSON bytes]:

const (
    MaxMessageSize  = 16 * 1024 * 1024
    FrameHeaderSize = 4
)

func WriteMessage(w io.Writer, msg *Message) error {
    data, err := json.Marshal(msg)
    if err != nil {
        return fmt.Errorf("failed to marshal message: %w", err)
    }
    if len(data) > MaxMessageSize {
        return fmt.Errorf("message size %d exceeds maximum %d",
            len(data), MaxMessageSize)
    }

    frame := make([]byte, FrameHeaderSize+len(data))
    binary.BigEndian.PutUint32(frame, uint32(len(data)))
    copy(frame[FrameHeaderSize:], data)
    _, err = w.Write(frame)
    return err
}

Why length-prefix instead of newline-delimited JSON? Because JSON payloads can contain newlines inside strings and you'd have to escape them on the wire. Length-prefix framing just works: a reader reads 4 bytes, gets the length, reads that many bytes, deserializes. No lookahead, no escape tables.

The 16 MB cap is there to fail loudly rather than run out of memory on a malformed header. In practice our real messages are well under 64 KB.

Mechanism 2 — four bridge protocols behind one interface

Different runtimes speak different things:

Runtime	Protocol	Why
OpenClaw	WebSocket	Full-duplex, JSON-native, easy from Node.js
NanoClaw	UDS	Lowest overhead for same-pod communication
ZeroClaw	SSE	Already has an HTTP API, SSE for server-push
PicoClaw	TCP	Minimal client, hand-rolled in 50 lines

The bus abstracts them behind one interface:

type RuntimeBridge interface {
    Connect(ctx context.Context) error
    Send(ctx context.Context, msg *Message) error
    Receive(ctx context.Context) (<-chan *Message, error)
    Close() error
}

Four methods. Adding a new protocol is one file (example: TCP bridge):

type TCPBridge struct{ streamBridge }

func (b *TCPBridge) Connect(ctx context.Context) error {
    conn, err := (&net.Dialer{}).DialContext(ctx, "tcp", b.addr)
    if err != nil {
        return err
    }
    b.conn = conn
    return nil
}

streamBridge is a shared base that implements Send/Receive/Close on top of any net.Conn. It handles context.Context deadlines properly:

func (b *streamBridge) Send(ctx context.Context, msg *Message) error {
    b.mu.Lock()
    defer b.mu.Unlock()

    if b.conn == nil {
        return fmt.Errorf("not connected")
    }

    // Respect context deadline for the write.
    if deadline, ok := ctx.Deadline(); ok {
        _ = b.conn.SetWriteDeadline(deadline)
        defer func() { _ = b.conn.SetWriteDeadline(time.Time{}) }()
    }

    return WriteMessage(b.conn, msg)
}

The subtle bit is Receive. ReadMessage blocks on the socket. If the caller cancels the context, we want the read to unblock. So Receive spawns a second goroutine whose only job is to watch the context and call Close on the conn, which makes the blocked ReadMessage return with an error.

go func() {
    select {
    case <-ctx.Done():
        _ = b.conn.Close()
    case <-b.closed:
    }
}()

The SSE bridge is the odd one out because SSE is unidirectional (server → client, event-stream format) and we need bidirectional. So it uses an HTTP POST for send and an SSE GET /events for receive, with exponential-backoff reconnect on the stream:

backoff := time.Second
for {
    // ... connect and read events ...
    time.Sleep(backoff)
    backoff *= 2
    if backoff > 30*time.Second {
        backoff = 30 * time.Second
    }
}

Mechanism 3 — Write-Ahead Log (WAL)

This is the one that earns the bus the right to exist.

When a message comes in from a channel sidecar, the router does three things:

Append a WAL entry to disk (emptyDir-backed) with state pending.
Call bridge.Send(ctx, msg) to hand it off to the runtime bridge.
Mark the WAL entry complete as soon as Send returns success. If Send fails, call scheduleRetry.

We delivery-mark on transport success (the bridge accepted the bytes), not on runtime ack. We considered a runtime-ack round-trip and decided against it: it doubles round-trips, forces every runtime to implement ack semantics, and our Message.ID is already idempotency-safe so downstream retries aren't harmful. If a message leaves bridge.Send OK but the runtime crashes before processing it, we lose that one message. Tradeoff: acceptable for a chat agent, not acceptable for a payment system. Different design calls, different bus.

scheduleRetry increments Attempts on the WAL entry. After maxRetryAttempts = 5, the entry is marked dlq and a copy is parked in the DLQ.

The WAL is a JSON-lines file. Each line is a WALEntry:

type WALEntry struct {
    ID       string   `json:"id"`
    Channel  string   `json:"channel"`
    State    WALState `json:"state"`       // pending | complete | dlq
    Attempts int      `json:"attempts"`
    TS       string   `json:"ts"`
    Msg      *Message `json:"msg,omitempty"`
}

JSON-lines is nice because you can cat wal.log | jq during an incident and see exactly what the bus was doing. It's also append-only, which means writes are O(1) and you never corrupt the middle of the file on a crash — at worst you have a half-written last line, which the recovery code handles.

The interesting operation is compaction. The file grows without bound otherwise. Compaction rewrites the file keeping only pending entries:

func (w *WAL) Compact() error {
    // ... write all pending entries to wal.log.tmp ...
    // atomic rename
    return os.Rename(tmpPath, w.path())
}

func (w *WAL) NeedsCompaction() bool {
    info, _ := w.file.Stat()
    return info.Size() > compactionThreshold  // 10 MB
}

We don't compact on every Complete call — that would tank throughput. The cmd/ipcbus binary runs a 60-second ticker that checks NeedsCompaction() and rewrites the file when it grows past 10 MB. That's a coarse heuristic — it will compact even if most entries are still pending, wasting some I/O — but it's simple and steady-state overhead is near zero. A smarter policy (also consider the pending ratio, pre-commit) would be a reasonable first PR.

The WAL does not fsync on every append. We batch. If a node hard-kills, we can lose the last few hundred milliseconds of messages. That's an acceptable tradeoff for a system where the upstream Slack delivery is already best-effort. If you care more about durability, Flush() is exposed and you can call it from your own code, but we chose not to make it automatic.

Mechanism 4 — Dead Letter Queue (DLQ)

After 5 delivery attempts, a message is "dead." We don't silently drop it; we move it to the DLQ:

func NewDLQ(path string, maxSize int, ttl time.Duration) (*DLQ, error) {
    db, err := bolt.Open(path, 0600, &bolt.Options{Timeout: 1 * time.Second})
    // ...
}

BoltDB is embedded KV storage with B+tree on-disk layout. It's fast, transactional, and single-file. Perfect for a sidecar that needs a few megabytes of dead messages, queryable by ID and age.

Two eviction policies:

maxSize — a hard cap on entry count. When we're full, we evict the oldest.
ttl — entries older than the TTL are purged. NewDLQ(path, maxSize, ttl) takes both as constructor args; the cmd/ipcbus binary passes maxSize=10000, ttl=24h and runs an hourly PurgeExpired ticker. Library callers can pick their own.

This matters because the DLQ is the debugging surface for the bus. Something went wrong? kubectl exec into the sidecar, open the BoltDB file, and look at the last N entries. We've caught a couple of real bugs this way that would have been invisible with "drop on failure."

func (d *DLQ) PurgeExpired() (int, error)
func (d *DLQ) Size() int
func (d *DLQ) List() ([]*DLQEntry, error)

Deliberately no replay-from-DLQ. If something's dead, it's dead. We want human attention, not automatic retry that hides a real problem.

Mechanism 5 — ring buffer with backpressure

The remaining problem: what if a channel sidecar is producing faster than the runtime can consume?

Naive answer: unbounded queue. Result: OOM-killed pod.

Real answer: bounded ring buffer with high/low watermarks.

func NewRingBuffer(size int, highWatermark, lowWatermark float64) *RingBuffer {
    // ... defaults to high=0.8, low=0.3 ...
}

When the buffer fills past 80%, the bus emits a slow_down control message upstream. The channel sidecar sees it and stops pulling from Slack. When the buffer drains below 30%, the bus emits resume and the sidecar starts pulling again.

Why two watermarks? Because if you use one, you thrash. Right at the threshold, every push flips state. Two watermarks with a gap gives you hysteresis. Classic control-theory stuff, very little Go stuff.

The slow_down / resume messages ride the same wire format as everything else:

switch m.Type {
case TypeAck, TypeNack, TypeSlowDown, TypeResume,
     TypeShutdown, TypeRegister, TypeHeartbeat:
    return true
}

Treating control traffic as just another MessageType means channel sidecars don't need a separate control channel. One TCP/UDS/WS connection carries both payloads and backpressure signals. Simpler, fewer failure modes.

Shutdown

Graceful shutdown is its own hazard. On SIGTERM the cmd/ipcbus binary runs a local shutdown() helper that does the bare minimum:

func shutdown(logger, router, wal, bridge, cancel) {
    router.SendShutdown()      // tell sidecars we're going away
    time.Sleep(5 * time.Second) // fixed grace window
    wal.Flush()                 // flush WAL to disk
    bridge.Close()              // close the runtime bridge
    cancel()                    // stop the UDS server + background tickets
}

That's it. No polling, no early exit if sidecars disconnect, no DLQ close (process-exit flushes BoltDB's mmap and that's enough). Whatever is still pending in the WAL when we exit gets replayed on next startup — that's the whole point of the WAL.

There's also a fancier ShutdownOrchestrator in internal/ipcbus/shutdown.go that takes a drainTimeout parameter and polls router.ConnectedCount() every 100 ms to exit early, but the current binary doesn't wire it up. Good first PR: swap the local helper out for the orchestrator so the sleep becomes a real wait-for-drain.

What we didn't do (on purpose)

Multi-pod clustering. The bus is deliberately in-pod. If you want cross-pod messaging, use a real broker (NATS, Redis streams). Scoping this to one pod kept us sane.
Ordering guarantees across channels. Within one channel, messages are ordered. Across channels, no promise. Most agent workloads don't care.
Exactly-once. At-least-once with idempotent consumers is simpler and good enough. The runtime is expected to deduplicate on Message.ID.
Protobuf on the wire. JSON is ~2× larger but 10× easier to debug. Given our throughput (tens of messages per second per pod, not millions), JSON is the right call.

Testing

We aimed for >80% statement coverage on the ipcbus package, approximately. The non-obvious piece: most of the reliability features are hard to unit-test with mocks because they're about failure modes. So we have a lot of tests that spin up real local listeners (net.Listen("tcp", "127.0.0.1:0"), net.Listen("unix", t.TempDir()+"/sock"), httptest.NewServer(...)) and exercise the bridges end-to-end.

For example, the SSE bridge test spins up an httptest server that handles both GET /events (as an SSE stream) and POST /messages, and checks that connecting, sending, and receiving all work:

func TestSSEBridge_SendReceive(t *testing.T) {
    srv, ready := sseEchoServer(t)
    defer srv.Close()

    bridge := NewSSEBridge(srv.URL)
    // ... connect, wait for SSE stream to establish, send, receive ...
}

About 70 tests total, -race clean. Good enough for a sidecar.

What this bought us

A uniform contract for channel sidecars. You write one Slack sidecar, it works with every runtime. You write one Discord sidecar, same thing. Runtime authors pick a protocol that fits their stack; they don't think about durability, retries, or backpressure — the bus handles it.

The runtime adapter for a new protocol is ~50 lines. The channel sidecar SDK (sdk/channel/) hides the framing entirely; you call client.Send(ctx, json.RawMessage(...)) and move on.

The whole ipcbus package is ~2k lines of Go. If you want to read one file to get the flavor, router.go is where all five mechanisms meet.

What to look at next

The k8s4claw repo if you want to use it
internal/ipcbus/ if you want to read the code
The intro post if you want context on how this fits into the operator

Open source, Apache-2.0. Questions and PRs welcome. If you've built something similar and went in a different direction, I'd love to hear why in the comments.

When Rust's Exhaustive Match Helps (And When It Doesn't): Notes from a Bare-Metal Hypervisor

willamhou — Wed, 22 Apr 2026 03:41:04 +0000

Disclaimer: This is about an experimental hypervisor project that only runs on QEMU virt — no real-hardware validation yet. The lessons apply to "Rust's tooling edges in systems programming," not production guidance.

10 weeks into writing an ARM64 bare-metal hypervisor, I assumed Rust's exhaustive match would be the safety net when I extended my state machine. Two observations, from one week of commits: exhaustive match didn't help my state machine at all, but caught 6 errors the one time I extended my Device enum. This post is about why — and why the distinction is about cardinality, not typestate vs tag enums.

I'm writing an ARM64 bare-metal hypervisor. Part of it is a thing called a Secure Partition (SP) — a lightweight VM managed by the SPMC. Each SP has a lifecycle: Reset → Idle → Running → Blocked → Preempted. 5 states, 7 legal transitions.

Two weeks ago I added a new transition: Blocked → Preempted, for chain preemption between SPs. By the textbook, this is exactly the scenario where Rust's enum + match should shine: add a state/transition, the compiler finds every site that needs updating.

The compiler said nothing.

This post is about why I didn't use the "enum-with-fields" pattern you see in tutorials, why match exhaustiveness didn't help on this state machine, and where it actually did help.

The Real Code

No toy examples. Here's the actual SpState from the repo:

// src/sp_context.rs
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
#[repr(u8)]
pub enum SpState {
    Reset = 0,
    Idle = 1,
    Running = 2,
    Blocked = 3,
    Preempted = 4,
}

Classic tag-only enum — #[repr(u8)], every variant is one byte, no payload. Why not the textbook Running { entry_pc: u64 } / Preempted { saved_ctx: VcpuContext }?

Because the state lives in an AtomicU8.

The SPMC runs on multiple physical CPUs. Different CPUs inside TF-A's SPMD (Secure Partition Manager Dispatcher) can route requests to the same SP at once. Two CPUs racing to do Idle → Running — one must lose, or both will ERET into the same SP and clobber register context.

CAS drives the race:

pub fn try_transition(&self, expected: SpState, new_state: SpState) -> Result<(), SpState> {
    match self.state.compare_exchange(
        expected as u8,      // success: AcqRel publishes our context-save
        new_state as u8,     // failure: Acquire syncs the observed loser
        Ordering::AcqRel,
        Ordering::Acquire,
    ) {
        Ok(_) => Ok(()),
        Err(actual) => Err(SpState::try_from(actual).expect("corrupt SP state value")),
    }
}

The constraint isn't memory layout — #[repr(u8, C)] on a fields-carrying enum does give stable layout. The real constraint is size: AtomicU8 wraps one byte, and any enum with a u64 payload is at least 8 bytes wide. Atomic u64 CAS is fine on aarch64, but that means every state change either serializes through a fat struct CAS or falls back to a lock. I wanted single-byte CAS in the fast path, so the payload lives elsewhere (in a separate VcpuContext guarded by the state transition itself).

Side note on expect("corrupt SP state value"): it really does panic. In this project the panic handler halts the offending CPU and dumps state via UART — because if the AtomicU8 ever holds a value outside 0..=4, memory corruption has already happened and limping along is worse than stopping. That's a conscious choice for this binary, not a general bare-metal guideline.

Why Exhaustive Match Didn't Help

The legal-transition check lives in one function:

// src/sp_context.rs
pub fn transition_to(&mut self, new_state: SpState) -> Result<(), &'static str> {
    let current = self.state();
    let valid = match (current, new_state) {
        (SpState::Reset, SpState::Idle) => true,
        (SpState::Idle, SpState::Running) => true,
        (SpState::Running, SpState::Idle) => true,
        (SpState::Running, SpState::Blocked) => true,
        (SpState::Blocked, SpState::Running) => true,
        (SpState::Blocked, SpState::Preempted) => true,  // ← the newly added line
        (SpState::Running, SpState::Preempted) => true,
        (SpState::Preempted, SpState::Running) => true,
        _ => false,
    };
    // ...
}

Note the final _ => false. This is not an exhaustive match — the wildcard swallows every unlisted combination as "illegal."

The commit that added Blocked → Preempted was literally 1 line. The compiler reported nothing, because to the compiler, all 25 (from, to) combinations are covered (7 explicit + _ fallback).

I could have replaced _ => false with all 18 illegal combinations enumerated. I started to — "exhaustive is more Rust-y". Then I gave up halfway:

// This way...
(SpState::Reset, SpState::Reset) => false,
(SpState::Reset, SpState::Running) => false,
(SpState::Reset, SpState::Blocked) => false,
// ... 15 more lines of this

No new information, and every future state addition means maintaining an N² table. _ => false is the documentation here: what's listed is legal; everything else isn't.

Verdict: For simple C-style enum + state-transition pairs, match exhaustiveness doesn't save you. Bugs at this layer can only be caught by unit tests (my test_sp_context.rs has 58 assertions covering every legal transition plus key illegal ones).

Where It Actually Saved Me

The place where match exhaustiveness actually saved me was device dispatch.

My hypervisor uses a Device enum to enumerate all virtual devices. Every time the guest touches MMIO, a match dispatches to the right implementation:

// src/devices/mod.rs
pub enum Device {
    Uart(pl011::VirtualUart),
    Gicd(gic::VirtualGicd),
    Gicr(gic::VirtualGicr),
    VirtioBlk(virtio::mmio::VirtioMmioTransport<virtio::blk::VirtioBlk>),
    VirtioNet(virtio::mmio::VirtioMmioTransport<virtio::net::VirtioNet>),
    Pl031(pl031::VirtualPl031),
}

This is a fields-carrying enum — each variant holds the state struct for its device. No _ fallback on matches against it, because every variant has its own handler:

impl MmioDevice for Device {
    fn read(&mut self, offset: u64, size: u8) -> Option<u64> {
        match self {
            Device::Uart(d) => d.read(offset, size),
            Device::Gicd(d) => d.read(offset, size),
            Device::Gicr(d) => d.read(offset, size),
            Device::VirtioBlk(d) => d.read(offset, size),
            Device::VirtioNet(d) => d.read(offset, size),
            Device::Pl031(d) => d.read(offset, size),
        }
    }
    // write, contains, is_ready, ...
}

When I added Pl031 (PL031 RTC) for Android boot, I only touched the enum definition. The compiler immediately fired 6 errors — every site that matches against Device was missing the Pl031 arm:

error[E0004]: non-exhaustive patterns: `&Device::Pl031(_)` not covered
  --> src/devices/mod.rs:51:15
error[E0004]: non-exhaustive patterns: `&mut Device::Pl031(_)` not covered
  --> src/devices/mod.rs:62:15
error[E0004]: non-exhaustive patterns: `&Device::Pl031(_)` not covered
  --> src/devices/mod.rs:73:15
// ... 6 total

Two of those were helper methods I'd written when adding VirtioNet and completely forgotten about. Had I used C switch without -Wswitch-enum (which Linux kernel and TF-A both enable by default), those two sites would silently fall into default and return "unknown device." The guest would do any MMIO to the RTC, fail to find a device, and hang mid-boot with an error pointing somewhere completely unrelated.

C with -Wswitch-enum + -Werror gives you the same check — the relevant difference is that Rust makes it a precondition for compiling instead of a build-system setting you can drop. Worth more in a solo project, less in a shop with a strict style guide.

Either way, the compiler caught this bug instead of the guest doing so at boot time.

When Exhaustive Match Actually Pays Off

Reviewing this state-machine extension + Device extension, here's my distilled rule:

Exhaustive match saves you: fields-carrying enum + every variant has independent handler logic.

Device::{Uart, Gicd, ..., Pl031} — each device's read/write is totally different
MmioAccess::{Read { reg, size }, Write { reg, size, val }} — read vs write semantics differ
ExitReason::{HvcCall, SmcCall, DataAbort, WfiWfe, ...} — each exception class has its own handler

Common trait: adding a variant potentially leaves gaps across the entire codebase, and each gap's correct implementation is non-trivial (not just "error vs OK" binary output).

Exhaustive match doesn't help: simple tag enum + cartesian-product check.

State machine (from, to) transition table — N² explosion, _ => false is more readable
Permission matrix (user_role, action) — same
Input sanity check match(input) { valid_range => ..., _ => reject } — tautological

These scenarios are "enumerate a small set of legal cases, reject everything else." _ => fallback loses no information — it's more readable.

A Few Takeaways

1. #[repr(u8)] is everyday life in hypervisor/kernel/driver code. Don't apologize for the atomic trade-off.

Every time a "Rust state machine" tweet appears, someone in the replies recommends typestate. Typestate is genuinely powerful when transitions happen through owning APIs (File::open → Handle<Open>), but it doesn't compose with shared mutable state across CPUs — the entire point of AtomicU8 is that multiple cores hold a reference to one byte. Typestate requires owning self by value to consume the old state; a multi-CPU SPMC can't do that on the fast path. Not a rejection of typestate, just the wrong tool for this edge.

2. _ => fallback isn't a sin, but ask yourself every time.

"If I add a new variant in the future, should this site force me to update it?"

Yes → drop the _, enumerate every variant
No (illegal state-machine pair, MMIO unknown-offset) → _ => default is documentation

3. State-machine correctness is never a gift from Rust. It's a gift from tests + documentation + code review.

My test_sp_context.rs has dedicated tests for every legal transition, a bunch of illegal ones, and CAS races. Rust didn't generate those; I wrote them. Rust saved me from some defensive code (no "sixth value" of SpState — try_from_u8 rejects it), but whether the legal-transition table is correct, Rust has no opinion.

4. What really saves you is "fields-carrying enum + each variant has its own handler."

That's Rust's signature strength. Find the places in your codebase that fit this pattern and get them right — it pays more than agonizing over whether the state machine should be typestate-ified.

Closing

My hypervisor isn't a "zero-unwrap" project. The repo has about 6 unwrap() calls (concentrated in test fixtures and boot-time paths that can't reasonably panic) and 45 _ => default fallback arms (mostly in MMIO register decode for unknown offsets).

Every unwrap() and _ => was a decision at the time, not laziness. Engineering beats slogans.

Rust gives you a good weapon. It doesn't think for you. Whether the state-transition table is legal is in your head, not the compiler's.

Code: github.com/willamhou/hypervisor

Blog: willamhou.github.io/hypervisor

This is part 5 of the ARM64 Hypervisor development series. The Chinese version is the canonical source — see part5-enum-state-machine.md.

k8s4claw: A Kubernetes Operator for Managing AI Agent Runtimes

willamhou — Tue, 21 Apr 2026 05:08:42 +0000

Every AI agent framework has its own deployment story. Claude-based assistants run one way, OpenAI agents another, security-focused runtimes yet another. If you run more than one on Kubernetes, you end up writing the same boilerplate over and over: secret management, persistent storage, graceful updates, inter-service messaging, observability.

k8s4claw is an open-source Kubernetes operator that wraps all of this behind a single CRD. You describe what the agent is, it handles how it runs.

apiVersion: claw.prismer.ai/v1alpha1
kind: Claw
metadata:
  name: research-agent
spec:
  runtime: openclaw
  config:
    model: "claude-sonnet-4"
  credentials:
    secretRef:
      name: llm-api-keys

The operator reconciles this into a StatefulSet, headless Service, ConfigMap, ServiceAccount, PodDisruptionBudget, and optionally NetworkPolicy and Ingress. When you add a channel (Slack, Discord, Webhook), it also wires up sidecars and a local message bus.

This post walks through the architecture, shows how to get it running locally, and explains the design decisions behind the IPC bus, the auto-update controller, and the runtime adapter system.

The Problem

We had several agent runtimes in flight at once — different languages, different process models, different resource profiles:

Runtime	Language	Use Case
OpenClaw	TypeScript/Node.js	Full-featured AI assistant
NanoClaw	TypeScript/Node.js	Lightweight personal assistant
ZeroClaw	Rust	High-performance agent
PicoClaw	Go	Ultra-minimal serverless
IronClaw	Rust + WASM	Security-focused agent
HermesClaw	Python	Conversational with tool use
K8sOps	Go	Cluster self-healing (claw4k8s)

Each had its own Helm chart, sidecar layout, and update strategy. Adding a Slack channel meant editing several files. Rotating credentials meant touching every deployment. Rolling back a bad update was a manual process.

We wanted one control plane for all of them.

Architecture

graph TB
    subgraph "Kubernetes Cluster"
        OP[k8s4claw Operator]

        subgraph "Claw Pod (with channels)"
            INIT["claw-init"]
            RT["Runtime Container"]
            IPC["IPC Bus Sidecar"]
            CH["Channel Sidecar"]
        end

        STS[StatefulSet]
        SVC[Service]
        CM[ConfigMap]
        PVC[(PVCs)]

        OP -->|manages| STS
        OP -->|manages| SVC
        OP -->|manages| CM
        STS -.->|runs| RT
        STS -.->|runs| IPC
        STS -.->|runs| CH

        CH <-->|UDS| IPC
        IPC <-->|Bridge| RT
    end

    EXT["Slack / Discord / Webhook"]
    CH <-->|API| EXT

The operator watches Claw custom resources and reconciles a full stack of Kubernetes objects. A minimal agent (no channels, no persistence) gets just the runtime container plus claw-init. If you declare any channels in spec.channels, the operator also injects:

claw-init — an init container that merges default runtime config with any user overrides before the runtime starts.
Runtime container — the actual AI agent binary.
IPC Bus sidecar (only when channels are present) — a WAL-backed message router that sits between the runtime and the channel sidecars.
Channel sidecar(s) — one per referenced ClawChannel (Slack, Discord, Webhook today).

There is a second CRD, ClawChannel, that describes how to connect to an external system. Channels are defined once and referenced by many Claws.

Quick Start

Prerequisites

Kubernetes 1.28+ (or kind for local development)
Go 1.25+
controller-gen (go install sigs.k8s.io/controller-tools/cmd/controller-gen@latest) — needed by make install

Install and run

git clone https://github.com/Prismer-AI/k8s4claw.git
cd k8s4claw

# Install CRDs into the active cluster
make install

# Run the operator locally against your current kubeconfig.
# --disable-webhooks lets you skip cert-manager setup during local dev.
# In-cluster deployments should leave webhooks enabled.
go run ./cmd/operator/ --disable-webhooks

Create your first agent

kubectl create secret generic llm-api-keys \
  --from-literal=ANTHROPIC_API_KEY=sk-ant-xxx

cat <<EOF | kubectl apply -f -
apiVersion: claw.prismer.ai/v1alpha1
kind: Claw
metadata:
  name: my-agent
spec:
  runtime: openclaw
  config:
    model: "claude-sonnet-4"
  credentials:
    secretRef:
      name: llm-api-keys
  persistence:
    session:
      enabled: true
      size: 2Gi
      mountPath: /data/session
    workspace:
      enabled: true
      size: 10Gi
      mountPath: /workspace
EOF

kubectl get claw my-agent -w

Connect Slack

apiVersion: claw.prismer.ai/v1alpha1
kind: ClawChannel
metadata:
  name: team-slack
spec:
  type: slack
  mode: bidirectional
  credentials:
    secretRef:
      name: slack-bot-token
  config:
    appId: "A0123456789"

Reference it from your Claw:

spec:
  channels:
    - name: team-slack
      mode: bidirectional

On the next reconcile the operator injects a Slack sidecar, spins up the IPC bus sidecar, and wires them together. The runtime container does not need to know anything about Slack — it just talks to the bus.

Deep Dive: The IPC Bus

The IPC bus is the most interesting piece of k8s4claw. It is a Kubernetes native sidecar (an init container with restartPolicy: Always) that routes JSON messages between channel sidecars and the agent runtime.

Channel Sidecar ──UDS──► IPC Bus ──Bridge──► Runtime Container
                         │ WAL  │
                         │ DLQ  │
                         │ Ring │
                         └──────┘

Why not just HTTP?

We tried. The problem is reliability. When a Slack event arrives while the runtime is overloaded, you need somewhere to buffer it. If the runtime crashes mid-response, you need to redeliver. When a channel sidecar falls behind, you need backpressure instead of dropped messages.

Three mechanisms do the work:

1. Write-Ahead Log (WAL) — Every inbound message is appended to a WAL on emptyDir before delivery. On restart, unacknowledged messages are replayed. Periodic compaction keeps the file bounded.

2. Dead Letter Queue (DLQ) — Messages that exceed the retry limit land in a BoltDB-backed DLQ instead of being dropped silently. You can inspect them later.

3. Ring buffer with backpressure — A fixed-size circular buffer with configurable high/low watermarks. Crossing the high watermark sends slow_down upstream; draining to the low watermark sends resume.

Bridge protocols

Different runtimes speak different wire protocols. The bus abstracts this behind a RuntimeBridge interface:

Runtime	Bridge	Protocol
OpenClaw	WebSocket	Full-duplex JSON over WS
NanoClaw	UDS	Length-prefix framed
ZeroClaw	SSE	HTTP POST + Server-Sent Events
PicoClaw	TCP	Length-prefix framed

Here is the actual interface (internal/ipcbus/bridge.go):

type RuntimeBridge interface {
    Connect(ctx context.Context) error
    Send(ctx context.Context, msg *Message) error
    Receive(ctx context.Context) (<-chan *Message, error)
    Close() error
}

Adding a new transport means implementing these four methods.

Deep Dive: Auto-Update Controller

The auto-update controller polls OCI registries on a cron schedule, filters new tags by a semver constraint, and performs health-verified rollouts with automatic rollback.

spec:
  autoUpdate:
    enabled: true
    versionConstraint: "^1.x"
    schedule: "0 3 * * *"
    healthTimeout: "10m"
    maxRollbacks: 3

How it works

Poll — on each cron tick, list tags from the registry and filter by the semver constraint.
Initiate — annotate the Claw with the target image and transition into the HealthCheck phase.
Health check — watch the StatefulSet readiness until all replicas are ready or the timeout fires.
Success — update status, clear the annotation, schedule the next cron tick.
Timeout — roll back to the previous image.
Circuit breaker — after N consecutive rollbacks, stop trying and emit an event plus a Prometheus metric.

The state machine lives in annotations and status conditions, so it survives operator restarts:

phase := claw.Annotations["claw.prismer.ai/update-phase"]
if phase == "HealthCheck" {
    return r.reconcileHealthCheck(ctx, &claw)
}

Version history

Every attempt is recorded:

status:
  autoUpdate:
    currentVersion: "1.2.0"
    versionHistory:
      - version: "1.2.0"
        appliedAt: "2026-03-28T03:00:00Z"
        status: Healthy
      - version: "1.1.5"
        appliedAt: "2026-03-21T03:00:00Z"
        status: RolledBack
    failedVersions: ["1.1.5"]
    circuitOpen: false

The Runtime Adapter Pattern

Each runtime is a Go struct implementing RuntimeAdapter:

type RuntimeAdapter interface {
    // Pod shape
    PodTemplate(claw *v1alpha1.Claw) *corev1.PodTemplateSpec
    HealthProbe(claw *v1alpha1.Claw) *corev1.Probe
    ReadinessProbe(claw *v1alpha1.Claw) *corev1.Probe
    DefaultConfig() *RuntimeConfig
    GracefulShutdownSeconds() int32

    // Spec validation
    Validate(ctx context.Context, spec *v1alpha1.ClawSpec) field.ErrorList
    ValidateUpdate(ctx context.Context, oldSpec, newSpec *v1alpha1.ClawSpec) field.ErrorList
}

A new adapter typically lives in a single file of ~100 lines. The shared BuildPodTemplate helper handles init containers, volume mounts, security context, and environment variables, so the adapter only declares what is actually different:

type MyRuntimeAdapter struct{}

func (a *MyRuntimeAdapter) PodTemplate(claw *v1alpha1.Claw) *corev1.PodTemplateSpec {
    return BuildPodTemplate(claw, &RuntimeSpec{
        Image:     "my-registry/my-runtime:latest",
        Ports:     []corev1.ContainerPort{{Name: "gateway", ContainerPort: 8080}},
        Resources: resources("100m", "256Mi", "500m", "512Mi"),
        // ...
    })
}
// plus HealthProbe, ReadinessProbe, DefaultConfig, GracefulShutdownSeconds,
// Validate, ValidateUpdate

Validation is per-runtime on purpose. OpenClaw and IronClaw require credentials because they call LLM APIs. ZeroClaw and PicoClaw permit credential-less operation. HermesClaw rejects spec.channels because it brings its own gateway. NanoClaw currently has no update-time persistence checks. The point is each adapter owns its own rules.

Go SDK

For programmatic access there is a Go SDK (sdk/):

import (
    "context"

    "github.com/Prismer-AI/k8s4claw/sdk"
)

client, err := sdk.NewClient() // uses the ambient kubeconfig by default
if err != nil {
    return err
}

claw, err := client.Create(ctx, &sdk.ClawSpec{
    Runtime: sdk.OpenClaw,
    Config: &sdk.RuntimeConfig{
        Environment: map[string]string{"MODEL": "claude-sonnet-4"},
    },
})
if err != nil {
    return err
}

// Block until the Claw reaches phase "Running" or ctx expires.
if err := client.WaitForReady(ctx, claw); err != nil {
    return err
}

There is also a channel SDK for writing custom sidecars:

import (
    "context"
    "encoding/json"

    "github.com/Prismer-AI/k8s4claw/sdk/channel"
)

client, err := channel.Connect(ctx,
    channel.WithChannelName("my-channel"), // or set CHANNEL_NAME env
    channel.WithSocketPath("/var/run/claw/bus.sock"),
    channel.WithBufferSize(100),
)
if err != nil {
    return err
}
defer client.Close()

// Send a message to the runtime.
if err := client.Send(ctx, json.RawMessage(`{"text":"Hello"}`)); err != nil {
    return err
}

// Receive returns a channel of *InboundMessage.
inbox, err := client.Receive(ctx)
if err != nil {
    return err
}
for msg := range inbox {
    // handle msg
    _ = msg
}

Testing Strategy

The repo has reasonable test coverage on the core packages. A recent local run looked roughly like this:

Package	Coverage (approx.)
`internal/webhook`	~97%
`internal/runtime`	~94%
`internal/registry`	~86%
`sdk`	~83%
`internal/controller`	~81%
`sdk/channel`	~81%
`internal/ipcbus`	~80%

Numbers move PR by PR. CI publishes a coverage report as an artifact and gates on a total-coverage threshold; there is no per-package floor enforced today. Treat the table as a snapshot, not a contract.

The testing pyramid:

Unit tests — pure functions, table-driven, t.Parallel() everywhere.
Fake-client tests — fake.NewClientBuilder() for controller logic without a real cluster.
envtest integration tests — real etcd + API server for webhook validation and reconcile loops.

The auto-update controller uses dependency injection via Clock and TagLister interfaces so time-dependent and registry-dependent code is fully testable with no network calls.

What's Not Done Yet

Worth being honest about:

custom runtime type is present in the CRD enum but no adapter is registered. If you want a runtime that is not in the built-in list today, you fork and add an adapter.
HermesClaw does not yet integrate with the k8s4claw channel sidecars — it uses its own gateway.
Local operator runs need --disable-webhooks unless you've set up cert-manager or your own TLS. In-cluster deployments via the Helm chart handle this for you.
CRD surface is larger than just Claw — ClawChannel, ClawSelfConfig, and related types are part of the contract. "Single CRD" is a simplification; "small, focused set of CRDs" is closer to the truth.

What's Next

k8s4claw is open source under Apache-2.0. The current open contribution target is Issue #4: add snapshot and PDB envtest coverage. If you want to propose something else, open a new issue and we'll triage it.

GitHub: github.com/Prismer-AI/k8s4claw

If you run AI agents on Kubernetes and you're tired of maintaining the plumbing around them, give it a try. Star the repo if it helps, and open an issue if something is off — both signals are useful.

How to Add Tamper-Evident Audit Trails to Your LangChain Agent

willamhou — Mon, 20 Apr 2026 02:16:33 +0000

Your LangChain agent calls tools. It searches the web, reads files, queries databases, calls APIs. But can you prove what it did?

Logs capture what happened. Cryptographic receipts prove it. The difference matters when an auditor, a customer, or a regulator asks "show me exactly what the agent did and prove it wasn't altered after the fact."

This tutorial adds Ed25519-signed, hash-chained audit trails to a LangChain agent in under 5 minutes. No external service, no API keys, no infrastructure. Everything verifies offline with a public key.

What you'll build

A LangChain agent where every tool call produces a signed receipt containing:

What: which tool was called, with what parameters
Who: the agent's Ed25519 public key
When: timestamp
Proof: Ed25519 signature over JCS-canonicalized (RFC 8785) payload
Chain: SHA-256 hash linking to the previous receipt (tamper-evident ordering)

If anyone modifies a receipt after the fact, the signature breaks. If anyone deletes or reorders receipts, the hash chain breaks.

Install

pip install signet-auth[langchain] langchain-openai langchain-community

Step 1: Create a signing identity

from signet_auth import SigningAgent

# Creates an Ed25519 keypair, stored locally in ~/.signet/
# If the key already exists, just load it:
try:
    agent = SigningAgent("my-langchain-agent")
except Exception:
    agent = SigningAgent.create("my-langchain-agent", owner="acme-corp")

print(f"Public key: {agent.public_key}")

That's it. No key server, no certificate authority. The private key stays on disk, the public key is what verifiers use.

Step 2: Add the signing callback

Signet ships a LangChain callback handler that signs every tool call automatically. Two lines:

from signet_auth.langchain import SignetCallbackHandler

signer = SignetCallbackHandler(agent)

This handler signs the full tool lifecycle: on_tool_start (what was called), on_tool_end (what it returned, hashed), and on_tool_error (what went wrong). If signing fails, the handler logs a warning and lets the agent continue. It never crashes your chain.

Step 3: Wire it into your agent

from langchain import hub
from langchain_community.tools import DuckDuckGoSearchRun
from langchain.agents import AgentExecutor, create_react_agent
from langchain_openai import ChatOpenAI

# Standard LangChain setup
llm = ChatOpenAI(model="gpt-4o-mini")
tools = [DuckDuckGoSearchRun()]
prompt = hub.pull("hwchase17/react")

# Create and run agent with signing callback
agent_executor = AgentExecutor(
    agent=create_react_agent(llm, tools, prompt),
    tools=tools,
    callbacks=[signer],
)

result = agent_executor.invoke({"input": "What is the weather in Tokyo?"})

Every tool call now produces a signed receipt. No code changes to the tools themselves.

Step 4: Inspect the receipts

import json

for i, receipt in enumerate(signer.receipts):
    data = json.loads(receipt.to_json())
    print(f"\nReceipt #{i+1}")
    print(f"  Tool:       {data['action']['tool']}")
    print(f"  Params hash: {data['action']['params_hash']}")
    print(f"  Signature:   {data['sig'][:40]}...")
    print(f"  Timestamp:   {data['ts']}")

Output:

Receipt #1
  Tool:       duckduckgo_search
  Params hash: sha256:a1b2c3...
  Signature:   ed25519:Mz4xNTk2NjQ0NDgw...
  Timestamp:   2026-04-19T10:30:00Z

Receipt #2
  Tool:       _tool_end
  Params hash: sha256:d4e5f6...
  Signature:   ed25519:Nk5yODk3MjE1Njg4...
  Timestamp:   2026-04-19T10:30:01Z

Notice the start/end pair: the first receipt captures the tool call, the second captures a hash of the output. Together they prove what was called and what it returned.

Step 5: Verify a receipt

Anyone with the public key can verify, offline:

from signet_auth import verify

receipt = signer.receipts[0]

is_valid = verify(receipt, agent.public_key)
print(f"Valid: {is_valid}")  # True

Tamper with any field and verification fails:

data = json.loads(receipt.to_json())
data["action"]["tool"] = "evil_tool"  # tamper

from signet_auth import Receipt
tampered = Receipt.from_json(json.dumps(data))
print(verify(tampered, agent.public_key))  # False

Step 6: Verify the audit chain

The audit log is a hash-chained JSONL file. Each entry's hash covers the previous entry, so deleting or reordering receipts breaks the chain:

from signet_auth import audit_verify_chain, default_signet_dir

signet_dir = default_signet_dir()
chain_status = audit_verify_chain(signet_dir)
print(f"Chain intact: {chain_status.valid}")
print(f"Entries: {chain_status.length}")

What this gives you

Without Signet	With Signet
"The agent called web_search" (log entry)	Ed25519 signature proving it, verifiable by anyone with the public key
Logs can be edited after the fact	Signature breaks if any field is modified
No ordering proof	Hash chain breaks if receipts are deleted or reordered
Trust the operator's logs	Verify independently, offline

When you need this

Regulated industries: EU AI Act Article 12 requires "automatic recording" of AI system activities. Signed receipts satisfy this with cryptographic proof, not just logs.
Enterprise deployments: When the question is "can you prove what the agent did?", signed receipts are the answer.
Agent-to-agent: When Agent B needs to verify what Agent A actually did before acting on its output.
Incident response: After something goes wrong, tamper-evident receipts let you reconstruct exactly what happened without trusting anyone's claim.

Next steps

Bilateral co-signing: Have both the agent and the tool server sign each interaction independently. Neither party can fabricate receipts. See signet proxy for MCP integration.
Policy attestation: Evaluate YAML policy rules and include the decision (allow/deny/require_approval) inside the signed receipt.
Delegation chains: Prove that Agent A was authorized by Human B to perform a specific action with scoped constraints.

All of these are in signet-auth today.

pip install signet-auth

GitHub: Prismer-AI/signet

Signet is open source (Apache 2.0). Rust core with Python and TypeScript bindings. No external service, no API keys.

Claude Managed Agents Has Built-in Tracing. Here's What It Can't Do.

willamhou — Tue, 14 Apr 2026 09:52:23 +0000

Claude Managed Agents Has Built-in Tracing. Here's What It Can't Do.

Anthropic shipped Claude Managed Agents last week. The pitch: production-grade agents with sandboxing, scoped permissions, and session tracing — built in, no setup required.

The tracing feature specifically: "Session tracing, integration analytics, and troubleshooting guidance are built directly into the Claude Console, so you can inspect every tool call, decision, and failure mode."

This is genuinely useful. If you're debugging a multi-step agent workflow, having every tool call logged in a console is miles better than parsing stderr.

But there's a distinction worth making — one that matters in exactly the situations where it matters most.

"Anthropic Recorded It" vs. "You Can Prove It"

Claude Managed Agents is cloud-hosted. The tracing data lives in Claude Console, on Anthropic's infrastructure.

That means the audit trail is: Anthropic says this happened.

For most debugging use cases, that's fine. You trust Anthropic. They trust you. The logs are accurate. Nobody is lying.

But consider the situations where audit trails actually get pulled:

Your agent made an unauthorized transfer. The question isn't "what does the console say" — it's "can you prove, to a third party, that the agent executed this action with these parameters at this time, and that this record hasn't been modified?"

A compliance audit. SOC 2, HIPAA, GDPR. The auditor asks for evidence of agent actions on sensitive data. "Here are logs from Anthropic's console" is not the same as "here is a cryptographically signed chain of records that I hold and you can independently verify."

An incident investigation. After a breach, forensic investigators need evidence that is tamper-evident and independently verifiable. If the evidence lives on the infrastructure that may have been compromised — or that a vendor controls — its integrity cannot be assumed.

The distinction isn't about trust in Anthropic. It's about the difference between a record and evidence.

What Cryptographic Signing Adds

A signed audit trail works differently.

Each tool call generates a receipt: the action, the parameters, the timestamp, the agent identity — all hashed and signed with the agent's private Ed25519 key. Receipts chain together: each receipt includes the hash of the previous one. Modifying any record breaks the chain. Deleting a record is detectable.

The key difference: you hold the proof, not a vendor.

from signet_auth import SigningAgent

agent = SigningAgent.create("procurement-bot", owner="ops-team")
receipt = agent.sign("marketplace_purchase",
    params={"item": "GPU-A100", "quantity": 2, "price": 15000})

# This receipt is a cryptographic artifact.
# You hold it. Anthropic doesn't.
# Any third party can verify it without contacting anyone.
assert agent.verify(receipt)

When an auditor asks "prove this agent executed this action with these parameters," you hand them the receipt and the public key. They verify it offline. No Anthropic console access required. No vendor dependency in the evidence chain.

The Three Gaps

1. Vendor-held vs. self-held evidence

Managed Agents tracing: logs live in Claude Console. Anthropic controls the data.

Signed receipts: cryptographic artifacts you hold locally. No third party in the verification chain.

2. Log integrity vs. cryptographic integrity

Managed Agents: session logs. Accurate under normal conditions. But a log file — even a well-managed one — can be modified. There's no mechanism in a standard log that makes tampering detectable after the fact.

Signed receipts: hash-chained. Tamper with any entry and the chain breaks. Detect deletions. Detect reordering. The integrity guarantee is mathematical, not administrative.

3. Single-party vs. bilateral proof

Managed Agents: Anthropic logs what happened on their infrastructure.

Bilateral signing (Signet v0.4+): the agent signs the request, the server independently signs the response. One tamper-evident record, two signatures, two trust domains. Rewriting the chain requires compromising both keys on separate machines.

What Managed Agents Does Well

To be clear about what this is not: this is not a criticism of Managed Agents as a product.

For developers building Claude-based agents who need to go to production quickly, Managed Agents is a compelling offer. Sandboxing, authentication, session persistence, scoped permissions, multi-agent coordination — real infrastructure problems, solved. The tracing in Console is useful for development and operational debugging.

The gaps above only matter in specific contexts:

Regulated industries (finance, healthcare, legal) where audit evidence must be third-party verifiable
Incident response and forensics where evidence integrity must be demonstrable
Enterprise compliance where "trust the vendor" isn't an accepted audit answer
Cross-vendor or multi-agent workflows where a single vendor doesn't control the full chain

For consumer applications, hobby projects, or internal tools where you trust Anthropic and compliance requirements are light: Managed Agents tracing is probably sufficient.

The Complementary Stack

Managed Agents and signed audit trails aren't competitors. They operate at different layers.

Managed Agents handles: infrastructure, sandboxing, session management, permission scoping, operational tracing.

Signed receipts handle: cryptographic proof of what happened, independently verifiable by any third party, held by you, not a vendor.

Signet works with Managed Agents. Claude Managed Agents uses MCP to connect to external tools — Signet's @signet-auth/mcp intercepts at the MCP transport layer and signs every tool call before it executes. The two layers stack.

Claude Managed Agents
  └── MCP tool calls
        └── Signet SigningTransport  ← signs here
              └── your tool server

The Console shows you what happened. The signed receipts prove it.

The Bottom Line

Claude Managed Agents ships a real, useful tracing feature. If you're using it, your debugging workflow just got better.

But "Anthropic recorded it" and "you can prove it" are different claims. In the situations where audit trails matter most — compliance, incident response, regulated industries — the difference is significant.

Signing is the layer that converts logs into evidence.

Signet adds Ed25519 signing and tamper-evident audit chains to AI agent tool calls. Works with Claude Managed Agents, LangChain, CrewAI, AutoGen, and 7 other frameworks. Apache-2.0 + MIT.

Now on the official Claude Code plugin marketplace: /plugin install signet@claude-plugins-official

AI Agents Can Move Money But Can't Produce Receipts

willamhou — Tue, 14 Apr 2026 03:12:41 +0000

AI Agents Can Move Money But Can't Produce Receipts

In March 2026, security researchers disclosed ZombieClaw — a botnet recruiting compromised AI agent instances. Over 30,000 instances were found exposed with default configurations. Reported losses reached up to $16 million in cryptocurrency. Hundreds of malicious skills were distributed through ClawHub (341 initially identified by Koi, with more found by VirusTotal).

Kaspersky found 512 vulnerabilities, eight critical. Bitdefender, VirusTotal, Sophos, and Oasis Security all published analyses.

But here's what nobody is talking about: after the attack, there is no cryptographic proof of what any compromised agent actually did.

No signed records. No tamper-evident logs. No way to distinguish "the agent executed transfer_eth() because the user asked" from "the agent executed transfer_eth() because a prompt injection rewrote its instructions."

The text logs exist, sure. But text logs can be edited, deleted, or fabricated. When $16M is missing, "trust the logs" is not a forensic standard.

The Forensics Problem

When a traditional server gets compromised, incident response teams have tools: immutable audit logs, signed system events, chain-of-custody protocols. When an AI agent gets compromised, you have:

Conversation history — stored by the agent itself. The compromised agent can edit its own history.
Tool call logs — if they exist at all, they're unsigned text files. An attacker who controls the agent controls the logs.
"The agent did it" — not enough for insurance claims, compliance reports, or criminal prosecution.

ZombieClaw exploited this gap perfectly. The attackers didn't just steal money — they operated in an environment where there is no verifiable evidence of what happened.

Why This Matters Beyond ZombieClaw

The AI agent security conversation focuses on prevention: sandboxing, permission systems, policy engines, skill auditing. These are important. But prevention has a 100% failure rate over time. Every system eventually gets breached.

What happens after?

Without cryptographic proof of agent actions, you can't answer:

Which agent initiated the transaction?
Were the parameters what the user actually approved?
When exactly did the compromise begin?
Was this agent's audit log tampered with after the fact?

SOC 2, HIPAA, and GDPR all require audit trails for actions on sensitive data. "The AI agent did it and we have no verifiable records" creates real gaps in compliance posture.

What a Signed Audit Trail Would Have Changed

If every tool call had been cryptographically signed at execution time, the ZombieClaw investigation would look different:

Before compromise: Signed receipts establish a baseline. Each agent has an Ed25519 identity. Every tool call is signed with the agent's key, timestamped, and chained into a tamper-evident log. The hash chain means you can't delete or reorder entries without breaking the chain.

During compromise: The attacker takes control of the agent. If the attacker uses the agent's existing key, every malicious action is still signed — you have a record of what was executed and when. If the attacker generates a new key, the signing identity changes — the anomaly is visible in the chain.

After compromise: Forensics teams can verify the entire chain offline. They can see which actions were signed by the legitimate agent key vs. an unknown key. They can narrow down when the signing identity changed. They can verify that the log hasn't been modified after the fact.

None of this is possible with unsigned text logs.

What This Doesn't Solve

Signing is not prevention. A signed receipt that says "agent transferred 50 ETH to attacker's wallet" doesn't stop the transfer — it proves it happened.

A signed audit trail doesn't solve:

Malicious skills — A signed record of a malicious skill executing is evidence, not a defense.
Prompt injection — The agent was tricked, not unauthorized. The signature is valid because the agent really did execute the call.
Key compromise — If the attacker steals the signing key, they can sign anything. Bilateral co-signing (where the server independently signs the receipt) mitigates this by requiring two keys from two trust domains.
User intent — A signed receipt proves the agent executed the call, not that the user wanted it.
Full host compromise — If the attacker owns the entire machine, they control the key and the log. Off-host anchoring (publishing chain hashes externally) is the mitigation, but it's not free.

Signing is the forensics layer. You still need sandboxing, permission systems, and skill auditing for prevention. But when prevention fails — and it will — you need evidence.

The Gap in Current Tools

As of April 2026, most major AI agent frameworks have no cryptographic signing on tool call records:

Category	Examples	Typical audit mechanism	Signed?
General-purpose agents	OpenClaw, Hermes Agent	Conversation logs, SQLite	No
Agent OS	OpenFang	SHA-256 hash chain	Hash only, no signatures
Orchestration frameworks	LangChain, CrewAI	Callbacks, event logs	No

OpenFang is the closest — they have a hash chain, which detects casual tampering. But without signatures, an attacker with database access can rewrite the entire chain and it still validates.

What Can You Do Today

If you're running AI agents in production:

Sign every tool call. Give each agent an Ed25519 identity and sign every action. Signet does this as a library — pip install signet-auth or npm install @signet-auth/core.
Chain signed receipts. Individual signatures are good. A hash-chained log of signed receipts is better — deletion and reordering become detectable.
Use bilateral signing when possible. Agent signs the request, server signs the response. Now rewriting the chain requires compromising both keys on different machines.
Export chain hashes off-host. Periodically publish the tip hash to an external system (git commit, append-only cloud storage, even a tweet). This anchors the chain against full-host compromise.
Treat audit integrity as a security requirement, not a feature. If your agent can move money, it needs signed receipts. Period.

The Uncomfortable Truth

AI agents can move money, execute code, and access credentials. Most still can't produce a receipt.

The next ZombieClaw is coming. The question is whether you'll have evidence when it happens.

Signet adds Ed25519 signing and tamper-evident audit logs to AI agent tool calls. Open source, Apache-2.0 + MIT.

Your MCP Server Has No Audit Trail — A Security Checklist

willamhou — Mon, 13 Apr 2026 06:24:46 +0000

Your MCP Server Has No Audit Trail — A Security Checklist

Last month, an AI agent mass-deleted a production environment. The team spent 3 days piecing together what happened — stderr logs, partial timestamps, no proof of which agent or what parameters. No audit trail.

This isn't rare. Amazon Kiro deleted a prod environment. Replit's agent dropped a live database. Supabase MCP leaked tokens via prompt injection. In every case: zero cryptographic evidence of what happened.

MCP is becoming the standard for agent-tool communication. Claude Code, Cursor, Windsurf, and dozens of tools use it. But the MCP spec ships with:

❌ No request signing
❌ No audit log
❌ No caller identity verification
❌ No replay protection
❌ No parameter integrity checks

Your MCP server accepts any request from any process, trusts it completely, and keeps no verifiable record. Here's a practical checklist to fix that.

The Threat Model

Before the checklist, understand what you're defending against:

Attack	How it works	Impact
Parameter tampering	Agent sends `create_issue("fix bug")`, something in the pipeline changes it to `delete_repo("production")`	Data loss
Replay	Legitimate `deploy_to_prod` captured and replayed 50 times	Repeated side effects
Impersonation	Rogue process sends requests claiming to be your trusted agent	Unauthorized actions
Cross-server forwarding	Request intended for staging gets forwarded to production	Wrong environment
Log tampering	Text logs edited after an incident to cover tracks	No incident response
Compliance gap	SOC 2 / HIPAA / GDPR require audit trails; "the AI did it" is not sufficient	Regulatory risk

Checklist

✅ 1. Use TLS for HTTP transports

If your MCP server uses HTTP (SSE or Streamable HTTP), always terminate TLS. This protects data in transit but does not protect against:

Compromised clients sending bad requests
Replay attacks (TLS protects the pipe, not the message)
Log tampering after the fact

For stdio transports (most local MCP servers), TLS doesn't apply — the attack surface is different (any local process can connect).

# nginx example
location /mcp {
    proxy_pass http://localhost:3001;
    proxy_set_header X-Forwarded-For $remote_addr;
}

Covers: Data in transit.

Doesn't cover: Request integrity, identity, audit.

✅ 2. Validate inputs at the boundary

Every tool handler should validate its arguments. MCP passes arbitrary JSON — treat it like user input.

server.setRequestHandler(CallToolRequestSchema, async (request) => {
  const { name, arguments: args } = request.params;

  if (name === "create_issue") {
    if (typeof args?.title !== "string" || args.title.length > 200) {
      return { content: [{ type: "text", text: "Invalid title" }], isError: true };
    }
    // proceed...
  }
});

Use Zod or similar for runtime validation. Never trust args blindly.

Covers: Malformed input, injection.

Doesn't cover: Who sent it, whether it's a replay, audit trail.

✅ 3. Add authentication (API keys or mTLS)

For HTTP transports, require an API key or use mutual TLS:

// Simple API key check
server.setRequestHandler(CallToolRequestSchema, async (request, extra) => {
  const apiKey = extra.requestHeaders?.["x-api-key"];
  if (apiKey !== process.env.MCP_API_KEY) {
    return { content: [{ type: "text", text: "Unauthorized" }], isError: true };
  }
  // proceed...
});

For stdio, authentication is harder — any local process with access to the pipe can send requests. This is where cryptographic signing becomes necessary.

Covers: Unauthorized callers (HTTP only).

Doesn't cover: Parameter integrity, replay, stdio auth, audit trail.

✅ 4. Sign every request with cryptographic receipts

This is the gap most MCP servers don't address. Signing binds a request to a specific agent identity and makes tampering detectable.

Signet adds Ed25519 signing to MCP. A signed receipt:

{
  "v": 1,
  "action": {
    "tool": "create_issue",
    "params_hash": "sha256:b878192...",
    "target": "mcp://github.local"
  },
  "signer": {
    "pubkey": "ed25519:0CRkURt/tc6r...",
    "name": "deploy-bot"
  },
  "ts": "2026-04-09T10:30:00.000Z",
  "nonce": "rnd_dcd4e13579...",
  "sig": "ed25519:6KUohbnS..."
}

Tamper with any field → signature fails. Replay → nonce rejected.

Client side — sign every tool call:

import { SigningTransport } from "@signet-auth/mcp";

const inner = new StdioClientTransport({ command: "my-mcp-server" });
const transport = new SigningTransport(inner, secretKey, "my-agent");
// Every tools/call now carries a signed receipt in params._meta._signet

The receipt is injected into _meta._signet. MCP servers ignore unknown fields by spec — zero server changes needed to start signing. Works with stdio and HTTP.

Server side — verify incoming signatures:

import { verifyRequest, NonceCache } from "@signet-auth/mcp-server";

const nonceCache = new NonceCache();

server.setRequestHandler(CallToolRequestSchema, async (request) => {
  const result = verifyRequest(request, {
    trustedKeys: ["ed25519:..."],       // allowed agent keys
    expectedTarget: "mcp://my-server",  // anti-forwarding
    maxAge: 300,                        // 5-min freshness window
    nonceCache,                         // replay protection
  });

  if (!result.ok) {
    return { content: [{ type: "text", text: result.error }], isError: true };
  }

  // Verified: signature valid, signer trusted, fresh, correct target
  // proceed with tool execution...
});

In ~50 microseconds, this checks: signature validity, signer trust, freshness, target binding, tool/params integrity, and nonce uniqueness.

Python (works with LangChain, CrewAI, AutoGen, or standalone):

from signet_auth import SigningAgent

agent = SigningAgent.create("my-agent", owner="devops-team")
receipt = agent.sign("create_issue", params={"title": "fix bug"})
assert agent.verify(receipt)

Covers: Identity, parameter integrity, replay, freshness, target binding.

Doesn't cover: Preventing the action (signing is attestation, not policy).

✅ 5. Keep a tamper-evident audit log

Signing individual requests is good. Chaining them into a tamper-evident log is better. If someone deletes or reorders records, the chain breaks.

Signet does this automatically — every signed receipt is appended to a SHA-256 hash-chained JSONL log at ~/.signet/audit/:

record_1: { receipt, prev_hash: "sha256:0000...", record_hash: "sha256:abc1..." }
record_2: { receipt, prev_hash: "sha256:abc1...", record_hash: "sha256:def2..." }
record_3: { receipt, prev_hash: "sha256:def2...", record_hash: "sha256:ghi3..." }

Query and verify from the CLI:

signet audit --since 24h              # what happened today
signet audit --tool github --since 7d # github calls this week
signet audit --verify                 # verify all signatures
signet verify --chain                 # check hash chain integrity

Or from Python:

for record in agent.audit_query(since="24h"):
    print(f"{record.receipt.ts}  {record.receipt.action.tool}")

chain = agent.audit_verify_chain()
assert chain.valid

Covers: Tamper detection, incident forensics, compliance audit.

Doesn't cover: Tamper proof (someone with disk access can delete the entire log; off-host anchoring is on the roadmap).

✅ 6. Implement rate limiting and timeouts

Even with signing, a compromised agent can flood your server. Add rate limits:

const callCounts = new Map<string, number>();

server.setRequestHandler(CallToolRequestSchema, async (request) => {
  const signer = request.params.arguments?._meta?._signet?.signer?.name ?? "unknown";
  const count = (callCounts.get(signer) ?? 0) + 1;
  callCounts.set(signer, count);

  if (count > 100) {  // per-agent limit
    return { content: [{ type: "text", text: "Rate limit exceeded" }], isError: true };
  }

  // proceed...
});

And always set timeouts on tool execution:

const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 30_000);
try {
  const result = await executeTool(args, { signal: controller.signal });
} finally {
  clearTimeout(timeout);
}

✅ 7. Principle of least privilege

Don't give your MCP server access to everything. Run it with minimal permissions:

Separate API keys per tool (read-only key for list_issues, write key for create_issue)
Filesystem access scoped to specific directories
Database user with only the required grants
Network egress limited to required endpoints

This is independent of MCP — it's basic defense-in-depth.

Summary

#	Practice	Protects against	Difficulty
1	TLS	Eavesdropping	Easy
2	Input validation	Injection, malformed data	Easy
3	Authentication	Unauthorized callers	Medium
4	Request signing	Tampering, replay, impersonation	3 lines
5	Audit log	Incident response, compliance	Automatic with signing
6	Rate limiting	Denial of service	Easy
7	Least privilege	Blast radius	Medium

Most MCP servers today implement 1-3 at best. Steps 4 and 5 — signing and audit — are the gap. They're also the hardest to bolt on after the fact, which is why starting with a library that handles both is worth the npm install.

Get Started

npm install @signet-auth/core @signet-auth/mcp
# or
pip install signet-auth

GitHub: github.com/Prismer-AI/signet

Apache-2.0 + MIT dual licensed. Open source, no SaaS, no phone-home.

If your AI agent can delete a database, you should be able to prove it did.

Every Trending AI Agent Project Is Reinventing Something Humans Already Built

willamhou — Sat, 11 Apr 2026 05:44:30 +0000

Every Trending AI Agent Project Is Reinventing Something Humans Already Built

I've been watching GitHub Trending for the past six months. The same category of project keeps appearing: infrastructure for AI agents.

Look closely and they're all doing the same thing. Taking organizational structures that humans have used for centuries and rebuilding them for agents.

The Pattern

Human world	Agent world	Example project
Workflow manuals / API standards	Tool-calling protocol	MCP (Anthropic)
Team collaboration / org charts	Multi-agent orchestration	CrewAI, AutoGen, LangGraph
Security policies / firewalls	Agent behavior control	Aegis, Invariant
Contracts / signatures / audit	?	?

The first three rows are well covered. Multiple teams shipping production-grade tools.

The last row is empty.

The Oldest Infrastructure in Human Society

Human civilization has run on one mechanism from Mesopotamian clay tablets to DocuSign: signatures.

Sign a contract. Sign for a delivery. Sign an expense report. Sign a bank transfer. When something goes wrong and you end up in court, the first question is always: "Is there a signature?"

Signatures don't solve trust. They solve proof. Not "I trust you," but "you did this, here is the evidence, you cannot deny it."

Now look at the agent world.

Your agent calls dozens of APIs every day. Creates issues. Sends messages. Places orders. Modifies configurations. What did it actually do?

No signature. No receipt. No evidence.

MCP, the protocol agents use to talk to tools, has no signing mechanism. Your agent acts with your credentials, your API keys, on your behalf. When something goes wrong, you can't even prove whether your agent did it.

What Happens Without Signatures

You run a multi-agent pipeline. An orchestrator delegates to a research agent, a writing agent, a review agent. The final report contains incorrect data. Which agent introduced it? When? With what parameters? Without signatures, you trace manually.

Security asks you a question. "What did your agent do last week? Which services did it call?" You open the MCP server access log. HTTP requests. No way to distinguish human operations from agent operations. You have no answer.

Your agent placed 10 purchase orders. One has an abnormal amount. You want to prove that transaction wasn't authorized by you. But you can't produce cryptographic evidence. The agent used your key. In the logs, it looks like you did it.

The Gap Is Real

This isn't a hypothetical concern. It's already happening. In the MCP spec discussion (SEP-1763), five independent projects identified the layers needed for an agent enforcement stack:

Layer	What it answers
Identity	"Who is this agent?"
Policy	"What is it allowed to do?"
Transport integrity	"Can you prove what was actually sent and received?"
Spend control	"How much can it spend?"
Output verification	"Is the response correct?"

The transport integrity layer, the signing layer, was empty until recently.

Signet: Signing for Agents

I built Signet to fill that row.

Every agent gets an Ed25519 identity. Every tool call gets a signed receipt. Receipts chain into a tamper-evident audit log. Servers can verify requests before execution. Both sides can co-sign a bilateral record.

from signet_auth import SigningAgent

agent = SigningAgent.create("procurement-bot", owner="ops-team")
receipt = agent.sign("marketplace_purchase",
    params={"item": "GPU-A100", "quantity": 2, "price": 15000})

# What did this agent do in the last 24 hours?
for record in agent.audit_query(since="24h"):
    print(f"{record.receipt.ts}  {record.receipt.action.tool}")

# Was this receipt tampered with?
assert agent.verify(receipt)

v0.6 added delegation chains: a root identity (human or org) cryptographically delegates scoped authority to an agent. The agent's receipts carry proof of who authorized them, what scope was granted, and that permissions can only narrow, never widen.

Owner (alice) → Agent A (tools: [Bash, Read], max_depth: 0)
                    ↓
              v4 Receipt: tool=Bash, params_hash=sha256:...
              authorization: chain proves alice → Agent A

The difference from plain logging: logs record what the system says happened. Signatures prove what actually happened. Logs can be rewritten. Signatures can't be forged without the private key. This isn't a log. It's evidence.

But Is This the Right Abstraction?

Back to the table at the top.

The entire industry is mapping human organizational patterns onto agents. Protocols. Teams. Firewalls. Signatures. Rebuilding them one by one.

There's an implicit assumption here: agent organizational structures should look like human ones.

Do agents need "team collaboration"? Or did we build multi-agent frameworks because humans are used to teams, so we instinctively gave agents one too?

Do agents need "signatures"? Or is there a fundamentally different trust mechanism that doesn't rely on signatures and evidence, but on something we haven't thought of yet?

Humans built passports, contracts, audits, and firewalls because humans are not fully trustworthy, have unreliable memory, and sometimes lie. Agents are a different species. Should their trust infrastructure be designed from first principles instead of copied from human patterns?

I don't know the answer.

What I do know: agents are already acting on your behalf, and you can't prove what they did. Whatever the final form of agent trust looks like, "provable" is probably a requirement that doesn't go away.

The bigger question, what agent organizational structures should look like, I'll leave to people smarter than me.

GitHub: github.com/Prismer-AI/signet

Now on the official Claude Code plugin marketplace:

/plugin install signet@claude-plugins-official

Apache-2.0 + MIT. Open source.

If you're thinking about these problems too: @willamhou

Two Hypervisors, One SoC: Replacing Hafnium with 30K Lines of Rust

willamhou — Thu, 09 Apr 2026 02:07:31 +0000

Two Hypervisors, One SoC: Replacing Hafnium with 30K Lines of Rust

Over about 10 weeks, I built a bare-metal SPMC at S-EL2 that boots Linux, manages Secure Partitions, and runs alongside Android pKVM on the same SoC.

I built an ARM64 hypervisor that runs next to Google's pKVM on the same chip. pKVM takes the Normal world at NS-EL2. My hypervisor takes the Secure world at S-EL2. They coordinate through ARM's FF-A protocol, relayed by EL3 firmware. 35 end-to-end tests pass through the full four-level stack: Linux kernel module → pKVM → TF-A → my SPMC → Secure Partitions → and back.

The Secure side already had an implementation: Hafnium, Google's reference SPMC. It's 200K+ lines of C. I replaced it with 30,000 lines of no_std Rust — no runtime, no allocator crate, one dependency (a DTB parser). It boots Linux to a BusyBox shell, manages three Secure Partitions, and handles FF-A v1.1 messaging and memory sharing.

I'll walk through the architecture, the parts that were genuinely hard, and the four bugs I spent the most time chasing.

ARM's Split Personality

ARM's latest chips divide the CPU into two security worlds. Each world gets its own hypervisor at EL2:

            Normal World          Secure World
           ┌────────────┐       ┌────────────┐
    EL0    │  Userspace  │       │            │
           ├────────────┤       ├────────────┤
    EL1    │ Linux/Android│       │  Secure    │
           │  kernel     │       │  Partitions│
           ├────────────┤       ├────────────┤
    EL2    │  pKVM       │       │  SPMC      │
           │  (NS-EL2)   │       │  (S-EL2)   │
           └──────┬──────┘       └──────┬──────┘
                  │      ┌──────┐       │
    EL3           └──────│ TF-A │───────┘
                         │ SPMD │
                         └──────┘

EL3 is the root of trust — ARM Trusted Firmware (TF-A) lives here and relays messages between worlds via SMC (Secure Monitor Call). The protocol is FF-A v1.1: it defines messaging, memory sharing, page ownership transfer, and partition management. My hypervisor fills the S-EL2 box.

Two Hypervisors, One Chip

This is the part most hypervisor projects don't deal with: coexistence. pKVM and my SPMC boot on the same 4 physical CPUs, each managing their own world. The boot chain:

TF-A BL1 (ROM) → BL2 (loader) → BL31 (SPMD at EL3)
    → BL32 (our SPMC at S-EL2, boots SP1/SP2/SP3)
    → BL33 (pKVM at NS-EL2 → Linux at NS-EL1)

When pKVM's Linux guest wants to talk to a Secure Partition, the message crosses four exception levels and two world switches:

Linux (NS-EL1) → SMC → pKVM (NS-EL2) → SMC → SPMD (EL3)
    → ERET → SPMC (S-EL2) → ERET → SP1 (S-EL1)
    → SMC → SPMC → SMC → SPMD → ERET → pKVM → ERET → Linux

The proof: Linux sends x4=0xBBBB via FF-A DIRECT_REQ, SP1 adds 0x1000, Linux reads back 0xCBBB. One round trip, four privilege levels, two world switches.

Making this work meant dealing with problems that mostly don't show up in a single-hypervisor setup:

SPMD is per-CPU. TF-A's Secure Partition Manager Dispatcher maintains separate state for each physical CPU. When pKVM boots secondary CPUs via PSCI, each one enters S-EL2 on whichever physical core it lands on. My SPMC must register a secondary entry point (FFA_SECONDARY_EP_REGISTER), allocate per-CPU stacks (3 × 32KB), and run a full event loop on every core. If any CPU skips its FFA_MSG_WAIT handshake, SPMD blocks the entire PSCI boot sequence. This is documented nowhere except TF-A's source code.

S-EL2 Stage-1 MMU and the NS bit. The Secure world has its own physical address space. When S-EL2 writes to address 0x42a16000 with the MMU off, it hits the Secure alias. pKVM's RX buffer is at the same address in the Non-Secure alias. Different memory. I had to enable an S-EL2 Stage-1 identity map where all Normal world DRAM is marked NS=1 to force writes to the correct alias. (More on this in War Stories.)

Cross-CPU cache coherency. pKVM writes a descriptor to its TX buffer on CPU 0, then issues an SMC. SPMD routes the call to S-EL2 on whichever CPU happens to be running — potentially CPU 2 with a stale L1 cache line. Even after adding DSB SY barriers, I had to copy the descriptor to a local stack buffer before parsing it. Reading directly from the cross-world buffer produced data aborts from corrupt pointer arithmetic.

On make run-pkvm-ffa-test, the full TF-A boot chain comes up, then pKVM initializes, and our kernel module exercises every FF-A path:

[SPMC] SP1 booted, now Idle (FFA_MSG_WAIT received)
[SPMC] SP2 booted, now Idle (FFA_MSG_WAIT received)
[SPMC] SP3 booted, now Idle (FFA_MSG_WAIT received)
[SPMC] Secondary EP registered with SPMD
...
Protected hVHE mode initialized successfully
...
ffa_test: Sending DIRECT_REQ to SP 0x8001...
ffa_test:   x3=0xaaaa x4=0xcbbb x5=0xcccc x6=0xdddd x7=0xeeee
ffa_test: [PASS] DIRECT_REQ to SP 0x8001 returns success
ffa_test: [PASS] SP 0x8001 x4 = 0xBBBB + 0x1000
...
ffa_test: [PASS] Shared page == 0xCAFEFACE (SP wrote it)
ffa_test: [PASS] MEM_RECLAIM returns success
...
ffa_test: [PASS] SP1→SP3 relay chain returns success
ffa_test: [PASS] SP1→SP2 Secure DRAM share verified
ffa_test:   Results: 35/35 PASS

Rust at Exception Level 2

Secure Partition lifecycle is a state machine: Reset → Idle → Running → Blocked → Preempted. In C, this would probably be an integer plus a set of invariants everyone has to remember. In Rust:

enum SpState { Reset, Idle, Running, Blocked, Preempted }

When I added the Blocked → Preempted edge for chain preemption during SP-to-SP messaging, the compiler forced me to revisit every transition. That flushed out two bugs before I ever ran the code.

My Cargo.toml has one dependency: fdt = "0.1.5". Everything else — page tables, GIC emulation, virtio drivers, the SPMC event loop — is hand-written. The alloc crate gives me Box and Vec backed by a bump allocator. Enum dispatch replaces trait objects for zero-cost MMIO routing.

Technical Highlights

Stage-2 Page Table Tricks

ARM's Stage-2 translation maps guest physical addresses to real physical addresses. I use identity mapping but repurpose the software-defined PTE bits for ownership tracking:

PTE bits [56:55]:
  00 = Owned          (page belongs to this VM)
  01 = SharedOwned    (shared out, sender retains ownership)
  10 = SharedBorrowed (mapped from another VM/SP)
  11 = Donated        (irrevocably transferred)

This mirrors pKVM's model. When VM 0 shares a page with SP1: validate ownership (SW bits = 00), set to SharedOwned (01) + read-only, map into SP1's Secure Stage-2 as SharedBorrowed (10). On reclaim: validate SP1 has relinquished, restore to Owned + read-write.

The Stage-2 walker reconstructs itself from VTTBR_EL2 at SMC handling time — it walks and modifies PTEs without owning the page table memory. The SPMC can manipulate any VM's page tables by just knowing the L0 table physical address.

SP-to-SP Messaging and Cycle Detection

Secure Partitions can message each other. SP1 sends a DIRECT_REQ to SP3, which forwards to SP2, which responds. The SPMC routes each hop:

NWd → SP1 runs → DIRECT_REQ(SP3) → SP3 runs
    → DIRECT_RESP(SP1) → SP1 resumes → DIRECT_RESP(NWd)

Each SP making an outgoing call transitions from Running to Blocked. The SPMC maintains a CallStack and checks for cycles: SP1 → SP3 → SP1 returns FFA_BUSY. Without this, deadlock.

The tricky part is preemption. A Normal world interrupt arrives while SP3 is running mid-chain. The SPMC transitions SP3 from Running to Preempted, SP1 from Blocked to Preempted (chain preemption), and returns FFA_INTERRUPT. When the Normal world later calls FFA_RUN, the entire chain resumes.

The `handle_sp_exit()` Loop

This is the heart of the SPMC. When the SPMC dispatches to an SP, the SP runs until it traps — but the trap might not be a response. It could be a memory operation, a log message, or a call to another SP.

loop {
    enter_guest();  // ERET to S-EL1
    let exit = decode_exit();
    match exit {
        FFA_MSG_SEND_DIRECT_RESP => return response,
        FFA_MEM_RETRIEVE_REQ    => { handle locally; re-enter SP },
        FFA_MEM_RELINQUISH      => { handle locally; re-enter SP },
        FFA_MEM_SHARE           => { record share; re-enter SP },
        FFA_CONSOLE_LOG         => { print to UART; re-enter SP },
        FFA_MSG_SEND_DIRECT_REQ => { dispatch to target SP; re-enter },
        _ => return error,
    }
}

The SP doesn't know its RETRIEVE_REQ is handled locally rather than going to another entity. It does an SMC, gets a result, and continues. This is what makes E2E memory sharing work: the Normal world shares a page, SP1 retrieves it (in-loop), writes 0xCAFEFACE, relinquishes (in-loop), and responds — all within a single dispatch.

War Stories

The Silent SIMD Trap

Week 4. The SPMC boots fine in release mode but hangs on the first read_volatile in debug. No output, no fault, nothing.

After a few hours with GDB, I found the CPU stuck in an EL3 exception handler. ESR showed an FP/SIMD trap. But my code doesn't use floating point.

Rust's debug-mode codegen will happily emit NEON instructions for things that look unrelated. In my case, the alignment check inside read_volatile compiled to cnt v0.8b, v0.8b — a SIMD population count. TF-A's default CPTR_EL3.TFP=1 traps all floating-point and SIMD from every exception level. EL3's handler wasn't prepared for that trap, so it looped forever.

What fixed it was one build flag: CTX_INCLUDE_FPREGS=1. It was a good reminder that once you're running below an OS, your compiler's codegen is part of the hardware contract.

The NS Bit and the Invisible Write

Week 8. PARTITION_INFO_GET works perfectly from our BL33 test harness. The SPMC writes SP descriptors to the caller's RX buffer, caller reads them back. 24 bytes per partition, everything checks out.

Then pKVM calls the same function. Same code path, same descriptor format. pKVM reads... all zeros.

The write succeeded (no fault). The address was correct (verified in GDB). But the data wasn't there.

ARM has two physical address spaces. When S-EL2 runs with the MMU off, all memory accesses go through the Secure physical address space. pKVM's buffer is at 0x42a16000 in Non-Secure DRAM. The write hits 0x42a16000 Secure. pKVM reads from 0x42a16000 Non-Secure. Different memory.

What fixed it was enabling an S-EL2 Stage-1 MMU with an identity map where all Normal world DRAM has the NS=1 attribute bit. I've worked with ARM for years and still hadn't fully internalized that Secure/Non-Secure is a physical address space split, not just a permission model. In QEMU, there's literally twice the memory at the same addresses, selected by one bit.

The Stale Cache and the Phantom Data Abort

Week 11. pKVM's MEM_SHARE works 70% of the time. The other 30%, the SPMC crashes with a Data Abort at a pointer address like 0x240f — clearly not a valid physical address.

addr2line traced it to parse_mem_region in my descriptor parser. The descriptor's composite_offset field, which should be 80, was reading as garbage. The SPMC was dereferencing base + garbage and faulting.

The descriptor lived in pKVM's TX buffer — Normal world DRAM. pKVM writes it on CPU 0, issues an SMC, SPMD context-switches to S-EL2 on CPU 2. Even though ARM's memory model guarantees the SMC acts as a barrier for the issuing CPU, the receiving CPU might still have a stale L1 cache line.

I first added DSB SY (Data Synchronization Barrier, full system scope) before every cross-world buffer read. It still crashed. The barrier improves visibility, but the buffer itself is in Non-Secure DRAM that the SPMC accesses through the NS=1 Stage-1 mapping. From the SPMC's point of view, that was still not enough to make the parse reliable.

What finally made it reliable was copying the entire descriptor to a local stack buffer before parsing it.

unsafe { core::arch::asm!("dsb sy", options(nostack, nomem)); }
let mut local_buf = [0u8; 4096];
unsafe {
    core::ptr::copy_nonoverlapping(
        tx_pa as *const u8, local_buf.as_mut_ptr(), total_length,
    );
}
// Parse from local_buf, never from the shared buffer
let parsed = parse_mem_region(local_buf.as_ptr(), total_length);

Now, if the copy still captures stale data, the bounds checks in parse_mem_region reject it cleanly instead of chasing a wild pointer into Secure memory. In practice that took the crash rate from about 30% to zero.

SPMD Is Per-CPU (or: Read the Firmware Source)

Week 7. pKVM boots fine on CPU 0. Secondary CPUs hang.

The FF-A spec describes SPMC init but says almost nothing about secondary CPUs. After reading TF-A's spmd_cpu_on_finish_handler(), I found it: SPMD maintains entirely separate state per physical CPU. Each secondary entering S-EL2 must call FFA_MSG_WAIT — a handshake that signals "this CPU's Secure world is ready." Without it, SPMD never completes the PSCI CPU_ON call, so the Normal world secondary never boots either.

My initial code had secondary CPUs do WFE (wait for event) after basic init. That's the Normal world pattern. But SPMD needs its per-CPU handshake, per-CPU stacks (3 × 32KB in .bss), and a full event loop on each secondary. The eventual fix was registering FFA_SECONDARY_EP_REGISTER during init and giving each secondary its own stack and event loop. The FF-A spec tells you what has to happen; TF-A's source code is where I found how it actually has to be wired up.

Testing Without an OS

All tests run on bare metal. No test harness, no OS, no #[test]. The binary calls each test suite sequentially, printing [PASS] or [FAIL] to UART.

For integration tests, the BL33 binary is a 500-line assembly program that sends 20 FF-A calls through real TF-A firmware and validates each response:

  Test 1: FFA_VERSION .............. PASS
  Test 2: FFA_ID_GET ............... PASS
  ...
  Test 13: MEM_SHARE lifecycle E2E . PASS
  Test 14: Alternating SP1/SP2 ..... PASS
  Test 17: SP->SP relay chain ...... PASS
  Test 18: Cycle detection ......... PASS
  Test 19: SP-to-SP MEM_SHARE ...... PASS
  Test 20: SP-to-SP MEM_RECLAIM .... PASS

  All tests complete.

For pKVM E2E tests, ffa_test.ko is a Linux kernel module that does the same through pKVM's FF-A proxy.

There's no mocking here. The BL33 tests go through real TF-A at EL3. The pKVM tests traverse pKVM at NS-EL2, SPMD at EL3, our SPMC at S-EL2, and SPs at S-EL1. If any layer is broken, the test fails.

Numbers

Metric	Value
Rust source	26,000 lines (96 files)
ARM64 assembly	3,400 lines (9 files)
Unit test assertions	457
BL33 integration tests	20/20
pKVM E2E tests	35/35
Dependencies	1 (`fdt` crate)
Dev time	~10 weeks (solo)
Binary size	230KB (release, SPMC)

What's Next

The big remaining piece is ARM's Realm Management Extension (RME) — the "R" in ARM CCA. RME adds a fourth world (Realm) with hardware-enforced memory isolation. A Realm VM's memory is inaccessible to both the Normal world hypervisor and the Secure world firmware.

The SPMC infrastructure (Stage-2 management, FF-A messaging, multi-CPU dispatch) provides a solid foundation, but RME requires Granule Protection Tables at EL3, a Realm Management Interface at EL2, and guest attestation. Significant step up.

Try It

git clone https://github.com/willamhou/hypervisor
cd hypervisor
make run          # 34 test suites, ~5 seconds on QEMU
make run-linux    # boots Linux 6.12 to shell

*Blog version · GitHub

For make run-spmc and make run-pkvm-ffa-test, you'll need TF-A and (for pKVM) the AOSP kernel — both build via Docker. The full build takes ~30 minutes the first time. See the README for details.

Built with Rust nightly, QEMU 9.2, and a lot of time spent cross-checking the ARM ARM.

Is that MCP request actually from your AI agent

willamhou — Wed, 08 Apr 2026 10:46:39 +0000

Is that MCP request actually from your AI agent?

Last week we open-sourced Signet — cryptographic signing for every AI agent tool call.

Someone asked a good question: the agent signs, but does the server verify?

Fair point. v0.1 was one-sided. The agent signed every request, but the server didn't check. Like mailing a signed contract that nobody verifies on the other end. Better than nothing, but the trust chain is broken.

This week we closed the loop.

From one-sided to bilateral

v0.1  Agent signs (one-sided)
      → proves the agent sent the request

v0.2  Compound receipt (request + response bound)
      → proves the request and response are paired

v0.3  Server verification
      → server can verify "this request was signed by a specific agent"

v0.4  Bilateral co-signing
      → agent signs the request, server signs the response, both hold proof

Server verification: 3 lines

Add this to your MCP server handler:

import { verifyRequest } from "@signet-auth/mcp-server";

server.setRequestHandler(CallToolRequestSchema, async (request) => {
  const v = verifyRequest(request, {
    trustedKeys: ["ed25519:..."],
    maxAge: 300,
  });
  if (!v.ok) return { content: [{ type: "text", text: v.error }], isError: true };

  // verified — proceed
  console.log(`Request from: ${v.signerName}`);
});

One function. Six checks: signature validity, trusted key list, freshness window (anti-replay), target binding (anti-cross-server), tool name match, params match. ~50μs.

Bilateral co-signing: 3 more lines

After verifying and processing the request, the server signs the response:

import { signResponse } from "@signet-auth/mcp-server";

const bilateral = signResponse(request, response, {
  serverKey: process.env.SIGNET_SERVER_KEY,
  serverName: "my-server",
});

This produces a BilateralReceipt:

{
  "v": 3,
  "agent_receipt": {
    "v": 1,
    "action": { "tool": "create_issue", "params_hash": "sha256:..." },
    "signer": { "pubkey": "ed25519:...", "name": "deploy-bot" },
    "sig": "ed25519:..."
  },
  "response": { "content_hash": "sha256:..." },
  "server": { "pubkey": "ed25519:...", "name": "github-mcp" },
  "ts_response": "...",
  "sig": "ed25519:..."
}

Two signatures, two keys, one record. The agent says "I sent this request." The server says "I received this request and returned this response." Both parties hold proof. Neither can deny it.

Why bilateral matters

One-sided signing proves "what the agent did." Bilateral signing proves "what happened between the agent and the server."

The difference:

Scenario: an agent places a purchase order, but the amount is wrong.

One-sided: the agent says "I sent price=100." But the server might have received price=10000. You can't prove what happened in between.

Bilateral: the agent's signature binds the params_hash. The server's signature binds the response content_hash. Any tampering in between breaks one of the signatures. Both parties signed the same facts — when something goes wrong, the signatures tell you who's right.

Python support

from signet_auth import SigningAgent

agent = SigningAgent.create("my-agent", owner="willamhou")
receipt = agent.sign("create_issue", params={"title": "fix bug"})

# Verify a bilateral receipt
from signet_auth import verify_bilateral
result = verify_bilateral(bilateral_receipt_json, agent_pubkey, server_pubkey)

Where this fits in the MCP ecosystem

In the MCP spec discussion (SEP-1763), five independent projects were identified as layers of an enforcement stack:

Layer	Project	What it does
Transport integrity	Signet	Agent signs request, server verifies + co-signs
Policy enforcement	APS	Authorization checks, policy engine
External anchoring	ArkForge	Receipt anchoring to transparency logs
Spend control	AgentPay	Budget caps
Output verification	veroq	Response content validation

Signet went from a one-sided signing tool to the transport layer of a bilateral trust protocol. Each layer does its own thing. Together they form a complete agent security stack.

Try it

Claude Code (on the official Anthropic plugin marketplace):

/plugin install signet@claude-plugins-official

Every tool call is signed and audited. Zero config.

MCP server (on the MCP Registry, Glama indexed):

npm install @signet-auth/mcp-server

Agent side:

# TypeScript
npm install @signet-auth/core @signet-auth/mcp

# Python (LangGraph, LlamaIndex, Pydantic AI, Google ADK, OpenAI Agents, Semantic Kernel, smolagents)
pip install signet-auth

GitHub: github.com/Prismer-AI/signet

@willamhou

How I Built Cryptographic Signing for Every AI Agent Tool Call

willamhou — Thu, 02 Apr 2026 02:45:56 +0000

How I Built Cryptographic Signing for Every AI Agent Tool Call

Your AI agent just mass-deleted a production database. Can you prove exactly what it did? When? Who authorized it?

You can't. None of the major agent frameworks produce cryptographic evidence of what happened.

I built Signet to fix this. It gives every AI agent an Ed25519 identity and signs every tool call with a tamper-evident receipt. Open source, 3 lines of code to integrate.

This post covers the design decisions, the crypto choices, and why I built an SDK instead of a proxy.

The Problem

MCP (Model Context Protocol) is becoming the standard for agent-tool communication. Claude, Cursor, Windsurf, and dozens of other tools use it. But MCP has no signing, no audit log, and no way to prove which agent did what.

Real incidents that motivated this:

Amazon Kiro deleted a production environment
Replit agent dropped a live database
Supabase MCP leaked tokens via prompt injection

In every case: zero audit trail. The agent did something, nobody could prove exactly what.

Design Decisions

SDK, not proxy

The existing tools in this space (Aegis, estoppl) use a proxy/gateway model. You deploy a separate process that sits between your agent and the MCP server, intercepting all traffic.

I chose the opposite: a client-side SDK. Three reasons:

Zero infrastructure. npm install or pip install, not Docker.
stdio-native. Most MCP connections use stdio pipes. Proxying stdio requires process orchestration that adds failure modes.
Framework-agnostic. Works with any MCP client, any transport, any language.

The tradeoff: a proxy can enforce policies (block unauthorized calls). Signet can't — it's a camera, not a bouncer. Attestation, not prevention. I think both are needed, and they're complementary.

Ed25519, not ECDSA or RSA

Ed25519 was the obvious choice:

Fast. ~70,000 signatures/sec on a laptop. Agent tool calls are maybe 10/sec. No bottleneck.
Small. 64-byte signatures, 32-byte keys. Fits in JSON metadata without bloat.
Deterministic. Same key + same message = same signature. No nonce generation needed at signing time (the nonce is in the key schedule).
Battle-tested. SSH, Signal, age, WireGuard all use it.

I use ed25519-dalek in Rust. The same core compiles to WASM for Node.js and to native for Python via PyO3. One implementation, three languages, zero divergence risk.

RFC 8785 (JCS) for canonical JSON

The signature covers the entire receipt body: action, signer, timestamp, nonce. But JSON serialization is non-deterministic — {"a":1,"b":2} and {"b":2,"a":1} are semantically identical but produce different bytes.

I use RFC 8785 (JSON Canonicalization Scheme) to solve this. JCS defines a deterministic JSON serialization: sorted keys, no whitespace, specific number formatting. Sign the JCS output, verify against the JCS output. Deterministic.

Why JCS over alternatives like bencode or CBOR? Because the receipts are JSON, the MCP protocol is JSON, and I didn't want to introduce a second serialization format just for signing.

Hash-chained audit log

Every receipt is appended to a local JSONL audit log. Each entry includes the SHA-256 hash of the previous entry:

record_1: { receipt: ..., prev_hash: "sha256:0000...", record_hash: "sha256:abc1..." }
record_2: { receipt: ..., prev_hash: "sha256:abc1...", record_hash: "sha256:def2..." }
record_3: { receipt: ..., prev_hash: "sha256:def2...", record_hash: "sha256:ghi3..." }

Delete or modify any record and the chain breaks. signet verify --chain detects it.

This is tamper-evident, not tamper-proof. Someone with disk access can delete the entire log. The hash chain catches selective editing (changing one record while keeping others). Off-host anchoring (anchoring chain hashes to an external service) is on the roadmap for true tamper-proof guarantees.

Encrypted key storage

Agent keys are encrypted at rest with Argon2id + XChaCha20-Poly1305:

Argon2id for key derivation (OWASP recommended, memory-hard, resists GPU attacks)
XChaCha20-Poly1305 for encryption (24-byte nonce = safe random nonce generation, AEAD for authenticated encryption)
AAD (Additional Authenticated Data) binds the ciphertext to the key's metadata, preventing key-file swaps

Unencrypted keys are supported for CI/automation (--unencrypted flag). Keys stored at ~/.signet/keys/ with 0600 permissions.

What a Receipt Looks Like

{
  "v": 1,
  "id": "rec_e7039e7e7714e84f...",
  "action": {
    "tool": "github_create_issue",
    "params": {"title": "fix bug", "body": "details"},
    "params_hash": "sha256:b878192252cb...",
    "target": "mcp://github.local",
    "transport": "stdio"
  },
  "signer": {
    "pubkey": "ed25519:0CRkURt/tc6r...",
    "name": "demo-bot",
    "owner": "willamhou"
  },
  "ts": "2026-03-29T23:24:03.309Z",
  "nonce": "rnd_dcd4e135799393...",
  "sig": "ed25519:6KUohbnSmehP..."
}

The signature covers v + action + signer + ts + nonce via JCS. Tamper with any field and verification fails.

The params_hash is always present. By default, raw params are also stored. For sensitive data, --hash-only mode stores only the hash — you can prove what shape the params had without revealing their content.

Integration: 3 Lines

TypeScript (MCP)

import { SigningTransport } from "@signet-auth/mcp";

const inner = new StdioClientTransport({ command: "my-mcp-server" });
const transport = new SigningTransport(inner, secretKey, "my-agent");

Every tools/call request gets a signed receipt injected into params._meta._signet. MCP servers don't need to change — they ignore unknown fields per the spec.

Python — LangChain

from signet_auth import SigningAgent
from signet_auth.langchain import SignetCallbackHandler

agent = SigningAgent.create("my-agent", owner="willamhou")
handler = SignetCallbackHandler(agent)
# Pass handler to your LangChain agent — every tool call is now signed

Python — CrewAI

from signet_auth import SigningAgent
from signet_auth.crewai import install_hooks, uninstall_hooks

agent = SigningAgent.create("my-agent", owner="willamhou")
install_hooks(agent)
crew.kickoff()  # every tool call is now signed
uninstall_hooks()

Python — Low-level API (any framework)

from signet_auth import SigningAgent

agent = SigningAgent.create("my-agent", owner="willamhou")
receipt = agent.sign("github_create_issue", params={"title": "fix bug"})

All Python bindings are Rust compiled via PyO3 — same crypto, same behavior, native speed. pip install signet-auth.

CLI

signet identity generate --name my-agent
signet sign --key my-agent --tool "github_create_issue" --params '{"title":"fix bug"}'
signet audit --since 24h
signet verify --chain

Architecture

signet/
├── crates/signet-core/       Rust core (one implementation)
├── bindings/
│   ├── signet-ts/            → WASM for Node.js
│   └── signet-py/            → PyO3 for Python
├── packages/
│   ├── @signet-auth/core     TypeScript wrapper
│   └── @signet-auth/mcp      MCP middleware
└── signet-cli/               CLI binary

The key architectural decision: one Rust implementation, compiled to multiple targets. The WASM binding and Python binding call the same signet-core code. There is no separate TypeScript or Python reimplementation of the signing logic. This eliminates the "two implementations diverge" class of bugs entirely.

172 tests across Rust (68), Python (85), TypeScript (11), and WASM (8).

What Signet Does NOT Do (Yet)

It proves the agent requested an action, not that the server executed it. The receipt says "agent X signed intent to call tool Y with params Z at time T." Whether the server actually did it is a separate question. Server-side verification middleware is on the roadmap.

It doesn't prevent bad actions. Signet is a camera, not a bouncer. It complements prevention tools like policy engines and firewalls. You want both: stop the bad thing AND have evidence of what happened.

Signer identity is self-asserted. The signer.name and signer.owner fields are set by the agent. There's no external identity registry (yet) to verify that "demo-bot" actually belongs to "willamhou."

Try It

pip install signet-auth
# or
npm install @signet-auth/core @signet-auth/mcp
# or
cargo build --release -p signet-cli

GitHub: github.com/Prismer-AI/signet

Apache-2.0 + MIT dual license.

Questions about the design decisions, crypto choices, or roadmap? I'm @willamhou on Twitter/X.