Forem: Shizuku

Beyond Artifact-Only Evaluation: A Case for Development-Session Attestation (DSA)

Shizuku — Tue, 24 Feb 2026 12:20:10 +0000

Introduction

AI-assisted development has dramatically increased implementation speed.

At the same time, I think it has made one thing harder: relying on artifact-only evaluation as a sufficient signal of engineering work (repositories, portfolio sites, demos, finished UI screenshots, etc.).

Artifacts still matter. Code review, testing, CI, and running systems will remain essential.

The problem is not that artifacts became useless.

The problem is that artifacts alone often do not answer questions like:

How much of the implementation was actually understood?
What tools/processes were executed during development?
Is the development process itself explainable after the fact?

This becomes especially important in situations where implementation output is not the whole story:

incident response and quality explanation
hiring assessments
constrained test environments (e.g., prohibited tools)
training and education contexts

This is the gap I am trying to frame with DSA (Development-Session Attestation):

an evidence-and-verification layer for the development session itself, not just the final output.

If you want to see a concrete implementation first:

GitHub (SessionAttested): https://github.com/shizuku198411/SessionAttested

DSA concept note (in repo): DSA.md

PoC workspace example with WebUI screenshots: attested_poc/README.md

What This Article Is (and Isn’t)

This is not an eBPF/LSM implementation deep dive.

I already have implementation-oriented material around SessionAttested, which is one implementation path toward DSA.

This article is about the higher-level idea:

why artifact-only evaluation is becoming less sufficient
what DSA is trying to add
how it relates to existing auditing/provenance approaches
how SessionAttested fits into that picture

What Is Changing in Artifact-Centric Evaluation

Artifact-centric evaluation is not “wrong.” It is simply becoming less complete in some scenarios.

1) Portfolio artifacts are becoming easier to produce without corresponding understanding

With modern AI tooling and developer assistants, it is increasingly possible to produce polished outputs without deep implementation understanding.

That does not mean the artifact is fake.

It means the artifact alone may no longer be enough to evaluate:

implementation comprehension
engineering judgment
ability to explain trade-offs and failure handling

2) Verifying prohibited tool usage is difficult in practice

In hiring exercises, training, competitions, or controlled environments, people may want to restrict certain tools.

The practical challenge is not only “detection,” but making the result explainable at the level of a development session and its commits.

Common monitoring approaches often leave gaps:

endpoint-wide monitoring is noisy
network monitoring does not map cleanly to file/commit changes
shell history does not capture all delegated/internal process activity (bash, node, helper processes, IDE internals)

3) Suspicious or unintended process activity during development is hard to explain later

IDE extensions, helper binaries, dependency/tool updates, and delegated subprocesses can all introduce process activity that is difficult to reason about after the fact.

The core problem is often not:

“Can we block everything?”

but rather:

“Can we later explain what ran, what wrote to the workspace, and what commit it relates to?”

DSA (Development-Session Attestation): What I Mean by It

By DSA, I mean:

an approach/framework for treating development sessions (the process of development) as evidence that can be collected, bound, verified, and reviewed — in addition to final artifacts

In other words, DSA extends the evaluation surface from:

artifact only

to:

artifact + development-session evidence

What Questions DSA Should Help Answer

A DSA-capable system should make it easier to answer questions like:

What processes executed in this development session?
Which executable identities wrote to the workspace?
Which files were changed, and what process lineage touched them?
Which commit(s) can this session be linked to?
Did a session violate policy (e.g., prohibited tools)?
Can these claims be verified later?

This is the part that is often missing if we only look at outputs or only look at CI/build provenance.

DSA Is an Evidence and Verification Layer, Not a Skill Scoring System

This distinction matters.

DSA is not:

an automatic “skill score” engine
a full replacement for code review or testing
a universal proof of behavior outside the audited environment

DSA is primarily about:

evidence collection
evidence binding
verification
reviewability

How that evidence is interpreted (hiring policy, education policy, compliance rules, etc.) remains an organizational decision.

Keeping this separation is important; otherwise the discussion quickly collapses into “monitoring” or “tool banning” only.

How DSA Relates to Existing Approaches

DSA is best understood as a complementary layer, not a replacement.

EDR/XDR / Host Auditing

Strong at:

endpoint-wide process visibility

Less strong at:

session-scoped, commit-linked, developer-facing explainability

Network Monitoring (FW / Proxy / NDR)

Strong at:

traffic visibility

Less strong at:

process/file/commit linkage for development work

CI / Provenance / Supply Chain Attestation

Strong at:

build/release provenance
artifact integrity and pipeline traceability

Less strong at:

developer-session provenance (what happened while code was being authored)

Code Review / Testing

Strong at:

output quality and behavior validation

Less strong at:

process evidence (what tools/processes actually ran)

In that sense, DSA sits in a gap between endpoint auditing and CI provenance:

development-session process evidence

A Practical Layer Model for DSA

To make DSA implementation-oriented (and not only conceptual), I find it useful to split it into layers:

Collection
- session evidence (exec, workspace writes, identities)
Binding
- link session evidence to commits/repositories
Verification
- policy checks, signature checks, integrity checks
Review
- human-facing UI/reports/artifacts
Evaluation Policy
- organizational interpretation and usage of evidence

This layering helps separate:

what the tooling should implement
what organizations/teams should define as policy

Where SessionAttested Fits

SessionAttested is one implementation path toward DSA.

It currently provides a concrete stack for:

host-side auditing of dev-container sessions
executable identity aggregation (exec / writer)
commit binding
signed attestations and verification
review-oriented outputs and WebUI

So while it can be used for “AI agent detection” policies, I think the more durable framing is:

SessionAttested is a DSA-oriented evidence and verification foundation.

What DSA Changes in Practice (Realistically)

DSA does not magically solve engineering evaluation.

What it can do is improve the quality of evidence available for review.

For example:

reviewing artifact + development-session evidence together
treating prohibited tool non-observation as a verifiable claim (within a managed session)
making incident/quality discussions more evidence-based (“what ran in this session?”)

That is a meaningful shift even before any “scoring” or large-scale standardization exists.

Limits (Important)

DSA should be discussed with explicit limits, otherwise it becomes over-claiming.

At least in the current framing (and in SessionAttested’s current implementation), DSA does not prove:

that no prohibited tools were used outside the audited environment/session
code quality or architecture quality by itself
a person’s skill level as a standalone conclusion

It provides:

high-confidence, verifiable process evidence for managed development sessions

That framing is narrower, but much more defensible.

Why I Think DSA Is Worth Exploring

I see DSA as a way to update our mental model of engineering evaluation in the AI-assisted era.

Not by abandoning artifact review, but by adding a process-evidence layer where it matters.

That is the motivation behind SessionAttested, and why I think “development-session attestation” is a useful concept to name and refine.

References

SessionAttested (GitHub): https://github.com/shizuku198411/SessionAttested
DSA concept note (repo): DSA.md
SessionAttested README: README.md
PoC example (VS Code forbidden-tool comparison + WebUI): attested_poc/README.md

What a “Development Session Proof” Workflow Looks Like with SessionAttested

Shizuku — Tue, 24 Feb 2026 03:20:08 +0000

This article is a hands-on / operational follow-up to my earlier write-up about SessionAttested.

Instead of focusing on architecture or implementation internals, this one focuses on:

how SessionAttested fits into a real development workflow
how policy refinement actually works in practice
what audit results look like in the built-in WebUI

The example used here is a real PoC workspace: attested_poc/.

In that PoC, I compare two cases under a policy that forbids VS Code (server/node executables):

a session without VS Code usage (PASS)
a session with VS Code usage (FAIL)

If you want to jump straight to the repository:

GitHub: https://github.com/shizuku198411/SessionAttested

PoC workspace example: https://github.com/shizuku198411/SessionAttested/blob/main/attested_poc/README.md

What This Article Tries to Show

SessionAttested is not only about “detecting forbidden tools.”

What becomes more interesting in practice is that it gives you a repeatable workflow for:

auditing development work in session-sized units
binding sessions to commits
evaluating them against a policy
producing signed outputs
reviewing the results later (including cumulative observations across sessions)

That is the angle I want to show here.

The PoC Scenario (VS Code Forbidden Policy)

In attested_poc/, I use a policy that treats VS Code-related executables (specifically VS Code Server / node) as forbidden.

This makes it easy to compare two sessions:

PASS case: development without VS Code
FAIL case: development with VS Code (Remote SSH workflow)

This is useful because it demonstrates both:

the policy verdict (PASS/FAIL)
the audit evidence (what was observed in exec / writer)

A Real Workflow (How It Feels to Use)

The workflow I ended up using repeatedly looks like this:

attested workspace init (auditor)
attested start (auditor)
Work in the dev container + attested git commit (auditee)
attested stop (auditor)
attested attest / attested verify (auditor)
Push (auditor or auditee, depending on policy)
Repeat steps 2–6 for the next session

The important point is:

workspace/container lifecycle is separate from
session lifecycle

So the container can be reused across sessions, while audit boundaries remain clear.

1) Workspace Initialization (`attested workspace init`)

You start from the project directory and run:

attested workspace init

Current behavior (interactive-friendly):

asks for workspace-id
uses current working directory as default workspace path
asks whether to build or pull the container image (default: build)
optionally mounts a host SSH key for Git push from the container
can capture GitHub repo / git user.name / git user.email

This does more than “register a workspace”:

creates SessionAttested scaffolding (attest/attested.yaml, attest/Dockerfile, attest/policy.yaml)
prepares .attest_run/
creates/registers the dev container (left stopped)

That means the workspace becomes immediately usable as a SessionAttested-enabled dev environment.

[IMAGE MARKER] workspace init prompt flow / generated files (optional screenshot)

2) Start a Session (`attested start`)

To begin audited work:

attested start

What happens:

the dev container starts (or is reused if already created)
the collector starts automatically (if configured)
a new session_id is issued
.attest_run/last_session_id is updated (used by no-argument flows)

This is the point where SessionAttested begins collecting:

exec events
/workspace write events

3) Work Inside the Dev Container + Commit (`attested git commit`)

Inside the container, the workflow feels close to normal development:

connect via SSH / VS Code Remote SSH / docker exec
edit code
run commands
commit changes

The key difference is that session-linked commits should use:

cd /workspace
attested git add -A
attested git commit -m "first commit"

This preserves normal Git behavior and records session/commit linkage:

commit_binding.json (latest)
commit_bindings.jsonl (append-only history for multiple commits in one session)

This becomes part of the later attestation and verification flow.

4) Stop the Session (`attested stop`)

When work is done:

attested stop

This finalizes the collector and writes aggregated outputs for that session:

audit_summary.json
event_root.json

Conceptually, this is the moment where “what happened in this dev session” becomes a closed, reviewable unit.

5) Attest + Verify (`attested attest` / `attested verify`)

Next, the auditor evaluates and verifies the session:

attested attest
attested verify --write-result

This produces / updates:

.attest_run/attestations/latest/attestation.json
.attest_run/attestations/latest/attestation.sig
.attest_run/attestations/latest/attestation.pub
ATTESTED
ATTESTED_SUMMARY
ATTESTED_POLICY_LAST
ATTESTED_WORKSPACE_OBSERVED

Two outputs are especially useful in day-to-day use:

`ATTESTED_SUMMARY`

Per-session verification results (append-only history):

pass/fail
reason
commit(s)
policy match status

`ATTESTED_WORKSPACE_OBSERVED`

Workspace-level cumulative observed identities across sessions:

what executables have been seen
what writer identities have been seen
unresolved counters

This is extremely useful when building or refining a policy over time.

6) How Policy Refinement Actually Works (`attested policy candidates`)

SessionAttested identifies tools by executable fingerprint (SHA256), not only by path strings.

That means a practical policy workflow is:

observe what actually ran
generate candidates
review and refine
apply policy
run another session and evaluate

Generate a Candidate Policy

attested policy candidates --include-exec

This produces something like:

.attest_run/policy.<session_id>.candidate.yaml

The candidate includes observed identities as forbidden_exec / forbidden_writers candidates.

Review Before Applying

In practice, this review step matters a lot.

Things to check:

Is this identity really a prohibited tool?
Is it just a shell/helper process (bash, sh, node) delegated by another tool?
Is this a temporary install/setup tool that should not be in policy?

In my PoC, I used this process to define a policy that flags VS Code Server / node executables as forbidden.

[IMAGE MARKER] policy snippet (attest/policy.yaml) showing VS Code-related forbidden entries

7) Reviewing Results in the WebUI (`attested webui`)

This is the new part in v0.1.1 that made the workflow much easier to inspect.

attested webui

It launches a local HTTPS WebUI (self-signed certificate), and renders the recorded audit/verification outputs.

Why This Matters

Before the WebUI, the workflow was already functional, but reviewing results meant opening several files manually:

ATTESTED_SUMMARY
audit_summary.json
attestation.json
ATTESTED_WORKSPACE_OBSERVED

The WebUI makes this much faster.

Session List (PASS/FAIL at a Glance)

The first thing I check is the session list (“See other sessions”):

session time window
session id
conclusion (PASS/FAIL)

This immediately tells you which sessions need attention.

PASS Case View

For PASS sessions, the useful checks are:

attestation/verification status
audit summary counts
executed/writer identities observed in that session

FAIL Case View (VS Code Forbidden)

For FAIL sessions, the WebUI makes it easier to answer:

Why did it fail? (reason code)
Which identities matched the policy?
Was it exec, writer, or both?

And with the detail view:

applied policy snapshot
conclusion details
workspace cumulative observed identities

This makes the result much easier to explain to humans compared to raw files alone.

Important Note About the WebUI

The WebUI is a viewer for recorded results, not a replacement for attest / verify.

machine decision: attest / verify
human review: attested webui

That split has worked well in practice.

8) Who Should Push? (Auditor vs Auditee)

One thing I liked in actual usage: Git push can be done by either side.

Auditee (inside container) pushes directly
Auditor (host side) pushes only after attest/verify passes

Both are valid. Which one you choose depends on policy.

Pattern A: Auditee Pushes

Good when:

speed matters
auditing is used as evidence/review, not a hard release gate

Pattern B: Auditor Pushes After Verify

Good when:

you want an explicit “verified before publication” workflow
the auditor controls what gets pushed/published

The attested_poc setup supports either pattern.

9) What I Learned from Using It in Actual Workflows

1. It standardizes the workflow nicely

The repeated flow:

workspace init
start
work + attested git commit
stop
attest / verify

creates clear audit boundaries without much extra mental load.

2. It is still useful even for solo development

Even when auditor = auditee, it remains valuable as a high-confidence process record:

what ran
what wrote
which session/commit it was tied to

And ATTESTED_WORKSPACE_OBSERVED is especially useful for post-hoc review and policy tuning.

3. It doesn’t get in the way too much

Since auditing runs on the host and development happens in a dev container:

the host stays cleaner
the dev environment remains flexible
IDE choice is not heavily constrained

This matters more than it sounds if you actually try to use a security/audit tool in daily development.

10) Closing Thoughts

Using attested_poc as a concrete example made one thing clear to me:

SessionAttested is more useful when seen as a development-session proof workflow than as “just a detection tool.”

The combination that makes it work in practice is:

dev-container-scoped work
host-side observation of exec / writer
commit binding
policy evaluation
human review via WebUI

There is still room to improve (for example, persisting per-session attestation.json instead of only latest), but the workflow already feels usable enough to evaluate in real development.

References

SessionAttested (GitHub): https://github.com/shizuku198411/SessionAttested
PoC Workspace Example (EN): https://github.com/shizuku198411/SessionAttested/blob/main/attested_poc/README.md

Forem: Shizuku

Beyond Artifact-Only Evaluation: A Case for Development-Session Attestation (DSA)

Introduction

What This Article Is (and Isn’t)

What Is Changing in Artifact-Centric Evaluation

1) Portfolio artifacts are becoming easier to produce without corresponding understanding

2) Verifying prohibited tool usage is difficult in practice

3) Suspicious or unintended process activity during development is hard to explain later

DSA (Development-Session Attestation): What I Mean by It

What Questions DSA Should Help Answer

DSA Is an Evidence and Verification Layer, Not a Skill Scoring System

How DSA Relates to Existing Approaches

EDR/XDR / Host Auditing

Network Monitoring (FW / Proxy / NDR)

CI / Provenance / Supply Chain Attestation

Code Review / Testing

A Practical Layer Model for DSA

Where SessionAttested Fits

What DSA Changes in Practice (Realistically)

Limits (Important)

Why I Think DSA Is Worth Exploring

References

What a “Development Session Proof” Workflow Looks Like with SessionAttested

What This Article Tries to Show

The PoC Scenario (VS Code Forbidden Policy)

A Real Workflow (How It Feels to Use)

1) Workspace Initialization (attested workspace init)

2) Start a Session (attested start)

3) Work Inside the Dev Container + Commit (attested git commit)

4) Stop the Session (attested stop)

5) Attest + Verify (attested attest / attested verify)

ATTESTED_SUMMARY

ATTESTED_WORKSPACE_OBSERVED

6) How Policy Refinement Actually Works (attested policy candidates)

Generate a Candidate Policy

Review Before Applying

7) Reviewing Results in the WebUI (attested webui)

Why This Matters

Session List (PASS/FAIL at a Glance)

PASS Case View

FAIL Case View (VS Code Forbidden)

Important Note About the WebUI

8) Who Should Push? (Auditor vs Auditee)

Pattern A: Auditee Pushes

Pattern B: Auditor Pushes After Verify

9) What I Learned from Using It in Actual Workflows

1. It standardizes the workflow nicely

2. It is still useful even for solo development

3. It doesn’t get in the way too much

10) Closing Thoughts

References

1) Workspace Initialization (`attested workspace init`)

2) Start a Session (`attested start`)

3) Work Inside the Dev Container + Commit (`attested git commit`)

4) Stop the Session (`attested stop`)

5) Attest + Verify (`attested attest` / `attested verify`)

`ATTESTED_SUMMARY`

`ATTESTED_WORKSPACE_OBSERVED`

6) How Policy Refinement Actually Works (`attested policy candidates`)

7) Reviewing Results in the WebUI (`attested webui`)