Forem: Bobai Kato

Stop Asking AI Agents to Guess How Your Repo Works

Bobai Kato — Sun, 24 May 2026 00:00:00 +0000

AI coding agents are getting better, but a lot of repos still make them start from guesswork.

They land in a codebase and have to infer:

which command installs dependencies
which command starts the app
which services need to be running
which env vars matter
which workflow is safe to run
whether the repo is actually ready
what “done” should look like after a change

That is a lot of operational knowledge to hide in scattered README sections, old shell scripts, CI files, tribal memory, and half-working local setup notes.

And when an agent guesses wrong, the problem is not always the agent.

Sometimes the repo never gave it a reliable answer.

AI Agents Amplify Repo Readiness

A human contributor can pause, read docs, ask questions, remember past setup quirks, and work around drift.

An AI agent usually does something else: it follows the strongest signal it can find.

If your README says one thing, your CI says another, your package scripts say a third, and your local setup depends on an undocumented service, the agent has to choose. Sometimes it chooses well. Sometimes it burns time. Sometimes it changes the wrong thing because the repo never made the safe path explicit.

That is why repo readiness matters.

Before asking an AI agent to work in a repo, the repo should be able to answer basic questions clearly:

What does this repo need?
How does it become ready?
What tasks are safe to run?
What checks prove the repo still works?
What paths should an agent avoid?
What workflow should be used for this kind of change?

If those answers are not explicit, every agent has to rediscover them from scratch.

README Instructions Are Not Enough

READMEs are useful for humans, but they are not enough as the only source of operational truth.

A README can explain intent, context, and contribution style. But repo readiness needs something more precise:

declared runtimes
declared tools
declared services
declared tasks
declared workflows
declared checks
declared agent boundaries

That information should be machine-readable.

Not because humans do not matter, but because humans, CI, and agents should not each maintain separate versions of the same setup logic.

When setup knowledge only exists as prose, it drifts. When it drifts, automation becomes fragile. When automation becomes fragile, agents become less trustworthy.

The Repo Should Tell The Agent What Ready Means

A good agent workflow should not begin with:

“Look around and figure out how this repo works.”

It should begin with:

“Here is the repo contract. Diagnose readiness first. Use the declared workflow. Run the safe task. Respect protected paths.”

That is the difference between guessing and operating.

This is the layer I care about with Ota.

Ota gives a repo one explicit readiness contract: ota.yaml.

That contract can declare the repo’s tasks, checks, services, workflows, toolchains, env requirements, and agent-safe boundaries. Then humans, CI, and AI agents can all use the same source of truth.

For example:

ota doctor
ota workflows
ota tasks --workflow app
ota up --workflow app
ota run test

The point is not to replace good documentation.

The point is to stop making documentation carry operational truth alone.

What Agents Need From A Repo

An AI agent does not need magic. It needs clear boundaries.

A solid repo should be able to say:

agent:
  entrypoint: test
  default_task: test
  safe_tasks:
    - test
    - lint
    - typecheck
  protected_paths:
    - .env
    - secrets/**
    - production/**

The agent boundary tells agents which task to start from, which tasks are safe, and which paths are protected. The instruction is simple: run ota doctor first, then use declared safe tasks instead of guessing from repo scripts.

It should also be able to say what ready means:

workflows:
  default: app
  app:
    setup:
      task: setup
    run:
      task: dev
    readiness:
      checks:
        - app-ready

Now the agent is not guessing from random scripts.

It has a contract.

It knows the default workflow. It knows the safe tasks. It knows what not to touch. It knows which checks matter.

That is a better starting point for any coding agent.

CI Should Prove The Same Truth

Repo readiness should not only help local agents. It should also show up in CI.

If a repo has an explicit readiness contract, CI can prove that contract across environments:

name: ota-readiness

on:
  pull_request:
  push:

jobs:
  readiness:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ota-run/setup@v1
      - run: ota doctor
      - run: ota validate .
      - run: ota up

That gives maintainers something concrete.

Not “the agent says it worked.”

A repeatable readiness check.

Why This Matters Now

AI agents are moving from demos into real repos.

That changes the bar.

A repo that is “understandable if you already know it” is not enough. A repo that “usually works after reading three docs pages” is not enough. A repo that depends on one maintainer’s memory is not enough.

If we want agents to make useful changes safely, repos need to become easier to reason about.

That means:

fewer hidden setup steps
fewer undocumented services
fewer ambiguous commands
fewer unsafe default paths
more explicit readiness
more machine-readable operational truth

The better your repo contract, the better every agent run becomes.

Get Started With Ota

Install Ota:

curl -fsSL https://dist.ota.run/install.sh | sh

Windows:

irm https://dist.ota.run/install.ps1 | iex

Then start with:

ota doctor
ota init --bootstrap
ota validate .
ota workflows
ota tasks

Useful links:

Install Ota: ota.run/docs/install
Ota docs: ota.run/docs
Ota source code: github.com/ota-run/ota
Example contracts: github.com/ota-run/examples
GitHub Action: github.com/ota-run/action
GitHub setup action: github.com/ota-run/setup

If your repo already has setup scripts, CI workflows, and contributor docs, Ota does not replace them blindly.

It helps turn the important parts into an explicit contract that humans, CI, and agents can all share.

Final Thought

AI agents should not have to reverse-engineer your repo every time they enter it.

Make the repo ready first.

Then let the agent work.

Pressure-testing Ota on Supabase: from setup prose to executable repo readiness

Bobai Kato — Sat, 23 May 2026 00:00:00 +0000

Supabase already has strong contributor documentation.

That is not faint praise. The docs are good. But good docs and executable readiness are not the same thing.

Before Ota, the www/docs contributor path still depended on a familiar manual loop:

read the docs
infer which workflow is the intended front door
install the right runtime and package manager versions
guess which commands are safe to run
decide whether the repo is actually ready, or only partially set up
if something fails on macOS, Windows, or Linux, figure out whether the problem is the repo, the machine, CI, or your shell

That is a normal setup experience in modern repos. It is also exactly where drift starts.

This post is about what changed when I added Ota to a scoped slice of the Supabase monorepo, how that improved the repo surface itself, and what the deeper pressure test exposed in Ota core.

Why this matters

Most repos still answer “why does this not run?” badly.

The failure mode usually looks like this:

local setup works differently from CI
one workflow is implied, but not explicitly declared
README setup and repo reality slowly diverge
native and container paths are present, but not clearly separated
readiness is inferred from “it seems to work” instead of declared checks
contributors and agents have to reverse-engineer what the repo expects

That is painful for humans. It is worse for agents.

If an AI agent is dropped into an unfamiliar repo, the first problem is usually not the code change itself. The first problem is operational ambiguity:

what this repo slice needs
which workflow is the front door
which tasks are safe
what “ready” means
what is actually blocking progress
what the next safe action should be

If the repo cannot expose that clearly, the agent is forced to guess from README prose, scripts, lockfiles, CI output, and convention.

That is what Ota is designed to remove.

What Ota added to Supabase

The upstream PR is intentionally scoped and intentionally lean:

PR: supabase/supabase#46269
Contract: ota.yaml

The PR adds:

an ota.yaml contract for the www/docs slice
a cross-OS contract matrix
Ota local artifact ignores
a small optional Ota section in CONTRIBUTING.md

That gives Supabase a machine-readable contract for:

the runtime and toolchain it expects
the workflows contributors should use
the setup tasks that make those workflows runnable
the readiness checks that prove the slice is actually up
the tasks and topology surfaces tooling can inspect directly

This is the shift from setup prose to executable repo readiness.

Before and after

Before Ota

The www/docs path was documented, but it was not encoded as repo truth.

That left both contributors and agents to reconstruct:

which commands matter
which runtime/toolchain versions are expected
whether native and container paths are meant to be equivalent
what should count as “ready”
how to distinguish a local issue from a repo contract issue

After Ota

That same slice can now answer those questions directly.

You can ask the repo:

ota validate .
ota doctor --workflow instant --native .
ota up --workflow instant --native --dry-run .
ota tasks --workflow app .
ota execution topology --json .

And the repo can answer from declared contract truth, not inferred setup lore.

For Supabase, that reduces four kinds of drift:

README drift

Setup prose can go stale. A contract is executable and testable.
Local vs CI drift

The same declared workflow can be pressure-tested in matrix CI.
Human vs agent drift

Humans and agents can consume the same operational truth surface.
Platform drift

Windows, macOS, and Linux differences get surfaced earlier instead of being rediscovered ad hoc.

Why the upstream PR stayed lean

This part was deliberate.

The public PR models the www/docs slice. It does not claim full Supabase readiness.

That is not a limitation of ambition. It is a review strategy.

A smaller honest contract is easier to review, easier to approve, and easier to trust than a broad fuzzy one. So the public change stays disciplined:

scoped slice
clear workflow surface
concrete CI pressure around that surface

But that did not mean I only did the minimum.

In parallel, I ran a deeper pressure test on a separate branch:

Supabase pressure branch

That separation mattered:

keep the upstream PR lean for higher approval odds
do the heavier product hardening outside the PR
fix real Ota gaps in Ota itself instead of hiding them in repo-local glue

That is the right way to use a serious repo as a product test surface.

The pressure test

The first PR branch was only the start.

The full pressure cycle expanded into:

native matrix coverage across Ubuntu, macOS, Windows PowerShell, and Windows Git Bash
container matrix coverage across supported runner surfaces
strict E2E parity checks
bounded verification tasks
a broader studio slice on the pressure branch
repeated reruns to separate repo noise from real Ota defects

Representative runs:

Lean PR branch matrix: run #26281955993
Full pressure branch green run: run #26300846166

This is where the value of Ota becomes clearer.

The point was not just to prove that ota.yaml could validate.

The point was to prove that the readiness model, workflow targeting, and platform behavior held up under real pressure.

What the contract actually models

At a high level, the contract models:

Node + pnpm toolchain expectations
native and container execution surfaces
workflow-specific setup paths
HTTP readiness for www and docs
safe bounded tasks for validation and verification

That gives the repo a surface that tools can inspect directly instead of inferring from scattered files.

For example:

ota validate .
ota doctor --workflow instant --native .
ota up --workflow instant --native --dry-run .
ota tasks --workflow app .
ota execution topology --json .

Those commands are not wrappers around documentation. They operate from declared repo truth.

What actually broke in Ota

Supabase exposed a real Ota bug.

The problem was selected workflow toolchain scoping.

In practice, Ota could let unrelated toolchain-owned tools leak into a selected workflow surface just because they were declared elsewhere in the contract.

That sounds small. It is not.

If a selected workflow does not require something, Ota should not silently activate or diagnose against it just because the broader repo declares it somewhere else.

If it does, you get:

noisy dry-runs
misleading readiness signals
false activation pressure
lower trust in workflow targeting

That bug was not fixed in the Supabase contract.

It was fixed in Ota core.

That distinction matters. If the platform is wrong, the correct move is to fix the platform, not teach users to work around it.

What did not count as an Ota bug

The pressure test also surfaced failures that looked suspicious at first but were not Ota defects.

The clearest example was the Docker-backed Studio path on hosted CI.

That failure came from runner and stack-specific behavior in the Supabase Docker environment, including hosted ulimit constraints. That is real repo/runtime behavior, but it is not an Ota orchestration bug.

So the right response was:

do not misclassify repo or runner behavior as an Ota platform defect
do not hide runtime limitations just to force more green boxes
keep the matrix useful for signal, not vanity

Trust is not built by making every box green. Trust is built by classifying failures correctly.

What value this adds to Supabase

The value here is not “Supabase now has another YAML file.”

The value is that a real slice of the repo now has:

an explicit operational contract
declared workflow entry points
repeatable readiness checks
machine-readable task and topology surfaces
cross-platform pressure testing around that declared truth

For Supabase maintainers, that means:

less rediscovery of the same setup blockers
earlier visibility into platform-specific drift
a clearer operational model for contributors

For contributors, that means:

less guessing
less README archaeology
less ambiguity around which path is actually supported

For agents, that means:

fewer blind setup guesses
a clearer answer to “what should I run?”
a more reliable signal for “is this repo ready, or am I about to make changes in a broken environment?”

Why Ota still adds value when a repo already has good docs

A fair question is: if Supabase already has solid docs, what does Ota add?

Docs and contracts do different jobs.

Docs explain.

Contracts declare.

Docs are excellent for onboarding context, caveats, workflow explanation, and human guidance. But docs do not usually answer machine-checkable questions like:

is this workflow ready right now?
which workflow is the front door?
which setup path belongs to this slice?
which checks prove readiness?
which tasks are safe for agents?
how does this differ across native and container execution?

That is the gap Ota fills.

It does not replace good docs. It gives the repo an executable readiness layer that docs alone cannot provide.

Why this matters beyond Supabase

The Supabase case study is really about a broader repo question:

What should happen when a repo does not run?

Most repos still answer that question badly.

They fail, and then they force the contributor to become the diagnosis layer.

Ota changes that shape.

A repo can declare:

what it needs
how it becomes runnable
what ready means
what is safe to run
what should block progress

That turns readiness from tribal knowledge into contract truth.

Supabase is a useful example because it is large enough to expose real pressure, but still scoped enough to model honestly.

Current state of the PR

The upstream PR is intact:

supabase/supabase#46269

From our side:

no unresolved actionable review threads
contract and matrix changes are clean
remaining red checks are upstream Vercel authorization and deploy noise, not Ota contract failures

So the Ota side of the work is sound.

Try it on your repo

If your repo already has setup docs, that is a good start.

The harder question is whether the repo can expose, in a machine-readable and testable way:

what it needs
how it becomes runnable
what ready means
what is safe to run
what should block progress

That is where Ota adds value.

If you want to pressure-test that on a real codebase, start here:

ota-run/ota

And if you want to start from the Supabase example specifically:

Note: at the time of writing, the upstream Supabase PR is still open. I’ll update this post with the final outcome once maintainers finish review.

References

The concrete artifacts from this work are here:

Pressure Testing Ota on n8n: A Closed PR That Still Proved the Point

Bobai Kato — Tue, 19 May 2026 13:54:52 +0000

I ran a real pressure test on one of the most visible OSS automation repos: n8n-io/n8n.

PR: n8n-io/n8n#30714

At first glance, a closed PR looks like a loss. It wasn’t.

Why this test mattered

n8n is a high-signal repo for readiness tooling:

large monorepo
multiple contributor paths
cross-platform contributors
native + Docker runtime surfaces
mature existing docs

If Ota only works on simple repos, it’s not infrastructure.

n8n is exactly the kind of repo that exposes whether Ota is real.

Before vs After (in the test branch)

Dimension	Before	After (Ota branch)
Readiness definition	Documented across markdown instructions	Explicit contract in `ota.yaml`
Cross-OS proof	Implicit, contributor-dependent	Matrix proof in GitHub Actions
Workflow entrypoints	Manual command selection	Named workflows (`app`, `backend`, `instant`, `docker`)
Machine-checkable status	Partial	Deterministic `ota proof` outcomes
Agent bootstrap metadata	None	`agent.bootstrap.ota` declared

This was additive. It did not replace n8n’s canonical setup flow.

What was added

In my branch I introduced:

ota.yaml with workflow-specific readiness paths
a smoke matrix workflow for Linux/macOS/Windows + Docker proof
pinned Ota version in CI for deterministic behavior
optional contributor-facing Ota guidance

Green proof run example:

Run 26092939099

Why maintainers closed it

The maintainers closed the PR (n8n-io/n8n#30714) for policy reasons, not technical failure:

they don’t currently need an extra readiness layer
they keep CI third-party dependencies narrow
they don’t want contributor docs to endorse an external tool

That’s a valid maintainer call.

The actual value

Even without merge, this test delivered high-value evidence:

Ota can model and prove readiness on a complex repo.
Cross-platform behavior held under matrix pressure.
Adoption boundaries are now clearer: technical fit and governance fit are separate gates.
Ota got sharper through real-world constraints, not synthetic demos.

So this is demonstrated value, not adopted value.

Final take

A closed PR can still be a product win.

This n8n test proved Ota’s technical posture under real load, clarified adoption constraints, and generated stronger evidence for the next integration target.

That’s exactly what pressure testing is for.

Get Started with Ota

Ready to pressure-test your own repo? Start here:

Install Ota: ota.run/docs/install
Ota docs: ota.run/docs
Ota GitHub: github.com/ota-run/ota
Contract examples: github.com/ota-run/examples

First commands (safe for any repo)

ota doctor
ota workflows
ota tasks --use
ota validate .

Prove a workflow (replace with one from `ota workflows`)

ota proof --workflow <workflow-name>

If your repo has no contract yet

ota init --bootstrap
ota validate .

Use the examples repo as a base, then define workflow names that match your repo.