Forem: ko-chan

Designing SaaS Commerce with State Machines [Part 11]

ko-chan — Mon, 23 Feb 2026 04:43:52 +0000

This article was originally published on Saru Blog.

What You Will Learn

Why "string status" breaks down in SaaS commerce
Implementation patterns for embedding state machines into Go domain models
Design techniques for coordinating multiple state machines
Handling edge cases: partial payments, expiration, and optimistic locking

The Status Column from Hell

When building web applications, you will almost certainly encounter "status" columns. Order status, invoice status, user account status. What starts as a simple active / inactive boolean grows into pending, processing, completed, cancelled, refunded... as the service evolves.

In the SaaS I'm building (a multi-tenant subscription management system), there are separate statuses for Quotes, Orders, Invoices, and Payments.

It starts simple.

CREATE TABLE invoices (
    id UUID PRIMARY KEY,
    status VARCHAR(20) NOT NULL DEFAULT 'draft',
    ...
);

On the application side:

// A common implementation
func (s *InvoiceService) MarkPaid(id uuid.UUID) error {
    invoice, _ := s.repo.Get(id)
    invoice.Status = "paid"  // ← direct string assignment
    return s.repo.Update(invoice)
}

This works. But as the service grows, problems emerge.

Problem 1: Can't prevent invalid transitions

// Is draft → paid really an allowed transition?
// Should "paid" be possible without going through "sent" first?
invoice.Status = "paid"

By business rules, invoices should transition draft → sent → paid. But with string assignment, any state can transition to any other state.

Problem 2: Typos aren't caught until runtime

invoice.Status = "piad"  // typo goes unnoticed

It compiles. Without tests, you'll only discover this in production.

Problem 3: Transition rules scatter across the codebase

// handler.go
if invoice.Status == "sent" || invoice.Status == "overdue" {
    invoice.Status = "paid"
}

// service.go
if invoice.Status != "void" && invoice.Status != "paid" {
    // payment processing
}

// worker.go
if invoice.Status == "sent" && time.Now().After(invoice.DueDate) {
    invoice.Status = "overdue"
}

The same transition rules are scattered across multiple locations, and missing an update in one place becomes a bug.

State Machines as the Solution

The state machine idea is simple: define which states can transition to which other states in one place, and forbid all other transitions.

In SaaS commerce (Quote-to-Cash), there are the following entities:

Cart                    Quote
  │                       │
  │ Convert()             │ Accept() → NewOrderFromQuote()
  └──────────┐   ┌───────┘
             ↓   ↓
        Order
             │
             │ Complete()
             ↓
     Subscription
             │
             │ billing cycle
             ↓
       Invoice
             │
             │ RecordPayment()
             ↓
       Payment

Orders are created from two entry points: Cart and Quote. From there, the flow continues through Subscription → Invoice → Payment. Cart is the self-service path where customers select products; Quote is the path where sales representatives present terms. The entry points differ, but the resulting Order has the same structure.

This article focuses on the Quote → Order → Invoice → Payment flow. Each has its own state machine.

Quote

draft ──→ sent ──→ accepted
                ├→ rejected
                └→ expired

draft: Just created. Line items can be added/removed
sent: Sent to customer. No modifications allowed
accepted: Customer accepted. Can be converted to an order
rejected / expired: Terminal states

Order

pending ──→ awaiting_payment ──→ confirmed ──→ processing ──→ completed

State	Transitions To
`pending`	`awaiting_payment`, `confirmed`, `cancelled`
`awaiting_payment`	`confirmed`, `cancelled`
`confirmed`	`processing`, `cancelled`
`processing`	`completed`, `cancelled`
`completed` / `cancelled`	(terminal states)

cancelled is reachable from any state except completed. Orders should be cancellable until the very last moment — a business requirement.

Invoice

draft ──→ sent ──→ paid
    │         ├→ overdue ──→ paid
    │         │         └→ void
    └→ void   └→ void

The key point: overdue can transition to paid. Late payments happen routinely

Payment

pending ──→ completed ──→ refunded
    │              └→ disputed ──→ completed (chargeback won)
    └→ failed                  └→ refunded (chargeback lost)

Implementation Patterns in Go

Pattern 1: Embed Transition Rules in Enum Types

Centralize transition rule definitions in one place. In Go, use custom types and methods.

type QuoteStatus string

const (
    QuoteStatusDraft    QuoteStatus = "draft"
    QuoteStatusSent     QuoteStatus = "sent"
    QuoteStatusAccepted QuoteStatus = "accepted"
    QuoteStatusRejected QuoteStatus = "rejected"
    QuoteStatusExpired  QuoteStatus = "expired"
)

// CanTransitionTo defines all transition rules.
// This single method shows which transitions are allowed.
func (s QuoteStatus) CanTransitionTo(target QuoteStatus) bool {
    switch s {
    case QuoteStatusDraft:
        return target == QuoteStatusSent
    case QuoteStatusSent:
        return target == QuoteStatusAccepted ||
               target == QuoteStatusRejected ||
               target == QuoteStatusExpired
    case QuoteStatusAccepted, QuoteStatusRejected, QuoteStatusExpired:
        return false // terminal states cannot transition
    default:
        return false
    }
}

The benefits of this design:

Transition rules live in one place. They don't scatter across handlers and service layers
default: return false automatically rejects unknown states
Easy to test. All transition patterns can be exhaustively verified

Tests verify both allowed and forbidden transitions:

func TestQuoteStatus_CanTransitionTo(t *testing.T) {
    tests := []struct {
        from     QuoteStatus
        to       QuoteStatus
        expected bool
    }{
        // Allowed transitions
        {QuoteStatusDraft, QuoteStatusSent, true},
        {QuoteStatusSent, QuoteStatusAccepted, true},
        {QuoteStatusSent, QuoteStatusRejected, true},
        {QuoteStatusSent, QuoteStatusExpired, true},

        // Forbidden transitions
        {QuoteStatusDraft, QuoteStatusAccepted, false},  // can't go directly from draft to accepted
        {QuoteStatusAccepted, QuoteStatusDraft, false},   // can't return from terminal state
        {QuoteStatusRejected, QuoteStatusSent, false},    // can't re-send after rejection
    }

    for _, tt := range tests {
        got := tt.from.CanTransitionTo(tt.to)
        if got != tt.expected {
            t.Errorf("%s → %s: got %v, want %v", tt.from, tt.to, got, tt.expected)
        }
    }
}

Table-driven tests that enumerate all transition patterns also serve as a specification document. When adding a new state, forgetting to add a test case becomes conspicuous.

For greater robustness, a test that covers all state combinations is effective:

func TestQuoteStatus_AllTransitions(t *testing.T) {
    allStatuses := []QuoteStatus{
        QuoteStatusDraft, QuoteStatusSent,
        QuoteStatusAccepted, QuoteStatusRejected, QuoteStatusExpired,
    }

    // Whitelist of allowed transitions
    allowed := map[QuoteStatus][]QuoteStatus{
        QuoteStatusDraft: {QuoteStatusSent},
        QuoteStatusSent:  {QuoteStatusAccepted, QuoteStatusRejected, QuoteStatusExpired},
    }

    for _, from := range allStatuses {
        for _, to := range allStatuses {
            expected := false
            for _, a := range allowed[from] {
                if a == to {
                    expected = true
                    break
                }
            }
            got := from.CanTransitionTo(to)
            if got != expected {
                t.Errorf("%s → %s: got %v, want %v", from, to, got, expected)
            }
        }
    }
}

This test verifies all 5×5=25 combinations. While forgetting to add a new state to allStatuses would leave the test incomplete, it at least guarantees existing transition rules haven't been broken.

Pattern 2: Execute Transitions Through Domain Model Methods

Rather than calling CanTransitionTo() directly, give the domain model methods that represent transitions.

// Send transitions the quote to the sent state.
func (q *Quote) Send() error {
    if !q.Status.CanTransitionTo(QuoteStatusSent) {
        return ErrInvalidQuoteTransition
    }
    if len(q.Items) == 0 {
        return ErrQuoteEmpty  // guard: can't send an empty quote
    }
    q.Status = QuoteStatusSent
    q.UpdatedAt = time.Now()
    return nil
}

// Accept transitions the quote to the accepted state.
func (q *Quote) Accept() error {
    if !q.Status.CanTransitionTo(QuoteStatusAccepted) {
        return ErrInvalidQuoteTransition
    }
    if q.IsExpired() {
        return ErrQuoteExpired  // guard: can't accept an expired quote
    }
    q.Status = QuoteStatusAccepted
    q.UpdatedAt = time.Now()
    return nil
}

Each method follows a common structure:

Check if the transition is allowed (CanTransitionTo)
Check guard conditions (business-rule-specific additional conditions)
Change state
Update timestamp

This keeps the handler layer simple:

// handler
func (h *QuoteHandler) Send(c echo.Context) error {
    quote, err := h.service.Get(c.Request().Context(), quoteID)
    if err != nil {
        return err
    }

    if err := quote.Send(); err != nil {
        return err  // transition rule violations automatically become errors
    }

    return h.service.Update(c.Request().Context(), quote)
}

The handler expresses only the intent to "send the quote" and doesn't need to know the details of transition rules.

Pattern 3: State-Dependent Operation Restrictions

Encapsulate "when changes are allowed" within the domain model as well.

// AddItem adds a line item to the quote.
func (q *Quote) AddItem(item *QuoteItem) error {
    if q.Status != QuoteStatusDraft {
        return ErrInvalidQuoteStatus  // can't modify items outside of draft
    }
    item.QuoteID = q.ID
    q.Items = append(q.Items, item)
    q.CalculateTotals()
    q.UpdatedAt = time.Now()
    return nil
}

// IsModifiable returns whether the quote can be modified.
func (q *Quote) IsModifiable() bool {
    return q.Status == QuoteStatusDraft
}

The rule "you can only add line items to a draft quote" is naturally expressed within the domain model. API handlers and frontends don't need to check this independently.

Coordinating Multiple State Machines

So far, we've been discussing single entities. In SaaS commerce, multiple state machines coordinate to form a single business flow. This is the hardest part of the design.

Quote → Order Conversion

When a quote is accepted, it can be converted to an order. But "accepted" alone isn't sufficient.

// CanConvertToOrder determines whether a quote can be converted to an order.
// All conditions must be met.
func (q *Quote) CanConvertToOrder() bool {
    return q.Status == QuoteStatusAccepted &&
           !q.IsExpired() &&
           len(q.Items) > 0
}

A combination of three conditions:

Condition	Reason
`Status == Accepted`	Can't convert unless accepted
`!IsExpired()`	Can't convert even if accepted, if the validity period has expired
`len(Items) > 0`	A quote with no line items can't become an order

The conversion is implemented as a factory method:

// NewOrderFromQuote creates an order from an accepted quote.
func NewOrderFromQuote(quote *Quote) (*Order, error) {
    if !quote.CanConvertToOrder() {
        return nil, ErrInvalidQuoteStatus
    }

    order := NewOrder(CreateOrderParams{
        ProviderID: quote.ProviderID,
        CustomerID: quote.CustomerID,
        QuoteID:    &quote.ID,  // ← keep a reference to the quote
        Currency:   quote.Currency,
    })

    // Copy line items from quote to order
    for _, qi := range quote.Items {
        oi := NewOrderItem(CreateOrderItemParams{
            PlanID:      qi.PlanID,
            Description: qi.Description,
            Quantity:    qi.Quantity,
            UnitPrice:   qi.UnitPrice,
        })
        oi.OrderID = order.ID
        order.Items = append(order.Items, oi)
    }

    order.Subtotal = quote.Subtotal
    order.TotalAmount = quote.TotalAmount
    return order, nil
}

Three key points:

Centralized guard conditions: Aggregate checks in CanConvertToOrder(). Don't check individually in the service layer
Preserve references: Track the original quote via QuoteID
Copy data: Copy quote line items to the order, giving them independent lifecycles

Invoice and Payment Coordination

The coordination between invoices and payments is slightly more complex.

// RecordPayment records a payment against the invoice.
func (inv *Invoice) RecordPayment(amount decimal.Decimal) error {
    if inv.Status != InvoiceStatusSent && inv.Status != InvoiceStatusOverdue {
        return ErrInvoiceNotPayable  // can't pay draft or void invoices
    }

    newAmountPaid := inv.AmountPaid.Add(amount)
    if newAmountPaid.GreaterThan(inv.TotalAmount) {
        return ErrPaymentExceedsAmount  // prevent overpayment
    }

    inv.AmountPaid = newAmountPaid
    inv.AmountDue = inv.TotalAmount.Sub(newAmountPaid)
    inv.UpdatedAt = time.Now()

    // Automatically transition to paid when fully paid
    if inv.AmountDue.LessThanOrEqual(decimal.Zero) {
        return inv.MarkPaid()
    }

    return nil
}

This implementation contains two design decisions.

1. Allow partial payments

By separating AmountPaid and AmountDue, the invoice amount can be satisfied through multiple payments. This supports installment payments and partial receipts.

2. Automatic transitions

When AmountDue reaches zero, the state automatically transitions to paid without explicitly calling MarkPaid(). The payment completion logic is contained within the domain model.

The Overall Flow

Cart
  │ Convert()
  ↓
Quote
  │ Accept() → CanConvertToOrder()
  ↓
Order
  │ Confirm() → Complete()
  ↓
Subscription
  │ billing cycle arrives
  ↓
Invoice
  │ Send() → RecordPayment()
  ↓
Payment
  │ Complete() or Fail()
  ↓
[Flow ends, next billing cycle begins]

Each entity maintains its own state machine while coordinating through conversion methods (NewOrderFromQuote) and guard conditions (CanConvertToOrder).

Edge Cases and Design Decisions

Time-Based Automatic Transitions

Quotes have validity periods. How should expiration be handled?

// IsExpired determines whether the validity period has passed.
func (q *Quote) IsExpired() bool {
    return time.Now().After(q.ValidUntil)
}

// Accept checks for expiration during acceptance.
func (q *Quote) Accept() error {
    if !q.Status.CanTransitionTo(QuoteStatusAccepted) {
        return ErrInvalidQuoteTransition
    }
    if q.IsExpired() {
        return ErrQuoteExpired
    }
    // ...
}

There's a design choice here:

Approach	Pros	Cons
Check on access (↑ implementation)	No cron job needed, simple	DB status and reality can diverge
Periodically transition to expired via cron	DB consistency maintained	Cron management required

The implementation above uses check-on-access. When a quote is accessed (when someone tries to accept it), the expiration is checked, and if expired, the request is rejected. The DB status remains sent, but IsExpired() determines the actual state.

Optimistic Locking

When multiple users simultaneously operate on the same quote or invoice, a version field detects conflicts.

type Quote struct {
    // ...
    Version int  // for optimistic locking
}

UPDATE quotes
SET status = $1, version = version + 1, updated_at = NOW()
WHERE id = $2 AND version = $3;  -- if version doesn't match, update count = 0

If the update count is 0, return ErrVersionConflict.

Designing Terminal States

Every state machine has terminal states. No transitions are possible from terminal states.

case QuoteStatusAccepted, QuoteStatusRejected, QuoteStatusExpired:
    return false // terminal states

However, Payment's completed is not a terminal state. Transitions to refunded and disputed are possible. Because even "completed" payments can have follow-up processing in business terms.

case PaymentStatusCompleted:
    return target == PaymentStatusRefunded || target == PaymentStatusDisputed

"Which states are terminal" is purely a business rules question, not a technical constraint. It's a design decision that requires discussion with domain experts.

When to Use This Pattern

Not every status column needs a state machine.

Situation	State Machine	Reason
2-state ON/OFF (active/inactive)	Not needed	A boolean is sufficient
3+ states with a defined transition order	Needed	There's value in preventing invalid transitions
Multiple entities coordinating	Needed	Guard condition management becomes complex
Transitions with side effects (notifications, billing)	Needed	Centralize side-effect execution conditions

Why not use DB CHECK constraints or triggers? PostgreSQL CHECK constraints can restrict allowed status values, and triggers can enforce transition rules. However, guard conditions ("can't send a quote with no line items", "can't accept an expired quote") depend on application context and can't be fully expressed on the DB side alone. It's practical to split responsibilities: CHECK constraints for status value restriction, and application-side for transition rules and guard conditions.

Why not use a library? Go has state machine libraries (looplab/fsm, etc.). However, the patterns above are sufficient in many cases. The CanTransitionTo() + domain method combination can be implemented without external dependencies, and integrates naturally with guard conditions and business logic. Libraries become useful when you need transition callbacks or state transition persistence (event sourcing).

Summary

Pattern	Overview
Centralize transition rules in enum types	Define allowed transitions in one place with `CanTransitionTo()`
Execute transitions via domain methods	Encapsulate guard conditions in `Quote.Send()`, `Invoice.MarkPaid()`
State-dependent operation restrictions	Use `IsModifiable()`, `IsCancellable()` for UI control as well
Centralized guard conditions	Aggregate multi-condition checks in `CanConvertToOrder()`
Factory methods for conversion	Type-safe entity conversion with `NewOrderFromQuote()`
Automatic transitions	Automatically transition to `paid` on full payment within `RecordPayment()`

Managing "status as strings" is easy but breaks down as entities multiply. Embedding state machines into domain models prevents invalid transitions at the type level and avoids scattered transition rules.

Particularly in SaaS commerce where multiple state machines coordinate, a design where each entity is responsible for its own transition rules while safely coordinating through guard conditions and factory methods is effective.

Series Articles

Evaluating GitHub Agentic Workflows — From a Claude Code User's Perspective [Part 10]

ko-chan — Tue, 17 Feb 2026 12:40:03 +0000

This article was originally published on Saru Blog.

What You Will Learn

What GitHub Agentic Workflows are
How they differ from traditional GitHub Actions
Benefits for Claude Code users
What can be automated in a 200K-line SaaS project
Key factors in the adoption decision

What Are GitHub Agentic Workflows?

On February 13, 2026, GitHub released this as a technical preview. Co-developed by GitHub Next, Microsoft Research, and Azure Core Upstream, it's open source under the MIT license.

In short, a mechanism for automatically running AI coding agents on GitHub Actions.

Traditional GitHub Actions strictly define "when X happens, do Y" in YAML. Agentic Workflows write "when X happens, make this kind of judgment" in Markdown. The AI makes the judgment.

Traditional GitHub Actions:
  Event → YAML-defined steps → Deterministic execution

Agentic Workflows:
  Event → Markdown-described objectives → AI judges and executes

How It Works

Workflow Definition

Place Markdown files in .github/workflows/. Markdown, not YAML.

---
on:
  issues: opened
permissions:
  contents: read
  issues: write
safe-outputs:
  add-comment: true
  add-labels: true
engine: claude
---

## Issue Triage

When a new issue is created, analyze its content and apply appropriate labels.

## Criteria

- Bug report → `bug` label
- Feature request → `enhancement` label
- Question → `question` label
- Security-related → `security` label + raise priority

## Comments

Leave triage results as a comment.

The frontmatter specifies the trigger, permissions, and AI engine to use. The body describes "what you want done" in natural language.

Compilation and Execution

# Compile with CLI (generates lock.yml from Markdown)
gh aw compile

# Manual trigger
gh aw run

gh aw compile parses the Markdown and generates a .lock.yml for GitHub Actions. This lock file is the actual workflow that runs. The Markdown is the human-readable specification; the lock.yml is the machine-executable procedure.

Available AI Engines

Engine	Authentication	Notes
Copilot CLI	Account auth tied to Copilot license	Default
Claude Code	`ANTHROPIC_API_KEY`	Requires Anthropic API key
OpenAI Codex	`OPENAI_API_KEY`	Requires OpenAI API key

The ability to choose Claude Code as the engine makes it a natural choice for developers already using Claude Code.

Differences from Traditional GitHub Actions

Aspect	GitHub Actions (YAML)	Agentic Workflows (Markdown)
Definition	YAML (strict syntax)	Markdown (natural language)
Execution nature	Deterministic (same input → same output)	Non-deterministic (AI judges)
Best suited for	Builds, tests, deploys	Triage, reviews, reports
Permissions	Specified in workflow definition	Read-only by default + safe outputs
Error handling	Explicitly defined	AI judges

The important point is that Agentic Workflows are not a replacement for CI/CD. GitHub's official blog states this explicitly:

Don't use agentic workflows as a replacement for GitHub Actions YAML workflows for CI/CD.

Builds, tests, and deploys remain with traditional YAML workflows. Agentic Workflows handle "ambiguous tasks" requiring AI judgment. Under the concept of "Continuous AI," they complement existing CI/CD.

Considering Application to the Saru Project

Current Automation

Saru already has the following automated via GitHub Actions:

Workflow	Purpose	Type
build-apis.yml	Go lint, unit tests, integration tests	CI (YAML)
build-portals.yml	Frontend type-check, lint, build	CI (YAML)
e2e-tests.yml	E2E tests for all portals	CI (YAML)
security-scan.yml	gosec, npm audit	CI (YAML)
cross-post.yml	Blog cross-posting to platforms	CD (YAML)

These are all deterministic processes — no reason to replace them with Agentic Workflows.

What Could Be Automated with Agentic Workflows

So what can be automated? Let me identify "manual tasks that require judgment."

1. Automatic Issue Triage

Current state: After creating an issue, I manually add labels and set priority. As a solo developer, I'm the only one doing this.

With Agentic Workflows: Trigger on issue creation to automatically analyze content, apply labels, set priority, and identify related files.

Assessment: Low impact for solo development. Little need to triage issues I wrote myself. Would be effective once the project goes OSS and external issues increase.

2. Automatic CI Failure Investigation

Current state: When CI fails, I read logs, investigate the cause, and fix it. As covered in Part 7, CI stabilization required enormous effort.

With Agentic Workflows: Trigger on CI failure to analyze logs, identify root causes, and automatically create fix PRs.

Assessment: The most compelling use case. Especially for E2E test flaky failures where root cause identification takes time. Even just having AI do the initial investigation would save significant time.

3. Automatic Dependabot PR Triage

Current state: When Dependabot PRs pile up, I review each one individually before merging.

With Agentic Workflows: Trigger on Dependabot PRs to review changes and make judgments: "patch version + tests pass → auto-merge," "major version → add needs-manual-review label."

Assessment: Effective. Dependabot PR handling is monotonous yet requires judgment — exactly what Agentic Workflows excel at.

4. Daily Status Report

Current state: None. Development status exists only in my head.

With Agentic Workflows: Auto-generate reports on daily issue/PR status, CI health, and outstanding items.

Assessment: Overkill for solo development. Would be effective for team development or when the project has OSS contributors.

Application Summary

Use Case	Impact	Priority
CI failure investigation	High	◎
Dependabot PR triage	Medium	○
Issue triage	Low (solo phase)	△
Daily status report	Low (solo phase)	△

Concerns About Adoption

1. Cost

Running Agentic Workflows incurs AI engine API calls.

Copilot: ~2 premium requests per execution (agent execution + safe outputs)
Claude Code: API billing via ANTHROPIC_API_KEY
Codex: API billing via OPENAI_API_KEY

If AI runs on every CI failure, monthly costs become unpredictable. E2E tests especially have many jobs, so failure frequency × API cost must be estimated.

2. Technical Preview Instability

As of February 2026, it's still a technical preview. GitHub's official documentation explicitly states "at your own risk." Too early to integrate into production CI/CD pipelines.

Documentation is still developing — details around Markdown frontmatter specifications and engine configuration require some trial-and-error exploration.

3. Trust in Non-Deterministic Execution

In the CI/CD world, "same input → same output" is a fundamental principle. Agentic Workflows are inherently non-deterministic — AI judgment may differ each time.

Safe outputs and read-only defaults provide safety margins, but handling cases like "AI applied the wrong label" or "created an irrelevant fix PR" becomes necessary.

4. Compatibility with Self-Hosted Runners

Saru runs parallel E2E tests on 15 self-hosted runners. Whether Agentic Workflows function correctly on self-hosted runners is unverified. Official documentation mostly assumes GitHub-hosted runners.

5. Coexistence with Claude Code CLI

This is the most important consideration. Saru already uses Claude Code CLI locally for development. If Claude Code also runs automatically on GitHub, clear role separation becomes essential:

Local development:
  Human + Claude Code CLI → Code implementation, test creation

On GitHub:
  Copilot → PR review (already in use)
  Agentic Workflows → CI failure investigation, triage (under consideration)

Multiple AIs operating on the same repository with different contexts requires clearly defined roles to avoid confusion.

Next Steps

This article stays at the investigation and evaluation level. In the next article, I plan to actually implement Agentic Workflows in the Saru repository and verify:

Building a CI failure auto-investigation workflow
Execution with the Claude Code engine
Operation on self-hosted runners
Actual cost measurement

Summary

Item	Detail
What are GitHub Agentic Workflows	A mechanism for auto-running AI agents on GitHub Actions
Definition method	Natural language in Markdown, not YAML
AI engines	Copilot CLI / Claude Code / OpenAI Codex
Relationship with CI/CD	Complement, not replacement (Continuous AI)
Effective use cases for solo dev	CI failure investigation, Dependabot PR triage
Current judgment	Worth evaluating, but too early for production given technical preview status

Series Articles

pnpm + Next.js Standalone + Docker: 5 Failures Before Success [Part 9]

ko-chan — Mon, 16 Feb 2026 14:22:21 +0000

This article was originally published on Saru Blog.

What You Will Learn

Why pnpm symlinks break in Next.js standalone Docker builds
When cp -rL is not enough
Symlink resolution patterns in Docker multi-stage builds
Lessons from 5 consecutive fix PRs

Background

Saru manages 5 Next.js frontends (Landing / System / Provider / Reseller / Consumer) in a pnpm monorepo. In development, we use volume mounts so Dockerfiles are not needed. But deploying to production or demo environments requires Docker images.

Next.js output: 'standalone' traces only the necessary files into .next/standalone/. Copy this into an Alpine-based runner stage and you get a lightweight image — or so I thought.

Failure 1: MODULE_NOT_FOUND (#557)

Symptom

Error: Cannot find module 'next/dist/compiled/next-server/app-page.runtime.prod.js'

Container crashes immediately on docker run.

Cause

pnpm builds node_modules using symlinks. For example:

standalone/node_modules/next → ../../node_modules/.pnpm/next@14.2.x/node_modules/next

Next.js's @vercel/nft (Node File Tracing) copies this symlink structure as-is into the standalone output. Inside the builder stage, the symlink targets exist, so everything works. But when COPY --from=builder brings this to the runner stage, the symlink targets don't exist.

builder (when standalone is created)    runner (after COPY)
├── standalone/                         ├── standalone/
│   └── node_modules/                   │   └── node_modules/
│       └── next → ../../.pnpm/...      │       └── next → ../../.pnpm/...
└── node_modules/.pnpm/... ✅ exists    └── (nothing) ❌

Attempted Fix

RUN cp -rL /app/apps/system/.next/standalone /app/standalone

cp -rL follows symlinks and copies real files. This should solve it in one command — I thought.

Result: ❌ Failed

Failure 2: Dangling Symlinks (#560)

Symptom

cp: can't stat '/app/apps/system/.next/standalone/node_modules/next': No such file or directory

cp -rL itself fails with an error.

Cause

Some symlinks in the standalone node_modules point to pnpm's virtual store outside the standalone directory. Symlinks pointing to paths not included in standalone become "dangling symlinks." cp -rL stops when it encounters a dangling symlink.

standalone/node_modules/
├── next → ../../node_modules/.pnpm/next@14.2.x/node_modules/next  ← dangling
├── react → ./react (real file exists in standalone)                ← OK
└── ...

Attempted Fix

"If bulk copy doesn't work, resolve one by one."

RUN cd /app/apps/system/.next/standalone/node_modules \
    && for mod in *; do \
         [ -L "$mod" ] || continue; \
         target=$(readlink -f "$mod" 2>/dev/null); \
         if [ -e "$target" ]; then \
           rm "$mod" && cp -r "$target" "$mod"; \
         else \
           rm "$mod"; \
         fi; \
       done

Check each symlink individually — if the target exists, copy it; if dangling, skip it.

Result: ❌ Failed (different reason)

Failure 3: Path Mismatch After Copy (#561)

Symptom

The build appeared to succeed, but MODULE_NOT_FOUND again — this time for different modules.

Cause

The Failure 2 approach resolved symlinks after copying to /app/standalone. But pnpm's symlinks use relative paths:

next → ../../node_modules/.pnpm/next@14.2.x/node_modules/next

This relative path resolves correctly from /app/apps/system/.next/standalone/node_modules/, but from /app/standalone/node_modules/ it points to a different location.

Additionally, some modules only existed in the root .pnpm store, not at the app level.

Attempted Fix

"Don't resolve after copying — resolve at the original location (where paths are valid), then copy."

RUN cd /app/apps/system/.next/standalone/node_modules \
    && for mod in *; do \
         [ -L "$mod" ] || continue; \
         target=$(readlink -f "$mod" 2>/dev/null); \
         if [ -e "$target" ]; then \
           rm "$mod" && cp -r "$target" "$mod"; \
         else \
           rm "$mod"; \
           real=$(find /app/node_modules/.pnpm -path "*/$mod/package.json" \
                  ! -path "*/node_modules/*/node_modules/*" 2>/dev/null | head -1); \
           [ -n "$real" ] && cp -r "$(dirname "$real")" "$mod" || true; \
         fi; \
       done \
    && cp -r /app/apps/system/.next/standalone /app/standalone

For dangling symlinks, search the root .pnpm store with find and copy from there.

Result: ❌ Failed (yet another reason)

Failure 4: Transitive Deps and Scoped Packages (#562)

Symptom

Error: Cannot find module 'styled-jsx'
Error: Cannot find module '@swc/helpers'

next itself was found, but styled-jsx and @swc/helpers — dependencies of next — were missing.

Cause

pnpm's store structure has an important characteristic:

node_modules/.pnpm/next@14.2.x/
└── node_modules/
    ├── next/           ← the package itself
    ├── styled-jsx/     ← next's dependency (sibling)
    ├── @swc/
    │   └── helpers/    ← scoped package dependency (sibling)
    └── react/          ← next's dependency (sibling)

pnpm places a package's dependencies as siblings in the same directory. The Failure 3 approach only copied the module itself, missing the siblings (transitive dependencies).

Furthermore, scoped packages like @swc/helpers live under an @swc/ directory that contains its own symlinks. A simple cp -r breaks these internal symlinks.

Fix

"Don't copy just the module — copy all siblings from the store entry using cp -rL."

RUN cd /app/apps/system/.next/standalone/node_modules \
    && for mod in *; do \
         [ -L "$mod" ] || continue; \
         rm "$mod"; \
         pkg=$(find /app/node_modules/.pnpm \
               -path "*/node_modules/$mod/package.json" 2>/dev/null | head -1); \
         if [ -n "$pkg" ]; then \
           store_nm=$(dirname "$(dirname "$pkg")"); \
           for dep in "$store_nm"/*; do \
             dep_name=$(basename "$dep"); \
             [ -e "$dep_name" ] && ! [ -L "$dep_name" ] && continue; \
             [ -L "$dep_name" ] && rm "$dep_name"; \
             cp -rL "$dep" "$dep_name" 2>/dev/null || true; \
           done; \
         fi; \
       done \
    && cp -r /app/apps/system/.next/standalone /app/standalone

Key points:

When a symlink is found, locate the module in the pnpm store
Once found, copy all siblings in that store entry's directory
Use cp -rL to also resolve nested symlinks within siblings

Result: ✅ MODULE_NOT_FOUND resolved

Failure 5: Static Asset 404s (#565)

Symptom

The container starts. HTTP requests get responses. But opening the page shows all CSS/JS returning 404.

Cause

# Before
COPY --from=builder --chown=nextjs:nodejs /app/apps/system/.next/static ./apps/system/.next/static

Next.js standalone's server.js looks for .next/static relative to its own directory — expecting /app/.next/static. But the Dockerfile was copying to /app/apps/system/.next/static, preserving the monorepo path structure.

Fix

# After
COPY --from=builder --chown=nextjs:nodejs /app/apps/system/.next/static ./.next/static

A one-line change. But finding this required checking 404s in the browser DevTools and reading the server.js source.

Result: ✅ Fully working

The Final Dockerfile

FROM node:20-alpine AS base

FROM base AS deps
RUN apk add --no-cache libc6-compat
WORKDIR /app
RUN corepack enable && corepack prepare pnpm@9.0.0 --activate
COPY pnpm-workspace.yaml package.json pnpm-lock.yaml* ./
COPY turbo.json ./
COPY apps/system/package.json ./apps/system/
# Copy all workspace package.json files
COPY packages/types/package.json ./packages/types/
COPY packages/ui/package.json ./packages/ui/
COPY packages/auth/package.json ./packages/auth/
COPY packages/api-client/package.json ./packages/api-client/
COPY packages/config/package.json ./packages/config/
COPY packages/env-validator/package.json ./packages/env-validator/
COPY packages/logger/package.json ./packages/logger/
RUN pnpm install --frozen-lockfile

FROM base AS builder
WORKDIR /app
RUN corepack enable && corepack prepare pnpm@9.0.0 --activate
COPY --from=deps /app/node_modules ./node_modules
COPY --from=deps /app/apps/system/node_modules ./apps/system/node_modules
COPY --from=deps /app/packages ./packages
COPY apps/system ./apps/system
COPY packages ./packages
COPY turbo.json pnpm-workspace.yaml package.json ./
ENV NEXT_TELEMETRY_DISABLED=1
ENV NODE_ENV=production
RUN pnpm turbo run build --filter=system

# Resolve pnpm symlinks (the core of this article)
RUN cd /app/apps/system/.next/standalone/node_modules \
    && for mod in *; do \
         [ -L "$mod" ] || continue; \
         rm "$mod"; \
         pkg=$(find /app/node_modules/.pnpm \
               -path "*/node_modules/$mod/package.json" 2>/dev/null | head -1); \
         if [ -n "$pkg" ]; then \
           store_nm=$(dirname "$(dirname "$pkg")"); \
           for dep in "$store_nm"/*; do \
             dep_name=$(basename "$dep"); \
             [ -e "$dep_name" ] && ! [ -L "$dep_name" ] && continue; \
             [ -L "$dep_name" ] && rm "$dep_name"; \
             cp -rL "$dep" "$dep_name" 2>/dev/null || true; \
           done; \
         fi; \
       done \
    && cp -r /app/apps/system/.next/standalone /app/standalone

FROM base AS runner
WORKDIR /app
ENV NODE_ENV=production
ENV NEXT_TELEMETRY_DISABLED=1
RUN addgroup --system --gid 1001 nodejs
RUN adduser --system --uid 1001 nextjs

COPY --from=builder /app/apps/system/public ./public
COPY --from=builder --chown=nextjs:nodejs /app/standalone ./
COPY --from=builder --chown=nextjs:nodejs /app/apps/system/.next/static ./.next/static

USER nextjs
EXPOSE 3000
ENV PORT=3000
ENV HOSTNAME="0.0.0.0"
CMD ["node", "server.js"]

Why 5 Failures?

Looking back, the root cause was insufficient understanding of pnpm's symlink structure.

Failure	Wrong Assumption
1st	Docker will resolve symlinks when COPYing
2nd	`cp -rL` will handle everything in one shot
3rd	Relative symlink paths will work after copying to a new location
4th	Copying just the module itself is enough for its dependencies
5th	Standalone preserves the monorepo directory structure

Each time I proceeded on assumption and only discovered the failure after actually building and running the Docker image.

Alternatives Considered

For reference, here are the alternatives I evaluated and why they were rejected.

Approach	Why Rejected
`node-linker=hoisted` (.npmrc)	Tried before, failed. Also requires `outputFileTracingRoot` in next.config.js
`pnpm deploy`	Workspace packages use raw TypeScript (relying on `transpilePackages`), path structure breaks
Copy all node_modules	Image size goes from ~50MB to ~500MB+

In the end, manually resolving standalone symlinks via shell script was the most reliable approach. Not elegant, but it works.

Lessons Learned

pnpm symlinks use relative paths. Resolving them after copying to a different location breaks paths. Always resolve at the original location.
pnpm's store structure is nested. A package's dependencies are placed as siblings in the same directory. Copying just one module leaves its dependencies behind.
Next.js standalone output is flat. The monorepo's apps/xxx/ structure is not preserved. server.js is placed at the standalone root.
cp -rL is not universal. It fails on dangling symlinks. Be prepared to handle them individually.
Docker production builds are hard to test locally. In development, volume mounts work fine, so symlink issues only surface during Docker build.

Summary

Item	Detail
Problem	pnpm symlinks break in Docker multi-stage builds
Cause	Next.js standalone copies symlink structure as-is
Solution	cp -rL all siblings from pnpm store entries, then COPY
Fix attempts	5 (#557 → #560 → #561 → #562 → #565)
Lesson	Resolve symlinks at source, copy siblings not just the module

Series Articles

Part 1: Fighting Unmaintainable Complexity with Automation
Part 2: Automating WebAuthn Tests in CI
Part 3: Next.js x Go Monorepo Architecture
Part 4: Multi-Tenant Isolation with PostgreSQL RLS
Part 5: Multi-Portal Authentication Pitfalls
Part 6: Developing a 200K-Line SaaS Alone with Claude Code
Part 7: Landmines and Solutions in Self-Hosted CI/CD
Part 8: Turning Solo Development into Team Development with Claude Code Agent Teams
Part 9: pnpm + Next.js Standalone + Docker: 5 Failures Before Success (this article)

Landmines and Solutions in Self-Hosted CI/CD: 15 Runners x Shared Docker Environment [Part 7]

ko-chan — Sun, 15 Feb 2026 09:04:31 +0000

This article was originally published on Saru Blog.

What You Will Learn

Problem and solution patterns in self-hosted runner environments
How to prevent resource contention on a shared Docker daemon
The port assignment problem killed by the birthday paradox
How to investigate when "CI that was working suddenly breaks"

Introduction

In Part 2, I wrote about automating E2E tests using WebAuthn and Mailpit. The tests themselves work fine. The problem was the CI infrastructure.

Saru has 4 frontends x 4 backend APIs. E2E tests run independently for each portal, plus there are cross-portal tests (integration tests between portals). Running these in parallel means 7+ jobs executing simultaneously.

Initially, I used GitHub-hosted runners, but E2E tests require a database, Keycloak, and a mail server—lots of Docker containers. GitHub-hosted runners are slow to set up each time and cost-inefficient for parallel execution.

So I migrated to self-hosted runners. The decision was correct, but a flood of new problems emerged.

This article chronicles the problems encountered in a self-hosted CI environment and their solutions, in chronological order. I hope it helps anyone adopting a similar configuration.

1. Docker Desktop/WSL2 Was Too Unstable

The Initial Setup

I initially ran runners on Docker Desktop (WSL2 backend) on Windows. The setup was simple:

Windows Host
  └─ WSL2
      └─ Docker Desktop
          └─ GitHub Actions Runner x N

The problem: "containers randomly die." During E2E tests, the PostgreSQL container would suddenly vanish, or Keycloak would become unresponsive. docker inspect showed Exit Code: 137 (SIGKILL).

Tracing the cause led to Docker Desktop/WSL2's virtualization layer:

Container → Docker Engine → WSL2 → Hyper-V → Windows

WSL2 itself runs as a lightweight Hyper-V VM, with Docker stacked on top. When memory pressure rises, WSL2 triggers the OOM Killer, indiscriminately killing Docker containers.

Migration to Hyper-V VM

The solution was to bypass WSL2 and create an Ubuntu VM directly on Hyper-V:

Container → Docker Engine → Ubuntu VM → Hyper-V → Windows

Item	Value
VM Name	saru-ci-runner
OS	Ubuntu 24.04
vCPU	16
Memory	64GB
Disk	200GB
Network	External Switch (bridged)

Hyper-V VM has fewer virtualization layers and more stable memory management than WSL2. WSL2 uses dynamic memory allocation "shared with the host" (defaulting to 50% of host RAM or up to 8GB), while Hyper-V VM allocates fixed memory, reducing the risk of OOM Killer strikes.

On top of this, I deployed 15 GitHub Actions Runners as systemd services:

# saru-hyperv-1 through saru-hyperv-15
for i in $(seq 1 15); do
  sudo systemctl status actions.runner.ko-chan-saru.saru-hyperv-$i
done

15 runners sharing a single Docker daemon. This "sharing" would later cause many problems.

2. Port Collisions via the Birthday Paradox

Problem

E2E tests have each job start its own PostgreSQL, Keycloak, frontend, and backend. To avoid port collisions, I calculated port numbers from the GitHub Actions RUN_ID:

# Initial implementation (problematic)
PORT_OFFSET=$(( RUN_ID % 3000 ))
POSTGRES_PORT=$(( 10000 + PORT_OFFSET ))

This looks fine, but since 15 runners share a single Docker daemon, ports from simultaneously running jobs can collide.

This has the same structure as the birthday paradox. With 3000 possible port offsets and 5 concurrent jobs, the collision probability is about 0.33% (1 - 3000!/(3000^5 × 2995!)). Seems trivial, but when CI runs dozens of times per day, collisions happen several times a week. And when ports collide, you get the cryptic error "container started but service unreachable."

Solution: RUNNER_NAME-based allocation

Instead of the random RUN_ID, I switched to deterministic port assignment from the runner name:

# Extract runner number from name (e.g., saru-hyperv-7 → 7)
if [[ "${RUNNER_NAME}" =~ saru-hyperv-([0-9]+) ]]; then
  RUNNER_NUM=${BASH_REMATCH[1]}
else
  RUNNER_NUM=$(( (RUN_ID % 15) + 1 ))
fi

# Allocate 200-port blocks per runner
RUNNER_BLOCK=$((RUNNER_NUM * 200))
PORTAL_OFFSET=$((PORTAL_INDEX * 10))

# Frontend/Backend: 20000 + (RUNNER_NUM × 200) + (PORTAL_INDEX × 10) + {0,1,2,3}
BASE_PORT=20000
OFFSET=$((RUNNER_BLOCK + PORTAL_OFFSET))
PORTAL_PORT=$((BASE_PORT + OFFSET + 1))
API_PORT=$((BASE_PORT + OFFSET + 2))

# Infra: 30000 + (RUNNER_NUM × 1000) + {0,100,200,...} + PORTAL_INDEX
BASE_INFRA_PORT=30000
INFRA_RUNNER_BLOCK=$((RUNNER_NUM * 1000))
POSTGRES_PORT=$((BASE_INFRA_PORT + INFRA_RUNNER_BLOCK + PORTAL_INDEX))
KEYCLOAK_PORT=$((BASE_INFRA_PORT + INFRA_RUNNER_BLOCK + 100 + PORTAL_INDEX))

The key insight: "each runner executes only one job at a time." Since the runner number uniquely determines the port block, collisions are impossible by design:

Runner	Frontend Range	Infra Range
saru-hyperv-1	20200–20313	31000–31510
saru-hyperv-2	20400–20513	32000–32510
...	...	...
saru-hyperv-15	23000–23113	45000–45510

All ports confirmed to fit within 65535.

3. `docker system prune` Kills Other Jobs' Containers

Problem

I had Docker cleanup at the end of each CI job:

# ⚠️ This was the problem
- name: Cleanup
  run: docker system prune -f

docker system prune deletes all stopped containers. Since 15 runners share a single Docker daemon, one job's cleanup can destroy containers another job is actively using.

Especially tricky is the timing right after container startup. If prune runs after Docker Compose or Run starts a container but before the health check passes, the starting container is judged "stopped" and deleted.

Solution: Targeted cleanup

# ✅ Safe cleanup
# docker system prune and docker container prune are FORBIDDEN
# They destroy concurrent jobs' containers
# Only remove dangling images older than 24 hours
docker image prune -f --filter "until=24h" 2>/dev/null || true

Container deletion targets only those tied to your own RUN_ID. Name patterns filter out persistent containers (saru-postgres-integ, etc.):

# Delete only this job's containers (protect persistent ones)
RUN_ID="${{ github.run_id }}"
for container in $(docker ps -a --format '{{.Names}}' \
  | { grep "^saru-" | grep -v "saru-postgres-integ\|saru-keycloak-dev\|saru-mailpit-dev" || true; }); do
  if [[ ! "$container" =~ "${RUN_ID}" ]]; then
    docker rm -f "$container" 2>/dev/null || true
  fi
done

Lesson: In shared Docker daemon environments, docker system prune is a banned weapon. Always use scoped deletion.

4. PostgreSQL Silently Crashes from Shared Memory Exhaustion

Problem

PostgreSQL containers in CI started crashing immediately after startup. pg_isready succeeds momentarily, but the following psql command returns "container is not running":

✓ pg_isready -U test → success
✗ psql -U test -c "CREATE DATABASE ..." → container is not running

This is extremely confusing. PostgreSQL internally restarts after initdb, so if pg_isready succeeds right before that restart, the process no longer exists when the next command runs.

But the real cause was different. Docker's default /dev/shm size (64MB) is insufficient for PostgreSQL.

Solution

docker run -d \
  --name saru-postgres-ci \
  --shm-size=256m \  # ← This is critical
  --restart=unless-stopped \
  -e POSTGRES_USER=test \
  -e POSTGRES_PASSWORD=test \
  -p 15432:5432 \
  --health-cmd "pg_isready -U test" \
  postgres:16-alpine \
  postgres -c max_connections=200

Specifying --shm-size=256m ensures adequate shared memory for PostgreSQL. Parallel tests (-parallel 2 or higher) especially need more shared buffers, and 64MB is not enough.

This was an intermittent issue, only reproducing under high test load. Root cause identification took a full day.

Whether OOM Killer is the cause can be verified with docker inspect:

docker inspect "${CONTAINER}" --format '{{.State.OOMKilled}}'
# true means memory exhaustion was the cause

5. The Persistent PostgreSQL Container Pattern

Problem

Initially, each job started and stopped its own PostgreSQL container. However:

Container startup takes 10–15 seconds each time
Ports are not released immediately on stop, causing collisions on next startup
Container lifecycle management becomes complex (forgotten stops, zombie containers, etc.)

Solution: Persistent Container + Per-Job Database

Combined with the --shm-size=256m from section 4, I switched to keeping the container running permanently and creating/dropping temporary databases per job:

# Start PostgreSQL container if not running (first time only)
- name: Start persistent PostgreSQL
  run: |
    POSTGRES_CONTAINER="saru-postgres-integ"
    if ! docker ps --format '{{.Names}}' | grep -qx "${POSTGRES_CONTAINER}"; then
      docker run -d \
        --name "${POSTGRES_CONTAINER}" \
        --shm-size=256m \
        --restart=unless-stopped \
        -e POSTGRES_USER=test \
        -p 15432:5432 \
        postgres:16-alpine
    fi

    # Create database for this job
    DB_NAME="integ_${{ github.run_id }}"
    docker exec "${POSTGRES_CONTAINER}" \
      psql -U test -h 127.0.0.1 -c "CREATE DATABASE \"${DB_NAME}\" OWNER test;"

# Delete only the database at job end
- name: Cleanup database
  if: always()
  run: |
    docker exec saru-postgres-integ \
      psql -U test -h 127.0.0.1 \
      -c "DROP DATABASE IF EXISTS \"integ_${{ github.run_id }}\";"

    # Also clean up stale databases from past failed jobs
    STALE_DBS=$(docker exec saru-postgres-integ psql -U test \
      -d postgres -h 127.0.0.1 -tAc \
      "SELECT datname FROM pg_database WHERE datname LIKE 'integ_%';")
    for DB in $STALE_DBS; do
      docker exec saru-postgres-integ psql -U test \
        -d postgres -h 127.0.0.1 \
        -c "DROP DATABASE IF EXISTS \"${DB}\";"
    done

Three key points:

--restart=unless-stopped: Container auto-recovers even when VM restarts
Job ID as database name: Prevents interference between concurrent jobs
Stale database cleanup: Periodically removes garbage left by failed jobs

6. Why Force TCP Connections in psql

Problem

When executing psql inside a PostgreSQL container without specifying the connection method, Unix sockets are used by default. However, PostgreSQL has an initdb → restart cycle on first startup, during which the Unix socket briefly disappears:

# ⚠️ Unix socket (default): may fail during restart
docker exec postgres psql -U test -c "SELECT 1"

# ✅ TCP connection: retries work during restart
docker exec postgres psql -U test -h 127.0.0.1 -c "SELECT 1"

Adding -h 127.0.0.1 forces TCP connection, making connection failure errors clearer (a definitive "connection refused" rather than an ambiguous "socket file not found").

Lesson: Always add -h 127.0.0.1 to psql calls in CI scripts.

7. Docker Network Pool Exhaustion

Problem

One day, all E2E jobs suddenly started failing. Error message:

Error response from daemon: could not find an available,
non-overlapping IPv4 address pool among the defaults to
assign to the network

Docker Compose creates a bridge network per project. Docker allocates /16 subnets from the default range 172.17.0.0/16 through 172.31.0.0/16, limiting available networks to about 30. When 15 runners simultaneously run E2E tests and each job creates multiple networks, this pool runs dry.

Solution

# Delete old networks (protect current RUN_ID's)
RUN_ID="${{ github.run_id }}"
for net in $(docker network ls --format '{{.Name}}' | { grep "^saru-ci-" || true; }); do
  if [[ ! "$net" =~ "${RUN_ID}" ]]; then
    containers=$(docker network inspect "$net" \
      --format '{{len .Containers}}' 2>/dev/null || echo "in-use")
    if [ "$containers" = "0" ]; then
      docker network rm "$net" 2>/dev/null || true
    fi
  fi
done

Delete unused old networks at the start of each job. Instead of docker network prune, filter by name and only remove those with "0 connected containers."

8. Preventing OTP Contention

Problem

E2E tests include OTP (one-time password) authentication. Each E2E job shares the same Mailpit instance, searching Mailpit's API for emails to retrieve OTP codes.

The problem: when multiple jobs log in simultaneously with the same email address (e.g., system-admin@saru.local), multiple OTP emails for the same recipient arrive in Mailpit. Timestamp filtering helps somewhat, but millisecond-level contention cannot be fully prevented.

Solution: Per-Job Email Addresses

matrix:
  portal:
    - name: system-auth
      system_email: "system-admin@saru.local"
    - name: system-entities
      system_email: "system-entities@saru.local"
    - name: system-products
      system_email: "system-products@saru.local"
    - name: system-misc
      system_email: "system-misc@saru.local"

Each job uses a different system admin account (email address), eliminating OTP email retrieval contention. Multiple system admin accounts are registered in the backend seed data.

9. The hashFiles Syntax Gotcha

Problem

I got stuck trying to use dynamic paths with GitHub Actions' hashFiles():

# ⚠️ This doesn't work (string concatenation can't be used inside hashFiles)
key: ${{ runner.os }}-turbo-${{ hashFiles('apps/' + matrix.app + '/**') }}

hashFiles() arguments only accept literal strings. Expressions are not evaluated.

Solution

# ✅ Use the format() helper
key: ${{ runner.os }}-turbo-${{ matrix.app }}-${{ hashFiles(format('apps/{0}/**', matrix.app), 'packages/**', 'pnpm-lock.yaml', 'turbo.json') }}

Use format() to build the path first, then pass the result to hashFiles(). This is not in the GitHub Actions documentation—I found it in a community discussion (#25718).

10. Migration Round-Trip Testing

CI tests a "migration round-trip" every time:

# Up → Down 1 step → Up again
DATABASE_URL="..." go run ./cmd/migrate -action up
DATABASE_URL="..." go run ./cmd/migrate -action down -steps 1
DATABASE_URL="..." go run ./cmd/migrate -action up

This ensures:

Down migrations are not broken
The "can never Up again after Up→Down" pattern is detected
Safety for production rollbacks is guaranteed

11. Automatic Diagnostics on Failure

To make root cause identification easier when CI fails, diagnostic information is collected in if: failure() steps:

- name: Diagnose PostgreSQL on failure
  if: failure()
  run: |
    # Container state
    docker inspect "${POSTGRES_CONTAINER}" --format '{{.State.Status}}'

    # Active connections
    docker exec "${POSTGRES_CONTAINER}" psql -U test -h 127.0.0.1 \
      -c "SELECT datname, count(*) FROM pg_stat_activity GROUP BY datname;"

    # Container logs (last 30 lines)
    docker logs "${POSTGRES_CONTAINER}" --tail 30

    # Host memory status
    free -h

Just having this breaks the loop of "CI failed → check logs → can't find the cause → re-run and pray."

12. Disk Management

Since 15 runners share the same 200GB disk, disk management is critical:

strategy:
  max-parallel: 2  # Limit concurrency to prevent disk exhaustion

Each job consumes about 2GB for node_modules installation, frontend builds, Playwright browser cache, etc. 8 concurrent jobs means 16GB, plus Docker images and build cache—200GB fills up quickly.

A periodic cleanup script is also prepared:

Target	Retention
CI artifacts	7 days
Runner _temp	1 day
.turbo cache	7 days
node_modules	3 days
Go build cache	14 days
Docker images	30 days
Playwright browsers	Keep (essential for E2E)

Summary: Lessons Learned from Self-Hosted CI

Lesson	Details
Shared resources collide	Docker daemon, ports, networks, disk
Determinism over randomness	Port assignment: `RUNNER_NUM`-based over `RUN_ID % N`
`prune` is a weapon	`docker system prune` is forbidden in shared environments
Persistent container + temporary DB	Simplifies container lifecycle management
Force TCP	psql with `-h 127.0.0.1` avoids Unix socket traps
Leave diagnostics on failure	Automate root cause identification with `if: failure()`
Don't trust Docker defaults	Explicitly specify `--shm-size`, `max_connections`

Self-hosted CI brings flexibility and speed that GitHub-hosted cannot match. But it also means accepting the complexity of "managing infrastructure yourself."

In solo development, when CI breaks, you are the only one who can fix it. That is precisely why it was important to pursue the "why" when problems occur and build systems that prevent recurrence. Every solution presented here was born from an actual incident.

Series Articles

Part 1: Fighting Unmaintainable Complexity with Automation
Part 2: Automating WebAuthn Tests in CI
Part 3: Next.js x Go Monorepo Architecture
Part 4: Multi-Tenant Isolation with PostgreSQL RLS
Part 5: Multi-Portal Authentication Pitfalls
Part 6: Developing a 200K-Line SaaS Alone with Claude Code
Part 7: Landmines and Solutions in Self-Hosted CI/CD (this article)
Part 8: Turning Solo Development into Team Development with Claude Code Agent Teams

Turning Solo Development into Team Development with Claude Code Agent Teams [Part 8]

ko-chan — Sun, 15 Feb 2026 09:04:28 +0000

This article was originally published on Saru Blog.

What You Will Learn

How Claude Code Agent Teams work and when to use them
Concrete use cases for solo developers using Agent Teams
Real examples of task lists and team configurations
Differences between regular subagents and Agent Teams, and how to choose

The Wall of Solo Development

In Part 6, I wrote about developing a 220,000-line SaaS application alone using Claude Code. AI functions as "ten extra pairs of hands," rapidly handling everything from specification drafting to implementation and testing.

However, the more I used Claude Code, the more one limitation became apparent.

Claude Code can only do one thing at a time in a single session.

In human team development, Person A implements the API while Person B writes tests and Person C does code review. But Claude Code can only work sequentially in a single session. Even merging 5 PRs had to be processed one at a time.

Agent Teams break this constraint.

What Are Agent Teams?

Agent Teams enable multiple Claude Code instances to work cooperatively as a single team. They were released as a Research Preview in February 2026.

Regular Claude Code:
  1 session ──→ sequential processing

Agent Team:
  Team Lead ──→ Teammate A ──→ parallel processing
             ├→ Teammate B ──→ parallel processing
             └→ Teammate C ──→ parallel processing

The differences from regular subagents (the Task tool) are clear:

Feature	Subagent (Task)	Agent Team
Communication	Reports to parent only	Members can talk directly to each other
Context	Shares parent's context	Each has independent context
Task management	None	Shared task list (with dependencies)
Human intervention	Through parent only	Can instruct each member directly
Use case	Research, one-off delegation	Multi-track parallel work

If subagents are "delegating work to a subordinate and getting results back," Agent Teams feel more like "assembling a team and running a project."

Example 1: PR Triage Team

Background

In Saru's development, feature implementation and CI fixes proceed in parallel, so PRs tend to pile up. One day, I faced this situation:

PRs awaiting merge: 5 (requiring rebase)
Stale worktrees: 12+ (leftovers from merged PRs)
PRs with CI failures: 2 (requiring investigation)

Processing these one by one would take an entire day. I used Agent Teams to clear them all at once.

Task List

The core of Agent Teams is the shared task list. Each task can have dependencies (blocks/blockedBy), allowing team members to autonomously pick up the next available task.

Here is the actual task list structure I used:

[
  {
    "id": "1",
    "subject": "Clean up stale worktrees for merged PRs",
    "description": "Delete 12+ worktrees. Also clean up remote tracking branches for merged PRs",
    "status": "completed"
  },
  {
    "id": "2",
    "subject": "Process PR #408 - fix consumer test timeout",
    "description": "Rebase onto main, investigate E2E CI failure, fix issues, push, request review",
    "status": "completed"
  },
  {
    "id": "3",
    "subject": "Process PR #435 - i18n plan selection page",
    "description": "Rebase onto main, investigate E2E CI failure, fix issues, push, request review",
    "status": "completed"
  },
  {
    "id": "4",
    "subject": "Process PR #521 - reseller/consumer creation E2E tests",
    "description": "Rebase, push, request Copilot re-review. Monitor CI results",
    "status": "completed"
  },
  {
    "id": "5",
    "subject": "Triage and process Dependabot PRs",
    "description": "Triage all Dependabot PRs (already closed)",
    "status": "completed"
  }
]

The key point is to write clear completion criteria in each task's description. Not just "rebase" but "rebase onto main, investigate CI failure, fix issues, push, request review." Vague instructions lead to half-finished work from AI.

Result

Processing 5 PRs + cleaning up 12 worktrees completed without human intervention. All I did was give the initial instructions and press the merge button at the end.

Example 2: CI Stabilization Merge Team

Background

In the final stage of CI stabilization described in Part 7, a series of tasks with complex dependencies emerged:

Merge PR #550 (E2E stability fix)
Merge PR #547 (signup fix) — would conflict without merging #550 first
Rebase 7 PR branches onto main — after #550 and #547 are merged
Force push rebased branches — after rebase completes
Verify CI runs triggered on all PRs — after push
Clean up worktrees — after push

Order matters, and a single mistake cascades into further problems. When doing this manually, it is easy to lose track: "Wait, did I already merge #547?"

Team Configuration

{
  "name": "ci-stability-merge",
  "description": "PR #550 merge, rebase all PRs on main, and re-trigger CI",
  "members": [
    {
      "name": "team-lead",
      "agentType": "team-lead",
      "cwd": "/opt/projects/saru/worktrees/fix-e2e-stability"
    }
  ]
}

Task List with Dependencies

Task 1: Merge PR #550          ── blocks → [2, 3]
Task 2: Merge PR #547          ── blocks → [3], blockedBy → [1]
Task 3: Rebase 7 PR branches   ── blocks → [4], blockedBy → [1, 2]
Task 4: Force push all         ── blocks → [5, 6], blockedBy → [3]
Task 5: Verify CI triggers     ── blockedBy → [4]
Task 6: Clean up worktrees     ── blockedBy → [4]

Tasks 5 and 6 are independent and can run in parallel. Agent Teams see the dependencies and automatically pick up tasks as their blockers are resolved.

This DAG (Directed Acyclic Graph) structure is Agent Teams' greatest weapon. No human needs to think about "what should I do next."

Example 3: Multi-Layered Quality Verification

Separately from Agent Teams, I routinely use specialist agents at each stage of the development workflow.

Orchestration via CLAUDE.md

In Saru, quality verification workflows are defined in CLAUDE.md (the instruction file for Claude Code):

Specification → /qa.verify-spec    → Spec verification
Design        → /qa.verify-design  → Design verification
Implementation → /qa.verify-impl   → Implementation verification
Testing       → /qa.verify-test    → Test verification

Specialist agents are automatically launched at each verification step:

Agent	Role	Trigger
security-engineer	Review RLS policies, authentication/authorization	Spec review, implementation verification
backend-architect	Review API design, data models	Design verification
quality-engineer	Check test coverage and comprehensiveness	Test verification

Value for Solo Developers

In team development, code reviews are done by colleagues. In solo development, there is no one else.

"Writing code and reviewing it yourself" is difficult for humans. Code you just wrote looks correct to you. You are biased.

Specialist agents bring a different perspective. The security-engineer points out "this input is not validated," and the backend-architect notes "this API design violates REST principles."

This is a different kind of value from Agent Teams' parallel execution, but it is an important component of the "alone yet a team" experience.

Example 4: Blog Article Pre-Publication Check

For this blog article and others, I run multiple verifications in parallel before publication:

Article draft complete
  ├→ security-engineer: Information leakage & security risk check
  ├→ Tavily search: Similar article search (plagiarism check)
  └→ Codex: Technical accuracy verification

These are independent of each other, so they can be run as parallel subagent calls. There is no need for Agent Teams — parallel Task tool (subagent) calls are sufficient.

Decision Criteria for Choosing

After hands-on experience, I settled on these decision criteria:

Condition	Choice
Independent research/analysis tasks	Subagent (Task)
Parallel work requiring coordination	Agent Team
Ordered task groups with dependencies	Agent Team (using task list)
One-off specialist review	Subagent
Large-scale refactoring	Agent Team (separate frontend/backend/test)

When in doubt, start with subagents. Agent Teams have overhead (team creation, task list management, inter-member messaging). Using Agent Teams for simple tasks wastes more time on setup than it saves.

Caveats and Limitations

1. High Token Consumption

Agent Teams consume several times the usual number of tokens because each member has an independent context. A 3-person team uses roughly 3x the tokens. Cost awareness is necessary.

2. Context Fragmentation

The Team Lead's conversation history is not inherited by members. All necessary context must be included in the initial instructions to each member. "Continuing from our earlier discussion" does not work.

3. File Conflict Risk

If multiple members edit the same file simultaneously, conflicts arise. You need to either isolate working directories with git worktrees or clearly divide file ownership. In Saru, worktrees are mandatory, so this problem is naturally avoided.

4. Experimental Feature

Agent Teams are in Research Preview as of February 2026. There are rough edges: session resumption is not supported, task status updates are sometimes forgotten, and so on. It is too early to integrate into production deployment pipelines.

Solo Development x Agent Teams = ?

In Part 6, I wrote that "AI provides ten extra pairs of hands, not ten extra brains." With Agent Teams, those hands now move in parallel.

In the world of solo development, Agent Teams function not as replacements for team members but as a work parallelization tool.

Process PR backlogs automatically with dependency tracking
Run quality verification from multiple perspectives simultaneously
Execute complex operations like CI stabilization in the correct order

"One-person team" may sound like a contradiction. But in practice, the most tedious parts of team development (communication, alignment, scheduling) disappear, leaving only the most valuable parts (parallel processing, specialist reviews, dependency management).

This may be the ideal form for solo developers.

Summary

Point	Content
What are Agent Teams	A system where multiple Claude Code instances work cooperatively
Difference from subagents	Direct inter-member communication, shared task list, independent context
Solo development use cases	PR triage, CI stabilization, parallel quality verification
Decision criteria	Use Task for no coordination needed, Agent Team for dependencies
Caveats	Increased token cost, context fragmentation, Research Preview

Series Articles

Part 1: Fighting Unmaintainable Complexity with Automation
Part 2: Automating WebAuthn Tests in CI
Part 3: Next.js x Go Monorepo Architecture
Part 4: Multi-Tenant Isolation with PostgreSQL RLS
Part 5: Multi-Portal Authentication Pitfalls
Part 6: Developing a 200K-Line SaaS Alone with Claude Code
Part 7: Landmines and Solutions in Self-Hosted CI/CD
Part 8: Turning Solo Development into Team Development with Claude Code Agent Teams (this article)

Building a 200K-Line SaaS Solo with Claude Code [Part 6]

ko-chan — Fri, 23 Jan 2026 18:33:47 +0000

This article was originally published on Saru Blog.

What You'll Learn

The reality of large-scale development with Claude Code
What to delegate to AI vs. what requires human judgment
Actual workflow with a 200K-line codebase

The Project in Numbers

Metric	Value
Lines of code	~200K (Go 84K + TypeScript 113K)
Development period	~3 months (Oct 2025–)
Commits	311
Developers	1
Portals	4 (System / Provider / Reseller / Consumer)
APIs	4 (dedicated to each portal)

One person. Three months. 200K lines.

To be honest, Claude Code wrote most of this code. My job is to set direction, review, and make decisions.

What is Claude Code?

A CLI-based AI coding agent from Anthropic. Unlike ChatGPT or Copilot:

Autonomous: Say "implement this feature" and it reads files, writes code, runs tests
Wide context: Understands 200K tokens (~150K lines) at once
CLI-native: No VS Code dependency, works entirely in terminal

Actual Workflow

A Typical Day

Morning: Pick one Issue
  ↓
Claude Code: "Implement this feature"
  ↓
Claude Code: Generates spec.md
  ↓
Me: Review, request changes
  ↓
Claude Code: Generates plan.md
  ↓
Me: Review, approve
  ↓
Claude Code: Implementation
  ↓
Me: Verify behavior, code review
  ↓
Claude Code: Create tests
  ↓
Me: Merge

My actual work time: 2-3 hours per day. The rest of the time, Claude Code is working while I do other things.

Concrete Example: Order Creation Feature

A recent implementation: "Order Creation UI"

My input (one line):

Add the ability to create Orders directly from the UI, without going through Cart/Quote.

Claude Code's output:

Specification (use cases, functional requirements, non-functional requirements)
API design (6 endpoints)
Frontend design (2 portals × 3 pages)
Implementation code (Go + TypeScript)
E2E tests
Internationalization (Japanese/English)

Time: ~4 hours (of which ~1 hour was my work)

What I Delegate to AI

1. Structuring Specifications

Transforming vague requirements into structured specifications.

Before (my input):

I want Providers to be able to create Orders directly for their Consumers

After (Claude Code's output):

## Use Cases
| Scenario | Actor | Portal | Description |
|----------|-------|--------|-------------|
| S2P | Provider | Provider Portal | Internal Order creation |
| P2C | Provider | Provider Portal | Order for Consumer |
| C2C | Consumer | Consumer Portal | Self-service purchase |

## Functional Requirements
- FR-1.1: Provider can create new Order from /orders/new
- FR-1.2: Customer selection (self or from Consumer list)
...

It organizes the vague ideas in my head into comprehensive specifications.

2. Following Existing Patterns

Reading 200K lines and generating code that matches existing patterns.

For example:

API endpoint naming conventions
Error handling patterns
Frontend directory structure
Test writing style

This is the most valuable part. New code stays consistent with existing code. When humans write, personal habits creep in and break codebase uniformity.

3. Repetitive Tasks

With 4 portals × 4 APIs, I often write similar code four times.

Example: Adding a new entity requires:

Handler (4 APIs)
Router config (4 APIs)
Frontend pages (4 portals)
Type definitions (shared package)
E2E tests (4 portals)

Doing this manually takes half a day. Claude Code does it in 30 minutes.

4. Test Creation

Deriving test cases from implementation code. Comprehensively covering edge cases.

// Test cases generated by Claude Code
test('should show error when quantity is 0', ...)
test('should show error when quantity is negative', ...)
test('should show error when quantity exceeds stock', ...)
test('should disable submit button while loading', ...)
test('should redirect to list page after successful creation', ...)

Writing tests is tedious work. Good decision to delegate to AI.

What Requires Human Judgment

1. What to Build

I don't let AI decide "what to implement next." Business priorities, technical debt, user feedback—synthesizing these is human work.

2. Trade-off Decisions

AI presents options, but final decisions are human.

Example: Choosing authentication method

Claude Code: "JWT or session-based?"
Me: "JWT. Because..."

AI can't decide "which is correct." Judgment considering context (team size, operations, future extensibility) can only be made by humans.

3. Final Security Review

RLS (Row-Level Security) policies, authentication/authorization logic—humans always review these.

AI can write "working code," but whether it's "secure code" is a different matter.

4. Deciding "Don't Do It"

AI tries to do what it's asked. "This feature isn't needed" or "this implementation is overkill" are human judgments.

In practice, I reject about 30% of features Claude Code proposes as "not needed now."

Impact on Productivity

Quantitative Changes

Metric	Before (pre-AI experience)	After (Claude Code)
Time per feature	2-3 days	4-6 hours
Daily commits	2-3	5-10
Test coverage	Depends on motivation	Always 80%+

Feels 3-5x faster.

Qualitative Changes

Less tedious work: Boilerplate, tests, documentation
More time for design: Delegate implementation to AI, spend more time on architecture
Maintained consistency: Code style unified across 200K lines

Caveats and Limitations

1. Review is Mandatory

Never merge Claude Code's output directly. Always read it.

In practice, about 1 in 10 times there's something "not quite right." Trust the AI, but don't skip verification.

2. Context Limitations

200K tokens is wide, but can't understand the entire 200K-line codebase at once.

Workarounds: explicitly specify related files, split sessions.

3. Weak on Latest Information

Claude Code's knowledge is frozen at training time. Latest library updates and best practices need human supplementation.

4. It Costs Money

Claude Code isn't free. Tens to hundreds of dollars per month. But considering the productivity gains, it more than pays for itself.

Solo Development × AI Possibilities

Three months ago, saying "I'll build 4 portals × 4 APIs alone" would have seemed insane.

Not anymore. With an AI agent, one person can build enterprise-scale systems.

However, AI is "10 pairs of hands," not "10 brains."

Direction is set by humans
Decisions are made by humans
Responsibility is taken by humans

AI is like an excellent junior engineer. It does whatever you ask, but you need to figure out what to ask.

Summary

Point	Detail
Scale	200K lines, 4 portals × 4 APIs
Period	3 months, 311 commits
AI's role	Structuring specs, following patterns, repetitive tasks, tests
Human's role	Priority decisions, trade-offs, security review, "don't do it" calls
Productivity	Feels 3-5x faster

Even solo, partnering with AI, you can build big things.

Series Articles

Part 1: Tackling Unmaintainable Complexity with Automation
Part 2: WebAuthn Testing in CI with Virtual Authenticators
Part 3: Next.js × Go Monorepo Architecture
Part 4: Multi-Tenant Isolation with PostgreSQL RLS
Part 5: Multi-Portal Authentication Pitfalls with Keycloak
Part 6: Building a 200K-Line SaaS Solo with Claude Code (this article)

3 Pitfalls of Multi-Portal Authentication with Keycloak [Part 5]

ko-chan — Mon, 19 Jan 2026 14:52:51 +0000

This article was originally published on Saru Blog.

The Setup

Saru has 4 portals: System, Provider, Reseller, Consumer. Each runs on a different subdomain, but they share one Keycloak realm.

system.saru.local   (port 3001)  →  Keycloak
provider.saru.local (port 3002)  →  (single realm,
reseller.saru.local (port 3003)  →   4 clients)
consumer.saru.local (port 3004)  →

Basic Keycloak + Auth.js integration is well-documented in existing tutorials. This article covers the problems those tutorials don't mention.

Pitfall 1: Cookie Collision Across Subdomains

The Problem

We wanted cross-subdomain session sharing for potential future use, so we set:

cookies: {
  sessionToken: {
    options: {
      domain: '.saru.local',  // Share across subdomains
    },
  },
}

Result: Login to System portal, Provider portal shows the same session. But it's the wrong user context. The System admin's token is being used on the Provider portal.

Why It Happens

Auth.js uses cookie names like authjs.session-token (or __Secure-authjs.session-token in HTTPS). With domain: '.saru.local', all subdomains share the same cookie. The first portal to set the cookie wins, and subsequent logins on other portals read that same cookie.

Note: Cookie names vary by Auth.js version and config (e.g., next-auth.session-token in older versions). Check your actual cookie names in browser devtools.

system.saru.local   → sets authjs.session-token (domain=.saru.local)
provider.saru.local → reads same cookie → wrong context!

The Fix

Portal-prefixed cookie names:

// packages/auth/src/config.ts

export function getAuthConfig({ portal }: { portal: PortalType }) {
  return {
    cookies: {
      sessionToken: {
        name: `authjs.${portal}-session-token`,  // e.g., authjs.system-session-token
        options: {
          domain: COOKIE_DOMAIN,
        },
      },
      callbackUrl: {
        name: `authjs.${portal}-callback-url`,
      },
      csrfToken: {
        name: `authjs.${portal}-csrf-token`,
      },
      // Also prefix state/pkceCodeVerifier if using OAuth flows
    },
    // ...
  };
}

Note: Auth.js uses additional cookies for OAuth flows (state, pkceCodeVerifier). If multiple portals perform concurrent logins, consider prefixing these as well to avoid intermittent auth failures.

Now each portal has its own session:

system.saru.local   → authjs.system-session-token
provider.saru.local → authjs.provider-session-token  ✓

Lesson Learned

If you don't need cross-subdomain sharing, just omit domain entirely. Cookies become host-only by default, and you avoid this problem.

Pitfall 2: Custom Claims Don't Appear in Tokens

The Problem

Our backend needs tenant context: account_id, account_type, capabilities. We stored these as Keycloak user attributes, but they weren't in the tokens.

// Expected in profile
profile.account_id      // undefined
profile.capabilities    // undefined

Why It Happens

Keycloak doesn't automatically include user attributes in tokens. You need Protocol Mappers. But even then, there are gotchas.

The Fix

Step 1: Create Protocol Mappers

For each attribute, create a mapper in Keycloak. Mappers can be added to a Client Scope (then assigned to your client) or directly to the client's "Dedicated Scope."

Setting	Value
Mapper Type	User Attribute
User Attribute	`account_id`
Token Claim Name	`account_id`
Add to ID token	ON
Add to access token	ON (if your API validates access tokens)
Add to userinfo	ON

Important: If you add mappers to a Client Scope, make sure that scope is assigned to your client (as default or optional). Otherwise, the mapper won't execute.

Step 2: Handle Multivalued Attributes Correctly

For capabilities (array of strings), we initially stored it as a JSON string:

// Wrong! This stores a literal string, not an array
{ "attributes": { "capabilities": "[\"CONSUME\", \"PROVIDE\"]" } }

Keycloak's "Multivalued" mapper expects separate values, not a JSON string. Important: Set "Multivalued" to ON in the mapper configuration.

// Correct: Keycloak Admin API user update payload
{ "attributes": { "capabilities": ["CONSUME", "PROVIDE"] } }

The Keycloak Admin API accepts arrays directly in the attributes map:

// When syncing capabilities from your app via Admin API
userUpdate := map[string]interface{}{
    "attributes": map[string][]string{
        "capabilities": {"CONSUME", "PROVIDE"},  // Array, not JSON string
    },
}

Step 3: Custom Scopes (Often Missed)

If you request scope: 'openid roles account_info', custom scopes like account_info need to exist in Keycloak. Standard OIDC only provides openid, profile, email.

Note: roles is not a standard OIDC scope - it's a Keycloak client scope with role mappers. Roles can appear in tokens even without explicitly requesting a roles scope, if the default client scopes include role mappers. However, if you explicitly request roles as a scope, verify the roles client scope is assigned to your client.

Create Client Scopes in Keycloak for custom scopes like account_info, then assign them to your clients. Non-existent or unassigned scopes are silently ignored - your token just won't include the expected claims.

Lesson Learned

Test your token contents early. Decode a token locally (e.g., jwt-cli, browser devtools, or a local script) and verify your claims are present before writing frontend code that depends on them.

Security note: Never paste production tokens into online decoders like jwt.io - they're third-party services. Use local tools for real tokens.

Pitfall 3: Token Exposure in Sessions

The Problem

We needed the access token on the client side to call APIs directly:

// Session callback - exposes token to client
async session({ session, token }) {
  return {
    ...session,
    accessToken: token.accessToken,  // Exposed to client JS
  };
}

Prerequisite: For token.accessToken to exist, you must first persist it in the jwt callback during sign-in (e.g., token.accessToken = account.access_token). The same applies to refreshToken.

This works, but it's a security tradeoff we didn't fully consider initially.

Why It's Risky

With accessToken in the session, any JavaScript on your page can access it:

const { data: session } = useSession();
console.log(session.accessToken);  // Any script can do this

If you have an XSS vulnerability, attackers can steal the token.

The Tradeoffs

Approach	Pros	Cons
Expose token	Simple, direct API calls from browser	XSS can steal token
BFF pattern	Token stays server-side, client calls BFF only	More complexity, all traffic through Next.js

Note: "Proxy all calls" is essentially the BFF pattern. The key question is whether your client ever holds a bearer token.

What We Chose

We expose the token, accepting the risk with defense-in-depth measures:

Strict CSP: Limits which scripts can run (not foolproof, but reduces attack surface)
Short expiry: Tokens expire in 5 minutes (limits damage window if stolen)
Refresh token rotation: Each refresh issues a new refresh token (requires "Revoke Refresh Token" enabled in Keycloak realm/client settings)

Honest assessment: These mitigations reduce risk but don't eliminate it. Any XSS vulnerability means full account compromise for the token's lifetime. If refresh tokens are also exposed in the session, attackers can extend access beyond the 5-minute window. We accept this tradeoff for our B2B context with trusted users and no user-generated content.

// Token refresh with rotation
return {
  accessToken: refreshedTokens.access_token,
  refreshToken: refreshedTokens.refresh_token ?? token.refreshToken,
  expiresAt: Math.floor(Date.now() / 1000) + refreshedTokens.expires_in,
};

Lesson Learned

There's no universally "correct" answer. Know your threat model. For a B2B SaaS with trusted users, token exposure with mitigations is often acceptable. For a consumer app with user-generated content (XSS risk), consider BFF.

Bonus: Token Refresh Error Handling

One more thing that bit us: handling refresh failures gracefully.

// Note: client_secret is for confidential clients (server-side).
// For public clients (SPAs), use PKCE without client_secret.
async function refreshAccessToken(token: JWT): Promise<JWT> {
  const response = await fetch(tokenEndpoint, {
    method: 'POST',
    headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
    body: new URLSearchParams({
      grant_type: 'refresh_token',
      refresh_token: token.refreshToken as string,
      client_id: clientId,
      client_secret: clientSecret,  // Confidential clients only
    }),
  });

  // Defensive: some error responses may not be JSON
  let refreshedTokens;
  try {
    refreshedTokens = await response.json();
  } catch {
    return { ...token, error: 'TokenRefreshFailed' };
  }

  if (!response.ok) {
    // Don't just throw - return an error state
    const error = refreshedTokens.error;
    const errorDesc = refreshedTokens.error_description || '';

    // Check for user-related errors
    if (errorDesc.includes('deleted') || errorDesc.includes('disabled')) {
      return { ...token, error: 'UserDeleted' };
    }
    // invalid_grant covers expired/revoked refresh tokens
    if (error === 'invalid_grant') {
      return { ...token, error: 'TokenRefreshFailed' };
    }
    return { ...token, error: 'TokenRefreshFailed' };
  }

  // Defensive: expires_in might be missing in some edge cases
  const expiresIn = refreshedTokens.expires_in ?? 300; // Default to 5 min

  return {
    ...token,
    accessToken: refreshedTokens.access_token,
    refreshToken: refreshedTokens.refresh_token ?? token.refreshToken,
    expiresAt: Math.floor(Date.now() / 1000) + expiresIn,
  };
}

Then handle it in your app:

const { data: session } = useSession();

if (session?.error === 'TokenRefreshFailed') {
  // Force re-login instead of showing cryptic errors
  signIn('keycloak');
}

Summary

Pitfall	Solution
Cookie collision	Portal-prefixed cookie names
Missing claims	Protocol Mappers + correct attribute format
Token exposure	Accept tradeoff with mitigations, or use BFF

The basics of Keycloak + Auth.js are well-documented. It's these edge cases that cost us debugging time. Hopefully this saves you some.

Series Articles

Part 1: Tackling Unmaintainable Complexity with Automation
Part 2: Automated WebAuthn Testing in CI
Part 3: Next.js × Go Monorepo Architecture
Part 4: PostgreSQL RLS for Multi-Tenant Isolation
Part 5: Multi-Portal Authentication Pitfalls (This article)

PostgreSQL RLS for Multi-Tenant Isolation: Protecting 4-Tier Data as a Solo Developer [Part 4]

ko-chan — Thu, 15 Jan 2026 17:20:04 +0000

This article was originally published on Saru Blog.

What You'll Learn

Comparison of data isolation patterns for multi-tenant SaaS
Practical usage of PostgreSQL Row-Level Security (RLS)
RLS policy design for 4-tier hierarchy (System/Provider/Reseller/Consumer)
Setting RLS context with Go + pgx
Detecting RLS leaks through testing

Introduction

As introduced in Part 1, Saru is a multi-tenant SaaS with a 4-tier account structure.

System Admin (manages the entire SMS platform)
    └── Provider (offers services)
            ├── Reseller (sells services)
            │       └── Consumer (purchases/manages)
            └── Consumer (direct sales)

In this structure, data isolation is critical.

Provider A's customer data must not be visible to Provider B
Reseller A's sales records must not be visible to Reseller B
Consumer A's subscription info must not be visible to Consumer B

Preventing this entirely at the application layer is difficult. Forgotten WHERE clauses and missing permission checks are especially common in solo development.

That's why I adopted PostgreSQL Row-Level Security (RLS).

1. Comparing Multi-Tenant Isolation Patterns

There are three main approaches to multi-tenant data isolation.

Pattern Comparison

Pattern	Isolation Level	Implementation Cost	Operational Cost	Scalability
Database per Tenant	Highest (physical)	High	High	Performance isolation excellent, operations challenging
Schema per Tenant	High (logical)	Medium	Medium (automation required)	Medium
Shared Schema + RLS	High (design-dependent)	Low	Low	High (design-dependent)

Database per Tenant

Each tenant has an independent database.

┌─────────────┐  ┌─────────────┐  ┌─────────────┐
│ Provider A  │  │ Provider B  │  │ Provider C  │
│   Database  │  │   Database  │  │   Database  │
└─────────────┘  └─────────────┘  └─────────────┘

Pros: Complete isolation, easy per-tenant backup/restore

Cons: DB instances grow with tenant count. Unmanageable for solo development.

Schema per Tenant

Separate schemas for each tenant within a single database.

┌────────────────────────────────────┐
│           Single Database          │
│  ┌──────┐  ┌──────┐  ┌──────┐     │
│  │ A.   │  │ B.   │  │ C.   │     │
│  │schema│  │schema│  │schema│     │
│  └──────┘  └──────┘  └──────┘     │
└────────────────────────────────────┘

Pros: Logical isolation, per-tenant operations possible

Cons: Schema creation and migration automation required for new tenants. Operations can get complex.

Shared Schema + RLS (Saru's Choice)

All tenants share the same schema, with RLS controlling access.

┌────────────────────────────────────┐
│           Single Schema            │
│  ┌──────────────────────────────┐  │
│  │     accounts, products,      │  │
│  │     subscriptions, ...       │  │
│  │     + RLS Policies           │  │
│  └──────────────────────────────┘  │
└────────────────────────────────────┘

Pros: Simple operations, easy migrations, adding tenants is just inserting rows

Cons: RLS policy design and testing are critical. Per-tenant recovery is difficult.

2. Why I Chose RLS

The Reality of Solo Development

Challenge	RLS Solution
Forgotten WHERE clauses	Automatic filtering at DB level
Missing permission checks	Policy violations = invisible data
Onboarding new developers	Policies ensure automatic isolation
Adding tenants	Just insert a new row

The Greatest Benefit: Last Line of Defense

Even if there are bugs in application code, RLS prevents data leaks.

-- Even if the app accidentally fetches all rows
SELECT * FROM accounts;

-- RLS returns only current tenant's data
-- (other tenants' data is invisible)

Prerequisites:

Tenant context set via session variables (e.g., SET app.account_id = '...')
RLS policies defined on target tables

Exceptions:

Superusers and roles with BYPASSRLS privilege can bypass RLS
DB admin roles need separate protection

Still, a design where "app bugs don't cause leaks" is reassuring for solo development.

3. RLS Policy Design for 4-Tier Hierarchy

3.1 RLS Context Variables

Use PostgreSQL's SET LOCAL to set context per request.

Variable	Description	Example
`app.account_id`	Current account ID	`550e8400-e29b-...`
`app.account_type`	Account type	`provider`, `consumer`
`app.bypass_rls`	RLS bypass flag (System only)	`true`, `false`

Important: SET LOCAL is only effective within a transaction. It automatically resets when the transaction ends.

3.2 Policies for the accounts Table

Define policies for each of the 4 tiers.

-- Enable RLS (FORCE applies to owner too)
ALTER TABLE accounts ENABLE ROW LEVEL SECURITY;
ALTER TABLE accounts FORCE ROW LEVEL SECURITY;

-- 1. System Admin: Access to everything
CREATE POLICY system_admin_all_accounts ON accounts
    FOR ALL
    USING (
        current_setting('app.bypass_rls', true) = 'true'
        OR current_setting('app.account_type', true) = 'system'
    );

-- 2. Provider: Self + subordinate Resellers/Consumers
CREATE POLICY provider_accounts ON accounts
    FOR ALL
    USING (
        current_setting('app.account_type', true) = 'provider'
        AND (
            id = current_setting('app.account_id', true)::UUID
            OR provider_id = current_setting('app.account_id', true)::UUID
        )
    );

-- 3. Reseller: Self + subordinate Consumers
CREATE POLICY reseller_accounts ON accounts
    FOR ALL
    USING (
        current_setting('app.account_type', true) = 'reseller'
        AND (
            id = current_setting('app.account_id', true)::UUID
            OR reseller_id = current_setting('app.account_id', true)::UUID
        )
    );

-- 4. Consumer: Self only
CREATE POLICY consumer_accounts ON accounts
    FOR ALL
    USING (
        current_setting('app.account_type', true) = 'consumer'
        AND id = current_setting('app.account_id', true)::UUID
    );

Notes:

current_setting(..., true) returns NULL when unset (no error)
Comparing with NULL yields NULL, which RLS treats as denied
Roles with BYPASSRLS privilege can bypass even with FORCE ROW LEVEL SECURITY

3.3 Policy Behavior Visualization

Logged in as Provider A (app.account_type = 'provider', app.account_id = A)
    ├── Provider A's data: ✓ Visible
    ├── Reseller A1's data: ✓ Visible (provider_id = A)
    ├── Consumer A1's data: ✓ Visible (provider_id = A)
    ├── Provider B's data: ✗ Not visible
    └── Consumer B1's data: ✗ Not visible

3.4 INSERT/UPDATE Control with WITH CHECK

USING filters reads, but WITH CHECK is needed for writes.
Saru uses this on the api_keys table.

-- Can only create own PATs
CREATE POLICY api_keys_user_insert ON api_keys
    FOR INSERT
    WITH CHECK (
        tenant_id = current_setting('app.account_id', true)::UUID
        AND user_id = current_setting('app.user_id', true)::UUID
    );

Current limitation: WITH CHECK is not defined on the accounts table.
Currently controlled by foreign key constraints and application layer, but considering adding it.

4. Implementation with Go + pgx

4.1 Tenant Context Struct

// backend/internal/domain/tenant/context.go

type AccountType string

const (
    AccountTypeSystem   AccountType = "system"
    AccountTypeProvider AccountType = "provider"
    AccountTypeReseller AccountType = "reseller"
    AccountTypeConsumer AccountType = "consumer"
)

// Context holds tenant-specific information for the current request.
type Context struct {
    AccountID   uuid.UUID
    AccountType AccountType
    UserID      *uuid.UUID
    ProviderID  *uuid.UUID
    BypassRLS   bool
    Scopes      []string
}

4.2 RLS Context Setting Function

Execute SET LOCAL within a transaction.

// backend/internal/infrastructure/postgres/tenant_context.go

// SetTenantContextTx sets the RLS context variables on a transaction.
func SetTenantContextTx(ctx context.Context, tx pgx.Tx, tc *tenant.Context) error {
    // Set account_id for RLS policies
    if _, err := tx.Exec(ctx,
        fmt.Sprintf("SET LOCAL app.account_id = '%s'", tc.AccountID.String()),
    ); err != nil {
        return fmt.Errorf("failed to set app.account_id: %w", err)
    }

    // Set account_type for RLS policies
    if _, err := tx.Exec(ctx,
        fmt.Sprintf("SET LOCAL app.account_type = '%s'", tc.AccountType),
    ); err != nil {
        return fmt.Errorf("failed to set app.account_type: %w", err)
    }

    // Set bypass_rls flag (only for system admin)
    bypassRLS := "false"
    if tc.BypassRLS {
        bypassRLS = "true"
    }
    if _, err := tx.Exec(ctx,
        fmt.Sprintf("SET LOCAL app.bypass_rls = '%s'", bypassRLS),
    ); err != nil {
        return fmt.Errorf("failed to set app.bypass_rls: %w", err)
    }

    return nil
}

Future improvement: String concatenation with fmt.Sprintf poses risks if values contain special characters.
Test code uses set_config($1, $2, true) format, and production code should be updated similarly.
// Safer implementation
_, err := tx.Exec(ctx,
    "SELECT set_config('app.account_id', $1, true)",
    tc.AccountID.String(),
)

4.3 Context Builders

Generate context for each account type.

// BuildSystemContext builds a tenant context for system admin operations.
func BuildSystemContext(userID *uuid.UUID) *tenant.Context {
    return &tenant.Context{
        AccountID:   uuid.Nil,
        AccountType: tenant.AccountTypeSystem,
        UserID:      userID,
        BypassRLS:   true,  // System Admin bypasses RLS
        Scopes:      []string{"*:*"},
    }
}

// BuildProviderContext builds a tenant context for provider operations.
func BuildProviderContext(accountID uuid.UUID, userID *uuid.UUID, scopes []string) *tenant.Context {
    return &tenant.Context{
        AccountID:   accountID,
        AccountType: tenant.AccountTypeProvider,
        UserID:      userID,
        ProviderID:  &accountID,
        BypassRLS:   false,
        Scopes:      scopes,
    }
}

5. Detecting RLS Leaks Through Testing

5.1 Integration Test Design Philosophy

RLS tests verify "what should be visible is visible, what shouldn't is not".

// backend/tests/integration/rls_isolation_test.go

// Verify Provider A cannot access Provider B's data
func TestRLSIsolation_ProviderCannotAccessOtherProvider(t *testing.T) {
    pc := testutil.SetupPostgres(t)
    defer pc.TruncateTables(t)

    // Create Provider A
    providerA := testutil.CreateProviderAccount(t, pc, "ProviderA")

    // Create Provider B
    providerB := testutil.CreateProviderAccount(t, pc, "ProviderB")

    // Try to access Provider B's account as Provider A
    count := pc.CountWithTenant(t,
        providerA.Account.ID.String(), "provider",
        "SELECT COUNT(*) FROM accounts WHERE id = $1",
        providerB.Account.ID,
    )

    // Not visible due to RLS
    assert.Equal(t, 0, count,
        "Provider A cannot see Provider B's account")

    // Own account is visible
    count = pc.CountWithTenant(t,
        providerA.Account.ID.String(), "provider",
        "SELECT COUNT(*) FROM accounts WHERE id = $1",
        providerA.Account.ID,
    )
    assert.Equal(t, 1, count,
        "Provider A can see their own account")
}

5.2 Test Helper

CountWithTenant executes queries in an RLS-enabled session.

// CountWithTenant executes a count query with tenant context.
// Runs queries with RLS properly applied.
func (pc *PostgresContainer) CountWithTenant(
    t *testing.T,
    accountID, accountType, query string,
    args ...interface{},
) int {
    ctx := context.Background()
    conn, err := pc.Pool.Acquire(ctx)
    require.NoError(t, err)
    defer conn.Release()

    // Set RLS context with set_config
    _, err = conn.Exec(ctx, `
        SELECT set_config('app.account_id', $1, false),
               set_config('app.account_type', $2, false)
    `, accountID, accountType)
    require.NoError(t, err)

    var count int
    err = conn.QueryRow(ctx, query, args...).Scan(&count)
    require.NoError(t, err)

    return count
}

5.3 Test Cases to Verify

Test Case	Verification
Same-tier isolation	Provider A cannot see Provider B
Parent-child relationship	Provider can see subordinate Resellers/Consumers
Cross-hierarchy	Reseller A cannot see Reseller B's Consumers
System Admin	Can access all data

Future additions: Leak tests for JOIN queries, verification with aggregate queries

6. Lessons Learned from RLS Design

6.1 FORCE ROW LEVEL SECURITY is Required

-- This alone lets table owner bypass RLS
ALTER TABLE accounts ENABLE ROW LEVEL SECURITY;

-- Include owner in RLS enforcement
ALTER TABLE accounts FORCE ROW LEVEL SECURITY;

If you see all data during development, check for FORCE.

Exception: Superusers and roles with BYPASSRLS attribute can bypass even with FORCE.

6.2 The Second Argument of current_setting

-- Second argument true: returns NULL when unset (no error)
current_setting('app.account_id', true)

-- Without second argument: errors when unset
current_setting('app.account_id')

Use true in RLS policies for fallback behavior when unset.
Comparing with NULL yields NULL, which RLS treats as denied.

6.3 SET LOCAL Only Works Within Transactions

// ❌ Doesn't work outside transaction
conn.Exec(ctx, "SET LOCAL app.account_id = '...'")
conn.Query(ctx, "SELECT * FROM accounts")

// ✅ Set within transaction
tx, _ := conn.Begin(ctx)
tx.Exec(ctx, "SET LOCAL app.account_id = '...'")
tx.Query(ctx, "SELECT * FROM accounts")
tx.Commit(ctx)

SET LOCAL automatically resets when the transaction ends.

pgx note: Connections borrowed from a pool must stay within BEGIN~COMMIT on the same connection.

7. RLS Performance

General Trends

RLS overhead varies significantly based on data volume, query patterns, and policy complexity.
External benchmarks report the following, though results vary by conditions:

Source	Conditions	Reported Value
Supabase	Simple policies + indexed	~5-15%
AntStack	No index, large data	10x+ degradation

Note: Queries with many JOINs can increase costs even with simple policies.

Saru's Approach

Data volume is currently small, so formal benchmarking hasn't been done.
However, these considerations were made during design to prevent degradation:

Indexes on RLS-target columns like provider_id, owner_id
Policies use only simple equality comparisons (no subqueries or function calls)
Cast comparison values to match column type (UUID), not the column itself

Future Plans

As data grows, the following will be implemented:

Regular checks with EXPLAIN ANALYZE
- Verify RLS policies are using indexes
- Check for plan degradation in JOIN queries
Comparative measurements on representative queries
- Simple SELECT / JOIN-heavy / Aggregate queries
- Impact when row counts are skewed by tenant
Optimization as needed
- Wrap current_setting() in (SELECT current_setting(...)) for initPlan
- Consider security barrier views

8. Future Improvements

The implementation described has room for improvement:

Item	Current State	Improvement
WITH CHECK on accounts	Not defined	Add constraints for INSERT/UPDATE
SET LOCAL implementation	`fmt.Sprintf`	Change to `set_config($1, $2, true)` format
bypass_rls protection	App-layer control	Restrict with DB role permissions for SET
JOIN tests	Not implemented	Add leak tests for related table joins

In solo development, I take the approach of "build something that works, then improve" rather than aiming for perfection.
Having RLS as a defense layer allows for confident incremental improvements.

Summary

Item	Implementation
Isolation method	Shared Schema + RLS
Context setting	`SET LOCAL app.*` via pgx
Policy design	Separate policies per tier
Bypass	`app.bypass_rls = 'true'` (System Admin only)
Testing	Integration tests verify "visible/not visible"

RLS is incredibly reassuring as a "last line of defense." Even if the app has bugs, the DB prevents data leaks.

For complex multi-tenant systems in solo development, RLS is a strong choice.

Series Articles

Part 1: Tackling Unmaintainable Complexity with Automation
Part 2: Automated WebAuthn Testing in CI
Part 3: Next.js × Go Monorepo Architecture
Part 4: PostgreSQL RLS for Multi-Tenant Isolation (This article)

Next.js + Go Monorepo: Managing 4 Portals 4 APIs as a Solo Developer [Part 3]

ko-chan — Wed, 14 Jan 2026 03:29:49 +0000

This article was originally published on Saru Blog.

What You'll Learn

Next.js + Go monorepo architecture patterns
Practical use of pnpm workspace + Turborepo
Sharing UI components across 4 portals
Package splitting strategies that don't break down in solo development

Introduction

As introduced in Part 1, Saru is a multi-tenant SaaS with a 4-tier account structure. To implement this, I adopted an architecture of 4 frontends + 4 backend APIs.

Normally, this would mean managing 8 repositories. For solo development, that would be unsustainable.

So I chose a monorepo. This article explains the architecture and design decisions.

1. Why Next.js × Go?

Technology Selection Rationale

Layer	Technology	Reason
Frontend	Next.js 14	App Router, RSC, rich ecosystem
Backend	Go + Echo	Simple, fast, type-safe, easy deployment
DB	PostgreSQL	Multi-tenant isolation via RLS

Why not use a full-stack framework (Next.js API Routes)?

Separation of concerns: Want independent deploy cycles for frontend and backend
Language strengths: Go handles complex business logic better (personal opinion)
Scalability: May want to scale APIs independently in the future

Authentication flow is divided as follows: Keycloak handles user authentication, NextAuth manages OAuth/sessions, and Go APIs validate Keycloak access tokens (JWT) for authorization.

2. Project Structure

saru/
├── apps/                    # 6 Next.js apps
│   ├── system/              # System Portal (admins)
│   ├── provider/            # Provider Portal (service providers)
│   ├── reseller/            # Reseller Portal (resellers)
│   ├── consumer/            # Consumer Portal (end users)
│   ├── customer/            # Customer Portal (legacy, merging with consumer)
│   └── landing/             # Landing pages
│
├── packages/                # Shared packages
│   ├── types/               # TypeScript type definitions
│   ├── ui/                  # Shared UI components
│   ├── api-client/          # API client + React Query hooks
│   ├── auth/                # NextAuth configuration
│   ├── config/              # ESLint, TypeScript config
│   └── env-validator/       # Environment variable validation
│
├── backend/                 # Go backend
│   ├── cmd/
│   │   ├── system-api/      # System API (port 8080)
│   │   ├── provider-api/    # Provider API (port 8081)
│   │   ├── reseller-api/    # Reseller API (port 8082)
│   │   ├── consumer-api/    # Consumer API (port 8083)
│   │   └── migrate/         # Migration CLI
│   └── internal/            # Shared logic
│
├── e2e/                     # Playwright E2E tests
├── pnpm-workspace.yaml      # pnpm workspace config
└── turbo.json               # Turborepo config

Why 4 Separate APIs?

You might ask: "Why not just one API for everything?"

Reasons for separation:

Clear permission boundaries: System API is admin-only, Provider API is provider-only
Independent deployability: Update Provider API without affecting others
Code clarity: One API with all endpoints becomes complex

Shared logic lives in internal/:

backend/internal/
├── domain/           # Domain models
├── application/      # Use cases
├── infrastructure/   # DB, external services
└── interfaces/       # Handlers, DTOs

All 4 APIs reference the same internal/, registering only the handlers they need.

3. pnpm workspace + Turborepo

pnpm-workspace.yaml

packages:
  - "apps/*"
  - "packages/*"

Simple. Everything under apps/ and packages/ becomes a workspace.

turbo.json

{
  "$schema": "https://turbo.build/schema.json",
  "tasks": {
    "build": {
      "dependsOn": ["^build"],
      "outputs": [".next/**", "dist/**"]
    },
    "dev": {
      "cache": false,
      "persistent": true
    },
    "lint": {
      "dependsOn": ["^lint"]
    },
    "type-check": {
      "dependsOn": ["^build"]
    }
  }
}

Key points:

"dependsOn": ["^build"] — Build dependencies first
dev has cache: false — Dev server shouldn't be cached
type-check depends on ^build — Type packages must build first

Common Commands

# Build all apps
pnpm build

# Develop specific app only
pnpm dev:system    # System Portal only

# Lint + type-check everything
pnpm lint && pnpm type-check

4. Shared Package Design

@repo/types — Type Definitions

// packages/types/src/product.ts
export interface Product {
  id: string;
  code: string;
  name: string;
  status: ProductStatus;
  providerId: string;
  // ...
}

export type ProductStatus = 'draft' | 'published' | 'archived' | 'discontinued';

Why a separate types package?

Multiple apps use the same types
Match backend response types
Changes propagate to all apps

@repo/ui — Shared UI Components

// packages/ui/src/components/ProductStatusBadge.tsx
import { ProductStatus } from '@repo/types';

const statusConfig = {
  draft: { label: 'Draft', variant: 'secondary' },
  published: { label: 'Published', variant: 'success' },
  archived: { label: 'Archived', variant: 'muted' },
  discontinued: { label: 'Discontinued', variant: 'destructive' },
};

export function ProductStatusBadge({ status }: { status: ProductStatus }) {
  const config = statusConfig[status];
  return <Badge variant={config.variant}>{config.label}</Badge>;
}

Sharing criteria:

Condition	Location
Same implementation in 2+ portals	`packages/ui/`
Single portal only	`apps/[portal]/src/components/`
Contains portal-specific business logic	Keep in each app

@repo/api-client — API Client + React Query

// packages/api-client/src/hooks/useProducts.ts
import { useQuery, useMutation } from '@tanstack/react-query';
import type { Product } from '@repo/types';

export function useProducts() {
  return useQuery({
    queryKey: ['products'],
    queryFn: () => apiClient.get<Product[]>('/products'),
  });
}

export function useCreateProduct() {
  const queryClient = useQueryClient();
  return useMutation({
    mutationFn: (data: CreateProductRequest) =>
      apiClient.post<Product>('/products', data),
    onSuccess: () => {
      queryClient.invalidateQueries({ queryKey: ['products'] });
    },
  });
}

Why share hooks?

Don't write the same API logic in each app
Unified caching strategy
Type-safe API calls

@repo/auth — NextAuth Configuration

// packages/auth/src/index.ts
import NextAuth from 'next-auth';
import Keycloak from 'next-auth/providers/keycloak';

export const { auth, handlers, signIn, signOut } = NextAuth({
  providers: [
    Keycloak({
      clientId: process.env.KEYCLOAK_CLIENT_ID!,
      clientSecret: process.env.KEYCLOAK_CLIENT_SECRET!,
      issuer: process.env.KEYCLOAK_ISSUER!,
    }),
  ],
  // ...
});

All 4 portals use the same Keycloak configuration.

5. Go Backend Structure

4 API Entry Points

// backend/cmd/system-api/main.go
func main() {
    e := echo.New()

    // System API specific routes
    systemRouter := router.NewSystemRouter(e, services)
    systemRouter.RegisterRoutes()

    e.Start(":8080")
}

// backend/cmd/provider-api/main.go
func main() {
    e := echo.New()

    // Provider API specific routes
    providerRouter := router.NewProviderRouter(e, services)
    providerRouter.RegisterRoutes()

    e.Start(":8081")
}

Shared Logic

// backend/internal/application/product_service.go
type ProductService struct {
    repo repository.ProductRepository
}

func (s *ProductService) Create(ctx context.Context, req CreateProductRequest) (*Product, error) {
    // Called by both System API and Provider API
}

All APIs use the same Services. The only difference is which endpoints each router registers.

Multi-Layer Authorization

Authorization check flow:
1. Go API: JWT signature verification + account type check
2. Go API: Business logic layer permission check
3. PostgreSQL RLS: Tenant isolation at data access (last line of defense)

API separation and RLS serve different purposes:

API separation: Endpoint-level access control (who can call which functions)
RLS: Data-level isolation (which data can be accessed)

This defense-in-depth ensures that even if API permission checks have gaps, RLS limits the blast radius (when properly configured).

6. Type Sharing: TypeScript ↔ Go

We don't have full automatic sync. Instead:

OpenAPI spec is the source of truth
TypeScript types generated via openapi-typescript
Go types are manually aligned (planning oapi-codegen integration)

# packages/types/package.json
"scripts": {
  "generate:types": "openapi-typescript ../../specs/shared-schemas.yaml -o src/generated/shared-schemas.ts"
}

Honest truth: Full automation isn't there yet. Go and TypeScript types can drift. E2E tests catch major flow issues, but aren't comprehensive. Planning to add OpenAPI schema CI checks for better coverage.

7. Development Server Startup

# Unified script starts all services
./scripts/start-dev.sh

Internally:

Start Docker (PostgreSQL, Keycloak, Mailpit)
Start 4 Go APIs (hot reload via air)
Start Next.js apps as needed

# Individual startup also possible
pnpm dev:system     # System Portal only
pnpm dev:provider   # Provider Portal only

Required Environment Variables

Key environment variables for each portal:

# Authentication (Keycloak)
KEYCLOAK_CLIENT_ID=xxx
KEYCLOAK_CLIENT_SECRET=xxx
KEYCLOAK_ISSUER=http://localhost:8180/realms/saru

# API Connection
NEXT_PUBLIC_API_URL=http://localhost:808x

# NextAuth
NEXTAUTH_SECRET=xxx
NEXTAUTH_URL=http://localhost:300x

package.json Script Examples

// Root package.json
{
  "scripts": {
    "dev": "turbo run dev",
    "dev:system": "turbo run dev --filter=system",
    "dev:provider": "turbo run dev --filter=provider",
    "build": "turbo run build",
    "lint": "turbo run lint",
    "type-check": "turbo run type-check"
  }
}

8. Avoiding Burnout in Solo Development

What We Do

Practice	Benefit
Shared packages	Don't write the same code in 4 places
Turborepo	Build caching for speed
E2E tests	Automatic regression detection
Unified scripts	No need to memorize startup procedures

What We Avoid

Avoided	Reason
Microservices	4 APIs is already complex enough
Over-abstraction	Only abstract after 3+ duplications
Auto type sync	High setup cost, E2E + future CI checks suffice

Summary

Component	Technology	Key Point
Workspace	pnpm workspace	Simple apps/ + packages/ structure
Build	Turborepo	dependsOn manages dependency order
Shared types	@repo/types	Type sharing across all apps
Shared UI	@repo/ui	Components used in 2+ portals
API client	@repo/api-client	Shared React Query hooks
Auth	@repo/auth	Shared NextAuth configuration
Backend	Go + Echo	4 APIs, shared logic in internal/

Monorepos have setup costs, but once established, "add this component to that app too..." becomes trivial.

For solo developers managing multiple apps, monorepo is a strong choice.

Series Articles

Part 1: Tackling Unmanageable Complexity with Automation
Part 2: Testing WebAuthn in CI
Part 3: Next.js + Go Monorepo (this article)

Testing WebAuthn in CI: E2E Automation with Virtual Authenticators and Mailpit [Part 2]

ko-chan — Tue, 13 Jan 2026 13:02:52 +0000

This article was originally published on Saru Blog.

What You'll Learn

How to test WebAuthn (passkey) authentication in CI environments
Automating OTP email retrieval with Mailpit API
Preventing email race conditions in parallel E2E tests
Locale-specific testing for multilingual UIs

Introduction

In Part 1, I introduced the overall architecture and automation strategy for "Saru," a multi-tenant SaaS platform. This article dives deeper into the E2E testing implementation that forms the core of that automation.

The most challenging aspect is testing authentication flows. Saru uses two authentication methods:

Portal	Auth Method	Challenge
System / Provider	OTP + Passkey	Email retrieval, WebAuthn
Reseller / Consumer	Keycloak OAuth	External IdP integration

This article explains how to automate testing all of these in CI.

1. WebAuthn Virtual Authenticator: Testing Passkeys in CI

The Challenge with Passkey Authentication

WebAuthn (passkeys) typically requires physical security keys or biometric authentication. At first glance, testing this in CI seems impossible.

Solution: Chrome DevTools Protocol (CDP) Virtual Authenticator

Playwright allows you to create virtual authenticators through CDP. This enables testing the full WebAuthn flow without physical devices.

Note: CDP virtual authenticators are Chromium-only. They don't work with Safari (WebKit) or Firefox. For cross-browser testing, run WebAuthn tests only on Chromium and mock authenticated state for other browsers.

Implementation Code

import { test, expect, type BrowserContext } from '@playwright/test';

test('should complete signup with Passkey registration', async ({ page, context }) => {
  // Enable virtual authenticator
  const cdpSession = await context.newCDPSession(page);
  await cdpSession.send('WebAuthn.enable');

  // Add virtual authenticator
  await cdpSession.send('WebAuthn.addVirtualAuthenticator', {
    options: {
      protocol: 'ctap2',           // CTAP2 protocol
      transport: 'usb',            // Emulate USB connection
      hasResidentKey: true,        // Passkey capable
      hasUserVerification: true,   // Emulate biometric auth
      isUserVerified: true,        // Always succeed verification
      automaticPresenceSimulation: true, // Auto-respond
    },
  });

  // ... Execute signup flow ...

  // Click Passkey registration button
  await page.getByRole('button', { name: 'Passkey' }).click();

  // Virtual authenticator responds automatically
  await expect(page.getByText('Passkey registered')).toBeVisible();

  // Cleanup
  await cdpSession.send('WebAuthn.disable');
});

Alignment Between Transport Settings and Server Configuration

When setting up the WebAuthn virtual authenticator, alignment with server-side settings is crucial.

In Saru's case, the backend generates WebAuthn registration options with AuthenticatorAttachment: CrossPlatform. This setting "prefers roaming authenticators (USB keys, etc.)."

Initially, I used transport: 'internal' (platform authenticator), which caused registration to fail.

// When server prefers CrossPlatform, alignment matters
transport: 'internal',  // Platform authenticator → may fail
transport: 'usb',       // Roaming authenticator → aligns with server

Key Point: The virtual authenticator's transport setting needs to align with the server's AuthenticatorAttachment setting. If registration fails, check the server configuration first. While WebAuthn spec doesn't require exact 1:1 correspondence, misalignment is a common cause of failures.

2. Automating OTP Email Retrieval: Mailpit API Integration

Problems with Traditional Approaches

Many E2E tests retrieve OTP from a test endpoint:

// Get OTP via test mode (not recommended)
const response = await request.get(`/signup/${sessionId}/test/otp`);
const { otp } = await response.json();

Problems:

Adds TEST_MODE branches to production code
Doesn't test actual email sending
Diverges from real user flows

Solution: Mailpit API

Saru uses Mailpit (development mail server) API to extract OTP from actually sent emails.

const MAILPIT_API_URL = 'http://localhost:8025/api/v1';

export async function waitForOtpEmail(
  email: string,
  type: 'login' | 'signup',
  maxAttempts = 30,
  sentAfter?: string
): Promise<string | null> {
  const subjectPatterns = {
    login: ['ログインコード', 'Login Code'],
    signup: ['認証コード', 'Verification Code'],
  };

  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    await new Promise(resolve => setTimeout(resolve, 1000));

    const response = await fetch(`${MAILPIT_API_URL}/messages`);
    const data = await response.json();

    // Search for email
    const otpEmail = data.messages.find(msg => {
      // Check recipient
      if (!msg.To.some(to => to.Address === email)) return false;
      // Check subject pattern
      if (!subjectPatterns[type].some(p => msg.Subject.includes(p))) return false;
      // Timestamp filter (explained below)
      if (sentAfter && new Date(msg.Created) < new Date(sentAfter)) return false;
      return true;
    });

    if (otpEmail) {
      // Extract 6-digit OTP
      const match = otpEmail.Snippet.match(/(\d{6})/);
      if (match) return match[1];
    }
  }
  return null;
}

Key Points:

Tests actual email sending flow
Supports both Japanese/English subject patterns
Polls for up to 30 seconds (handles SMTP cold start delays)

3. Preventing Email Race Conditions in Parallel Tests

The Problem: OTP Mix-ups in Parallel Execution

When running multiple tests in parallel in CI, tests may accidentally retrieve another test's OTP.

For example:

Test A: Sends OTP to user-a@example.com
Test B: Sends OTP to user-b@example.com
Test A: Searches Mailpit → Gets Test B's OTP

Solution: Timestamp + Unique Address Filtering

Saru combines two methods to prevent race conditions:

Unique email addresses: Each test uses a different email address
Timestamp filtering: Record time before OTP request, search only emails after that time

// Record timestamp before login
const sentAfter = new Date().toISOString();

// Submit email address (unique per test)
await page.fill('input[type="email"]', email);
await page.getByRole('button', { name: 'Login' }).click();

// Get OTP with timestamp filtering
const otp = await waitForOtpEmail(email, 'login', 30, sentAfter);

// Filtering inside waitForOtpEmail
const otpEmail = data.messages.find(msg => {
  // Recipient check (narrow down by unique address)
  if (!msg.To.some(to => to.Address === email)) return false;

  // Timestamp filter (exclude old emails)
  if (sentAfter) {
    const emailTime = new Date(msg.Created).getTime();
    const filterTime = new Date(sentAfter).getTime();
    if (emailTime < filterTime) return false;
  }
  return true;
});

Alternative Approaches for Parallel Testing

More robust methods to consider:

Method	Pros	Cons
Unique address + timestamp (Saru's approach)	Simple, no backend changes	Vulnerable to clock skew
Embed X-Request-ID in email	Uniquely identifies email	Requires backend changes
Mailpit Search API	Direct filtering by conditions	Depends on API features

Saru's approach prioritizes "simple and works well enough."

Deprecated: clearMailpit()

Previously, clearMailpit() deleted all emails before each test, but in parallel execution this deletes other tests' emails too. Timestamp filtering made this function deprecated.

/**
 * @deprecated Use timestamp-based filtering instead.
 * This function causes race conditions in parallel tests.
 */
export async function clearMailpit(): Promise<void> {
  await fetch(`${MAILPIT_API_URL}/messages`, { method: 'DELETE' });
}

4. Appendix: Locale-Specific Testing

Not directly related to authentication testing, but a useful technique for E2E testing multilingual apps.

The Challenge with Multilingual E2E

Common approach:

// Regex to support multiple languages (old approach)
await expect(page.getByRole('button', {
  name: /(ログイン|Login|登录)/
})).toBeVisible();

Problem: Regex must be updated every time a language is added.

Locale-Specific Testing Pattern

In Saru, we fix the language at test time and directly verify that language's text.

// e2e/utils/locale.ts
export async function setLocale(
  context: BrowserContext,
  locale: 'ja' | 'en'
): Promise<void> {
  // Note: domain: 'localhost' may behave differently across browsers
  // Consider using url option if issues arise
  await context.addCookies([{
    name: 'locale',
    value: locale,
    domain: 'localhost',
    path: '/',
  }]);
}

// Test file
const TEXT = {
  LOGIN: 'ログイン',
  PRODUCT_NAME: '商品名',
  CREATE: '作成',
} as const;

test.beforeEach(async ({ context }) => {
  await setLocale(context, 'ja');
});

test('should create a product', async ({ page }) => {
  await page.getByLabel(TEXT.PRODUCT_NAME).fill('テスト商品');
  await page.getByRole('button', { name: TEXT.CREATE }).click();
});

Benefits: Text is explicit and readable; impact scope is clear when adding languages.

5. CI Configuration: Parallel Execution on Self-hosted Runners

Matrix Strategy for Parallelization

GitHub Actions uses matrix for parallel execution by portal.

# .github/workflows/e2e-tests.yml
jobs:
  e2e:
    runs-on: [self-hosted, linux, x64]
    strategy:
      fail-fast: false
      matrix:
        portal:
          - name: system
            tests: "e2e/system-*.spec.ts e2e/system-portal/*.spec.ts"
            api_port: 8080
          - name: provider
            tests: "e2e/provider-portal/*.spec.ts"
            api_port: 8081
          # ... other portals

Separating Cross-Portal Tests

Tests spanning multiple portals (e.g., Provider→Reseller integration) run in a separate job.

Reasons:

Tests logging in as the same user compete
OTP retrieval timing overlaps

e2e-cross-portal:
  needs: [db-setup, e2e]  # Run after other E2E tests complete
  runs-on: [self-hosted, linux, x64]
  steps:
    - name: Run cross-portal tests
      run: |
        pnpm exec playwright test \
          e2e/auth.spec.ts \
          e2e/dashboard.spec.ts \
          e2e/search-filters.spec.ts

6. Running Cross-Portal Tests Locally

Since it takes 15-20 minutes to reach cross-portal tests in CI, we have scripts for local verification first.

# Run all cross-portal tests
./scripts/run-e2e-cross-portal.sh

# Smoke tests only
./scripts/run-e2e-cross-portal.sh smoke

# Run with visible browser
./scripts/run-e2e-cross-portal.sh --headed

# Playwright UI mode
./scripts/run-e2e-cross-portal.sh --ui

Summary

Challenge	Solution	Constraints/Notes
Testing WebAuthn authentication	CDP virtual authenticator	Chromium only
OTP email retrieval	Mailpit API integration	Requires polling
Email race conditions in parallel tests	Unique address + timestamp	Watch for clock skew
Multilingual UI testing	Locale-specific testing	Cookie setup dependent
CI execution time	Matrix parallelization + cross-portal separation	Complex job design

With these mechanisms, Saru's main authentication flows are automated in CI. Production-specific issues (external IdP outages, browser update behavior changes, etc.) still require manual verification, but manual testing in the daily development cycle has been significantly reduced.

Series Articles

Part 1: Tackling Unmanageable Complexity with Automation
Part 2: Testing WebAuthn in CI (this article)

Tackling Unmaintainable Complexity with Automation: Building a Multi-Tenant SaaS Solo - Part 1

ko-chan — Sat, 27 Dec 2025 15:51:00 +0000

This article was originally published on Saru Blog.

What You'll Learn

How to approach building systems that exceed "what one person can maintain"
Overview of complex multi-tenant system design
Automation strategy to achieve zero manual testing

Introduction

"Don't build what you can't maintain alone."

I believe this is a fundamental rule of solo development. But I wanted to challenge that limit.

After using AI coding agents (Claude Code, GitHub Copilot, etc.) in production for several months, I started thinking, "Maybe I can build something complex on my own." But there's a condition: thorough automation.

In this blog, I'll document the development of "Saru," a multi-tenant subscription management system. I'll share what I automated and how I designed it to build a complex system as a solo developer.

Why Challenge "Unmaintainable Complexity"?

I want to build something "serious" with AI coding agents, not just "simple stuff"
I want to test how far automation can take me against systems normally impossible for one person
Even if I fail, it's a learning experience. If I succeed, it becomes reproducible know-how

Saru's Complexity

4-Tier Account Structure

Typical SaaS has 2 tiers: "Admin → User." Saru has 4.

Tier	Role	Description
System	Platform Management	Oversees everything
Provider	Service Provision	Provides SaaS and products
Reseller	Sales Agency	Sells Provider's services
Consumer	Purchase & Use	Purchases subscriptions

Additionally:

Resellers with PROVIDE capability can offer their own services
Consumers with PROVIDE capability become Creators (individual entrepreneurs)

This flexibility is both the source of complexity and a differentiating point.

4 Portals × 4 APIs

Each tier has its own dedicated frontend and API.

Portal	API	Ports
System Portal	system-api	3001 / 8080
Provider Portal	provider-api	3002 / 8081
Reseller Portal	reseller-api	3003 / 8082
Customer Portal	customer-api	3004 / 8083

Portals are separated by dynamic subdomains (e.g., provider-xxx.example.com)

Other Complexities

Authentication: Keycloak integration, WebAuthn passkeys, OTP authentication, portal session isolation
Data Isolation: Multi-tenant isolation via PostgreSQL Row-Level Security (RLS)
Permission Control: Capability model (CONSUME/PROVIDE/RESELL/ADMINISTER)

Automation Strategy: Zero Manual Testing

To maintain this complexity solo, I adopted a policy of completely eliminating manual testing.

E2E Testing (Playwright)

WebAuthn (passkey) authentication testing is also automated
Executable in CI environments using virtual authenticators
Covers major flows across all portals

CI/CD

Self-hosted GitHub Actions Runner (on WSL2)
Automatic E2E test execution on PR creation
Lint (ESLint, golangci-lint) & type checking (TypeScript)

Development Flow

Using Claude Code's speckit workflow, I consistently perform specification → design → implementation → verification.

/speckit.specify (Create specification)
    ↓
/qa.verify-spec (Verify specification) ← Run after specification
    ↓
/speckit.clarify + Expert verification (Resolve ambiguities)
    ↓
/speckit.plan (Design planning)
    ↓
/qa.verify-design (Verify design) ← Run after design
    ↓
/speckit.tasks (Generate tasks)
    ↓
/speckit.analyze (Check consistency) ← Always run before implementation
    ↓
/speckit.implement (Implementation)
    ↓
/qa.verify-* (Various verifications)
    ↓
Test → Lint → Commit → PR → CI

Tools by Phase

By deciding which tools to use for each phase, I prevent oversights.

Phase	Supporting Tools
Specification	Context7, Tavily, Sequential Thinking
Clarification	backend-architect, security-engineer
Design Planning	backend-architect, Context7
Task Generation	Serena, Sequential Thinking
Consistency Check	Serena, Codex
Implementation	Serena, security-engineer

Verification Commands

Specification & Design Verification:

Command	Timing	Purpose
`/qa.verify-spec`	After specify	Verify specification (requirement coverage, feasibility, contradictions)
`/qa.verify-design`	After plan	Verify design (architecture validity, consistency with existing patterns)

Post-Implementation Verification:

/qa.verify-impl → /qa.verify-test → Test → Lint → Commit → PR → CI

Command	Purpose
`/qa.verify-impl`	Verify implementation code (design consistency, code style, security)
`/qa.verify-test`	Verify test code (coverage, test patterns)
`/qa.verify-migration`	Verify DB migrations (Up/Down consistency, RLS, indexes)
`/qa.verify-rls`	Verify RLS policies (tenant isolation, cross-access prevention)
`/qa.verify-api`	Verify API design (REST design, authentication, error handling)

Problem Investigation:

Command	Purpose
`/qa.investigate`	Multi-faceted investigation (using MCP, expert agents, sub-agents)

Tech Stack

Area	Technology
Frontend	Next.js 14 (App Router), TypeScript, TanStack Query, shadcn/ui
Backend	Go, Echo, sqlc
Database	PostgreSQL with Row-Level Security
Auth	Keycloak
Test	Playwright (E2E), WebAuthn Virtual Authenticator
CI/CD	GitHub Actions (Self-hosted Runner)

Screenshots

System Portal Login Screen

Enter your email address and send an OTP.

OTP Input Screen

Enter the 6-digit one-time password to authenticate.

Provider Portal Signup Screen

Screen for registering as a new service provider.

Features Implemented So Far

Basic UI for all 4 portals
Basic endpoints for all 4 APIs
Keycloak authentication integration (WebAuthn + OTP)
Account & User CRUD
Data isolation via RLS
Permission control via Capability model
Playwright E2E tests (WebAuthn-compatible)
CI/CD pipeline

Future Plans

Internationalization (i18n)
Multi-currency support
Subscription management features
Payment integration (Stripe, etc.)
Automatic provisioning via Webhook integration