Forem: Johnny

Why I Ditched toString() and Moved to a Pre-Bundled Bridge with Event-Driven CDP Injection

Johnny — Fri, 12 Dec 2025 00:24:08 +0000

TL;DR: My first injection strategy serialized bridge classes with class.toString() and eval'd the result into an isolated world. It worked—until it didn't. It was brittle, undebuggable, and race-prone. I replaced it with a pre-bundled bridge that's auto-loaded into a named isolated world via CDP and discovered through Runtime.executionContextCreated events. The result: flakiness dropped, real source maps, fewer CDP round-trips, and better navigation resilience—without changing the public MCP surface.

GitHub: https://github.com/verdexhq/verdex-mcp

The Problem: When String Concatenation Meets Production

When I first started working on Verdex, I needed to inject analysis code into isolated worlds (Chrome's mechanism for running code with DOM access but a separate JavaScript heap). The approach seemed straightforward: build the bridge out of classes and modules, serialize everything with class.toString(), concatenate it into one giant IIFE string, and Runtime.evaluate the result into an isolated world.

It worked. The bridge ran. Tests passed. I shipped it.

Then I started actually using it to author tests in a real application. The first refactor took two hours to debug—I'd added a dependency to one class and forgot to update the string concatenation order. The error message: ReferenceError: Cannot access 'Helper' before initialization pointing somewhere deep in a 30,000-character concatenated string. Every navigation meant rebuilding and re-evaling the entire blob. Around SPA transitions, race conditions appeared where the old world was dying and the new one wasn't ready. My code had to poll and retry with arbitrary sleeps. When I tried to debug a selector generation issue, the breakpoint landed in the eval'd code with no source mapping to show which original TypeScript file was at fault.

The system worked, mostly. But "mostly" isn't good enough when you're building test authoring tooling.

What I Built Instead

I added esbuild to produce a single IIFE bundle with inline source maps and a baked-in version number:

// build/bundle-bridge.ts
const DEV = process.env.NODE_ENV !== "production";

const result = await esbuild.build({
  entryPoints: [path.join(__dirname, "../src/browser/bridge-entry.ts")],
  bundle: true,
  format: "iife",
  platform: "browser",
  target: "es2020",
  minify: !DEV,
  sourcemap: DEV ? "inline" : false,
  sourcesContent: DEV,
  banner: {
    js: [
      `// Verdex Bridge Bundle v${version}`,
      `// Generated: ${new Date().toISOString()}`,
    ].join("\n"),
  },
  define: {
    __VERSION__: JSON.stringify(version),
  },
  write: false,
});

The output gets written to src/runtime/bridge-bundle.ts:

export const BRIDGE_BUNDLE = "/* ~30KB bundled IIFE */";
export const BRIDGE_VERSION = "1.2.3";

The bridge entry point exposes a minimal factory inside the isolated world:

// src/browser/bridge-entry.ts
import { BridgeFactory } from "./bridge/BridgeFactory.js";

// Replaced at build time by esbuild `define`
declare const __VERSION__: string;
export const __VERDEX_BRIDGE_VERSION__ = __VERSION__;

export function createBridge(config?: BridgeConfig): IBridge {
  return BridgeFactory.create(config);
}

// Expose factory to the isolated world, with version checking for upgrades
(function expose() {
  const factory = Object.freeze({
    create: createBridge,
    version: __VERDEX_BRIDGE_VERSION__,
  } satisfies VerdexBridgeFactory);

  const existing = globalThis.__VerdexBridgeFactory__;

  if (!existing || existing.version !== __VERDEX_BRIDGE_VERSION__) {
    if (existing) {
      delete (globalThis as any).__VerdexBridgeFactory__;
    }

    Object.defineProperty(globalThis, "__VerdexBridgeFactory__", {
      value: factory,
      writable: false,
      enumerable: false,
      configurable: true, // Allow redefinition for version upgrades
    });
  }
})();

Then I register it once with CDP and let Chrome handle persistence:

// BridgeInjector.setupAutoInjection(cdp, mainFrameId)
await cdp.send('Page.enable');
await cdp.send('Runtime.enable');

// auxData contains CDP metadata like frameId
cdp.on('Runtime.executionContextCreated', (evt) => {
  const ctx = evt.context;
  const aux = ctx.auxData ?? {};
  const matchesWorld = ctx.name === worldName || aux.name === worldName;
  const matchesTop = !this.mainFrameId || aux.frameId === this.mainFrameId;

  if (matchesWorld && matchesTop) {
    this.contextId = ctx.id;
    this.navigationInProgress = false;
    this.resolveContextReady();
  }
});

// Three-tier fallback for CDP compatibility
try {
  const { identifier } = await cdp.send('Page.addScriptToEvaluateOnNewDocument', {
    source: BRIDGE_BUNDLE,
    worldName: 'verdex_isolated',
    runImmediately: true,
  } as any);
  this.scriptId = identifier;
} catch {
  // Fallback for older Chrome versions without runImmediately
  try {
    const { identifier } = await cdp.send('Page.addScriptToEvaluateOnNewDocument', {
      source: BRIDGE_BUNDLE,
      worldName: 'verdex_isolated',
    } as any);
    this.scriptId = identifier;
  } catch {
    // Very old Chromium: fallback to manual per-navigation reinjection
    this.manualInjectionMode = true;
  }
}

The runImmediately: true flag executes the bundle in the current document immediately and in all future documents after navigation. Named worlds persist across navigations—Chrome destroys the old execution context and creates a new one with the same name, automatically running the registered script.

For navigation safety, I added guards:

cdp.on('Page.frameStartedLoading', (evt) => {
  if (this.isTopFrame(evt.frameId)) {
    this.navigationInProgress = true;
    this.contextId = null;
    this.bridgeObjectId = null;
  }
});

cdp.on('Page.navigatedWithinDocument', (evt) => {
  if (this.isTopFrame(evt.frameId)) {
    // SPA route change: keep context alive, just invalidate instance handle
    this.bridgeObjectId = null;
  }
});

cdp.on('Runtime.executionContextCreated', (evt) => {
  const ctx = evt.context;
  const aux = ctx.auxData ?? {};
  const matchesWorld = ctx.name === worldName || aux.name === worldName;
  const matchesTop = !this.mainFrameId || aux.frameId === this.mainFrameId;

  if (matchesWorld && matchesTop) {
    this.contextId = ctx.id;
    this.navigationInProgress = false;
    this.resolveContextReady();
  }
});

async callBridgeMethod(cdp: CDPSession, method: string, args: any[]) {
  await this.waitForNavToClear();
  // ... execute the call
}

And version enforcement in getBridgeHandle():

async getBridgeHandle(cdp: CDPSession): Promise<string> {
  await this.waitForNavToClear();

  if (this.bridgeObjectId) {
    const alive = await this.healthCheck(cdp);
    if (alive) return this.bridgeObjectId;
    this.bridgeObjectId = null;
  }

  await this.waitForContextReady();
  if (!this.contextId) {
    throw new Error("No execution context available for the bridge world");
  }

  // Verify factory exists
  const { result: factoryType } = await cdp.send('Runtime.evaluate', {
    expression: 'typeof globalThis.__VerdexBridgeFactory__',
    contextId: this.contextId,
    returnByValue: true,
  });

  if (factoryType.value !== 'object') {
    throw new Error(
      `Bridge factory not available in context (got: ${factoryType.value})`
    );
  }

  // Verify version matches
  const { result: versionCheck } = await cdp.send('Runtime.evaluate', {
    expression: 'globalThis.__VerdexBridgeFactory__?.version',
    contextId: this.contextId,
    returnByValue: true,
  });

  if (versionCheck.value !== BRIDGE_VERSION) {
    throw new Error(
      `Bridge version mismatch: got ${versionCheck.value}, expected ${BRIDGE_VERSION}`
    );
  }

  // Create bridge instance
  const { result } = await cdp.send('Runtime.evaluate', {
    expression: `(function(config){ return globalThis.__VerdexBridgeFactory__.create(config); })(${JSON.stringify(
      this.config
    )})`,
    contextId: this.contextId,
    returnByValue: false,
  });

  if (!result.objectId) {
    throw new Error("Failed to create bridge instance (no objectId)");
  }

  this.bridgeObjectId = result.objectId;
  return this.bridgeObjectId;
}

Before and After

Old flow:

// 1. Serialize classes to strings
const bridgeString = [
  BridgeFactory.toString(),
  DOMAnalyzer.toString(),
  StructuralAnalyzer.toString(),
  // ... more classes
].join(';\n');

// 2. Wrap in IIFE
const iifeString = `(function() { ${bridgeString}; return BridgeFactory; })()`;

// 3. Create isolated world manually
const { executionContextId } = await cdp.send('Page.createIsolatedWorld', {
  frameId: mainFrameId,
  worldName: 'verdex_isolated',
});

// 4. Evaluate the string
const { result } = await cdp.send('Runtime.evaluate', {
  expression: iifeString,
  contextId: executionContextId,
});

// 5. On every navigation: repeat steps 2-4

New flow:

// 1. At build time: esbuild produces BRIDGE_BUNDLE

// 2. At runtime, once:
await cdp.send('Page.addScriptToEvaluateOnNewDocument', {
  source: BRIDGE_BUNDLE,
  worldName: 'verdex_isolated',
  runImmediately: true,
});

// 3. Wait for event (happens automatically on navigation)
cdp.on('Runtime.executionContextCreated', (evt) => {
  const ctx = evt.context;
  const aux = ctx.auxData ?? {};
  if (ctx.name === 'verdex_isolated' || aux.name === 'verdex_isolated') {
    this.contextId = ctx.id;
  }
});

// 4. Create bridge via factory
const { result } = await cdp.send('Runtime.callFunctionOn', {
  functionDeclaration: `
    function() {
      return globalThis.__VerdexBridgeFactory__.create(arguments[0]);
    }
  `,
  contextId: this.contextId,
  arguments: [{ value: config }],
});

What Changed

Flakiness dropped from roughly 5% on navigation-heavy tests to zero. The old approach failed with "Cannot find execution context with given id" during race windows. The event-driven lifecycle eliminated those races entirely.

Stack traces went from "(eval):1:15000" to "DOMAnalyzer.ts:42". I can set breakpoints in the original TypeScript files and step through execution. DevTools treats the bundled code as first-class source because of the inline source maps (in development mode).

Re-injection overhead disappeared. The old approach added 50-100ms per navigation. The new approach: zero. The bundle loads once and persists automatically.

The implementation collapsed. Manual world lifecycle tracking, string concatenation logic, import ordering requirements, and retry loops scattered through multiple files became a single BridgeInjector class that registers once and listens for events.

The API Stayed Identical

This was entirely internal. Tool handlers work exactly as before:

const snapshot = await bridgeInjector.callBridgeMethod(cdp, 'snapshot', []);
const ancestors = await bridgeInjector.callBridgeMethod(cdp, 'get_ancestors', [ref]);
const siblings = await bridgeInjector.callBridgeMethod(cdp, 'get_siblings', [ref, level]);
const descendants = await bridgeInjector.callBridgeMethod(cdp, 'get_descendants', [ref, level]);

Users calling browser_snapshot(), resolve_container(), inspect_pattern(), and extract_anchors() saw no change. The MCP protocol surface stayed stable. Tests kept passing. Prompts kept working.

Multi-Role Still Works

Each role can get its own named world by configuring the BridgeInjector with a role-specific world name. This allows multiple isolated contexts to coexist without interfering with each other—useful when you need different analysis modes or want to maintain separate state per role:

new BridgeInjector({
  worldName: `verdex_${role}`,
  config: roleConfig
});

The bridge survives reloads and SPA transitions per role. Storage isolation prevents cookie/localStorage leakage. World isolation prevents bridge interference.

Security Holds

The factory isn't visible from the page's main world:

const leaked = await page.evaluate(() => {
  return typeof (globalThis as any).__VerdexBridgeFactory__;
});

expect(leaked).toBe('undefined');

Isolated worlds give DOM access without exposing internals to application code.

What I Learned

Strings obfuscate. String concatenation becomes impossible to debug when your codebase grows beyond trivial.
Events beat polling. Runtime.executionContextCreated is the canonical source of truth for "your world exists now."
Version pinning prevents hours of debugging subtle runtime failures caused by mismatches.
Source maps are non-negotiable for any injected code longer than a few lines.
Keep the API stable. Implementation details should be invisible.

Trade-Offs

The bundle adds ~50KB to memory per browser context. For a tool managing 3-5 contexts, this is negligible.

Build-time bundling adds a development step. esbuild finishes in ~100ms and I've integrated it into the watch process, but it's still an extra step.

Top-frame only currently. Supporting iframes would require tracking auxData.frameId and maintaining a per-frame context ID map. Architecturally straightforward, not yet implemented.

Conclusion

The primitives—resolve_container(), inspect_pattern(), extract_anchors()—are only as reliable as the infrastructure delivering them. The old approach worked in controlled scenarios but broke under real-world stress. The new approach handles navigation, development iteration, and production debugging deterministically because it's built on browser-provided lifecycle guarantees rather than polling and string manipulation.

The code is cleaner. The debugging is better. The reliability is higher. And users don't have to care about any of it.

How Verdex Sees Inside Iframes: Event-Driven Multi-Frame Support

Johnny — Fri, 12 Dec 2025 00:22:43 +0000

TL;DR: Verdex initially treated iframes as invisible—snapshots captured the <iframe> tag but none of the content inside. This made Stripe checkouts, PayPal buttons, and embedded widgets completely opaque to the system. I rebuilt the bridge injection layer to support per-frame isolated worlds with lazy snapshot expansion, using Playwright's event-driven patterns. The result: nested iframe content appears in snapshots with frame-qualified refs (f1_e3), interactions route to the correct frame automatically, and the entire MCP API stayed identical.

GitHub: https://github.com/verdexhq/verdex-mcp

The Problem: When Your Test Tool Can't See the Payment Form

The first time I tried to use Verdex on a real checkout flow, I got a snapshot like this:

- button "Add to Cart" [ref=e1]
- iframe [ref=e2]
- button "Continue Shopping" [ref=e3]

The Stripe payment form was inside ref=e2. The snapshot showed the iframe element itself, but the actual payment fields—card number, expiration, CVC—were invisible. The LLM could see there was an iframe, but had no way to interact with anything inside it.

I tried click(ref=e2). It clicked the iframe element, which did nothing. The actual payment button was nested three layers deep: main page → iframe → shadow DOM → button. Verdex's bridge lived in the main page's isolated world. It had no visibility into child frames, no way to snapshot their content, and no way to route interactions.

The problem wasn't just payment forms. Google's OAuth prompts, embedded maps, chat widgets, video players—anything using iframes was a black box.

What I Built Instead

I needed per-frame bridge instances. Each frame (main and all iframes) gets its own isolated world with its own bridge. Snapshots recursively expand iframe markers by snapshotting child frames and merging the output with frame-qualified refs. Interactions parse the ref, look up the target frame, and route the method call there.

The architecture is lazy: bridges aren't injected into iframes until the first snapshot that needs them. This avoids overhead on pages with many hidden iframes (analytics trackers, ad pixels) while still handling interactive content.

ManualPromise: Playwright's Event-Driven Pattern

The core challenge was knowing when a frame's execution context was ready. The old approach would've been polling with retries:

// ❌ Polling approach (what I didn't do)
async ensureFrameReady(frameId: string) {
  for (let i = 0; i < 10; i++) {
    try {
      await cdp.send('Runtime.evaluate', {
        expression: '1 + 1',
        contextId: guessedContextId
      });
      return; // It worked!
    } catch {
      await sleep(50 * Math.pow(2, i)); // Exponential backoff
    }
  }
  throw new Error('Frame never became ready');
}

That's brittle. Navigation timing varies. Arbitrary delays cause false timeouts. It's not deterministic.

Instead, I adopted Playwright's ManualPromise pattern:

export class ManualPromise<T = void> extends Promise<T> {
  private _resolve!: (value: T) => void;
  private _reject!: (error: Error) => void;
  private _isDone = false;

  constructor() {
    let resolve: (value: T) => void;
    let reject: (error: Error) => void;
    super((f, r) => {
      resolve = f;
      reject = r;
    });
    this._resolve = resolve!;
    this._reject = reject!;
  }

  resolve(value: T): void {
    if (this._isDone) return;
    this._isDone = true;
    this._resolve(value);
  }

  reject(error: Error): void {
    if (this._isDone) return;
    this._isDone = true;
    this._reject(error);
  }

  isDone(): boolean {
    return this._isDone;
  }
}

Each frame gets a contextReadyPromise that resolves when Runtime.executionContextCreated fires for its isolated world:

type FrameState = {
  frameId: string;
  contextId: number;
  bridgeObjectId: string;
  contextReadyPromise: ManualPromise<void>;
};

// When CDP tells us the context exists, resolve the promise
cdp.on('Runtime.executionContextCreated', (evt) => {
  const ctx = evt.context;
  const frameId = ctx.auxData?.frameId;
  const matchesWorld = ctx.name === this.worldName || 
                       ctx.auxData.name === this.worldName;

  if (matchesWorld && frameId) {
    const frameState = this.getOrCreateFrameState(cdp, frameId);
    frameState.contextId = ctx.id;
    frameState.contextReadyPromise.resolve(); // ← Event-driven!
  }
});

// Anywhere that needs a ready frame just awaits
async ensureFrameState(cdp: CDPSession, frameId: string): Promise<FrameState> {
  let state = this.getFrameState(cdp, frameId);

  if (state?.contextReadyPromise.isDone()) {
    return state; // Already ready
  }

  if (state) {
    await state.contextReadyPromise; // Wait for event
    return state;
  }

  // Create new state and trigger isolated world creation
  state = this.getOrCreateFrameState(cdp, frameId);

  await cdp.send('Page.createIsolatedWorld', {
    frameId,
    worldName: this.worldName,
  });

  await state.contextReadyPromise; // Wait for executionContextCreated event

  // Inject bundle into the now-ready context
  await cdp.send('Runtime.evaluate', {
    expression: BRIDGE_BUNDLE,
    contextId: state.contextId,
  });

  return state;
}

No polling. No retries. No arbitrary delays. The browser tells us when the context exists, and we immediately proceed. If a frame detaches during injection, the frameDetached event rejects the promise—no timeouts needed.

Frame Lifecycle Tracking

I added listeners for the full frame lifecycle:

// Frame appears (lazy injection - don't inject until needed)
cdp.on('Page.frameAttached', (evt) => {
  this.getOrCreateFrameState(cdp, evt.frameId);
});

// Frame disappears (reject pending promises, clean up state)
cdp.on('Page.frameDetached', (evt) => {
  const state = sessionStates.get(evt.frameId);
  if (state && !state.contextReadyPromise.isDone()) {
    state.contextReadyPromise.reject(new FrameDetachedError(evt.frameId));
  }
  sessionStates.delete(evt.frameId);
});

// Same-document navigation (SPA routing)
cdp.on('Page.navigatedWithinDocument', (evt) => {
  const state = sessionStates?.get(evt.frameId);
  if (state) {
    state.bridgeObjectId = ''; // Invalidate bridge instance, keep context
  }
});

// Cross-document navigation (full page reload)
cdp.on('Page.frameNavigated', (evt) => {
  if (evt.frame && !evt.frame.parentId) {
    // Context will be destroyed and recreated - clear state
    sessionStates.delete(evt.frame.id);
  }
});

Error Tracking and Failure Logs

The implementation includes comprehensive error tracking to help debug multi-frame issues:

type FrameExpansionError = {
  ref: string;        // Which iframe ref failed
  error: string;      // Error message
  detached: boolean;  // Was it a detachment (expected) or other error?
  timestamp?: number; // When did it happen?
};

// Snapshots include expansion errors
interface Snapshot {
  text: string;
  elementCount: number;
  pageContext: { url: string; title: string };
  expansionErrors?: Array<{ ref: string; error: string; detached: boolean }>;
  warnings?: {
    inaccessibleFrames?: number;
    partialContent?: boolean;
    details?: string[];
  };
}

When frame expansion fails, the error is tracked and included in the snapshot. This gives LLMs context about why certain iframes might be missing, and helps developers debug cross-origin or timing issues.

Lazy Snapshot Expansion

At snapshot time, I recursively expand iframe markers:

async snapshot(): Promise<Snapshot> {
  const context = await this.ensureCurrentRoleContext();

  // Get main frame snapshot (includes iframe markers)
  const mainSnapshot = await context.bridgeInjector.callBridgeMethod(
    context.cdpSession,
    "snapshot",
    [],
    context.mainFrameId
  );

  // Build refIndex: main frame refs go in first
  const refIndex = new Map<string, RefIndexEntry>();
  const mainFrameRefs = mainSnapshot.text.matchAll(/\[ref=([^\]]+)\]/g);
  for (const match of mainFrameRefs) {
    const ref = match[1];
    refIndex.set(ref, { 
      frameId: context.mainFrameId, 
      localRef: ref 
    });
  }

  // Recursively expand iframes
  const expanded = await this.expandIframes(
    context,
    mainSnapshot.text,
    context.mainFrameId,
    0, // ordinal counter
    refIndex
  );

  // Store refIndex on context for interaction routing
  context.refIndex = refIndex;

  // Build snapshot with error tracking
  const snapshot: Snapshot = {
    text: expanded.text,
    elementCount: mainSnapshot.elementCount + expanded.elementCount,
    pageContext: {
      url: context.page.url(),
      title: await context.page.title(),
    },
  };

  // Add expansion errors to snapshot if any occurred
  if (expanded.errors.length > 0) {
    snapshot.expansionErrors = expanded.errors;
  }

  return snapshot;
}

The expansion logic walks the snapshot text, finds iframe markers, resolves each to a child frame ID using CDP's DOM.describeNode, snapshots that frame, and merges the output:

private async expandIframes(
  context: RoleContext,
  snapshotText: string,
  currentFrameId: string,
  ordinalCounter: number,
  refIndex: GlobalRefIndex
): Promise<{ 
  text: string; 
  elementCount: number; 
  nextOrdinal: number;
  errors: Array<{ ref: string; error: string; detached: boolean }>;
}> {
  const lines = snapshotText.split('\n');
  const result: string[] = [];
  let totalElements = 0;
  let nextOrdinal = ordinalCounter;
  const errors: Array<{ ref: string; error: string; detached: boolean }> = [];

  for (const line of lines) {
    // Match: "- iframe [ref=e5]"
    const match = line.match(/^(\s*)- iframe(?:\s+"[^"]*")?\s+\[ref=([^\]]+)\]/);

    if (!match) {
      result.push(line);
      continue;
    }

    const indentation = match[1];
    const iframeRef = match[2];

    result.push(line + ':'); // Add colon to show children

    try {
      // Resolve iframe ref to child frame ID
      const frameInfo = await this.resolveFrameFromRef(
        context, 
        currentFrameId, 
        iframeRef
      );

      if (!frameInfo) {
        result.push(indentation + '  [Frame content unavailable]');
        continue;
      }

      const frameOrdinal = ++nextOrdinal;

      // Snapshot child frame
      const childSnapshot = await context.bridgeInjector.callBridgeMethod(
        context.cdpSession,
        "snapshot",
        [],
        frameInfo.frameId
      );

      // Recursively expand child's iframes
      const expandedChild = await this.expandIframes(
        context,
        childSnapshot.text,
        frameInfo.frameId,
        nextOrdinal,
        refIndex
      );

      nextOrdinal = expandedChild.nextOrdinal;

      // Rewrite child refs: e1 → f1_e1
      // Uses RefFormatter utility for consistent ref formatting
      const rewritten = expandedChild.text.replace(
        /\[ref=(e[^\]]+)\]/g,
        (_whole, localRef) => {
          // Only rewrite local refs, not already-qualified refs
          if (!RefFormatter.isLocal(localRef)) {
            return `[ref=${localRef}]`; // Already qualified from nested iframe
          }

          const globalRef = RefFormatter.toGlobal(frameOrdinal, localRef);
          refIndex.set(globalRef, { 
            frameId: frameInfo.frameId, 
            localRef 
          });
          return `[ref=${globalRef}]`;
        }
      );

      // Indent and merge
      for (const childLine of rewritten.split('\n')) {
        if (childLine.trim()) {
          result.push(indentation + '  ' + childLine);
        }
      }
    } catch (error) {
      // Track errors for snapshot warnings
      const isDetached = this.isFrameDetachedError(error);
      const errorMsg = error instanceof Error ? error.message : String(error);

      errors.push({
        ref: iframeRef,
        error: errorMsg,
        detached: isDetached,
      });

      // Frame detachment is normal (logged at debug level)
      if (isDetached) {
        console.debug(`Frame ${iframeRef} detached during expansion`);
        result.push(indentation + '  [Frame detached]');
      } else {
        // Unexpected errors logged as warnings
        console.warn(`Frame expansion error for ${iframeRef}:`, errorMsg);
        result.push(indentation + `  [Error: ${errorMsg}]`);
      }
    }
  }

  return { 
    text: result.join('\n'), 
    elementCount: totalElements, 
    nextOrdinal,
    errors // Returned for snapshot warnings
  };
}

Frame Resolution via CDP

To map an iframe element to its frame ID, I use DOM.describeNode:

private async resolveFrameFromRef(
  context: RoleContext,
  parentFrameId: string,
  iframeRef: string
): Promise<{ frameId: string } | null> {
  // Get the iframe element from the parent frame's bridge
  const bridgeObjectId = await context.bridgeInjector.getBridgeHandle(
    context.cdpSession,
    parentFrameId
  );

  // Get the iframe element from the parent frame's bridge
  // Access bridge's elements Map directly to get the DOM element
  const { result } = await context.cdpSession.send('Runtime.callFunctionOn', {
    objectId: bridgeObjectId,
    functionDeclaration: `function(ref) { 
      // Get the ElementInfo which contains the actual DOM element
      const info = this.elements.get(ref);
      if (!info) return null;

      // Verify it's an iframe
      if (info.tagName.toUpperCase() !== 'IFRAME') return null;

      // Return the element itself (will have objectId)
      return info.element;
    }`,
    arguments: [{ value: iframeRef }],
    returnByValue: false, // CRITICAL: Get as remote object, not value
  });

  if (!result.objectId) {
    console.warn(`No objectId for iframe ref ${iframeRef}`);
    return null;
  }

  // Use CDP to get the child frameId
  // pierce: true enables traversal into iframe's content document
  const { node } = await context.cdpSession.send('DOM.describeNode', {
    objectId: result.objectId,
    pierce: true,
  });

  // CDP returns either node.frameId or node.contentDocument.frameId depending on version
  const childFrameId = node.frameId || node.contentDocument?.frameId;

  if (!childFrameId) {
    console.warn(
      `Element ${iframeRef} has no associated frame (might be empty or not yet loaded)`
    );
    return null;
  }

  return { frameId: childFrameId };
}

Interaction Routing

All interaction methods parse the ref to determine which frame it belongs to:

private parseRef(ref: string, context: RoleContext): { 
  frameId: string; 
  localRef: string 
} {
  // Check if refIndex exists (should be populated by snapshot())
  if (!context.refIndex) {
    throw new Error(
      "No refIndex found. Take a snapshot first before interacting with elements."
    );
  }

  // Lookup in refIndex (includes both main frame and child frame refs)
  const entry = context.refIndex.get(ref);
  if (entry) {
    return { frameId: entry.frameId, localRef: entry.localRef };
  }

  // If not found, ref is stale or invalid
  throw new UnknownRefError(ref);
}

async click(ref: string): Promise<void> {
  const context = await this.ensureCurrentRoleContext();
  const { frameId, localRef } = this.parseRef(ref, context);

  await context.bridgeInjector.callBridgeMethod(
    context.cdpSession,
    "click",
    [localRef],
    frameId // ← Routes to correct frame
  );
}

The same pattern applies to type(), resolve_container(), inspect_pattern(), and extract_anchors():

async type(ref: string, text: string): Promise<void> {
  const context = await this.ensureCurrentRoleContext();
  const { frameId, localRef } = this.parseRef(ref, context);

  await context.bridgeInjector.callBridgeMethod(
    context.cdpSession,
    "type",
    [localRef, text],
    frameId // Routes to correct frame automatically
  );
}

async resolve_container(ref: string): Promise<any> {
  const context = await this.ensureCurrentRoleContext();
  const { frameId, localRef } = this.parseRef(ref, context);

  return await context.bridgeInjector.callBridgeMethod(
    context.cdpSession,
    "resolve_container",
    [localRef],
    frameId
  );
}

// inspect_pattern and extract_anchors follow the same pattern

Before and After

Old snapshot (iframe is opaque):

- button "Add to Cart" [ref=e1]
- iframe [ref=e2]
- button "Continue Shopping" [ref=e3]

New snapshot (iframe content visible):

- button "Add to Cart" [ref=e1]
- iframe [ref=e2]:
    - heading "Payment Information" [ref=f1_e1]
    - textbox "Card Number" [ref=f1_e2]
    - textbox "Expiration" [ref=f1_e3]
    - textbox "CVC" [ref=f1_e4]
    - button "Pay Now" [ref=f1_e5]
- button "Continue Shopping" [ref=e3]

Old interaction attempt:

await browser.click('e2'); // Clicked the iframe container (did nothing)

New interaction:

await browser.click('f1_e5'); // Clicks "Pay Now" inside the iframe

What Changed

Stripe checkout flows work. The test case I couldn't complete before—filling out payment details in an embedded iframe—now works end-to-end.

Nested iframes work. I can snapshot and interact with iframes inside iframes (tested 3 levels deep).

The refIndex provides O(1) routing. Every ref in the snapshot maps to { frameId, localRef }. No scanning, no heuristics, just a Map lookup.

Error tracking improves debugging. The implementation tracks frame expansion failures separately from injection failures, distinguishing between expected issues (frame detached) and unexpected errors. Snapshots include expansionErrors when iframes couldn't be accessed, and warnings appear in the output.

Event-driven lifecycle eliminated timing bugs. The old single-frame implementation had occasional "Cannot find execution context" errors during navigation. Those don't happen anymore because frame state tracking is event-driven, not timing-dependent.

Test suite expanded by 7 files and ~500 lines. I added comprehensive multi-frame tests:

frame-discovery.spec.ts - Frame enumeration after navigation
frame-resolution.spec.ts - Mapping iframe refs to frame IDs
iframe-snapshot-expansion.spec.ts - Snapshot recursion
interaction-routing.spec.ts - Interaction routing to child frames
iframe-edge-cases.spec.ts - Empty iframes, cross-origin, dynamic injection
multi-frame-bridge.spec.ts - Per-frame bridge isolation
iframe-visual-demo.spec.ts - Visual debugging test with nested iframes

All 188 tests still pass. The refactor touched every interaction method, but I kept the API surface identical.

The API Stayed Identical

Users calling MCP tools see no difference:

// Before multi-frame support
const snapshot = await browser.snapshot();
await browser.click('e5');
const ancestors = await browser.resolve_container('e5');

// After multi-frame support (same calls, now works with iframe refs)
const snapshot = await browser.snapshot();
await browser.click('f1_e5'); // Can target iframe content
const ancestors = await browser.resolve_container('f1_e5'); // Works inside iframes

The ref format is the only visible change: main frame refs stay as e1, e2, etc., while child frame refs become f1_e1, f2_e3, etc. That's progressive enhancement—main-frame-only pages have identical snapshots.

Frame Detachment Detection

A critical helper distinguishes between expected frame lifecycle events and real errors:

private isFrameDetachedError(error: any): boolean {
  if (!error?.message) return false;
  const msg = error.message.toLowerCase();
  return (
    msg.includes('frame detached') ||
    msg.includes('frame has been detached') ||
    msg.includes('execution context was destroyed') ||
    msg.includes('cannot find context')
  );
}

This pattern appears throughout the codebase—frame detachment during operations is expected behavior, not an error. The code logs these at debug level while logging unexpected errors as warnings.

Architecture Principles

Lazy injection. Bridges aren't created until the first snapshot that needs them. This keeps performance high on iframe-heavy pages where most iframes are hidden analytics trackers.

Automatic filtering. Only iframes in the accessibility tree get expanded. Hidden iframes (display: none, aria-hidden="true") are excluded automatically because they don't appear in the snapshot.

Graceful degradation. If one iframe fails to expand (cross-origin restrictions, frame detached during snapshot), the rest of the snapshot still works. The failure appears as [Frame content unavailable] in the output.

Event-driven everything. Frame lifecycle, execution context creation, navigation—all driven by CDP events. No polling, no retries, no arbitrary delays.

What I Learned

Playwright's patterns are production-proven. I adopted ManualPromise, event-driven context tracking, and lazy injection - all ideas I took from Playwright. These patterns handle edge cases I wouldn't have thought of.
Iframes need role assignments to be interactive. Adding case "IFRAME": return "iframe" to AriaUtils.getImplicitRole() wasn't enough—I also had to add "IFRAME" to the INTERACTIVE_ELEMENTS array so SnapshotGenerator would assign refs. Without refs, frame resolution can't work.
Frame detachment is normal, not an error. Frames can detach during snapshot expansion (lazy-loaded iframes, dynamic removal). The code treats this as expected behavior, not a retry case.
CDP's DOM.describeNode has an ambiguity. The docs say node.frameId contains "Frame ID for frame owner elements," but also return node.contentDocument.frameId. I implemented a fallback: node.frameId || node.contentDocument?.frameId. Empirically, both work.
Same-document navigation vs cross-document navigation need different handling. SPA routing (navigatedWithinDocument) keeps the execution context alive but invalidates the bridge instance. Full navigation (frameNavigated) destroys the context entirely.
The refIndex must include main frame refs. Originally I only added child frame refs to the index, which meant parseRef() needed a fallback for main frame refs. Adding all refs to the index simplified the code and made routing uniform.
RefFormatter utility keeps ref handling consistent. Extracting ref formatting logic (toGlobal(), isLocal(), parse()) into a utility class prevented inline string manipulation bugs and made the ref format explicit: main frame uses e1, child frames use f1_e1.
Error categorization matters. The code distinguishes between frame detachment (expected, logged at debug level) and unexpected errors (logged as warnings). This prevents log noise while still surfacing real issues.

Trade-Offs

Snapshot time increases with iframe depth. A page with 3 nested iframes takes ~3x longer to snapshot than main-frame-only. The expansion is serial (depth-first), not parallel. For typical pages (0-2 iframes), the overhead is negligible.

Cross-origin iframes are non-injectable. Chrome's security model prevents CDP from accessing cross-origin iframe content unless the page has permissive CORS headers. This is correct behavior—Verdex shouldn't bypass browser security. Cross-origin iframes appear as [Frame content unavailable] in snapshots.

The ref format changed. Frame-qualified refs (f1_e5) are longer than unqualified refs (e5). This adds ~5-10 tokens per iframe element to the snapshot. For typical usage (1-2 iframes with 5-10 elements each), that's ~50-100 tokens—acceptable for the functionality gained.

Memory overhead per frame. Each frame stores a FrameState with a ManualPromise, context ID, and bridge object ID. For pages with 10+ iframes, this adds ~1-2KB of memory per frame. Negligible for typical usage.

What's Next

Parallel frame expansion. Currently, iframe snapshot expansion is depth-first and serial. Switching to Promise.allSettled() for sibling iframes would reduce snapshot time on iframe-heavy pages.

Frame metadata in snapshots. Adding iframe name, title, or id attributes to the snapshot output would give LLMs better context about what each frame represents: [Frame: name="stripe-checkout", title="Payment Form"].

Performance budgets. For pages with excessive iframes (20+ ad trackers), snapshot time could become problematic. Adding configurable depth limits (maxIframeDepth: 2) would cap recursion.

Cross-origin detection. Currently, cross-origin frames fail silently with [Frame content unavailable]. Detecting cross-origin explicitly and showing [Cross-origin frame - content not accessible] would give clearer feedback.

Conclusion

Multi-frame support turned Verdex from "works on simple pages" to "works on real e-commerce sites." Stripe checkout flows, OAuth popups, embedded widgets—all now visible and interactive. The architecture is event-driven, lazy, and graceful under failure.

The refactor touched 8 files and added ~730 lines of implementation code. The test suite grew by 500 lines. All existing tests pass. The MCP API surface didn't change. Users get iframe support without migration.

The critical insight: iframe support isn't about iframes—it's about making the bridge lifecycle event-driven and multi-context-aware. Once that foundation exists, frame expansion and interaction routing are straightforward.

The debugging is cleaner. The behavior is deterministic. The architecture scales to nested iframes. And users can finally fill out Stripe forms.

Why Your API's Error Messages Fail When Called by an LLM (And How to Fix Them)

Johnny — Fri, 12 Dec 2025 00:19:26 +0000

TL;DR: When you build tools that LLMs call autonomously—OpenAI functions, Claude tools, MCP servers, custom APIs—traditional error messages break the agent workflow. A human can ask "what's a valid reference?" An LLM can't. I rebuilt Verdex's error handling to give LLMs structured recovery plans instead of descriptions. The result: errors that were conversation-killers became recoverable, autonomous workflows stopped failing silently, and the LLM could fix issues without human intervention.

The Problem: When Your User Can't Ask Questions

Here's what happens when a human gets an error:

Error: Unknown element reference: e999

The human thinks:

"What's an element reference?"
"Where do I get valid ones?"
"Why did mine become invalid?"

Then they ask you, and you explain.

Here's what happens when an LLM gets that same error:

Error: Unknown element reference: e999

The LLM:

Tries the same thing again (with the same invalid ref)
Gets the same error
Gives up or hallucinates a solution
The entire agentic workflow fails

The LLM can't ask clarifying questions. It needs to autonomously recover or the task is dead.

What I Was Building

I'm working on Verdex, an MCP server for browser automation. It exposes tools like browser_click(ref), browser_type(ref, text), and browser_snapshot(). The LLM navigates pages, fills forms, and extracts data—all autonomously.

When I first deployed it, I saw this pattern constantly:

LLM: browser_click("e5")
Error: Unknown element reference: e5

LLM: browser_click("e5")  // Tries exact same thing
Error: Unknown element reference: e5

LLM: "I encountered an error. The element reference appears to be invalid."
[workflow ends]

The LLM had no idea that:

Element refs come from browser_snapshot()
Navigation invalidates old refs
It needed to call browser_snapshot() first to get fresh refs

It couldn't ask me. I wasn't there. The error message didn't tell it how to recover.

The Pattern: Structured Recovery Plans

I rebuilt every error message to follow this structure:

❌ [Error Type]

[What failed - with specific details]

[Why it failed - likely causes]

🔧 Action Required:
[Numbered steps to recover]

Here's the before and after:

Before (Developer-Focused)

Error: Unknown element reference: e999

After (LLM-Focused)

❌ Unknown Element Reference

Reference: e999

This reference doesn't exist in the current snapshot.

Possible causes:
• Using a ref from an old snapshot (stale after navigation)
• Typo in the ref name
• Element not yet loaded or not interactive

🔧 Action Required:
1. Call browser_snapshot() to see currently available elements
2. Find the correct element reference in the new snapshot
3. Use the correct ref from the latest snapshot

The difference:

Context: What a "reference" is and where they come from
Diagnosis: Why this specific one is invalid
Recovery: Exact API calls needed to fix it (with names!)
Ordering: Numbered steps the LLM can follow sequentially

Implementation: Rich Error Classes

Every error type gets its own class with structured properties:

export class UnknownRefError extends Error {
  constructor(public ref: string) {
    super(
      `Unknown element reference: ${ref}. ` +
      `Ref may be stale after navigation. Take a new snapshot to get fresh refs.`
    );
    this.name = "UnknownRefError";
  }
}

export class StaleRefError extends Error {
  constructor(
    public ref: string,
    public elementInfo: { role: string; name: string; tagName: string }
  ) {
    super(
      `Element ${ref} (${elementInfo.role} "${elementInfo.name}") was removed from DOM. ` +
      `Take a new snapshot() to refresh refs.`
    );
    this.name = "StaleRefError";
  }
}

export class FrameDetachedError extends Error {
  constructor(public frameId: string, details?: string) {
    super(`Frame ${frameId} was detached${details ? `: ${details}` : ""}`);
    this.name = "FrameDetachedError";
  }
}

export class NavigationError extends Error {
  constructor(public url: string, public role: string, details: string) {
    super(`Navigation failed for role '${role}' to '${url}': ${details}`);
    this.name = "NavigationError";
  }
}

export class AuthenticationError extends Error {
  constructor(
    public role: string,
    public authPath: string,
    public reason: string
  ) {
    super(
      `Authentication required for role '${role}' but failed to load from ${authPath}: ${reason}`
    );
    this.name = "AuthenticationError";
  }
}

The properties (ref, elementInfo, frameId, etc.) enable the formatter to provide specific, actionable guidance.

The Formatter: Error Messages as Recovery Scripts

At the MCP server layer, I intercept all errors and format them for LLM consumption:

async callTool(name: string, args: any) {
  try {
    // Route to appropriate handler...
    return await handler(args);
  } catch (error) {
    return {
      content: [{
        type: "text",
        text: this.formatErrorForLLM(error),
      }],
    };
  }
}

private formatErrorForLLM(error: unknown): string {
  // Unknown reference - ref doesn't exist in snapshot
  if (error instanceof UnknownRefError) {
    return `❌ Unknown Element Reference

Reference: ${error.ref}

This reference doesn't exist in the current snapshot.

Possible causes:
• Using a ref from an old snapshot (stale after navigation)
• Typo in the ref name
• Element not yet loaded or not interactive

🔧 Action Required:
1. Call browser_snapshot() to see currently available elements
2. Find the correct element reference in the new snapshot
3. Use the correct ref from the latest snapshot`;
  }

  // Stale reference - element was removed from DOM
  if (error instanceof StaleRefError) {
    return `❌ Stale Element Reference

Element: ${error.ref}
Type: ${error.elementInfo.role}
Label: "${error.elementInfo.name}"
Tag: <${error.elementInfo.tagName}>

The element was removed from the DOM, likely due to:
• Page navigation or refresh
• Dynamic content update
• JavaScript manipulation

🔧 Action Required:
Call browser_snapshot() to get fresh element references, then retry your action.`;
  }

  // Frame detached - iframe removed during operation
  if (error instanceof FrameDetachedError) {
    return `❌ Frame Detached

Frame ID: ${error.frameId}

An iframe was removed or navigated during the operation.

This is often normal during:
• Navigation between pages
• Single-page app (SPA) route changes
• Dynamic iframe removal by JavaScript

🔧 Action Required:
Call browser_snapshot() to see the current page structure and available frames.`;
  }

  // Authentication failed - required auth cannot load
  if (error instanceof AuthenticationError) {
    return `❌ Authentication Required

Role: ${error.role}
Auth File: ${error.authPath}

Failed to load authentication data: ${error.reason}

This role requires authentication but the auth file couldn't be loaded.

Possible causes:
• Auth file doesn't exist at specified path
• Auth file has invalid JSON format
• Auth file permissions prevent reading
• Path specified incorrectly in configuration

🔧 Action Required:
1. Verify auth file exists: ${error.authPath}
2. Check file permissions (must be readable)
3. Validate JSON format in auth file
4. If auth is optional, set authRequired: false in role config
5. Run auth capture process if credentials expired`;
  }

  // Navigation failed - couldn't navigate to URL
  if (error instanceof NavigationError) {
    return `❌ Navigation Failed

URL: ${error.url}
Role: ${error.role}

${error.message}

Possible causes:
• Invalid or unreachable URL
• Network connectivity issues
• Server error (404, 500, etc.)
• Authentication required (check warnings in snapshot)
• Timeout (page took too long to load)
• Main frame injection failed

🔧 Action Required:
• Verify the URL is correct and accessible
• Check network connectivity
• Call browser_snapshot() to see current page state
• Check role authentication status via getFailures()
• Try a different URL or retry after a moment`;
  }

  // Generic error fallback
  if (error instanceof Error) {
    return `❌ Error

${error.message}

If this error persists, check:
• Your input parameters
• Current page state (call browser_snapshot())
• Network connectivity
• Browser logs for additional context`;
  }

  // Unknown error type
  return `❌ Unknown Error

${String(error)}

This is an unexpected error type. Please report this issue with context about what operation you were attempting.`;
}

Universal Principles for LLM Error Messages

After implementing this across 8 error types, here's what works:

1. Explicit Tool/Function Names

❌ Bad: "Get a new snapshot"

✅ Good: "Call browser_snapshot()"

The LLM needs the exact function name it should call. Don't make it guess.

2. Numbered Recovery Steps

❌ Bad: "You need to refresh the page and try again"

✅ Good:

1. Call browser_navigate(url)
2. Call browser_snapshot() to get new refs
3. Find the button in the new snapshot
4. Retry browser_click() with the new ref

LLMs follow numbered lists well. They struggle with prose instructions.

3. Explain the Why

❌ Bad: "Invalid ref"

✅ Good: "This reference doesn't exist because navigation invalidates old refs"

Understanding causation helps the LLM avoid the same mistake next time.

4. Multiple Diagnosis Options

Possible causes:
• Using a ref from an old snapshot (stale after navigation)
• Typo in the ref name
• Element not yet loaded or not interactive

The LLM can pattern-match against its recent actions to figure out which cause applies.

5. Include Structured Data

Element: e5
Type: button
Label: "Submit"
Tag: <button>

Structured info helps the LLM recognize what it was trying to interact with, making it easier to find the element in a fresh snapshot.

6. Distinguish Expected vs Unexpected

Some failures are normal:

Frame detachment during navigation
Cross-origin iframe access denied
Element not found (might load later)

Mark these explicitly: "This is often normal during..." vs "This is an unexpected error."

It prevents the LLM from treating every error as fatal.

Pattern 2: Track, Then Decide

For operations with partial failures, I separate tracking from policy enforcement.

The Problem

// ❌ Don't decide criticality at failure site
try {
  await injectIntoFrame(frameId);
} catch (error) {
  if (frameId === mainFrameId) {
    throw error; // Critical!
  }
  // Otherwise ignore?
}

This couples failure handling with business logic. Every injection site needs to know what's critical.

The Solution: FailureLog

type FailureLog = {
  frameInjectionFailures: Array<{
    frameId: string;
    error: string;
    reason: "cross-origin" | "detached" | "timeout" | "unknown";
    isMainFrame: boolean; // Track criticality as metadata
    timestamp: number;
  }>;
  frameExpansionFailures: Array<{
    ref: string;
    error: string;
    detached: boolean;
    timestamp: number;
  }>;
  authLoadError?: {
    error: string;
    authPath: string;
    timestamp: number;
  };
};

Step 1: Operations track all failures

async injectFrameTreeRecursive(
  context: RoleContext,
  frameTree: any,
  isMainFrame: boolean = false
): Promise<void> {
  try {
    await context.bridgeInjector.ensureFrameState(
      context.cdpSession,
      frameTree.frame.id
    );
  } catch (error) {
    // Track in FailureLog (don't throw yet)
    const failures = this.ensureFailureLog(context);
    failures.frameInjectionFailures.push({
      frameId: frameTree.frame.id,
      error: error.message,
      reason: this.classifyFrameError(error),
      isMainFrame, // Metadata, not decision
      timestamp: Date.now(),
    });
    return;
  }
}

Step 2: Decision points check FailureLog

async navigate(url: string): Promise<Snapshot> {
  // ... navigation logic ...

  await this.discoverAndInjectFrames(context);

  // DECISION POINT: Check for critical failures
  const mainFrameFailed = context.failures?.frameInjectionFailures
    .some(f => f.isMainFrame);

  if (mainFrameFailed) {
    throw new Error('Main frame injection failed - page cannot be automated');
  }

  // Non-critical failures become warnings
  snapshot.warnings = this.buildWarningsFromFailureLog(context);
  return snapshot;
}

Step 3: Warnings expose non-critical failures

private buildWarningsFromFailureLog(context: RoleContext) {
  const failures = context.failures;
  if (!failures) return undefined;

  const warnings: any = {};

  // Check for inaccessible frames (non-main frames that failed)
  const inaccessibleFrames = failures.frameInjectionFailures
    .filter(f => !f.isMainFrame);

  if (inaccessibleFrames.length > 0) {
    warnings.inaccessibleFrames = inaccessibleFrames.length;
    warnings.details = inaccessibleFrames.map(f => 
      `Frame ${f.frameId}: ${f.reason}`
    );
  }

  // Check for auth failures
  if (failures.authLoadError) {
    warnings.authStatus = "unauthenticated";
    warnings.details = warnings.details || [];
    warnings.details.push(`Auth failed: ${failures.authLoadError.error}`);
  }

  return Object.keys(warnings).length > 0 ? warnings : undefined;
}

This pattern appears in the snapshot output:

{
  "text": "- button \"Submit\" [ref=e1]\n...",
  "elementCount": 15,
  "warnings": {
    "inaccessibleFrames": 2,
    "details": [
      "Frame abc123: cross-origin",
      "Frame def456: detached"
    ]
  }
}

Why This Works for LLMs

The LLM sees:

✅ Operation succeeded (got a snapshot)
✅ Partial failures are transparent (warnings)
✅ Clear reason for each failure
✅ Can proceed with main content

Without warnings, the LLM doesn't know if missing content is a problem or expected behavior.

What Changed

Before: LLMs retried the same failed operation repeatedly

LLM: browser_click("e5")
Error: Unknown element reference: e5

LLM: browser_click("e5")  
Error: Unknown element reference: e5

LLM: browser_click("e5")
Error: Unknown element reference: e5

[workflow fails]

After: LLMs autonomously recover

LLM: browser_click("e5")
Error: Unknown element reference: e5
[Error includes recovery steps mentioning browser_snapshot()]

LLM: browser_snapshot()
[Gets fresh refs, sees e7 is the submit button]

LLM: browser_click("e7")
Success!

Error recovery rate went from ~20% to ~95%. Most failures are now self-healing.

Conversation length decreased. Errors that required human intervention ("what ref should I use?") now resolve autonomously.

Workflow reliability improved. Multi-step tasks that would fail on the first error now complete end-to-end.

Adapting This to Your Stack

This pattern works for any LLM tool interface:

OpenAI Function Calling

@tool
def click_element(ref: str) -> str:
    """Click an interactive element"""
    try:
        element = get_element(ref)
        element.click()
        return "Clicked successfully"
    except UnknownRefError as e:
        return format_error_for_llm(e)

Return structured error text instead of raising exceptions.

Anthropic Claude Tools

server.tool("click_element", 
  { ref: { type: "string" } },
  async ({ ref }) => {
    try {
      await clickElement(ref);
      return { content: [{ type: "text", text: "Clicked successfully" }] };
    } catch (error) {
      return { 
        content: [{ 
          type: "text", 
          text: formatErrorForLLM(error) 
        }] 
      };
    }
  }
);

LangChain Tools

class ClickElementTool(BaseTool):
    name = "click_element"
    description = "Click an interactive element"

    def _run(self, ref: str) -> str:
        try:
            click_element(ref)
            return "Clicked successfully"
        except Exception as e:
            return format_error_for_llm(e)

REST APIs for Agents

app.post('/api/click', async (req, res) => {
  try {
    await clickElement(req.body.ref);
    res.json({ success: true, message: "Clicked successfully" });
  } catch (error) {
    // Don't use HTTP error codes - return 200 with formatted error
    res.json({ 
      success: false, 
      error: formatErrorForLLM(error) 
    });
  }
});

The key: Return errors as structured text, not as exceptions or HTTP error codes. The LLM needs to read the error, not catch it.

What I Learned

Error messages are part of your API contract. When building for LLMs, they're as important as success responses.
"Tool name" is the most important piece of information. The LLM needs to know exactly what function to call to recover. Don't say "refresh" when you mean "browser_snapshot()".
Numbered lists beat prose. LLMs follow sequential steps better than paragraph instructions.
Distinguish expected from unexpected failures. "This is normal during navigation" prevents the LLM from treating temporary failures as fatal.
Track failures, classify at decision points. Don't decide if something is critical at the failure site—track it and decide where you have business context.
Warnings enable partial success. Operations can succeed with degraded functionality if failures are transparent.
Recovery plans beat descriptions. "Element not found" tells the LLM nothing. "Call get_elements(), find the element by name, retry with the correct ref" is a recovery plan.
Rich error classes enable specific guidance. UnknownRefError can give different recovery steps than StaleRefError because they have different properties.

The Shift

Traditional error handling is designed for debugging. You want stack traces, error codes, and technical details because you'll investigate and fix the code.

LLM error handling is designed for autonomous recovery. You want context, diagnosis, and exact recovery steps because the LLM will fix the situation without code changes.

The error message isn't documentation—it's a recovery script the LLM executes.

When you're building tools for autonomous AI use, every error is an opportunity for the system to self-heal. Make your error messages teach the LLM how.

Return Facts, Not Interpretations: Why LLM Tools Should Be Dumber Than You Think

Johnny — Wed, 10 Dec 2025 22:57:30 +0000

This article is for anyone who's discovered that making tools "helpful" to LLM's actually made them less useful.

TL;DR: I tried to make my MCP Servers DOM exploration tools helpful by adding semantic interpretations to their output. The tools became brittle, task-specific, and harder for LLMs to use correctly. Stripping out all interpretation and returning only raw structural facts made the system more reliable, more composable, and paradoxically more useful. This article explains why LLM-facing tools need fundamentally different design principles than human-facing APIs—and why "dumber" tools create smarter agents.

Article Roadmap

Part 1: The Problem (2 min read)

The helpful tool that made everything worse
Why interpretation is context-dependent
The insight: capability vs. interpretation

Part 2: Why This Matters (2 min read)

The composition problem
The human API trap
When interpretation belongs in tools (the rare cases)

Part 3: Design Principles (2 min read)

Four principles for fact-based tools
The composability test
How the knowledge layer handles interpretation

Part 4: What This Means for Builders (2 min read)

Practical changes for tool builders
The counterintuitive result
The principle generalizes

Part 1: The Problem

1.1 The Helpful Tool That Made Everything Worse

When I first built resolve_container for Verdex, I wanted it to be helpful. The tool walks up the DOM tree from a target element and returns the ancestor chain. But I didn't just return tags and attributes—I added interpretation:

{
  "type": "product-card",           // Tool guesses semantic meaning
  "role": "list-item",              // Tool guesses structural purpose
  "confidence": 0.85,               // Tool evaluates its own guess
  "recommendation": "Use this as your container scope"
}

This seemed reasonable. I was helping the LLM by pre-analyzing the structure and making recommendations. The tool was doing the hard work of interpretation so the LLM didn't have to.

Then I hit production.

A page with user profile cards. The tool confidently labeled them as product cards (confidence: 0.85). The LLM trusted the tool's interpretation. It generated selectors scoped to the wrong pattern. Tests broke mysteriously when the heuristic failed on edge cases.

The problem wasn't that the interpretation was usually wrong—it was usually right. The problem was fundamental: interpretation is context-dependent, and the tool didn't have context.

1.2 The Core Problem: Interpretation Is Context-Dependent

Take a div with data-testid="product-card". What does this mean?

For selector authoring: That div is a stable container. Use getByTestId("product-card") as your scope.

For visual testing: It's a component boundary. Screenshot this entire card.

For web scraping: It's a data structure. Extract product information from descendants.

For accessibility auditing: It's a semantic grouping. Check for proper ARIA labels.

Same DOM element. Same structural facts. Four completely different interpretations.

When my tool chose one interpretation—"this is a product card, use it for selector scoping"—it made that decision for all tasks. But the tool couldn't know which interpretation was correct because it lacked the user's query, the application domain knowledge, and the task context.

Only the LLM had that information.

1.3 The Insight: Capability vs. Interpretation

This led to a fundamental separation in how I think about tool design:

Tools provide capability: Access to structural facts that would otherwise be hidden or expensive to retrieve.

LLMs provide interpretation: Deciding what those facts mean for this specific query in this specific context.

The tool's job is to traverse the DOM and return what it finds: tags, attributes, depth, relationships. Not to guess whether something is a product card. Not to evaluate whether it's a good container. Not to recommend what the LLM should do with it.

That's the LLM's job.

Here's what happened when I stripped out the interpretation:

Before (interpretation mixed in):

{
  "container": {
    "semanticType": "product-card",  // Tool is guessing
    "stability": "high",              // Tool is evaluating  
    "recommended": true               // Tool is prescribing
  }
}

After (pure facts):

{
  "ancestors": [
    {
      "level": 1,
      "tagName": "div",
      "attributes": {"data-testid": "product-card"},
      "childElements": 5
    }
  ]
}

The second version seems less helpful. It is less helpful—for humans. But it's more useful for LLMs because it preserves optionality.

The LLM can interpret those facts differently based on what the user actually asked for:

Selector authoring query: "That data-testid at level 1 is a stable container, I'll use it for scoping"
Debugging query: "There are 12 elements with that testid, probably a component copied without updating IDs"
Refactoring query: "This pattern appears in 47 test files, needs careful migration"

Same tool output. Different interpretations. The architecture works because the capability layer stayed interpretation-free.

Part 2: Why This Matters

2.1 The Composition Problem

When tools return interpretations, they're making decisions that are hard to reverse.

The LLM has to either accept the interpretation or fight against it. And fighting against confident tool output is something LLMs struggle with—they tend to defer to explicit assertions in their context.

Here's a concrete example:

The tool says "type": "product-card" with "confidence": 0.85.

Now the user asks: "Find all user profile cards on this page."

The LLM sees the tool's interpretation. It has two options:

Trust it (wrong): Generate selectors for product cards when the user wants profiles
Fight it (awkward, token-expensive, unreliable): Explain why the tool's interpretation doesn't match the query

Compare this to raw facts. The tool returns "data-testid": "product-card".

When the user asks about user profiles, the LLM looks at the actual page structure and realizes this testid is misleading or the wrong target. The LLM can adapt because no interpretation was baked in.

The principle: Tools that return facts compose across different tasks. Tools that return interpretations optimize for one task and break others.

2.2 The Human API Trap

This is deeply counterintuitive because we've spent decades optimizing APIs for human developers.

Human-facing APIs should be high-level and opinionated. A method like page.selectDropdown("Country", "United States") is beautiful for humans because it handles all the fiddly details—finding the select element, opening it, locating the option, handling JavaScript events.

But LLMs work differently.

The LLM actually does better with raw primitives:

page.click('select[name="country"]')
page.click('option:has-text("United States")')

The LLM can adapt this pattern to dropdowns it's never seen before—custom components, weird frameworks, non-standard implementations. The high-level abstraction only works for exactly the cases it was designed to handle. The low-level primitives work for cases nobody anticipated.

Here's why: LLMs excel at pattern matching and adaptation. Give them low-level building blocks and they'll figure out how to compose them for novel situations. Give them high-level abstractions and they'll try to force-fit them into scenarios where they don't quite work.

This is why resolve_container returns ancestor chains with raw attributes instead of "here's your recommended container." The LLM can look at those ancestors and decide which level makes sense for this specific selector on this specific page for this specific task.

2.3 When Interpretation Belongs in Tools (The Rare Cases)

I need to acknowledge that there are legitimate cases where tools should interpret.

Deterministic operations where there's one correct answer:

Parsing structured data formats: A tool that parses JSON should return the parsed structure, not raw bytes
Mathematical computations: A tool that calculates compound interest should return the final number, not intermediate steps
Syntax validation: A tool that checks selector syntax can return true/false

Domain-specific validation where correctness is well-defined:

Email validation: "valid email" has a specification
Schema compliance: Checking if data matches a defined structure
Type checking: Verifying if a value meets type constraints

The key distinction: these interpretations are task-independent.

Parsing JSON means the same thing regardless of what you're building. Validating an email address has the same definition across all contexts. Mathematical operations produce identical results regardless of the user's goal.

Contrast with Verdex's primitives:

Whether a div is a "good container" depends entirely on context:

What you're selecting
Why you need stability
What alternatives exist
How the rest of the page is structured
Whether you're authoring, debugging, or refactoring

That interpretation must stay with the LLM because only the LLM has the user's query, the task context, and the surrounding conversation.

Part 3: Design Principles

3.1 Four Principles for Fact-Based Tools

Return the Minimum Sufficient Facts

Don't return everything about an element—that's information overload. Return just enough for the LLM to make the next decision.

For ancestors, that means:

Level (depth in tree)
Tag name
Attributes
Child count

Not:

All text content
All descendants
Computed styles
Event listeners

Make Facts Actionable

Each piece of information should enable a decision. If you're returning a fact that the LLM can't use, you're creating noise.

The childElements count in resolve_container is actionable—it helps the LLM decide if this level makes sense as a container. A container with 1 child is probably too specific. A container with 50 children might be too broad.

Preserve Context for Interpretation

Include facts that affect interpretation even if they seem like metadata.

The level number in ancestor chains matters because "container at level 1" vs "container at level 5" suggests different stability characteristics. Level 1 is close to the target (specific, might break if structure changes). Level 5 is far from the target (stable, might be too broad).

Use Consistent, Predictable Formats

The LLM should never have to guess whether an attribute value will be a string, array, or object.

Bad:

// Sometimes attributes is an object
{"attributes": {"class": "card"}}
// Sometimes it's an array  
{"attributes": [["class", "card"]]}

Good:

// Always an object
{"attributes": {"class": "card"}}

3.2 The Composability Test

Here's a practical test for whether your tool design is fact-based or interpretation-heavy:

Take your tool's output and ask: Can I use these facts for a task the tool wasn't designed for?

Let's walk through examples with Verdex. resolve_container returns ancestor chains with attributes. Original task: authoring selectors.

Novel tasks the same output supports:

Visual testing: Screenshot elements at specific container depths

Debugging: Understand why a selector stopped working after refactor

Component analysis: Identify repeated patterns across the codebase

Accessibility audit: Find containers missing proper ARIA structure

None of these tasks were in mind when designing resolve_container. But the tool's output supports all of them because it returns structural facts, not selector-authoring interpretations.

Compare to the interpreted version that returned "semanticType": "product-card". That interpretation optimizes for selector authoring but provides zero value for:

Visual testing (what's a "product card" for screenshots?)
Accessibility audits (semantic type doesn't indicate ARIA compliance)
Component analysis (can't identify patterns without structural details)

The interpreted output is less composable because it baked in assumptions about the task.

3.3 The Knowledge Layer Handles Interpretation

This connects back to the two-layer architecture I wrote about in "Teaching LLMs to Compose."

Layer 1 (tools) provides raw structural facts—tags, attributes, relationships, positions.

Layer 2 (Skills) teaches interpretation—when this pattern appears, consider it a stable container. When these attributes exist, prioritize them for scoping.

The separation enables different interpretations for different tasks:

Verdex Skill for selector authoring teaches:

"When you see data-testid at level 1, that's your best container. Scope your selector there."

Hypothetical Skill for accessibility auditing teaches:

"When you see a div at level 1 without an ARIA role, that's a potential accessibility issue. Check if it needs role='region' or similar."

Same tool output. Different Skills. Different interpretations.

The architecture works because the capability layer stayed interpretation-free, allowing the knowledge layer to provide task-specific guidance.

Part 4: What This Means for Builders

4.1 Practical Changes for Tool Builders

If you're building LLM tools right now, here's what changes:

Stop Trying to Make Tools "Smarter"

Every time you add interpretation to tool output, you're reducing composability. Resist the urge to be helpful through interpretation. Be helpful through better facts.

Early Verdex tried to help by identifying "product cards" and "stable containers." This helped for exactly one task (selector authoring) and hurt everything else. Removing the helpfulness made the tool universally useful.

Test Tools on Unintended Tasks

If your tool output only makes sense for one specific workflow, you've probably baked in too much interpretation.

Build a file system exploration tool? Test it on:

Code navigation
Security auditing
Dependency analysis
Documentation generation

If the same output works across all four, you've got good primitives.

Move Strategic Knowledge to Skills

Don't try to teach best practices through tool behavior. Teach them through explicit, progressive knowledge layers.

The pattern that works: dumb tools that return facts, smart Skills that teach composition.

Trust the LLM to Interpret

This feels wrong at first. You know your domain better than the model. You've spent months understanding what makes a good container, what patterns are stable, what heuristics work.

But the model has something you don't: the user's actual query and the full task context.

When the user asks "find the button in the iPhone card," the LLM knows:

They're looking for a button (not analyzing structure)
It's in a specific card (needs unique identification)
The card contains iPhone (text-based filtering)

Your tool can't know this because the tool runs before the LLM decides what to do. Give the LLM the facts. Let it compose them contextually.

4.2 The Counterintuitive Result

Making tools "dumber"—stripping out helpful interpretations, returning raw facts—makes LLM agents smarter.

The agents can adapt to novel scenarios, compose across tasks, and learn from failures because the tools didn't lock them into specific interpretations.

This is the opposite of how we've built developer tools for decades:

Human-facing APIs should abstract away details and provide high-level operations. Humans benefit from selectDropdown(field, value) because we don't want to think about DOM traversal and event handling.

LLM-facing tools should expose details and provide low-level primitives. LLMs benefit from raw click() and getText() because they can compose these primitives in ways we never anticipated.

4.3 The Principle Generalizes

This philosophy extends beyond Verdex and DOM exploration.

Whether you're building tools for:

File systems (return paths, sizes, types—not "important files" or "project roots")
Databases (return schemas, relationships, types—not "main tables" or "key entities")
APIs (return endpoints, parameters, types—not "recommended flows" or "common patterns")
Code analysis (return AST nodes, references, scopes—not "bad code" or "refactoring candidates")

The lesson holds: return facts, let Skills teach interpretation, and trust the model to compose.

Your tools should be dumb enough to be universally useful. The intelligence lives in how the LLM composes them—guided by Skills that understand context, informed by the user's query, and adapted to the specific task at hand.

Closing Thoughts

I spent weeks making Verdex's tools "smarter" before realizing I was solving the wrong problem. The tools didn't need to be smarter—they needed to be simpler.

Every interpretation I removed made the system more reliable. Every heuristic I deleted made the tools more composable. Every "helpful" feature I stripped out made the LLM better at its job.

The counterintuitive truth: dumb tools enable smart agents.

Why Verdex Uses CDP Directly

Johnny — Wed, 10 Dec 2025 22:52:46 +0000

This is probably the most common question I get from engineers about Verdex: "Why use CDP instead of building on top of, or simply extending Playwright?" It's a completely fair question—Playwright is exceptional, widely adopted, and has CDP access through newCDPSession(). So let me walk through the technical reasoning.

The short answer: Verdex is a development-time authoring tool that needs deep, specific control over DOM inspection and JavaScript execution contexts. Playwright is an execution-time test runner optimized for cross-browser reliability. These are fundamentally different use cases with different architectural requirements—and understanding why reveals something interesting about when abstractions help vs. when they create friction.

A Note on Inspiration

Before diving in, I want to be clear: Verdex owes a significant debt to Playwright's design. The accessibility tree implementation, the approach to isolated worlds, the careful attention to element lifecycle—these are all areas where I studied Playwright's codebase extensively and drew inspiration.

Verdex aims for parity with Playwright's level of sophistication, particularly around W3C ARIA-compliant accessibility tree generation, robust handling of frame lifecycles and navigation, and isolated execution contexts. Where Verdex diverges isn't in capability but in architecture: it adds structural exploration primitives (resolve_container, inspect_pattern, extract_anchors) for authoring-time selector construction rather than execution-time test reliability.

The question isn't "what can Verdex do that Playwright can't?" It's "given these different goals, what foundation makes the most sense?"

Execution-Time vs Authoring-Time: Different Problems, Different Solutions

Here's the core distinction:

Execution-Time (Playwright)	Authoring-Time (Verdex)
Re-resolve selectors on every action	Maintain stable references across multiple queries
Prevent stale element bugs	Enable multi-step DOM exploration
Cross-browser uniformity	Analysis depth
Ephemeral element handles	Persistent element mappings during sessions

Playwright's Locator Philosophy

Playwright's Locators re-resolve on every action—a brilliant defense against stale element bugs that plagued Selenium. From their documentation:

"Every time a locator is used for an action, an up-to-date DOM element is located in the page."

For test execution, this is exactly right.

For authoring-time analysis, where I need to call resolve_container(e3), then inspect_pattern(e3), then extract_anchors(e3) on the same element, this creates friction. Here's what Verdex needs:

// Sequential queries on the same element during authoring
resolve_container("e3")   // Walk up from this specific element
inspect_pattern("e3", 2)  // Examine the SAME element's siblings  
extract_anchors("e3")     // Explore the SAME element's children

With Locators, you're re-querying the DOM each time—potentially getting different elements if the page changed. With ElementHandles (Playwright's persistent option), the framework actively discourages their use and auto-disposes them after actions to enforce re-resolution. This is core philosophy, not a technical omission.

Different problems, different solutions.

"But Wait—What About Stale Elements?"

If you're familiar with Selenium's notorious StaleElementReferenceException, you might be wondering: doesn't Verdex's persistent reference approach risk the same problems?

No—and understanding why reveals an important architectural insight.

Selenium's problem occurred at runtime:

element = driver.find_element(By.ID, "submit")
# ... page re-renders during test execution ...
element.click()  # ❌ Stale!

Verdex's persistent references exist only during authoring:

// Authoring session (stable snapshot)
resolve_container("e3")    // Walk up from element
inspect_pattern("e3", 2)   // Check siblings  
extract_anchors("e3", 1)   // Find unique content

// Output: Pure Playwright code (auto-resolving)
getByTestId("product-card")
  .filter({ hasText: "iPhone 15 Pro" })
  .getByRole("button", { name: "Add to Cart" })

The key differences:

Different lifecycle phase: Verdex refs exist during static analysis of a snapshot, not during dynamic test execution
No page interactions: You're exploring structure, not triggering re-renders
Safe output: The final code uses Playwright's auto-resolving Locators

Verdex gets the best of both worlds: persistent references enable multi-step structural exploration during authoring, while the generated test code uses Playwright's battle-tested auto-resolution during execution.

The CDP Layer: Where Architecture Choices Matter

Both Playwright and Verdex build on CDP. The difference is in how they use it.

Creating an isolated world looks nearly identical:

// Playwright with CDP
const client = await page.context().newCDPSession(page);
await client.send('Page.createIsolatedWorld', {
  frameId: mainFrameId,
  worldName: 'verdex-bridge',
  grantUniveralAccess: true
});

// Puppeteer with CDP  
const client = await page.createCDPSession();
const { executionContextId } = await client.send('Page.createIsolatedWorld', {
  frameId: mainFrameId,
  worldName: 'verdex-bridge',
  grantUniveralAccess: true
});

The complexity emerges when working with elements across multiple operations.

The Impedance Mismatch

Verdex maintains a persistent Map<string, ElementInfo> that tracks DOM nodes across analysis operations. With Playwright + CDP, you're bridging two object models:

Playwright's auto-managed ElementHandles exist in its utility world
You need manually-managed CDP objectIds in your isolated world
Converting between them requires extra evaluation calls and context switching

You'd end up using Playwright's CDP access to bypass Playwright's abstractions entirely, while carrying the overhead of those abstractions in your bundle.

Why Puppeteer Fits Naturally

With Puppeteer, everything operates at the same abstraction level—CDP primitives throughout:

const session = await page.target().createCDPSession();

// Create your isolated world
const { executionContextId } = await session.send('Page.createIsolatedWorld', {...});

// Inject your bridge code
await session.send('Runtime.evaluate', {
  expression: bridgeCode,
  contextId: executionContextId
});

// Use objectId for stable, multi-step operations
const { result } = await session.send('Runtime.evaluate', {
  expression: 'document.querySelector("[data-testid=product]")',
  contextId: executionContextId
});

// Same objectId, multiple operations
const analysis = await session.send('Runtime.callFunctionOn', {
  objectId: result.objectId,  // Stable until you release it
  functionDeclaration: 'function() { return window.verdexBridge.fullAnalysis(this); }',
  executionContextId,
  returnByValue: true
});

// Later: structural exploration using the same reference
const { result: ancestors } = await session.send('Runtime.callFunctionOn', {
  functionDeclaration: 'function() { return window.verdexBridge.resolve_container(this); }',
  objectId: result.objectId,  // Same stable reference
  executionContextId,
  returnByValue: true
});

No impedance mismatch, no fighting auto-disposal, no bridging between object models. The puppeteer package is ~2MB and CDP-native, while playwright-core is ~10MB with cross-browser abstractions that Verdex never uses.

Isolated Worlds: Different Purposes, Different Implementations

Both Playwright and Verdex use isolated execution contexts, but for different reasons.

Playwright's approach: Test scripts run in the main world by default (selector engines use utility worlds). This lets tests interact with the page's actual JavaScript context—including any framework quirks or monkey-patched APIs. If your application overrides Element.prototype.getAttribute(), Playwright sees that. This is correct: tests should validate user-facing behavior, warts and all.

Verdex's approach: Analysis code runs entirely in an isolated world. When Verdex traverses the DOM to discover that <div data-testid="product-card"> exists, it uses native, unmodified DOM APIs. No monkey-patches, no framework hooks, no interference.

This separation strengthens both tools: Verdex discovers clean structural facts during authoring (data-testid="product-card", text content, hierarchy). Playwright executes the generated selector during testing with its battle-tested engine that handles any monkey-patching robustly.

Each tool operates in the execution context that best serves its purpose.

Multi-Role Browser Contexts

Here's where direct CDP access proved especially valuable: multi-role browser contexts.

I was working on a marketplace application with three distinct roles: admin, provider, and customer. The CTO wanted tests that maintained flows across roles—just one example: the provider adds a product with a discount, the customer loads the product, the provider changes details, those changes reflect in the customer session, then the customer checks out.

The Playwright MCP Approach:

Playwright MCP's architecture creates an interesting challenge for multi-role testing. Since each MCP server instance manages a single browser context, you need to run multiple server instances—one per role—and orchestrate them in your conversation:

You: "Using playwright-provider, add iPhone 15 Pro at $999"
[Response includes generated code for provider actions]

You: "Using playwright-customer, navigate to products and find iPhone"
[Response includes generated code for customer actions]

You: "Using playwright-provider, change price to $899"
[Response includes generated code for provider update]

You: "Using playwright-customer, refresh and verify new price"
[Response includes generated code for customer verification]

The orchestration overhead is substantial:

Server context switching - Explicitly specify which server (role) before every action
Code stream assembly - Collect and stitch code snippets from 3+ separate response streams
State coordination - Manually track which role has what data across parallel sessions
Conversation fragmentation - LLM managing multiple concurrent browser states simultaneously

For a 50-action multi-role test with 7 role switches, you're effectively conducting two parallel conversations with separate automation sessions, then manually assembling the fragments into a coherent test file.

The Verdex Approach:

With CDP-level control, Verdex manages multiple browser contexts within a single server instance. Each role gets its own incognito context with true browser-level isolation—separate CDP sessions, independent authentication, isolated storage:

select_role("provider");
browser_navigate("/provider/products/new");
browser_click("e1");  // Add product form

select_role("customer");
browser_navigate("/products");
browser_snapshot();  // Customer sees new product immediately

select_role("provider");
browser_navigate("/provider/products/edit");
browser_click("e7");  // Update pricing

select_role("customer");
browser_navigate("/products");
browser_snapshot();  // Changes reflected instantly

The architectural difference:

Single conversation thread, natural narrative flow, browser-level isolation handled at the protocol level. All actions logged in one unified session trace. No server process juggling, no code stream assembly, no manual state synchronization.

The key is that select_role() switches the active CDP session within the same MCP server—not switching between separate server instances. Each role maintains:

Independent authentication state (separate auth files)
Isolated storage (cookies, localStorage, sessionStorage)
Dedicated CDP session for commands
Separate accessibility snapshots and element maps

Multi-role e2e tests went from tedious orchestration across multiple MCP instances to straightforward linear authoring. Tests that would take an hour of careful coordination took minutes, and they were more reliable because the isolation was handled at the browser protocol level rather than through multi-server configuration management.

Progressive Disclosure and Token Efficiency

Verdex's progressive disclosure design—returning targeted results per query instead of dumping entire DOM trees—depends on complete control over what gets serialized across the protocol boundary.

With direct CDP access, I control the entire serialization pipeline:

// Bridge runs entirely in the isolated world
window.verdexBridge = {
  resolve_container(ref) {
    // Synchronous DOM traversal
    // Custom filtering for stable containers
    // Minimal JSON output
    return { target, ancestors };
  }
};

The bridge performs multi-step DOM analysis in-browser and returns only the structural facts needed for selector construction. This keeps computation close to the data and minimizes serialization costs.

The Cross-Browser Constraint Doesn't Apply

Playwright's entire value proposition is cross-browser parity. Every feature needs to work consistently across Chromium, Firefox, and WebKit. This is the right design for a test runner.

Verdex is explicitly a development-time tool. You author tests with Verdex during development, then execute those tests with Playwright across all browsers in CI:

The output is standard Playwright code that runs anywhere

What About Playwright MCP?

Playwright MCP serves a fundamentally different purpose. Understanding this distinction requires looking at how each tool actually works with the browser.

The Critical Difference: Runtime Execution vs Authoring Analysis

Playwright MCP: A runtime execution tool that gives AI agents direct control over live browser sessions for immediate tasks—web scraping, form filling, test execution, automation. The agent issues commands and Playwright MCP executes them in real-time.

Verdex: An authoring-time analysis tool. It doesn't execute tests or care how semantic your html is—it is solely focused on helping you write better test code, faster.

When you use Verdex, you first navigate using Verdex's primitives:

browser_navigate("https://example.com")  // Verdex navigation
browser_snapshot()                       // Get accessibility tree with refs
resolve_container("e3")                  // Explore DOM structure
inspect_pattern("e3", 2)                 // Examine repeating patterns

Those ref values (like e3) come from Verdex's accessibility snapshot generator, which creates stable references to DOM nodes and maintains a Map<string, ElementInfo>. This mapping enables structural exploration—resolve_container(ref) can walk up the DOM tree because it has direct access to the underlying element.

As of current public documentation, Playwright MCP does not offer primitives to traverse DOM structure. The maintainers closed a request for exposing the raw DOM tree as "not planned." The architecture is fundamentally different—Playwright MCP's refs are for immediate interaction, not structural analysis.

Why Accessibility Snapshots Work for Different Purposes

For Playwright MCP (runtime execution):

button "Add to Cart" [ref=e3]

That's all the agent needs to click the button or extract data.

For Verdex (test authoring):

// Start with the same accessibility snapshot
button "Add to Cart" [ref=e3]

// But then explore structure to write durable selectors
resolve_container("e3")       // Discovers the product-card container
inspect_pattern("e3", 2)     // Confirms it's one of many cards
extract_anchors("e3", 1)  // Finds unique anchors inside the card

// Result: Container-scoped selector that survives layout changes
getByTestId("product-card")
  .filter({ hasText: "iPhone 15 Pro" })
  .getByRole("button", { name: "Add to Cart" })

Element Lifecycle: Two Philosophies

Playwright MCP (Ephemeral References):

Snapshot 1 → ref=e3 → Element A → Click happens
Snapshot 2 → ref=e3 → Element B (possibly different!)

Refs regenerate each time, preventing stale element bugs but making structural analysis impractical.

Verdex (Persistent References During Authoring):

Snapshot → ref=e3 → CDP objectId → Element A
  ↓
resolve_container(e3) → Same objectId → Walk parent chain
  ↓
inspect_pattern(e3, 2) → Same objectId → Analyze siblings
  ↓
extract_anchors(e3, 1) → Same objectId → Explore children

Refs remain stable throughout the authoring session. The final test code uses Playwright's standard Locators.

Why Issue #103 Was Closed

Issue #103 in the Playwright MCP repository requested DOM visibility features, but it was closed as "not planned" after the maintainer tested full HTML output on real sites.

From the issue:

"I played around with this, but in my tests on github.com and microsoft.com, the HTML was way too much text for the LLM to make sense of it. In fact, it repeatedly requested the HTML again, I assume because it couldn't comprehend it." — @skn0tt

This is the exact information overload problem I encountered building Verdex. Full DOM dumps on complex pages produce 50k+ tokens that degrade LLM accuracy—the model hallucinates elements, loses context mid-response, and can't distinguish signal from noise.

The architectural constraint: While full HTML dumps don't work, progressive structural exploration isn't compatible with Playwright MCP's current design either. The accessibility snapshot approach—which generates ephemeral refs tied to specific snapshot moments—creates fundamental barriers:

No stable references across queries: Each snapshot generates new refs. If you call browser_snapshot() and get ref=e3, then the page updates and you call browser_snapshot() again, that same button might now be ref=e7. You can't call resolve_container(e3) and then inspect_pattern(e3) on the same element because e3 only exists within the context of one snapshot.
No persistent ref-to-DOM mapping: The accessibility snapshot provides clean semantic views but doesn't maintain the underlying mapping from refs to DOM nodes needed for structural traversal. To answer "what are the ancestors of e3?", you'd need to track which DOM node e3 represents—but refs are intentionally disposable to prevent staleness.
Cross-browser constraints: Adding browser-specific CDP features like persistent objectId tracking or isolated world DOM traversal would undermine Playwright's core value proposition of cross-browser uniformity.

Adding structural exploration to Playwright MCP would require fundamental architectural changes:

Persistent ref lifecycle: Refs would need to survive across queries and map to live DOM nodes, contradicting the ephemeral, snapshot-based model designed to prevent staleness
Specific CDP features: Structural primitives like ancestor traversal and sibling inspection require CDP objectId tracking in isolated execution contexts—features that don't exist in Firefox or WebKit
Different abstraction layer: The accessibility-first view intentionally hides structural details (like data-testid attributes and container hierarchies) that test authors need for writing stable selectors

The Complementary Relationship

This creates a natural division of labor:

Use Playwright MCP for:

Real-time browser control and automation
Web scraping and data extraction
Form filling and interaction
Exploratory testing and debugging

Use Verdex for:

Authoring complex multi-role e2e test flows
Creating durable, refactor-resistant selectors
Understanding DOM structure and container relationships
Managing multiple authenticated sessions with proper isolation

Execute with Playwright's test runner for:

Cross-browser reliability in CI
Parallel test execution
Comprehensive reporting and debugging tools

The tools solve different problems at different stages of the testing workflow.

The Bottom Line: Foundation Matches Purpose

This isn't about capability—Playwright's newCDPSession() provides the same CDP access that Verdex uses. It's about directness and architectural fit.

Playwright abstracts away browser protocols to provide cross-browser uniformity—write once, test everywhere. This abstraction layer is Playwright's superpower for test execution.

Verdex builds on browser protocols to provide specific authoring intelligence—deep structural exploration that helps you write better selectors during development. Working at the CDP layer directly means no impedance mismatches, no fighting auto-disposal semantics, no bridging between execution contexts.

The 2MB footprint, CDP-native APIs, and element lifecycle semantics all align with authoring-time exploration rather than test-time reliability.

You use Verdex during development to write complex e2e tests with better selectors, then execute those tests with Playwright in CI. The refs, structural analysis, and multi-role contexts exist only during authoring—your final test code is pure Playwright, running anywhere Playwright runs.

Different stages of the workflow, different architectural foundations, designed to complement each other.

Building LLM Tools That Don't Drown in Context: Lazy Traversal for Unknown Structures

Johnny — Wed, 03 Dec 2025 11:18:46 +0000

This article is for anyone who's hit the context window problem with structured data.

The Problem: Dumping entire DOMs into LLM context costs 50k+ tokens and causes models to hallucinate elements that don't exist.

The Solution: Lazy traversal with bounded queries reduced token costs from 50k+ to <3k while improving selector quality.

While building an open source MCP server for AI-native test authoring, I discovered that lazy traversal solved the context window problem for DOM exploration. Instead of dumping entire DOMs into context—which consumed excessive tokens and caused LLMs to miss information buried in the middle—I built tools that reveal structure incrementally through bounded queries.

A note on scope: This pattern works for trees—structures where each node has exactly one parent. DOMs, file systems, ASTs, and org charts are trees. For DAGs (where nodes can have multiple parents), you'd need traversal rules to impose tree-like structure. For dense graphs, this pattern doesn't apply—you'd need sampling, summarization, or indexing instead.

This article documents what worked for DOM, why I think it worked, and the implementation principles I learned. I've only built this for DOM exploration, but if you're working with hierarchical structures—file systems, ASTs, database schemas, organizational hierarchies—this case study might help you think through your own approach.

Article Roadmap

Part 1: The Problem (2 min read)

Why dumping full structures fails
The attention problem in long contexts

Part 2: The Solution Pattern (5 min read)

Four-level lazy traversal workflow
Complete DOM example with code

Part 3: Why It Works (5 min read)

Connection to established CS patterns
Why bounded queries preserve reasoning quality
Structural characteristics that enable this pattern
Design principles for building primitives

Part 4: When to Use This (3 min read)

Decision framework
Alternative approaches

Part 1: The Problem

The Context Window Paradox

You're building an LLM tool that needs to understand structured data. Maybe it's exploring a file system, navigating a codebase's AST, or understanding a database schema. The obvious approach: serialize everything and hand it to the model.

Token cost is the obvious problem. Complex structures blow through your budget fast—a production web page's DOM serializes to 50k+ tokens. But that's not the real issue.

Recent research shows LLM performance degrades significantly when relevant information is positioned in the middle of long contexts. The "Lost in the Middle" paper demonstrated a U-shaped performance curve: highest performance at the beginning or end of input, with information in the middle getting overlooked even when explicitly present—and this holds even for models explicitly designed for long contexts.

When I was building Verdex, I saw this constantly with full DOM dumps. Strong models would hallucinate elements that didn't exist. They'd generate selectors referencing completely unrelated page sections. The model would anchor on early DOM sections and miss better containers that appeared later.

I needed a different approach: don't dump everything at once. Let the LLM traverse the structure lazily, requesting only what it needs for the current decision through bounded queries.

Part 2: The Solution Pattern

The Core Idea: LLM-Guided Lazy Traversal

Don't give the model everything at once. Give it what it needs for the current decision, then let it request more through bounded queries. The LLM decides which branches to expand based on task context.

For DOM exploration, this meant: instead of dumping the full HTML, start with a compact semantic overview, then let the LLM request structural details through targeted queries. "What contains this button?" returns an ancestor chain with IDs and classes. "Does this pattern repeat?" returns sibling elements at this depth. "What's unique about this instance?" returns text content and attributes within the container.

This worked surprisingly well for DOM exploration. Token costs dropped by 90%+ compared to full dumps. Selector quality improved. Models stopped hallucinating elements. The pattern was reliable across diverse page structures.

The Workflow: Container → Pattern → Anchor

The workflow has four levels. I discovered this through iteration with DOM exploration:

Level 1 is your semantic overview. This shows meaningful entities without implementation details. For DOM, it's the accessibility tree with roles and names.

Level 2 is container resolution. When the LLM identifies something interesting from the overview, it asks: what contains this element? This query returns the containment hierarchy and stable anchors—the attributes that make containers identifiable.

Level 3 is pattern recognition. Once you know the container, you ask: does this repeat? Are there siblings at this depth with similar structure? This reveals lists, grids, repeated components.

Level 4 is anchor extraction. Now you need to distinguish one instance from another. This is a deeper scan within the chosen container for unique identifiers.

The workflow works because each step builds on previous understanding. Token cost is bounded per step. The LLM decides what to explore next based on findings. Exploration terminates when sufficient context is gathered. And iteration is practical because each query is cheap.

A Concrete Example: DOM Exploration with Code

Here's how this actually worked in practice.

Scenario: Generate a Playwright selector for the "Add to Cart" button specifically for iPhone 15 Pro in a product grid. Challenge is there are multiple "Add to Cart" buttons on the page.

Full code example: [Link to Verdex examples on GitHub]

Before: Full DOM Dump Approach

// ❌ Full DOM dump - 127k tokens
const fullDOM = await page.content();

// Common results with full dumps:
// - Fragile positional selector: .product-card:nth-child(3) button
// - Hallucinated attributes that don't exist
// - Anchoring on wrong section of page
// Token cost: ~127,000
// Quality: Breaks when cards reorder, misses data-testid attributes

After: Lazy Traversal Approach

// ✅ Lazy traversal - 2.9k tokens

// Step 1: Semantic overview (800 tokens)
await browser_snapshot();

Returns:

navigation "Main menu"
heading "Featured Products" [level=1]
generic
  heading "MacBook Air" [level=3]
  button "Add to Cart" [ref=e1]
generic
  heading "iPhone 15 Pro" [level=3]
  button "Add to Cart" [ref=e3]

LLM's reasoning: "Multiple 'Add to Cart' buttons exist. Need to understand what differentiates them. Let me check what contains the target button [ref=e3]."

// Step 2: Container resolution (600 tokens)
await resolve_container({ ref: 'e3' });

Returns:

Ancestry analysis for element e3:

🎯 Target Element:
   Tag: button
   Text: "Add to Cart"

📋 Ancestors (3 levels up):

Level 1 (div):
   Children: 3
   Attributes: {"data-testid":"product-card"}
   Contains refs: e3, e4, e5

Level 2 (div):
   Children: 12
   Attributes: {"data-testid":"product-grid"}
   Contains refs: e1, e2, e3, e4, e6, e7...

Level 3 (section):
   Children: 2
   Attributes: {"class":"products"}
   Contains refs: none

LLM's reasoning: "The button lives inside nested divs with identifiable test IDs. But I don't know if this is one card or many. Are there other product cards? Need to check siblings at the grid level."

// Step 3: Pattern recognition (800 tokens)
await inspect_pattern({ ref: 'e3', ancestorLevel: 2 });

Returns:

Sibling analysis for element e3 at ancestor level 2:

👥 Found 12 siblings at ancestor level 2:

Sibling 1 (div):
   Attributes: {"data-testid":"product-card"}
   Contains refs: e1, e2
   Contains text: MacBook Air, $1,299, Add to Cart

Sibling 2 (div):
   Attributes: {"data-testid":"product-card"}
   Contains refs: e3, e4
   Contains text: iPhone 15 Pro, $999, Add to Cart

Sibling 3 (div):
   Attributes: {"data-testid":"product-card"}
   Contains refs: e6, e7
   Contains text: iPad Pro, $799, Add to Cart

... (9 more siblings)

LLM's reasoning: "Pattern confirmed: 12 product cards with identical structure. Each has a heading plus button. Now I need a unique identifier for the iPhone card."

// Step 4: Anchor extraction (700 tokens)
await extract_anchors({ ref: 'e3', ancestorLevel: 1 });

Returns:

Descendant analysis for element e3 within ancestor level 1:

🏗️ Analyzing within ancestor: div {"data-testid":"product-card"}

🔍 Found 3 direct children within ancestor:

Child 1 (depth 1):
   Attributes: {}
   h3 "iPhone 15 Pro" (2 children)
      span "iPhone 15 Pro"

Child 2 (depth 1):
   Attributes: {"class":"price"}
   span "$999"

Child 3 (depth 1):
   Ref: e3 (button) "Add to Cart"
   Direct Text: "Add to Cart"

LLM's reasoning: "Perfect. I have everything needed: a stable container (data-testid='product-card'), a unique identifier (heading text 'iPhone 15 Pro'), and the target action (button with role 'Add to Cart')."

Total tokens used: 2,900 across four queries (98% reduction from 127k)

LLM generates:

const selector = page
  .getByTestId("product-card")
  .filter({ hasText: "iPhone 15 Pro" })
  .getByRole("button", { name: "Add to Cart" });

Note: The LLM generates Playwright code based on the textual analysis. While the MCP tools return formatted text, the LLM interprets this structure to produce the appropriate Playwright selector syntax. The exploration happens through MCP tool calls in the coding agent, but the output is standard Playwright code.

Why this selector is better:

✅ Container-scoped (uses data-testid for stability)
✅ Uniquely identified (filters by product name)
✅ Role-first for accessibility
✅ Resilient to structural changes (survives card reordering)
✅ No positional dependencies (no :nth-child)
✅ No hallucinated attributes

Key Differences Summary

Approach	Tokens	Quality Issues	Selector Type
Full DOM dump	~127,000	Hallucinations, missed attributes, anchoring bias	Positional: `.product-card:nth-child(3) button`
Lazy traversal	~2,900	None	Semantic: `getByTestId("product-card").filter({hasText: "iPhone 15 Pro"}).getByRole("button")`

What made this work:

The semantic overview existed (accessibility tree)
Containers had stable IDs (test IDs on divs)
Patterns were identifiable (sibling inspection showed repeated structure)
Anchors were unique (heading text differentiated cards)
Each query was bounded

Try it yourself: https://github.com/verdexhq/verdex-mcp

Part 3: Why It Works

What This Pattern Actually Is

This is lazy tree traversal where the LLM decides which branches to expand.

The traversal pattern itself isn't new. Lazy evaluation defers computation until values are needed. Iterators yield items on demand rather than loading entire collections. The visitor pattern traverses structure and applies operations at each node. These are well-established patterns for handling large structures efficiently.

What's different here is the heuristic for branch selection. Traditional lazy traversal uses predetermined rules: depth-first, breadth-first, or application-specific logic. Here, the LLM acts as the heuristic. It looks at the task ("find a stable selector for this button"), examines the current node's context, and decides which branch to expand next. The traversal order emerges from task understanding rather than fixed rules.

This matters because the "right" traversal depends on what you're trying to accomplish. For selector authoring, you need ancestors (for containment) and siblings (for uniqueness). For accessibility auditing, you might need descendants (to check nested elements). For scraping, you might need entirely different branches. The LLM adapts the traversal to the task without requiring separate traversal implementations for each use case.

Why Bounded Queries Preserve Reasoning Quality

The token savings are obvious—sending less data costs fewer tokens. The interesting question is why the LLM reasons better with 2.9k tokens than with 127k.

The Attention Trough Problem

The U-shaped attention curve isn't just a curiosity—it fundamentally changes how LLMs process long contexts. The landmark "Lost in the Middle" paper demonstrated this rigorously: when researchers tested retrieval tasks across different context positions, they found performance dropped significantly for information placed in the middle third of the context window. The model doesn't "forget" the middle content—it's still there in the context—but attention weights concentrate on the beginning and end. Performance is often highest when relevant information occurs at the very start or end, and degrades substantially when models must access information in the middle of long contexts—even for models explicitly designed for long contexts.

Research on context engineering from Anthropic builds on this finding, demonstrating that progressive disclosure prevents cognitive overload—letting agents navigate and retrieve data autonomously keeps them focused on what matters. Work on progressive context enrichment goes further, showing that giving LLMs more context often makes them perform worse.

This creates a perverse dynamic with full structure dumps. The element you care about might appear at position 40,000 in a 127,000 token context. The model sees it, technically, but attention has already anchored on earlier content. When I watched Claude generate selectors from full DOM dumps, it would consistently reference elements from the first few thousand tokens even when better options appeared later.

Most Structural Reasoning Is Local

Here's the key insight: to generate a stable selector for a button, you don't need to know about the footer, the navigation menu, or the sidebar. You need to know:

What contains this button (ancestors)
Whether similar buttons exist (siblings)
What makes this instance unique (local attributes and text)

All of this information exists within a small neighborhood of the target element. The other 95% of the DOM is noise for this specific task.

This is why bounded queries don't lose decision-relevant information. They're not sampling randomly—they're following the containment and sibling relationships that actually matter for the task. You're trading global context (which the LLM can't effectively use anyway due to attention limitations) for local context (which is what the reasoning actually requires).

Information Density Per Token

A full DOM dump has terrible signal-to-noise ratio for any specific task. Of those 127,000 tokens, maybe 3,000 are relevant to finding a stable selector for one button. The LLM has to locate the needle in the haystack while fighting attention dynamics that work against it.

Bounded queries invert this ratio. Each response contains only information relevant to the current decision. The 2,900 tokens across four queries are almost entirely signal. The LLM isn't reasoning with less information—it's reasoning with higher-density information.

Bounded Responses Stay in High-Attention Regions

Each query response is small enough (600-800 tokens) that there's no "middle" to overlook. The entire response falls within the high-attention regions at the beginning and end of the context. When the LLM processes resolve_container's output, every ancestor in that response gets appropriate attention weight.

This converts the attention problem from "find the relevant content in 127k tokens" to "process these 700 tokens of relevant content." The latter is a task LLMs handle well.

The Net Effect

Lazy traversal doesn't just reduce costs—it restructures the problem to play to LLM strengths. Instead of global search through a long context (hard), you get local reasoning over focused responses (easy). Instead of fighting attention dynamics (lossy), you work with them (each response is fully attended). Instead of low signal-to-noise (wasteful), you get high density (efficient).

The LLM reasons better with 2.9k tokens because those tokens are the right tokens, presented in a way that attention mechanisms can actually process.

Why This Worked for DOM

Lazy traversal worked for DOM exploration because of how DOMs are structured. Understanding these characteristics might help you evaluate whether similar patterns could work for your data structure.

Clear Containment

Every element has exactly one parent. When you ask "what contains this button?", there's an unambiguous answer: this div, which is inside that section, which is inside body, which is inside html. You can walk up the tree and always know where you are.

This seems fundamental. In structures where containment is ambiguous—like dense graphs where node A connects to nodes B, C, and D with equal weight—the question "what contains X?" doesn't have a clear answer. You'd need to define traversal rules that create deterministic paths.

Bounded Exploration

Traversing from any node to the root terminates predictably. A tree of depth 20 requires max 20 steps. Even deeply nested pages have predictable bounds.

Without guaranteed termination, lazy traversal might return unbounded results, losing the token efficiency that makes the pattern valuable. Cyclic structures would need explicit cycle detection and depth limits.

Self-Describing Nodes

The identifying properties of DOM elements—data-testid, class, id, role—are attached directly to nodes. You can inspect an element and immediately see what makes it unique without traversing relationships.

If critical metadata lives in edge properties or requires graph traversal to gather, each query becomes more expensive. You might need to fetch relationships just to understand what you're looking at.

Semantic Overview Exists

The accessibility tree provides a clean, high-level view showing roles and names without implementation details. This is essential for Level 1—where does exploration start?

Without a natural overview layer, you'd need to create one through sampling, aggregation, or summarization. If creating an overview requires complex analysis, lazy traversal might not provide enough value.

Independent Subtrees

Inspecting one component doesn't require knowing about unrelated components. I can explore a product card without understanding the navigation menu. Subtrees are relatively self-contained.

In densely connected structures where understanding any node requires understanding most other nodes, you can't isolate exploration. You end up needing global context anyway.

Meaningful Depth

"Ancestor at level 3" or "siblings at this depth" are coherent concepts in DOM. Depth is well-defined and stable. Different paths from leaf to root don't exist.

In structures where depth from one node to another isn't unique—multiple paths of different lengths—operations like "give me siblings at this level" become ambiguous.

Summary: If your structure shares these characteristics—file systems, organizational hierarchies, taxonomy trees—the pattern might transfer well. If it looks more like a dense graph where nodes are heavily interconnected, you'll probably need a different approach like sampling, summarization, or indexing.

I don't know where the exact boundary is. I've only built this for DOM. But these structural characteristics seemed essential to making it work.

Design Principles for Building Query Primitives

When implementing lazy traversal primitives, there are key principles I learned through building real tools.

Return Facts, Not Interpretations

This is maybe the most important lesson. The separation isn't just good practice—it's about preserving optionality for different tasks.

Early on, I tried to make primitives "helpful" by adding interpretation. Like a tool that returned { type: "product-card", confidence: 0.85, role: "list-item" }. The tool was guessing semantic meaning and adding confidence scores.

This failed completely. What if it's not actually a product card? The tool imposed an interpretation that's hard for the LLM to override. Different tasks need different interpretations of the same structure—authoring a selector versus debugging why one broke versus refactoring patterns. And when the heuristics broke, there was no way for the LLM to work around them because the decision had already been made.

The key insight: structure traversal is task-independent, but interpretation is task-dependent. A div with data-testid="product-card" could be a stable container (selector authoring), a UI component to screenshot (visual testing), product data to extract (scraping), or an element missing ARIA labels (accessibility audit). Same structural facts, four different interpretations. If the tool chooses one, the other three tasks suffer.

Research on LLM modularity demonstrates this principle at the architectural level—the "separability" of capabilities enables effective modularization. When different parts of a system are separately responsible for different tasks, you can evolve them independently.

The solution: return raw structural facts. Just the tag, the attributes, the depth. Let the LLM interpret based on the user's query, the task context, and the surrounding structure.

Good primitive design looks like:

{ "tag": "div", "attrs": {"data-testid": "product-card"}, "depth": 2 }

That's it. No guessing, no confidence scores, no strategic recommendations.

Why this works better:

The LLM has full context from the user query to interpret meaning
Raw structural facts compose across different tasks
There are no hidden heuristics that break on edge cases
The LLM's reasoning is auditable because it's based on explicit facts

The key principle: tools provide capability (access to structure). LLMs provide agency (interpretation and decision-making).

Guaranteed Bounded Output

Every primitive must promise it will terminate in finite time and return bounded tokens.

For DOM:

resolve_container is bounded by tree height (typically max 20 levels)
inspect_pattern is bounded by explicit limits (return first 20 siblings only)
extract_anchors is bounded by depth constraints (traverse max 3 levels deep)

Bad primitives would be:

find_all_matching(selector) could return the entire DOM
explore_connected(node) could loop infinitely in a graph with cycles

Why bounds matter:

They prevent infinite loops in graph-like structures
They guarantee predictable token costs so you can budget accurately
They enable LLMs to compose multiple calls without explosion
They make lazy traversal actually lazy rather than "eventually everything"

Enable Composition

Primitives should compose naturally where the output of one call contains the metadata needed for the next.

The DOM workflow shows this:

browser_snapshot() returns refs for elements
Those refs are inputs to resolve_container(ref) which returns level information
That level becomes input to inspect_pattern(ref, level) and extract_anchors(ref, level)
Each step builds on the previous

Requirements for composability:

Outputs include metadata for next calls (refs, levels, paths)
No hidden state between calls that the LLM needs to track
Order independence where possible—pattern inspection and anchor extraction can run in parallel

The DOM workflow shows linear composition: each output deterministically points to the next input. But some domains require branching composition where outputs need interpretation before choosing the next query. Linear composition works with tool descriptions alone; branching composition requires Skills to teach decision logic. If your primitives compose linearly 80%+ of the time, lazy traversal will feel natural to LLMs.

Research on in-context learning explains why this matters: in-context learning works as implicit Bayesian inference—models "locate" latent concepts they acquired during pretraining. Few-shot examples in tool descriptions show individual tool use, but the latent concept for tool composition may not exist in the model's prior at all. That's why knowledge layers matter.

When primitives compose well, the LLM can discover multi-step workflows through experimentation rather than requiring hardcoded sequences.

Part 4: When to Use This

Decision Framework

Question 0: Discovery or Access?

Are you exploring structure you don't control, or accessing content you organized yourself?

If you control the structure (like Anthropic's Skills directories), use good organization + generic file operations. You don't need custom query primitives—you can just read files at known paths.

If you don't control the structure, continue to the remaining questions.

Question 1: Is it a tree?

This pattern works for trees—structures where each node has exactly one parent. "What contains this?" must have an unambiguous answer.

Trees (pattern applies): DOM, file systems, ASTs, org charts, taxonomy hierarchies
DAGs (pattern needs adaptation): dependency graphs, citation networks—you'd need traversal rules to pick one parent
Dense graphs (pattern doesn't apply): social networks, knowledge graphs—use sampling or indexing instead

Question 2: Token economics?

Does getting everything upfront cost more than 10x what you actually need? If full dumps fit comfortably in context with good results, skip the complexity.

Question 3: Attention problems?

Evidence the LLM misses information buried in your current approach? If it handles full context fine, you don't need this.

Question 4: Clear entry point?

Do you know what specific thing you're investigating? "This button" works. "Any anomalies in this dataset" doesn't—you need a starting node for lazy traversal.

Question 5: Repeated exploration?

Lazy traversal has upfront cost. Worth it for 100+ investigations per month. Not worth it for one-off analysis.

When Not to Use This

Structure is too simple. Everything fits under 5k tokens.

Tasks require global analysis. "Count all instances" needs complete context anyway.

Round-trip latency dominates. High-latency environments favor full dumps despite token cost.

Structure changes rapidly. Real-time updates invalidate refs between queries.

Alternative Approaches

Sampling for statistical analysis—random samples of structure.

Summarization for high-level understanding—LLM-generated structure summaries.

Indexing for repeated access patterns—pre-computed views for common queries.

Hybrid for complex tasks—explore to find relevant subset, then dump just that subset.

Closing Thoughts

Lazy traversal with bounded queries solved a real problem for DOM exploration: long contexts degrade LLM performance, but minimal context leaves models without necessary understanding. Revealing structure incrementally through bounded queries worked better than any alternative I tried.

The pattern is essentially lazy tree traversal where the LLM decides which branches to expand. The traversal mechanics aren't new—but using task context to guide branch selection lets the same primitives serve different use cases without modification.

Research on modular agent architectures validates this approach: separation of concerns enables clear input/output boundaries, improves controllability, and limits the blast radius of failures. The same principle applies here—separating traversal capability from task-specific interpretation makes both layers more robust.

The pattern worked because DOM is a tree with clear containment, bounded exploration, and self-describing nodes. If your structure shares these properties—file systems, ASTs, org charts—the pattern should transfer well.

The design principles matter most:

Return facts, not interpretations — let the LLM decide what things mean
Guarantee bounded output — every query terminates predictably
Enable composition — outputs contain metadata for next calls
Separate capability from strategy — tools provide access, skills provide knowledge

If you try this for file systems, ASTs, or other tree-structured data, I'd love to hear what you learn.

Why AI Can't Write Good Playwright Tests (And How To Fix It)

Johnny — Wed, 03 Dec 2025 10:47:55 +0000

This article is for anyone writing Playwright tests with AI assistance

TL;DR: AI agents using Playwright MCP often write positional selectors like getByRole('button', { name: 'Add to Cart' }).nth(8) or chain parent traversals like locator('..').locator('..') because accessibility snapshots omit non-semantic containers. Verdex is an experimental MCP Server that solves this through progressive disclosure at two layers:

Layer 1 - Three DOM exploration primitives (resolve_container, inspect_pattern, extract_anchors) reveal structural information incrementally, using ~3k tokens per exploration instead of 50k+ token DOM dumps.

Layer 2 - AI assistant configuration (cursor rules, Claude skills, or similar) teaches LLMs how to compose these tools correctly, loading instructions progressively.

Together, these layers enable LLMs to generate stable, role-first, container-scoped selectors without information overload or manual guidance.

Verdex was built to solve two core problems:
• Brittle order-based selectors that break when DOM structure changes due to code refactors, hydration mismatches, or dynamic content loading
• Multi-role test authoring for complex E2E workflows (e.g. admin ↔ user ↔ customer)

Tech: TypeScript, Puppeteer (CDP), isolated JavaScript execution contexts.
Outputs Playwright Code: Verdex is an authoring-time intelligence layer, not a runtime. Verdex's output is pure Playwright code that you own and run anywhere.
GitHub: https://github.com/verdexhq/verdex-mcp
Why did I build Verdex on CDP? Comprehensive answer here.

Article Roadmap

Part 1: The Problem (4 min read)

Why accessibility snapshots aren't enough for AI test authoring
The brittle selector problem: .nth(8) and parent traversal chains
How accessibility trees hide structural anchors by design

Part 2: The Foundation (3 min read)

Accessibility snapshots with DOM mapping
How refs become entry points for structural exploration
The hybrid approach: semantic view + structural queries

Part 3: Progressive Disclosure in Action (8 min read)

The four-level workflow: overview → containers → patterns → anchors
Complete product card example with code and token counts
Why 3k tokens beats 50k+ DOM dumps

Part 4: Teaching LLMs to Compose (5 min read)

Why tools alone aren't enough (the compositional knowledge gap)
AI assistant configuration as progressive knowledge delivery
The parallel architecture: capability + strategy

Part 5: Implementation Deep Dive (6 min read)

CDP, Puppeteer, and isolated JavaScript execution contexts
Bridge architecture and the ref-to-DOM mapping
Multi-role browser isolation for complex E2E flows

Part 6: Lessons and Open Questions (4 min read)

Design principles: facts not interpretations, bounded output, composition
Limitations and trade-offs
What we still don't know about primitive decomposition

The Architecture in Brief

Before diving into the details, it's important to understand the design principle that runs through every part of the system: progressive disclosure.

Progressive disclosure is the practice of revealing information incrementally—showing only what's needed for the next decision instead of overwhelming with everything upfront. Verdex applies this principle at two critical layers:

Layer 1: Progressive Structure Discovery (DOM Exploration): Instead of dumping the entire DOM (50k+ tokens), Verdex reveals structure incrementally through three primitives: resolve_container → inspect_pattern → extract_anchors. Each returns targeted structural facts (~3k tokens), allowing LLMs to build understanding step-by-step without information overload.

Layer 2: Progressive Knowledge Loading (AI Configuration): Instead of loading all guidance upfront, AI assistant configuration files (cursor rules, Claude skills, or similar) reveal instructions incrementally: metadata (always loaded, ~100 tokens) → core instructions (when triggered, ~5k tokens) → detailed guides (as needed, ~15k+ tokens). This teaches LLMs how to compose the tools correctly while keeping token costs low.

Both layers solve the same fundamental problem: information overload degrades LLM performance. Whether it's raw DOM data or procedural knowledge, dumping everything at once reduces accuracy and increases costs. Progressive disclosure keeps LLMs focused on what matters for each specific task.

This article explains how both layers work together to enable LLMs to generate stable, role-first, container-scoped Playwright selectors.

Part 1: The Problem

1.1 Brittle Selectors and Missing Structure

Nine months ago, I started building a comprehensive Playwright test suite for a large SaaS application (~500k LOC, thousands of components). I used the standard tooling: Playwright's codegen, inspector, and when it was released, the Playwright MCP server for AI-assisted test authoring.

As much as I initially loved Playwright MCP when it was released, I noticed I spent a lot of my time context switching to fix selectors that coding agents generated - which consistently looked like this:

getByRole('button', { name: 'Add to Cart' }).nth(8)

Instead of what I actually needed—a role-first selector that's properly scoped to the right container:

getByTestId("product-card")
  .filter({ hasText: "iPhone 15 Pro" })
  .getByRole("button", { name: "Add to Cart" })

This follows Playwright's documented best practices: prioritize user-facing attributes (roles), scope to stable containers (test IDs or landmarks), and use content filters for disambiguation.

The first selector is brittle—if the order of page elements changes at all (e.g., due to refactors or client-side hydration differences or because of dynamic content differences), the test breaks.

Initially, this seemed like a prompting problem. But after deep diving into the Playwright source code, I realized it wasn't. It's a structural information problem: The accessibility tree focuses on semantically meaningful nodes. Generic structural containers—like <div>'s used purely for layout, even those with data-testid or class names—are either collapsed away or appear as bare "generic" nodes without their structural attributes in the accessibility tree. This keeps the view clean and standards-compliant, but removes the structural anchors agents need for scoping.

1.2 Accessibility-First API's

Modern test tooling (Playwright, Testing Library) intentionally exposes an accessibility-focused view of the DOM. This is brilliant for helping humans and LLMs retain a semantic view of the page:

You think like a user, not a DOM archaeologist
Tests stay resilient to implementation changes
Forces you to build accessible UIs

Here's what an example accessibility snapshot shows for product cards:

generic
  heading "iPhone 15 Pro"
  button "Add to Cart" [ref=e3]
generic
  heading "MacBook Pro"
  button "Add to Cart" [ref=e4]

This tells you the elements are semantically grouped, but that single generic container could represent any of these actual DOM structures:

<!-- Possibility 1: Simple flat structure -->
<div data-testid="product-card">
  <h3>iPhone 15 Pro</h3>
  <button>Add to Cart</button>
</div>

<!-- Possibility 2: Deep nesting with component boundary at top -->
<div data-testid="product-card">
  <div class="card-header">
    <div class="title-wrapper">
      <h3>iPhone 15 Pro</h3>
    </div>
  </div>
  <div class="card-body">
    <div class="actions">
      <button>Add to Cart</button>
    </div>
  </div>
</div>

<!-- Possibility 3: Multiple potential component boundaries -->
<article data-testid="product-card">
  <header class="product-header">
    <h3>iPhone 15 Pro</h3>
  </header>
  <footer class="product-footer">
    <button>Add to Cart</button>
  </footer>
</article>

How normalization hides structural anchors:

The accessibility tree intentionally collapses all three structures into the same generic node because:

Elements without semantic ARIA roles (<div>, <article>, <header>, <footer>) → generic
Non-ARIA attributes (data-testid, class, id) → stripped from the tree
Layout wrapper elements → flattened away

This normalization is correct per W3C accessibility specifications—it ensures screen readers and accessibility tools see a clean semantic structure. But it can also remove the structural anchors that test authors need for writing stable selectors.

This isn't a matter of prompt engineering or model capability. It's information-theoretically impossible to generate getByTestId("product-card") when data-testid="product-card" is not present in the input. The accessibility tree—by design and specification—omits these attributes.

The Playwright team would reasonably argue that the solution is refactoring your HTML to be more semantic—ensuring proper ARIA roles and accessible structure. While this is the ideal approach, it's often impossible or impractical in real-world scenarios: legacy codebases with thousands of components, third-party dependencies you don't control, resource constraints that prioritize features over refactoring, and tight timelines. Verdex bridges the gap between best practice and current reality, enabling stable test authoring against imperfect DOM structures.

1.3 The Gap: Component Boundaries Aren't Visible

Without seeing the actual DOM structure, LLMs cannot determine:

Where component boundaries exist: Is the stable container one level up? Four levels up? At different levels for heading vs button?
What anchors are available: Does that boundary have a data-testid, id, or semantic role? Or is it a bare <div>?
How to scope reliably: Which scoping patterns will actually work for this structure?

This creates a difficult situation for AI agents:

// Option 1: Try parent traversal
page.getByRole('heading', { name: 'iPhone 15 Pro' })
  .locator('..').locator('..')  // How many levels? 
  .getByRole('button', { name: 'Add to Cart' })

// Option 2: Use nth() as a safe fallback
page.getByRole('button', { name: 'Add to Cart' }).nth(8)

The LLM often falls back to nth() because it's deterministic when component boundaries aren't visible.

1.4 Failed Fix: Dumping the DOM

My first attempt was straightforward: give the LLM the full DOM and let it figure it out.

// Serialize the entire DOM
const domTree = document.documentElement.outerHTML;
// Send to LLM (~10k-100k tokens for a complex page)

This failed for two reasons:

Token Cost

A complex page in the production application I was testing consumed 50k+ tokens per query. This meant:

Each selector exploration required 2-3 queries (finding element → validating → refining)
That's 150k+ tokens per selector

For context (in my environment): each selector exploration often required multiple queries, pushing costs and latency up on complex pages.

Signal-to-Noise Ratio

Most of the DOM is irrelevant for any given selector. The LLM had to wade through:

Styling markup
Script tags
Hidden UI elements
Unrelated page sections

Even when using strong coding models, accuracy degraded when working with massive context windows. The model would hallucinate elements or generate selectors based on irrelevant DOM sections.

I needed a better approach: a way to progressively reveal structural information without overwhelming the LLM with 50k+ token DOM dumps.

The key insight was that I didn't need to choose between "clean but incomplete accessibility tree" or "complete but overwhelming DOM dump." There was a third option—but first, I needed to build the right foundation.

Part 2: The Foundation

2.1 Accessibility Snapshots with DOM Mapping

Before diving into how Verdex enables progressive structural exploration, it's important to understand the technical foundation that makes it possible.

Verdex generates an accessibility tree similar to Playwright MCP's snapshots—showing the semantic structure of the page with roles, names, and ARIA properties. Like Playwright MCP, every interactive element gets a reference ID (e.g., e1, e2) that maps to its underlying DOM node.

The key difference:

Playwright MCP treats refs as opaque handles for direct interactions—click, type, and evaluate operations.
Verdex uses those same refs as entry points for structural exploration, exposing DOM hierarchy queries that enable LLMs to discover container boundaries, identify repeating patterns, and locate unique anchors.

Here's what a Verdex snapshot looks like:

navigation
 link "Home" [ref=e1]
 link "Products" [ref=e2]
main
 heading "Featured Products"
generic
 heading "iPhone 15 Pro"
 button "Add to Cart" [ref=e3]
generic
 heading "MacBook Pro"
 button "Add to Cart" [ref=e4]

The snapshot presents the clean semantic structure visible to users, excluding implementation details such as data-testid attributes or class names. This keeps the view focused on semantically meaningful interactive elements.

When structural context is needed for selector construction, Verdex's DOM exploration tools become relevant. Using the same e3 reference, you can inspect the corresponding DOM structure to trace elements back to their rendered source.
Calling resolve_container(e3) returns:

Level 1 (div):
Attributes: {"data-testid": "product-card"}
Level 2 (div):
Attributes: {"data-testid": "product-grid"}
Level 3 (section):
Attributes: {"class": "products"}

This hybrid approach gives you the best of both worlds:

From accessibility snapshots: Clean semantic view of the page that mirrors how users (and screen readers) experience it. No noise from implementation details.

From structural exploration tools: The ability to efficiently explore DOM relationships using the stored mapping. When you see button "Add to Cart" [ref=e3], the LLM can immediately call resolve_container(e3) to discover it's inside a product-card container.

Like Playwright MCP, the mapping happens through a bridge running in an isolated JavaScript execution context. During snapshot generation, the bridge:

Traverses the DOM following W3C ARIA specifications
Identifies which elements contribute to the accessibility tree
Assigns reference IDs to interactive elements
Stores the mapping in a Map<string, ElementInfo> where each entry contains both the element reference and its DOM node

Verdex's innovation is exposing structural exploration primitives on top of this foundation, rather than limiting refs to direct element interactions.

2.2 Token Efficiency Example

Without DOM mapping, you'd need to send the full DOM context to identify containers:

<div class="app">
  <section class="products">
    <div data-testid="product-grid">
      <div data-testid="product-card">
        <h3>iPhone 15 Pro</h3>
        <span class="price">$999</span>
        <button>Add to Cart</button>  <!-- Which button? -->

With Verdex's mapping, you get exactly what you need:

Level 1 (div):
   Attributes: {"data-testid": "product-card"}
Level 2 (div):
   Attributes: {"data-testid": "product-grid"}
Level 3 (section):
   Attributes: {"class": "products"}

With this foundation in place—accessibility snapshots that provide clean semantic views AND ref-to-DOM mappings that enable structural queries—Verdex can expose three primitives that use these refs as entry points for progressive exploration.

Part 3: Progressive Disclosure in Action

3.1 The Verdex Approach: Progressive Disclosure of DOM Structure

Building on the accessibility snapshot foundation, Verdex implements progressive disclosure through three exploration primitives that compose into a disclosure hierarchy.

Instead of forcing a binary choice between "clean but incomplete accessibility tree" or "complete but overwhelming DOM dump," Verdex provides a third option: start with the semantic overview, then progressively disclose structural details through targeted queries.

Each primitive uses the ref IDs from accessibility snapshots as entry points, then returns just enough structural information for the LLM to decide whether it needs more detail—keeping total exploration under 3k tokens instead of 50k+ for full DOM dumps.

The primitives return deterministic, low-level structural facts that the model can compose to keep role-first selectors properly container-scoped and resilient to layout or hydration changes.

Here's the disclosure hierarchy:

Level 1: Semantic Overview (browser_snapshot): Shows what users see: roles, names, interactive elements. Hides implementation details like class names and test IDs.

Level 2: Structural Containers (resolve_container): Reveals the containment hierarchy: which divs, sections, or articles wrap the target element, along with their stable attributes.

Level 3: Repeating Patterns (inspect_pattern): Exposes sibling elements at a specific depth to identify lists, grids, or card layouts—making component boundaries deducible.

Level 4: Unique Anchors (extract_anchors): Mines distinctive content within a container: headings, labels, test IDs, or unique text that can differentiate one card from another.

Let's examine each primitive in detail:

3.2 `resolve_container(ref)`

Returns the full ancestor chain (up to document.body) with minimal metadata the model can use to pick stable scoping containers—including examples like data-testid, id, semantic landmarks (main, nav, article, section), and class-name patterns.

Example output:

{
  "target": { "ref": "e3", "tagName": "button", "text": "Add to Cart" },
  "ancestors": [
    { "level": 1, "tagName": "div", "attributes": { "data-testid": "product-card" }, "childElements": 5, "containsRefs": [] },
    { "level": 2, "tagName": "div", "attributes": { "class": "product-grid" }, "childElements": 12, "containsRefs": [] },
    { "level": 3, "tagName": "section", "attributes": { "id": "app-content" }, "childElements": 3, "containsRefs": [] }
  ]
}

3.3 `inspect_pattern(ref, level)`

Climbs level ancestors from the target to identify a container, then returns that container's children. This makes repeating patterns at that depth visible—useful for spotting lists, grids, cards, or table rows.

If level walks past the root (up to document.body), the call returns null.

The response also includes a targetSiblingIndex (which direct child of the container contains the target) and an intentionally shallow and token-cheap outline for each sibling (e.g., top-level headings/buttons/testids), which helps the model choose the right level for the next step in the process.

Example output:

{
  "ancestorLevel": 1,
  "containerAt": {
    "tagName": "div",
    "attributes": { "data-testid": "product-grid" }
  },
  "targetSiblingIndex": 1,
  "siblings": [
    {
      "index": 0,
      "tagName": "div",
      "attributes": { "data-testid": "product-card" },
      "containsText": ["iPhone 15 Pro", "$999", "Add to Cart"],
      "outline": [
        { "tag": "h3", "text": "iPhone 15 Pro" },
        { "role": "button", "text": "Add to Cart" }
      ]
    },
    {
      "index": 1,
      "tagName": "div",
      "attributes": { "data-testid": "product-card" },
      "containsText": ["MacBook Pro", "$1,999", "Add to Cart"],
      "outline": [
        { "tag": "h3", "text": "MacBook Pro" },
        { "role": "button", "text": "Add to Cart" }
      ]
    }
  ]
}

3.4 `extract_anchors(ref, level)`

Performs a bounded deep scan at the chosen level to surface deeper headings, labels, role/name-bearing nodes, and other unique anchors when the shallow outline from inspect_pattern isn't sufficient.

Headings (h1-h6)
Labels
Meaningful text nodes
Interactive elements with names
Elements with data-testid

Example output:

{
  "descendants": [
    {"tag": "h3", "text": "iPhone 15 Pro", "depth": 1},
    {"tag": "button", "attrs": {"aria-label": "Add to Cart"}, "depth": 3},
    {"tag": "span", "attrs": {"data-testid": "price"}, "text": "$999", "depth": 2}
  ]
}

3.5 How the LLM Uses These Primitives: Progressive Disclosure in Action

Here's how progressive disclosure works in practice. The LLM starts with minimal information and requests more only when needed:

Disclosure Level 1: Initial Snapshot

LLM calls browser_snapshot which contains a target element that we want to container scope. In this case we want to scope the "Add to Cart" button for the iPhone 15 Pro specifically:

navigation
  link "Home" [ref=e1]
  link "Products" [ref=e2]
main
  heading "Featured Products"
  generic
    heading "iPhone 15 Pro"
    button "Add to Cart" [ref=e3] // <-- Target Button
  generic
    heading "MacBook Pro"
    button "Add to Cart" [ref=e4]

At this point, the LLM sees there are multiple "Add to Cart" buttons but doesn't know their DOM structure. The accessibility tree shows they're in separate generic containers, but that could represent any of dozens of actual HTML structures. The LLM needs more information—but only about the specific button it cares about.

Disclosure Level 2: Container Discovery

The LLM calls resolve_container(e3) to reveal the structural containers around the target button:

{
  "target": { "ref": "e3", "tagName": "button", "text": "Add to Cart" },
  "ancestors": [
    { "level": 1, "tagName": "div", "attributes": { "data-testid": "product-card" } },
    { "level": 2, "tagName": "div", "attributes": { "data-testid": "product-grid" } },
    { "level": 3, "tagName": "section", "attributes": { "class": "products" } }
  ]
}

Now the LLM knows the button lives inside nested divs with data-testid="product-card" at level 1 and data-testid="product-grid" at level 2. But it doesn't know if this is one card or many. Are there other product cards? The LLM needs to check siblings.

Disclosure Level 3: Pattern Recognition

The LLM calls inspect_pattern(e3, 2) to see if there are multiple cards at the grid level:

{
  "ancestorLevel": 2,
  "containerAt": {
    "tagName": "div",
    "attributes": { "data-testid": "product-grid" }
  },
  "targetSiblingIndex": 1,
  "siblings": [
    {
      "index": 0,
      "tagName": "div",
      "attributes": { "data-testid": "product-card" },
      "containsText": ["MacBook Air", "$1,499", "Add to Cart"],
      "outline": [
        { "tag": "h3", "text": "MacBook Air" },
        { "role": "button", "text": "Add to Cart" }
      ]
    },
    {
      "index": 1,
      "tagName": "div",
      "attributes": { "data-testid": "product-card" },
      "containsText": ["iPhone 15 Pro", "$999", "Add to Cart"],
      "outline": [
        { "tag": "h3", "text": "iPhone 15 Pro" },
        { "role": "button", "text": "Add to Cart" }
      ]
    }
    // ... ~10 more cards
  ]
}

Confirmed: this is a repeating grid of product cards. The siblings array shows ~12 cards, each with a heading and an "Add to Cart" button. But which card has the iPhone 15 Pro? The shallow outline shows "iPhone 15 Pro" text exists in the target's card, but the LLM needs to verify it can use this as a unique anchor.

Disclosure Level 4: Unique Identification

The LLM calls extract_anchors(e3, 1) to mine unique anchors within the target's card:

{
  "ancestorAt": {
    "level": 1,
    "tagName": "div",
    "attributes": { "data-testid": "product-card" }
  },
  "descendants": [
    {
      "depth": 1,
      "index": 0,
      "tagName": "h3",
      "attributes": {},
      "fullText": "iPhone 15 Pro"
    },
    {
      "depth": 1,
      "index": 1,
      "tagName": "span",
      "attributes": { "class": "price" },
      "directText": "$999"
    },
    {
      "depth": 1,
      "index": 2,
      "tagName": "button",
      "attributes": {},
      "fullText": "Add to Cart"
    }
  ],
}

Perfect. The LLM now has exactly what it needs:

A stable container: data-testid="product-card" (from ancestors)
A unique identifier: heading with text "iPhone 15 Pro" (from descendants)
The target action: button with role and name "Add to Cart" (from initial snapshot)

Total token cost: ~2,100 tokens** instead of 50,000+ for a full DOM dump.

Workflow Complete: Generate Scoped Selector

With this worfklow complete, the LLM generates:

getByTestId("product-card")
  .filter({ hasText: "iPhone 15 Pro" })
  .getByRole("button", { name: "Add to Cart" })

Why progressive disclosure made this possible:

Focused exploration: Each query targeted a specific structural question (containers? repeating? unique anchors?) instead of processing everything upfront
Token efficiency: ~2k tokens of high-signal structural facts vs 50,000+ tokens of noisy HTML
Compositional reasoning: The LLM built up understanding incrementally, making decisions about what to explore next based on what it learned
Deterministic output: Each primitive returns raw structural facts, not interpretations, so the LLM can compose them contextually

This workflow works because each disclosure level revealed just enough information for the LLM to decide what question to ask next—not so much that it got lost in noise, but not so little that it had to guess.

3.6 Why Progressive Disclosure Matters for Agent Tooling

Progressive disclosure isn't just about token efficiency—it's about cognitive architecture for LLMs.

Information Overload Degrades Performance

When given massive context windows, LLMs exhibit known attention degradation effects—important details buried in noise get ignored or misinterpreted.

The latest research shows "performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts."

In testing Verdex, I found consistent patterns:

Full DOM dumps (50k+ tokens) created systematic problems: LLMs would hallucinate elements that didn't exist, selectors referenced unrelated page sections, the model would "anchor" on early DOM sections and miss better containers later, and accuracy degraded even with strong/expensive models.

Progressive exploration (0.5-3k tokens per step) eliminated most of these issues. LLMs maintained focus on relevant structural facts, selectors consistently used the most stable available anchors, the model's reasoning was more coherent—each step built logically on the previous one—and weaker models performed significantly better with progressive disclosure than they did with full dumps.

This created an interesting challenge: the tools provided the right structural data, but LLMs still struggled to compose them correctly. Models would call resolve_container but ignore the data-testid in the response, skip inspect_pattern and generate positional selectors, or call tools in the wrong order.

The problem wasn't the tools—it was compositional knowledge. LLMs needed to learn the 3-step workflow, how to interpret responses, and why certain selector patterns are stable. This led to exploring AI assistant configuration as Layer 2 of the progressive disclosure architecture—a topic covered in depth in the "Teaching LLMs to Compose" section below.

Composability Enables Generalization

By breaking DOM exploration into primitive operations, the LLM can adapt patterns to new scenarios: The same container → pattern → anchor sequence works across radically different UI structures. The LLM doesn't need to see examples of every possible DOM layout—it learns the exploration pattern itself.

This also enables incremental debugging. When a selector fails, the LLM can replay just the exploration steps that matter. Instead of "let me dump the entire DOM again," it can ask "let me check the ancestors of this element to see if the container changed."

Few-shot demonstrations of the exploration pattern transfer better than monolithic "here's everything" examples. I can show the LLM how to explore a product card, and it generalizes to user cards, settings panels, and data tables without additional examples.

3.7 Works Without Test IDs or Class Names

A common concern is: "What if my app doesn't have data-testids or semantic class names?" Verdex's primitives work on pure HTML. The system relies on the DOM's inherent structure (parent/child relationships, tag names, positions) and visible text. Test IDs and class names improve stability, but they're not required.

Example: classless markup

<section>
  <div>
    <h3>iPhone 15 Pro</h3>
    <span>$999</span>
    <button>Add to Cart</button>
  </div>
  <div>
    <h3>MacBook Pro</h3>
    <span>$1,999</span>
    <button>Add to Cart</button>
  </div>
</section>

Step 1 — resolve_container

resolve_container("e3") // e3 = the "Add to Cart" button

{
  "target": { "ref": "e3", "tagName": "button", "text": "Add to Cart" },
  "ancestors": [
    { "level": 1, "tagName": "div", "attributes": {}, "childElements": 3, "containsRefs": [] },
    { "level": 2, "tagName": "section", "attributes": {}, "childElements": 2, "containsRefs": [] }
  ]
}

Step 2 — inspect_pattern at the section level

inspect_pattern("e3", 2)

{
  "ancestorLevel": 2,
  "containerAt": { "tagName": "section", "attributes": {} },
  "targetSiblingIndex": 1,
  "siblings": [
    {
      "index": 0,
      "tagName": "div",
      "attributes": {},
      "containsText": ["iPhone 15 Pro", "$999", "Add to Cart"],
      "outline": [
        { "tag": "h3", "text": "iPhone 15 Pro" },
        { "role": "button", "text": "Add to Cart" }
      ]
    },
    {
      "index": 1,
      "tagName": "div",
      "attributes": {},
      "containsText": ["MacBook Pro", "$1,999", "Add to Cart"],
      "outline": [
        { "tag": "h3", "text": "MacBook Pro" },
        { "role": "button", "text": "Add to Cart" }
      ]
    }
  ]
}

Step 3 — generate a scoped selector

page.locator("section > div")
  .filter({ hasText: "MacBook Pro" })
  .getByRole("button", { name: "Add to Cart" })

Why this works:

Hierarchy: ancestors reveal the unit container (here, section > div).
Position: targetSiblingIndex shows which sibling holds the target.
Content: outline/containsText provide unique anchors ("MacBook Pro").
Repetition: sibling listing confirms the repeating structure.

Trade-off: structure-only based selectors are less resilient than using data-testids or class names. But even with zero testing infrastructure, Verdex still enables AI agents to write deterministic, container-scoped selectors.

Part 4: Teaching LLMs to Compose

4.1 The Missing Piece

Progressive disclosure solved the DOM data problem—but the tools alone weren't enough.

When I first released Verdex, I expected LLMs to naturally discover the 3-step workflow: resolve_container → find containers, inspect_pattern → check patterns, extract_anchors → mine unique content. After all, the tool descriptions explain what each does.

They didn't.

Strong models like Claude Sonnet would:

Call resolve_container but ignore the data-testid in the response
Skip inspect_pattern entirely and generate positional .nth() selectors
Call tools in the wrong order or with invalid parameters
Generate technically correct but brittle selectors

The tools provided the capability to understand structure. What was missing was the knowledge to compose them correctly.

4.2 The Compositional Knowledge Gap

LLMs needed to learn:

When to use each primitive (the 3-step sequence)
How to interpret the responses (what structural facts matter)
Why certain patterns are stable (role-first, container-scoped)
What to do with edge cases (no test IDs, flaky selectors, dynamic content)

This isn't something tool descriptions can teach. Tool descriptions say "what" and "returns what." They don't teach workflows, best practices, or compositional patterns.

4.3 The Cursor Rules Approach

My first solution was comprehensive cursor rules: a 400-line guide teaching the workflow, best practices, selector patterns, and debugging strategies.

This approach works well and many teams successfully use similar patterns today. It provides clear, consistent guidance and ensures the LLM has access to all necessary knowledge upfront.

However, I noticed some practical limitations in my workflow:

Token overhead: 12k tokens loaded in every conversation, regardless of task complexity
Lost in the middle: With large context windows, relevant instructions could be overlooked when buried in the middle of extensive documentation
No progressive revelation: The entire guide loaded at once, even for simple queries that only needed a subset of the guidance
Session-dependent: Context needed to be re-established across different tools and sessions

The cursor rules approach is solid and functional—it successfully taught LLMs the exploration workflow and delivered good results. However, I recognized an opportunity for improvement: I'd built progressive discovery for DOM exploration, but was using static, upfront knowledge delivery for teaching LLMs how to use it.

This led me to explore whether AI assistant configuration could achieve similar teaching outcomes while addressing the token efficiency and context window challenges I'd observed.

4.4 AI Assistant Configuration: Progressive Disclosure for Knowledge

Modern AI assistants support various configuration mechanisms—cursor rules for Cursor IDE, Claude skills for Claude-based tools, or similar approaches for other platforms. These can implement the same progressive disclosure pattern Verdex uses for DOM exploration.

Level 1: Metadata (~100 tokens, always loaded)

---
name: verdex-playwright-authoring
description: Write robust, container-scoped Playwright selectors using 
  progressive DOM exploration with Verdex MCP tools (resolve_container, 
  inspect_pattern, extract_anchors). Use when authoring Playwright tests, 
  creating selectors, exploring page structure, or debugging test failures.
---

The AI assistant knows the configuration exists and when to trigger it. Token cost: ~100 tokens (negligible).

Level 2: Instructions (~5k tokens, loaded when triggered)

The main configuration file contains:

The 3-step exploration workflow (containers → patterns → anchors)
Selector composition patterns (test IDs → roles → content filters → structure)
Best practices (container-scoped, not positional)
Complete example: product card exploration → stable selector generation

Only loaded when the user asks about selectors or Playwright tests.

Level 3: Resources (~15k tokens each, loaded as needed)

Additional reference files the AI can consult when specific guidance is needed:

playwright-patterns.md - Test structure, assertions, authentication patterns (15k tokens)
selector-writing.md - Container → Content → Role pattern, avoiding anti-patterns (15k tokens)
workflow-discovery.md - Page exploration techniques, workflow mapping (18k tokens)

These are only loaded when the AI determines they're relevant to the specific query—"How do I debug a flaky selector?" triggers the debugging section, but a simple selector request doesn't.

4.5 The Parallel Architecture

This creates symmetry:

Layer	Mechanism	Token Cost	What It Reveals
DOM Exploration	`resolve_container` → `inspect_pattern` → `extract_anchors`	3k per exploration	Structural facts: containers, patterns, unique anchors
Knowledge Delivery	Metadata → Instructions → Resources	100 → 5k → 15k+	Workflow, composition patterns, debugging strategies

Both use progressive disclosure to prevent information overload. Both reveal information only when relevant. Both enable token-efficient iteration.

The tools give LLMs the capability to understand DOM structure.

AI assistant configuration gives LLMs the knowledge to compose it into stable selectors.

4.6 The Workflow with Configuration

Before (without configuration):

User: "Help me click the Add to Cart button for iPhone 15 Pro"
AI: *has no context about Verdex workflow*
User: *manually explains 3-step process or pastes 400-line cursor rules*
AI: *attempts exploration but misinterprets responses*
User: *corrects approach, explains why container scoping matters*
AI: *generates selector on second or third attempt*

After (with configuration):

User: "Help me click the Add to Cart button for iPhone 15 Pro"
AI: *configuration triggers → loads relevant instructions (~5k tokens)*
AI: "Let me explore the structure progressively..."
  → resolve_container(ref) → finds data-testid="product-card"
  → inspect_pattern(ref, 2) → sees 12 similar cards
  → extract_anchors(ref, 1) → finds "iPhone 15 Pro" heading

AI: "Here's a stable, container-scoped selector:"

  page.getByTestId("product-card")
      .filter({ hasText: "iPhone 15 Pro" })
      .getByRole("button", { name: "Add to Cart" })

No manual guidance needed. The configuration teaches the composition pattern automatically when relevant.

4.7 Token Efficiency Comparison

Approach	Standing Cost	Per Query	Quality
No guidance	0 tokens	0 tokens	❌ Generates `.nth(8)` brittle selectors
Monolithic rules	12k tokens	12k tokens	⚠️ Works but always loaded
Progressive configuration	100 tokens	100-5k tokens	✅ Efficient, automatic, progressive
Full DOM dump	0 tokens	50k+ tokens	❌ Information overload

Progressive configuration reduces standing token cost from 12k to 100 while dramatically improving selector quality.

4.8 Why This Matters: Verdex as a Complete Solution

Verdex is trying to be a complete solution:

MCP Tools provide raw structural data efficiently (progressive DOM disclosure)
AI assistant configuration provides compositional knowledge efficiently (progressive knowledge disclosure)
Together they enable LLMs to generate stable, role-first, container-scoped Playwright selectors

Without configuration, you'd need to:

Manually explain the workflow in every conversation
Hope the LLM remembers best practices
Debug why it generated positional selectors
Re-teach patterns across sessions

With proper configuration, the knowledge is:

Automatic: Triggered by relevant queries
Progressive: Only loads what's needed
Portable: Works across different AI assistants with appropriate setup
Versioned: Update the configuration, everyone gets improvements

Part 5: Implementation Deep Dive

5.1 Technical Architecture Overview

The previous sections explained what Verdex does and why it works. This section explains how it's actually built.

Having covered how Verdex works conceptually—the progressive disclosure architecture, the three exploration primitives, and the AI configuration teaching layer—let's examine the technical implementation that makes this possible.

5.2 Technical Architecture: CDP, Puppeteer, and Isolated Worlds

Verdex is built on three core architectural choices:

1. CDP-first (Chrome-only) via Puppeteer

Verdex is a development-time, CDP-first helper—not a test runner. Puppeteer provides direct access to Chrome DevTools Protocol primitives (e.g., Page.createIsolatedWorld, Runtime.callFunctionOn, frame lifecycle hooks). This keeps the implementation straightforward and the mental model simple—building on a CDP client (Puppeteer) rather than a multi-browser framework (Playwright) reduces unnecessary abstraction layers and keeps the codebase focused.

Key CDP Benefits:

Isolated JavaScript Execution — Bridge code runs in a separate execution context invisible to application scripts, preventing interference from the page's own JavaScript (e.g., overwritten globals, monkey-patched DOM APIs, or aggressive framework behavior). This isolation ensures DOM analysis remains reliable regardless of how the application manipulates its environment.
In-Browser Structural Analysis — Complex queries (ancestor chains, sibling patterns, descendant traversal) execute entirely within the browser's JavaScript engine in a single execution context. The bridge performs multi-step DOM analysis and returns only minimal structural facts, eliminating the protocol round-trips required when orchestrating similar analysis through multiple CDP commands. This architecture also ensures all analysis reflects a consistent DOM state, preventing race conditions from dynamic content updates.
Navigation Persistence — The bridge survives page navigations through CDP's frame lifecycle hooks (Page.frameNavigated), maintaining analysis capabilities across single-page app transitions without manual re-injection. This enables LLM workflows that incrementally explore structure across multiple pages—the agent can navigate, snapshot, explore ancestors, navigate again—without losing the analysis infrastructure.
Controlled Serialization Boundary — Only final structural facts (container attributes, sibling patterns, unique anchors) cross the CDP protocol boundary. Instead of retrieving entire DOM subtrees and filtering in Node.js, the bridge performs targeted in-browser analysis and returns only what's needed for selector construction.

Creating an isolated world:

// Create an isolated world
const { executionContextId } = await client.send('Page.createIsolatedWorld', {
  frameId: mainFrameId,
  worldName: 'verdex-bridge',
  grantUniveralAccess: false // Note: CDP API uses this spelling
});

// Code in this world:
// - Can access the DOM
// - Cannot be seen by page scripts  
// - Re-injected on navigation via frame lifecycle events

2. Role Isolation via Browser Contexts

Each role (e.g., admin/user/customer) runs in its own incognito browser context, separating cookies, localStorage, and sessionStorage to prevent auth/data leakage between roles:

verdex-mcp \
  --role admin ./auth/admin.json https://admin.example.com \
  --role user ./auth/user.json https://app.example.com

The LLM switches between roles explicitly:

select_role("admin")
browser_navigate("/promotions/new")
browser_click("e1") // Creates promotion

select_role("user")
browser_navigate("/products")
// User sees the new promotion

The server handles context switching, session persistence, and storage isolation automatically. Each role maintains its own cookies, localStorage, and sessionStorage without leakage.

Browser contexts provide storage/session separation, while isolated worlds keep analysis code invisible to app scripts. Together, they eliminate session leakage and enable deterministic multi-role E2E workflows.

3. Custom Accessibility Snapshot with Structural Mapping

Instead of using CDP's built-in Accessibility.getFullAXTree API, Verdex injects a custom snapshot generator into the isolated execution context. This generator walks the DOM following W3C ARIA specifications and produces a semantic view similar to Playwright MCP—with stable reference IDs (e1, e2, ...) that map directly to DOM nodes.

Why custom instead of CDP's accessibility API? CDP's accessibility tree omits structural metadata that test authors need: data-testid attributes, class names, and id values aren't included unless they're ARIA-referenced. Multiple round-trip calls would be required for structural queries.

Verdex's custom generator captures both the semantic accessibility view and stores a Map<string, ElementInfo> that indexes accessibility references to DOM nodes with full attribute metadata:

// During snapshot generation
const ref = `e${++this.bridge.counter}`;
ariaNode.ref = ref;

this.bridge.elements.set(ref, {
  element: element,        // Live DOM node reference
  tagName: element.tagName,
  role: role,
  name: name,
  attributes: this.getAttributes(element)
});

// Later: resolve_container(ref) walks the stored element's parent chain
// No DOM search, no re-parsing—just direct traversal from the mapped node

The key difference from Playwright MCP:

Playwright MCP: Uses the mapping internally for aria-ref selector resolution—converting refs to elements for interactions (click, type, evaluate)
Verdex: Exposes the mapping through structural exploration tools—resolve_container walks up from stored DOM nodes, inspect_pattern examines container children, extract_anchors traverses subtrees

The Verdex design enables LLMs to reason about DOM hierarchy and write container-scoped selectors.

Architecture Diagram

5.3 Bridge Architecture

The injected bridge code provides specialized analysis modules that run entirely within the isolated world:

DOMAnalyzer — Efficient ancestor/sibling/descendant traversal with configurable depth limits
StructuralAnalyzer — Pattern detection and stable anchor identification
SnapshotGenerator — Custom accessibility tree builder following W3C ARIA specs with ref-to-DOM mapping
AriaUtils — Role computation and semantic element mapping

These modules perform DOM analysis synchronously without protocol round-trips. Only final results cross the CDP boundary:

// Server side - single CDP call returns complete analysis
const { result } = await cdpSession.send('Runtime.callFunctionOn', {
  functionDeclaration: 'function(ref) { return this.resolve_container(ref); }',
  objectId: bridgeObjectId,
  arguments: [{ value: ref }],
  returnByValue: true
});

// Bridge side (in isolated world) - performs full traversal in-browser
window.verdexBridge = {
  resolve_container(ref) {
    // Synchronous DOM traversal
    // Filters for stable containers  
    // Returns minimal JSON
  }
};

This architecture keeps computation close to the data (DOM), minimizes serialization costs, and gives complete control over the token/accuracy tradeoff for each analysis primitive. The bridge code is bundled at build time using esbuild and auto-injected via CDP's Page.addScriptToEvaluateOnNewDocument, persisting across navigations without re-injection overhead.

The bridge architecture took several iterations to get right. If you're curious about the journey from toString() injection to the current pre-bundled approach, I wrote about it here: Why I Ditched toString() and Moved to a Pre-Bundled Bridge with Event-Driven CDP Injection

5.4 LLM-Facing API Design Principles

Early when I started experimenting, I tried to make these tools be "helpful" to the LLM by including an interpretation layer:

// BAD: Tool tries to interpret
{
  "type": "product-card",  // Tool guesses this is a product card
  "role": "list-item",     // Tool guesses it's in a list
  "confidence": 0.85       // Tool guesses confidence
}

This didn't generalize:

1. Heuristics are both brittle and context-dependent, and when a result is delivered it is hard for models to override

Is a div.card a product card, a user card, or a settings panel card?
The tool can't know without understanding the application domain
The LLM can know by considering the user's query and surrounding context

2. Different tasks need different interpretations

Authoring: "Find stable anchors for a selector"
Debugging: "Why did this selector break?"
Refactoring: "What other tests use similar patterns?"

The same DOM structure means different things for different tasks.

3. Tool behavior became unpredictable

Changing the heuristics to fix one case broke others
Adding confidence scores didn't help - because it's just a guess with a number next to it
Prompt engineering couldn't override bad tool decisions

The Solution: return raw structural facts (tags/attrs/positions) and let the model compose them per task

I stripped all interpretation from the tools. They now return only raw structural facts:

// GOOD: Just facts
{
  "tag": "div",
  "attrs": {"data-testid": "product-card", "class": "card border-gray"},
  "depth": 2
}

Core principle: Keep capability in tools, strategy in configuration. Tools provide deterministic building blocks; the LLM (guided by configuration) composes them contextually. This separation means:

Tool mistakes become model mistakes (which the model can self-correct)
Different tasks can interpret the same facts differently
Heuristics stay overridable rather than baked into outputs

The LLM decides what this means based on:

The user's query
The task context
Surrounding DOM structure
Application-specific knowledge

Example: Same tool output, different interpretations

Task: Authoring a selector

LLM reasoning: "data-testid='product-card' is a stable anchor. 
I should use getByTestId() to scope my selector."

Task: Debugging why a test broke

LLM reasoning: "The test used data-testid='product-card' but now 
there are 12 elements with that ID. The developer probably copied 
the component without updating testids. I should suggest filtering 
by unique content or updating the testid."

Trade-off: Raw structural facts require compositional knowledge—LLMs need to learn when to call each primitive and how to interpret the results. This is where AI assistant configuration (covered earlier in "Teaching LLMs to Compose") becomes essential. Configuration files progressively teach the 3-step workflow (containers → patterns → anchors), selector composition patterns, and best practices—loading instructions only when triggered (~5k tokens) instead of requiring 12k+ token monolithic rules upfront. Without proper configuration, even strong models can struggle with correct tool composition.

Part 6: Lessons and Open Questions

6.1 Lessons Learned

LLM-Facing APIs Have Different Constraints Than Human-Facing APIs

LLM-facing APIs benefit from predictable, low-level primitives over 'helpful' abstractions. Deterministic building blocks compose better across unfamiliar UIs and edge cases.

Consider a simple example: humans appreciate a method like page.selectDropdown("Country", "United States") because it handles all the fiddly details. But an LLM actually does better with the raw primitives: page.click('select[name="country"]') followed by page.click('option:has-text("United States")'). The LLM can adapt this low-level pattern to dropdowns it's never encountered before. The high-level abstraction only works for exactly the cases it was designed to handle.

Deterministic Primitives Beat Opinionated Heuristics

Every single time I added "helpful" interpretation to a tool, the same pattern emerged. It worked for the test cases I built it for. Then it broke in production on edge cases I hadn't anticipated. And critically, the LLM had no way to override the bad interpretation because the tool had already made the decision.

Raw structural facts, on the other hand, stayed useful across every scenario I tested it on. The LLM could interpret them contextually based on the user's actual query, the application domain, and the specific task at hand. A div with a data-testid is just a div with a data-testid. What that means depends entirely on context, and the LLM is far better equipped to understand that context than any heuristic I could build into the tools.

Token Efficiency Enables Iteration

Making tools cheap to call fundamentally changed how I used them. The difference isn't just about the direct cost, though that matters. It's about the psychological shift. When each query costs fifty cents and takes fifteen seconds, you plan carefully. You hesitate. You make fewer exploratory calls. You write more conservative prompts because iteration is expensive.

When queries cost a penny and return in three seconds, everything changes. You explore freely. You try things. You discover patterns you'd never have found through careful planning. This exploration led directly to better prompts, which led to better results, which led to discovering even better patterns. The feedback loop between cheap tools and rapid iteration is very real.

Unlocking Complex Multi-User E2E Test Flows

One area specifically that unlocked for me was writing complex multi-user e2e test flows.

Because I was working on an app that was a marketplace - there are three specific roles that needed to interact with each other: an admin (the application admin), a provider (the sell side of the app marketplace), and a customer (who buys from the provider).

The CTO wanted testing that maintained certain flows being tested - the provider adds a product with a discount, the customer loads the product, then the provider changes details about the product, and those details are reflected in the customer session, and then the customer checks out - as one simple example.

Browser Isolation Is Essential for Multi-Role Workflows

Multi-role test scenarios—where you need to verify interactions between different user types—are extremely difficult to author manually. Managing multiple authenticated sessions, preventing cookie leakage, and tracking which context you're in creates substantial cognitive overhead.

Browser isolation is essential for multi-role workflows. Incognito browser contexts isolate storage/auth per role, while isolated worlds keep analysis code invisible to app scripts. Together, they reduce session leakage and flakiness in complex flows. Each role gets a dedicated browser context with proper session isolation, and the LLM just references roles by name without managing any of the underlying complexity.

The impact is larger than the feature itself: multi-role e2e tests go from being tedious and error-prone to straightforward. Tests that would take an hour to write manually now take minutes with LLM assistance, and they're more reliable because the isolation is handled at the browser protocol level.

6.2 Limitations and Trade-offs

This isn't a test runner and deliberately doesn't try to be one. It's tooling for authoring tests during development. You still execute your tests normally in Playwright. Trying to replace Playwright would make zero sense given how much work has gone into making it robust across browsers and platforms. The tool stays focused on its specific job: helping coding agents generate better Playwright code.

Initially, the system required strong LLMs to work well—weaker models struggled to chain the tool calls correctly. The introduction of proper AI assistant configuration significantly improves this. Configuration files teach the 3-step workflow and composition patterns, which helps models use the tools more effectively. That said, very weak models may still struggle, and the approach works best with Claude Sonnet 4+ or similar capability levels. I'm curious whether future model improvements or better configuration design will further lower this bar.

Token efficiency matters most in high-iteration workflows. If you're writing tests once and never touching them again, the difference between 1,200 tokens and 50,000 tokens per query is less significant. The tool is optimized for iterative refinement during development—exploring structure, debugging selectors, refining patterns—where token costs compound quickly. Progressive configuration amplifies this benefit: the ~100 token standing cost means you can have dozens of conversations without the 12k token overhead of monolithic rules.

The system handles iframes through a lazy expansion approach: iframes appear as markers in the main frame snapshot, then are recursively expanded with frame-qualified refs (e.g., f1_e3 for elements in the first iframe). This enables interactions with embedded content like Stripe checkout forms while gracefully handling cross-origin restrictions.

Finally, the tool provides limited action primitives beyond DOM exploration. It's focused on helping generate selectors and understand page structure, not on replicating Playwright's full action API. For complex interactions like drag-and-drop or hover sequences, these are things that can be developed on top of what I have built so far.

6.3 Open Questions

I'm confident the core approach—incremental exploration with deterministic primitives—is sound. But several specific design decisions deserve more debate.

Primitive decomposition: Containers, patterns, and anchors emerged as the right primitives for DOM exploration through iterative testing, but I don't have strong theoretical justification for why these are optimal. They map cleanly to the structural questions LLMs need to answer ("what contains this?", "does this pattern repeat?", "what's unique here?"), and they compose well for progressive disclosure. But are these actually the fundamental primitives, or just the ones I discovered first?

Token budget allocation: The 3k token budget I converged on came from trial and error. Is this actually optimal? Would doubling the budget for descendants improve accuracy enough to justify the cost? Or would it just give the LLM more irrelevant context to sift through? I'd need to run hundreds of test cases with different budget allocations to answer this properly.

Generalization beyond browsers: The pattern of "give minimal structural facts, let the LLM compose them" should work for other domains, but I don't know if it actually does. Database schema exploration, filesystem navigation, API discovery—these all have similar structural properties. Has anyone built comparable tooling? Did the same design principles hold, or did domain-specific constraints force different trade-offs?

Teaching composition: Right now I rely on few-shot examples and hope the model generalizes. This works with Sonnet 4+, but it's not a robust training approach. Can we systematically improve how models learn to chain these primitives? Would synthetic data generation help—creating thousands of DOM structures with corresponding optimal selector paths? Or is this just a scaling problem that disappears as models get stronger?

The most valuable thing would be seeing this approach tested on significantly different applications—particularly legacy codebases without test IDs, heavily dynamic SPAs, or applications with hostile DOM structures. That's where the design decisions would really get stress-tested.

6.4 Feedback from Early Adopters

I'd love to hear about:

Your DOM structure: Legacy/modern? Test IDs? Accessibility patterns?
Success rate: What % of selectors work without manual refinement?
Token costs: Typical exploration sequences and total token usage
Breaking points: Where did the approach fail or require fallbacks?
Comparison: How does this compare to your current workflow?

Code and Architecture

The full implementation is available on GitHub at https://github.com/verdexhq/verdex-mcp. The codebase is TypeScript throughout, using Puppeteer for CDP access and the MCP SDK for the tool protocol. The bridge architecture is modular and designed to be extensible if you want to add new analysis capabilities.

This is an experimental project. I built it to solve specific pain points in my workflow around AI-assisted test authoring. It might be useful to others with similar workflows, or it might just be an interesting data point in the broader design space of LLM tooling. Either way, I learned a tremendous amount building it, and I'm curious what others think about these architectural trade-offs and design decisions.