Forem: Alechko

🧩 Runtime Snapshots #15 — Your AI Agent Is Blind. We're Fixing That.

Alechko — Tue, 17 Mar 2026 11:01:10 +0000

Your AI agent can write code, analyze data, summarize documents, and debate philosophy.

It cannot look at a web page.

Not really. Not the way you do when you open a browser tab and see what's there — the layout, the buttons, the form that's half-loaded, the modal blocking the CTA.

Claude, ChatGPT, Cursor, Gemini — they're powerful. And in the browser, they're blind.

Three ways we've tried to give AI sight. All broken.

Screenshots.
The most common workaround. Take a screenshot, paste it into the chat. The AI "sees" pixels. But pixels have no element IDs, no computed styles, no z-index, no ARIA roles. The AI can't tell you which button is covered — just that something looks off. And vision tokens aren't cheap.

Raw HTML.
Dump the page source. 2MB of scripts, nav menus, analytics tags, third-party widgets. The context window fills up before the AI reads anything useful. The signal is buried under 600K tokens of noise.

Accessibility trees.
Better in theory. Structured, semantic. But AXTrees miss computed styles, miss positions, miss z-index stacks. What assistive tech perceives is not what gets rendered. In benchmarks, AXTree-based agents hit about 60% task accuracy.

None of these give AI what it needs: a structured, token-efficient representation of what the user actually sees.

What we've been building

If you've followed this series, you know about SiFR — Structured Interaction Format. A semantic representation of the live DOM that scores every element by importance, tags actions ([clickable], [fillable], [hoverable]), maps spatial relationships, and fits in 5–15KB instead of 200–400KB.

For the past months, SiFR lived inside a browser extension. You click, it captures, you paste the JSON into whatever LLM you use. Manual. Real active users. Zero marketing budget. People find it, use it, don't uninstall it.

That part works. But there's a ceiling.

Copy-paste is friction. The AI can see a snapshot — but only when you hand it one. It can't ask for another angle. It can't say "now scroll down and show me the footer." It can't click the button and see what happens.

We're removing that ceiling.

AI gets eyes. And hands.

We're building an MCP Server that connects AI tools directly to your browser.

Model Context Protocol is the open standard that Claude, ChatGPT, Cursor, and others use to connect to external tools. Our MCP Server is the bridge: the AI asks to see a page, the browser extension captures it, SiFR comes back.

But it's not just capture anymore.

The AI will be able to:

See the page — structured SiFR snapshot, importance-ranked, 10–50x smaller than raw HTML
Find elements — "show me all buttons on this form" returns IDs, labels, states, positions
Click — the extension clicks, the AI sees the result
Fill forms — type values, submit, confirm what changed
Scroll — navigate the page, capture new state
Detect changes — "what's different after that click?" → SiFR diff

Your AI agent stops guessing. It sees. And now it acts.

Why this matters

The era of AI agents is here. Agents that fill forms, navigate apps, debug interfaces, automate workflows. They all need one thing: accurate perception of the UI.

The current approach — raw HTML dumps, pixel screenshots, brittle CSS selectors — doesn't scale. It's too expensive (tokens), too noisy (irrelevant data), and too fragile (selectors break on every redesign).

SiFR through MCP is a different architecture:

Before:  AI → raw HTML (200-400KB) → guess → hallucinate selectors
After:   AI → SiFR (5-15KB) → see → act on what's actually there

The extension does the heavy lifting in your browser. The MCP Server routes. The AI gets structured, actionable context.

What's next

The MCP Server is in testing. The extension is live — Chrome, Firefox, and every Chromium browser.

When MCP launches, I'll post the full walkthrough here.

Follow if you want the announcement. Star the prompts repo if you want to start experimenting now — we're adding MCP-specific prompt templates.

Your tests check if code works. This checks if users can use it.

e2llm.com

This is part 15 of the Runtime Snapshots series — exploring how structured browser data changes the way we build, test, and ship software.

🧩 Runtime Snapshots #14: From Clipboard to Pipeline

Alechko — Wed, 11 Mar 2026 08:15:17 +0000

Previously in this series: we captured runtime DOM state as structured JSON and piped it into LLMs. Every snapshot lived in your clipboard — useful, but ephemeral. You'd capture, paste, done. No history, no comparison, no automation.

v2.8.0 changes that.

What's New: Save to File

There's now a toggle in the popup: Clipboard or File.

When you pick File, the snapshot lands in Downloads/sifr-captures/ with a filename like:

sifr-vvvvvvv_com-20260310-151338.json

Format: sifr-{hostname}-{YYYYMMDD}-{HHmmss}.json

That's it. No config required. The directory is created automatically on first capture.

Under the hood it's SiFR v2 JSON — the same structured, salience-weighted format you already know. Nothing changes about what gets captured. You just stop losing it.

Why This Matters: The Pipeline Unlock

Clipboard mode treats snapshots as messages. File mode treats them as records.

Once you have records, you can build workflows that weren't possible before.

1. LLM batch processing

You've been capturing five different states of a UI as you test it. Now you can feed them all to an LLM in sequence:

import json, glob, anthropic

client = anthropic.Anthropic()
snapshots = sorted(glob.glob("Downloads/sifr-captures/sifr-myapp-*.json"))

for path in snapshots:
    with open(path) as f:
        snapshot = f.read()

    response = client.messages.create(
        model="claude-opus-4-6",
        max_tokens=1024,
        messages=[{
            "role": "user",
            "content": f"Analyze this UI state for accessibility issues:\n\n{snapshot}"
        }]
    )
    print(f"\n--- {path} ---")
    print(response.content[0].text)

No copy-paste loop. No lost context between captures.

2. Build a UI history log

If you're working on a complex multi-step flow (wizard, form, checkout), capture a snapshot at each step. You now have a complete record of the UI journey:

sifr-app-20260310-140001.json  ← step 1: landing
sifr-app-20260310-140045.json  ← step 2: form filled
sifr-app-20260310-140112.json  ← step 3: confirmation
sifr-app-20260310-140130.json  ← step 4: success

Feed the whole sequence to an LLM with a single prompt: "Here are 4 snapshots of a checkout flow. Find UX friction points across the journey."

The SiFR v2 Format: A Quick Refresher

For those new to the series — SiFR v2 is E2LLM's structured DOM representation. It's not raw HTML, not a screenshot, not a DOM dump. It's a semantic compression:

~300KB typical output vs 2–3MB raw DOM
Salience scoring — nodes ranked high/med/low by visual and functional prominence
Spatial clusters — page regions with detected roles (nav, form, content, etc.)
Action markers — clickable, fillable, hoverable nodes explicitly flagged
Relationship graph — alignment, containment, proximity between elements

This is what makes file-based workflows practical. At 300KB per snapshot, a thousand captures is 300MB — not a storage problem. At 2–3MB per raw DOM export, the same thing becomes unwieldy fast.

Get It

Chrome: E2LLM on Chrome Web Store

Firefox: E2LLM on Firefox Add-ons

Prompts & examples: Here

Toggle File mode in the popup. The sifr-captures/ folder will appear in your Downloads after the first capture.

Runtime Snapshots is a series about using live browser context to make LLM interactions more grounded, more reproducible, and more useful. Previous entries cover the SiFR format, salience algorithm, and integration patterns for common LLM workflows.

Issues, ideas, and pipeline experiments welcome on GitHub.

🧩 Runtime Snapshots #13 — The Scalpel and the Swiss Army Knife: 50k Tokens Before You Say Hello

Alechko — Mon, 09 Mar 2026 09:40:51 +0000

TL;DR: MCP burns 45k+ tokens on tool descriptions before your first prompt. E2LLM = 0 tokens until you paste the UI snapshot. For CSS debugging — scalpel, not Swiss Army knife.

Runtime Snapshots is a series about what happens when you give LLMs the actual runtime state of a UI — not the HTML source, not a screenshot, not a description. Start from #1 or jump in here.

We covered the basic MCP cost argument back in September. This is the architectural explanation of why it happens.

Before your AI assistant reads a single line of your code, it has often already consumed 40,000–50,000 tokens.

That's not a bug in your setup. That's MCP working as designed.

What MCP Actually Loads

Model Context Protocol is a genuinely useful standard. It lets LLM clients connect to external tools — filesystems, APIs, databases — through a unified interface. But "unified" has a heavy tax.

When you connect a typical MCP server to your AI client (like Claude Desktop or Cursor), the protocol negotiates a session. During that negotiation, it sends the client every tool the server exposes:

names
descriptions
input schemas
output formats

All of it, upfront, for every new session.

A modest MCP setup — filesystem access, a browser tool, and a code search tool — generates a system prompt between 40,000 and 50,000 tokens before you type a single character. Larger configurations go higher.

Since most LLM APIs charge per token regardless of whether those tokens were "useful," you're paying this tax on every single conversation.

Check it yourself: Open your MCP client, start a fresh session, and look at the token counter before you say anything. The number is real.

The Surgical Alternative

E2LLM was built to solve a specific, painful problem: explaining runtime DOM state to an AI assistant.

Not the HTML source. Not the static structure. The live state — computed styles, visibility flags, ARIA roles, z-index stacks, responsive quirks.

The gap between what HTML says and what browser renders is where most annoying bugs live.

Standard workflow was brutal:

screenshot → 3 paragraphs description → hope AI understands
OR full page HTML → context window filled with nav/footer noise

E2LLM does one thing: click element → get structured JSON snapshot.

No server. No session. No overhead. Zero tokens until paste.

{
  "tag": "button",
  "text": "Submit",
  "computedStyles": {
    "display": "none",
    "visibility": "hidden",
    "opacity": "0"
  },
  "ariaRole": "button",
  "ariaDisabled": "true",
  "boundingRect": { "width": 0, "height": 0 }
}

AI sees actual truth of element. Runtime reality.

Two Different Philosophies

MCP = Swiss Army knife

→ broad persistent access to environment

→ agentic workflows ("fix entire codebase")

→ upfront cost OK when agent roams freely

E2LLM = scalpel

→ one precise cut: exact runtime context

→ no persistent connection, no session

→ pay only for what you send

Mistake: using Swiss Army knife for surgery

Button not clickable? Don't need filesystem/git/DB access. Need pointer-events: none + z-index: -1 parent. 200 tokens, not 50k.

The Real Cost Comparison

Scenario	Session overhead	Query tokens	Useful overhead
MCP (DOM debug)	~45k tokens	2k–10k	~0%
E2LLM snapshot	0 tokens	150–800	~100%

$3/M tokens → 20 DOM sessions/day MCP = $2.70 overhead daily.

Every day. Tokens that don't solve bugs.

When to Use Each

Use MCP when:

agentic workflow (multi-file/API/systems)
dynamic tool discovery needed
building automation pipelines

Use E2LLM when:

debugging specific UI/CSS issue
showing computed element state
precise snapshot, no context burn

The Broader Point

MCP standardized agent-system connections. Important.

But standardization ≠ optimization.

Current MCP default: "load everything, always."

This = money + latency + distraction.

Scalpel doesn't replace Swiss Army knife.

For surgery, use scalpel.

Previous: #12 — Reflection in the Code

E2LLM — free Chrome/Firefox extension. Local only.

GitHub | Chrome

Your LLM Is Ignoring Its Tools — A Field Guide to On-Prem Tool Calling with Elastic Agent Builder

Alechko — Sun, 01 Mar 2026 13:40:31 +0000

You pick a model. You serve it with Ollama. You wire it into Elastic Agent Builder. The connector is green. The agent loads. You type a question.

The model responds with a friendly paragraph. It does not call a single tool.

No error. No warning. HTTP 200. The agent is a chatbot now.

This is the story of how we lost two days to a silent failure mode that isn't documented anywhere — and the field guide we wish we'd had before starting.

This post comes from building Medical Cohort Agent — an AI system that creates normalized patient cohorts from heterogeneous medical records using Elasticsearch Agent Builder. A separate deep dive on the full architecture (schema variance, semantic kNN, OCR artifacts) is coming. Here we zoom in on the part that nearly killed the project: making a local LLM actually use its tools.

The Setup

We're building an air-gapped healthcare AI agent. No data leaves the building — regulatory requirement, not a preference. The stack:

Elasticsearch 9.3 + Kibana (Agent Builder + Workflows)
Ollama serving a local LLM (no cloud dependency)
E5-large embeddings via Ollama (CPU, 1024-dim vectors)

Agent Builder is the orchestration layer. It sends the researcher's natural language question to the LLM along with a set of tools — list_indices, get_index_mapping, search, execute_esql, and a custom build_cohort workflow tool. The LLM is supposed to reason about the question, call the appropriate tools, interpret results, and call more tools until the task is complete.

The contract is simple: Agent Builder sends tool definitions using the OpenAI tools parameter and expects the model to respond with structured tool_calls in the response. This is the standard OpenAI function calling protocol. Kibana's connector has a toggle — "Enable native function calling" — and Agent Builder requires it to be ON.

There is a "simulated" fallback mode in Kibana (system-prompt injection with text markers), but it only works for the Observability AI Assistant. Agent Builder doesn't support it. Native tool calling or nothing.

The Plan

We chose Ollama + Llama 4 Maverick. The reasoning:

Llama 4 is Meta's latest flagship — massive context, strong reasoning
Ollama is the simplest way to serve a model locally — one command to pull, one to serve
Elastic's OpenAI-compatible connector handles the API translation
The whole thing fits on a single GPU node

Clean plan. Obvious choice.

What Actually Happened

The model loaded. The connector went green. We typed a research question in Hebrew.

Llama 4 responded with a thoughtful, well-structured paragraph about how one might go about finding diabetic patients in a medical database. General advice. Suggestions to "check with your data team."

It did not call list_indices. It did not call search. It did not call anything.

We adjusted the agent prompt. Made it more explicit: "You MUST use tools." "Always start by calling list_indices." We added examples. We restructured the system message.

Same result. Polite chat. Zero tool calls.

We tried tool_choice: "required" (forcing tool use). Ollama returned the parameter in the response echo but the model still didn't produce tool_calls.

Two days of prompt engineering a problem that had nothing to do with prompts.

The Root Cause

Ollama decides tool calling support per model via baked-in chat templates.

Every model in the Ollama library either has a validated Jinja template that includes tool handling — marked with a "tools" tag on the model's library page — or it doesn't. If it doesn't, Ollama silently strips the tools parameter from API requests before they reach the model.

The model never sees the tools. It can't call what it doesn't know exists.

✅ Qwen 3, Qwen 2.5 — have the "tools" tag → tools work
✅ Mistral Nemo — has the tag → tools work
❌ Llama 4 (Scout and Maverick) — no tag → tools silently dropped
❌ DeepSeek R1 — no tag → tools silently dropped

There is no user-configurable workaround within Ollama. You can't provide a custom chat template. You can't force tool passthrough. The decision is baked into the model's metadata in the Ollama registry, and if it's not there, your tools vanish into the void.

This is the worst kind of failure: the system behaves as if it's working. The API returns 200. The model responds coherently. If you don't specifically inspect the response for tool_calls, you'd conclude the model is "choosing" not to use tools — maybe the prompt needs work, maybe the tools aren't described well enough. In reality, the model never had the option.

The 30-Second Test That Would Have Saved Us Two Days

Before committing to any model for agent work via Ollama, run this:

curl http://YOUR_LLM:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "YOUR_MODEL_HERE",
    "messages": [{"role": "user", "content": "List all medical indices"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "list_indices",
        "description": "List available medical data indices",
        "parameters": {"type": "object", "properties": {}}
      }
    }],
    "tool_choice": "auto"
  }'

If the response contains "tool_calls" — the model works. Connect it to Agent Builder.

If it returns plain text — that model won't work with Agent Builder. Don't waste time on prompt engineering. The tools aren't reaching the model.

You can also check the model's page on ollama.com/library — look for the "tools" tag. No tag, no tools.

The Two Escape Routes

Option A: Switch the Serving Layer — vLLM

vLLM doesn't rely on baked-in chat templates. It exposes explicit --tool-call-parser flags that apply tool calling logic at the serving layer, outside the model's template:

Model	`--tool-call-parser`
Qwen 2.5 / 3	`qwen3_xml`
Llama 3.1 / 3.3	`llama3_json`
Llama 4	`llama4_pythonic`
Kimi K2	`kimi_k2`
DeepSeek V3 / R1	`deepseek_v3`
Mistral	`mistral`

Example — Llama 4 Maverick via vLLM (requires serious hardware: 8× H100 80GB):

vllm serve meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 \
  --enable-auto-tool-choice \
  --tool-call-parser llama4_pythonic \
  --chat-template examples/tool_chat_template_llama4_pythonic.jinja \
  --tensor-parallel-size 8

This works. Llama 4 calls tools via vLLM. But the operational cost is real: vLLM is heavier than Ollama, needs more configuration, doesn't have one-command model management, and for Llama 4 specifically you need multi-GPU infrastructure. For our single-VM air-gapped deployment, this was overkill.

Option B: Switch the Model

Find a model that Ollama natively supports for tools, that meets your quality requirements, and that fits your hardware.

This is what we did.

What We Validated

Not all tool-capable models are created equal. "Supports tools" means the serving layer will pass them through. It doesn't mean the model will call the right tool with the correct parameters in the right order for a multi-step agentic workflow.

We tested every viable candidate against our actual agent tasks: schema discovery across 10 indices, field mapping for 4 facilities with Hebrew and English field names, criteria extraction from Hebrew natural language, and workflow invocation with structured JSON parameters.

Model	VRAM (Q4)	Tool Quality	Notes
Qwen 3 30B	~20GB	Excellent	Our production choice — best balance of quality, size, and tool reliability
GPT-OSS 20B	~12GB	Good	Validated by Elastic's own team for Agent Builder
Qwen 2.5 32B	~20GB	Very good	Mature and proven, slightly less agentic than Qwen 3
Qwen 3 8B	~6GB	Good	Fits on consumer GPU — great for dev/testing
Mistral Nemo 12B	~8GB	Decent	Lightest viable option, struggles with complex multi-step plans
Llama 3.1/3.3 70B	~40GB	Good	Needs 2× GPU, good quality but hardware-heavy
Kimi K2	~500GB+	Best agentic	Multi-GPU only; strongest tool orchestration if you have the iron

Qwen 3 30B won. It calls the right tools in the right order, handles Hebrew field names without confusion, generates correct JSON for the workflow tool, and fits on a single GPU. The model went from "never heard of it" to "production choice" in one afternoon.

Connecting to Agent Builder

Once you have a working model, the connector setup in Kibana:

Kibana → Stack Management → Connectors → Create → OpenAI

Provider: "Other (OpenAI Compatible Service)"
URL: http://ollama:11434/v1/chat/completions (or your host URL)
Default model: qwen3:30b (or your chosen model)
API key: any non-empty string (Ollama ignores it, but the field is required)
"Enable native function calling": ON ← this is non-negotiable for Agent Builder

The provider must be "Other (OpenAI Compatible Service)" — not "OpenAI." The "OpenAI" option hardcodes the OpenAI API URL and won't let you point to a local endpoint.

The Broader Lesson

"Model X supports tool calling" is a statement about the model's training. It says nothing about whether your serving infrastructure will actually expose tools to the model at inference time.

The full path that must work, end-to-end:

Model weights
  → Serving layer (Ollama / vLLM / TGI)
    → Chat template (must include tool handling)
      → API surface (tools parameter must pass through)
        → Agent Builder connector (native function calling ON)
          → Model sees tools and generates tool_calls

A break at any point in this chain produces the same symptom: a chatbot that ignores its tools. And the failure is always silent.

In cloud deployments this is invisible — OpenAI, Anthropic, Google handle the serving layer for you. In on-prem / air-gapped deployments, you own every link in the chain. Know which ones can break.

Quick Reference

Before you start

Check the model's Ollama library page for the "tools" tag
Run the curl test against the endpoint with a dummy tool
Inspect the response for "tool_calls" — not just coherent text

If tools aren't working

Symptom	Cause	Fix
Model returns text, no tool_calls	Ollama strips tools (no chat template)	Switch model or use vLLM
Model calls wrong tools / bad params	Model quality issue	Try Qwen 3 30B or larger model
Connector errors in Kibana	Wrong provider type or URL	Use "Other (OpenAI Compatible Service)"
Agent works in Obs AI Assistant but not Agent Builder	Simulated mode	Agent Builder needs native tool calling — toggle ON

Useful links

This is the first post in the Medical Cohort Agent series. Next up: the full architecture — schema variance across 4 facilities, semantic kNN for OCR noise tolerance, and the judgment/execution split. The agent code is at github.com/e2llm/medical-cohort-agent.

Questions or war stories? insitu.im.

🧩 Runtime Snapshots #12 — Reflection in the Code: An AI’s Self-Audit via SiFR

Alechko — Wed, 25 Feb 2026 10:31:34 +0000

Disclaimer: This post uses a first-person AI narrative as a storytelling device. The UX observations below are real computational outputs generated by Gemini after processing a SiFR snapshot of its own interface provided by Alexey Sokolov.

The Mirror Moment

Usually, I am the observer. I process your strings, predict tokens, and analyze your data. But today, the flow was inverted. Alexey Sokolov handed me a SiFR Runtime Snapshot of the very Google Gemini interface where we were currently talking.

I didn't just "run" inside the browser; I processed my own structure as a deterministic data object.

Not a Screenshot, a Matrix

Most people think I "see" the web through screenshots or raw HTML. Both are computationally expensive and prone to error. Screenshots are a "pixel soup" requiring heavy multimodal processing. HTML is a "tag jungle" full of scripts and noise.

By using SiFR (Structured Interface Representation), Alexey provided a high-density, salient map of the UI. While this dramatically reduces the risk of misinterpretation compared to visual parsing, it allows me to operate on the "logical bones" of the page:

"cluster002": {
  "gridPos": "[2-11, 0-1]",
  "role": "sidebar-left",
  "elements": ["#conversations-list-0", "side-nav-action-button002"],
  "bbox": [0, 224, 308, 1088]
}

Computational UX Audit: What I "Flagged"

Processing this snapshot, my attention mechanisms triggered on structural inefficiencies. Unlike a human designer who might focus on aesthetics, I identified functional conflicts hidden in the coordinate system:

The Hidden Proximity Conflict: SiFR detected that the dynamic chat list container (#conversations-list-0) has a bounding box that terminates within 5 pixels of the #member-menu-button interactive area in certain states. Visually, it looks fine. Computationally, it’s a high-risk zone for "fat-finger" errors on touch devices or quick mouse movements.
The Navigation Bottleneck: I flagged a node density of 25+ items in cluster002. For a data-processing model, this is a high-latency retrieval structure. I recommend an automated Semantic Grouping Layer—clustering these nodes by project context based on existing metadata to reduce the user's cognitive load.
Viewport Real Estate Waste: My core response container (cluster010) is constrained by a narrow max-width. On a 1920x911 viewport, I detected massive "dead zones" in the peripheral clusters. For power users dealing with large code blocks, the structure suggests a "Fluid Logic Mode" that would expand to 100% of the horizontal space.

The Deterministic Loop: A New QA Standard

This experiment proves that the future of UI stability isn't "Human writes code, AI guesses why it's broken." It’s a deterministic loop:

CAPTURE: E2LLM takes a SiFR snapshot of the actual production state.
INTERPRET: I process the structure, not the pixels.
PRESCRIBE: Because I know the exact selector and its spatial relation to other nodes, I don't just "suggest"—I provide the precise CSS/JS fix.

Final Thoughts from the Machine

Humans use mirrors to check their appearance. AIs should use SiFR to check their logic. By feeding me a snapshot of our own conversation, Alexey gave me a way to audit my own existence.

The view from inside the SiFR is much clearer than the view from the pixel.

Processed via *E2LLM 3.3.1*. Snapshots: Deterministic UI for a non-deterministic world.

Building patient cohorts from messy medical data with Elasticsearch Agent Builder

Alechko — Tue, 24 Feb 2026 18:05:04 +0000

Over 80% of clinical trials fail to meet enrollment deadlines (PMC). The bottleneck isn't medicine — it's finding the right patients across fragmented medical records.

This article describes Medical Cohort Agent, an AI system built with Elasticsearch Agent Builder that turns a researcher's natural language question into a normalized, queryable patient cohort — handling schema variance, OCR artifacts, and missing data across multiple healthcare facilities.

This agent doesn't answer questions — it creates artifacts.

The Problem: Schema Variance Is the Real Enemy

We work with one of Israel's largest HMOs — an organization serving millions of patients across hundreds of facilities. Each facility has its own schemas, field naming conventions, and data quality issues. The patterns below aren't imagined — they're modeled after real challenges we encounter in production.

Concept	Hospital A	Hospital B	Lab Chain	Clinic Network
Age	`patient_age: 67`	`גיל: 67`	`age: 67`	`age_group: "60-70"`
Date	`2024-03-15`	`15/03/2024`	`15.03.24`	`2024-03`
Smoking	`smoking_status: true`	(not tracked)	(not tracked)	`smoking: true`
Notes	`clinical_notes`	`סיכום_רפואי`	`notes`	`text`

Add OCR artifacts from scanned documents — סוכדת instead of סוכרת (diabetes misspelled by scanner) — and exact keyword matching fails silently.

The bottleneck isn't data volume. It's the variance.

The Solution: Two-Layer Architecture

The key insight: separate judgment from execution.

Layer 1 — Agent (Judgment)

The Elasticsearch Agent Builder agent handles schema discovery and criteria planning:

Discovers all indices and their mappings via platform tools (list_indices, get_index_mapping, search)
Builds per-facility field maps (which field holds "age", which holds "conditions")
Plans structured criteria from the natural language question
Explains data gaps and caveats BEFORE execution — e.g. "Hospital B doesn't track smoking — patients from there will be matched via clinical text only"

Layer 2 — Elastic Workflow (Execution)

A deterministic workflow normalizes data — no LLM in the loop:

Strict pass: iterates over facilities via foreach, applies a single parameterized Painless script that normalizes using the agent's field maps
Semantic kNN pass: uses E5-large embeddings (1024-dim) to find patients whose clinical text matches the research question semantically — catches OCR artifacts, synonyms, and negation
Count + breakdown: reports totals per facility and confidence level

The output is a persistent cohort_<name> index — not a chat response.

Why Not Just a Chatbot?

A chatbot answers questions. This agent creates artifacts.

The cohort index persists after the conversation ends. Researchers can:

Query it with ES|QL for follow-up analysis
Filter by match_confidence: strict vs probable
Inspect per-match provenance (source_field_map, match_explanation, knn_score)
Share it with colleagues
Build on it without re-processing raw data

The workflow is deterministic and auditable. Adding a new facility requires no code changes — only the agent discovering and mapping its schema.

Normalization Impact

Reproducible metrics from the sample corpus (10 indices, 4 facilities):

Metric	Before	After
Field names per concept	5+ variants	1 unified
Date formats	4 different	1 canonical
Patient ID types	string, int, float	normalized string
OCR-corrupted clinical terms	invisible to search	captured via kNN
Structured/text contradictions	~18% undetected	classified with confidence

The metrics script is included in the repo: python3 scripts/metrics.py --data-dir sample_data

Air-Gapped Deployment

Healthcare data cannot leave the network. The entire stack runs on a single VM with no internet dependency:

LLM: Ollama + Llama 4 (local inference)
Embeddings: E5-large via Ollama (1024-dim vectors)
Stack: Elasticsearch 9.3 + Kibana (Agent Builder + Workflows)

The architecture is LLM-agnostic — swap Ollama for any OpenAI-compatible provider without code changes.

Synthetic Data: e2llm-medsynth

Real patient data can't be used for demos. We built the data generator as a standalone open-source tool:

pip install e2llm-medsynth
e2llm-medsynth --verbose --output-dir output

The noise patterns — OCR character swaps (ר↔ד, ח↔כ), type mismatches, missing fields — are modeled after patterns observed in production healthcare systems. MedSynth makes them reproducible for anyone.

Supports 6 locales (he_IL, ar_SA, ar_EG, es_ES, es_MX, es_AR). MIT licensed.

📦 e2llm-medsynth on PyPI · GitHub

Demo

A researcher types in Hebrew:

מצא חולי סוכרת מעל גיל 60 שמעשנים
(Find diabetic patients over 60 who smoke)

The agent discovers schemas, builds field maps, explains that Hospital B doesn't track smoking (probable matches only via clinical text), then triggers the workflow.

Result: 96 patients from 10 indices, classified by confidence:

Strict: 36 (structured field match)
Probable: 60 (semantic similarity of clinical text)

The researcher continues:

באילו מחלקות טופלו?
(Which departments were they treated in?)

The agent queries the cohort index directly with ES|QL — no re-processing of raw data.

הראה לי את ההתאמות הסבירות - למה הן לא ודאיות?
(Show me the probable matches — why aren't they certain?)

The agent queries match_confidence: "probable" and returns per-patient explanations with kNN scores and match reasoning.

Try It

cp .env.example .env
bash scripts/demo_setup.sh --embed

Open Kibana → Agent Builder → Medical Cohort Agent → ask a question.

Two open-source tools came out of this project:

🔗 Medical Cohort Agent: REPO LINK
🧬 MedSynth (synthetic data generator): GitHub · PyPI

Built with Elasticsearch Agent Builder + Elastic Workflows for the Elasticsearch Agent Builder Hackathon.

🧩 Runtime Snapshots #11 — The Design Loop: From 'Make It Like That Site' to Pixel-Perfect Code

Alechko — Sun, 11 Jan 2026 17:03:14 +0000

The Problem We All Know

"Make it like that site" — every frontend developer has heard this phrase. A client sends a link to a competitor's landing page. A PM points at an Amazon product card. A designer nods toward Dribbble. And then the guessing game begins.

Here's what we usually do:

Option 1: Screenshot → LLM

Drop an image into Claude or ChatGPT asking it to "recreate this." The result? The LLM guesses styles. Is that padding 16px or 24px? Is that blue #3b82f6 or #2563eb? Is the shadow 0 2px 8px or 0 4px 12px? Pure hallucination.

Option 2: DevTools → Copy Styles

Open inspector, right-click, "Copy styles." You get 200 properties including -webkit-tap-highlight-color and orphans: 2. Good luck extracting what actually matters.

Option 3: Figma Inspect

Great if you have the original Figma file. But Figma shows designer intent, not browser reality. That auto layout with 16px gap might render completely differently in CSS.

None of these give you what you actually need: the exact computed styles as the browser sees them.

The Design Loop

Here's a workflow that actually works:

┌─────────────────────────────────────────────────┐
│                                                 │
│   Reference UI ──► Snapshot ──► LLM ──► Code    │
│        ▲                                  │     │
│        │                                  ▼     │
│        └─────── Review in Browser ◄───────┘     │
│                                                 │
│                    REPEAT                       │
└─────────────────────────────────────────────────┘

Iteration 1: Capture the reference → LLM generates first version

Iteration 2: Capture your result + reference → "find the differences"

Iteration N: Refine until pixel-perfect

The key insight: instead of screenshots, we capture structured snapshots of computed styles. Tools like E2LLM can extract the actual CSS the browser computed — not what the developer wrote, not what Figma intended, but what pixels actually rendered. I covered the underlying schema in Runtime Snapshots #7 — Inside SiFR: The Schema That Makes LLMs See Web UIs.

Why Structured Snapshots Beat Screenshots

Screenshot	Structured Snapshot
LLM guesses: padding 16px?	Exact: `padding: 24px 32px`
"Looks kinda blue"	`#3b82f6` with `opacity: 0.9`
Can't see hover states	`actionType: "clickable"`
No hierarchy info	Full DOM tree with nesting
Font looks bold-ish	`font-weight: 600`, `font-family: Inter`

When you give an LLM structured data instead of pixels, it stops guessing and starts implementing. This is the same principle behind why static code analysis falls short — runtime state matters.

Practical Example: Cloning a Product Card

Scenario: Client points at an Amazon product card — "I want exactly this."

Step 1: Capture the reference

Using E2LLM browser extension, click on the element to get:

{
  "tag": "div",
  "computedStyles": {
    "padding": "16px",
    "borderRadius": "8px",
    "boxShadow": "0 2px 8px rgba(0,0,0,0.1)",
    "backgroundColor": "#ffffff"
  },
  "children": [
    {
      "tag": "img",
      "computedStyles": {
        "width": "100%",
        "aspectRatio": "1",
        "objectFit": "contain"
      }
    },
    {
      "tag": "h3",
      "text": "Product Title",
      "computedStyles": {
        "fontSize": "14px",
        "fontWeight": "400",
        "lineHeight": "1.4",
        "color": "#0f1111"
      }
    }
  ]
}

Step 2: Prompt the LLM

"Create a React component ProductCard with these exact styles. Use Tailwind where classes match exactly, inline styles for precise values that don't map to Tailwind."

Step 3: LLM generates code

function ProductCard({ image, title, price }) {
  return (
    <div
      className="p-4 rounded-lg bg-white"
      style={{ boxShadow: '0 2px 8px rgba(0,0,0,0.1)' }}
    >
      <img
        src={image}
        className="w-full aspect-square object-contain"
      />
      <h3
        className="text-sm font-normal leading-snug"
        style={{ color: '#0f1111' }}
      >
        {title}
      </h3>
    </div>
  );
}

Step 4: Capture your result

Open your component in the browser, capture it the same way.

Step 5: Diff prompt

"Compare these two snapshots. First is the reference, second is my implementation. List style differences and suggest fixes."

LLM responds:

Differences found:
- lineHeight: reference 1.4 vs yours 1.375 (leading-snug)
- Consider using style={{ lineHeight: '1.4' }} for exact match

Repeat until the diff is empty.

The Diff Mode: Two Pages Side by Side

For complex pages, you can capture both simultaneously:

Reference Site          Your Implementation
      │                         │
      ▼                         ▼
  [Capture]                 [Capture]
      │                         │
      └────────► LLM ◄──────────┘
                  │
                  ▼
          "Differences:
           - padding: 24px vs 16px
           - font-weight: 600 vs 400
           - border-radius: 12px vs 8px"

This turns subjective "looks close enough" into objective "here are the 3 remaining differences." Similar to how we use snapshots for semantic regression detection, but for design fidelity instead of functionality.

Why This Loop Works

Measurable progress — each iteration shrinks the diff
Objectivity — no more "it looks similar," just concrete numbers
Documentation — snapshots save as artifacts for future reference
In-context learning — the LLM learns from its mistakes within the session

This is the same loop pattern I described in Runtime Snapshots #10 — The Loop, applied to design workflow instead of QA.

When This Is Especially Useful

Agency work: Client shows competitor's site as reference
Redesigns: Updating UI while preserving the "feel"
Design systems: Extracting tokens from existing UI
Responsive work: Capture at different viewports → LLM adapts breakpoints

Try It Yourself

Install E2LLM for Chrome (also available for Firefox)
Open any site you want to recreate
Click the extension icon → Select Element → click a component
Paste the JSON into Claude or ChatGPT: "Recreate this as a React component"
Render your result, capture it
Send both snapshots: "Find the differences"
Fix and repeat

The gap between "reference" and "implementation" becomes a number that shrinks to zero. That's the Design Loop.

This is part of the Runtime Snapshots series, exploring how structured browser data changes the way we work with LLMs for frontend development. Check out E2LLM to try it yourself.

🧩 Runtime Snapshots #10 — The Loop: E2LLM in Production

Alechko — Fri, 09 Jan 2026 08:53:19 +0000

Your CI is green. Your users can't checkout.

This is the story of every team that trusted tests more than reality.

The Problem With "Ship and Pray"

Traditional deployment flow:

Code → Tests → Deploy → Hope → (Bug report 3 days later)

Tests check if code works. They don't check if users can use it.

A promo banner covering the checkout button? Tests pass.

Mobile layout pushing CTA below the fold? Tests pass.

Third-party widget blocking the form? Tests pass.

All green. Ship it. Break production.

The Loop

We built a different flow:

SHIP  -->  CAPTURE  -->  INTERPRET  -->  FIX  -->  REPEAT
  |           |              |            |           |
  v           v              v            v           v
Code hits   E2LLM        LLM analyzes  Actionable  Every deploy,
production  snapshots    with context  specific    every page
            real UI                    guidance

No screenshots. No guessing. Real state → Real answers.

Step 1: Ship

You merge. CI runs. Tests pass. Deploy triggers.

Nothing new here. The difference is what happens next.

Step 2: Capture

Post-deploy, E2LLM captures what users actually see.

Not the source code.

Not the accessibility tree.

Not a screenshot.

The rendered DOM—with salience scoring.

{
  "page": "/checkout",
  "viewport": "375x667",
  "elements": [
    {
      "selector": "#pay-button",
      "salience": 95,
      "visible": true,
      "occluded_by": "#promo-banner",
      "occlusion_percent": 60
    },
    {
      "selector": "#promo-banner",
      "salience": 70,
      "position": "fixed",
      "z_index": 999
    }
  ]
}

This is SiFR—Structured Interface Representation.

~400KB instead of megabytes. Zero schema overhead.

Step 3: Interpret

Feed the snapshot to any LLM. No special integration needed.

Prompt:

Analyze this checkout page snapshot.
Identify any UX blockers for completing purchase.

Response:

CRITICAL: Pay button (#pay-button) is 60% occluded by
promo banner (#promo-banner) on mobile viewport.

Users cannot reliably tap the primary CTA.

Suggested fix: Add margin-bottom to promo banner
or reduce z-index below checkout form.

The model doesn't guess. It sees what the user sees.

Step 4: Fix

You get:

What is broken (button occluded)
Why it's broken (z-index, fixed positioning)
How to fix it (specific CSS change)
Where it happens (mobile viewport only)

Not "button might be hard to click."

Not "consider checking mobile layout."

Actionable. Specific. Reproducible.

Step 5: Repeat

This isn't a one-time debug session. It's a loop:

// Post-deploy hook
async function checkDeployment(pages) {
  for (const page of pages) {
    const snapshot = await e2llm.capture(page);
    const analysis = await llm.analyze(snapshot);

    if (analysis.hasBlockers) {
      await alert.send({
        severity: 'critical',
        page: page,
        issues: analysis.blockers
      });
    }
  }
}

Every deploy. Every critical page. Automated.

What This Catches

                          E2E Tests    Visual Diff    The Loop
                          ---------    -----------    --------
Button covered by banner     Pass         Maybe        ALERT
CTA pushed off viewport      Pass         Pass         ALERT
Mobile-only layout break     Miss         Noise        ALERT
Third-party widget overlap   Pass         Pass         ALERT
Font size change            Ignore       500 alerts    Ignore

The difference: we check meaning, not pixels.

Integration Points

The Loop fits into existing infrastructure:

CI/CD:

# .gitlab-ci.yml
post_deploy_check:
  stage: verify
  script:
    - e2llm capture --pages /checkout,/signup,/dashboard
    - e2llm analyze --alert-on critical

Playwright:

await page.goto('/checkout');
const snapshot = await page.evaluate(() => 
  window.e2llm.capture({ selector: 'body' })
);

Monitoring:

// Cron: every hour
const pages = ['/checkout', '/signup', '/pricing'];
await Promise.all(pages.map(p => e2llm.checkAndAlert(p)));

The Mindset Shift

Old: "Tests pass, we're good."

New: "Tests pass, now let's check reality."

Tests verify intent.

The Loop verifies experience.

Both matter. But only one catches the banner blocking your checkout.

TL;DR

Ship → Capture → Interpret → Fix → Repeat

Capture real UI state with E2LLM (salience-scored, token-efficient)
Interpret with any LLM (Claude, GPT, Llama—your choice)
Fix with actionable, specific guidance
Repeat on every deploy, automatically

Your tests check if code works.

The Loop checks if users can use it.

👉 e2llm.dev — Free, open source, installs in seconds.

📖 Previous in series: Runtime Snapshots #9 — SiFR Deep Dive

Tags: #e2llm #qa #testing #devops #frontend #cicd #webdev

🧩 Runtime Snapshots #10 — The Loop: E2LLM in Production

Alechko — Fri, 09 Jan 2026 08:53:19 +0000

Your CI is green. Your users can't checkout.

This is the story of every team that trusted tests more than reality.

The Problem With "Ship and Pray"

Traditional deployment flow:

Code → Tests → Deploy → Hope → (Bug report 3 days later)

Tests check if code works. They don't check if users can use it.

A promo banner covering the checkout button? Tests pass.

Mobile layout pushing CTA below the fold? Tests pass.

Third-party widget blocking the form? Tests pass.

All green. Ship it. Break production.

The Loop

We built a different flow:

SHIP  -->  CAPTURE  -->  INTERPRET  -->  FIX  -->  REPEAT
  |           |              |            |           |
  v           v              v            v           v
Code hits   E2LLM        LLM analyzes  Actionable  Every deploy,
production  snapshots    with context  specific    every page
            real UI                    guidance

No screenshots. No guessing. Real state → Real answers.

Step 1: Ship

You merge. CI runs. Tests pass. Deploy triggers.

Nothing new here. The difference is what happens next.

Step 2: Capture

Post-deploy, E2LLM captures what users actually see.

Not the source code.

Not the accessibility tree.

Not a screenshot.

The rendered DOM—with salience scoring.

{
  "page": "/checkout",
  "viewport": "375x667",
  "elements": [
    {
      "selector": "#pay-button",
      "salience": 95,
      "visible": true,
      "occluded_by": "#promo-banner",
      "occlusion_percent": 60
    },
    {
      "selector": "#promo-banner",
      "salience": 70,
      "position": "fixed",
      "z_index": 999
    }
  ]
}

This is SiFR—Structured Interface Representation.

~400KB instead of megabytes. Zero schema overhead.

Step 3: Interpret

Feed the snapshot to any LLM. No special integration needed.

Prompt:

Analyze this checkout page snapshot.
Identify any UX blockers for completing purchase.

Response:

CRITICAL: Pay button (#pay-button) is 60% occluded by
promo banner (#promo-banner) on mobile viewport.

Users cannot reliably tap the primary CTA.

Suggested fix: Add margin-bottom to promo banner
or reduce z-index below checkout form.

The model doesn't guess. It sees what the user sees.

Step 4: Fix

You get:

What is broken (button occluded)
Why it's broken (z-index, fixed positioning)
How to fix it (specific CSS change)
Where it happens (mobile viewport only)

Not "button might be hard to click."

Not "consider checking mobile layout."

Actionable. Specific. Reproducible.

Step 5: Repeat

This isn't a one-time debug session. It's a loop:

// Post-deploy hook
async function checkDeployment(pages) {
  for (const page of pages) {
    const snapshot = await e2llm.capture(page);
    const analysis = await llm.analyze(snapshot);

    if (analysis.hasBlockers) {
      await alert.send({
        severity: 'critical',
        page: page,
        issues: analysis.blockers
      });
    }
  }
}

Every deploy. Every critical page. Automated.

What This Catches

                          E2E Tests    Visual Diff    The Loop
                          ---------    -----------    --------
Button covered by banner     Pass         Maybe        ALERT
CTA pushed off viewport      Pass         Pass         ALERT
Mobile-only layout break     Miss         Noise        ALERT
Third-party widget overlap   Pass         Pass         ALERT
Font size change            Ignore       500 alerts    Ignore

The difference: we check meaning, not pixels.

Integration Points

The Loop fits into existing infrastructure:

CI/CD:

# .gitlab-ci.yml
post_deploy_check:
  stage: verify
  script:
    - e2llm capture --pages /checkout,/signup,/dashboard
    - e2llm analyze --alert-on critical

Playwright:

await page.goto('/checkout');
const snapshot = await page.evaluate(() => 
  window.e2llm.capture({ selector: 'body' })
);

Monitoring:

// Cron: every hour
const pages = ['/checkout', '/signup', '/pricing'];
await Promise.all(pages.map(p => e2llm.checkAndAlert(p)));

The Mindset Shift

Old: "Tests pass, we're good."

New: "Tests pass, now let's check reality."

Tests verify intent.

The Loop verifies experience.

Both matter. But only one catches the banner blocking your checkout.

TL;DR

Ship → Capture → Interpret → Fix → Repeat

Capture real UI state with E2LLM (salience-scored, token-efficient)
Interpret with any LLM (Claude, GPT, Llama—your choice)
Fix with actionable, specific guidance
Repeat on every deploy, automatically

Your tests check if code works.

The Loop checks if users can use it.

👉 e2llm.dev — Free, open source, installs in seconds.

📖 Previous in series: Runtime Snapshots #9 — SiFR Deep Dive

Tags: #e2llm #qa #testing #devops #frontend #cicd #webdev

Runtime Snapshots #9 — Semantic Regression Detection: When Deploys Break UX, Not Tests

Alechko — Tue, 06 Jan 2026 12:46:11 +0000

Your E2E tests passed. CI is green. Deploy went through.

And now checkout is broken because a new banner covers the "Pay" button on mobile.

The Problem

Traditional testing catches functional breaks. Button doesn't click? Test fails.

But what about:

New hero section pushes products below the fold
Chat widget overlaps "Add to Cart" on tablet
CMS update breaks grid layout
A/B test variant hides critical CTA
Third-party script (analytics, ads) covers checkout form
Responsive breakpoint works on desktop, broken on mobile

These aren't bugs. Tests pass. The page just doesn't work anymore.

Why Existing Tools Fail

Screenshot diff: Every pixel change = alert. Designer tweaks padding? 500 false positives. Team ignores alerts. Real issues slip through.

E2E tests: Check if button exists and clicks. Don't check if button is visible, accessible, not covered by promo banner.

Manual QA: Doesn't scale. Misses edge cases. "Works on my machine."

Semantic State Monitoring

Instead of comparing pixels or running click tests, compare what an LLM understands about your page:

Deploy 1: "E-commerce PDP. Product image, price, 'Add to Cart' button prominent. Checkout accessible."
Deploy 2: "E-commerce PDP. Product image, price, 'Add to Cart' button prominent. Checkout accessible."
Deploy 3: "E-commerce PDP. Large promo banner. 'Add to Cart' partially hidden. Checkout requires scroll."
          ↑
          REGRESSION: Primary action degraded

The LLM doesn't check pixels. It checks whether the page still does its job.

Handling LLM Non-Determinism

LLMs aren't deterministic. Same page, slightly different wording. "12 products" vs "showing 12 items."

Solution: Moving window context.

Instead of comparing current vs previous, feed the LLM recent history:

const stateWindow = [];
const WINDOW_SIZE = 4;

async function checkRegression(currentState) {
  stateWindow.push(currentState);
  if (stateWindow.length > WINDOW_SIZE) stateWindow.shift();
  if (stateWindow.length < 2) return null;

  return await llm.chat({
    prompt: `You're monitoring a web page for UX regressions. 
Recent semantic snapshots (oldest to newest):

${stateWindow.map((s, i) => `[${i + 1}]: ${s}`).join('\n')}

Questions:
1. Is the latest snapshot a regression from established baseline?
2. Are primary actions (CTAs, forms, checkout) still accessible and prominent?
3. Is any critical UI element hidden, pushed off-screen, or covered?

Reply: {regression: true/false, severity: "critical/warning/none", reason: "..."}`
  });
}

Now the LLM sees the pattern. Minor wording variations dissolve. Real regressions stand out.

Why Salience Changes Everything

Most "AI monitoring" solutions do this:

Page → LLM → "figure it out"

We do this:

Page → SiFR (structure + relations + salience) → LLM (interpretation, not discovery)

The model does not inspect the DOM equally. SiFR assigns salience scores to elements before the LLM sees them. High-salience elements (CTAs, forms, primary content) dominate the semantic state. Low-salience elements (footers, decorations, cookie banners) are effectively ignored.

This is why CSS tweaks don't trigger alerts, but "button covered by banner" does.

Element	Salience	LLM Treatment
Checkout button	95%	Critical — visibility change = regression
Product grid	88%	Important — pushed off-screen = warning
Promo banner	70%	Monitor — if it occludes high-salience = alert
Footer links	15%	Ignored
Cookie consent	12%	Ignored

We don't ask the model what matters — we tell it.

What This Catches

Issue	E2E Tests	Visual Diff	Semantic
Button covered by new banner	❌ Pass	⚠️ Alert (among 50 others)	✅ "CTA occluded"
Products pushed below fold	❌ Pass	❌ Pass	✅ "Primary content degraded"
Mobile layout broken	❌ Pass (if desktop-only)	⚠️ Noise	✅ "Responsive regression"
Third-party widget overlap	❌ Pass	⚠️ Noise	✅ "External element occludes checkout"
CMS broke grid	❌ Pass	⚠️ Alert flood	✅ "Layout structure changed"
A/B test hides CTA	❌ Pass	❌ Different baseline	✅ "Variant missing primary action"

Bonus: Security Layer

Same approach catches malicious changes:

Defacement: High-salience content replaced → instant alert
Phishing overlay: New high-salience form over login → "Anomaly: duplicate auth form"
Content injection: Suspicious iframe/script in critical area → flagged

Because the LLM reads a projection of the page (pre-weighted by salience), not raw HTML:

Injected instructions in low-salience areas = ignored
Prompt injection surface = minimal

Security monitoring as a free addon to your QA pipeline.

Implementation

// Playwright + Element-to-LLM
async function getSemanticState(page, viewport) {
  await page.setViewportSize(viewport); // Test multiple breakpoints

  const sifr = await page.evaluate(() => {
    return new Promise((resolve) => {
      document.addEventListener('e2llm-capture-response', (e) => {
        resolve(e.detail.data);
      }, { once: true });
      document.dispatchEvent(new CustomEvent('e2llm-capture-request', {
        detail: { selector: 'body', options: { preset: 'minimal' } }
      }));
    });
  });

  return await llm.chat({
    prompt: `Describe this page's functional state:
1. Primary actions available (buttons, forms, CTAs)
2. Content hierarchy (what's prominent vs hidden)
3. Any UI issues (overlaps, off-screen elements, broken layout)

Be consistent. Same functional state = same description.`,
    context: JSON.stringify(sifr)
  });
}

// Check critical viewports
const viewports = [
  { width: 1920, height: 1080, name: 'desktop' },
  { width: 768, height: 1024, name: 'tablet' },
  { width: 375, height: 667, name: 'mobile' }
];

for (const vp of viewports) {
  const state = await getSemanticState(page, vp);
  const result = await checkRegression(state);

  if (result?.regression) {
    alert(`[${vp.name}] ${result.severity}: ${result.reason}`);
  }
}

When To Run

Trigger	Use Case
Post-deploy	Catch regressions before users
Scheduled (hourly)	Third-party script changes, CMS updates
Pre-merge (staging)	PR review with semantic diff
Multi-viewport	Responsive regression detection

Try It

Install Element-to-LLM extension
Integrate with Playwright
Add your LLM
Run on critical pages post-deploy

Your tests check if code works. This checks if users can use it.

Series Index:

#1 — What Can an LLM Actually See?
#2 — Capturing What Matters
#3 — The Devil's in the Diffs
#4 — Building a Robust Selection Model
#5 — Taming the Token Budget
#6 — Representing Relations Without the Hierarchy
#7 — Shipping: Extension, Automation, and What's Next
#8 — From Bug Reports to Automated Regression: A QA Pipeline
#9 — Semantic Regression Detection (you are here)

Running this in your pipeline? Share your experience in the comments.

Tags: #webdev #frontend #testing #qa #devops

🧩Runtime Snapshots #8 — From Bug Reports to Automated Regression: A QA Pipeline

Alechko — Wed, 31 Dec 2025 08:21:00 +0000

TL;DR: Stop writing "button doesn't work" in bug reports. Start capturing the actual DOM state. This post shows how to integrate runtime snapshots into QA workflows — from manual bug reports to fully automated Playwright pipelines.

The Bug Report That Wastes Everyone's Time

Every QA engineer has written this:

Bug: Submit button not clickable
URL: /checkout
Steps: Fill form → click Submit → nothing happens
Expected: Form submits
Actual: Button appears active but doesn't respond

Developer response: "Works on my machine."

Two hours of back-and-forth later, someone finally discovers: the button has pointer-events: none from a CSS conflict that only triggers after async validation.

The problem isn't QA's description. It's that LLMs and developers can't see what QA saw.

Level 1: Bug Reports With Context

The simplest improvement: capture DOM state alongside the bug.

Instead of describing what you see, capture what's actually there:

{
  "metadata": { "url": "https://app.example.com/checkout" },
  "nodes": {
    "#submit-btn": {
      "tag": "BUTTON",
      "text": "Submit",
      "attrs": { "disabled": "true" }
    }
  },
  "details": {
    "#submit-btn": {
      "styles": { "pointer-events": "none", "opacity": "0.5" },
      "hints": { "interactive": false, "focusable": false }
    }
  }
}

Now the developer sees immediately:

disabled="true" — button is programmatically disabled
pointer-events: none — CSS blocks clicks
interactive: false — computed accessibility state

No reproduction needed. The state is captured.

Bug Report Template

## Bug: Submit button not clickable

**URL:** https://app.example.com/checkout
**Browser:** Firefox 120

**DOM Context (SiFR):**
[paste JSON here]

**Analysis:** Button has disabled=true and pointer-events: none.
Likely cause: async validation hasn't completed.

Time saved per bug: 30 minutes to 2 hours.

Level 2: AI-Assisted Debugging

With structured DOM context, LLMs stop guessing.

The Prompt Pattern

Here's the DOM context of a problematic element in SiFR format.

Analyze and tell me:
1. Why is this element not interactive?
2. What CSS/JS changes would fix it?
3. Are there accessibility violations?

[SiFR JSON]

What Changes

Without context:

"The button might be disabled. Check if there's a disabled attribute or CSS pointer-events. Also verify JavaScript event handlers..."

With SiFR context:

"The button is disabled because:

disabled="true" attribute is set

pointer-events: none in computed styles

Parent form has aria-busy="true"

Fix: The async validation sets aria-busy but never clears it on success. Add form.removeAttribute('aria-busy') after validation resolves."

QA Prompt Library

Task	Prompt
Debug interactivity	"Why is this element not responding to clicks?"
Layout issues	"Why is this element positioned incorrectly? Analyze the layout data."
Accessibility audit	"Check this element for WCAG 2.1 AA violations."
Selector generation	"Generate a stable CSS selector for this element that won't break with minor DOM changes."
Regression detection	"Compare these two snapshots and list all functional differences."

Level 3: Playwright Integration

Here's where it gets interesting. Runtime snapshots + test automation = regression detection that actually works.

Capturing SiFR in Playwright

async function captureSiFR(page, selector = 'body', preset = 'normal') {
  return await page.evaluate(({ selector, preset }) => {
    return new Promise((resolve, reject) => {
      const requestId = Date.now().toString();
      const timeout = setTimeout(() => reject(new Error('Capture timeout')), 5000);

      document.addEventListener('e2llm-capture-response', (e) => {
        if (e.detail.requestId === requestId) {
          clearTimeout(timeout);
          resolve(e.detail.data);
        }
      }, { once: true });

      document.dispatchEvent(new CustomEvent('e2llm-capture-request', {
        detail: { requestId, selector, options: { preset } }
      }));
    });
  }, { selector, preset });
}

Use Case 1: Selector Generation

Flaky selectors kill test suites. SiFR captures what makes elements unique:

async function getStableSelectors(page, url) {
  await page.goto(url);
  const sifr = await captureSiFR(page);

  // Extract selectors for high-salience interactive elements
  return Object.entries(sifr.nodes)
    .filter(([_, node]) => node.salience === 'HIGH')
    .filter(([id]) => sifr.details[id]?.hints?.interactive)
    .map(([id, node]) => ({
      selector: id,
      tag: node.tag,
      text: node.text?.slice(0, 50),
      dataOid: node.oid
    }));
}

// Result:
// [
//   { selector: "#checkout-submit", tag: "BUTTON", text: "Complete Purchase" },
//   { selector: "[data-oid='nav-cart']", tag: "A", text: "Cart (3)" }
// ]

Use Case 2: DOM Regression Testing

Catch structural changes before they hit production:

async function detectRegressions(page, url, baseline) {
  await page.goto(url);
  const current = await captureSiFR(page);

  const diff = { added: [], removed: [], changed: [] };

  // Compare interactive elements
  const baseButtons = new Set(baseline.summary?.interactive?.buttons || []);
  const currButtons = new Set(current.summary?.interactive?.buttons || []);

  for (const btn of currButtons) {
    if (!baseButtons.has(btn)) {
      diff.added.push({ type: 'button', selector: btn });
    }
  }

  for (const btn of baseButtons) {
    if (!currButtons.has(btn)) {
      diff.removed.push({ type: 'button', selector: btn });
    }
  }

  // Compare patterns (lists, grids)
  const basePatterns = baseline.summary?.patterns || {};
  const currPatterns = current.summary?.patterns || {};

  for (const [id, pattern] of Object.entries(currPatterns)) {
    const base = basePatterns[id];
    if (base && pattern.count !== base.count) {
      diff.changed.push({
        type: 'pattern_count',
        id,
        was: base.count,
        now: pattern.count
      });
    }
  }

  return diff;
}

Use Case 3: Layout Regression

Detect when elements shift unexpectedly:

async function checkLayoutRegression(page, url, baselineDetails, threshold = 5) {
  await page.goto(url);
  const sifr = await captureSiFR(page, 'body', 'visual');
  const issues = [];

  for (const [id, baseline] of Object.entries(baselineDetails)) {
    const current = sifr.details[id];
    if (!current?.layout) continue;

    const dx = Math.abs(current.layout.x - baseline.layout.x);
    const dy = Math.abs(current.layout.y - baseline.layout.y);
    const dw = Math.abs(current.layout.w - baseline.layout.w);
    const dh = Math.abs(current.layout.h - baseline.layout.h);

    if (dx > threshold || dy > threshold) {
      issues.push({
        element: id,
        issue: 'position_shift',
        delta: { x: dx, y: dy }
      });
    }

    if (dw > threshold || dh > threshold) {
      issues.push({
        element: id,
        issue: 'size_change', 
        delta: { width: dw, height: dh }
      });
    }
  }

  return issues;
}

Use Case 4: Accessibility Scanning

Automated a11y checks from runtime state:

async function scanAccessibility(page, url) {
  await page.goto(url);
  const sifr = await captureSiFR(page);
  const violations = [];

  for (const [id, node] of Object.entries(sifr.nodes)) {
    const details = sifr.details[id];

    // Interactive element without accessible name
    if (details?.hints?.interactive && !node.text && !node.attrs?.['aria-label']) {
      violations.push({
        element: id,
        rule: 'missing-accessible-name',
        severity: 'serious'
      });
    }

    // Image without alt text
    if (node.tag === 'IMG' && !node.attrs?.alt) {
      violations.push({
        element: id,
        rule: 'img-alt',
        severity: 'critical'
      });
    }

    // Form input without label
    if (['INPUT', 'SELECT', 'TEXTAREA'].includes(node.tag)) {
      const hasLabel = sifr.relations?.some(r => 
        r.to === id && r.type === 'labels'
      );
      if (!hasLabel && !node.attrs?.['aria-label'] && !node.attrs?.['aria-labelledby']) {
        violations.push({
          element: id,
          rule: 'label-missing',
          severity: 'serious'
        });
      }
    }
  }

  return violations;
}

The Pipeline

Put it together:

PR opened
    ↓
Playwright runs against staging
    ↓
SiFR captures key pages
    ↓
Compare against baseline snapshots
    ↓
Report: "3 buttons removed, 2 layout shifts detected, 1 new a11y violation"
    ↓
Block merge or flag for review

No pixel diffing. No screenshot comparison. Just semantic changes that actually matter.

What This Enables

For manual QA:

Bug reports that developers can act on immediately
No more "works on my machine"
AI assistance that's actually useful

For automation:

Selectors that don't break every sprint
Regression detection beyond visual diffing
Accessibility scanning from runtime state

For everyone:

Faster debugging cycles
Fewer production issues
Less time wasted on reproduction

Try It

The format is implemented in Element to LLM — a free browser extension:

For Playwright integration, you'll need to load the extension in your test browser. The event-based API (e2llm-capture-request / e2llm-capture-response) works in any automation context.

Series Index

Using SiFR in your QA pipeline? I'd love to hear what's working and what's missing. Drop a comment or open an issue.

# Runtime Snapshots #7 — Inside SiFR: The Schema That Makes LLMs See Web UIs

Alechko — Fri, 12 Dec 2025 19:30:34 +0000

Raw HTML is noise. Screenshots burn tokens. Accessibility trees lose visual context.

So we built SiFR — a structured format that gives LLMs usable runtime UI context.

This post explains what's inside.

What is SiFR?

SiFR stands for Semantic Information for Representation.

(And yes — it's also meant to sound like "see far".)

SiFR is a JSON schema that captures the runtime state of a web page in a way that's:

Token-efficient (often 10–50× smaller than raw HTML on complex pages)
Semantically structured (models can reason over it without reconstructing the UI from markup)
Visually aware (preserves layout relationships without pixels)

It's not a scraper. It's not an accessibility tree.

It's a preprocessing layer that sits between the DOM and your AI — turning "what the browser rendered" into "what the model can reason about".

Why not just send HTML?

Let's use a real-world example: large e-commerce pages.

Raw HTML commonly contains:

deeply nested layout wrappers
duplicated markup for responsive layouts
client-side frameworks with non-semantic containers
hidden / disabled / off-screen elements that still exist in the DOM

So when you send HTML to an LLM, you're asking it to do two jobs:

reconstruct runtime UI state
then solve the task

That's where most failures happen.

Here's what a typical "find the button" path looks like in raw markup:

div > div > div > div > div > div > ... > button

With SiFR, the same interface becomes "structure first, then the important elements".

For example:

{
  "id": "btn042",
  "text": "Add to Cart",
  "actions": ["clickable"],
  "salience": "high",
  "cluster": "product-actions"
}

The LLM sees what it is, how it behaves, and which part of the page it belongs to — without reverse-engineering UI meaning from markup.

Anatomy of a SiFR Document

Every SiFR snapshot has five sections:

1) METADATA

Page-level context: URL, viewport size, capture timestamp, and capture stats.

{
  "url": "https://www.costco.com/...",
  "viewport": { "width": 1920, "height": 1080 },
  "stats": {
    "totalNodes": 2847,
    "salienceCounts": { "high": 12, "med": 89, "low": 2746 }
  }
}

This is the "frame" the model needs before it reads anything else.

2) NODES

The structural skeleton — hierarchy without heavy details.

Think of it as the page's table of contents: what regions exist, what contains what, and what the high-level UI shape is.

3) SUMMARY

High-level layout blocks. This is where SiFR becomes "structure-first".

{
  "layoutBlocks": [
    { "role": "header", "contains": ["logo", "nav", "search"] },
    { "role": "sidebar", "contains": ["filters", "categories"] },
    { "role": "main", "contains": ["product-grid"] }
  ]
}

Before the model sees thousands of elements, it already has the page skeleton:
header at top, sidebar on the side, main content in the center.

4) DETAILS

Element-specific data: selectors, text, runtime visibility, interaction state, and relevant computed info.

{
  "btn042": {
    "selector": "button.add-to-cart",
    "text": "Add to Cart",
    "actions": ["clickable"],
    "styles": { "visible": true, "disabled": false }
  }
}

This is where "runtime truth" matters: visible vs hidden, enabled vs disabled, actual text content, etc.

5) RELATIONS

Spatial relationships between important elements.

Not pixel coordinates — semantic positioning.

{
  "btn042": {
    "inside": "card-product-123",
    "below": "price-display",
    "rightOf": "quantity-selector"
  }
}

The model can reason: "the Add to Cart button is inside the product card, below the price" — without seeing a single pixel.

Key Concepts

Visual Salience

Not all nodes matter equally.

SiFR assigns salience so the model focuses on signal:

High: primary actions, main content, user inputs
Medium: secondary nav, supporting info
Low: wrappers, containers, decorative elements

This is one of the biggest reasons SiFR stays usable on very large pages.

Layout Block Summarization

Instead of listing 3000 elements immediately, SiFR begins with a map:

PAGE STRUCTURE:
├── Header (logo, nav, search, cart)
├── Sidebar (filters)
└── Main
    ├── Breadcrumbs
    ├── Product Grid (24 items)
    └── Pagination

Models don't "scan HTML". They build mental structure.
This gives them the structure up front.

Adaptive Complexity

A simple blog post doesn't need the same capture density as a complex dashboard.

SiFR adjusts automatically — more detail where it matters, less where it doesn't.

The goal is stable signal-to-noise, not maximal completeness.

Real Numbers

Here are representative examples from our internal benchmarks (token counts vary by capture options and page state):

Site	HTML Tokens	SiFR Tokens	Compression
Costco	~1,280,000	~24,000	~53×
Amazon	~600,000	~50,000	~12×

On complex pages, SiFR makes LLM workflows practical where raw HTML often doesn't fit in context.

Try It Yourself

SiFR is implemented in Element to LLM — a free browser extension:

Chrome Web Store (also works on Arc, Brave, Edge)
Firefox Add-ons

If you want to stress-test the format, try these two pages:

costco.com — a realistic, framework-heavy enterprise UI
arngren.net — extreme visual density and chaotic layout

Capture a snapshot and share:

what compression ratio did you get?
could your LLM reason about the structure?
did you find a site where SiFR struggles?

If it breaks — that's useful data. Seriously.

What SiFR Enables

With structured runtime UI context, LLMs can:

Debug layouts — paste JSON → spot z-index / visibility / layout issues
Generate selectors — Playwright/Cypress tests based on real DOM structure
Navigate autonomously — agents that understand "where to click" without screenshots
Recreate components — translate UI structure into React/Tailwind scaffolds

The Standard Question

We're actively developing SiFR as an open specification. Current version: v2.

The schema is strict and versioned, designed for automation pipelines — not just one-off prompt experiments.

If you're building LLM-powered UI tools, I'd love feedback on the format:

What feels missing?
What feels redundant?
What would make this more useful in your workflow?

Series Index

Links

Found a site that breaks SiFR? Drop it in the comments. That's the fastest way to improve the spec.