Forem: David Aronchick

The Brownian Ratchet for Data

David Aronchick — Fri, 23 Jan 2026 00:35:31 +0000

Monday I wrote about how multiclaude and GasTown converged on nearly identical primitives for multi-agent orchestration. The key insight wasn't about prompts or models or agent personas. It was about infrastructure: CI is the ratchet. Let chaos reign. Multiple agents, overlapping work, duplicated effort, whatever. As long as you have a mechanism that only captures forward progress, you're good.

That phrase has been rattling around my head ever since. Because here's the thing: we have this for code. What's the equivalent for data?
The Missing Ratchet
CI transformed software development by giving us a one-way gate. Code either passes or it doesn't. No negotiations, no exceptions, no "we'll fix it later." The ratchet clicks forward, and it never clicks back.

Data has no such mechanism.

Oh, we have tools. We have great expectations (pun intended). We have dbt tests and schema validators and anomaly detectors. But none of them function as the arbiter-the single, uncompromising source of truth that says "this data is real now, and we're never going backward."

Instead, we have... hope? Process? Tickets that say "data quality issue" that sit in someone's backlog for three sprints while the dashboard keeps serving numbers that everyone knows are wrong but nobody can prove?
What Would a Data Ratchet Look Like?
Let's steal the multiclaude architecture and apply it to data:

Code Ratchet
Data Ratchet

CI tests
Schema validation + semantic checks

Passing tests
Data meeting quality thresholds

Merged PRs
Verified, immutable records

Git history
Data lineage with provenance

Multiple agents
Multiple validators / transformation paths

The principle is the same: chaos is fine, as long as we ratchet forward.

Multiple data sources can feed into your system. They can be messy, inconsistent, formatted in ways that make you question whether the upstream team has ever heard of ISO 8601. That's the Brownian motion: the random thermal energy of the real world generating data in a thousand incompatible ways.

But the ratchet, the verification layer, only lets validated data through. And once it's through, it's permanent. Immutable. Part of the record.
The Four Components
I think a data ratchet needs four things:

The Pawl: Schema as Contract JSON Schema (or Avro, or Protobuf, whatever floats your boat) isn't just documentation. It's the pawl that prevents backward motion. Data either conforms or it doesn't. No partial credit.

Here's what a schema-as-pawl actually looks like:
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "SensorReading",
"type": "object",
"required": ["device_id", "timestamp", "value", "unit"],
"properties": {
"device_id": {
"type": "string",
"pattern": "^[A-Z]{2}-[0-9]{6}$"
},
"timestamp": {
"type": "string",
"format": "date-time"
},
"value": {
"type": "number",
"minimum": -273.15
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit", "kelvin"]
}
},
"additionalProperties": false
}

Notice additionalProperties: false. That's the pawl. You can't sneak extra fields through. You can't send "value": "hot" instead of a number. You can't omit the timestamp and promise to fill it in later.

But here's where most systems fail: they treat schema validation as a warning, not a wall. "Schema violation detected, logging and continuing." That's not a ratchet. That's a turnstile with a broken lock.

A real data ratchet rejects non-conforming data. Full stop. The data can go back to the source, get transformed, get remediated, whatever it needs to do. But it doesn't get through until it conforms.

The Wheel: Idempotent Checkpoints In multiclaude, git worktrees give each agent isolation. If an agent's work fails, it fails in its own branch. The main branch (the ratcheted progress) stays untouched.

Data pipelines need the same thing: checkpoints that are idempotent and isolated. If a transformation fails, you can retry from the last checkpoint without corrupting the verified data downstream.
class CheckpointedPipeline:
def init(self, checkpoint_store: str):
self.checkpoint_store = checkpoint_store

def process_batch(self, batch_id: str, records: list[dict]) -&gt; str:
    # Check if we already processed this batch
    checkpoint = self.load_checkpoint(batch_id)
    if checkpoint and checkpoint[&quot;status&quot;] == &quot;completed&quot;:
        return checkpoint[&quot;output_path&quot;]  # Idempotent: return existing result

    # Process in isolation (write to temp location)
    temp_path = f&quot;{self.checkpoint_store}/pending/{batch_id}&quot;
    validated = []
    for record in records:
        if self.validate(record):
            validated.append(record)
        else:
            self.quarantine(record, batch_id)  # Don&apos;t lose it, just don&apos;t let it through

    self.write_records(temp_path, validated)

    # Only after success: commit the checkpoint
    final_path = f&quot;{self.checkpoint_store}/verified/{batch_id}&quot;
    self.atomic_move(temp_path, final_path)
    self.save_checkpoint(batch_id, {&quot;status&quot;: &quot;completed&quot;, &quot;output_path&quot;: final_path})

    return final_path

The key moves: write to a temp location first, only move to the verified path after success, and the checkpoint makes retries safe. If the process dies mid-batch, we start over. No partial state leaking into the verified dataset.

Most pipelines I've seen treat state as something that happens to them rather than something they manage. They're stateless in theory and stateful in practice, which is the worst of both worlds.

The Arbiter: Automated Verification with Teeth Here's the multiclaude rule that matters: agents are forbidden from weakening CI to make their work pass.

Translate that to data: no one can weaken the validation rules to make bad data pass. Not the data team, not the business stakeholder with a deadline, not the executive who needs the dashboard updated yesterday.

What does "CI for data" actually look like? Something like this:

data-ci.yaml

name: Data Quality Gate

on:
data_ingestion:
sources: ["sensor-feed", "partner-api", "user-uploads"]

jobs:
validate:
steps:
- name: Schema Validation
run: |
jsonschema --instance ${{ inputs.data_path }} \
--schema schemas/${{ inputs.source }}.json
fail_on_error: true # This is the ratchet. No exceptions.

  - name: Semantic Checks
    run: |
      python checks/semantic_validator.py \
        --data ${{ inputs.data_path }} \
        --rules rules/${{ inputs.source }}.yaml
    # Example rules:
    # - timestamp must be within last 24 hours
    # - device_id must exist in device registry
    # - value must be within 3 std devs of rolling mean

  - name: Lineage Recording
    if: success()
    run: |
      record-lineage \
        --input ${{ inputs.data_path }} \
        --schema-version ${{ inputs.schema_hash }} \
        --validator-version ${{ github.sha }} \
        --output verified/${{ inputs.batch_id }}

on_failure:
steps:
- name: Quarantine Bad Data
run: |
move-to-quarantine ${{ inputs.data_path }} \
--reason "${{ job.failure_reason }}"
- name: Alert Source System
run: |
notify-upstream ${{ inputs.source }} \
--batch ${{ inputs.batch_id }} \
--errors ${{ job.validation_errors }}

The critical bit is fail_on_error: true with no escape hatch. No continue-on-error. No "warn and proceed." The data either passes or it goes to quarantine.

This is culturally difficult. It requires the same organizational commitment that "we don't ship if tests fail" required for software teams. But it's the only way the ratchet works.

Reproducibility: The Secret Ingredient There's one more piece that makes the code ratchet work: reproducibility. When CI fails, you can reproduce the failure. When it passes, you can reproduce the pass. Same inputs, same outputs, every time.

Data systems are notoriously bad at this. The pipeline that worked yesterday fails today because someone changed an upstream schema. Or because the source system had a hiccup. Or because Mercury is in retrograde. (I've debugged all three. The Mercury one was actually a timezone issue in a system named "Mercury." I wish I was kidding.)

A real data ratchet needs what I'd call a "usability signature":
{
"batch_id": "2026-01-22-sensor-feed-042",
"verified_at": "2026-01-22T14:32:01Z",
"input_hash": "sha256:a1b2c3d4...",
"schema": {
"name": "SensorReading",
"version": "2.1.0",
"hash": "sha256:e5f6g7h8..."
},
"validators": {
"semantic_checks": "v1.4.2",
"anomaly_detector": "v0.9.1"
},
"result": {
"status": "passed",
"records_in": 10482,
"records_verified": 10479,
"records_quarantined": 3
},
"output_path": "verified/2026-01-22/sensor-feed-042.parquet",
"output_hash": "sha256:i9j0k1l2..."
}

This signature is an artifact, not just a log line. You can take this signature, grab the input data by its hash, run the exact versions of the validators, and you'll get the same result. If you can't do that, you don't have a ratchet. You have a coin flip.
The Uncomfortable Implication
Here's what this means in practice: a lot of data that's currently flowing through your systems wouldn't make it through a real ratchet.

That's not a bug. That's the point.

The Brownian ratchet works because it's uncompromising. The pawl doesn't care that you really need this data for a quarterly review. It doesn't care that the source system "usually" sends valid records. It doesn't care about your deadline.

CI transformed software quality not by being smart, but by being stubborn. It created a culture where "works on my machine" stopped being an excuse because there was an objective arbiter that didn't care about your machine.

Data needs the same stubbornness. The same willingness to say "no" and mean it.
What This Looks Like in Practice
I've been thinking about this in the context of what we're building at Expanso: intelligent data pipelines that can process data at the edge. The edge is where the Brownian motion is strongest. Sensors, devices, user inputs, all generating data in a thousand formats with a thousand failure modes.

The traditional answer is to centralize. Pull everything to a data lake, clean it up, validate it there. But that's expensive, slow, and loses context. By the time you've moved the data, you've lost the ability to remediate at the source.

What if the ratchet lived at the edge? Validation happens where data is generated. Non-conforming data gets rejected immediately, while there's still context to fix it. Only verified data propagates upstream.

That's the vision. Not a single central ratchet, but a distributed network of ratchets. Each one small and stubborn. Each one clicking forward, never back.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.*

NOTE: I'm currently writing a book based on what I have seen about the real-world challenges of data preparation for machine learning, focusing on operational, compliance, and cost. I'd love to hear your thoughts!

Originally published at Distributed Thoughts.

The Brownian Ratchet and the Chimpanzee Factory

David Aronchick — Tue, 20 Jan 2026 00:34:00 +0000

Two weeks ago, Steve Yegge released GasTown, a multi-agent orchestrator he describes as "an industrialized coding factory manned by superintelligent chimpanzees." A few days later, Dan Lorenc quietly pushed multiclaude, built on what he calls the "Brownian Ratchet" principle: chaos is fine, as long as we ratchet forward.

While the projects are separate, Dan says he was deeply inspired by Gas Town. However, they, and many others like them, landed on almost identical foundational architecture: detached UI for observability, git worktrees for isolation, external state persistence, and CI as the final arbiter. That convergence tells us something important about where agent tooling is heading.
The Problem Both Solve
Running one Claude Code instance is straightforward, but running twenty in parallel on the same codebase is a distributed systems problem. The challenges are familiar to anyone who's operated infrastructure at scale: agent sessions crash, work gets duplicated, changes conflict, and state disappears when processes restart. Without proper isolation, a single runaway agent can corrupt shared resources, and without observability you can't debug what's happening. If state doesn't persist, progress evaporates the moment something fails.

In source code, people saw this same problem (lots of people working on the same thing), and solved it incrementally with things like SVN, and then Git (with many others as well).

Every multi-agent orchestration system has to answer these questions about multiple things working on a single thing, and what's interesting is watching how different systems answer them.
Two Philosophies, Same Primitives
GasTown takes the comprehensive approach, with seven distinct agent roles that divide responsibilities across the system. The Mayor coordinates overall work, Polecats handle ephemeral tasks, the Refinery manages merge queues, and so on through Witness, Deacon, Dogs, and Crew. Work flows through what Yegge calls the MEOW stack (Molecules, Epics, Work orders), with state persisting through git-backed "hooks" and the Beads memory framework. Agents maintain persistent identities that survive session crashes via the GUPP principle: "If there is work on your hook, YOU MUST RUN IT." This is ... confusing ... but i have to respect explicitly naming everything. Gives you an unequivocal way to know what (and who) is doing what, at the cost of a lot of UX affordance up front.

multiclaude takes the minimalist path with just three roles: a supervisor that coordinates, workers that execute tasks, and a merge-queue agent that handles CI. State lives in a JSON file and the filesystem, communication happens through simple message passing, and the philosophy is explicit: "Trying to perfectly coordinate agent work is both expensive and fragile. Instead, we let chaos happen and use CI as the ratchet that captures forward progress."

The rhetoric differs dramatically between the two projects. Yegge's documentation reads like a manifesto, complete with warnings that you shouldn't use GasTown if you "care about money" or are "more than 4 feet tall." Lorenc's README is Unix-philosophy spare, with clean diagrams and matter-of-fact explanations. But underneath the different personalities, you find the same primitives.

Primitive
GasTown
multiclaude

Process isolation
tmux windows
tmux windows

Code isolation
git worktrees
git worktrees

State persistence
Git-backed hooks
JSON + filesystem

Quality gate
CI (automated merging)
CI ("the ratchet")

Observability
Attach to any session
Attach to any session

Both folks recognized that you can't just spawn Claude instances and hope for the best. You need boundaries.
What the Convergence Reveals
When two experienced engineers arrive at the same architectural primitives, that's signal worth paying attention to. It suggests these aren't arbitrary choices but structural requirements for the problem space.

Isolation requires more than process boundaries. Both projects use git worktrees rather than just separate directories because a worktree gives each agent its own branch, its own working copy, and its own commit history. Conflicts become merge conflicts, which git already knows how to surface, and the blast radius of any single agent is bounded by what it can do to its own worktree.

Observability can't be an afterthought. Both chose tmux as the primary interface rather than a web dashboard or log aggregator. A terminal multiplexer lets you attach to any agent's session, watch it work, and intervene if needed. This is distinctly different from how most "AI agent frameworks" approach the problem, with their emphasis on structured outputs and API-driven orchestration.

State must survive failures. GasTown invests heavily in crash recovery through git-backed hooks while multiclaude keeps it simpler with filesystem persistence, but both reject the idea of ephemeral agent state. When a session dies, the work shouldn't die with it.

CI becomes the coordination mechanism. In both systems, CI isn't just a quality check but the arbiter of what counts as progress. Lorenc is explicit: "If it passes, the code goes in. If it fails, it doesn't. The automation decides." Yegge's Refinery agent serves the same function, and this approach shifts coordination from real-time synchronization (expensive, fragile) to asynchronous validation (robust, scalable).
The Deeper Pattern: Scoped Autonomy
Strip away the specific implementations and you find a design pattern emerging for AI agent systems: scoped autonomy with external persistence. Give agents freedom to act within clear boundaries, let them fail without cascading damage, capture successful outcomes permanently, and accept that coordination is expensive and often unnecessary if your ratchet mechanism is good enough.

This isn't a new idea. It's how we've learned to build reliable distributed systems over the past two decades, and the insight here is that agent orchestration is distributed systems with the same principles applying. Kubernetes asks "Is it running?" and reconciles toward desired state while GasTown asks "Is it done?" and persists completed work. Both are control loops operating over unreliable workers, and both accept that perfect coordination is impossible and design around it.
Where They Diverge: Single-Player vs. MMORPG
The most interesting philosophical difference isn't about orchestration complexity but about the human model.

Lorenc frames multiclaude explicitly: "Gastown treats agents as NPCs in a single-player game. You're the player, agents are your minions. multiclaude treats software engineering as an MMORPG. You're one player among many."

In multiclaude, your workspace persists. You spawn workers, go to lunch, come back, and check what merged while you were away. Other humans can have their own workspaces on the same repo, and the system keeps running when you're not watching.

GasTown is designed around intensive engagement. Yegge describes watching 20-30 agents in parallel, making $100/hour decisions about what work to greenlight, experiencing "palpable stress" as the system runs at speeds too fast to comprehend. It's a powerful multiplier for an engaged operator rather than a fire-and-forget system.

Neither model is wrong since they're optimizing for different workflows, but the MMORPG framing points toward something important: these systems need to work when humans aren't actively supervising.
What This Means for the Industry
We're watching the orchestration layer crystallize in real time, and the patterns that emerge now will shape how agent systems get built for years.

The "19-agent trap" (simulating an org chart with Analyst → PM → Architect → Dev → QA handoffs) is giving way to operational models where agents have specific, bounded roles. The emphasis shifts from elaborate prompting frameworks to infrastructure primitives: isolation, persistence, observability.

The tooling will mature as costs drop. Right now, GasTown burns $100/hour in tokens, partly because the models are expensive and partly because the coordination overhead is high. Both factors will improve, and the architectural patterns being established now will outlast the current pricing structure.

For teams thinking about agent infrastructure, the lesson isn't "adopt GasTown" or "adopt multiclaude" since both are weeks old and explicitly experimental. The lesson is to watch what primitives they converged on, because if you're building agent systems you'll probably need them too: git worktrees for isolation, something tmux-like for observability, persistent state that survives session failures, and CI or some equivalent as the ratchet that captures forward progress.

The chimpanzee factory and the Brownian ratchet arrived at the same answer. That's worth paying attention to.

Repos:
- GasTown: github.com/steveyegge/gastown
- multiclaude: github.com/dlorenc/multiclaude

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.*

NOTE: I'm currently writing a book based on what I have seen about the real-world challenges of data preparation for machine learning, focusing on operational, compliance, and cost. [I'd love to hear your thoughts**](https://github.com/aronchick/Project-Zen-and-the-Art-of-Data-Maintenance?ref=distributedthoughts.org)!**

Originally published at Distributed Thoughts.

The Upstream Problem: Why Context Graphs Are Starving

David Aronchick — Sat, 17 Jan 2026 00:33:27 +0000

Foundation Capital just published what they're calling AI's trillion-dollar opportunity: context graphs. They argue that enterprise value is shifting from systems of record (Salesforce, Workday, SAP) to systems of agents. The new crown jewel isn't the data itself. It's the context graph: a living record of decision traces stitched across entities and time, where precedent becomes searchable.

They're right about the destination. But Greg Ceccarelli's response on LinkedIn caught something important that their framing misses. Foundation Capital focuses on capturing decisions at execution time. That matters, but it's the last mile. The first mile is still bleeding out.
The Telephone Game (But With Developers)
Decisions don't originate at execution time. They originate in conversations.

A PM pattern-matches across customer interviews. Engineering debates constraints in Slack. A VP makes a call on a Zoom that nobody documents. By the time any of this hits a system of record, the context has been compressed, lossy-encoded, and re-interpreted three times. It's a game of telephone where the prize is a barely articulated card in your Kanban roadmap.

Recording meetings is table stakes now. The raw material exists. But most of it vanishes. It's searchable in theory and useless in practice. You can find that the decision was made, but you can't find why it made sense given everything else that was happening at the time.

Jamin Ball's piece "Long Live Systems of Record" pushed back on the "agents kill everything" narrative, arguing that agents don't replace systems of record, they raise the bar for what a good one looks like. I think he’s right, but the problem is no one is the voice for the downstream consumers. The reasoning, the exceptions, the context that justified a decision in the moment isn’t in any form that a human (let alone an agent) can find or consume. That's what's missing.

Context graphs need to be fed. The feed is conversations, and, right now, conversations evaporate.
The Same Problem, Worse, in Data
Greg's framing focuses on software development, and that's where his company SpecStory and their new Intent product are building. I think these are awesome, and deserve a lot of attention. In fact, so much so that I want to take it further. Software development is just one domain where decisions get lost upstream.

Data pipelines, our world, are another, and arguably worse.

When a data engineer decides which fields to drop during transformation, how to handle null values in a critical column, why a particular join strategy was chosen over another, what "clean" means for this specific dataset... where does that reasoning live? In a PR comment that gets archived. A Slack thread that disappears. Someone's head who leaves the company.

The data observability market has exploded. Gartner estimates data observability will be a $2.5B+ market by 2027. But all of it focuses on detecting problems after they happen. The upstream intent, why the pipeline was designed this way, what tradeoffs were considered, what the original constraints were, remains uncaptured.

Another favorite company of mine,Great Expectations, does a great job capturing what should be true. And dbt moves documentation closer to the code. And we have standards

, for example, captures the what of transformations. But almost nothing captures the WHY.

When an ML model makes a bad prediction, you can trace back to the training data. But can you trace back to why the training data was prepared that way? Who decided to impute missing values with medians instead of dropping rows? What was the conversation that led to that feature engineering choice? What did the team know at the time that isn't written down anywhere?

The decision trace doesn't exist because nobody captured it when it happened.
Intent Has Locality
This connects to something I've been thinking about for years. Intent has locality, just like data.

The richest context about a decision exists at the moment it's made, in the place it's made. Move it somewhere else, a different system, a different time, a summary written later, and you lose fidelity. This is true whether you're moving bytes across a network or moving reasoning into documentation.

Think about what happens when you try to document a decision after the fact. You're reconstructing. You remember the outcome but not the three alternatives you considered. You remember the constraint that mattered most but not the secondary factors that shaped the final call. You remember that someone raised an objection but not exactly what shifted the conversation.

The further you get from the moment of decision, the more context you lose. And unlike data, you can't just store a copy closer to where it's needed. The moment passes. The reasoning evaporates. What remains is the artifact without the intent.
What SpecStory Is Building
This is why what Greg and the SpecStory team are building with Intent matters. They started where decisions turn into code: the conversation between developers and coding agents. Intent records every exchange with Claude Code, Cursor, GitHub Copilot, Codex, Gemini. The transcript of how software actually gets built.

But as they asked where the intent came from, the answer kept pointing upstream. Team calls. Architecture discussions. Pairing sessions. The decisions that happen before anyone opens an IDE.

Their solution has three layers:

Capture: Every agent prompt and IDE session, recorded automatically. Not just the code that got written, but the back-and-forth that produced it.

Arena: Real-time collaboration with automatic decision extraction. Not verbose summaries nobody will read. The actual decision linked to the exact moment in the conversation.

Repo: Decisions versioned alongside your source code. Consumable by humans and agents. Searchable forever.

Full context lineage: Team discusses → Decision extracted → Agent builds → Session reasoning preserved → Code ships. Every line of code traceable back to the exchange where the decision was made.

That's the upstream feed layer context graphs need. The bridge from conversation to context to code.
The Parallel Problem Nobody's Solving for Data
The same pattern applies to data infrastructure, and the gap is even wider.

Here's an example I come back to constantly. You're looking at point-of-sale data from a retail chain, and one store shows zero transactions for six hours. What happened?

Maybe the system wasn't connected. Maybe there was a hurricane. Maybe it was midnight and the store was closed. Maybe there was a police action in the area. Maybe the pipeline is connected but stopped running. Maybe someone unplugged the wrong cable during a renovation.

The data looks identical in every case: zeros. But the appropriate response is completely different. If it's a hurricane, you adjust your forecasts and check on your employees. If it's a pipeline failure, you fix the pipeline and backfill the data. If it's midnight, you do nothing because everything is working correctly.

The "what" is the same. The "why" determines everything that matters.

This is the context gap in data infrastructure. Data lineage tools tell you what transformations happened. They don't tell you why someone chose that approach over alternatives. Data catalogs describe what datasets contain. They don't capture the discussions that shaped how those datasets were structured. Data quality tools flag when something looks wrong. They can't explain what "right" was supposed to mean based on the original requirements conversation.

Every data team has experienced this: you inherit a pipeline, something breaks, and you spend days reverse-engineering decisions that took the original author five minutes to make. The 2024 Stack Overflow survey found developers spend 30%+ of their time understanding existing code. For data engineers working with inherited pipelines, I'd bet that number is higher.

A few teams are starting to explore how to capture intent at the data layer, not just the code layer. The ones who figure out how to preserve decision context where data actually lives, at the edge, in pipelines, across distributed infrastructure, might be building something important. But right now, the tooling barely exists.
Why This Matters for AI Agents
Foundation Capital is right that agents need decision traces to exercise judgment. But consider what happens when we only capture traces at execution time.

An agent can follow a rule. It can look up a policy. But it can't understand why an exception was made last quarter unless someone captured the reasoning when it happened. It can see that a certain transformation was applied to a dataset but not why that approach was chosen over three alternatives that were discussed and rejected.

Research on AI decision-making keeps surfacing the same challenge: agents struggle with edge cases because they lack the contextual reasoning that humans use to navigate ambiguity. We've been trying to solve this with better prompts, more examples, refined guardrails. But the fundamental problem is upstream. The reasoning that would help agents handle edge cases was never captured in the first place.

Agents inherit our documentation debt. Every undocumented decision, every lost conversation, every piece of reasoning that exists only in someone's memory becomes a gap in the context graph. And agents can't exercise judgment across gaps.
The Compounding Problem
Context loss compounds in ways that aren't obvious until you're deep in a system you didn't build.

Every undocumented decision becomes a landmine for the next person (or agent) who encounters that code, that pipeline, that system. They see what was built but not why. So they either preserve it blindly (accumulating technical debt they don't understand) or change it without understanding the original constraints (breaking things the original author anticipated but never wrote down).

I've seen this pattern repeatedly. A team inherits a data pipeline with a seemingly arbitrary filter. They remove it because it doesn't match current requirements. Three months later, they discover it was preventing a subtle data quality issue that only surfaces under specific conditions. The original author knew about this. They even discussed it extensively with the team. But that conversation happened on a Zoom call that was never transcribed, and the person who made the decision left the company two years ago.

Multiply this across every team, every pipeline, every codebase. The DORA research shows that elite teams ship faster partly because they spend less time reverse-engineering past decisions. They've somehow preserved more context. Usually through heroic documentation efforts that don't scale.
The Path Forward
Foundation Capital's context graph thesis is right about the destination. Greg Ceccarelli and the SpecStory team are right about the first mile.

The platforms that win won't just capture decisions at execution time. They'll capture intent upstream, in the conversations, the debates, the reasoning that happens before anyone writes a line of code or builds a pipeline.

And they'll keep that intent close to where it matters. Versioned with the code. Traveling with the data. Available when someone (or something) needs to understand not just what happened, but why it was allowed to happen.

We're good at storing what happened. We're terrible at capturing why. The next trillion-dollar platforms will be the ones that figure out how to close that gap, not at execution time, but upstream, where the decisions actually get made.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.*

Originally published at Distributed Thoughts.

The $1B AI Drug Lab That Can't Touch Its Own Data

David Aronchick — Tue, 13 Jan 2026 06:12:34 +0000

Nvidia and Eli Lilly announced a $1 billion AI drug discovery lab today at the J.P. Morgan Healthcare Conference. The press releases are full of the expected language: "reinvent drug discovery," "accelerate medicine development," "foundation models for biology." Lilly's CEO David Ricks said they're "combining our volumes of data and scientific knowledge with Nvidia's computational power."

But, MAN, there is a phrase in there that is doing an INSANE amount of hard word: Combining our volumes of data.

How, exactly?
The Missing Paragraph
The coverage has something conspicuously absent: any discussion of how pharma data actually moves. The lab will be in South San Francisco. Lilly's clinical trial data, compound libraries, and patient information live in facilities scattered across Indiana, Ireland, and dozens of research sites worldwide. The announcement talks about "lab-in-the-loop" systems linking wet labs and dry labs in "24/7 AI-assisted experimentation."

That's a lovely vision. It also assumes data flows like water between these locations. In pharma, it doesn't.
Why Pharma Data Is Different
Clinical trial data contains protected health information under HIPAA. Proprietary compound structures represent billions in R&D investment and competitive advantage. Manufacturing process data falls under FDA's 21 CFR Part 11, which mandates complete audit trails for every electronic record: who touched it, when, why, and what changed.

These aren't bureaucratic inconveniences that clever engineering can route around. They're structural constraints that exist because the consequences of failure are measured in patient safety and billion-dollar regulatory actions.

I've been talking to teams that operate in these environments. The pattern is consistent: they don't lack compute. They lack the ability to make their data usable without making it movable.
The Air Gap Paradox
Traditional security thinking offers two options. Lock data down completely in air-gapped environments where nothing gets out. Or open it up for analysis and accept the exfiltration risk. Pharma has mostly chosen the first option, which is why so much valuable data sits in protected directories that researchers can barely access.

The promise of AI drug discovery assumes you can train models on this data. But training requires moving data to compute, or moving compute to data. The first option triggers every compliance alarm in the building. The second option is what the press releases hand-wave past.

Security teams need something in between: protected environments where data scientists can actually work, but where every attempted data movement gets logged, analyzed, and blocked if it violates policy. Not just access controls (who can log in) but egress controls (what can leave). The ability to process data, transform it, analyze it, without ever letting raw records escape the protected perimeter.

This is remarkably hard to build. It's also not a GPU problem.
The Audit Trail Problem
21 CFR Part 11 requires that regulated companies maintain computer-generated, time-stamped audit trails recording every modification to electronic records.

Let’s say that again. Every Modification

The trail must include the operator identity, date/time, and the nature of the change.

Now imagine training a foundation model on clinical trial data. The model sees millions of records. It learns patterns. It generates new molecular structures based on those patterns. What's the audit trail for that? When the model suggests a compound, which training records influenced that suggestion? When a researcher uses an AI-generated insight to make a decision, how do you document the provenance?

These aren't hypothetical concerns. The FDA released draft guidance on AI in drug development in January 2025, outlining a risk-based credibility assessment framework for AI models across nonclinical, clinical, and manufacturing phases. Regulators are actively figuring out how to apply existing frameworks to machine learning systems. Companies that can demonstrate clean data lineage from source through model to output will have a structural advantage in regulatory discussions.
What Nvidia's Billion Dollars Actually Buys
Nvidia and Lilly aren't naive about these challenges. The announcement mentions that researchers will "generate large-scale data" in the lab itself, creating new datasets specifically designed for AI training. That sidesteps some of the legacy data problems.

The collaboration will (likely) use Nvidia's BioNeMo platform, an open framework for building and training deep learning models for drug discovery that's been adopted by over 200 techbios and large pharma companies. They're also focusing initial efforts on applications where data constraints are less severe: manufacturing optimization, process simulation, early-stage compound screening. These are real opportunities where GPU compute genuinely is the bottleneck.

But the highest-value problems in drug discovery involve the data that's hardest to access: longitudinal patient records from clinical trials, real-world evidence from treatment outcomes, proprietary biological assay results accumulated over decades of R&D. That data can't just be copied to a shiny new lab in South San Francisco. And the estimated $1-3 billion cost to develop a single new drug happens largely because of failures that better data access might prevent.
The Actual Hard Problem
The companies that figure out "compute over data" for regulated industries will eat this market. Not by building bigger GPU clusters, but by solving the governance layer that lets valuable data become usable without becoming vulnerable.

What does that look like in practice? Tagging data at the source with cryptographic fingerprints so you can always verify provenance. Processing pipelines that run inside protected perimeters with whitelist-only egress. Audit systems that log not just access but every transformation, every query, every attempted export. The ability to prove, at any point, exactly what happened to every record.

This is boring infrastructure work. It doesn't make for exciting keynote demos. But it's the actual constraint on AI-driven drug discovery, and throwing more GPUs at it doesn't help.
What I'd Watch For
If you're evaluating pharma AI investments, look past the compute announcements. Ask instead:
How does the company handle data that can't leave its current location?What's their approach to federated learning or on-premises model training?How do they maintain audit trails through AI-assisted workflows?What's their story for regulatory submissions that involve AI-generated insights?
The GPU buildout is the visible part of the iceberg. The governance layer underneath is where the actual differentiation happens.

Nvidia's bet will work for some use cases. Public datasets, synthetic data, newly generated experimental results. But the highest-value pharma AI problems live behind walls that compute power alone can't scale. The billion dollars is impressive. The missing paragraph about data governance is the real story.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.*

Originally published at Distributed Thoughts.

Your 2026 Resolution: Add Context to Your Data (Before It Breaks You)

David Aronchick — Sun, 11 Jan 2026 06:11:42 +0000

Last week I sat in an executive review where two teams spent forty minutes arguing about "active users." Not about strategy. Not about growth. About what the number meant.

One team counted anyone who logged in. The other excluded users who bounced in under 30 seconds. Neither knew which experiment flags were active when the data was pulled. The dashboard just showed a number. No definition. No lineage. No context.

This happens constantly. And it's about to get significantly worse.

Gartner predicts that 60% of AI projects will be abandoned by 2026 because organizations lack "AI-ready data." Not because models failed. Not because compute was too expensive. Because the data traveling through these systems carries no meaning beyond the raw values.

The models can't tell the difference between a deprecated pricing page and current policy. They can't distinguish a test account from a real customer. They retrieve answers confidently, cite sources correctly, and still get everything wrong.

This is the year we stop treating context as optional documentation and start treating it as infrastructure.
The Context Engineering Pivot
Something shifted in 2025. The industry stopped talking about "prompt engineering" and started talking about "context engineering."

Andrej Karpathy called it "the delicate art and science of filling the context window with just the right information for each step." MIT Technology Review documented the transition from "vibe coding" to systematic context management. Google's December release of their Agent Development Kit was entirely focused on context architecture.

The terminology change matters. "Prompt" implies a single instruction you craft carefully. "Context" implies an entire information environment you engineer deliberately.

And it turns out most organizations have been engineering that environment with all the care of a teenager cleaning their room by shoving everything under the bed.

David Lanstein, CEO of Atolio, put it bluntly in IBM's 2026 predictions: "The solution isn't bigger models, but smarter data. True value will come from feeding models high-quality, permission-aware structured data to generate intelligent, relevant and trustworthy answers."

The race for bigger context windows missed the point. A 200K token context window filled with undifferentiated garbage produces undifferentiated garbage outputs, just with more confident citations.
What Context Actually Means
When I talk about context, I don't mean adding a few comments to your SQL. I mean four distinct layers that most data systems ignore entirely.

Semantic context is what a value actually represents. Not just "this column is called revenue" but "this is recognized revenue under ASC 606, calculated monthly, excluding deferred amounts, as defined by the finance team's Q3 2025 policy update." When the definition changes, the context changes with it. When someone queries the data six months from now, they see what it meant then, not what it means today.

Operational context is the health and provenance of the data at query time. Is this number fresh? Did the upstream pipeline fail overnight? Is there an active incident affecting the source system? A dashboard that shows revenue without showing "by the way, the payment processor had a three-hour outage last night" is lying by omission.

Experimental context is which flags and tests were active when the data was generated. Your MAU number is meaningless if you don't know that 40% of users were in an onboarding experiment that changed the activation flow. The number isn't wrong. It's just uninterpretable without the experiment metadata traveling alongside it.

Human context is ownership and decision history. Who defined this metric? What decisions have been made based on it? Where's the design doc? When someone has a question, they shouldn't have to play archeologist in Slack to figure out who to ask.

Most data systems capture maybe one of these. Usually the semantic layer, poorly, in a data catalog that nobody updates and fewer people read.
The Kubernetes Lesson I Should Have Learned Sooner
When I was the first product manager on Kubernetes at Google, we thought we'd solved the orchestration problem. Pods, services, deployments. State reconciliation. Declarative configuration. Ship your containers and let the scheduler figure out the rest.

What we hadn't solved was context.

A customer came to us wanting to run a global footprint of clusters, one per region, with synchronized jobs. Low-latency application, workloads coordinated across continents. We had an internal project called "Ubernetes" that was supposed to handle this, but the complexity was brutal. We ended up helping them build a custom solution.

The problem wasn't deploying the workloads. GitOps handles that fine now. The problem was that when data crossed cluster boundaries, all the context about that data evaporated. Each cluster was internally consistent. The global system was broken because nobody knew what the data meant in aggregate.

I've watched the same pattern repeat across every data problem I've worked on since. The compute orchestration is largely solved. The data orchestration is still a mess, and it's a mess because context doesn't travel with the data. This is actually why I'm writing a book on exactly this: the hidden complexity of data preparation that causes 80% of AI projects to fail.
Why RAG Doesn't Fix This
The popular assumption has been that Retrieval-Augmented Generation solves the context problem. Point your model at your documents, let it retrieve what it needs, problem solved.

InfoWorld's analysis last week explains why this breaks at scale: "RAG breaks at scale because organizations treat it like a feature of LLMs rather than a platform discipline. Models generate confidently incorrect answers because the retrieval layer returns ambiguous or outdated knowledge."

The failure mode is insidious. RAG with good retrieval but no context governance produces what I've started calling "hallucination with citations." The model cites a real document. The citation is accurate. The document is from 2023 and contradicts current policy. The answer is wrong, but it looks impeccably sourced.

CX Today reported on exactly this pattern: "If the knowledge base is outdated, RAG just retrieves the wrong answer faster. If content is unstructured, like PDFs, duplicate docs, or inconsistent schemas, the model struggles to pull reliable context."

The problem isn't retrieval. The problem is that the documents themselves carry no context about their validity, scope, or temporal bounds. A PDF is just a PDF. It doesn't know that it was superseded by a newer version, that it only applies to EMEA customers, or that the pricing section was invalidated by a board decision last quarter.

When VentureBeat declared "RAG is dead" in their 2026 predictions, they were being provocative. But the underlying point stands: RAG without context governance is dying. The organizations that will succeed with retrieval-augmented systems are the ones treating their knowledge bases as living, context-rich assets rather than static document dumps.
The Toll Booth Is Coming
There's a harder version of this problem emerging, and most organizations haven't noticed it yet.

Constellation Research warns that "enterprise data tolls and API economics are going to be a headache" in 2026. Celonis is suing SAP over data access. The Information reported that Salesforce is raising prices on apps that tap into its data. Connector fees are trickling down to IT budgets.

"Connection fees are going to be the new cloud egress," Constellation writes.

Here's what this means: if you don't own the context layer for your own data, you'll rent it from someone else. Every vendor building "AI-ready" connectors is essentially building a context layer on top of your data and charging you for access to the meaning of information you already own.

The Solutions Review predictions roundup makes this explicit: "By the end of 2026, connectivity, governance, and context provisioning for AI agents will be built into every serious data platform."

Built in. Not optional. Not a nice-to-have catalog project. Core infrastructure.

The organizations that treat context as someone else's problem will find themselves paying tolls to access the semantic meaning of their own customer data. The ones that invest now will own that layer.
Resolution #1: Ship Context With Every Event
The practical version of this starts at ingestion.

Every event entering your system should carry enough metadata that a reader (human or machine) can interpret it without external lookups. Not "user_id, timestamp, action" but "user_id, timestamp, action, schema_version, experiment_flags, source_system, data_classification."

This isn't aspirational. Anthropic's context engineering guide describes exactly this pattern: maintaining lightweight identifiers that allow systems to "dynamically load data into context at runtime using tools."

A transformation editor should show, live, which downstream dashboards and models will break if you drop a column. A query should surface its lineage alongside its results. A dashboard shouldn't just display a number; hovering over it should reveal the definition, the upstream tables, the freshness, and the last incident that affected it.

This requires tooling changes, yes. But mostly it requires treating context as a first-class output of every pipeline stage rather than an afterthought someone might add later.
Resolution #2: Make Context the Default in AI and Agents
TechCrunch's 2026 analysis identifies the Model Context Protocol as "quickly becoming the standard" for agent interoperability. OpenAI and Microsoft have embraced it. Google is standing up managed MCP servers.

The infrastructure for context-aware agents is arriving. The question is whether your data is ready to participate.

That means storing valid_from/valid_to timestamps on policy documents. It means tagging content with scope limitations (region, customer tier, product line). It means encoding data classification and retention rules at the source, not in a compliance spreadsheet nobody maintains.

Stanford HAI's predictions note that "2026 will hear more companies say that AI hasn't yet shown productivity increases." The organizations that do show productivity increases will be the ones whose agents can distinguish current reality from historical noise without human intervention.

An agent that refuses to take high-impact actions without verifying the environment, cohort, and guardrails is not cautious. It's correctly engineered. An agent that charges ahead on stale data with high confidence is the expensive kind of wrong.
Resolution #3: Measure Time to Trustworthy Insight
I wrote about Nicole Forsgren's new book last month. Her frameworks for developer productivity apply directly to data work, but with a crucial modification.

For data teams, the north star isn't deployment frequency or cycle time. It's time to trustworthy insight: from raw logs or events to a result you would put in front of an executive with confidence.

Most organizations can't measure this because they don't know when insight becomes trustworthy. The data arrives, transformations run, dashboards update, but confidence accrues informally. Someone senior enough eventually blesses the number based on vibes and experience.

Context infrastructure makes this measurable. If every metric carries its lineage, freshness, and incident history, you can ask: how long did it take from data landing to a metric with full provenance, no upstream incidents, and a defined owner? That's the number that matters.

When that number shrinks, you're actually improving. When people are just shipping dashboards faster without context, you're accumulating debt.
The Year We Stop Arguing About Definitions
Most New Year's resolutions fail by February. The gym membership lapses. The meditation app goes unused. The ambitious reading list gathers dust.

Data resolutions fail for the same reason: they're framed as one-time efforts rather than infrastructure changes. "We'll document our metrics" becomes a Q1 project that never gets maintained. "We'll improve data quality" becomes a dashboard that nobody checks.

Context isn't a project. It's a property of how data moves through your organization. It either travels with its story or it doesn't.

The organizations that treat 2026 as the year context becomes default will stop having the same arguments in every meeting. The exec review becomes a discussion of strategy instead of a debate about what "active users" means. The AI agent produces answers that come with their own credibility assessment. The data team ships products instead of debugging why downstream consumers don't trust the numbers.

Gartner says 60% of AI projects will fail for lack of AI-ready data. The projects that succeed will be the ones that stopped treating data as numbers and started treating it as knowledge.

That's the resolution. Make the data know what it is.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.

NOTE: I'm currently writing a book called "Zen and the Art of Data Maintenance" based on what I've seen about the real-world challenges of data preparation for machine learning, focusing on operational, compliance, and cost. I'd love to hear your thoughts!

Originally published at Distributed Thoughts.

The Natasha Problem: Why Your Data Pipeline Only Fits One Person

David Aronchick — Sun, 11 Jan 2026 06:11:42 +0000

For most folks, you probably don’t think about clothing sizes. There’s a number, you pick it, you try on the clothes, and if they fit, then congrats, you’re that number. But how’d they pick that number? And why does every style/line/person fit slightly differently?

Turns out there's a woman named Natasha in Los Angeles whose body is used to design jeans for seven or eight major clothing brands. She's what the industry calls a "fit model." When Levi's or H&M or whoever designs a new pair of jeans, they don't start with measurements of the general population. They design them on a mannequin, then bring in Natasha, and adjust everything until the jeans fit her perfectly.

She's the only person in the design process with an actual human body.

Every other size is mathematical extrapolation. The size 2 isn't designed for any real person. Neither is the 4, the 8, or the 12. They're all proportional adjustments from Natasha's measurements, calculated by formula. If you're not built exactly like Natasha, your clothes don't actually fit you. They fit a mathematical projection of you derived from someone else's body.

This is, as Radiolab's Heather Radke discovered, why the dressing room feels like a personal judgment. We've internalized the idea that clothes should fit, and when they don't, we assume the problem is our body. But the clothes were never designed for our bodies. They were designed for one body, then scaled with arithmetic.

I kept thinking about this because it's exactly how we build our IT infrastructure.
The Ruth O'Brien Problem
The Natasha situation has a history. In the 1930s, a woman named Ruth O'Brien at the Bureau of Home Economics tried to solve the sizing problem scientifically. She hired "measuring squads" through the WPA to travel the country and measure women's bodies across twenty-six different dimensions: elbow to wrist, thigh girth, heel length, and on and on.

She was going to create the definitive dataset of American women's bodies.

The resulting dataset became the basis for women's clothing sizes for decades. But there never was an “average” person. It was a statistical fiction.
Your Pipeline Has a Fit Model Too
Every data pipeline has its own Natasha. Usually it's the data from headquarters, or the first customer deployment, or whatever clean dataset was available when the system was designed.

I've watched this pattern play out dozens of times. A team builds an ETL pipeline that works beautifully on their development data, where the schema is clean, the timestamps are consistent, and the sensor readings arrive in perfect intervals. They deploy to production, and 40% of their edge sites start throwing errors.

The problem isn't that the edge data is wrong. The problem is that the pipeline was designed for one shape of data, then mathematically extrapolated to handle everything else.

Consider what happens with ML training data. ImageNet became the de facto standard for computer vision benchmarks. Models trained on ImageNet achieve remarkable accuracy on ImageNet test sets. Deploy those same models to a factory floor, and they struggle with the lighting, the angles, the dust on camera lenses, the specific way that this particular production line differs from the curated images in the training set.

The model was fit to ImageNet. Everything else is extrapolation.

Or look at timestamp handling. A pipeline designed on data from a single timezone assumes UTC normalization is someone else's problem. Works fine until you're ingesting from devices across twelve timezones, some of which handle daylight saving time transitions differently, some of which have clock drift, and one of which is running firmware from 2019 that uses a different epoch.

The pipeline wasn't wrong. It was designed for one reality and scaled mathematically to others.
The Measurement Squads Never Left
Ruth O'Brien's original sin was the belief that you could measure a population once, derive a standard, and apply it forever.

We do this constantly with data.

A team defines a schema based on current requirements. They encode assumptions about data types, nullable fields, value ranges, and relationships. Then they treat that schema as ground truth, and any data that doesn't conform is "dirty" and needs to be "cleaned."

But the data isn't dirty. The data is reality. The schema is the idealized projection that reality keeps refusing to match.

I saw a project once where the sensor format had been standardized across the organization with beautiful documentation and clear specifications. Every new deployment was supposed to conform. In practice, about 60% of edge sites had made local modifications: different firmware versions, custom calibration routines, integration with legacy equipment that predated the standard by a decade.

The central data team spent enormous effort "fixing" the non-conforming data with transformations to coerce it into the standard shape, imputation for missing fields, and interpolation for different sampling rates.

By the time the data reached the analytics layer, it had been mathematically adjusted to fit a shape it never had. The "cleaned" data was a fiction, no more real than a size 2 extrapolated from Natasha's measurements.
Bodies Resist Standardization. So Does Data.
The reality is that bodies cannot be forced into interchangeable parts. The entire apparatus of industrial manufacturing assumes standardization. You take raw materials, process them into uniform components, and assemble them into identical products. It works for cars and electronics, but it fundamentally doesn't work for human bodies.

The closer you get to where data is generated, the more specific and contextual it becomes. A sensor on a drilling rig in the Permian Basin produces readings shaped by the specific geology, equipment age, and operational patterns of that site. A sensor in the North Sea produces data shaped by entirely different conditions. Both might be "pressure readings," but they're not interchangeable.

The centralization assumption says: bring all the data to one place, normalize it, and then analyze. This works if the data is genuinely similar. It falls apart when the normalization process destroys the very information you needed.
The Dressing Room Moment
Radke describes the experience of trying on clothes that don't fit as a moment of internalized judgment. The clothes were never designed for your body, but you blame yourself anyway. The sizing system has convinced you that "normal" is a real thing, and you're the deviation.

Data teams have the same experience. The pipeline breaks on edge cases, and the team treats it as a data quality problem to be solved with more transformation logic, more coercion, more normalization, more effort to force reality into the shape the system expected.

But the pipeline was designed for one type of data. Everything else is extrapolation. When the extrapolation fails, that's information. It's telling you that your model of reality was incomplete.

The question isn't "how do we clean this data to fit our schema?" The question is "why did we assume all data would look like our development dataset?"

Martha Skidmore was the closest match to Norma out of 3,864 women, and she still wasn't Norma. She was a real person, with a real body, that happened to approximate a statistical fiction slightly better than 3,863 others.

Your edge data isn't defective. It's real. The pipeline is the fiction.

Originally published at Distributed Thoughts.

A Picture Is Worth Ten Thousand Tokens

David Aronchick — Tue, 30 Dec 2025 00:34:11 +0000

"A picture is worth a thousand words" has been greeting-card wisdom for a century, the kind of thing we nod along to while understanding it metaphorically because images convey emotion, show rather than tell, and bypass the limitations of language. What we didn't expect was for this to become literally, computationally true.

DeepSeek released a paper this fall that made a lot of people rethink what we know about LLM efficiency, At the core, one of the findings seems obviously wrong until you work through the math: if you render text onto an image and have a vision-language model decode it, you can achieve 97% accuracy while using one-tenth the tokens. Take a document with 1,000 text tokens, turn it into an image, and the model can reconstruct that text using just 100 vision tokens. For those that aren’t researchers, this seems insane: You’re telling me that words, representing by simple characters, are HARDER to make sense of than the same words but represented by pixels? Nuts.

What this possibly reveals under the hood could be pretty foundational. If something computationally "heavy" modality turns out to be the efficient one while the computationally "light" modality is actually the expensive one, then we’ve been thinking about a lot of things wrong. And, according to the page, it looks like we have been.
The Architecture That Makes This Work
DeepSeek-OCR pairs two components: an encoder called DeepEncoder (about 380M parameters) and a decoder based on their 3B MoE model with 570M active parameters. The encoder is the interesting part because it chains together a SAM-base model for local perception using window attention, a 16x convolutional compressor, and a CLIP-large model for global understanding in a sequence that exploits how these different attention mechanisms scale.

The key insight is that window attention (that’s the thing the model “looks” at to predict the right answer) processes lots of tokens cheaply because it only looks at local neighborhoods, which means you can afford to have thousands of tokens at that stage without blowing up your compute budget. Then the compressor aggressively reduces token count before the expensive global attention (where it compares its predictions with everything else) kicks in so you're only paying the quadratic attention cost on the compressed representation rather than the full input.

Feed it a 1024x1024 image and you get 4,096 patch tokens from the initial segmentation, but after compression that becomes just 256 vision tokens entering the decoder, and a 640x640 image yields only 100 tokens.

The magic isn't in any single component but in recognizing that compression can happen inside the pipeline rather than fighting against the text representation after the fact. And because the computational characteristics of vision encoders (cheap local processing followed by expensive global processing on a much smaller token set) happen to be more favorable than the characteristics of text transformers (expensive global processing on the full token count from the start), it gives DeepEncoder the ability to specifically to exploit that gap.
The Numbers That Matter
On their Fox benchmark testing documents with 600-1,300 text tokens, the results show a graceful degradation curve that suggests we're not just getting lucky on easy cases: at 100 vision tokens (compression ratio around 7-10x) they hit 87-98% OCR precision depending on document complexity, and at 64 vision tokens (pushing toward 20x compression) precision drops to 59-96%, which is still surprisingly usable for applications where you need the gist rather than perfect fidelity.

On OmniDocBench, a practical document parsing benchmark, DeepSeek-OCR with 100 vision tokens beats GOT-OCR2.0 which uses 256 tokens, and with fewer than 800 vision tokens it outperforms MinerU2.0 which averages over 6,000 tokens per page. In production they're processing 200,000+ pages per day on a single A100-40G, which isn't a research demo but a training data pipeline running at scale.
Why This Works (And Why It's Counterintuitive)
We've built our mental models around text as the native format for language understanding, and for good reason: text is what LLMs were designed for, text is lightweight, text is structured, and vision is the bolt-on capability we added later for multimodal tasks. But attention mechanisms don't care about our intuitions because they care about sequence length, and attention scales quadratically with sequence length, which means a document with 5,000 tokens pays the O(n²) cost across all 5,000 tokens regardless of how "lightweight" we think text ought to be.

Vision encoders, particularly modern ones with the window-then-global architecture DeepSeek uses, have fundamentally different computational characteristics because you're essentially buying cheap local processing at the window attention stage and only paying the quadratic cost on a much smaller token count after compression, which means the image isn't a burden you're adding to the model but a compression layer that happens to be more efficient than operating directly on text tokens. This is counterintuitive until you remember that efficiency depends on the compute path, not just the data size.
The Memory Decay Proposal
The paper includes a fascinating speculation in their discussion section that I think deserves more attention than the OCR results themselves. They draw a parallel between human memory decay over time, visual perception degradation over distance, and text compression at different resolutions, and their proposal is elegant: for multi-turn conversations, render older dialogue turns as images at progressively lower resolutions.

Recent context stays high-resolution (their "Gundam" mode, 800+ tokens) while older context gets progressively downscaled to Large mode for yesterday's conversation, Base mode for last week, and Small or Tiny for anything older.

The information doesn't disappear but compresses into something lossy and gist-preserving, which mirrors something real about how memory actually works: you don't remember conversations from a year ago at the same fidelity as conversations from an hour ago, but the information is still there in some form, accessible if you need it but not consuming the same cognitive resources as recent experience.

The engineering implication is that instead of choosing between "keep full context" and "truncate old context," you get a spectrum where the context window becomes a memory system with natural decay characteristics built into the architecture itself, giving you a gradient rather than a cliff.
The Distributed Systems Angle
There's a pattern here that feels familiar from data infrastructure because the optimal representation for processing isn't always the optimal representation for storage or transmission. We compress video for streaming then decompress for playback, we convert data to columnar formats for analytics even though row-oriented formats are more natural for transactional workloads, we build materialized views that trade storage for query performance, and we ship compute to data when moving the data would be more expensive than moving the code.

What DeepSeek is demonstrating is that text-to-image-to-text can be a legitimate processing pipeline, not because images are somehow "better" but because the computational characteristics of the vision encoder path happen to be more favorable for certain workloads and the transformation overhead pays for itself in reduced attention costs. This is compute-over-data thinking applied to tokens themselves: instead of asking "how do we process this text efficiently," you ask "what representation makes the compute most efficient, and is the transformation cost worth it?"

For long documents, the answer might genuinely be to render to image, process visually, and reconstruct text.
What This Means
DeepSeek is careful to call this "early-stage work that requires further investigation," and they're right to be cautious because OCR is a specific task where you have ground-truth text to measure against and general language understanding is considerably harder to evaluate. But the direction is suggestive in ways that go beyond document processing.

If you're building systems that process long documents, analyze historical conversations, or maintain persistent context across sessions, the architecture you're probably using (longer context windows, sliding attention, memory banks, retrieval augmentation) might be competing with an approach that seems absurd on its face: just turn the text into pictures. The optimal representation for text processing in an LLM might not be text, which is either a profound insight about the nature of these systems or a temporary artifact of current architectures that will disappear when we build better text processing.

I genuinely don't know which, but when the counterintuitive result has solid empirical backing, that's usually where the interesting questions live.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.*

Originally published at Distributed Thoughts.

NVIDIA Bought the Bouncer: SchedMD and Where Lock-In Actually Lives

David Aronchick — Mon, 29 Dec 2025 00:37:14 +0000

On December 15, 2025, NVIDIA acquired SchedMD, a 40-person company based in Lehi, Utah. The price wasn't disclosed, the press release emphasized a commitment to open source, and most coverage focused on NVIDIA’s expanding software portfolio, thereby missing the point entirely. Most folks missed how huge this was.

SchedMD maintains Slurm, the workload manager running on 65% of the TOP500 supercomputers, including more than half of the top 10 and more than half of the top 100. Every time a researcher submits a training job, every time an ML engineer queues a batch inference run, every time a national lab allocates compute for a simulation, there's a decent chance Slurm is deciding which GPUs actually run it.

Everyone's been watching the CUDA moat. Judah Taub's recent Substack piece frames it perfectly: the programming model as the source of lock-in, with five potential escape routes ranging from OpenAI's Triton to Google's TPUs to AMD's ROCm to Modular's Mojo to Tenstorrent's RISC-V approach. All of which are valid competitive threats.

But NVIDIA, to their credit, saw through the programming model debates and identified one of the key ways to accelerate the scale-out. They bought the bouncer.
What Slurm Actually Does
If you've never submitted a job to an HPC cluster, Slurm is invisible infrastructure, and that's intentional. Researchers type sbatch my_training_job.sh and their code runs on GPUs. Still, how those GPUs get allocated, when the job actually starts, which nodes handle which portions of distributed training, how competing jobs get prioritized, whether your experiment runs tonight or next Tuesday—that's all Slurm.

The formal description sounds almost TOO basic: "allocating exclusive and/or non-exclusive access to resources, providing a framework for starting, executing, and monitoring work, and arbitrating contention for resources by managing a queue of pending jobs."

The reality is that Slurm is the layer that translates organizational policy into compute allocation. This includes things like: * Fair-share scheduling across research groups * Priority overrides for deadline-sensitive projects * Resource limits that prevent any single user from monopolizing a cluster * Preemption policies that balance throughput * Responsiveness to Hilbert curve scheduling that optimizes for network topology

And lots more. Or just launching a job without requiring SSH!

Every organization running Slurm has encoded its resource management philosophy into its configuration over years of tuning, with institutional knowledge baked into partition definitions and quality of service policies, accounting systems tied to grants and budgets, and user training built around Slurm commands. This isn't a program you swap out over a weekend.
Why Slurm Won
Slurm wasn’t the obvious choice. When development began at Lawrence Livermore National Laboratory in 2001, the HPC world ran on proprietary schedulers: PBS (Portable Batch System) had variants everywhere, IBM's LoadLeveler dominated their ecosystem, Quadrics RMS handled specialized clusters, and Platform Computing's LSF (Load Sharing Facility) served enterprise HPC.

LLNL wanted something different because they were moving from proprietary supercomputers to commodity Linux clusters and needed a resource manager that could scale to tens of thousands of nodes, remain highly portable across architectures, and stay open source. The 2002 first release was deliberately simple, and the name originally stood for "Simple Linux Utility for Resource Management" (the acronym was later dropped, though the Futurama reference remained).

What happened next is a case study in how open source wins infrastructure markets.

PBS fragmented into OpenPBS, Torque, and PBS Pro (now Altair), with each fork diluting the community and scattering innovation, leaving organizations that chose PBS to pick which one, and none had the whole community behind them. THEN LSF went commercial when IBM acquired Platform Computing in 2012. While enterprise support is valuable, licensing costs matter when you're scaling to thousands of nodes, which makes the open-source alternative increasingly attractive. THEN Grid Engine's ownership bounced between Sun Microsystems, Oracle, and Univa, with each transition eroding community trust as development priorities shifted with corporate strategy.

Slurm stayed focused on one codebase with GPLv2 licensing that couldn't be closed and a plugin architecture that let organizations customize without forking. And in 2010, Morris Jette and Danny Auble, the lead developers, left LLNL to form SchedMD (https://en.wikipedia.org/wiki/SchedMD), creating a commercial support model that kept the software free while funding continued development—the Red Hat playbook, applied to HPC scheduling.

Hyperion Research data from 2023 shows that 50% of HPC sites use Slurm, while the next closest, OpenPBS, sits at 18.9%, PBS Pro at 13.9%, and LSF at 10.6%. The gap isn't closing, and it's widening.
The Two-Door Strategy
In parallel with ALL that noise, NVIDIA wasn’t sitting around flat-footed.

In April 2024, NVIDIA acquired Run:ai for approximately $700 million. Run:ai builds Kubernetes-based GPU orchestration, and if Slurm is how supercomputers and traditional HPC clusters manage GPU workloads, Run:ai is how cloud-native organizations do the same thing on Kubernetes—different paradigms serving the same function, and NVIDIA now owns the scheduling layer for both.

Run:ai handles the world that emerged from containers and microservices: * Organizations running on GKE, EKS, or on-prem Kubernetes clusters * Data science teams whose workflows are built around Jupyter notebooks, Kubeflow, and MLflow * And companies that think in pods and deployments rather than batch queues and node allocations.

Slurm handles the world that emerged from supercomputing: national labs, research universities, pharmaceutical companies running molecular dynamics, financial firms running risk simulations, organizations where HPC predates the cloud, and where "scale" means dedicated clusters with thousands of nodes.

Both roads lead to GPUs, and NVIDIA now controls traffic on both.
What Lock-In Actually Looks Like
Judah Taub's CUDA analysis is correct that the programming model creates real lock-in, because rewriting GPU kernels for a different platform is expensive, and the ecosystem of libraries, tools, and community knowledge around CUDA represents decades of accumulated investment.

But programming models can be abstracted, compilers translate, and compatibility layers exist. PyTorch runs on AMD GPUs via ROCm, JAX runs on TPUs, and the code you write doesn't have to be tied permanently to CUDA, even if the transition has friction.

Orchestration creates a different kind of stickiness, because your workflows are encoded in Slurm through every batch script, every job array definition, every dependency chain that says "run step B only after step A completes successfully,” and that's not just code but institutional memory. Your accounting systems integrate with Slurm through reports that show department heads how their GPU allocation was used, chargeback systems that bill internal projects, and compliance logs that verify your government-funded research ran on approved infrastructure. Your users know Slurm through the commands they type without thinking, the debugging instincts for when jobs hang or fail, the training materials your HPC team developed, and the Stack Overflow answers they Google at 2 AM. Your cluster topology is optimized for Slurm's algorithms through a network configuration that aligns with Slurm’s understanding of a fat-tree topology, a partition structure that reflects your organizational hierarchy, and node groupings that balance locality and fairness.

Switching schedulers isn't a recompile; it's a reorganization.
The Promise and the Pattern
NVIDIA says Slurm will remain open source and vendor-neutral, and the GPLv2 license makes closing the source legally problematic anyway, so SchedMD's existing customers aren't about to get cut off.

But control of the roadmap is different from control of the code.

When NVIDIA prioritizes features, which hardware gets first-class Slurm support? When performance optimizations ship, which GPUs benefit most? When integrations between Slurm and the rest of NVIDIA’s software stack tighten, does the "vendor-neutral" promise mean equal optimization for AMD and Intel accelerators?

The pattern exists in enterprise software: Oracle doesn't prevent you from running MySQL, Microsoft doesn't prevent you from using GitHub with non-Azure clouds, but the integration points, the polish, and the performance optimizations flow toward the owner's products.

NVIDIA's official line emphasizes that Slurm "forms the essential infrastructure used by global developers, research institutions, and cloud service providers to run massive scale training infrastructure," which is true—and now NVIDIA owns that essential infrastructure.
The Distributed Gap
There's a less-discussed implication in all this.

Traditional HPC scheduling, whether Slurm or its competitors, assumes a particular architecture: a big, centralized cluster where jobs are scheduled across nodes, making the optimization problem one of matching jobs to resources within a unified system.

This architecture works well when data and compute are co-located, with training runs pulling from high-speed parallel file systems and simulations operating on datasets staged to local storage, making the cluster a world unto itself.

But increasingly, that's not the world in which organizations operate.

Data sovereignty requirements mean datasets can't always move to where the GPUs are; edge deployments generate data that shouldn’t traverse networks just to run inference; federated learning needs to coordinate training across institutions without centralizing sensitive information; and multi-cloud strategies mean compute is scattered across providers, regions, and architectures.

Run:ai helps with Kubernetes-based orchestration but assumes Kubernetes, while Slurm helps with HPC workloads but assumes a traditional cluster architecture. Neither solves the problem of "I have data in 50 locations, compute in 12 different configurations, and regulatory constraints that prevent me from pretending this is one big cluster."

NVIDIA's acquisitions reinforce the gravitational pull toward centralization: bigger clusters, more GPUs, bring your data to us. That's a valid architecture for many workloads, and for foundation model training at hyperscale, it might be the only architecture.

But it's not the only architecture that matters, and the orchestration gap for truly distributed computing remains wide open. (We have some thoughts if you’re interested :))
What NVIDIA Actually Understood
Credit where it's due: NVIDIA read the landscape correctly.

The hardware competition gets the attention, with AMD's MI300X, Intel's Gaudi, Google's TPUs, and startups raising hundreds of millions to build custom silicon, keeping everyone focused on the chip.

NVIDIA looked one layer up and recognized that whoever owns the orchestration layer owns the decision about which chips run which workloads, because the scheduler doesn't just allocate resources; it also encodes assumptions about what resources exist and how they should be used.

By acquiring both Slurm and Run:ai, NVIDIA ensures that, regardless of which paradigm you use (traditional HPC or cloud-native Kubernetes), the software layer that schedules your GPU workloads comes from NVIDIA, meaning alternatives to CUDA still need to run through NVIDIA's orchestration. It's like owning both the road and the traffic lights: the cars might be different, but they all stop at the same intersections.
Where This Leaves Everyone Else
For organizations already running Slurm, not much changes immediately because the software remains open source, SchedMD's support contracts presumably continue, and the 40 employees who built their careers around making Slurm work are now NVIDIA employees with presumably NVIDIA resources.

For organizations building alternatives to NVIDIA's hardware dominance, the landscape has grown harder: your new accelerator needs software ecosystem support, which now means either convincing NVIDIA-owned Slurm to treat your hardware as a first-class citizen or building your own orchestration layer from scratch.

For anyone thinking about distributed computing that doesn't fit the cluster model, the message is clear: the major players aren't building for you, and the orchestration layer for truly distributed, heterogeneous, data-gravity-respecting deployments doesn't exist in their portfolio.

That's both a challenge and an opportunity.

The CUDA moat is real, but it was always visible, always discussed, always the focus of competitive energy. The orchestration moat is quieter because Slurm doesn't make headlines like GPUs do, and scheduling software isn't sexy; it's just where the actual decisions get made.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.*

Originally published at Distributed Thoughts.

Edge ML Has a Size Obsession

David Aronchick — Tue, 16 Dec 2025 00:34:14 +0000

UPS could deliver your Amazon package on a cargo e-bike. In most cities, for most packages, this would actually be faster. No parking. No traffic. Straight to your door.

Instead, a 16,000-pound truck idles outside your apartment building while the driver walks up three flights of stairs with an envelope containing a phone charger.

It's not that UPS is stupid. The truck handles the complicated cases: bulk deliveries, heavy items, and commercial routes with 200 stops. Once you've built infrastructure for complex cases, running it for easy cases feels free. Same truck, same driver, same route. Why optimize?

But "feels free" isn't free. The truck burns diesel at idle. It needs a commercial parking spot that doesn't exist. The driver spends 30% of their day not delivering packages but managing the logistics of operating a vehicle designed for a more complex problem than most stops actually present.

Edge ML has the same problem. We built the infrastructure for complex cases (language models, multimodal reasoning, generative AI), and now we're using it for everything. Sensor classification? Deploy a neural network. Anomaly detection? Fine-tune a transformer. Predictive maintenance? Surely this needs deep learning.

A quantized Llama 3B takes 2GB on disk and 4GB in memory. A 4-bit quantized 7B model still needs roughly 4GB. Want to run a 70B model? Even with aggressive quantization, you're looking at 35GB minimum.

A scikit-learn random forest for the same classification task takes 50KB.

The industry spent three years figuring out how to squeeze the truck into tighter parking spaces. Most deliveries never needed the truck.
Two Mistakes, Not One
The size obsession hides two distinct problems. First: teams often reach for the wrong vehicle entirely. Second: even with the right vehicle, the route planning determines whether packages arrive.

Most edge deployments handle predictive maintenance, anomaly detection, sensor classification, and quality control. These are tabular data problems. A NeurIPS 2022 paper confirmed what practitioners already suspected: tree-based models such as XGBoost and Random Forests outperform deep learning on tabular data across 45 benchmark datasets. A study on industrial IoT found that XGBoost achieved 96% accuracy in predicting factory equipment, reducing downtime by 45%. Random forests hit 98.5% on equipment failure classification.

TensorFlow Lite Micro fits in 16KB. TinyML gesture recognition runs at 138KB and 30 FPS. These aren't compromised models. They're right-sized for their problems. The e-bike, not the truck.

But here's what matters more: whether you're dispatching e-bikes or trucks, you need routes that work. And route planning is why 70% of Industry 4.0 AI projects stall in pilot.

The models work in demos. Deployment breaks them.
The Orchestration Gap
In the early days of Kubrenetes, we made a version of this mistake. We thought container scheduling was the hard part. The hard part was everything after scheduling: networking, storage, observability, updates, rollbacks. The entire operational lifecycle.

Edge ML is learning this lesson now. Where MLOps ends with a packaged model, orchestration begins. And orchestration is where edge ML goes to die.

Think about what makes delivery logistics hard. It's not the vehicles. It's coordinating thousands of them across changing conditions. And when you're operating with vehicles/artifacts that are too big for the use case, you're just making everything harder.

Model staleness hits regardless of model size. Edge models, once deployed, might not be frequently updated. A classifier trained on 2024 patterns doesn't recognize 2025 anomalies. Rolling out updates across thousands of devices is nontrivial, whether you're pushing 50KB or 5GB. This is the equivalent of route maps that don't know about new neighborhoods. Your drivers show up, but they can't find the addresses.

Fleet heterogeneity compounds everything. Devices don't update uniformly. You end up managing fragmented fleets in which different nodes run different model versions with varying capabilities. Cloud deployments update in minutes. Edge deployments take weeks, sometimes months. Some devices never update at all. Imagine dispatching trucks, vans, and bikes from the same warehouse without a unified system that tracks which vehicle has which capabilities. Version skew creates subtle bugs that only manifest at scale.

Energy constraints create hard limits that benchmarks ignore. Thermal throttling kicks in when you stress mobile CPUs. Even a small model running continuous inference drains batteries and generates heat. Academic papers report Joules per prediction. Users report their phone dying by 2pm. The diesel cost that accumulates invisibly on every route, whether you're carrying one envelope or a hundred boxes.

Network variability breaks every cloud-native assumption. Traditional MLOps assumes stable, high-bandwidth connections. That assumption doesn't hold when inference pipelines need to survive outages, intermittent connectivity, or bandwidth that costs real money. What happens when your edge device goes offline for a week? When does it reconnect with stale models and queued data? It's like planning routes that assume every road is always open. The moment a bridge closes, your whole system breaks.
The Data Pipeline Problem
This is where "feels free" really isn't free. Edge ML isn't failing because models are too big. It's failing because data pipelines weren't designed for bidirectional flow.

Delivery networks learned this decades ago. Packages flow out from warehouses, but returns flow back. Damage reports flow back. Delivery confirmations flow back. Reverse logistics are just as important as forward logistics, and often harder.

The traditional ML assumption:
Edge Device → Cloud → Inference → Response

What edge ML actually needs:
Edge Device ↔ Local Inference ↔ Selective Sync ↔ Model Updates ↔ Back to Edge

This bidirectionality creates problems most teams don't anticipate. As IBM's edge deployment guide notes: "In an edge deployment scenario, there is no direct reason to send production data to the cloud. This may create the issue that you'll never receive it, and you can check the accuracy of your training data. Generally, your training data will not grow."

Your model improves based on the data it sees. If that data never leaves the edge, your model never improves. But if all data goes to the cloud, you've rebuilt the centralized architecture you were trying to escape, with extra latency and bandwidth costs. It's like routing every return through your main distribution center instead of handling them at local hubs. Technically correct, but operationally a nightmare.

Edge-to-cloud ETL pipelines are emerging as critical infrastructure. They need real-time ingestion, adaptive transformation, graceful degradation when connectivity fails, and respect for data sovereignty constraints. A 50KB model and a 5GB model face identical challenges here. The pipeline doesn't care about parameter count, just like the route doesn't care whether you're driving a truck or riding a bike.
What Actually Works
The teams succeeding with edge ML have stopped optimizing vehicles and started optimizing routes.

Tiered inference separates quick decisions from complex reasoning. Vector search at the edge runs in 5-10ms using in-memory indexes. No GPU required. Simple classifications and caching happen locally. Complex reasoning routes selectively when network allows. This is the e-bike for last-mile delivery, the truck for bulk warehouse transfers. Match the vehicle to the delivery, not the other way around.

Edge MLOps mirrors replicate minimal cloud capabilities locally. When the network disappears, edge nodes still manage model lifecycle, handle updates from local cache, and queue telemetry for later sync. This approach acknowledges what cloud-native architectures ignore: networks fail. Devices go offline. The question isn't whether your deployment loses connectivity. It's whether it keeps working when it does. Local dispatch centers that function when headquarters goes dark.

Data locality as first principle means processing where data lives, not where servers are convenient. By 2025, over 50% of enterprise data will be processed at the edge, up from 10% in 2021. This shift is already happening in manufacturing, retail, healthcare, and logistics. Organizations adapting successfully treat edge deployment as first-class infrastructure, building intelligent data orchestration that moves compute to data rather than data to compute. Deliver from the nearest warehouse, not the central hub.

Selective synchronization solves the training data problem. Not all edge data needs to reach the cloud, but representative samples do. Anomalies do. Edge cases that challenged local models do. Smart filtering at the edge, with policies that adapt based on model confidence and data novelty, keeps training pipelines fed without overwhelming bandwidth or centralized storage. Send back the damage reports. Don't send back confirmation that every package arrived fine.

This is exactly why we built Expanso around data orchestration rather than model serving. The model isn't the bottleneck, whether it's a 50KB decision tree or a 4GB quantized LLM. The bottleneck is getting the right data to the right place at the right time, coordinating updates across heterogeneous fleets, and maintaining observability when half your nodes are intermittently connected. Our approach treats edge nodes as first-class participants in data pipelines, not afterthoughts bolted onto cloud architectures. Route planning, not vehicle engineering.
Where This Is Heading
$378 billion in projected edge computing spending by 2028. IDC expects edge AI deployments to grow at 35% CAGR over the next three years. That investment isn't going into building better trucks. The quantization problem is largely solved. The money is going into the logistics layer that makes edge deployment actually work.

Federated learning is moving from research curiosity to production requirement. It's the only practical way to improve models from edge data without centralizing that data, solving the training feedback loop that IBM's guide warned about. Standardized edge-cloud orchestration protocols are emerging to simplify deployment across heterogeneous environments. The security surface is expanding dramatically as AI distributes across thousands of devices rather than sitting in secured data centers.

The companies navigating this successfully aren't the ones with the smallest vehicles or the fastest engines. They're the ones who recognized early that vehicle optimization was table stakes, not competitive advantage. The hard problems were always about fleet management, route planning, package tracking, and graceful degradation when conditions change.
The Right Questions
Not "how do I compress my neural network to fit on edge hardware?"

Start with "what's the simplest model that solves my actual problem?" For sensor data, that's often a decision tree. Kilobytes, not gigabytes. Proven to outperform neural networks on tabular data. For language tasks, yes, you need transformers. Adobe's SlimLM shows what's possible: 125M-1B parameters, document assistance on smartphones.

Then ask "can my infrastructure actually deploy and maintain this?" Can you push updates to a fragmented fleet? Can your edge nodes operate when disconnected? Does your data pipeline support bidirectional flow? Can you monitor inference quality across thousands of distributed nodes?

The size obsession missed the point twice: once by reaching for complex models when simple ones work better, and again by focusing on compression when deployment was the actual bottleneck.

UPS isn't going to start delivering envelopes on e-bikes anytime soon. The truck infrastructure exists. The routes are planned. The drivers are trained. Switching has costs.

But if you're building edge ML from scratch, you get to choose. You can build the truck fleet because trucks are what serious logistics companies use. Or you can look at what you're actually delivering and pick the right vehicle for the job.

A 50KB model that deploys beats a 50MB model that doesn't. But even an e-bike needs a route that works.

The edge isn't where ML projects go to die. It's where the logistics need to grow up.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.*

Originally published at Distributed Thoughts.

Emergence vs. Engineering: The Industry Just Bet Against the God Model

David Aronchick — Sat, 13 Dec 2025 18:09:33 +0000

Monday, OpenAI, Anthropic, Google, Microsoft, and AWS jointly donated their agent infrastructure to the Linux Foundation. If any of them actually believed a single model would achieve AGI in 2-3 years, this would be the dumbest move in corporate history.

You don't standardize the plumbing when you're about to build God.

The Agentic AI Foundation launched with three projects: Anthropic's Model Context Protocol (MCP) for connectivity, Block's goose for execution, and OpenAI's AGENTS.md for instructions. Together they form a complete stack for building composable AI systems, many specialized tools working through standard interfaces.

This isn't a technical footnote. It IS a recognition that no one is going to be able to do it all themselves.

For many MANY years, we've tried to engineer general intelligence from first principles. The results are impressive but bounded. This week, you could argue, the AI industry quietly bet on a different approach: letting intelligence emerge from simpler components.
The Physics Problem
Tim Dettmers published "Why AGI Will Not Happen" the day after the MCP announcement. His argument is remarkably clear.

"Computation isn't abstract. It happens in silicon, constrained by the speed of light, thermodynamics, and the square-cube law. Moving global information to local neighborhoods scales quadratically with distance. Memory becomes more expensive relative to compute as transistors shrink. "If you want to produce 10 exaflops on a chip, you can do that easily," Dettmers writes, "but you will not be able to service it with memory."

GPUs maxed out their performance-per-dollar around 2018. The gains since then came from one-off features: 16-bit precision, Tensor Cores, HBM, 8-bit quantization, 4-bit inference. Those tricks are exhausted. Dettmers estimates maybe one or two more years of meaningful scaling improvements before we hit the wall.

The transformer architecture itself is already near physically optimal. There doesn't appear (BUT I HAVE BEEN WRONG MANY TIMES BEFORE) to be a clever redesign waiting in the wings to unlock another order of magnitude.

Superintelligence? Fantasy. Recursive self-improvement still obeys scaling laws. An AI improving itself faces the same diminishing returns as engineers improving it externally. You're filling gaps in capability, not extending the frontier.

If you can't engineer your way to general intelligence through scale, what's the alternative?

The same thing that produced intelligence in nature: emergence.
More Is Different
In 1972, physicist Philip Anderson published "More is Different" in Science. It became one of the most cited papers in complexity research and helped establish the Santa Fe Institute.

Anderson's argument was profound: reductionism doesn't imply constructionism. You can break a system down into its fundamental parts, but you cannot rebuild complex behavior by assembling those parts. "At each new level of complexity," he wrote, "entirely new properties appear."

Consciousness isn't hiding in neurons. Traffic patterns don't exist in individual cars. The economy isn't a property of any single transaction. These phenomena emerge from interactions between simpler component, and they can't be predicted or engineered from first principles.

This isn't mysticism. It's how complex systems actually work.

The Santa Fe Institute defines emergence as "properties at one scale that are not present at another scale." Complex adaptive systems share common features: many agents, each intelligent and adaptive within their domain, none possessing complete information about the whole. Global patterns arise from local interactions without central control.

You don't engineer emergence. You create conditions for it.
The Ant Colony Test
Deborah Gordon at Stanford has spent decades studying ant colonies. Her description of individual ants is memorable: "I probably wouldn't hire them."

And yet collectively, ants build complex nests, find food sources efficiently, coordinate defense, and adapt to changing environments. Zero central control. The queen doesn't manage; she reproduces. As Gordon puts it, "Tasks allocate workers, rather than a manager allocating tasks to workers."

The mechanism is stigmergy: coordination through environmental modification. Ants leave pheromone trails that influence other ants' behavior. Simple rules at the individual level (follow strong trails, lay pheromones when successful)produce sophisticated collective intelligence.

Gordon draws the parallel explicitly: "In many ways, understanding the behavior of ant colonies could teach us about the way billions of relatively simple neurons work together in our brains."

The brain follows the same pattern. Neurons aren't conscious. They fire or don't fire based on local inputs. Consciousness emerges from billions of these simple interactions. There's no central "intelligence unit" directing traffic, no homunculus watching the show.

Decentralized control. Simple rules. Local interactions producing global behavior. Resilience through redundancy. Adaptation without central planning.

This is how nature builds intelligence. Not by engineering a god, but by enabling a swarm.
The Pattern Repeats
The internet works the same way.

David Clark's 1988 paper on DARPA's design philosophy reveals remarkably minimal assumptions: the network can transport a datagram with reasonable, not perfect, reliability. That's it. Everything else emerges from endpoints following simple protocols.

TCP/IP split responsibility deliberately. Keep IP simple and flexible. Push complexity to the edges. "Fate-sharing" means intelligent endpoints, dumb pipes. The result: a decentralized system that scaled beyond anyone's imagination and survives failures that would destroy centralized alternatives.

Unix philosophy follows the same template. Ken Thompson and Doug McIlroy: "Make each program do one thing well. Expect the output of every program to become the input to another." Small tools, standard interfaces, emergent capability from composition.

Nobody said "let's build one giant program that does everything." That was the mainframe mentality, and it lost.

I watched this pattern win with Kubernetes. We didn't build bigger VMs. We built smaller containers with standard interfaces and let orchestration handle the complexity. The sophisticated behavior emerged from composition, not from engineering a monolith.
What MCP Actually Means
The MCP donation makes sense through this lens.

With 97 million monthly SDK downloads and adoption by Claude, ChatGPT, Gemini, Microsoft Copilot, Cursor, and VS Code, MCP has become TCP/IP for AI agents: the standard protocol for connecting models to tools, data, and services.

David Soria Parra, MCP's lead maintainer: "The main goal is to have enough adoption in the world that it's the de facto standard."

Nick Cooper from OpenAI: "We need multiple protocols to negotiate, communicate, and work together to deliver value for people, and that sort of openness and communication is why it's not ever going to be one provider, one host, one company."

Read that again. OpenAI's own engineer saying it's not ever going to be one company.

When your fiercest competitors agree on a protocol, they're hedging. They're building for a world where no single system wins. They're betting on emergence over engineering.
The Honest Assessment
This doesn't mean AI won't be transformative. It means the path isn't "scale until AGI."

It's: build composable tools, let emergence do the heavy lifting.

Dettmers contrasts the US "winner-take-all" philosophy (that is betting everything on frontier models) with China's "economic diffusion" approach, integrating AI capabilities throughout the economy. The diffusion strategy doesn't require AGI. It requires useful, composable tools that produce emergent value when combined.

The MCP ecosystem is infrastructure for exactly this. Specialized agents handling narrow tasks, connected through standard protocols, producing collective intelligence that no individual component possesses.

Ant colonies. Neural networks. The internet. Unix. Kubernetes. Now AI agents.

The pattern keeps winning because it's how complexity actually works.
The Kicker
Fifty years ago, Philip Anderson argued that you can't construct complexity from simple parts through pure engineering. Emergence requires different tools, different thinking. You don't build intelligence; you create conditions for it to arise.

This week, the AI industry admitted he was right.

When OpenAI, Anthropic, Google, Microsoft, and AWS all agree on something, pay attention. They're not building for a world where one model solves everything. They're building for emergence.

The god model was always a fantasy.

The swarm is real.

Originally published at Distributed Thoughts.

🎄 On the First Day of Debugging: The Twelve Characters of Christmas

David Aronchick — Fri, 12 Dec 2025 06:12:14 +0000

🎵 On the first day of debugging, production gave to me:
An emoji that broke awk's field count tree 🎵
A Holiday Horror Story
Friday morning. Coffee in hand. You commit a documentation change before the holiday break. You've decided to get EXTRA cool, nothing fancy, just adding some friendly emoji to your metrics. The GitHub Actions workflow fails:
Error: Invalid format '100'
Expected: number
Got: string '100'

Four hours later (goodbye, early weekend), you've discovered that 📊 breaks awk '{print $3}' because emoji count as fields and your clever metric extraction just imploded.

Welcome to production, where every character is a potential landmine wrapped in festive paper.

This holiday season, let me gift you the knowledge of The Twelve Characters of Christmas. Twelve special characters that will ruin your week, test your patience, and teach you why pipelines are terrible at telling you what's actually wrong.

Think of this as your technical advent calendar. Behind each door: a character that breaks things in fascinating ways.
🎁 The First Gift: The Emoji 📊
What you unwrapped: Emojis have variable width.

The problem wasn't the emoji itself; it was the assumption that field extraction works on visual spacing.

What we see:

📊 Metric: 100%

What awk sees with '{print $3}':

Field 1: 📊
Field 2: Metric:
Field 3: 100%

After adding emoji:

📊 📈 Metric: 100%

Now awk sees:

Field 1: 📊
Field 2: 📈
Field 3: Metric:
Field 4: 100% # Wrong field! 🎁

Your gift that keeps giving:
String length lies ("📊".length = 2 in JavaScript, not 1)Byte count ≠ character countSorting alphabetically becomes... interestingEvery string operation you thought you understood is now probabilistic
The fix: awk '{print $NF}' always grabs the last field.
🎁 The Second Gift: The Zero-Width Joiner ‍
What you unwrapped: Authentication bypass wrapped in invisibility.

These characters are invisible glue between emoji, but they work anywhere:
"hello" vs "he‍llo"

Those look identical. They're not. The second has U+200D (zero-width joiner) between the 'e' and 'l'.

Your gift that keeps giving:
user_input = "admin‍" # has ZWJ at end
if user_input == "admin": # nope! 🎁
grant_access()

Also fails:

db.query("SELECT * FROM users WHERE username = ?", user_input)

Authentication bypass via invisible character. Your WAF can't see it. Your logs look fine. Your security audit finds nothing.

The discovery: Someone's "username not found" ticket escalates to a database investigation revealing two "identical" usernames with different bytes.
🎁 The Third Gift: Right-to-Left Override ‮
What you unwrapped: A trojan horse with a bow on top.

Unicode includes directionality controls. U+202E reverses text rendering:
Filename: "image.txt‮gpj.evil"
Displays as: "image.txtlive.jpg"
Actual bytes: "image.txt[RLO]live.jpg"

Your gift that keeps giving:

Your file viewer shows a JPG. Your security scanner checks JPG extensions. Your pipeline processes a JPG. What actually executes? evil.txt 🎁

This isn't theoretical—Trojan Source attacks use this for malicious code injection that passes code review because it looks fine on screen.

The discovery: Your image processor starts executing arbitrary code. Merry Christmas!
🎁 The Fourth Gift: The Null Byte \0
What you unwrapped: The gift of dual realities.

In C, \0 terminates strings. In everything built on C (which is everything), null bytes create two parallel universes:

What you check:

filename = "safe.txt\0../../etc/passwd"
if filename.endswith(".txt"): # True! 🎁
process_file(filename)

What actually happens:

String terminates at \0

Opens: "safe.txt"... or does it?

Your gift that keeps giving:
-- What you think you're querying:
SELECT * FROM files WHERE name = 'safe.txt\0'

-- What the SQL parser sees:
-- String terminated, rest ignored 🎁

SQL injection's sneaky cousin. Your input validation passes. Your database dies. Your logs show "safe.txt" and nothing suspicious.

The discovery: After your security audit, when the penetration tester shows you what they did.
🎁 The Fifth Gift: The BOM
What you unwrapped: The invisible file prefix that breaks everything.

UTF-8 files sometimes start with U+FEFF (byte order mark). It's invisible in most editors. It destroys your scripts:

!/bin/bash

echo "Hello World"

Looks fine. Won't execute:
$ ./script.sh
bash: ./script.sh: /bin/bash: bad interpreter: No such file or directory 🎁

Your gift that keeps giving:

The shebang is actually #!/bin/bash. Different bytes. Bash doesn't recognize it.

CSV files with BOM? Your parser thinks the first column is named "id" instead of "id". Joins fail mysteriously. Someone suggests "just trim whitespace" and it still doesn't work because there is no whitespace.

The discovery: After you've checked file permissions, reinstalled bash, rebooted the server, and finally run xxd script.sh to see the bytes.
🎁 The Sixth Gift: The Soft Hyphen
What you unwrapped: The invisible line-break suggestion.

U+00AD is a "suggestion to break here if needed." Invisible until line-wrapping occurs:
"supercalifragilistic" appears as "supercalifragilistic"

But:
"supercalifragilistic".includes("cali") // true
"supercalifragilistic".includes("cali") // false! 🎁

Your gift that keeps giving:

Copy-paste from a website into your pipeline, and suddenly:
Searches failLog aggregation misses matchesDeduplication creates duplicatesYour debugging session: "I can literally SEE the string matches. Why isn't it finding it?"
The discovery: After you paste the same text directly into your code, and THAT works.
🎁 The Seventh Gift: Turkish İ
What you unwrapped: The gift of internationalization nightmares.

Turkish has four 'i' letters: i, ı, İ, I. Case conversion depends on locale:
"file".upcase # "FILE" in English
"file".upcase # "FİLE" in Turkish (tr_TR locale) 🎁

"FİLE".downcase # "fi̇le" (with combining dot)

Your gift that keeps giving:

Your case-insensitive comparison just became locale-sensitive. Germans debate whether ß.upcase should be "SS" or "ẞ". Greeks have three lowercase sigmas (σ, ς, Σ).

File systems differ:
Windows: case-insensitive, case-preservingmacOS: optionally case-sensitiveLinux: case-sensitive always
The discovery: Your pipeline works in dev (macOS), breaks in staging (Linux), works differently in production (Windows containers)..
🎁 The Eighth Gift: The Combining Accent ́
What you unwrapped: Two ways to write the same letter.

Unicode has two ways to write é:
é # U+00E9 (single character)
é # U+0065 + U+0301 (e + combining acute accent) 🎁

Your gift that keeps giving:

Visually identical. Different bytes. macOS normalizes to decomposed (NFD). Linux doesn't (NFC).

Create file on macOS:

touch café.txt # stored as cafe\u0301.txt

Access from Linux:

ls -l café.txt # File not found 🎁

Why?

$ ls -lb
-rw-r--r-- 1 user staff 0 Nov 24 10:00 cafe\314\201.txt

The discovery: Your cross-platform pipeline mysteriously loses files between systems.
🎁 The Ninth Gift: The Surrogate Pair 💩
What you unwrapped: Emoji that need TWO UTF-16 code units.
"💩".length // 2, not 1 🎁
"💩"[0] // � (invalid Unicode)
"💩".substring(0, 1) // � (broken character)

// Counting is hard:
[..."Hello 💩 World"].length // 13 (correct)
"Hello 💩 World".length // 14 (wrong)

Your gift that keeps giving:

Your pipeline truncates strings at byte boundaries. Emoji get cut in half. JSON becomes invalid. Logs show question marks. Everyone blames the database collation.

Reversing destroys emoji:

"Hello 💩 World"[::-1] # "dlroW �� olleH" 🎁

The discovery: When a customer complains that their emoji-filled messages look like gibberish.
🎁 The Tenth Gift: The Homoglyph а
What you unwrapped: Letters that look identical but aren't.

Latin 'a' (U+0061) and Cyrillic 'а' (U+0430) look identical. Different bytes:
twitter.com # real
twіtter.com # Cyrillic 'і' (U+0456) 🎁

Your gift that keeps giving:
expected = "admin"
actual = "аdmin" # First character is Cyrillic

if actual == expected: # False! 🎁
grant_access()

But to humans reading logs:

print(f"Login attempt: {actual}") # looks like "admin"

Your domain validation passes. Your email verification passes. Your phishing attack succeeds.

The discovery: During your security incident post-mortem.
🎁 The Eleventh Gift: The Newline \n
What you unwrapped: Three standards for ending a line.
Unix: \n (LF)
Windows: \r\n (CRLF) 🎁
Old Mac: \r (CR)

Your gift that keeps giving:
"line1\r\nline2\r\nline3".count('\n') # 2 (wrong)
"line1\r\nline2\r\nline3".splitlines() # 3 (right) 🎁

Git helpfully converts line endings based on .gitattributes. Now every line in your diff is "changed." Your PR is 10,000 lines. The actual change was one word.

Your pipeline hash-checks files for integrity. Same content, different line endings, different hashes. False positive failure alert at 3am.

The discovery: When your coworker opens your file and their editor "fixes" all the line endings.
🎁 The Twelfth Gift: The Tab \t
What you unwrapped: The final gift—visual width that lies.

These look identical:

"key: value" # 4 spaces
"key:\tvalue" # 1 tab character 🎁

But:

"key: value".split('\t') # ['key: value']
"key:\tvalue".split('\t') # ['key:', 'value']

Your gift that keeps giving:

YAML treats tabs and spaces differently. One is indentation. One is death:
config:
key: value # spaces, valid
config:
key: value # tab, invalid YAML 🎁

Your config looks fine in your editor (which converts tabs to spaces). Your pipeline reads the raw file (which has tabs). Your deployment fails with "invalid YAML" and the error message points to a line that looks perfectly fine.

The discovery: After you copy-paste the "broken" YAML into a validator and it works fine.
🎄 What All Twelve Gifts Have in Common
Notice what these twelve characters share:
Invisible to humans: Your eyes can't distinguish them*ASCII-land works fine: English text with basic punctuation passes every timeDeterministic but unpredictable: Given the character, it always fails the same way, but you have no idea which characters your pipeline will encounterProduction discovery: Test data is sanitized, user input is chaosBinary debugging: Remove pieces until something changes, like unwrapping boxes to find which gift is broken
This isn't fundamentally about Unicode complexity. It's about **pipeline observability*.
🎁 The Real Problem (Unwrapped)
Your pipeline has stages:
[Input] → [Parse] → [Transform] → [Validate] → [Store]

Each stage has opinions about text encoding. None agree:
Input accepts UTF-8Parse assumes ASCIITransform uses locale-sensitive operationsValidate checks byte lengthStore expects UTF-8 but doesn't verify
When something breaks, you get: Error: Invalid format '100'

What you need:
Stage: Parse
Input bytes: [F0 9F 93 8A 20 4D 65 74 72 69 63 3A 20 31 30 30 25]
Encoding: UTF-8
Character count: 13
Byte count: 17
Field extraction: Expected 3 fields, got 4
Problem character: U+1F4CA (📊) at position 0
Suggestion: Use byte-based parsing or normalize input

But you don't get that. You get silence until complete failure.
🎁 Three Gifts for Better PipelinesGift 1: Declare Encoding Everywhere# Not this:
with open('file.txt') as f:
data = f.read()

This:

with open('file.txt', encoding='utf-8', errors='strict') as f:
data = f.read()

errors='strict' means fail immediately on invalid bytes. Don't guess. Don't substitute. Fail with the exact byte position.
Gift 2: Normalize at Boundariesimport unicodedata

def sanitize_input(text):
# Pick ONE canonical form and enforce it
normalized = unicodedata.normalize('NFC', text)

# Remove invisibles
visible = &apos;&apos;.join(c for c in normalized 
                  if unicodedata.category(c)[0] != &apos;C&apos;)

# Verify what&apos;s left
try:
    visible.encode(&apos;ascii&apos;)  # Or &apos;utf-8&apos;, be explicit
except UnicodeEncodeError as e:
    raise ValueError(
        f&quot;Invalid character at position {e.start}: &quot;
        f&quot;{repr(visible[e.start])}&quot;
    )

return visible

Gift 3: Make Intermediate State Visible
Between pipeline stages, log:
Byte count vs character countEncoding declarationCharacter categories presentSample of problematic characters
Your monitoring should show:
"Stage 2 processed 1M records, 47 contained non-ASCII, 3 contained control characters, 0 validation failures."

Not: "Stage 2 complete."
🎄 The Holiday Season Lesson
Every data pipeline has two modes:
Development: All data is well-formed (like your holiday wish list)Production: The real world exists (like your actual gifts)
There's no smooth transition. Your pipeline either handles emoji, zero-width joiners, null bytes, and right-to-left overrides, or it silently corrupts data until someone notices.

These twelve characters didn't cost you hours of debugging because character encoding is hard. They cost you those hours because your pipeline's feedback loop is binary: success or mysterious failure, nothing in between.

Design-time testing uses sanitized data. Production sends you the real world. The gap between them is where you spend your Friday afternoon debugging why 📊 appears in your error message instead of going to your holiday party.
🎵 And a Pipeline That Actually Works Reliably 🎵
Happy Holidays from everyone at **Expanso.io. May your deployments be clean, your pipelines be observable, and your error messages be specific.

P.S. If you're reading this during the December code freeze/during a holiday week/when every one else is drunk on "egg-nog" : I'm sorry. The Further Reading section below might help. Or at least commiserate.
Further Reading (Stocking Stuffers)Trojan Source: Invisible Vulnerabilities - RLO attacks in real codeThe Absolute Minimum Every Software Developer Must Know About Unicode - Joel SpolskyUTF-8 Everywhere - Why UTF-8 should be your defaultUnicode Security Considerations - Official security implicationsThe Turkey Test - Case conversion nightmare
Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.

Originally published at Distributed Thoughts.

Multi-Everything: Why Your Data Strategy Is Harder Than Your Cloud Strategy

David Aronchick — Tue, 09 Dec 2025 12:13:43 +0000

An Uber engineer gave a great talk at Kubecon I have wanted to write about: “...we end up having to think about use cases that can either reside entirely within one cloud provider so that I can put training and serving together, or I need to think about the use cases where it makes sense to actually pull the data from one provider to another, in order to facilitate being able to leverage that compute. It doesn’t make it quite as seamless as it could be, and you have to be purposeful in how you think about what workloads you’re going to be converging together.”

Two sentences. They explain why multicloud is complicated. It works! But it's not just "spread stuff everywhere and balance."

And when you're thinking about it for yourself, put this in context. Uber has dedicated platform engineering teams, specialized GPU infrastructure groups, and the budget to build custom observability solutions. They still struggle with this. If you're not dedicated just as many resources to address these challenges, they reality of the complexity is going to hit you even harder.
The Conversation We've Been Having
For the past decade, the infrastructure industry focused relentlessly on making compute portable. Kubernetes runs anywhere. Containers abstract away the underlying machine. We've built elaborate systems to ensure that a workload running in AWS can, in theory, run identically in GCP or Azure.

And it worked! The compute layer genuinely is more portable than it was in 2015.

But what Uber was talking about here reveals a the immutable truth. Compute can PRETTY READILY between clouds because the commodity layer is portable. But when it comes to running in multiple locations, it's not the containers you need to worry about, it's the data.
Why Data Gravity Wins
Data gravity, the phenomenon where data accumulates mass and attracts applications toward it, isn't a new concept. Dave McCrory coined the term back in 2010, and there have been many other great pieces covering it. But the implications have become dramatically more severe as AI workloads have grown.

Uber's engineering teams maintain a data lake on one cloud provider. They run inference workloads on a different provider. Training happens somewhere else entirely. Each choice was rational in isolation; optimize for the best GPU availability here, the best storage economics there.

The result? "You have to be purposeful in how you think about what workloads you're going to be converging together."

That's a diplomatic way of describing a constraint that dominates every architectural decision. When considering whether a use case can leverage GPUs efficiently, the first question isn't "do we have the compute?" It's "where does the data live, and what does it cost to move it?"

This isn't a Kubernetes problem. It's not even really a cloud provider problem. It's physics meeting economics.

Moving a petabyte of training data from one cloud to another isn't JUST a technical challenge, it's a business calculation. Egress fees, while expenalone can run into six figures. AWS charges $0.09 per GB for the first 10 TB transferred to the internet. Do the math on a 50TB training dataset and you're looking at $4,500 just in network transfer—before you've stored anything, processed anything, or extracted a single insight.

Add latency considerations for real-time inference. Add compliance requirements that may prohibit certain data from crossing certain boundaries. Add the simple fact that a model optimized with TensorRT for a specific NVIDIA GPU configuration doesn't just "run" on different hardware.

The container is portable. Everything the container needs is not.
You Don't Have Uber's Teams
Uber's response to these challenges involves building custom metrics APIs to abstract away GPU vendor differences. Their teams revamped their entire observability stack when they discovered that CAdvisor-based GPU metrics didn't support newer models. They're actively working on making GPU capacity more fungible across clusters.

They have the engineering headcount to do this. Data quality issues are consistently cited as a primary cause of AI-project failure, and when your data lives in multiple places with different access patterns, different compliance requirements, and different cost structures, maintaining quality becomes exponentially harder.

Uber has spent 15+ years building dedicated teams to solve this problem. Most organizations are asking their existing platform engineers to figure it out alongside everything else they're already doing.
The Cluster Was the Wrong Abstraction
Uber made an observation that cuts to the heart of how we've been thinking about infrastructure: "We ended up with a Kubernetes infrastructure focused on batch and a Kubernetes infrastructure focused on microservices... The distinction has been that the hardware was segregated at the cluster level."

Their teams built dedicated GPU clusters for GPU workloads. Dedicated CPU clusters for CPU workloads. The cluster became the organizing principle for hardware allocation.

This made sense when clusters were the primary unit of deployment. But it created silos. GPU capacity sat isolated from CPU capacity. When GPU clusters were underutilized, that capacity couldn't easily flow to other workloads. When CPU-bound services accidentally landed on GPU nodes, expensive hardware sat wasted running authentication checks.

"We've been over-indexing on a Kubernetes cluster as an abstraction for hardware," Uber noted, "rather than leveraging a lot of what we can do internally from Kubernetes itself."

The cluster was supposed to abstract away infrastructure complexity. Instead, it became another boundary; another wall between resources that could, in principle, be fungible but in practice are not.

If this is happening at Uber, with their dedicated platform teams and infrastructure budgets, what does your cluster architecture actually look like?
GPUs Make Everything Harder
The challenges with multi-cloud compute get significantly worse when GPUs enter the picture. CPUs are MOSTLY fungible (the entire stack may not be, but it's fairly close). An x86-compatible chip in AWS behaves essentially the same as a x86-compatible chip in Azure (MOSTLY). You can pack multiple workloads onto a single CPU. Failovers are straightforward.

None of this applies to GPUs.

"GPU workloads aren't quite as fungible as CPU workloads. I can't as easily just dynamically pack eight workloads onto one GPU now, where I could have just squeezed things onto a single CPU."

Choosing the right GPU for training versus inference is already complex enough when you're working with a single provider. Training requires massive compute throughput and high memory bandwidth. Inference optimizes for latency and cost-per-query. The hardware choices are fundamentally different.

Now multiply that complexity across providers. Different cloud GPU offerings have different availability, different pricing models, different networking characteristics. An H100 on AWS isn't quite the same as an H100 on GCP when you factor in interconnect speeds, memory configurations, and the software stack surrounding it.

And worse, disaster recovery math changes even further. With CPUs, you might provision 20% overhead for failover capacity. With GPUs, given their cost, their scarcity, and the fact that models are often optimized for specific hardware configurations, that overhead becomes genuinely painful to justify.

If the workload was optimized for this GPU, with this memory configuration, using this specific NVIDIA architecture. Moving it isn't just a scheduling decision; it's potentially a retraining decision.
Observability in a Multi-Vendor World
One thing that PARTICULARLY stood out for me was observability. In Uber's case, they are exploring building their own metrics API to abstract away GPU vendor differences.

Why? Because they use NVIDIA hardware but are also evaluating AMD. Each vendor exposes different metrics. Teams built dashboards around low-level Cadvisor metrics that don't even support newer GPU models. When they tried to migrate to updated metrics, they discovered the entire organization had built dependencies on the old metric set.

"You're going to end up with a mix of a variety of different metrics and with nuances about what each of them means." They're now trying to build "metrics almost as an API" => a platform-level abstraction that can source data from vendor-specific implementations without requiring every team to understand GPU model differences.

This is the kind of problem that doesn't show up in multicloud architecture diagrams. It's the accumulated weight of real decisions made by real teams trying to get actual work done.

And again: Uber has dedicated teams building custom solutions for this. What's your plan?
The Real Problem: Multi-Everything
What Uber's experience reveals is that "multicloud" was always the wrong frame for this conversation.

The challenge isn't running compute across multiple cloud providers. Kubernetes solved that. The challenge is that modern AI workloads exist in a multi-everything environment:

Multi-region: Data generated in Europe may have different residency requirements than data generated in Asia. Training might happen in a region with GPU availability. Inference might need to happen close to users.

Multi-provider: Not just AWS vs. GCP vs. Azure, but also on-premises data centers that still hold sensitive datasets, edge locations that generate real-time data, and specialized AI clouds that offer unique hardware.

Multi-compliance-zone: Regulatory boundaries don't align with cloud provider boundaries. GDPR, HIPAA, financial regulations, and industry-specific requirements create a patchwork of constraints that have nothing to do with where your Kubernetes clusters run. Some EU member states have enacted additional residency requirements beyond GDPR for specific sectors like healthcare and public services.

Multi-format: Data lakes, data warehouses, streaming platforms, feature stores, vector databases. Each optimized for different access patterns. Each with its own replication and consistency guarantees.

Over 80% of enterprises with multicloud environments experience interoperability and connectivity problems. The compute layer was the easiest part of this puzzle to solve. We've been celebrating that victory while the harder problems compound.
What Comes Next
Uber's agentic AI workflows are still experimental, representing a minority of GPU usage. But they noted that if they "unlock some agentic workflow and put it everywhere," it would represent "a considerable increase in what they need to support with GPUs."

That's the trajectory the entire industry is on. More AI workloads. More models. More demand for training and inference capacity. And every one of those workloads will inherit all the multi-everything constraints that already make enterprise architecture so complex.

The industry spent a decade making compute portable. The next decade's problem is fundamentally different: making data accessible without necessarily making it mobile.

That's not a Kubernetes upgrade. It's not a new cloud service. It's a rethinking of how we architect systems when the data—not the compute—is the constraint that matters.

Uber's engineers are living in that future right now, with dedicated teams and substantial budgets to figure it out. The rest of us need to start thinking about how we'll solve the same problems with a fraction of the resources.

Because the data isn't going to move itself. And frankly, given the economics, you probably don't want it to.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.*

Originally published at Distributed Thoughts.