Build a Multi-Agent Data Pipeline in 50 Lines of Neam

Praveen Govindaraj — Wed, 01 Apr 2026 12:38:13 +0000

In this tutorial, you'll build a working multi-agent data pipeline using Neam, an agentic AI programming language. By the end, you'll have a DIO orchestrating five agents through a churn prediction workflow.

Step 1: Define Your Infrastructure Profile. This tells every agent where data lives and what compliance rules apply:

infrastructure_profile MyInfra {
    data_warehouse: {
        platform: "postgres",
        connection: env("DB_URL")
    },
    governance: { regulations: ["GDPR"] }
}

Step 2: Declare Your Agents. Each agent is a specialist. Note the budget constraints:

budget B { cost: 50.00, tokens: 500000 }

databa agent MyBA { provider: "openai",
    model: "gpt-4o", budget: B }
datascientist agent MyDS { provider: "openai",
    model: "gpt-4o", budget: B }
datatest agent MyDT { provider: "openai",
    model: "gpt-4o", budget: B }

Step 3: Wire Up the DIO. The orchestrator coordinates everything:

budget DioBudget { cost: 500.00, tokens: 2000000 }

dio agent MyDIO {
    mode: "hybrid",
    task: "Predict customer churn, identify drivers",
    infrastructure: MyInfra,
    agent_md: "./my_domain.agent.md",
    provider: "openai", model: "gpt-4o",
    budget: DioBudget
}

let result = dio_solve(MyDIO, task)
print(result)

Step 4: Create Your Agent.MD. This is the secret weapon — encode domain knowledge:

## @organization-context
Company: My E-Commerce Co
Scale: 500K customers, 5M orders

## @known-data-issues
- signup_date timezone drift before 2024-03
- Product ratings skew positive (self-reported)

## @agent-preferences
DataScientist: XGBoost for tabular, AUC-ROC metric

Run it: neam-cli run my_pipeline.neam

Welcome to Neam Ecosystem

Neam DIO: Orchestrate 14 AI Agents for Your Data Lifecycle

Praveen Govindaraj — Tue, 31 Mar 2026 06:30:49 +0000

85% of ML Projects Fail.

We Built 14 AI Agents to Fix That.

How the Neam Data Intelligent Orchestrator manages the entire data lifecycle — from requirements to production — with spec-driven agent coordination.

The Number That Should Trouble Every Data Leader
Here is a statistic that should keep every VP of Data awake at night: 85% of machine learning projects never reach production. Not 85% that deliver poor results. Eighty-five percent that never ship at all.

For every six ML initiatives your organization launches, five will consume budget, occupy engineers, generate excitement in steering committees — and then quietly die. The models sit in notebooks. The pipelines rot. The business case gets revisited “next quarter,” which is corporate shorthand for never.

This is not a technology problem. The algorithms work. The cloud scales. The tooling has never been better. The problem is organizational. It lives in the gaps between the business analyst who writes the requirements and the data engineer who builds the pipeline. Between the data scientist who trains the model and the MLOps engineer who deploys it.

These gaps have a name: handoff failures. And they are where data projects go to die.

The Shrimp Tank Insight
In mid-2019, a $70 shrimp tank in a Singapore shop made me rethink how systems should be designed. The shopkeeper explained: no water changes, no filter cleaning. The shrimp eat the vegetation, the vegetation grows back. A self-sustaining ecosystem. You buy it, you keep your hands clean. It just… lives.

💡 Key Insight

That question became the design philosophy behind the Neam DIO: 14 agents, each with a distinct role, each producing outputs that others consume, each making the system stronger simply by doing their job. A data ecosystem that, like that $70 shrimp tank, just… works.

Introducing the Data Intelligent Orchestrator (DIO)
The DIO is the central coordination layer of Neam’s Intelligent Data Organization. It is not a chatbot. It is not a prompt chain. It is a compiled, spec-driven orchestrator that coordinates 14 specialist AI agents across the complete data lifecycle.

The Four-Layer Architecture
LayerAgentsWhat They DoInfrastructureData Agent, ETL Agent, Migration AgentSource discovery, SQL-first warehousing, zero-downtime platform movesPlatformDataOps, Governance, Modeling, AnalystSRE for data, compliance enforcement, architecture intelligence, NL-to-SQLAnalyticalData-BA, DataScientist, Causal, DataTest, MLOpsRequirements, EDA-to-AutoML, causal reasoning, quality validation, production opsOrchestrationDIODynamic crew formation, RACI assignment, 8 auto-patterns, error recovery

Each agent has a defined personality, authority boundary, and trait-based capabilities. The Data-BA Agent is “inquisitive and traceability-obsessed.” The DataTest Agent is “skeptical, adversarial, never rubber-stamps.” The Causal Agent is “correlation-is-not-causation embodied.”

These are not marketing descriptions. They are system prompts compiled into bytecode.

How the DIO Actually Works

Step 1: Task Understanding
When a task arrives — say, “Predict which customers will churn in 90 days and identify the causal drivers” — the DIO classifies the intent and matches it against 8 pre-defined auto-patterns.

Step 2: Crew Formation

Not every task needs all 14 agents. The DIO scores each agent on four dimensions:

Capability match (40%) — Can this agent do the required work?
Cost efficiency (20%) — How much budget does it consume?
Infrastructure compatibility (20%) — Does it work with the declared platform?
Historical performance (20%) — How well has it performed on similar tasks?
For churn prediction, the DIO forms a crew of 7 agents and skips DataOps, Analyst, Modeling, and Migration entirely.

Step 3: RACI Delegation

Every sub-task gets a RACI assignment: who is Responsible (does the work), Accountable (owns the outcome — always the DIO), Consulted (provides input), and Informed (receives results).

Step 4: Execute with Quality Gates

The DataTest Agent — architecturally separated from all builder agents — must approve artifacts before they flow downstream. The agent that trains the model cannot be the agent that validates it. This is a trust boundary.

Step 5: Error Recovery

Retry → Fallback → Graceful Degradation → Human Escalation. Exhaust automated options before involving humans, but involve humans before producing incorrect results.

The Trait System

TraitWhat It MeansAgentsDataProducerCreates data artifactsData Agent, ETL, Migration, Data-BA, DataScientist, DeployDataConsumerReads artifacts from other agentsETL, Modeling, Analyst, DataScientist, Causal, MLOpsCausalReasonerPerforms causal inferenceCausal Agent (exclusively)QualityGatekeeperCan block downstream progressData Agent, DataOps, Governance, DataTest, MLOps

The Causal Agent: The Missing Role

SHAP values tell you which features were important to the model’s prediction. They do not tell you which features cause the outcome. The Causal Agent reveals that “support_ticket_resolution_time” is the actual driver — not “days_since_last_order.” One is chasing symptoms. The other identifies the lever you can actually pull.

The Evidence: DataSims
MetricTraditional TeamNeam Agent StackCost$548,000$34,700Phases completedVaries (often incomplete)7/7Model AUCVaries0.847Test coverageVaries94%ReproducibilityLow100% (50/50 runs)Cost reduction — 93.7%