Forem: Blaine Elliott

State of Data Engineering 2026: Why Data Teams Spend 60% of Their Time Firefighting

Blaine Elliott — Sun, 12 Apr 2026 17:43:27 +0000

It's 9am. You planned to build a new pipeline today. Instead you're debugging why the revenue dashboard shows zeros, tracing a stale table through three upstream dependencies, and explaining to a VP that yesterday's numbers were wrong. By noon you've fixed the fire but built nothing.

This is normal for most data teams. And the 2026 State of Data Engineering Survey (1,101 respondents) now has the numbers to prove it. The interactive explorer lets you query the raw data yourself.

Key findings from the 2026 survey

Before the deeper cut, here's what the survey found across 1,101 data professionals:

82% use AI tools daily (code generation dominates at 82%, documentation at 56%)
42% expect their teams to grow in 2026
43.8% run on cloud data warehouses, 26.8% on lakehouses
90% report data modeling pain points
52.2% say organizational challenges are their biggest bottleneck (vs 25.4% technical debt)

The AI and team growth numbers got the headlines. The time allocation data tells a more important story.

How data engineers actually spend their time in 2026

Two stats from the survey:

34% of time goes to data quality and reliability
26% goes to firefighting

That's 60% of a data engineer's week reacting to problems. Not building pipelines. Not designing models. Reacting.

When asked about their biggest bottleneck, only 10.1% cited data quality. Legacy systems (25.4%), lack of leadership direction (21.3%), and poor requirements (18.8%) all ranked higher.

Data engineers spend most of their time on reactive data quality work but don't identify it as their biggest problem. They've normalized it. Firefighting isn't a crisis. It's the job.

Ad-hoc data modeling doubles firefighting time

The survey's most actionable finding: ad-hoc data modeling (17.4% of respondents) correlates with 38% of time spent firefighting. Teams using canonical or semantic models spend 19%. Half the fires, same job.

But 59.3% of respondents cited "pressure to move fast" as their top modeling pain point, followed by "lack of clear ownership" at 50.7%.

The cycle: pressure to move fast leads to ad-hoc decisions, which create data quality issues, which create fires, which consume the time needed to do things properly. The pressure increases because you're behind.

How to reduce data engineering firefighting

Three things the survey data supports:

1. Assign data quality ownership. 50.7% cited lack of ownership as a top pain point. When quality is everyone's responsibility, it's nobody's responsibility.

2. Invest in data modeling. Teams with canonical models spend half as much time firefighting. The "move fast" pressure is self-defeating when it creates the fires that slow you down.

3. Automate the detection layer. This is the highest-leverage fix for teams that can't reorganize overnight. You can't prevent every schema change, stale table, or anomaly. But you can find out about them in minutes instead of hours.

The difference between a 30-minute fire and a half-day fire is almost always detection speed. A schema change that breaks a pipeline at 2am is a 5-minute fix if you get an alert at 2:05am. It's a 4-hour investigation if the CFO finds it at 9am. (For a deeper look at how this works in practice, see how data freshness monitoring catches stale tables and setting up data quality monitoring for Snowflake and Databricks.)

Automated schema change detection, freshness monitoring, and anomaly alerts compress the gap between "something broke" and "we know about it." That's the gap where firefighting time lives. AnomalyArmor is built specifically for this: monitoring across Snowflake, Databricks, BigQuery, Redshift, and PostgreSQL with alerts in minutes. Email support@anomalyarmor.ai for a trial code.

How to Set Up Data Quality Monitoring in Minutes, Not Hours

Blaine Elliott — Sun, 12 Apr 2026 17:37:55 +0000

You sign up for a data quality tool. You land on an empty dashboard. There's a button that says "Add Connection." You click it, paste your credentials, wait for discovery to finish, and then... nothing obvious to do next.

You poke around. Maybe you find a freshness tab. Maybe you set up an alert. Maybe you close the tab and never come back.

This is how most data observability tools lose customers. Not because the product is bad, but because nobody showed you what to do with it.

We measured the gap. Without guidance, the median time to configure a first freshness monitor in AnomalyArmor was over 40 minutes. With our new guided onboarding, it's under 8. That's the difference between a tool that gets adopted and a tool that gets abandoned during the trial.

TL;DR: AnomalyArmor now has guided onboarding that gets you to your first live data monitor in under 8 minutes. A pre-loaded demo database lets you learn without connecting production. No guesswork, no empty dashboards, no "figure it out yourself."

Why data quality tools have an onboarding problem

Data tools have a unique setup challenge. Unlike a project management app where you create a board and start dragging cards, data observability requires multiple sequential steps before you see any value:

Connect a database
Run schema discovery
Understand what was found
Configure monitoring
Set up alerts
Wait for something to happen

Most users drop off somewhere between steps 2 and 4. They connected their database. Discovery ran. Now there are 200 tables on the screen and no clear next step.

According to Appcues research, 40-60% of users who sign up for a SaaS product will use it once and never come back. For data tools, that number is likely higher because the setup complexity is steeper. Every minute between "signed up" and "seeing value" increases the probability that someone closes the tab and moves on to the next tool in their evaluation.

We decided to fix this.

How AnomalyArmor's guided onboarding works

Instead of dropping you into an empty dashboard, AnomalyArmor starts a guided walkthrough the moment you sign up. It's built around a chapter system where each chapter teaches one capability by having you actually use it.

This is not a product tour. Product tours are overlays that point at every button on the screen and say "this is the sidebar" while you click "Next" fourteen times. Nobody learns anything from those.

GIF: Record the Intro or Connect chapter. Show the spotlight overlay dimming the rest of the screen while highlighting a specific UI element (like the navigation sidebar or the "Add Connection" button). The tooltip popover should be visible with a title, description, and "Next" or action button. Capture 2-3 steps advancing to show the flow of moving through a chapter.

Each chapter uses a spotlight overlay to highlight specific UI elements, explain what they do, and guide you through real actions. Steps don't advance until you've completed the required action, so you're building hands-on familiarity, not just reading tooltips.

A demo database you can explore on day one

The first thing we did was remove the cold start problem entirely.

When you sign up, you get a pre-configured demo database called BalloonBazaar. It has 4 schemas, 24 tables, and 147 columns of realistic e-commerce data. It comes pre-loaded with actual issues: stale tables, schema changes, anomalous patterns, the kinds of problems you'd find in a real data pipeline.

SCREENSHOT: The asset list page with the BalloonBazaar demo database expanded. Should show the schema tree (bronze, silver, gold, raw) with tables nested underneath. Ideally capture a state where at least one table shows a freshness violation badge or a schema change indicator, so the reader can see that the demo data comes with real issues out of the box.
You don't need to connect your own database to start learning. You can explore schema drift on the demo data, set up freshness monitors, configure alerts, and see what AnomalyArmor catches. All without risking your production credentials during a tire-kicking session.

The demo data is flagged internally so it doesn't count against your usage. It's there for learning, not billing.

Want to try it right now? Sign up and the demo database is waiting. No sales call.

The core onboarding path: first monitor in minutes, full coverage when you're ready

The core path has five chapters. The first four get you to a live freshness monitor in under 8 minutes. The fifth adds alerting so issues reach you where you work. Here's the breakdown:

Chapter	What you do	What you'll have when it's done
Intro	Quick orientation: navigation, alerts overview, getting help	Familiarity with the AnomalyArmor interface
Connect	Walk through the database connection form	Understanding of how to add your own databases later
Discover	Run schema discovery, explore tables and columns	Visibility into every table, column, and type in your database
Freshness	Configure a freshness monitor, set intervals and thresholds	Live freshness monitoring that tells you when tables go stale
Alerts	Set up email, Slack, or webhook notifications	Alert delivery so issues reach you where you already work

Once you've got monitoring and alerts running, nine optional chapters let you go deeper: alert routing rules, data quality metrics, correctness checks, lineage tracking, AI-powered intelligence, data tagging, team administration, and CLI/agent workflows. Tackle them at your own pace, in any order.

SCREENSHOT: The chapter selection / learning page showing all 14 chapters. The core path chapters (Intro, Connect, Discover, Freshness, Alerts) should show as completed or in-progress with checkmarks or progress bars. The optional chapters (Alert Rules, Metrics, Correctness, Lineage, Intelligence, Tags, Admin, MCP) should show as available but not started, so the reader can see the breadth of coverage and the progress tracking.

Three step types that teach, not just tour

Each step in a chapter is one of three types, and the distinction matters:

Observation steps highlight something on the screen and explain what it does. You read, you understand, you move on. These are for context, like understanding what the freshness chart axes represent.

Action steps require you to actually do something: click a button, fill in a form, make a selection. The step doesn't advance until you've taken the action. This is where the learning happens, because you're building muscle memory, not just reading instructions.

Wait steps pause while something async completes. When you trigger schema discovery, the step waits for discovery to finish before advancing. No "click here after it's done" guesswork. The system knows when the job is done and moves you forward automatically.

GIF: Record the Freshness chapter. Start from the step where the spotlight highlights the freshness configuration panel on a demo table (e.g. bronze_orders). Show the user setting a check interval, defining a staleness threshold, and clicking save/enable. Then show the freshness check kicking off and the step auto-advancing once the check completes. This is the "aha moment" where the user sees live monitoring working for the first time.

The system tracks your progress per chapter. You can pause mid-chapter, close the browser, come back next week, and pick up where you left off. You can also replay any chapter if you want a refresher.

Why onboarding quality decides which data tool your team adopts

Data observability is not a solo activity. You set it up, your team uses it. If the person who signed up can't get to value quickly, the tool never reaches the rest of the team.

The evaluation pattern is predictable: one engineer evaluates three tools over a week, picks the one they figured out fastest, and rolls it out. The product with the best onboarding wins the evaluation, even if a competitor has more features on paper.

Pendo's 2024 State of Software report found that feature adoption, not feature count, is the strongest predictor of retention. Users who activate three or more features in their first session are 3x more likely to convert. That's exactly what guided onboarding is designed to do: get you to schema discovery, freshness monitoring, and alerting in a single sitting.

Our target: within minutes of signing up, you should have freshness monitoring running on real tables with alerts going to your Slack channel. Everything in the onboarding flow is designed to get you there.

GIF: Record the Alerts chapter. Show the spotlight guiding the user to add a new alert destination (Slack is the most visual). Walk through selecting Slack, connecting the channel, and sending a test alert. End with the test notification appearing in the Slack channel preview or the success confirmation in the UI. This shows the full loop: monitoring detects an issue, alert reaches you where you work.

How we keep improving it

We track onboarding analytics internally: chapter completion rates, drop-off points, time to complete each chapter, and completion trends over time. This isn't vanity metrics. When we see a chapter with a high drop-off rate, we know the steps are confusing and we rewrite them.

Every chapter is scored against a quality rubric with six dimensions: clarity, value demonstration, action quality, pacing, error recovery, and completion momentum. If a chapter scores below our threshold, it gets reworked before it ships.

We treat onboarding like a product feature, not an afterthought. For most users evaluating data quality tools, onboarding IS the product. If they don't get through it, nothing else matters.

Get started with data quality monitoring in minutes

AnomalyArmor's guided onboarding starts automatically when you sign up. The demo database is pre-loaded. You'll have your first live monitor running in under 8 minutes, with alert delivery configured shortly after.

No credit card. No sales call. No staring at an empty dashboard wondering what to click.

Start the guided onboarding now

Key takeaways:

Most data observability tools lose users between "connected" and "configured" because setup is complex and unguided
AnomalyArmor's guided onboarding uses interactive chapters with spotlight overlays, not passive product tours
A pre-loaded demo database (BalloonBazaar) eliminates the cold start problem, so you can learn without connecting production
First live freshness monitor in under 8 minutes (down from 40+ without guidance)
Full core path covers connection, discovery, monitoring, and alerting
Nine optional chapters cover the full product surface: alert rules, metrics, correctness, lineage, AI intelligence, tagging, admin, and CLI workflows

Have questions about setting up data quality monitoring? Email blaine@anomalyarmor.ai. I'll walk you through it.

AI Data Quality Monitoring: Why Most Tools Stop at Tactical AI

Blaine Elliott — Sun, 12 Apr 2026 17:37:53 +0000

Your data observability tool just sent you 47 alerts. Three dashboards are showing anomalies. A stakeholder is asking why the numbers in their report changed. You open your "AI-powered" monitoring tool, and it waits for you to ask the right question.

This is tactical AI. And it's where most data quality tools stop.

The real opportunity is strategic AI: monitoring that thinks proactively about your data problems, surfaces patterns you didn't know to look for, and tells you what to fix before anyone notices something is broken.

Understanding the difference explains why some AI data quality features feel genuinely useful while others feel like marketing checkboxes.

What is Tactical AI in Data Quality Monitoring?

Tactical AI handles reactive observations and analysis. You ask a question, it retrieves information and presents it clearly.

Examples of tactical AI in data observability:

"What columns does the orders table have?"
"When was user_events last updated?"
"What freshness violations do I have right now?"
"What's the blast radius if dim_customers goes down?"

This is AI as an intelligent interface to your data catalog. It saves you from clicking through dashboards, writing queries, or holding complex lineage relationships in your head. Good tactical AI can even correlate information across domains, connecting a schema change to a downstream freshness issue.

But tactical AI is fundamentally reactive. You ask, it answers. You have to know what questions to ask. You have to initiate every interaction. You have to do all the thinking about what might be wrong.

When you have 47 alerts and an angry stakeholder, tactical AI makes you play detective. It hands you a magnifying glass and wishes you luck.

What is Strategic AI in Data Quality Monitoring?

Strategic AI does something fundamentally different. It doesn't wait for questions. It thinks about your data problems autonomously.

Here's a concrete example:

The scenario: Your revenue_daily table failed a freshness check this morning. Three dashboards are showing stale data. The CFO is asking questions.

Tactical AI response: You ask "why is revenue_daily stale?" It tells you the upstream orders table hasn't updated. You ask "why hasn't orders updated?" It tells you there was a schema change yesterday. You ask "what changed?" It shows you a column rename. Fifteen minutes of detective work to find a two-minute fix.

Strategic AI response: You open your monitoring tool and it tells you: "The freshness failure in revenue_daily was caused by yesterday's schema change in orders, when order_status was renamed to status. This broke the ETL job at line 47 of transform_orders.sql. Similar pattern to the incident on January 3rd, which was resolved by updating the column reference. Here's the specific change needed."

Same incident. One approach makes you investigate. The other hands you the answer.

Strategic AI for data observability reasons about:

Root causes, not symptoms. Instead of telling you what's broken, it hypothesizes why things keep breaking. It identifies systemic data quality issues across your entire data estate.

Behavioral patterns over time. Which tables are high-risk based on historical incident rates? Which pipelines are fragile? Which data producers cause the most downstream issues? Strategic AI tracks these patterns and surfaces them unprompted.

Options and tradeoffs. When something needs fixing, strategic AI doesn't just flag the problem. It proposes solutions, explains the tradeoffs, and helps you decide.

Proactive alerts before incidents. Strategic AI notices that a table's null rate is trending upward over three days, or that a schema change is about to break two downstream consumers, and warns you before the incident happens.

Learning from your resolutions. When you fix an alert, strategic AI remembers how. When similar patterns emerge, it suggests the same resolution. When you consistently ignore certain alert types, it asks if those rules should be adjusted.

The difference is autonomy. Tactical AI is a tool you use. Strategic AI is a collaborator that thinks alongside you.

Why Most AI Data Observability Tools Are Stuck on Tactical

Almost every "AI-powered" data quality tool today is purely tactical. They've added chat interfaces to their metadata catalogs. Some can answer sophisticated questions. A few can correlate across domains.

But none of them think proactively:

They don't tell you "here are the three issues you should worry about today, and here's why"
They don't notice that your data quality is degrading in a specific pattern
They don't learn from how you resolve incidents and apply those patterns to new situations
They don't warn you about problems before they become incidents

Tactical AI is useful. It's where everyone has to start. It's where AnomalyArmor is starting. But it's also becoming table stakes. Every tool will have a chat interface within a year. The real differentiation in AI data quality monitoring comes from AI that understands your data deeply enough to be proactive. We're building a path to reach that objective.

The cost of staying tactical: A 2024 study found data teams spend 40% of their time on data quality issues. Most of that time is investigation, not resolution. Strategic AI compresses investigation from hours to seconds.

Building Proactive AI Data Quality Monitoring

You can't skip tactical AI to get to strategic. The foundation matters.

Strategic AI requires rich context: schema metadata, lineage graphs, historical incidents, resolution patterns, freshness trends, validity rules, team ownership. If the tactical layer can't access and correlate this information, the strategic layer has nothing to reason about.

The path to proactive data monitoring:

Phase 1: Comprehensive context. The AI needs access to everything: schema changes, freshness status, alert history, lineage relationships, data quality metrics, user actions. Most tools only expose a fraction of this to their AI layer.

Phase 2: Cross-domain correlation. The AI connects information across domains. A schema change in orders caused a freshness failure in revenue_daily which triggered anomalies in the CFO dashboard. This requires deep understanding, not keyword matching.

Phase 3: Pattern recognition over time. The AI needs memory. What happened last month? What patterns recur? Which resolutions worked? This is where tactical becomes strategic.

Phase 4: Autonomous reasoning. The AI synthesizes patterns into recommendations without being asked. It surfaces what matters before you know to look for it.

What Strategic AI Data Quality Looks Like in Practice

Proactive AI data monitoring looks different from today's chat interfaces.

Morning briefings. You open your data observability tool at 9am and it tells you:

"Three things need attention today:

user_events has had increasing null rates in session_id for 5 days. Downstream tables session_metrics and user_journeys are starting to show anomalies. Likely cause: the mobile app update on Monday.

The ETL job for inventory_snapshot failed twice this week with the same timeout pattern I saw last month. That was resolved by increasing the batch size. Here's the config change.

Team Platform pushed a schema change to api_logs that will break the error_rates dashboard when it propagates tonight. They should coordinate with the analytics team first."

No questions asked. No investigation required. Just: here's what matters, here's why, here's what to do.

Automated incident analysis. When something breaks, the AI doesn't just show you what's broken. It investigates automatically:

"This freshness failure in revenue_daily correlates with yesterday's schema change in orders by user jsmith. The column order_status was renamed to status. This matches the pattern from the January 3rd incident, which was resolved by updating line 47 of transform_orders.sql. Suggested fix: change order_status to status in the SELECT clause."

Proactive risk identification. After observing your data estate for months, the AI notices:

"Your three highest-risk tables are orders, user_events, and payments. Combined, they've caused 73% of downstream incidents this quarter. None have SLAs defined. Adding freshness SLAs would reduce incident impact by an estimated 60%. Here's a suggested configuration."

Resolution learning. The AI tracks how you fix things:

"You've resolved 12 freshness alerts for daily_aggregates in the past month by re-running the Airflow DAG. Should I suggest automatic retry as the first resolution step for this table?"

This is AI as a thinking partner for data engineering teams, not just a query interface.

The Future of AI in Data Observability

Data engineering teams are drowning in signals. Every monitoring tool produces alerts. Every dashboard shows metrics. The job isn't collecting more data quality information. The job is knowing what matters and what to do about it.

Tactical AI helps you find information faster. Strategic AI helps you understand what the information means and what actions to take.

The data observability platforms that win will be the ones that make the leap from reactive to proactive. From answering questions to anticipating them. From flagging problems to solving them.

Where AnomalyArmor Fits

We're building toward strategic AI for data quality monitoring. Today, we have a strong tactical foundation. Tomorrow, we're aiming for something more ambitious.

What's live today:

AI Q&A across your schema, lineage, freshness, and alerts
Cross-domain correlation that connects schema changes to downstream impact
Natural language investigation: "What changed in orders this week?" "Why are there nulls in customer_id?"
Git blast radius that links data issues to the commits and authors responsible

What we're building toward:

Proactive daily briefings that surface issues before you look for them
Pattern recognition across your incident history
Autonomous recommendations based on how you've resolved similar issues
Predictive alerts that warn you before the incident happens

We're not just adding chat to a dashboard. We're building the foundation for AI that thinks about your data quality so you can focus on building.

Try AnomalyArmor and see the difference between AI that waits for questions and AI that has answers ready.

Questions about our AI approach? Email blaine@anomalyarmor.ai. I'll show you exactly where we are on the tactical-to-strategic journey.

Why We Open-Sourced Our Database Query Layer

Blaine Elliott — Sun, 12 Apr 2026 17:32:21 +0000

When you connect a data quality tool to your database, you're trusting that tool with access to your data. Most tools ask you to just trust them. We decided to show our work.

Every query AnomalyArmor runs against your database goes through our Query Security Gateway. The gateway is open source. You can read every line of code. You can verify exactly what we're allowed to do.

GitHub: https://github.com/anomalyarmor/anomalyarmor-query-gateway
PyPI: https://pypi.org/project/anomalyarmor-query-gateway/

The trust problem

Data quality tools need database access to do their job. Schema discovery requires reading metadata. Freshness monitoring requires checking timestamps. Anomaly detection requires looking at distributions.

But customers have legitimate concerns. What queries are you actually running? Could you read our customer data? How do we know you're not doing more than you say?

"Trust us" isn't a good enough answer. Especially when the data is sensitive.

Three access levels

We built the gateway around three access levels. You choose how much access to grant based on your security requirements.

Schema Only: The most restrictive. We can query metadata tables (information_schema, pg_catalog, system tables) but nothing else. You get schema discovery and basic tagging. No access to actual table data.

Aggregates: We can run aggregate functions: COUNT, SUM, AVG, MIN, MAX. No raw values. This enables freshness monitoring (checking MAX(updated_at)), row counts, null rates, and statistical distributions. We never see individual records.

Full: Unrestricted read access. This enables improved tagging and intelligence features that sample values to detect patterns. For example, detecting that a column named "data" actually contains Social Security numbers.

Most customers use Aggregates. You get the monitoring features without exposing raw data.

How it works

The gateway sits between AnomalyArmor and your database. Every query passes through it. The gateway parses the SQL, validates it against your access level, and blocks anything that doesn't comply.

Your Query → Gateway → Parser → Validator → Database
                          ↓
                    Audit Logger

If you've set Aggregates access and something tries to run SELECT email FROM users, the gateway blocks it. Doesn't matter if it's a bug in our code or a misconfigured feature. The query never reaches your database.

Every query attempt is logged. You can audit what we ran and what we tried to run.

Why open source

We published the gateway code for a few reasons.

First, transparency. You shouldn't have to take our word for how the access levels work. Read the code. The validator logic is right there. If we say "aggregates mode only allows aggregate functions," you can verify that claim yourself.

Second, security review. Open source means security researchers can audit it. If there's a bypass or a flaw in our logic, someone can find it and report it. Closed source security is security through obscurity.

Third, trust through verification. When your security team asks "how does this tool handle database access," you can point them to a GitHub repo instead of a marketing page.

Defense in depth

We don't just rely on the gateway. There are two layers of enforcement.

The first layer checks features. Before any SQL is constructed, we check if your access level permits that feature. Trying to run freshness monitoring with Schema Only access? Blocked at the feature layer. You never even see a query.

The second layer is the gateway. It parses and validates the actual SQL. This catches anything that somehow bypasses the feature layer. If a bug in our code constructs a query it shouldn't, the gateway stops it.

Both layers have to allow the operation. If either blocks, nothing runs.

What this means for you

When you connect AnomalyArmor to your database, you choose your access level. The default is Full, for maximum monitoring capability. But you can restrict it at any time.

Some customers use Schema Only on production databases and Full on staging. Some use Aggregates everywhere. You can set a company-wide default and override it per data source.

You can change levels whenever you want. Downgrading disables features that require higher access. Upgrading enables them. No migration, no reconfiguration.

The features at each level

Schema Only gets you:

Schema discovery (tables, columns, types)
Basic tagging (inferred from column names and types)
Basic intelligence (metadata-based insights)

Aggregates adds:

Row counts
Freshness monitoring
Null and completeness checks
Cardinality (distinct counts)
Numeric statistics (min, max, average)

Full adds:

Improved tagging (samples values to detect patterns)
Improved intelligence (value-based insights)

Most data quality monitoring works fine with Aggregates. Full is for when you want the AI to analyze actual values to find things like PII in unexpected columns.

Check it yourself

The gateway code is at https://github.com/anomalyarmor/anomalyarmor-query-gateway. It's Apache 2.0 licensed. Read it, fork it, run the tests.

If you find a security issue, email security@anomalyarmor.ai. We take reports seriously.

This is how we think data tools should work. Not "trust us," but "verify us."

Ready to try data observability with transparent security? Sign up for AnomalyArmor and choose your access level when you connect your database.

Data Quality Tools in 2026: What to Actually Look For

Blaine Elliott — Sun, 12 Apr 2026 17:32:19 +0000

Every data quality vendor has a features page with the same checkboxes. Schema monitoring. Freshness tracking. Anomaly detection. Column profiling. The features are table stakes. What separates the good tools from the mediocre ones is everything else.

Time to value

How long from signup to seeing your first useful alert? This is the single most important question and almost nobody talks about it.

Some tools require a week of configuration before they're useful. You need to define every monitor. Set every threshold. Map every relationship. By the time you're done, you've spent more time setting up the tool than you would have spent just writing SQL checks yourself.

Good tools should give you value in hours, not weeks. Connect your database. Let the tool figure out what normal looks like. Get your first alert when something breaks. You can fine-tune later.

When evaluating, ask: "If I connect my database right now, what will I learn in the next 24 hours?" If the answer is "nothing until you configure monitors," keep looking.

Noise level

A tool that alerts on everything is worse than a tool that alerts on nothing. Alert fatigue is real. If your data quality tool sends fifty alerts a day and forty-eight of them don't matter, you'll start ignoring all of them.

Good tools give you control over what matters. Tags and data classification let you prioritize critical tables and ignore the noise. AI-powered intelligence helps you understand context and triage issues quickly. And integrations with your existing workflow, whether that's Slack, your orchestrator, or AI agents via MCP, mean alerts reach you where you actually work.

Ask vendors: "How do I control which alerts I see and where they go?" If the answer is complicated, expect frustration.

Database coverage

You probably have more than one database. Maybe Postgres for your application, Snowflake for analytics, and some vendor data landing in BigQuery. Your data quality tool needs to work across all of them.

Watch out for tools that technically support your databases but treat some as second-class citizens. "We support MySQL" might mean "we can connect to MySQL but half our features don't work." Ask for specifics. Which features work on which databases?

Pricing model

Most data quality tools price per table. This makes sense: more tables means more monitoring. But the per-table rate varies wildly, from $5 to $20 per table.

Do the math for your actual usage. If you have 200 tables, the difference between $5 and $15 per table is $24,000 a year. That's a real budget item, not a rounding error.

Also watch for hidden costs. Some tools charge extra for features that should be standard. Some charge for users. Some charge for alerts. Get a complete quote, not just the headline price.

Integration with your workflow

Where do your alerts go? If your team lives in Slack, the tool better have good Slack integration. Not just "can send to Slack" but "sends useful, actionable messages that you can respond to."

Same for your orchestration tools. If you're running dbt, can the tool integrate with your dbt tests? Can it trigger alerts based on dbt run failures? Can it show lineage from your dbt models?

The best tool in the world is useless if it doesn't fit into how your team actually works.

AI and agent integration

Data quality tools are starting to add AI features, but most stop at chat interfaces for querying metadata. That's useful, but it's just the beginning.

The real question is whether the tool fits into how AI agents work. Does it expose an MCP server so your AI coding assistant can check data quality before making changes? Can an agent query freshness status or schema changes programmatically? Can it trigger monitors or pull context into your existing AI workflows?

This matters because data engineering workflows are increasingly agent-assisted. If your data quality tool can't participate in those workflows, you're stuck copying and pasting between systems. Look for tools that treat AI integration as a first-class feature, not an afterthought.

What I'd actually evaluate

If I were evaluating data quality tools today, here's my process:

Day 1: Sign up. Connect one database with maybe 50 tables. How long until you have working monitors? If you're still configuring after an hour, that's a red flag. Good tools make setup simple enough that you can be monitoring real tables in minutes, not days.

Day 2-3: Look at the alerts. Are they useful? Are they noise? Intentionally break something in a test environment and see how long it takes to get an alert.

Week 1: Try the integrations you actually need. Set up Slack alerts. Connect to your orchestrator. See if it feels native or bolted-on.

Week 2: Do the pricing math. How much will this cost at your current scale? What about double that scale? Are there features you need that cost extra?

Questions to ask every vendor

Before you buy, get answers to these:

How long does initial setup take for a database with 100 tables?
What's your actual per-table price at my expected scale?
Which features work on which databases?
How does alerting integrate with Slack/Teams/PagerDuty?
Do you support dbt integration? What does it include?
Do you have an MCP server or API for AI agent integration?
What happens if I exceed my plan limits?

The bottom line

Every tool will tell you they have the features you need. What matters is whether those features actually work in practice, whether the tool fits your workflow, and whether the price makes sense for your scale.

Don't buy based on a demo. Run a real trial with real data. See how it performs in your actual environment. That's the only way to know if a tool is good or just good at demos.

AnomalyArmor is built for fast time-to-value. Connect your database and get automated data quality scoring, null rate monitoring, anomaly detection, and schema drift alerts in minutes. Pricing starts at $5/table, roughly half what competitors charge. Sign up.

Schema Drift: The Silent Pipeline Killer

Blaine Elliott — Sun, 12 Apr 2026 17:26:46 +0000

Schema drift is when your database schema changes in ways your downstream systems don't expect. It sounds boring. It will ruin your week.

Unlike a crashed server or a failed deployment, schema drift doesn't announce itself. There's no error page. No alert. Your pipelines keep running. Your dashboards keep updating. The numbers just quietly become wrong.

How it happens

Schema drift happens because databases are shared infrastructure. Your data warehouse isn't just used by your team. Backend engineers add columns. Product teams rename fields. Someone decides user_id should be customer_id for consistency. An intern drops a table they thought was unused.

None of these changes are malicious. Most of them are reasonable in isolation. The problem is that nobody told the data team. And why would they? To the person making the change, it's just a database column. They don't know it feeds into seventeen downstream tables and a board reporting dashboard.

The five types of schema drift

Not all schema changes are equally dangerous. Here's what to watch for:

Column renames are the worst. They look like dropped columns to your queries, but the data is still there under a different name. If you're selecting amount and someone renamed it to total_amount, you get nulls. Not an error. Nulls.

Column drops are at least obvious. Your query fails. You get an error. You can trace the problem immediately.

Type changes are subtle. A varchar becomes a text. An int becomes a bigint. Sometimes it doesn't matter. Sometimes your aggregations start returning slightly different results and nobody notices for weeks.

Column additions are usually safe, but they can break SELECT * queries in unexpected ways. More columns means more memory, slower queries, and occasionally hitting column limits in downstream systems.

Table drops or renames are the nuclear option. Everything downstream breaks loudly. At least you'll notice.

A real example

Last year, a SaaS company I worked with had their entire customer churn model break. The ML team spent three days debugging before they found the issue: a column called last_activity_date had been renamed to last_active_at in the production database.

The rename happened as part of a Rails convention cleanup. Totally reasonable. The backend team did it in a migration with proper deprecation warnings in the API. What they didn't know was that the data warehouse was syncing that table directly, and the churn model was using last_activity_date to calculate days since last login.

When the column disappeared, the pipeline kept running. The null values got coerced to some default date. Suddenly every customer looked like they'd been inactive for decades. The churn model started predicting 100% churn for everyone.

Three days of debugging. One column rename.

Why traditional monitoring misses it

Most monitoring focuses on "is the system up" and "are the jobs running." Those are good things to monitor. They won't catch schema drift.

Your dbt job ran successfully. Great. It just produced wrong data because the source schema changed. Your Airflow DAG is green. Wonderful. It's now loading nulls into a column that shouldn't have nulls.

You need monitoring that understands what the schema looked like yesterday and what it looks like today. You need something that can tell you "column user_status changed from varchar(50) to varchar(20)" before your pipeline truncates half your status values.

Detecting schema drift

The simplest approach is to snapshot your schema periodically and diff it. Every hour, run a query against information_schema, store the results, compare to the previous snapshot. Any differences trigger an alert.

This works. It's also tedious to build and maintain. You need to handle every database type differently. You need to store the snapshots somewhere. You need alerting infrastructure. You need to filter out the noise (not every schema change is a problem).

This is exactly the kind of problem that makes sense to outsource to a dedicated tool. Let someone else deal with the cross-database compatibility. Let someone else figure out which changes are breaking versus benign. You have actual work to do.

What good detection looks like

When a schema change happens, you should know immediately. Not tomorrow. Not when the weekly report looks wrong. Immediately.

The alert should tell you exactly what changed: which table, which column, what the old definition was, what the new definition is. It should tell you when the change happened. And ideally, it should tell you what downstream systems might be affected.

That last part is hard. It requires lineage tracking, knowing which tables feed into which other tables and reports. But even without lineage, just knowing about the change within minutes instead of days is a massive improvement.

Prevention vs detection

In a perfect world, schema changes would go through a review process. Backend teams would notify data teams before making changes. There would be a deprecation period. Downstream systems would be updated first.

In the real world, changes happen fast. Startups move quickly. People forget. Communication breaks down. You can't rely on perfect process to prevent schema drift.

Detection is your safety net. Good process is great. Detection catches everything that process misses.

Key takeaways:

Schema drift happens when database schemas change without downstream systems knowing
Column renames are the most dangerous because they don't cause obvious errors
Traditional job monitoring won't catch schema drift
You need schema-aware monitoring that diffs your database structure over time
Detection is your safety net when process fails

AnomalyArmor detects schema drift automatically, plus monitors data quality metrics like null rates, row counts, and distribution shifts. Connect your database and get alerts within minutes. Sign up.

Why I Built AnomalyArmor

Blaine Elliott — Sun, 12 Apr 2026 17:26:44 +0000

I've done data engineering over the years at CJ, Savings.com, MySpace, Chegg, LinkedIn, Microsoft, One Medical, and AbnormalAI. The thing that's always stuck with me is how the job gets harder in a way that sneaks up on you.

When you build a pipeline, you're not just creating one thing to maintain. You're creating a machine that generates new things to maintain. Every run, every interval, every partition of data that pipeline produces becomes another touch point you're responsible for. One pipeline running hourly for a year is 8,760 data points you now own. Scale that across dozens of pipelines feeding into each other, and you've got an exponential maintenance problem.

This is the part nobody warns you about when you start in data engineering. The pipelines themselves aren't that hard. It's everything they produce that buries you.

The problem without a solution

I spent years looking for elegant tooling to handle this. Something that could watch all those touch points without requiring me to manually define what "good" looks like for each one. The solutions I found were either too simple (just run some SQL tests), too complex (six-week implementations that needed a dedicated admin) or too expensive (out of reach for our budget or company size).

What I wanted was analysis at scale. Limited human interaction to set up, comprehensive coverage across all my data, and smart enough to distill thousands of potential issues into a small set of things I actually needed to look at. Signal, not noise.

The hackathon that started it

A few years back I built a hackathon project around this idea. The core concept was automated statistical profiling: connect to a database, analyze the distributions, detect when something changed meaningfully, and surface only the stuff worth investigating. And do all this at scale with a little I/O as possible to achieve the desired outcome: does my data have any land mines in it?

It worked better than I expected. Not because the statistics were novel, but because it removed the manual effort. I didn't have to write a test for every column. I didn't have to define thresholds for every metric. The system figured out what normal looked like and told me when things deviated.

That project sat in a repo for a while. But the idea kept nagging at me.

Building for myself

AnomalyArmor came from recognizing voids in the industry that nobody was filling. The expensive enterprise tools were overkill for most teams. The lightweight open source options required too much manual configuration. There was a middle ground that didn't exist: something that worked out of the box, scaled with your data, and didn't cost a fortune.

I also just wanted better tooling for myself. Every data engineering job I've had, I've ended up building some version of this internally. Schema change detection scripts. Freshness monitoring cron jobs. Anomaly alerts cobbled together from Airflow sensors. AnomalyArmor is what all of that should have been from the start.

What it does

The pitch is simple: connect your database, get alerts when something's wrong.

Schema drift detection tells you when columns change before your pipelines break. Freshness monitoring tells you when tables stop updating before anyone asks why the dashboard is stale. Data quality metrics catch null spikes, distribution shifts, and anomalies before they corrupt your analytics. Lineage extends these offerings to give you a blast radius of what should be monitored, then does that monitoring for you.

Why $5 per table

I priced it at roughly half what competitors charge because I know what data team budgets look like. At 100 tables, you're paying $475 a month. That's affordable for a real team, not just enterprises with unlimited spend.

If AnomalyArmor saves you one fire drill per month, one late-night debugging session, one embarrassing "why are these numbers wrong" conversation, it's paid for itself.

Try it yourself

If you're tired of the exponential maintenance problem and want tooling that actually helps, sign up and connect your first database in under 5 minutes.

No sales pitch. Just see if it solves a problem you have.

— Blaine

The 6 Dimensions of Data Quality: Definitions, Examples, and How to Monitor Each

Blaine Elliott — Sun, 12 Apr 2026 17:15:39 +0000

The six dimensions of data quality are accuracy, completeness, consistency, timeliness, validity, and uniqueness. Each dimension measures a different aspect of whether data is fit for its intended use. Together they define whether a dataset can be trusted for analytics, machine learning, or customer-facing applications.

This guide defines each dimension with practical examples, SQL detection patterns, and monitoring strategies for production data pipelines.

What are the dimensions of data quality?

Data quality dimensions are measurable attributes that describe different ways data can be wrong. The widely accepted framework includes six core dimensions:

#	Dimension	Question it answers
1	Accuracy	Does the data reflect real-world truth?
2	Completeness	Is any expected data missing?
3	Consistency	Does the same fact match across systems?
4	Timeliness	Is the data current enough to be useful?
5	Validity	Does the data conform to expected formats and rules?
6	Uniqueness	Are there duplicate records where there shouldn't be?

These six dimensions come from the DAMA International Data Management Body of Knowledge (DMBOK) and are used by organizations including the UK Government Data Quality Hub, Monte Carlo, Collibra, and Informatica. Different sources sometimes add dimensions like integrity or conformity, but the core six cover the vast majority of data quality failures.

Why do data quality dimensions matter?

Without a framework, data teams describe quality problems anecdotally: "the data looks off," "something's wrong with customer IDs," "the numbers don't match the dashboard." These complaints are hard to prioritize and harder to fix systematically.

The six dimensions convert vague complaints into measurable categories. A data team that says "we have a completeness problem on 3% of rows and a timeliness problem on 2 tables" can write monitoring rules, assign owners, and track improvement over time. A team that just says "data quality is bad" cannot.

1. Accuracy

Definition: Accuracy measures how closely data reflects the real-world entity or event it describes.

A customer's street address stored as "123 Mai Street" when it should be "123 Main Street" is inaccurate. A transaction recorded as $100 when the actual amount was $1000 is inaccurate. A birth date of 1900-01-01 for a 30-year-old customer is inaccurate.

Accuracy is the hardest dimension to verify automatically because it requires comparing data to an authoritative external truth. Most teams verify accuracy through:

Cross-reference with source systems: Compare warehouse data against the upstream OLTP database
Sampling and manual review: Audit a random subset against original documents
Reference data checks: Compare against a trusted master data source (e.g., a zip code database)
Statistical sanity checks: Flag values that are impossibly high or low

-- Detect impossibly old ages (accuracy check)
SELECT customer_id, birth_date, DATE_DIFF(CURRENT_DATE(), birth_date, YEAR) AS age
FROM customers
WHERE DATE_DIFF(CURRENT_DATE(), birth_date, YEAR) > 120
   OR DATE_DIFF(CURRENT_DATE(), birth_date, YEAR) < 0;

2. Completeness

Definition: Completeness measures whether all expected data is present. It covers both row-level completeness (no missing rows) and column-level completeness (no missing values in required fields).

A daily sales table that should contain one row per store per day but is missing rows for three stores has a row-level completeness problem. A customers table with email IS NULL for 15% of records has a column-level completeness problem.

Completeness checks are straightforward to automate:

-- Column-level completeness: null rate for required fields
SELECT
  COUNT(*) AS total_rows,
  COUNT(email) AS rows_with_email,
  COUNT(*) - COUNT(email) AS null_emails,
  ROUND(100.0 * (COUNT(*) - COUNT(email)) / COUNT(*), 2) AS null_rate_pct
FROM customers;

-- Row-level completeness: missing expected records
SELECT store_id, sale_date
FROM expected_stores_and_dates
LEFT JOIN daily_sales USING (store_id, sale_date)
WHERE daily_sales.store_id IS NULL;

The hard part isn't writing the query. It's deciding what "expected" means. You need a ground truth for what should exist, which usually comes from a reference table, a calendar, or a contract with the upstream source.

3. Consistency

Definition: Consistency measures whether the same fact matches across different systems, tables, or timestamps.

If the customer table shows 10,000 active users and the billing table shows 9,850 active users, there's a consistency problem. If a transaction amount appears as $100 in one system and $100.00 in another, that's usually formatting, not a consistency failure. But if the same transaction appears as $100 in one system and $1000 in another, that's a critical consistency failure.

Consistency checks compare aggregate or row-level values across data sources:

-- Cross-system consistency: customer count reconciliation
WITH crm_count AS (
  SELECT COUNT(*) AS n FROM crm_customers WHERE status = 'active'
),
warehouse_count AS (
  SELECT COUNT(*) AS n FROM dim_customers WHERE is_active = TRUE
)
SELECT
  crm_count.n AS crm_active_customers,
  warehouse_count.n AS warehouse_active_customers,
  ABS(crm_count.n - warehouse_count.n) AS delta
FROM crm_count, warehouse_count;

Consistency problems often stem from timing: one system was updated, the other hasn't synced yet. The monitoring question is whether the gap is within an acceptable SLA or has exceeded it.

4. Timeliness

Definition: Timeliness measures whether data is fresh enough to be useful. A timely dataset is updated on its expected schedule and is current relative to the real-world events it describes.

A dashboard showing "sales last hour" that's actually showing data from 6 hours ago has a timeliness problem. A machine learning model trained on data that's 3 months stale may produce incorrect predictions. A fraud detection system running on yesterday's transactions is useless.

Timeliness is measured in two ways:

Freshness lag: How long since the last update? (CURRENT_TIMESTAMP - MAX(inserted_at))
Schedule adherence: Did the expected update happen on time?

-- Freshness: hours since last row was added
SELECT
  TIMESTAMP_DIFF(CURRENT_TIMESTAMP(), MAX(inserted_at), HOUR) AS hours_since_last_insert,
  MAX(inserted_at) AS most_recent_row
FROM orders
HAVING hours_since_last_insert > 2;  -- alert if stale beyond SLA

Timeliness is the easiest dimension to monitor at scale because it only requires a single max-timestamp query per table. This is why freshness monitoring is typically the first data quality check teams implement.

5. Validity

Definition: Validity measures whether data conforms to defined formats, types, ranges, and business rules.

An email field containing "not-an-email" is invalid. A phone number field with "call my cell" is invalid. A country field with "Martian Empire" is invalid. A percentage field with 150 is invalid. A timestamp in the year 9999 is invalid.

Validity is the most rule-heavy dimension. It requires explicit definitions of what "valid" means for each field:

-- Validity: email format check
SELECT customer_id, email
FROM customers
WHERE email IS NOT NULL
  AND NOT REGEXP_CONTAINS(email, r'^[^@\s]+@[^@\s]+\.[^@\s]+$');

-- Validity: range check
SELECT order_id, discount_pct
FROM orders
WHERE discount_pct < 0 OR discount_pct > 100;

-- Validity: enum check
SELECT order_id, status
FROM orders
WHERE status NOT IN ('pending', 'paid', 'shipped', 'delivered', 'refunded');

Modern data quality tools automate validity checks by profiling historical data to learn expected formats, then flagging new records that deviate.

6. Uniqueness

Definition: Uniqueness measures whether records that should be unique are unique. It covers both primary key uniqueness and business-level deduplication.

A customers table should have exactly one row per customer. A transactions table should have exactly one row per transaction. When the same customer appears twice with slightly different spellings, or the same transaction appears twice because of a retry bug, you have a uniqueness failure.

Uniqueness checks are simple to write:

-- Primary key uniqueness
SELECT customer_id, COUNT(*) AS occurrences
FROM customers
GROUP BY customer_id
HAVING COUNT(*) > 1;

-- Business-level uniqueness (same email, different IDs = probable duplicate)
SELECT LOWER(TRIM(email)) AS normalized_email, COUNT(*) AS dup_count,
       ARRAY_AGG(customer_id) AS customer_ids
FROM customers
WHERE email IS NOT NULL
GROUP BY LOWER(TRIM(email))
HAVING COUNT(*) > 1;

The hard part is defining the business rule for uniqueness. Primary keys are enforced by the database. Business-level deduplication (same person, different spellings) requires fuzzy matching, normalization, or entity resolution algorithms.

How do these dimensions relate to each other?

The six dimensions overlap and interact. A single data quality failure often affects multiple dimensions at once:

Duplicate records violate uniqueness, but also affect accuracy (counts are wrong) and sometimes completeness (aggregates miss data)
Schema drift violates validity (new values don't match expected format), often triggers completeness failures (previously required columns become null), and degrades accuracy (wrong values flow through)
Pipeline delays violate timeliness, but also create consistency problems between source and destination systems

Good monitoring tracks all six dimensions because a problem in one often predicts problems in others. A sudden spike in uniqueness failures for customer IDs is often an upstream completeness problem (nulls being converted to a default value).

How do you measure data quality across all six dimensions?

The standard approach is to calculate a quality score per table per dimension, then aggregate:

Per-dimension score: For each table and each dimension, compute pass/fail against defined rules
Rollup to table score: Average the six dimension scores (or weight by business importance)
Rollup to dataset score: Average across all tables in a dataset
Track over time: Plot the score daily to catch degradation trends

For production data pipelines, modern data observability tools automate this by:

Profiling historical data to learn baselines (typical null rates, value distributions, update frequencies)
Detecting anomalies in new data against those baselines
Tagging each anomaly by the dimension it violates
Rolling up to dashboards that show quality over time per table and per dimension

The key insight is that you cannot manually write rules for every edge case across 500 tables. You need statistical baselines that learn from the data itself, with explicit rules for the invariants that matter most to the business.

Data Quality Dimensions FAQ

What are the 6 dimensions of data quality?

The six dimensions of data quality are accuracy, completeness, consistency, timeliness, validity, and uniqueness. Accuracy measures truth against reality, completeness measures missing data, consistency measures cross-system agreement, timeliness measures freshness, validity measures conformance to rules, and uniqueness measures duplicate records.

Are there more than 6 dimensions of data quality?

Yes. Some frameworks add dimensions like integrity (referential relationships), conformity (adherence to standards), reasonableness (within expected bounds), or auditability (traceable to source). The DAMA DMBOK lists six core dimensions that cover the most common failure modes, which is why the "six dimensions" framework is the most widely cited.

Which data quality dimension is most important?

It depends on the use case. For financial reporting, accuracy and consistency matter most. For real-time dashboards, timeliness is critical. For machine learning features, completeness and validity drive model performance. Most production data teams treat timeliness and completeness as the top two because their failures are easiest to detect and most visible to downstream users.

How do you measure data quality dimensions?

Each dimension is measured by running rule-based or statistical checks and counting pass/fail rates. Accuracy is typically measured by sampling and cross-reference. Completeness is measured as null rate or row-count against expectation. Consistency is measured by reconciling aggregates across systems. Timeliness is measured as lag from expected update. Validity is measured by format and range checks. Uniqueness is measured by primary key and business-level dedup queries.

What is the difference between data quality and data integrity?

Data quality is the broader concept covering accuracy, completeness, consistency, timeliness, validity, and uniqueness. Data integrity is a narrower concept focused on referential relationships and constraint enforcement (foreign keys resolve, required fields aren't null, allowed values are enforced). Integrity is sometimes listed as a seventh dimension of quality, but most frameworks treat it as a subset of validity and completeness.

Can you have high data quality in one dimension and low in another?

Yes, and this is common. A table can have perfect uniqueness (no duplicates) but terrible timeliness (updated weekly when it should be hourly). A dataset can be perfectly complete (no missing rows) but inaccurate (values are wrong). Monitoring each dimension separately reveals these patterns. A single "data quality score" that averages all six hides the specific failure modes you need to fix.

How is data quality different from data observability?

Data quality is the outcome: whether data is fit for use. Data observability is the practice: continuously monitoring data pipelines to detect quality issues in production. You can have high data quality without observability (if nothing ever breaks), but in practice you need observability to maintain quality over time as systems evolve and upstream sources change.

What tools automate data quality dimension monitoring?

Modern data observability platforms including AnomalyArmor, Monte Carlo, Metaplane, Bigeye, and Datafold automate monitoring across all six dimensions by profiling historical baselines and flagging anomalies. Open-source tools like Great Expectations, Soda Core, and dbt tests cover rule-based validity and completeness checks but require manual rule writing. Most production teams combine both: automated baseline monitoring for the long tail plus explicit rules for business-critical invariants.

How much historical data do you need to monitor data quality dimensions?

Statistical baselines typically require 7-14 days of historical data for basic anomaly detection. Weekly seasonality needs at least 4 weeks. Yearly seasonality requires 12-18 months. For rule-based checks (validity, uniqueness, primary key enforcement), no history is needed, you can run them on any new data as it arrives.

Can you fix low data quality after the fact?

Sometimes yes, often no. Validity and uniqueness problems can often be fixed retroactively by cleaning and deduplication. Completeness problems can sometimes be fixed by re-running upstream loads. Accuracy problems usually can't be fixed without access to the original source, which may have been lost. Timeliness problems can't be fixed at all: once data is late, it's late. Prevention through monitoring is always cheaper than retroactive cleanup.

Data quality dimensions are only useful if you can measure them in production. See how AnomalyArmor automatically monitors accuracy, completeness, consistency, timeliness, validity, and uniqueness across your data pipelines.

Data Anomaly Detection: The Complete Guide for Data Engineers

Blaine Elliott — Sat, 11 Apr 2026 22:48:30 +0000

Data anomaly detection is the process of identifying data points, patterns, or values that deviate from expected behavior. It catches schema changes, stale tables, row count spikes, and statistical outliers before they break dashboards or corrupt downstream analytics. Modern data anomaly detection combines statistical methods like z-scores and Welford's algorithm with machine learning models that learn seasonal patterns from historical data.

This guide explains the four types of data anomalies, the algorithms used to detect each one, and how to implement detection in Snowflake, Databricks, and PostgreSQL.

What is data anomaly detection?

Data anomaly detection is the automated identification of unexpected values, patterns, or changes in a dataset. In data engineering, it monitors production tables for problems like:

A column gets renamed, dropped, or changes type (schema drift)
A daily-updated table hasn't received new rows in 36 hours (freshness failure)
Row counts drop by 80% overnight (volume anomaly)
Null rate in a critical column spikes from 2% to 40% (quality anomaly)
A customer ID in a fact table references a non-existent record (referential anomaly)

The goal is to catch these problems before they reach dashboards, ML models, or customer-facing applications.

The four types of data anomalies

1. Schema anomalies

Schema anomalies occur when the structure of a table changes unexpectedly. Common examples:

Column added: A new column appears upstream, which can break SELECT * queries
Column dropped: A column disappears, breaking any query that references it
Column renamed: The column exists under a different name, causing silent NULL returns
Type changed: A VARCHAR becomes an INTEGER, causing cast failures

Schema anomalies are the most common cause of silent data failures because queries often continue to run without error, returning wrong results.

2. Freshness anomalies

Freshness anomalies happen when a table stops updating on its expected schedule. A table that normally updates every hour but hasn't received new rows in 6 hours has a freshness anomaly. These are caused by:

Upstream pipeline failures
Source system outages
Broken scheduled jobs
Permission changes

Freshness is typically measured as "time since last insert" or "max(timestamp_column)".

3. Volume anomalies

Volume anomalies are unexpected changes in row counts. A daily sales table that normally receives 10,000-12,000 rows suddenly receiving 500 rows (or 100,000) is a volume anomaly. Causes include:

Upstream filter changes
Duplicate data ingestion
Failed partial loads
Fraud or bot activity

4. Value anomalies

Value anomalies are statistical outliers in column values. Examples:

A revenue column where 5% of rows are negative when they should always be positive
A foreign key column where null rates spike from 2% to 40%
A timestamp column with future dates

Value anomalies are detected using statistical methods applied to specific columns.

How data anomaly detection works

Anomaly detection uses three main approaches: static thresholds, statistical methods, and machine learning.

Static thresholds

The simplest approach. You define the expected range manually:

SELECT 'anomaly' AS status
FROM orders
WHERE COUNT(*) < 1000 OR COUNT(*) > 50000;

Static thresholds work for stable metrics but fail for anything with seasonality (weekend traffic drops, end-of-month spikes).

Statistical methods

Statistical anomaly detection uses historical data to compute expected ranges automatically. The most common approach is the z-score:

z = (current_value - historical_mean) / historical_stddev

If the absolute z-score exceeds a threshold (typically 2 or 3), the value is flagged as anomalous. A z-score of 2 catches values more than 2 standard deviations from the mean, which is roughly the top or bottom 2.5% of a normal distribution.

Welford's algorithm is the most efficient way to compute running mean and standard deviation for anomaly detection. It maintains three numbers (count, mean, and sum of squared deviations) and updates them incrementally with each new data point, requiring constant memory:

def update_stats(count, mean, m2, value):
    count += 1
    delta = value - mean
    mean += delta / count
    delta2 = value - mean
    m2 += delta * delta2
    return count, mean, m2

def get_variance(count, m2):
    return m2 / (count - 1) if count > 1 else 0

This is the foundation of most production anomaly detection systems because it scales to high-volume event streams without storing historical data.

Machine learning methods

For data with complex seasonality (weekly patterns, business hours, holiday effects), machine learning models outperform simple statistics. The most common approach is Prophet (Facebook's time-series forecasting library), which decomposes a series into trend, weekly seasonality, and yearly seasonality, then flags values outside the prediction interval.

Prophet requires at least 14 data points to detect weekly patterns and 365 points to detect yearly patterns. For tables with less history, fall back to z-scores.

How to detect data anomalies in Snowflake

Snowflake provides metadata views that make anomaly detection straightforward.

Schema anomalies: Track column changes via INFORMATION_SCHEMA.COLUMNS:

SELECT table_name, column_name, data_type, last_altered
FROM information_schema.columns
WHERE table_schema = 'PRODUCTION'
  AND last_altered > DATEADD(hour, -24, CURRENT_TIMESTAMP());

Freshness anomalies: Check ACCOUNT_USAGE.TABLES for last DML operation:

SELECT table_name,
       last_altered,
       DATEDIFF(hour, last_altered, CURRENT_TIMESTAMP()) AS hours_stale
FROM snowflake.account_usage.tables
WHERE table_schema = 'PRODUCTION'
  AND DATEDIFF(hour, last_altered, CURRENT_TIMESTAMP()) > 24;

Volume anomalies: Compare today's row count against a rolling 30-day average:

WITH daily_counts AS (
  SELECT DATE(created_at) AS day, COUNT(*) AS row_count
  FROM orders
  WHERE created_at >= DATEADD(day, -30, CURRENT_DATE())
  GROUP BY DATE(created_at)
),
stats AS (
  SELECT AVG(row_count) AS mean, STDDEV(row_count) AS stddev
  FROM daily_counts
  WHERE day < CURRENT_DATE()
)
SELECT row_count,
       (row_count - mean) / stddev AS z_score
FROM daily_counts, stats
WHERE day = CURRENT_DATE()
  AND ABS((row_count - mean) / stddev) > 2;

How to detect data anomalies in Databricks

Databricks offers Delta Live Tables expectations for inline anomaly detection:

import dlt
from pyspark.sql.functions import col

@dlt.table
@dlt.expect_or_drop("valid_order_total", "order_total > 0")
@dlt.expect_or_fail("recent_data", "created_at > current_date() - interval 2 days")
def clean_orders():
    return spark.read.table("raw_orders")

For volume and statistical anomalies, use Unity Catalog's lineage tracking combined with scheduled queries:

SELECT table_name,
       COUNT(*) AS row_count,
       MAX(ingestion_time) AS last_update
FROM production.orders
GROUP BY table_name;

How to detect data anomalies in PostgreSQL

PostgreSQL doesn't have built-in anomaly detection, but you can implement it with pg_stat_user_tables and custom queries:

SELECT relname AS table_name,
       n_live_tup AS row_count,
       last_autoanalyze
FROM pg_stat_user_tables
WHERE schemaname = 'public'
  AND last_autoanalyze < NOW() - INTERVAL '24 hours';

For value anomalies, use window functions to compute rolling statistics:

WITH rolling_stats AS (
  SELECT order_id,
         amount,
         AVG(amount) OVER (ORDER BY created_at ROWS BETWEEN 30 PRECEDING AND 1 PRECEDING) AS rolling_mean,
         STDDEV(amount) OVER (ORDER BY created_at ROWS BETWEEN 30 PRECEDING AND 1 PRECEDING) AS rolling_stddev
  FROM orders
)
SELECT order_id, amount, rolling_mean, rolling_stddev,
       (amount - rolling_mean) / NULLIF(rolling_stddev, 0) AS z_score
FROM rolling_stats
WHERE ABS((amount - rolling_mean) / NULLIF(rolling_stddev, 0)) > 3;

Build vs buy: data anomaly detection tools

Building anomaly detection in-house gives you control but requires engineering time to maintain. Most data teams outgrow custom solutions because:

Alert fatigue: Static thresholds fire too often and get ignored
Seasonality blindness: Simple statistics miss weekly and yearly patterns
Cross-platform monitoring: Different code for Snowflake, Databricks, and Postgres
Incident triage: No unified view of which alerts matter most

AnomalyArmor is a data observability platform that uses AI to configure anomaly detection automatically. You connect your data warehouse, describe what you want to monitor in plain English, and the AI agent sets up schema drift alerts, freshness schedules, and statistical anomaly detection across all your tables. It works on Snowflake, Databricks, PostgreSQL, and BigQuery.

Data anomaly detection FAQ

What is the difference between anomaly detection and data validation?

Data validation checks if data matches explicit rules (e.g., "order_id is not null"). Anomaly detection uses statistical methods to identify values that deviate from historical patterns. Validation catches known problems. Anomaly detection catches unknown ones.

What is the best algorithm for data anomaly detection?

For most production use cases, z-scores computed with Welford's algorithm work well. For data with strong weekly or yearly seasonality, Prophet or similar time-series models are better. For high-dimensional data, isolation forests outperform statistical methods.

How do I detect schema drift automatically?

Query your database's INFORMATION_SCHEMA or metadata views on a schedule, store the previous state, and diff the current state against the stored version. When columns change, type definitions change, or tables are added or removed, fire an alert. AnomalyArmor does this automatically for Snowflake, Databricks, and PostgreSQL.

What is a z-score and how is it used in anomaly detection?

A z-score measures how many standard deviations a value is from the historical mean. A z-score of 2 means the value is 2 standard deviations above the mean, which occurs in roughly 2.5% of a normal distribution. Most anomaly detection systems use z-scores between 2 and 3 as thresholds.

How much historical data do I need for anomaly detection?

Statistical methods like z-scores need at least 7-10 data points to produce meaningful baselines. Machine learning methods like Prophet need at least 14 points for weekly seasonality and 365 points for yearly seasonality. During the learning phase, most systems don't fire alerts.

What is the difference between data observability and anomaly detection?

Anomaly detection is one component of data observability. Data observability also includes lineage tracking, impact analysis, schema change detection, and root cause analysis. Anomaly detection tells you something is wrong. Observability tells you what, where, and why.

Can AI improve data anomaly detection?

Yes. AI improves anomaly detection in three ways. First, AI agents can configure monitoring rules from natural language instead of YAML or GUI forms. Second, LLMs can analyze alert patterns to reduce false positives. Third, AI can correlate anomalies across tables to identify root causes faster than manual investigation.

How do I avoid alert fatigue in anomaly detection?

Use adaptive thresholds that learn from historical patterns instead of static rules. Set sensitivity per table based on how critical it is. Group related alerts so a single upstream failure generates one notification instead of ten. Suppress alerts during known maintenance windows.

What data platforms support anomaly detection natively?

Snowflake has data metric functions and ACCOUNT_USAGE views. Databricks has Delta Live Tables expectations and Unity Catalog lineage. BigQuery has table metadata and scheduled queries. PostgreSQL has pg_stat_user_tables. None of these are full anomaly detection systems, but they provide the raw metrics needed to build one.

How real-time should anomaly detection be?

It depends on the use case. Schema drift and freshness checks should run every 5-15 minutes. Row count and statistical anomalies should run hourly for most tables and daily for slower-changing ones. Real-time streaming anomaly detection (sub-second) is rarely needed for data warehouses but is critical for fraud detection and security monitoring.

Summary

Data anomaly detection catches schema changes, freshness failures, volume spikes, and statistical outliers before they break downstream analytics. The four main types of anomalies require different detection approaches: schema changes need metadata diffs, freshness needs time-since-update checks, volume needs historical baselines, and value anomalies need statistical methods like z-scores or machine learning models like Prophet.

Modern data observability platforms combine all four detection methods with AI-powered configuration to make anomaly detection practical at scale. Whether you build in-house or buy a tool, the fundamental algorithms are the same: maintain historical baselines, compute expected ranges, and flag deviations beyond your sensitivity threshold.

Want to see data anomaly detection in action? Watch a 30-second demo of AI configuring schema drift monitoring in real time.

You Don't Need to Write Data Tests

Blaine Elliott — Sat, 11 Apr 2026 22:47:37 +0000

Spend five minutes in any data engineering forum and you'll find the same confession repeated in different words: "We just eyeball row counts and pray." It shows up on Reddit, Hacker News, the dbt Community Forum, Stack Overflow. The phrasing changes but the story doesn't.

Data engineers know they should be testing. They're not skipping tests because they're lazy or because they don't understand the value. They're skipping tests because everything else in their environment conspires against it.

Why data engineers don't test

If you talk to enough practitioners (or read enough forum threads), the same reasons surface over and over:

Nobody gives them time. Organizations reward fast delivery, not reliable delivery. If decision makers don't prioritize testing, it never becomes a standard. The incentive structure actively punishes thoroughness. You get more credit for shipping a pipeline in two days than for spending a week making it bulletproof.

Data changes faster than tests can keep up. This is what separates data testing from software testing. Your code doesn't change overnight. Your data does. A source team renames a column. A third-party API changes its response format. A bulk operation shifts row counts by 40%. Tests written last month don't account for changes that happened last night.

Data quality is invisible until it breaks. The fundamental problem in data engineering is that a bad query still returns results. Results, but not necessarily correct ones. If nobody can see when things are broken, nobody builds the political will to prevent breakage.

Data is inherently hard to test. You can test code. Data is another story. Unit tests verify that your transformation logic works. They don't verify that the data you received is what you expected. These are fundamentally different problems, and the second one causes far more real-world failures.

Code testing vs data testing

This is the distinction the industry has been dancing around for years. Unit tests and data quality checks are different things, and conflating them is why most testing advice falls flat for data teams.

Unit tests verify your code does what you intended. They answer: "Does my transformation produce the right output given known input?"

Data quality checks verify the data you received is what you expected. They answer: "Did 50,000 rows actually arrive? Is the schema the same as yesterday? Are null rates within normal bounds? Did the distribution shift?"

In data engineering, the second category catches far more production failures than the first. Your dbt model can be perfectly correct and still produce garbage if the source data changed underneath it.

Most testing advice aimed at data engineers focuses on the first category. Write unit tests for your transformations. Test your SQL with fixtures. Use dbt tests. This is useful, but it misses the failures that actually page people at 3am.

"Make testing easier" is the wrong frame

The conventional wisdom is: testing is too hard, so let's make it easier. Better frameworks. Better test runners. Better dbt test macros. AI-assisted test generation.

That's genuinely helpful for teams that have the bandwidth to maintain a test suite. But it doesn't address the actual constraint. The problem isn't that testing is too hard. The problem is that testing is another thing to maintain in an environment where there's already not enough time.

Making tests 50% easier to write doesn't help when nobody has time to write them at all. And even if you find time to write them, data changes faster than tests can keep up.

The better frame: don't make testing easier. Make it unnecessary.

Automated data testing: tests you never write

Automated data testing flips the model. Instead of engineers defining what "correct" looks like for every table, the system learns what normal looks like and alerts when something deviates.

This covers the checks that catch the majority of real incidents:

Schema change detection. A column gets renamed, removed, or changes type. This breaks downstream models, joins, and dashboards. You don't need a handwritten test for this. You need a system that tracks schema state and alerts on any change.

Freshness monitoring. A table that updates every hour hasn't been touched in six hours. The pipeline didn't error. It just silently stopped. A system that learns update patterns and flags deviations catches this without any configuration.

Volume anomalies. A table that normally loads 100,000 rows per day suddenly loads 1,000. Or zero. Or 500,000. Anomaly detection against historical baselines catches this without anyone defining thresholds.

Distribution shifts. A column's null rate jumps from 2% to 35%. A numeric field's average drops by half. These are the subtle failures that pass a "did it run?" check but corrupt downstream analytics.

None of these require writing tests. They require connecting to your data warehouse and letting the system build baselines.

What this looks like in practice

You connect your Snowflake, Databricks, BigQuery, PostgreSQL, or Redshift warehouse. The system runs discovery: what tables exist, what schemas they have, when they typically update, what their normal row counts and distributions look like.

From that point, monitoring is automatic. Schema changes trigger alerts. Stale tables trigger alerts. Volume and distribution anomalies trigger alerts. All of this happens without writing a single line of test code.

When something fires, you get context: which table, what changed, when it changed, and which downstream assets are affected. The alert isn't "test failed." The alert is "the orders_fact table hasn't updated in 4 hours, and 12 downstream models depend on it."

This is what AnomalyArmor does. Five-minute setup, no test authoring, no test maintenance. It watches your warehouse and tells you when something looks wrong. The coverage scales with your warehouse, not with your team's bandwidth to write tests. See the quickstart guide to connect your first data source.

This doesn't replace all testing

To be clear: automated data testing doesn't eliminate the need for all handwritten tests. If you have specific business rules (revenue must be positive, email must contain @, every order must have a customer), those still need explicit validation.

But most data teams don't have any testing at all. They're eyeballing row counts and praying. For those teams, automated data testing provides 80% of the coverage with 0% of the authoring effort.

Start with automated monitoring. Add handwritten tests for your most critical business rules. That's the order that matches reality for time-constrained data teams.

The real question

The real question isn't whether every possible scenario has been tested. It's how much uncertainty your organization is willing to tolerate before it starts verifying the numbers it depends on.

For most data teams, the answer has been: a lot of uncertainty. Because the alternative was writing and maintaining tests they didn't have time for.

Automated data testing changes that tradeoff. The cost of coverage drops to near zero. The question stops being "can we afford to test?" and becomes "why aren't we?"

Sources

Joe Reis, 2026 State of Data Engineering Survey (2026). 1,101 respondents. Found data teams spend 34% of time on data quality, 26% on firefighting.
AnomalyArmor, Quickstart Guide. Connect your first data source and set up automated monitoring.
AnomalyArmor, Schema Monitoring Docs. How automated schema change detection works.
AnomalyArmor, Data Quality Monitoring Docs. Volume, distribution, and anomaly monitoring reference.

Automated Data Testing FAQ

What is automated data testing?

Automated data testing is software that continuously validates data without requiring engineers to write explicit test cases. It learns patterns from historical data (volume, schema, distributions, freshness) and alerts when new data deviates from those patterns. It's the opposite of manual test writing like dbt tests or custom SQL assertions.

How is automated data testing different from dbt tests?

dbt tests are deterministic rules you write manually: "this column is unique", "this foreign key exists". Automated data testing learns baselines from historical data and flags statistical deviations. dbt tests catch known problems. Automated testing catches unknown problems. Most production teams use both.

Do I still need to write data tests if I use automated testing?

Yes, for business-critical invariants. Some rules must be enforced explicitly: "revenue must never be negative", "user_id in orders must exist in users". Write these as dbt tests or validation rules. Use automated testing for everything else (statistical anomalies, freshness, schema changes, volume drops).

What can automated data testing detect that manual tests can't?

Automated testing catches things you didn't know to look for: a column's null rate drifting from 2% to 15% over two weeks, row count dropping by 30% on Tuesdays only, a new category appearing in an enum column, a schema change that silently returns NULL for one in a million rows. These are invisible to explicit rules unless you already anticipated them.

Why don't data engineers write more tests?

Three reasons. First, writing tests requires knowing what to test, and data changes faster than test coverage. Second, test maintenance scales linearly with the number of tables, so a team with 500 tables drowns in test code. Third, the ROI of manual tests is unclear until something breaks, so writing them feels like prevention against unknown risks.

How do automated data tests learn what's normal?

They compute baselines from historical data using statistical methods: running mean and standard deviation (often via Welford's algorithm), distribution fingerprints, seasonality models like Prophet, and moving averages. The baselines update incrementally as new data arrives. Most systems require 7-14 days of history before alerts start firing.

What's the false positive rate of automated data testing?

Well-tuned systems run at 5-15% false positive rates using z-scores with sensitivity thresholds of 2-3 standard deviations. Poorly tuned systems can exceed 50%. The key factors are: enough historical data to establish stable baselines, seasonality-aware models for data with weekly or daily patterns, and sensitivity tuning per table based on business criticality.

Can AI replace data engineers writing tests?

AI can configure and maintain monitoring based on patterns it learns from your data. It can't replace business logic validation. A data engineer still needs to specify what matters to the business. But AI removes the grunt work of writing 500 tests for 500 tables, which is where most test-writing effort is wasted.

What tools provide automated data testing?

Leaders in this space include AnomalyArmor, Monte Carlo, Metaplane, Bigeye, and Datafold. Each uses statistical methods to learn baselines and detect anomalies. Open-source options include re_data and Elementary. Traditional tools like Great Expectations require manual test writing but can be combined with profiling to semi-automate.

How much historical data do I need before automated testing works?

Minimum 7 days for basic z-score detection on daily data, 14 days for weekly seasonality detection, and 365 days for yearly seasonality. During the initial learning period, alerts should be suppressed or warnings only. Most tools have a "learning phase" flag that prevents false alerts until the baseline is stable.

Stop writing and maintaining data tests. See how AnomalyArmor's AI agent configures monitoring from a single sentence.

Data Pipeline Monitoring: How to Stop Silent Failures Before They Hit Production

Blaine Elliott — Sat, 11 Apr 2026 22:32:03 +0000

Your Airflow DAG shows all green. Every task completed. No errors in the logs.

But the revenue dashboard is showing yesterday's numbers. A downstream ML model is training on stale features. The finance team is about to close the quarter using incomplete data.

This is the most dangerous type of pipeline failure: the one that doesn't look like a failure at all. And it's far more common than the kind that throws an error.

Data pipeline monitoring exists to catch exactly this. Not job-level "did it run?" checks. Outcome-level "did the data actually arrive, and does it look right?" checks. The difference between those two questions is where most data incidents live.

What is data pipeline monitoring?

Data pipeline monitoring is continuous validation that data is flowing correctly through every stage of your pipeline, from ingestion to transformation to the tables your stakeholders query.

It covers five dimensions:

Freshness: Is data arriving on schedule?
Volume: Are the expected number of rows landing?
Schema: Have columns been added, removed, or changed type?
Distribution: Do the values look normal, or has something shifted?
Lineage: When something breaks, which downstream tables and dashboards are affected?

Most teams start with the first two and add the rest as they scale. But even basic freshness and volume checks catch the majority of incidents that slip past orchestration tools.

The 5 types of pipeline failures (and which ones your tools miss)

1. The successful failure

A DAG runs to completion. Zero errors. But the source API returned an empty response, so the pipeline wrote zero rows. The orchestrator sees a successful run. The table is now empty or stale.

What catches it: Volume monitoring. If a table that normally receives 50,000 rows per load suddenly gets zero, that's an alert.

2. The schema surprise

Someone on the source team renames a column from user_id to userId. Your pipeline doesn't error, it just silently drops the column or fills it with nulls. Downstream joins break. Metrics go wrong. Nobody notices for three days.

What catches it: Schema change detection. Any added, removed, or type-changed column triggers an alert before downstream transformations run.

3. The slow drift

Data volumes gradually decrease by 5% per week. No single day looks alarming. But after a month, you're missing 20% of your records. A filter change upstream, a timezone bug, a partition misconfiguration.

What catches it: Distribution and volume trend monitoring. Anomaly detection that compares today's load against historical patterns, not just a static threshold.

4. The partial load

The pipeline runs, but only processes data from 3 of 5 source partitions. Row counts look lower than normal, but not dramatically. The missing data is from one region, so the aggregate metrics look "close enough" to pass a quick glance.

What catches it: Volume monitoring with granular baselines, comparing expected vs actual row counts at the partition or segment level.

5. The delayed cascade

A source table updates 4 hours late. Downstream transformations ran on schedule and processed stale input. The numbers are technically "fresh" (the downstream table updated on time) but wrong (it used yesterday's source data).

What catches it: Freshness monitoring on source tables, combined with lineage awareness that understands the dependency chain. The downstream table looks fresh, but tracing upstream reveals the root cause.

Why orchestration alerts aren't enough

Airflow, Dagster, Prefect, and similar tools monitor the process: did the job start, run, and finish? They answer "did my code execute?" not "did my data arrive correctly?"

Three specific gaps:

1. Successful jobs that produce wrong output. A job can complete with exit code 0 and write garbage. The orchestrator has no opinion about data content. It ran your code. That's its job.

2. No cross-system visibility. Your pipeline pulls from a Postgres source, transforms in dbt, and lands in Snowflake. The orchestrator sees the dbt run. It doesn't know the Postgres source stopped updating two hours before the dbt run kicked off.

3. No historical baselines. Orchestration tools tell you about this run. They don't tell you whether this run's output looks normal compared to the last 30 runs. A table loading 1,000 rows isn't alarming, unless it normally loads 100,000.

Data pipeline monitoring sits on top of orchestration. It checks what the orchestrator can't: the actual data that landed.

What good data pipeline monitoring looks like

Effective monitoring has four properties:

1. It monitors outcomes, not processes

Check the table, not the job. Did rows arrive? Are the columns intact? Do the values fall within expected ranges? This is the fundamental shift from orchestration monitoring.

2. It adapts to patterns

A static threshold of "alert if fewer than 10,000 rows" breaks when your table legitimately receives 2,000 rows on weekends. Good monitoring learns the pattern and alerts on deviations from it, not from a fixed number.

3. It maps dependencies

When a source table is late, you need to know which downstream tables, dashboards, and reports are affected. Without lineage, you're manually tracing dependencies across systems during an incident, which is the worst time to be doing it.

4. It routes alerts to the right people

A freshness alert on the marketing analytics table should go to the data engineering team that owns that pipeline, not to a shared #data-alerts channel that everyone has muted. Alert routing by ownership turns monitoring from noise into action.

How to set up data pipeline monitoring

Step 1: Identify your critical tables

You don't need to monitor everything on day one. Start with the 10-20 tables that power:

Executive dashboards
Customer-facing data products
Financial reporting
ML model features

These are the tables where a silent failure causes the most damage.

Step 2: Set freshness and volume baselines

For each critical table:

Freshness: How often should this table update? Set the SLA slightly longer than the expected interval. A table that updates hourly gets a 2-hour SLA.
Volume: How many rows does a typical load produce? Set a range based on the last 30 days, accounting for weekday/weekend variation.

Step 3: Enable schema change detection

Schema changes are the most common cause of silent pipeline failures. Any column added, removed, renamed, or type-changed should generate an alert. This catches problems at the source before they propagate downstream.

Step 4: Connect your alert channels

Route alerts to Slack, PagerDuty, or email based on table ownership. The person who gets the alert should be the person who can fix it.

Step 5: Expand gradually

Once your critical tables are monitored, expand to the next tier. Most teams reach full coverage within a few weeks, not months.

The build vs buy decision

You can build basic monitoring with SQL queries and a scheduler. Check INFORMATION_SCHEMA for freshness, run COUNT(*) for volume, compare schemas against a stored baseline.

This works for 5-10 tables. At 50+ tables across multiple databases, you're maintaining:

A custom scheduler running checks every 15-60 minutes
Per-table configurations for thresholds and SLAs
Historical storage for baselines and trend comparison
Alert routing logic by table ownership
A UI for your team to see monitoring status

At that point, the monitoring system is its own engineering project. The question is whether your team's time is better spent maintaining monitoring infrastructure or building data products.

Purpose-built tools like AnomalyArmor handle this out of the box. Connect your warehouse, and freshness, volume, and schema monitoring start automatically. AI-powered analysis explains what changed and why, so you spend less time investigating and more time fixing. Setup takes minutes, not weeks.

Common mistakes to avoid

Setting thresholds too tight. A freshness SLA of 61 minutes on a table that updates hourly will fire every time there's a minor delay. Start generous and tighten over time.

Monitoring everything equally. Not every table is critical. A staging table that only you use doesn't need PagerDuty integration. Prioritize by downstream impact.

Ignoring weekends and holidays. Many pipelines have legitimately different patterns on weekends. Your monitoring needs to account for this or you'll get false alerts every Saturday.

Alert channel sprawl. Sending every alert to a shared Slack channel guarantees they'll be ignored. Route alerts to the specific team that owns the pipeline.

Treating monitoring as a one-time setup. Your pipelines change. New tables get added, old ones get deprecated, schedules shift. Monitoring configuration needs to evolve with your data stack.

FAQ

What's the difference between data pipeline monitoring and data observability?

Data pipeline monitoring focuses on whether data is flowing correctly through your pipelines: freshness, volume, schema. Data observability is the broader discipline that includes monitoring plus lineage, root cause analysis, and historical context. Monitoring is the foundation. Observability is the full picture.

Do I need monitoring if I already use dbt tests?

Yes. dbt tests validate data at transformation time. They check "is this data correct right now?" Monitoring checks "is this data arriving on schedule, in the expected volume, with the expected schema?" They answer different questions. dbt tests catch logic bugs. Monitoring catches infrastructure and upstream failures.

How many tables should I monitor?

Start with your 10-20 most critical tables. Expand from there. Most teams reach full coverage (all production tables) within a few weeks. The goal is 100% coverage of anything that powers a decision, dashboard, or downstream system.

What's the right alert threshold for freshness?

Set it at 1.5-2x your expected update interval. A table that updates every hour should alert at 2 hours. A daily table should alert at 25-26 hours. This avoids false alarms from minor delays while catching real failures.

Can I build my own pipeline monitoring?

You can, and many teams start there. SQL queries checking freshness and row counts are straightforward for a handful of tables. The maintenance burden grows quickly at scale. Most teams that start DIY either invest significant engineering time maintaining it or switch to a purpose-built tool within 6-12 months.

Data Observability vs Data Quality: What's the Difference and Do You Need Both?

Blaine Elliott — Sat, 11 Apr 2026 22:31:15 +0000

Data observability and data quality get used interchangeably, but they solve different problems. Confusing them leads to buying the wrong tool, building the wrong monitors, and missing the issues that actually break things.

Here's the short version: data observability tells you whether your pipelines are working. Data quality tells you whether the data itself is correct. One watches the plumbing. The other checks the water.

Data observability: watching the pipes

Data observability monitors the infrastructure that moves data. It answers questions like:

Did this table update on schedule? (freshness)
Did the number of rows change unexpectedly? (volume)
Did someone add, remove, or rename columns? (schema changes)
Where did this data come from, and what depends on it? (lineage)

These are all things you can measure without knowing anything about what the data means. You don't need business logic. You don't need to know that revenue should always be positive or that email should contain an @ sign. You're just watching patterns and alerting when they break.

Data observability catches problems like:

An Airflow DAG failed silently at 3am and your morning dashboards show stale data
A backend engineer renamed user_id to account_id and broke 12 downstream models
A bulk delete wiped 40% of your rows and nobody noticed for two days
A table that normally updates every hour hasn't been touched in six hours

These are infrastructure failures. The data pipeline broke, and observability tells you where and when.

Data quality: checking the water

Data quality validates the actual content of your data. It answers questions like:

Is email always a valid email address? (validity)
Are there duplicate rows in the orders table? (uniqueness)
Does every order_id in the line items table exist in the orders table? (referential integrity)
Is price always positive? (range/business rules)
Are null rates for critical columns within expected bounds? (completeness)

These checks require domain knowledge. Someone has to decide that price should be positive, that email should match a pattern, that country_code should be in a known list. The tool can automate the checking, but a human has to define what "correct" means.

Data quality catches problems like:

A third-party API started sending prices in cents instead of dollars
A form change allowed empty email addresses into the database
Duplicate records from a retry bug inflated conversion metrics by 15%
A timezone bug shifted all timestamps by 5 hours

These are data content failures. The pipeline worked fine. The data arrived on time, with the right schema, in the right volume. It was just wrong.

Where they overlap

The line between observability and quality isn't always clean. Some examples:

Volume anomalies sit in both camps. A sudden drop in row count could be a pipeline failure (observability) or a business change (quality). The monitoring is the same. The response is different.

Null rate spikes are technically a quality metric, but a sudden increase in nulls for a column that's always been 100% populated usually means something broke upstream. That's an observability signal.

Schema changes are pure observability, but they can cause data quality problems downstream. A column type change from int to varchar might not break the pipeline, but it could produce garbage in your aggregations.

Most modern tools handle both to some degree. The question is emphasis.

When to use which

Start with observability if:

You don't have any monitoring today and want coverage fast
Your biggest pain point is stale dashboards and broken pipelines
You want automated detection without writing rules for every table
You have hundreds of tables and can't manually define quality checks for all of them

Observability tools can start monitoring the day you connect. They learn what "normal" looks like and alert on deviations. No configuration needed for the basics.

Add quality checks when:

You have specific business rules that must always hold (prices > 0, no duplicate orders)
You're dealing with data from external sources you don't control
Regulatory compliance requires you to prove data accuracy
Your data powers ML models where subtle incorrectness compounds

Quality checks are more effort to set up but catch problems that observability misses. A table can be perfectly fresh, with the right schema and normal volume, and still be full of wrong data.

The practical answer: Start with observability for broad coverage, then layer quality checks on your most critical tables. You get 80% of the value from observability with 20% of the setup effort. Quality checks fill the gap for the tables where correctness actually matters.

How the tools stack up

Most tools in this space started on one side and expanded toward the other.

Observability-first tools (AnomalyArmor, Bigeye, Metaplane) give you automated schema, freshness, and volume monitoring out of the box. You connect a database, and within minutes you have baseline coverage across every table. Quality features were added later: custom metrics, validity rules, referential checks.

Governance-first tools (Monte Carlo) started with enterprise data governance, cataloging, and compliance, then expanded into observability and monitoring. They're comprehensive but come with enterprise pricing and longer setup times. If your primary need is pipeline monitoring, you're paying for a lot of surface area you don't use.

Quality-first tools (Great Expectations, Soda, dbt tests) start with explicit validation rules that you write. You define expectations ("this column should never be null," "row count should be between 1000 and 5000") and the tool checks them on a schedule. Observability features like freshness monitoring and lineage are bolted on or require additional setup.

The trend is convergence. Every observability tool now adds quality metrics. Every quality tool now has some form of freshness monitoring. Governance tools are expanding down-market. The difference is which side is mature and which side feels like an afterthought.

What to look for in practice

Skip the category debate and focus on what actually matters:

Time to first alert. How fast can you go from zero monitoring to getting notified when something breaks? If the answer is weeks of configuration, that's a quality-first tool pretending to do observability. If the answer is hours, that's observability-first, which is what you want for starting out.

False positive rate. A tool that alerts on everything is worse than no tool. AI-powered anomaly detection that learns your data's patterns produces fewer false alarms than static thresholds.

Custom rule support. At some point you'll need business-specific checks. Can you define custom SQL metrics? Can you set validity rules? Can you do referential integrity checks across tables?

Lineage. When something breaks, can you see what's affected downstream? Lineage turns a "this table looks weird" alert into "this table looks weird and it feeds your executive dashboard, the churn model, and the finance report."

Integration with your stack. Alerts should go where your team works (Slack, PagerDuty). The tool should connect to what you already run (dbt, Airflow, Snowflake, Databricks, PostgreSQL). Bonus points for AI agent integration via MCP so your coding assistant can check data health.

The bottom line

Data observability and data quality are complementary, not competing. Observability gives you broad, automated coverage across your entire data estate. Quality gives you precise, rule-based validation on critical data.

If you're starting from zero, start with observability. Connect your databases, get baseline monitoring, and stop finding out about broken pipelines from angry stakeholders. Then add quality checks where they matter most.

If you already have dbt tests or Great Expectations running, you have quality covered. Add observability to catch the problems that explicit tests can't: the pipeline that failed silently, the schema that changed without notice, the table that stopped updating on a holiday.

Either way, the goal is the same: find out about data problems before your stakeholders do.

Data Observability vs Data Quality FAQ

What is data observability?

Data observability is the practice of monitoring data systems end-to-end to understand the health, reliability, and performance of data pipelines. It tracks freshness, volume, schema changes, lineage, and incidents across your data stack. The term is borrowed from software observability but applied to data infrastructure.

What is data quality?

Data quality is the measure of how well data meets the needs of its users. It covers dimensions like accuracy, completeness, consistency, timeliness, uniqueness, and validity. Data quality focuses on the data itself, while observability focuses on the systems producing and moving the data.

Do I need both data observability and data quality?

Most production data teams need both. Observability catches pipeline failures, stale tables, and schema drift. Quality catches bad values, missing records, and business rule violations. They overlap in some areas (freshness, volume anomalies) but diverge in others (lineage vs validation rules). The cleanest approach is to use observability for infrastructure monitoring and quality rules for content validation.

What's the difference between data observability and data monitoring?

Data monitoring is a subset of data observability. Monitoring tracks specific metrics and fires alerts. Observability adds context: lineage showing which pipeline caused a problem, incident history, cross-system correlation, and root cause analysis. Observability is what you do with monitoring data to understand the why, not just the what.

What are the five pillars of data observability?

The five commonly cited pillars are: Freshness (is the data up to date?), Volume (is the expected amount of data arriving?), Schema (has the structure changed?), Lineage (what depends on what?), and Distribution (are the values within expected ranges?). Some vendors add Quality as a sixth pillar.

How does data observability differ from application observability?

Application observability tracks request latency, error rates, and resource usage in services. Data observability tracks data characteristics: freshness, volume, schema, and statistical properties. The underlying principle is the same (instrument everything so you can diagnose problems), but the metrics and tools are different.

What tools provide data observability?

Popular data observability platforms include AnomalyArmor, Monte Carlo, Metaplane, Bigeye, Datafold, Soda, and Databand. Open-source options include Great Expectations, Elementary, and re_data. Each has different strengths in terms of platform support, setup complexity, and price.

Is dbt enough for data quality?

dbt provides tests (schema tests, custom SQL tests) that work for deterministic validation inside your transformation layer. dbt is not enough for production data quality because it doesn't monitor raw source tables, doesn't track freshness across jobs, doesn't provide cross-pipeline lineage, and doesn't detect statistical anomalies. Most teams pair dbt tests with a data observability tool.

How much does data observability cost?

Pricing varies widely. Enterprise tools like Monte Carlo start at $15-25k/year for small deployments. Mid-market tools like Metaplane and AnomalyArmor price per monitored table, typically $5-10/table/month. Open-source tools have no license cost but require engineering time to maintain. Budget based on your number of tables and the criticality of your data.

Can I build data observability in-house?

Yes, but most teams outgrow custom solutions within 6-12 months. In-house data observability typically covers 2-3 pillars well (usually freshness and volume) but falls short on lineage, incident management, and statistical anomaly detection. If you have <20 critical tables and a strong data engineering team, in-house can work. Past that, buying a tool is cheaper than building.

AnomalyArmor combines data observability and quality monitoring in one platform. Try the schema drift demo to see how the AI agent handles both.