Forem: Taro.Matsui

Part 1: Snowflake's Autonomous Future

Taro.Matsui — Tue, 18 Nov 2025 15:52:58 +0000

Introduction

I'm Taro Matsui, Head of Technology Strategy at CCCMK Holdings, the company behind V Point, Japan's largest loyalty platform. I'm also a Snowflake Data Superhero—a recognition for community members who actively share knowledge and insights, particularly around data platform strategies.

While reviewing our technology roadmap recently, I noticed something significant: Snowflake's 2025 releases weren't just feature additions—they revealed a coherent vision for autonomous data platforms.

This article explores that vision. Rather than predict exact timelines—which inevitably shift—I'm mapping the trajectory toward true platform autonomy and what it means for us as data engineers.

On Terminology: I distinguish between automation (rule-based execution) and autonomy (AI-driven learning and decision-making). Snowflake's recent capabilities—Optima, Adaptive Warehouse, Query Insights—represent genuine autonomy, not mere automation.

Let's start by understanding where we are today.

In August 2025, Snowflake released Snowflake Optima to general availability (GA). While the initial release received limited attention, Snowflake published an official blog post about Optima on October 2nd that revealed its broader significance.

Snowflake Optima Documentation
Official Optima Blog Post

This wasn't just another feature release. Optima signaled something far more significant: the first step in Snowflake's long-term vision to move beyond "autonomous performance tuning" toward autonomous data platform operations.

The blog also discusses Warehouse Gen2's intelligent DML capabilities, demonstrating Snowflake's relentless pursuit of performance improvements.

Warehouse Gen2

This article examines two core autonomous capabilities—Snowflake Optima and Adaptive Warehouse—and explains how they, together with Query Insights as a transparency layer, shape Snowflake’s trajectory toward platform autonomy:

Current State: Today's capabilities
Structural Analysis: Drivers of this evolution
Future Scenarios: Transformation of data engineering
Strategic Implications: Preparation strategies

Important Context: This article presents my analysis based on official Snowflake releases and industry trends, not definitive predictions. My goal is to spark discussion about where data engineering is headed.

Key distinction:

Autonomy represents AI-powered advanced automation
Automation < Autonomy (autonomy encompasses automation)

Now, let's assess the current state.

Accelerating AI Integration in Snowflake's Platform

Since Snowflake Summit in June 2025, we've witnessed rapid AI integration into Snowflake's core platform capabilities. Let's start with Snowflake Optima.

What is Snowflake Optima?

Snowflake Optima is a new capability designed to autonomously handle query optimization. It reached general availability (GA) in August 2025. While the initial release received limited attention, Snowflake's October 2nd blog post revealed its broader strategic significance.

Key Features

Optima Indexing

Identifies frequently executed point lookup queries and automatically generates and maintains hidden indexes in the background.

Dynamic Workload Distribution

Analyzes query execution load in real-time and dynamically adjusts compute resources and parallelism as needed.

Automatic Plan Optimization

Enhances optimization during query compilation, including join order and improved statistical accuracy (number of distinct values (NDV) estimation).

Critical Point: No Additional Cost

With Gen2 Warehouses —Snowflake's next-generation compute architecture—Optima activates automatically at no additional cost. Existing Snowflake users benefit without any action required.

Official Documentation

How Optima Relates to Search Optimization Service

You might wonder: "Isn't Optima just Search Optimization Service (SOS) automation?" I initially thought so too.

However, detailed analysis of the official documentation reveals key differences.

Architectural Positioning

The official documentation states that "Optima Indexing is built on top of the Search Optimization Service."

This indicates extension, not replacement.

Comparison	Search Optimization Service	Snowflake Optima
Activation	Manual configuration required (`ALTER TABLE ADD SEARCH OPTIMIZATION`)	Automatically enabled with Gen2
Cost	Storage and compute costs incurred	No additional cost
Guarantee level	Reliable index maintenance	Best effort
Scope	Entire table or specific columns	Limited to repetitive query patterns
Optimization decision	User decides upfront	System autonomously decides based on workload analysis

Usage Guidelines

The official documentation clearly recommends differentiated usage:

Use Optima when: You want general workloads to benefit from automatic optimization—no configuration, no cost
Use SOS when: Mission-critical workloads require guaranteed index freshness and consistent performance (e.g., real-time threat detection)

In essence, Optima handles general workloads autonomously, while SOS ensures guaranteed performance for mission-critical operations.

Projected Roadmap for Snowflake Optima

Currently, Optima appears positioned as a managed service layer over SOS. However, I predict it will evolve into a more comprehensive service encompassing clustering keys and automatic materialized view generation for frequent query patterns.

If these capabilities become available at no additional cost—or minimal cost (clustering key and MV creation only)—we'll achieve "performance without tuning."

This will eliminate most performance tuning work traditionally handled by expert data engineers.

What This Means for Data Teams

Optima's autonomy marks a fundamental shift: performance optimization moving from specialized expertise to platform intelligence. For data teams, this means:

Junior engineers gain senior-level optimization capabilities
Senior engineers redirect focus from tuning to strategy
Organizations reduce dependency on scarce expertise

This isn't just a feature—it's a catalyst for workforce transformation.

How much time does your team spend on performance tuning today? What could they accomplish if freed from this burden?

Query Insights Release

Another critical capability, Query Insights in Snowsight, reached general availability on October 7th. This feature identifies performance bottlenecks in detected queries, explains their impact, and provides improvement recommendations to engineers.

Query Insights

This democratizes what was once a specialized skill—performance bottleneck investigation—previously limited to expert data engineers. Given the limited number of expert engineers, the ability to efficiently investigate and address even non-critical workloads is extremely valuable.

Adaptive Warehouse (Future Release)

A third critical capability, Adaptive Warehouse (Adaptive Compute), was announced before Optima.

The Role of Adaptive Warehouse

Introduced to select enterprises in private preview in June 2025, this capability automates warehouse-level workload optimization:

Automatic warehouse size adjustment
Dynamic parallelism changes
Automatic cluster scaling
Query routing optimization

Traditionally, humans configured these settings and periodically reviewed them. Adaptive Warehouse learns workload patterns and autonomously adjusts them.

A Complete Autonomy Stack: Three Integrated Capabilities

Combining these evolving capabilities with Adaptive Warehouse enables autonomous optimization across all query execution layers.

Layer	Traditional Manual Optimization	Autonomous Capability
Data Access	Clustering key design, SOS configuration, materialized views	Snowflake Optima Builds on SOS
Resource Allocation	Warehouse size/cluster adjustment, QAS configuration	Adaptive Warehouse
Workload Analysis	Query performance, cost management, resource monitors	Query Insights

Query Insights functions as the "transparency layer" for these capabilities. By presenting how Optima optimized and how Adaptive Warehouse adjusted resources in a human-readable way, it facilitates human-AI collaboration.

When these become standard features available at minimal cost, the performance tuning work traditionally handled by expert data engineers will be dramatically reduced.

Snowflake's "Simplicity" Philosophy

These autonomous capabilities aren't accidental. They embody the "Simplicity" philosophy emphasized by CEO Sridhar Ramaswamy at Snowflake Summit 2025.

"Complexity is the Root of All Problems"

CEO Ramaswamy stated in his keynote:

"Complexity creates risk. Complexity creates cost. Complexity creates friction. Simplicity drives results, and that is why Snowflake holds simplicity at the heart of our design."

This philosophy stems from recognizing that:

Complex systems are prone to failures
Troubleshooting takes extensive time
Customer trust erodes as a result

The worst outcome is a negative spiral: adding more complexity to restore trust.

Snowflake's Solution: Hiding Complexity Through Abstraction

Snowflake's strategy isn't to eliminate complexity but to absorb complexity into the platform and hide it from users:

Optima hides index design complexity (eventually clustering keys too)
Adaptive Warehouse hides resource management complexity
Query Insights provides support and transparency for query evaluation
Together, they abstract and simplify the specialized skill of performance tuning

A Veteran Engineer's Dilemma

Over recent years, my role has expanded beyond data platforms to overseeing development across many systems, leaving me with far less time for hands-on data work.

As Snowflake adoption grew across the organization, I ran into a persistent challenge.

I wanted my team focused on data management and business contribution. Yet cost control demanded performance tuning skills—skills that don't come easily. Teaching "clustering key selection" in training sessions didn't translate to true understanding without practical experience.

"How do I transfer this expertise?" This question kept me up at night.

Snowflake offered an elegantly simple solution:

"We'll handle the complexity. You focus on what matters."

Initially skeptical, recent feature releases have made this feel increasingly realistic.

This raises a fundamental question:

How far can autonomous platforms really go?

Part 2: Structural Analysis - What Lies Beyond This Evolution

From "Performance Without Tuning" to "Engineering Without Operations"

Snowflake's autonomy strategy evolves in stages. Based on current releases and Snowflake's historical development cycle, I see this evolution unfolding across three distinct phases—each building on the autonomy achieved before.

Phase 1: Autonomous Performance Optimization (~2026)

Goal: "Performance without tuning"

Key capabilities:

Automatic micro-partition management
Query Acceleration Service (QAS)
Search Optimization Service (SOS)
Gen2 Warehouse (May 2025 GA)
Snowflake Optima (August 2025 GA)
Query Insights (October 2025 GA)
Adaptive Warehouse (June 2025 Private Preview)

This phase introduces foundational capabilities for automatically improving query performance:

Gen2 Warehouse: Enhanced baseline performance
Snowflake Optima: Autonomous optimization services
Query Insights: Query optimization transparency
Adaptive Warehouse: Optimal resource allocation

As autonomy advances, cost evaluation capabilities become critical. Snowflake has already rolled out FinOps features—monitoring, alerts, and budget controls—demonstrating this commitment.

FinOps Foundation Membership

This trajectory feels increasingly inevitable.

Phase 2: Autonomous Operations Management (2026-2027 Projection)

Once low-level tasks like query performance become autonomously optimized, what comes next?

Based on current capabilities—Cortex AI's data profiling and Universal Search's metadata integration—the technical foundations for autonomous data management are already in place. The next autonomy frontier will be applying these AI capabilities to automate data quality checks, pipeline monitoring, and metadata management.

Snowflake's historical release cycle shows preview-to-GA typically takes 6-12 months. Preview launches in 2026, selective GA in 2027, and major Phase 2 capabilities GA by 2028 seem realistic.

Expected timeline:

2026: limited feature previews (automatic metadata collection, etc.)
2027: Staged GA rollouts (starting with simpler capabilities)
2028: Major Phase 2 features reach GA

Goal: "Engineering without operations"

Projected capabilities:

Autonomous data pipeline operations: AI-driven error diagnosis and auto-remediation, dynamic scheduling optimization
Autonomous data quality management: Automatic anomaly detection rule generation, autonomous observability, table lifecycle management
Autonomous metadata management: Automated data profiling and metadata, semantic ecosystems via OSI (Open Semantic Interface)
Unstructured data handling: Automatic vectorization of internal documents, images, audio, and video
Advanced data modeling support: Automatic materialized view generation based on query patterns, automatic clustering key creation

Snowflake Openflow (data movement and ingestion service) announced in 2025 will further accelerate this autonomy.

Snowflake Openflow

This phase autonomizes the data platform operation itself.
At this level of autonomy, a new question emerges: "What will data engineers actually do?"

My answer would redefine the role:

"I want to use data to drive greater business impact!"

Phase 3: Autonomous Data Strategy (2027-2028 Staged GA Projection)

Goal: Democratizing data-driven value creation

In this phase, data engineers shift focus to data strategy, enabling all business users to create value with data.

Applicability to Other Platforms

While this article focuses on Snowflake, the automation and autonomy trends apply across the entire data platform landscape:

These examples represent just the beginning. AI-driven autonomous optimization will become a competitive battleground among all major platforms.

Vendor lock-in remains a valid concern—over-reliance on Snowflake carries risks, requiring careful attention to other platforms' developments.

We ourselves use Databricks and BigQuery alongside Snowflake, maintaining a multi-platform approach.

That said, Snowflake's vibrant community provides invaluable information and insightful analysis, making it hard not to rely on them.

In Part 2, we'll dive into specific scenarios and career implications.

What Comes Next

This concludes Part 1's analysis of Snowflake's autonomous platform trajectory. We've seen how Optima, Adaptive Warehouse, and Query Insights form the foundation for platform autonomy.

But what does this actually mean for your career?

In Part 2, we'll explore:

Three specific Phase 3 capabilities (2027-2028)
Three emerging career paths for data engineers
How to prepare for the autonomous era

Continue to Part 2 →

Author's Note: As a Snowflake Data Superhero, I engage regularly with Snowflake's product teams and community. However, all predictions, timelines, and interpretations in this article represent my personal analysis, not Snowflake's official roadmap or strategy. I write as an independent technologist and community contributor, not as a company representative.

Part 2: Snowflake's Autonomous Future

Taro.Matsui — Mon, 13 Oct 2025 13:35:17 +0000

Part 2: The Data Engineer of 2028 - Three Career Paths in the Age of Autonomous Platforms

Part1 Post

Introduction

Part 1 mapped Snowflake's three-phase evolution toward autonomous platforms. But what does this actually mean for you as a data engineer? And more importantly—how should you prepare?
In Part 2, we answer that question by diving into Phase 3's concrete capabilities and the three career trajectories they unlock.

Future Scenarios: The Phase 3 World

Let me share three specific capabilities I anticipate in Phase 3 (2027–2028).

Prediction 1: Natural Language Data Platform Design

The Transformation
Business users describe requirements in plain natural language (no SQL or technical syntax required), and AI automatically proposes data architectures.

In Practice
A request like "I want to analyze customer purchase history every Monday morning and generate reports" will be fulfilled within minutes:

Identifying required data sources
Proposing pipeline designs
Recommending optimal data models
Estimating costs and performance

The Shifting Role of Data Engineers

Traditional approach: Listen to requirements, spend days to weeks on design and implementation

Future approach: Review AI proposals and validate business logic—a shift to oversight and approval

Why This Is Feasible

Cortex Analyst already generates SQL from natural language. Snowflake Copilot handles DDL/DML operations. Integrating these capabilities makes this technically achievable.

That said, capturing all organizational context in Snowflake remains challenging. Full autonomy won't be achieved—data engineers will retain critical decision-making responsibilities.

Prediction 2: Self-Evolving Data Models

The Transformation
Data platforms learn workload changes and autonomously evolve their data models.

In Practice
Traditional approach: Humans periodically decide, "Query patterns are increasing—let's create a materialized view."

Future approach: AI detects query pattern changes and automates responses:

Proposing optimizations (automatic MV generation)
Estimating cost reduction impacts
Validating post-implementation effectiveness
Automatically rolling back when benefits don't materialize

The Revolutionary Impact

This marks a shift from "static data models" to "dynamic architectures that evolve continuously with business needs."

Data engineers move from "daily optimization tasks" to "long-term data strategy."

Prediction 3: Automated Semantic Layer Construction

The Transformation
Business terminology automatically maps to data, enabling everyone to work with data naturally.

In Practice
For a question like "What were the top 10 products by revenue in last year's Q4?":

Automatically interprets "revenue," "quarter," and "products"
Applies organization-specific definitions (fiscal year, etc.)
Generates consistent answers

Why This Is Feasible
This foundation is expected to emerge by 2026.

Beyond that, these definitions will unify across all tools—Tableau, Power BI, Looker—eliminating the "numbers vary by tool" problem.

Open Semantic Interface (OSI) standardized in 2025, and Snowflake Horizon Catalog already provides metadata management. Integration should enable this by 2026–2027.

Organizational Impact

Data engineers shift from "terminology translators" to strategic value creators.

Upstream data modeling—before platform integration—becomes increasingly critical. Data engineers will govern data architecture and modeling across the entire organization and business, not just the platform.

This shift elevates data engineering's strategic importance to the business.

What This Phase Delivers

The New Role of Data Engineers

Traditional: "Data maintainers" (focused on keeping systems running)
2027: "Data strategists" (focused on driving business outcomes)

Specifically:

Translators: Communicating business vision to AI
Strategists: Evaluating and approving AI decisions
Innovators: Creating new business value through data

The following outlines an ideal scenario. In practice, organizational maturity and technical constraints will likely extend the timeline.

Yet within a few years, pioneering organizations will begin realizing this vision.

A Data Engineer's Week (2027–2028 Vision)

Operations (~20%):

Reviewing and approving AI-generated optimization proposals (not implementing them)

Strategy (~80%):

Engaging with business units to clarify new requirements
Evaluating new technologies and business applicability
Building and operating data governance frameworks
Proposing data strategies to leadership

These percentages illustrate a fundamental inversion: from "operations-heavy, strategy-light" to strategy-dominant workloads.

This operational-to-strategic shift transforms data engineers from cost centers to profit centers.

"Data is the oil of the 21st century," as the saying goes. But like crude oil, raw data doesn't drive business. It must be refined, processed, and productized to create value.

Managing the production process? That's a cost center.

Conceiving the product itself and managing its creation? That's a profit center.

The Evolution of Data Engineer Value:

Traditional: Evaluated by "technical skills"
2027: Evaluated by "business impact"

"How many pipelines built?" → "How much revenue generated?"
"Achieved zero incidents?" → "Created new business opportunities?"
"Reduced costs?" → "Established competitive advantage through data?"

Snowflake's Ultimate Question:

Will you transform the world with data—or merely manage it?

Our answer:

We will transform business and the world through data.

Long-term Strategy (Beyond 2028): Evolution to Data Strategist

Vision: From "Data Gatekeepers" to "Business Creators"

In 2028, the data engineering profession transforms fundamentally.

Traditional Responsibilities:

Building and operating data platforms
Performance optimization
Incident response

Post-2028 Responsibilities:

Proposing data-driven business strategies
Creating value through AI agent collaboration
Planning and launching data-enabled new ventures

This represents elevated status for data engineers.

Emerging Career Paths

I once viewed data engineering as "backstage work." But participating in executive meetings changed my perspective. Watching executives analyze data to formulate strategy and observing diverse approaches to generating new data shifted my understanding: data engineers should partner with leadership.

This realization led me to identify three emerging career patterns.

Path A: Data Visionary

Role: Designing future businesses powered by data

Core Responsibilities:

Formulating data strategies through executive dialogue
Planning new data-enabled ventures
Discovering and leveraging data assets for competitive advantage

Required Skills:

Business strategy comprehension
Data-driven insight extraction
Executive and business communication skills

Path B: AI Orchestrator

Role: Overseeing multiple AI agents and optimizing the entire data platform

Core Responsibilities:

Supervising AI agent decisions (Optima, Adaptive Warehouse, Cortex, etc.)
Managing anomalies and edge cases
Proposing AI decision logic improvements

Required Skills:

Foundational AI/ML understanding
Deep data architecture knowledge
Anomaly detection and analysis
A holistic understanding of system integration

Path C: Data Product Manager

Role: Treating data as a product for internal and external delivery

Core Responsibilities:

Planning, developing, and operating data products
Advancing Data as a Service
Operating data marketplaces

Required Skills:

Product management
User experience design
Business model development
Marketing and sales capabilities

Recommended Team Structure (2028)

Data Strategy Team (10 members):

Data Visionaries: 2 members
- Business strategy, new venture planning
AI Orchestrators: 3 members
- Platform oversight, edge case handling
Data Product Managers: 2 members
- Internal/external data product operations
Data Analysts: 2 members
- Business analysis, dashboard development
Platform Architect: 1 member
- Deep technical expertise, new technology evaluation

Note: In this autonomous future, dedicated roles for traditional infrastructure operations and performance tuning shift toward AI oversight and business strategy. Organizations in transition may maintain hybrid models.

The Essential Mindset Shift

Question: Will autonomy eliminate our jobs?

Answer: No. Your role gets elevated.

Traditional: "Optimize this slow query" → Takes days
2026: AI autonomously optimizes → Time freed for new business planning

Traditional: "Pipeline broke" → Half-day investigation and fix
2027: AI automatically remediates → Time freed for strategy formulation

Traditional: Incident response consumes all bandwidth
2028: Delegate to AI, discuss future with executives

Dispelling a Critical Misconception

"Autonomy makes data engineers obsolete" is false. Autonomy drives the evolution of the profession, not its extinction.

Data engineers aren't disappearing. They're moving toward more strategic, higher-value work.

How We Measure Value Is Changing

Traditional: "Technical Skills" × "Implementation Speed"

2028: "Business Acumen" × "AI Collaboration" × "Strategic Thinking"

Autonomy isn't a threat—it's an ally liberating you from routine work to focus on what truly matters.

Autonomy's Limits and the Human Role

Autonomy has boundaries.

What AI Still Can't Decide:

Deep business context understanding
Data quality judgment
Stakeholder interest reconciliation
Ethical and legal decisions
Security risk assessment
Creative problem-solving

These remain human domains beyond 2028. Data engineers’ value doesn’t vanish—it shifts to higher-level, more creative territories across the data landscape.

If I paraphrase the three message pillars from Snowflake World Tour Tokyo:

Trust Snowflake, liberate yourself from complexity, connect across boundaries—and transform the world.

A Note on These Predictions

As a Data Superhero, I have the privilege of early access to some features, regular conversations with Snowflake's product teams, and deep engagement with the global data community. These insights inform my predictions—but they remain educated speculation, not guaranteed timelines.

The value isn't in being precisely correct about "when." It's in having a directional framework to:

Prepare your skills and career
Advocate for the right capabilities at your organization
Engage productively with platform evolution

Whether Snowflake, Databricks, BigQuery, or others lead specific capabilities, the industry trajectory toward autonomy is clear.

Conclusion

I've worked as a data engineer for many years. One thought persisted:

"Performance tuning matters. Incident response matters. Data management matters. But what I truly want is to transform business and society through data."

Snowflake's "Simplicity" philosophy—hiding complexity—sends us a message: "Focus on what truly matters."

I'm an optimist who embraces change. For better or worse, I'm a dreamer—and my wild ideas occasionally confuse those around me.

These timelines are speculation. Actual releases may shift due to technical challenges or market dynamics. What matters isn't 'when' but 'direction'.

AI's evolution is remarkable. While we feel our own productivity gains, mega-platforms are accelerating development even faster.

Predictable futures become achievable. They arrive faster than anticipated.

The Future This Article Envisions

Snowflake's features are being reverse-engineered from their vision of data engineering's future. This trend has accelerated significantly this year.

This is the dawn of a new data engineering era.

Phase 1 (~2026):
Autonomous performance tuning → Liberation from technical optimization

Phase 2 (2026–2027):
Autonomous operations → Minimizing data management tasks

Phase 3 (2027–2028):
Autonomous data strategy → Platforms automatically built from business requirements

The ultimate destination:

Data engineers fully liberated from reactive work, empowered to drive proactive innovation.

This forecast may prove inaccurate. But this kind of directional thinking deepens our understanding of what's possible. And if I'm wrong? I'll have plenty of specific feature requests ready to share with Snowflake!

To You, the Reader

If you worry autonomy will eliminate your job, I hope this article offers hope.

If daily operations and tuning prevent you from doing what you truly want:

Autonomy isn't your enemy. It's your ally.

To drive greater business impact within limited time, we're delegating more to AI—and that's strategic.

Drive meaningful change with data.
Build successful businesses through data.
Improve lives with data.

We need time for this. Snowflake is giving us that time.

So here's my final question to you:

Data engineers will become more valuable than ever—will you join me in shaping that future?

One Last Thing: A Request for Snowflake

If anyone from Snowflake reads this—one request:

Please accelerate Adaptive Warehouse's public preview!
(We're waiting eagerly!)

I'd love to hear your thoughts—share your perspective in the comments below, or connect with me on LinkedIn.