Forem: Alex Merced

Apache Data Lakehouse Weekly: April 9–15, 2026

Alex Merced — Wed, 15 Apr 2026 19:54:05 +0000

The Iceberg Summit wrapped in San Francisco, leaving behind a set of in-person alignments that are now surfacing as concrete proposals on the dev lists. Parquet's ALP encoding vote closed, Polaris 1.4.0 planning accelerated, and Arrow's engineering community tackled two interlinked decisions about its future Java baseline and AI tooling policy. The post-summit week is when talk becomes code.

Apache Iceberg

The two days in San Francisco established alignment on the discussions that have dominated the dev list all spring. The V4 metadata.json optionality thread drew the largest in-person audience of any design session, with Anton Okolnychyi, Yufei Gu, Shawn Chang, and Steven Wu working through the portability and static-table implications of making the root JSON file optional when a catalog manages metadata state. The direction that emerged favors catalog-managed metadata as a first-class supported mode, with portability guarantees preserved through explicit opt-in semantics rather than the current default assumption.

The one-file commits design — the work Russell Spitzer and Amogh Jahagirdar have been advancing through multiple proposals — is heading toward a formal spec write-up following alignment reached at the summit. The approach replaces manifest lists with root manifests and uses manifest delete vectors to enable single-file commits, promising dramatic reductions in commit latency and metadata storage footprint. This is one of the most consequential V4 changes for high-frequency write workloads, and the in-person sessions cleared the remaining design disagreements about inline versus external manifest delete vectors.

Péter Váry's efficient column updates proposal for AI and ML workloads drew real interest at the summit. The design targets wide tables where only a subset of columns change on each write — embedding vectors, model scores, feature values — allowing Iceberg to write only the updated columns to separate files and merge at read time. For teams managing petabyte-scale feature stores, the I/O savings are significant. Péter indicated that a formal proposal with POC benchmarks would land on the dev list in the days following the summit.

The AI contribution policy that pulled in Holden Karau, Kevin Liu, Steve Loughran, and Sung Yun over the preceding weeks moved toward practical resolution. The summit provided the in-person clarity that async debate rarely does, and a working policy covering disclosure requirements and code provenance standards for AI-generated contributions is expected to be published on the dev list this week.

Apache Polaris

Polaris is one month past its February 18 graduation as a top-level Apache project, and the governance machinery is running. Jean-Baptiste Onofré's first board report as a TLP covers the March 26 ASF board meeting, documenting community health, development progress, and strategic direction under Polaris's own PMC. JB also joined the Apache Software Foundation board itself as a Dremio-nominated director, a governance milestone that deepens the open-source commitment across the entire ecosystem.

The Apache Ranger authorization RFC from Selvamohan Neethiraj remained the most active technical discussion thread. The design allows organizations running Ranger alongside Hive, Spark, and Trino to manage Polaris security within a unified governance framework, eliminating the policy duplication that arises when teams bolt separate authorization systems onto each engine. The plugin is opt-in and backward compatible with Polaris's existing internal authorization layer, a design choice that lowers the enterprise adoption barrier considerably.

The 1.4.0 release — Polaris's first as a graduated project — is now in active scope finalization. Credential vending for Azure and Google Cloud Storage is the headline feature, alongside catalog federation design that lets Polaris front for multiple catalog backends in multi-cloud deployments. With incubator overhead behind it, release velocity is expected to accelerate. Watch the dev list this week for a 1.4.0 milestone thread and vote timeline.

Apache Arrow

Jean-Baptiste Onofré's thread proposing JDK 17 as the minimum version for Arrow Java 20.0.0 is approaching decision. Contributors including Micah Kornfield and Antoine Pitrou have been weighing in, and the practical rationale is compelling: setting JDK 17 as the floor would align Arrow's Java modernization with Iceberg's own upgrade timeline, effectively raising the minimum across the entire lakehouse stack in a single coordinated move. The decision is expected to land before the 20.0.0 release cycle formally opens.

The arrow-rs 58.2.0 release was on track for April, following the 58.1.0 shipment in March, which arrived with no breaking API changes. The Rust implementation has become one of the most actively maintained segments of the Arrow ecosystem, with a release cadence that matches growing adoption in query engines that want Arrow's columnar format without a JVM dependency.

Nic Crane's thread on using LLMs for Arrow project maintenance continued to generate thoughtful discussion. The framing — AI as a resource for maintainers rather than just contributors — is distinct from how Iceberg and Polaris are approaching the same question. Arrow's angle is practical: a lean maintainer group managing a growing issue backlog needs help triaging, and LLMs can do that work without introducing the code-provenance concerns that matter for contributions. Google Summer of Code 2026 student proposals arrived this week, with interest concentrated in compute kernels and language bindings for Go and Swift, adding bandwidth to a project that will need it as the 20.0.0 cycle opens.

Apache Parquet

The ALP (Adaptive Lossless floating-Point) encoding specification vote closed this week, marking one of the most meaningful additions to the Parquet specification in recent memory. ALP encodes floating-point exponents and mantissas separately, delivering significantly better compression ratios for float-heavy columns. The practical beneficiaries are ML feature stores and scientific computing workloads, where columns full of embedding coordinates and model outputs are common. Months of careful spec review paid off.

The Variant type that shipped in February has been generating follow-on integration discussion across engine teams. Spark, Trino, and Dremio contributors compared notes on their implementation experiences this week, working through edge cases in semi-structured data handling that the spec leaves partially open. Getting these implementations to converge matters: Parquet's value as a cross-engine format depends on consistent behavior, and Variant is novel enough that divergence between engines would fragment the ecosystem.

The File logical type proposal — which would allow Parquet files to natively embed unstructured data including images, PDFs, and audio as columnar records — continued advancing through community discussion. Alongside Variant, this proposal signals a deliberate effort to evolve Parquet from a purely analytical format into a unified storage layer capable of managing the diverse data shapes that AI and ML pipelines produce. The direction is ambitious and the community engagement is substantive.

Cross-Project Themes

The post-summit week is when the conversations that happened in person translate back into the formal proposals and vote threads that actually change the projects. Across all four lists, expect the next two weeks to be among the most active of 2026 as in-person alignments hit the dev list in concrete form.

The second theme connecting all four projects is the deliberate expansion of format scope to meet AI workload demands. Parquet's ALP acceptance, the File logical type proposal, Iceberg's efficient column updates for wide ML tables, Polaris's Ranger integration and federation work, and Arrow's JDK 17 modernization are all responses to the same underlying pressure: the lakehouse stack is being asked to power AI pipelines, not just analytical queries. The pace of that evolution is accelerating, and the summit put the community's roadmap on the same page.

Looking Ahead

Watch the Iceberg dev list for the V4 metadata optionality formal proposal, the single-file commits spec write-up, and a published AI contribution policy. The Polaris 1.4.0 milestone thread and vote timeline should also land this week. Arrow's JDK 17 decision for Java 20.0.0 will likely follow close behind. The summit session recordings will appear on YouTube in the weeks ahead — an excellent resource for anyone who missed San Francisco.

Resources & Further Learning

Get Started with Dremio

Try Dremio Free — Build your lakehouse on Iceberg with a free trial
Build a Lakehouse with Iceberg, Parquet, Polaris & Arrow — Learn how Dremio brings the open lakehouse stack together

Free Downloads

Apache Iceberg: The Definitive Guide — O'Reilly book, free download
Apache Polaris: The Definitive Guide — O'Reilly book, free download

Books by Alex Merced

AI Weekly: Agents, Models, and Chips — April 9–15, 2026

Alex Merced — Wed, 15 Apr 2026 19:52:21 +0000

Three stories shaped the past week: AI coding tools are merging into unified agentic stacks, a wave of new language models raised the multimodal baseline across the industry, and chipmakers moved hardware designed specifically for agentic workloads into general availability. Here is what you need to know.

AI Coding Tools: One Stack Nobody Planned

The first week of April confirmed a trend that has been building all year: Cursor, Claude Code, and OpenAI Codex are converging into a single development environment rather than competing as standalone tools. Cursor shipped a rebuilt interface for orchestrating parallel agents, and OpenAI published an official plugin that runs inside Claude Code. Early adopters are already running all three together, treating Cursor as the interface layer, Claude Code as the reasoning engine, and Codex for code-specific generation.

The numbers back the urgency of this convergence. A Stack Overflow Developer Survey released this week puts daily AI coding tool usage at 84% of developers — but only 29% trust AI-generated code in production without review. That trust gap is the product problem the new integrated stacks are designed to solve, giving teams a single debuggable environment instead of three black boxes.

Claude Desktop and Cursor both shipped full MCP v2.1 support during this period, making tool discovery and invocation consistent across both clients. Microsoft also shipped Agent Framework 1.0 this week with stable APIs, a long-term support commitment, and full MCP support built in, along with a browser-based DevUI that visualizes agent execution and tool calls in real time. For enterprise teams, this is the most concrete sign yet that the MCP-plus-A2A architecture is becoming the default for production agentic systems.

AI Models: Multimodal Is Now the Baseline

April 2026 has become the most packed month for LLM releases on record, and the defining pattern is that pure-text models no longer ship. Every major release this week handles text, images, and at minimum one additional modality.

The headline model is Claude Mythos Preview, which Anthropic announced on April 7, available to roughly 50 partner organizations through Project Glasswing. Focused on cybersecurity vulnerability detection, reasoning, and coding, Mythos scores 93.9% on SWE-bench Verified and 94.6% on GPQA Diamond. Anthropic describes it as a step change above Claude Opus 4.6. Preview pricing sits at $25 per million input tokens and $125 per million output tokens, reflecting the gated early-access nature of the program. No public release date has been announced.

Google released the Gemma 4 family on April 2 under Apache 2.0, delivering four variants purpose-built for different deployment scenarios. Zhipu AI shipped GLM-5.1, a 744B mixture-of-experts model under MIT license, and GLM-5V-Turbo adds vision-to-code capability. Alibaba's Qwen 3.6-Plus targets agentic coding with a 1 million token context window. The gap between proprietary and open-weight models has narrowed significantly — Chinese labs are shipping models that rival the best US offerings on many benchmarks while publishing weights under permissive licenses.

AI Chipsets: Blackwell Reaches More Desks

The NVIDIA RTX PRO 5000 72GB Blackwell GPU reached general availability on April 9, expanding memory options for desktop agentic AI workloads. The 72GB variant joins the existing 48GB model, giving AI developers and data scientists the option to right-size memory for larger context windows and heavier fine-tuning runs without moving to a data center rack. Demand for Blackwell-class compute is at an all-time high.

Nvidia's Rubin platform is in full production, with partners scheduled to deploy Rubin-based instances in the second half of 2026. AWS, Google Cloud, Microsoft, and OCI are among the first cloud providers lined up. The Vera Rubin NVL72 rack-scale system, which packs 72 Rubin GPUs, will feature in Microsoft's next-generation AI data centers. The Rubin platform combines six new chips targeting training, inference, and networking in a single coordinated architecture designed for environments that may eventually reach one million GPUs.

On the design side, Nvidia revealed this week that AI has compressed a 10-month, eight-engineer GPU design task into an overnight job. The company is applying AI across every stage of chip design, though engineers emphasize there is still a long way to go before humans are removed from the process entirely.

Standards and Protocols: A2A Turns One

April 9, 2026 marked the one-year anniversary of Google's Agent-to-Agent Protocol. The numbers tell a strong adoption story: more than 150 organizations now participate, the GitHub repo has passed 22,000 stars, and production deployments exist inside Azure AI Foundry and Amazon Bedrock AgentCore. A year ago, A2A launched with 50 partners. Today it functions as the horizontal coordination bus for inter-agent communication across Microsoft, AWS, Salesforce, SAP, and ServiceNow.

The v1.0 release introduced Signed Agent Cards, which let agents cryptographically verify each other's identities before delegating tasks. The AP2 extension, which ties A2A into payment and commerce transaction workflows, arrived as a formal extension alongside the anniversary. Combined with IBM's Agent Communication Protocol merging into A2A in 2025, the protocol now covers the full lifecycle from tool access to inter-agent delegation to commerce.

The Linux Foundation's Agentic AI Foundation now serves as the permanent governance home for both MCP and A2A, co-founded by OpenAI, Anthropic, Google, Microsoft, AWS, and Block. For practitioners, the layered model is now clear: MCP handles the vertical connection from agent to tools and data sources; A2A handles the horizontal coordination between agents. Any production agentic system you build in 2026 needs both.

Resources to Go Further

The AI landscape changes fast. Here are tools and resources to help you keep pace.

Try Dremio Free — Experience agentic analytics and an Apache Iceberg-powered lakehouse. Start your free trial

Learn Agentic AI with Data — Dremio's agentic analytics features let your AI agents query and act on live data. Explore Dremio Agentic AI

Join the Community — Connect with data engineers and AI practitioners building on open standards. Join the Dremio Developer Community

Book: The 2026 Guide to AI-Assisted Development — Covers prompt engineering, agent workflows, MCP, evaluation, security, and career paths. Get it on Amazon

Book: Using AI Agents for Data Engineering and Data Analysis — A practical guide to Claude Code, Google Antigravity, OpenAI Codex, and more. Get it on Amazon

Agentic Analytics on the Apache Lakehouse

Alex Merced — Mon, 13 Apr 2026 22:39:29 +0000

Read the complete Open Source and the Lakehouse series:

If you grant a Large Language Model direct access to a raw Amazon S3 bucket filled with Parquet files, it will fail to answer your business questions. AI agents possess immense processing power, but they lack inherent business knowledge.

To execute agentic analytics safely and accurately, an AI agent requires three things: deep business context, universal governed access, and interactive speed. The Apache open-source data lakehouse stack provides the foundation for those requirements, but you must bridge the gap between raw data and machine intelligence.

The Hallucination Trap

Consider a raw data table containing a column named cst_act_flg. A human analyst working at the company for five years knows this stands for "Customer Account Flag." An AI agent does not. If a user asks the agent to "Show me active customers," the agent guesses meaning from the abbreviation. Guessing leads directly to hallucinations.

Raw data lakes optimize for machine storage, not semantic understanding. To prevent hallucinations, you must teach the AI your specific business language.

Teaching AI with the Semantic Layer

The semantic layer acts as a translation layer between technical schemas and business logic. It provides the context that transforms a generic LLM into an accurate agentic analyst.

In the Dremio platform, the Semantic Layer is built through Virtual Datasets. Engineers create logical views that rename cst_act_flg to Active_Customer_Status. Dremio takes this a step further by using generative AI to automatically document these datasets. By sampling table data and analyzing schemas, Dremio generates detailed Wikis and Tags for your Apache Iceberg tables.

When an AI agent receives a user prompt, it first reads these semantic Wikis. The documentation effectively teaches the AI agent the definitions of your specific business metrics before it attempts to write SQL, ensuring remarkably high accuracy.

Autonomous Reflections: AI Accelerating AI

Agentic analytics creates a massive new compute burden. When executives and business lines can ask natural language questions, the volume of unpredictable SQL queries skyrockets. Human database administrators cannot manually tune indexes or write materialized views fast enough to support this scale.

You need AI to accelerate AI. Dremio tackles this with Autonomous Reflections. The platform continuously monitors query patterns—originating from both humans and AI agents—over a seven-day rolling window.

When Dremio identifies a bottleneck, it automatically acts. It creates, maintains, and drops "Reflections" (pre-computed, highly optimized Iceberg materializations of the data) entirely in the background. Performance becomes an automated byproduct of the architecture, rather than a manual engineering chore.

Text-to-SQL and Native AI Functions

With context and speed resolved, users can interact directly with the agentic interfaces. Dremio includes a built-in AI Agent capable of discovering datasets, exploring relationships, and visualizing answers. Because the agent is grounded in the AI Semantic Layer and the open Apache Polaris catalog, Text-to-SQL translations actually hit the right tables.

But agentic analytics is not limited to text-to-SQL. Dremio exposes LLM capabilities directly inside the SQL engine itself.

Using native AI SQL functions like AI_CLASSIFY or AI_GENERATE, analysts can run sentiment analysis on unstructured product reviews directly within a standard SELECT statement. This eliminates the need to export data into external Python pipelines just to leverage modern generative AI models.

The Fully Realized Agentic Lakehouse

This 7-part series mapped the evolution of the modern data architecture.

It starts with the strict vendor-neutral governance of the Apache Software Foundation. You store data highly compressed using Apache Parquet. You map those files into relational, transactional tables using Apache Iceberg. You expose those tables to multiple engines securely using Apache Polaris. You execute queries with zero-copy, in-memory speed using Apache Arrow.

Finally, you layer the semantic context and Autonomous Reflections over that stack to create the Agentic Lakehouse.

You can build this stack yourself, or you can use a unified platform. Deploy agentic analytics directly on Apache Iceberg data with no pipelines and no added overhead. Try Dremio Cloud free for 30 days.

Assembling the Apache Lakehouse: The Modular Architecture

Alex Merced — Mon, 13 Apr 2026 22:36:47 +0000

Read the complete Open Source and the Lakehouse series:

For decades, the standard data architecture was monolithic. When you bought a data warehouse, you bought a single box where the vendor tightly coupled the storage format, the database rules, the metadata catalog, and the compute engine. If you wanted to query your data with a different tool, you had to physically extract the data from the warehouse and pay to store it somewhere else.

The modular Apache Lakehouse breaks that monolith apart. By using open standards for every defining layer of the data stack, you can decouple your storage from your compute entirely.

The Four Pillars of the Open Stack

The true power of the modern data lakehouse emerges when you assemble the four foundational open-source components into a single, cohesive architecture.

The Storage Layer (Apache Parquet): At the base, you have raw object storage (like Amazon S3 or Google Cloud Storage) filled with highly compressed, columnar Parquet files. This minimizes your storage footprint and guarantees rapid I/O for analytical queries.
The Table Format (Apache Iceberg): Because Parquet files are immutable, they cannot function natively as a database. Iceberg sits directly above the storage layer, mapping those files into relational tables. It provides the ACID transactions, schema evolution, and time travel necessary to keep data highly structured.
The Governance Layer (Apache Polaris): To prevent catalog fragmentation, Polaris acts as the central brain. It securely manages access to the Iceberg tables, using credential vending to ensure that different compute engines can hit the same data safely and transparently via a REST API.
The Execution Layer (Apache Arrow): When a BI dashboard or a query engine needs the data, it processes it in RAM using Apache Arrow. This in-memory columnar format ensures zero-copy reads, eliminating the massive CPU penalties of the legacy serialization tax.

This stack ensures complete vendor neutrality. Because every layer relies on an Apache Software Foundation standard, you own your data. You can swap compute engines tomorrow without migrating a single byte.

The Trap of the DIY Lakehouse

When engineering teams first understand this modular stack, the instinct is to build it manually. They stitch together open-source Spark clusters, deploy standalone Polaris containers, and point everything at their S3 buckets.

That Do-It-Yourself approach provides absolute control over the infrastructure, but it introduces a massive operational trap.

Apache Iceberg is incredibly powerful, but it is not self-maintaining. Every time you insert or update rows, Iceberg creates new snapshots, new manifest files, and tiny new Parquet files. If left unchecked, this bloat degrades query performance to a crawl. In a DIY build, your team must manually write, schedule, and monitor heavy Spark jobs to regularly compact small files, rewrite manifests, and vacuum expired snapshots. Your team becomes a database maintenance firm instead of a data analytics firm.

The Open Platform Approach

The enterprise alternative to a DIY build is a managed, open platform.

Choosing a managed platform does not violate the "no vendor lock-in" mandate—provided the platform honors the open architecture. Dremio, for example, natively integrates all four of these Apache projects out of the box.

When you deploy Dremio, you get a fully featured engine running Apache Arrow in its memory layer, querying Apache Iceberg tables stored in Apache Parquet formats, tracked by an internal Apache Polaris catalog.

Crucially, Dremio handles the operational burden. Features like Automatic Table Optimization quietly compact files and vacuum expired snapshots in the background, ensuring sub-second query performance without demanding custom maintenance scripts. Because the underlying data remains in open Iceberg REST formats, you are never locked into the execution engine.

To bypass the engineering headaches of a DIY build and start analyzing data on a production-ready Apache architecture on day one, try Dremio Cloud free for 30 days.

What is Apache Arrow? Erasing the Serialization Tax

Alex Merced — Mon, 13 Apr 2026 22:34:25 +0000

Read the complete Open Source and the Lakehouse series:

If you pull a million records from a database into a Python notebook, the query runs instantly, but the transfer feels endlessly slow. Your compute engine wastes the majority of that time quietly translating data layouts.

Historically, moving data between two analytical systems required paying a massive "serialization tax." Apache Arrow eliminates that tax by establishing a universal, open-source standard for how computer memory holds columnar data.

The Hidden Cost of Moving Data

When an analytical system queries legacy architectures via JDBC or ODBC, it encounters a severe bottleneck. The database holds data in its own proprietary layout. To send the data over a network, the database must serialize it—converting it into a generic row-based format like a JSON array or a proprietary buffer stream.

When the receiving system (like a pandas DataFrame or a Spark cluster) catches the stream, it must deserialize the rows. It reads the row, pulls out the individual strings and integers, and places them into its own internal columnar arrays for processing.

This cycle of formatting, converting, and parsing consumes up to 80% of the CPU time in data workflows. It slows down queries, burns compute credits, and bottlenecks machine learning pipelines.

The Standardized In-Memory Format

Apache Arrow changes the physics of data movement. While Apache Parquet defines how to store columnar data on a slow hard drive, Arrow defines how to structure columnar data inside high-speed RAM.

Arrow provides a standardized, language-agnostic, in-memory columnar format. Whether your system uses Java, Python, C++, or Rust, it structures the data identically in memory. Because the format is columnar, it natively supports vectorization. Modern CPUs can use Single Instruction, Multiple Data (SIMD) hardware acceleration to process entire chunks of Arrow arrays in a single clock cycle.

Zero-Copy Sharing

Standardizing the memory layout unlocks Arrow's most powerful trait: Zero-Copy data sharing.

Imagine a Java-based query engine and a Python-based data science tool running on the same machine. In a pre-Arrow world, the Java tool translates its data to a middle format, hands it to Python, and Python copies it into a new memory space. It doubles the memory footprint and wastes time.

With Apache Arrow, both tools understand the exact same memory layout. The Java engine creates an Arrow buffer in RAM. When Python asks for the data, Java simply hands Python the memory address pointer. Python begins reading the data instantly. Zero serialization. Zero copying.

Taking Flight: Arrow over the Network

Arrow's speed is not restricted to single machines. The project introduced Arrow Flight, a high-performance Remote Procedure Call (RPC) protocol for transmitting large datasets across networks.

Instead of converting data to REST or row-based streams, Arrow Flight transports the native Arrow memory buffers directly over the wire. The receiving client gets the buffer and immediately begins executing analytics on it.

To finalize the death of the serialization tax, the Apache Arrow community created ADBC (Arrow Database Connectivity). ADBC replaces legacy JDBC and ODBC drivers with an API standard explicitly designed for columnar analytics. ADBC allows databases to deliver native Arrow streams directly to clients, bypassing row-conversion entirely.

Arrow on the Lakehouse

Apache Arrow is the execution memory moving through the central nervous system of the lakehouse.

By stacking Parquet for storage, Iceberg for tables, Polaris for metadata routing, and Arrow for memory processing, you create an open data architecture capable of outperforming expensive proprietary data warehouses.

Dremio co-created Apache Arrow. It uses Arrow natively as its internal execution engine to eliminate the serialization tax that slows down traditional platforms. Try Dremio Cloud free for 30 days to query your object storage with zero-copy analytics.

What is Apache Polaris? Unifying the Iceberg Ecosystem

Alex Merced — Mon, 13 Apr 2026 22:28:10 +0000

Read the complete Open Source and the Lakehouse series:

Treating thousands of Parquet files as a unified database table requires a brain. Apache Iceberg provides the metadata structure to do this, but the Iceberg specification alone does not spin up a server, manage security roles, or handle network requests. You need a catalog service to orchestrate those root metadata pointers.

Until recently, that catalog layer threatened to fragment the entire lakehouse vision. Vendors began building their own proprietary catalogs to track Iceberg tables, trapping users in the exact data silos Iceberg promised to eliminate. Apache Polaris solves that fracture.

The Catalog Fragmentation Problem

Apache Iceberg ensures you do not have to copy data from Amazon S3 to Azure or Google Cloud just to query it. But if the pointer deciding which file is the "current" version of a table lives inside a vendor-locked ecosystem, engine independence becomes a myth.

If your data ingestion pipeline uses Apache Flink writing to a proprietary catalog, your business intelligence tool querying via Apache Trino or Dremio cannot see those updates unless they share the exact same catalog protocol.

The industry realized that to maintain true decoupling of compute and storage, the catalog itself had to become an open standard. That standard materialized as the Iceberg REST Catalog API.

The Iceberg REST API Standard

Apache Polaris is a vendor-neutral, open-source backend implementation of the Iceberg REST Catalog specification.

Because Polaris strictly adheres to the REST spec, any compute engine that speaks Iceberg REST can connect to it. A Spark job can create a table, a Flink job can stream records into it, and a Dremio cluster can instantly query the results.

This architecture guarantees true interoperability. Polaris becomes the single source of truth for your lakehouse. It tracks the latest metadata pointers and ensures that concurrent read and write operations across different engines maintain transactional consistency.

Enterprise Security with Credential Vending

Centralizing metadata also centralizes security. If multiple disconnected engines access the same object storage bucket, managing cloud identity roles becomes a nightmare of overly broad permissions.

Polaris implements robust Role-Based Access Control (RBAC) to solve this. Administrators define access policies for individual catalogs, namespaces, and tables directly inside Polaris. When an analyst runs a query on an engine, they don't use their own cloud credentials.

Instead, Polaris utilizes Credential Vending. The engine asks Polaris for access to a table. Polaris verifies the user's RBAC privileges. If approved, Polaris vends a temporary, highly scoped security token back to the engine. The engine uses that temporary token to read the specific Parquet files from S3. This eliminates the risk of issuing permanent, root-level S3 access keys across dozens of compute clusters.

Guaranteed Vendor Neutrality Under the ASF

A catalog is the brain of the lakehouse. If a single vendor owns the code running that brain, they quietly control the lakehouse. They dictate the roadmap, licensing, and integration pace.

Donating Polaris to the Apache Software Foundation as an incubating project legally shielded its interoperability. Open governance guarantees that Polaris remains neutral territory. No single cloud provider or query engine vendor can monopolize the definition of your table metadata.

The Evolving Architecture

If Apache Parquet provides high-performance storage, Apache Iceberg acts as the relational file system, and Apache Polaris serves as the brain, resolving engine traffic and handling access control. Together, they form the foundation of a modern data architecture.

Dremio’s built-in Open Catalog is built natively on Apache Polaris. When you sign up, you get a production-ready, vendor-neutral Polaris catalog deployed instantly. Try Dremio Cloud free for 30 days to query your data without creating proprietary metadata silos.

What is Apache Iceberg? The Table Format Revolution

Alex Merced — Mon, 13 Apr 2026 22:21:13 +0000

Read the complete Open Source and the Lakehouse series:

If you drop ten thousand Parquet files into an S3 bucket, you have a data swamp. You do not have a database. To run SQL queries against those files safely, your engine needs to know exactly which files belong to which table, what the columns are, and which files to ignore. Historically, Apache Hive solved this by tracking directories. Apache Iceberg solves this by tracking files.

That shift from directory-listing to file-level metadata fundamentally changes how organizations scale analytics. Iceberg brings the reliability of a transactional database to cloud object storage.

The Directory Listing Bottleneck

Legacy data architectures treated cloud storage like a local hard drive. If an engine like Hive wanted to read a table, it asked the cloud provider to list all the files inside a specific directory.

Listing millions of files in Amazon S3 or Google Cloud Storage takes an incredibly long time. Worse, cloud providers aggressively throttle high-frequency listing requests. When concurrent writers update a heavily partitioned Hive table, metadata synchronization operations cause readers to see inconsistent, partial data. Scaling meant hitting a hard wall.

Iceberg architects recognized that the file system is the wrong place to store database state. They moved the state into a dedicated metadata tree.

The Iceberg Metadata Tree Architecture

When an engine queries an Iceberg table, it never asks S3 to list directories. File discovery becomes an instant, O(1) metadata lookup. The architecture works through a strict hierarchy of pointers.

The query begins at the Catalog, which holds a single pointer to the current metadata.json file. This ensures atomic commits; whichever engine successfully updates the catalog pointer wins the transaction. The metadata.json tracks the table schema and points to a Manifest List.

The Manifest List acts as a table of contents for a specific point in time (a snapshot). It points to multiple Manifest Files. Finally, these Manifest Files contain the explicit paths to the individual Parquet data files, along with statistics like minimum and maximum values for every column.

This strict tree structure means the engine knows exactly which Parquet files it needs to read before touching the raw data.

Schema and Partition Evolution

Data shapes change. In traditional data lakes, renaming a column or changing a partition strategy required a total table rewrite. Iceberg executes these changes in milliseconds as metadata operations.

Iceberg achieves Schema Evolution by assigning a unique ID to every column. It tracks schema changes against the ID, not the string name. If you delete a column named user_id and create a new column named user_id, Iceberg knows they are entirely different fields. You can add, drop, rename, and reorder columns with zero side effects on existing files.

Similarly, Iceberg features "hidden partitioning". Engineers do not have to create physically derived columns just to partition data (e.g., extracting the year from a timestamp). Iceberg tracks the partition logic entirely in metadata. If you decide to change a table from monthly partitioning to daily partitioning, old data remains partitioned by month, and new data partitions by day. The engine handles the difference transparently.

Time Travel and Atomic Snapshots

Because Iceberg uses a tree of files where data is never updated in place, every write operation creates a brand new, immutable snapshot of the table.

When you run an UPDATE statement, Iceberg writes a new Parquet file containing the updated records, creates a new Manifest pointing to the new data, and generates a new Manifest List. The previous snapshot remains completely intact.

This architecture unlocks Time Travel. Analysts can append FOR SYSTEM_TIME AS OF to their SQL queries to read previous table states. If a faulty pipeline writes bad data, you do not need to rebuild the table from backups. You simply roll back the catalog pointer to the previous, healthy snapshot. Time travel does not duplicate data; the metadata simply points back to the underlying files that were valid at that exact moment.

Scaling the Open Source Lakehouse

Apache Iceberg provides the structure necessary to treat raw Parquet files like high-performance relational tables. However, a table format alone is incomplete. You need a centralized catalog mechanism to manage the root pointers, enforce security access, and resolve interoperability between multiple query engines.

That requirement leads directly to Apache Polaris, the open catalog standard designed to unify the Iceberg ecosystem.

Dremio executes natively against Iceberg tables, managing the metadata optimization lifecycle automatically. To see Iceberg transactions and time travel in action without building infrastructure, try Dremio Cloud free for 30 days.

What is Apache Parquet? Columns, Encoding, and Performance

Alex Merced — Mon, 13 Apr 2026 22:16:34 +0000

Read the complete Open Source and the Lakehouse series:

If you ask a data analyst to calculate the average transaction amount for the month of July using a massive CSV file, the compute engine must read every single line of that file. It reads the customer name, the address, the item SKUs, and the timestamps, just to find the single column it actually needs. At the petabyte scale, this row-based reading pattern guarantees slow analytics and high compute bills.

In 2013, engineers at Twitter and Cloudera collaborated to solve this fundamental storage bottleneck. Inspired by Google's Dremel paper on querying nested data, they created Apache Parquet. Since becoming a top-level project at the Apache Software Foundation in 2015, Parquet has emerged as the baseline storage format for the modern data lakehouse.

The Columnar Architecture of Parquet

Unlike CSV or JSON files that store data row by row, Apache Parquet heavily reorganizes data horizontally to support parallel analytics.

When a query engine writes a Parquet file, it horizontally slices the table into "Row Groups" (typically between 128 MB and 1 GB in size). Within each row group, the data is physically stored column by column. A "Column Chunk" holds all the values for a single column within that row group. Finally, the column chunk is split into smaller "Pages," which serve as the base unit for compression.

This architecture immediately solves the CSV problem through "Column Pruning." If you run a SELECT statement targeting only the transaction amount, the query engine completely ignores the chunks containing addresses and names. It only reads the specific column chunks requested. This drastically reduces disk I/O, generating faster query responses and lowering costs.

Dictionary Encoding and Compression

Data analytics often involves reading repetitive categorizations. Consider a status column containing millions of rows that say either "Active", "Pending", or "Cancelled". Storing those full strings over and over wastes massive amounts of space.

Parquet handles low-cardinality repetitive data using Dictionary Encoding. Instead of writing "Cancelled" millions of times, Parquet creates a small dictionary in the file's metadata mapping "Active" to 0, "Pending" to 1, and "Cancelled" to 2. The actual data pages simply store a list of these tiny integers.

Beyond encoding, columnar storage inherently improves compression. Algorithms like Snappy, Zstd, and GZIP search for repeating patterns to compress data. A column of integers looks incredibly repetitive and compresses tightly. A row containing an integer, a string, a date, and a boolean does not. Storing homogeneous data together allows Parquet files to consume a fraction of the space of their dense CSV equivalents.

Predicate Pushdown and Row Group Skipping

Perhaps Parquet's greatest distinct advantage is that its files are entirely self-describing. When a system writes Parquet data, it also computes and stores statistical metadata in the file's footer.

The footer contains the minimum value, maximum value, and null counts for every column within every row group. When you issue a query with a filter—like WHERE transaction_amount > 1000—the query engine reads the footer first. This process is called Predicate Pushdown.

If the footer reveals that the highest transaction amount in Row Group 1 is 500, the engine simply skips reading Row Group 1 entirely. The engine only pulls data from row groups containing values that might satisfy the query. This optimization turns broad multi-gigabyte table scans into highly targeted micro-reads.

Parquet's Role in the Open Source Lakehouse

Apache Parquet provides the physical storage engine for the data lakehouse. It ensures that data remains highly compressed and brutally efficient to read.

However, pure Parquet files are immutable. You cannot natively issue an UPDATE or DELETE statement against a raw Parquet file to fix a typo. To treat these static, high-performance files like a living, mutating database, you need a table format running on top of them. That is the role of Apache Iceberg.

To experience query execution directly against Parquet data stored in your own object storage, try Dremio Cloud free for 30 days. Dremio's vectorized query engine reads Parquet data aggressively, allowing you to ask questions in plain English and receive instant analytical results.

Apache Software Foundation: History, Purpose, and Process

Alex Merced — Mon, 13 Apr 2026 22:13:16 +0000

Read the complete Open Source and the Lakehouse series:

If you build a modern data lakehouse, you inevitably stack Apache Iceberg, Apache Parquet, and Apache Arrow. These projects dictate how you store, query, and govern petabytes of data. But the code itself is only half the story. The legal and operational framework supporting that code dictates whether a project survives for decades or gets hijacked by a single vendor.

That framework is the Apache Software Foundation. The ASF provides the structural immunity that prevents any one company from controlling the open source stack. Understanding how the ASF operates helps you evaluate the longevity and neutrality of the tools powering your lakehouse.

The Origins of the Apache Software Foundation

The web runs on software. In 1995, an informal collective of eight developers began collaborating on patches for the NCSA HTTPd web server. They called themselves the "Apache Group." Their work eventually became the Apache HTTP Server, which powered the early internet expansion.

As the software gained massive corporate adoption, the group faced a structural problem. An informal collective cannot legally hold copyrights, accept corporate donations, or shield individual volunteer developers from lawsuits.

To solve this, the group incorporated the Apache Software Foundation in 1999 as a U.S. 501(c)(3) non-profit public charity. The foundation exists to provide software for the public good. It acts as an independent legal shield that takes taking legal and financial ownership so that developers can focus entirely on code. Today, the ASF stewards hundreds of projects spanning big data, artificial intelligence, and cloud infrastructure.

The Apache Way: Community Over Code

The ASF operates on a unique philosophy known as "The Apache Way." The core tenet is simple: a healthy community is more important than good code. A toxic but brilliant contributor poses a greater risk to a project's survival than a mediocre codebase.

Meritocracy drives the Apache Way. You cannot buy a seat on a project's decision-making board. Contributors must earn authority by submitting code, writing documentation, and helping others on the mailing lists.

Crucially, individuals participate in the ASF as individuals. They do not act as representatives of their employers. This strict firewall prevents corporations from buying influence. Projects make decisions openly on public mailing lists through consensus. If an action is not recorded on the mailing list, it did not happen.

The Apache Incubator Process

You cannot simply hand an existing codebase to the ASF and declare it an Apache project. Every incoming project must pass through the Apache Incubator.

When a project enters the incubator, it becomes a "podling." The incubator Project Management Committee assigns experienced Apache members as mentors to guide the podling.

During incubation, the project community must prove they can operate under The Apache Way. They must transition all intellectual property to the ASF, which involves relicensing the code under the permissive Apache License 2.0. They also must demonstrate that their contributor base is diverse and not dominated by a single company.

Once a podling proves its community is resilient, legally clear, and self-governing, it applies for graduation. The ASF board grants approval, elevating the project to a Top-Level Project (TLP). The project then operates autonomously under its own Project Management Committee.

Apache Software Foundation vs. Linux Foundation

The ASF and the Linux Foundation frequently appear alongside each other, but they operate under entirely different models. Both are vital to open source software, but they serve different purposes.

The ASF is a 501(c)(3) public charity focused on grassroots community incubation. The Linux Foundation is a 501(c)(6) trade organization that acts as a consortium for massive industry collaboration.

Feature	Apache Software Foundation (ASF)	Linux Foundation (LF)
Organizational Model	501(c)(3) charity	501(c)(6) trade organization
Members	Individuals	Corporations
Governance	Decentralized Project Management Committees	Centralized Technical Steering Committees
Financial Influence	Financial donors hold zero influence	Large corporate sponsors often hold structured governance seats

The Linux Foundation excels at gathering competing corporate giants to fund and stabilize core internet infrastructure like Kubernetes. Companies pay membership fees, and those fees often secure them seats on a governing board to help direct the project.

The ASF strictly prohibits pay-to-play governance. A company can donate millions of dollars to the ASF, but they receive exactly zero influence over any project's technical direction. Only individual code contributors earn votes.

Why ASF Governance Matters for the Lakehouse

When you design a data lakehouse, you commit to a storage and query architecture that will last five to ten years. If a single vendor controls your data format, they can change the licensing model, slow down innovation, or force you into expensive proprietary compute engines.

By building your stack on Apache Parquet for storage, Apache Iceberg for table formats, and Apache Arrow for memory processing, you mitigate that risk. Because these are Top-Level Projects at the ASF, no single company can hijack their roadmaps.

The ASF ensures that the standards remain genuinely open. Competing query engines can all integrate with Iceberg and Arrow under equal conditions. Your data stays in your storage, in an open format, accessible by any engine. No lock-in.

If your team is ready to run analytics on these open standards without manual tuning, start by querying your Iceberg tables centrally. Try Dremio Cloud free for 30 days to deploy agentic analytics directly on your data lakehouse with zero vendor lock-in.

AI Tools Race Heats Up: Week of April 3–9, 2026

Alex Merced — Thu, 09 Apr 2026 13:24:19 +0000

Microsoft shipped Agent Framework 1.0 this week with full MCP and A2A support, AMD posted record MLPerf Inference 6.0 results, and a JetBrains survey put hard numbers on how fast Claude Code is climbing the professional adoption charts. The agentic stack is snapping together fast.

AI Coding Tools: Microsoft Agent Framework 1.0 Ships

On April 7, Microsoft released Agent Framework 1.0, the production-ready unification of Semantic Kernel and AutoGen into a single open-source SDK. The release delivers stable APIs, a long-term support commitment, and enterprise-grade multi-agent orchestration out of the box. The headline capability is cross-runtime interoperability: Agent Framework 1.0 ships with full MCP support for tool discovery and invocation, plus A2A 1.0 support arriving imminently for cross-framework agent collaboration. A browser-based DevUI debugger lets teams visualize agent execution, message flows, and tool calls in real time.

The release lands the same week JetBrains published research from its January 2026 AI Pulse survey of more than 10,000 developers. The numbers tell a clear story: 90% of professional developers now use at least one AI tool at work regularly. GitHub Copilot leads work adoption, but Claude Code has risen to share second place alongside Copilot, each used by 18% of developers in professional settings. That is a significant jump from its position just two surveys ago. The JetBrains data also shows Claude Code scoring 80.8% on the SWE-bench Verified benchmark, which measures real bug fixes across actual GitHub repositories — the highest published score for complex debugging and large-codebase work.

Google also drove meaningful activity this week on the open-weights coding side. Gemma 4 launched April 2 under an Apache 2.0 license, built from the same research as Gemini 3. The 31B Dense variant ranks third on Arena AI's open model leaderboard. AMD confirmed day-zero support for all Gemma 4 models across its Instinct GPUs, Radeon GPUs, and Ryzen AI processors — covering everything from cloud data centers to AI PCs — through vLLM, SGLang, llama.cpp, Ollama, and LM Studio.

AI Processing: AMD Posts Record Inference Results

AMD published its MLPerf Inference 6.0 results this week, anchored by the Instinct MI355X GPU. Built on CDNA 4 architecture with a 3nm process, the MI355X carries 185 billion transistors, supports FP4 and FP6 data types, and pairs all of that with up to 288GB of HBM3E memory. The submission covered a range of generative AI workloads from single GPU to multi-node scale, and AMD's ecosystem of partners reproduced the results across four different Instinct GPU types, a first for an MLPerf submission that gives customers real confidence in the numbers.

The broader hardware shift is toward heterogeneous inference architectures. Intel and SambaNova announced a collaboration this week that combines GPUs for the prefill phase of inference, SambaNova Reconfigurable Dataflow Units for the decode phase, and Xeon CPUs for orchestration. The design challenge they are addressing is real: GPU resources are expensive and poorly suited to the decode phase of token generation, which has different memory-bandwidth and compute characteristics than prefill. By mapping each phase of inference to the hardware dataflow it is best suited for, the collaboration targets a meaningful reduction in cost-per-token at production scale. Intel frames this as an ecosystem-first strategy rather than a single-chip bet, and the modular architecture maps naturally to the way cloud providers assemble rack-scale AI systems.

AMD also released PACE (Platform Aware Compute Engine) on April 8, an optimization framework for LLM inference on 5th Generation EPYC CPUs. PACE targets throughput improvement and latency reduction by adapting inference execution to the specific NUMA topology and cache hierarchy of the CPU it is running on. For organizations running inference on CPU-only infrastructure — a common pattern for privacy-sensitive workloads and edge deployments — this is a practical tool for squeezing more tokens per second out of existing hardware.

Looking further ahead, NVIDIA's Vera Rubin platform is in full production and scheduled to reach cloud providers in the second half of 2026. AWS, Google Cloud, Microsoft, and OCI are confirmed as among the first to deploy Vera Rubin NVL72 rack-scale systems. The platform targets a 10x reduction in inference token cost and a 4x reduction in the number of GPUs needed to train MoE models, compared to the Blackwell generation. Microsoft's Fairwater AI superfactories are the flagship deployment, scaling to hundreds of thousands of Vera Rubin Superchips.

Standards & Protocols: MCP v2.1 and the Agent Stack Solidifies

The agentic protocol stack continued maturing this week. MCP has crossed 97 million monthly SDK downloads in Python and TypeScript combined, and has been adopted by every major AI provider — Anthropic, OpenAI, Google, Microsoft, and Amazon. The MCP v2.1 specification adds Server Cards, a standard for exposing structured server metadata via a .well-known URL, enabling registries and crawlers to discover server capabilities without connecting to them. Major host applications including Claude Desktop and Cursor have shipped full MCP v2.1 support. The Linux Foundation's Agentic AI Foundation (AAIF) — co-founded by OpenAI, Anthropic, Google, Microsoft, AWS, and Block in December 2025 — now serves as the permanent governance home for both MCP and A2A.

Microsoft's Agent Framework 1.0 is the most concrete evidence yet that the MCP-plus-A2A architecture is becoming the production-ready default for enterprise agentic systems. The framework treats MCP as the resource layer — connecting agents to tools, APIs, and data sources through standardized servers — and A2A as the networking layer, enabling agents built on different frameworks to delegate tasks and coordinate workflows. The Elastic team published a two-part implementation guide this week walking through how MCP and A2A complement each other in a practical newsroom multi-agent example, with Elasticsearch providing the data substrate. The pattern of MCP for tool access and A2A for agent coordination is becoming the standard vocabulary for describing agentic architecture, and the practical implementation resources are finally catching up to the conceptual work.

For data engineering teams, the intersection of these protocols with the lakehouse stack is the most interesting frontier. MCP servers for Dremio and Apache Iceberg catalogs let agents query and reason over live data without custom integration code. As A2A matures, the pattern of orchestrator agents delegating to specialist data agents — each with MCP-backed access to specific catalog namespaces or table subsets — becomes a plausible production architecture for agentic analytics workflows.

Resources to Go Further

The AI landscape changes fast. Here are tools and resources to help you keep pace.

Try Dremio Free — Experience agentic analytics and an Apache Iceberg-powered lakehouse. Start your free trial

Learn Agentic AI with Data — Dremio's agentic analytics features let your AI agents query and act on live data. Explore Dremio Agentic AI

Join the Community — Connect with data engineers and AI practitioners building on open standards. Join the Dremio Developer Community

Book: The 2026 Guide to AI-Assisted Development — Covers prompt engineering, agent workflows, MCP, evaluation, security, and career paths. Get it on Amazon

Book: Using AI Agents for Data Engineering and Data Analysis — A practical guide to Claude Code, Google Antigravity, OpenAI Codex, and more. Get it on Amazon

Apache Data Lakehouse Weekly: April 3–9, 2026

Alex Merced — Thu, 09 Apr 2026 13:09:40 +0000

The open lakehouse community gathered in San Francisco this week for the biggest Iceberg Summit yet, two full in-person days at the Marriott Marquis, while Arrow's release engineering hummed along, Polaris settled into its first full month as a top-level project, and Parquet's ALP encoding vote moved toward a close. The summit didn't just celebrate what the community has built; it provided the forum to hash out the debates that have defined the dev lists all spring.

Apache Iceberg

Iceberg Summit 2026, the third edition of the ASF-sanctioned gathering, ran April 8–9 at San Francisco's Marriott Marquis, growing to two full in-person days after last year's sold-out success drew nearly 500 attendees. The community warmed up at a Pre-Summit Meetup hosted by Bloomberg's Engineering Department on April 7, organized by Sung Yun, with lightning talks and networking before the main event. Speakers from Apple, Bloomberg, Pinterest, Wells Fargo, and contributors from across the vendor ecosystem took the stage, making this the most industry-spanning Iceberg event to date.

The V4 design direction was front and center. Ryan Blue and the core contributor group have spent months laying groundwork through the dev list, and the summit provided an in-person venue to align on what V4 will actually look like in practice. The metadata.json optionality thread — asking whether the root JSON file can be made optional when a catalog manages metadata state — drew contributions from Anton Okolnychyi, Yufei Gu, Shawn Chang, and Steven Wu, debating portability concerns and the implications for static table and Spark driver behavior. The one-file commits discussion that Russell Spitzer and Amogh Jahagirdar advanced across multiple proposals is similarly headed toward resolution, promising dramatic reductions in commit latency and metadata storage footprint.

The AI contribution guidelines debate, which pulled in Holden Karau, Kevin Liu, Steve Loughran, and Sung Yun on the dev list over the preceding weeks, was a natural candidate for in-person resolution at the summit. The community has been converging on disclosure requirements and code provenance standards for AI-generated contributions; with many of the same contributors in the same room, a working policy is likely to emerge from this week's discussions.

Péter Váry's efficient column updates proposal, targeting AI/ML workloads with wide tables that need to update embeddings and model scores without rewriting entire rows, was among the talks submitted to the summit program. The approach, which writes only the updated columns to separate files and stitches them at read time, addresses a real pain point for teams managing petabyte-scale feature stores on Iceberg. Watch for a formal proposal and POC benchmarks to land on the dev list in the days following the summit.

Apache Polaris

Polaris spent this week in its first full month of life as a graduated Apache top-level project after the February 18 graduation. Jean-Baptiste Onofré filed the project's first board report as a TLP at the March 26 ASF board meeting, documenting community health and strategic direction under Polaris's own PMC, a governance milestone that marks the project's full independence.

The Apache Ranger authorization RFC from Selvamohan Neethiraj continued drawing feedback this week. The design allows organizations already running Ranger alongside Hive, Spark, and Trino to manage Polaris security within a unified governance framework, eliminating the policy duplication and role explosion that arise when teams bolt separate authorization systems onto each engine. The plugin design is opt-in and backward compatible with Polaris's existing internal authorization layer, a thoughtful approach that should lower the adoption barrier for enterprises evaluating Polaris in regulated environments.

The 1.4.0 release, which will be Polaris's first as a graduated project, remains in active planning. Credential vending for Azure and Google Cloud Storage backends is the headline feature in the release cycle. The catalog federation design, allowing Polaris to serve as a front for multiple catalog backends in multi-cloud deployments, is also advancing, addressing the needs of large enterprises running Iceberg tables across AWS, Azure, and GCS simultaneously. With Polaris now holding its own dev list, JIRA, and PMC, expect the release velocity to accelerate now that the project is no longer navigating incubator overhead.

Apache Arrow

Arrow's engineering focus this week centered on release preparation and language-binding consistency. The arrow-rs 58.2.0 release was scheduled for April, following the 58.1.0 shipment in March, which arrived with no breaking API changes. The Rust implementation has become one of the most actively maintained parts of the Arrow ecosystem, with a release cadence that matches the project's growing adoption in data lakehouse query engines.

The JDK 17 minimum version discussion that Jean-Baptiste Onofré launched continued gaining traction. Setting JDK 17 as the floor for Arrow Java 20.0.0 would coordinate Arrow's modernization trajectory with Iceberg's own Java upgrade timeline, effectively raising the Java minimum across the entire lakehouse stack in a single coordinated move. Contributors including Micah Kornfield and Antoine Pitrou have been weighing in, and the decision is expected to crystallize before the 20.0.0 release cycle formally opens.

Nic Crane's thread on using LLMs to aid Arrow's project maintenance, framing AI tools as a resource for the maintainers themselves rather than just contributors, continued generating discussion. The Arrow community's angle is slightly different from Iceberg's: less about contribution disclosure policy and more about how a lean maintainer group can responsibly use AI to triage a growing issue backlog. Google Summer of Code 2026 student proposals also arrived this week, with interest concentrated in compute kernels and language bindings for Go and Swift.

Apache Parquet

Parquet's week was defined by two major technical milestones reaching final stages. The ALP (Adaptive Lossless floating-Point) encoding specification completed its review period, and the formal acceptance vote was expected to close this week. ALP delivers significantly better compression ratios for floating-point data by encoding the exponent and mantissa separately, a direct performance benefit for ML feature stores and scientific computing workloads where float-heavy columns dominate. The encoding has been the subject of months of careful review, and its acceptance marks one of the most meaningful additions to the Parquet specification in recent memory.

The Variant type that shipped in February continued to see integration discussion across engine teams. Spark, Trino, and Dremio contributors compared notes on their implementation experiences, working through edge cases in semi-structured data handling that the spec leaves partially open to interpretation. Getting these implementations to converge is critical: Parquet's value as a cross-engine format depends on consistent behavior across the ecosystem, and Variant is novel enough that divergence is a real risk.

The File logical type proposal, which would allow Parquet files to natively embed unstructured data like images, PDFs, and audio as columnar records, advanced through community discussion this week. Combined with Variant, the proposal signals a deliberate effort to evolve Parquet from a purely analytical format into a unified storage layer capable of managing the diverse data shapes that AI/ML pipelines produce alongside the structured features they consume.

Cross-Project Themes

The Iceberg Summit was, by design, where the open lakehouse community takes stock and sets direction. The threads that dominated all four dev lists in the months leading up to it, AI contribution policies, V4 metadata design, column-level updates for ML workloads, Polaris's enterprise integration roadmap, all converged in San Francisco this week. What happens on the dev lists in the next two to three weeks will reflect what was decided in person, and readers should expect a burst of formal proposals, updated design documents, and new voting threads as the summit's in-person alignment translates back into async collaboration.

The second theme running beneath all four projects is the expansion of format scope to meet AI workload demands. Parquet's ALP and Variant additions, Iceberg's efficient column updates for wide ML tables, Polaris's Ranger and federation work, and Arrow's modernization to JDK 17 are all responses to the same underlying pressure: the lakehouse stack is being asked to power AI/ML pipelines, not just analytical queries. The projects are evolving in coordination, and the pace of that evolution is accelerating.

Looking Ahead

Post-summit, watch the Iceberg dev list for formal proposals on V4 metadata optionality and single-file commits, along with a published AI contribution policy. The Parquet ALP vote result should arrive within days. Polaris 1.4.0 scope finalization and the Arrow 20.0.0 JDK decision are the other near-term milestones to track. If the summit follows the pattern of 2025's event, the community will also release session recordings on YouTube in the weeks that follow, an excellent resource for anyone who couldn't make it to San Francisco.

Resources & Further Learning

Get Started with Dremio

Try Dremio Free — Build your lakehouse on Iceberg with a free trial
Build a Lakehouse with Iceberg, Parquet, Polaris & Arrow — Learn how Dremio brings the open lakehouse stack together

Free Downloads

Apache Iceberg: The Definitive Guide — O'Reilly book, free download
Apache Polaris: The Definitive Guide — O'Reilly book, free download

Books by Alex Merced

AI Tools Race Heats Up: Week of March 16 – April 2, 2026

Alex Merced — Fri, 03 Apr 2026 02:45:40 +0000

The past two weeks brought a major hardware shakeup at Nvidia GTC, a pricing war across AI coding tools, and the first MCP Dev Summit. Here is what matters most for developers and data practitioners.

AI Coding Tools: The Three-Lane Market Takes Shape

The AI coding tool market split into three clear lanes during this period. Claude Code, Cursor, and GitHub Copilot now represent three distinct approaches: terminal-native agents, AI-native IDEs, and multi-editor extensions.

GitHub Copilot shipped agentic code review in March 2026. The feature gathers full project context before suggesting changes and can pass those suggestions directly to the coding agent for automatic fix PRs. Agent mode also reached general availability on both VS Code and JetBrains during this window. Copilot Pro at $10/month now includes 300 premium requests, multi-model support including Claude Opus 4.6, and the full coding agent.

Windsurf (formerly Codeium) overhauled its pricing on March 19, switching from credits to daily and weekly quotas. The change sparked debate among users. Heavy users now face daily limits even on the $20/month Pro plan. A new Max tier at $200/month targets developers who hit throttling mid-day.

Across the market, the $20/month price point has become the new standard. Cursor Pro, Windsurf Pro, Claude Code Pro, and v0 Premium all sit at $20/month. Power users now budget $60–$200/month. Developer surveys show 95% of developers use AI tools weekly. The average experienced developer now uses 2.3 tools, often pairing Cursor for daily editing with Claude Code for complex tasks.

AI Processing: Nvidia GTC Rewrites the Hardware Playbook

Nvidia GTC 2026 ran March 16–19 in San Jose and delivered the biggest hardware announcements of the quarter. The headline: Nvidia is no longer just a GPU company.

The Groq 3 chip is the first product from Nvidia's $20 billion Groq acquisition last December. The chip focuses on AI inference — running trained models rather than training them. This marks a departure from Nvidia's long-standing philosophy that one class of GPU can handle every AI workload. CEO Jensen Huang described inference demand as going "exponential" on the company's recent earnings call.

Nvidia also unveiled standalone Vera CPU racks aimed directly at Intel and AMD. The Vera CPU targets agentic AI workloads that require heavy data orchestration alongside GPU inference. The data center CPU market now faces what analysts call a "quiet supply crisis." CPU delivery lead times stretch to six months. Prices rose more than 10%. AMD's data center head Forrest Norrod called demand increases "unprecedented over the last six to nine months."

The Vera Rubin NVL72 rack-scale platform promises a 10x reduction in inference token cost and 4x fewer GPUs needed to train mixture-of-experts models compared to Blackwell. AWS, Google Cloud, Microsoft, and OCI will deploy Vera Rubin instances in the second half of 2026. AI labs including Anthropic, Meta, OpenAI, Mistral, and xAI plan to use the Rubin platform for next-generation model training.

Standards & Protocols: MCP Dev Summit and A2A Hit v1.0

The first MCP Dev Summit took place April 2–3 in New York City. The two-day event featured 95+ sessions from protocol maintainers, security researchers, and production deployers. MCP now has over 97 million monthly SDK downloads across Python and TypeScript and is adopted by every major AI provider.

The Agentic AI Foundation (AAIF) under the Linux Foundation now has 146 members. Platinum members include AWS, Anthropic, Block, Bloomberg, Cloudflare, Google, Microsoft, and OpenAI. Gold members include IBM, Salesforce, SAP, Snowflake, Docker, JetBrains, and Oracle. This broad backing means both MCP and A2A are governed by a neutral foundation rather than any single company.

A2A (Agent-to-Agent) reached v1.0 as its first production-ready release, adding gRPC transport, signed Agent Cards for cryptographic identity, and multi-tenancy support. SDKs now ship in Python, Go, JavaScript, Java, and .NET. The Technical Steering Committee includes representatives from Google, AWS, Microsoft, IBM, Cisco, Salesforce, SAP, and ServiceNow.

The MCP 2026 roadmap focuses on enterprise readiness: better authentication, observability, and horizontal scaling for HTTP transport. Working groups drive each area, with specs expected throughout the year. Google also introduced the Universal Commerce Protocol (UCP) for agent-to-business transactions, pairing with its Agent Payments Protocol for purchases within defined guardrails.

Resources to Go Further

The AI landscape changes fast. Here are tools and resources to help you keep pace.

Try Dremio Free — Experience agentic analytics and an Apache Iceberg-powered lakehouse. Start your free trial

Learn Agentic AI with Data — Dremio's agentic analytics features let your AI agents query and act on live data. Explore Dremio Agentic AI

Join the Community — Connect with data engineers and AI practitioners building on open standards. Join the Dremio Developer Community

Book: The 2026 Guide to AI-Assisted Development — Covers prompt engineering, agent workflows, MCP, evaluation, security, and career paths. Get it on Amazon

Book: Using AI Agents for Data Engineering and Data Analysis — A practical guide to Claude Code, Google Antigravity, OpenAI Codex, and more. Get it on Amazon