Forem: Prithvi S

How OpenSearch Plugins Really Work: Architecture & Extension Points

Prithvi S — Wed, 15 Apr 2026 00:32:11 +0000

How OpenSearch Plugins Really Work: Architecture & Extension Points

OpenSearch is powerful out of the box, but its true flexibility comes from plugins. Yet most developers treat plugins as black boxes: you install them, they work, and you move on. But what if you need to build one? Or understand why a plugin broke after an upgrade? Or design a system that integrates with OpenSearch's plugin ecosystem?

In this post, I'll walk you through how plugins actually work: compilation, packaging, installation, and the extension points that make customization possible. By the end, you'll understand the mechanics well enough to build your own.

The Plugin Lifecycle: From Source to Running Code

Step 1: Writing and Compiling a Plugin

A plugin is a Java project with dependencies on OpenSearch core. At minimum, you need:

dependencies {
    compileOnly "org.opensearch:opensearch:${opensearch_version}"
}

That compileOnly is critical: your plugin compiles against OpenSearch, but doesn't bundle it. The plugin will run inside the OpenSearch JVM, using the host's core libraries.

Your plugin entry point is a class that extends Plugin. For example:

public class MyCustomPlugin extends Plugin implements SearchPlugin {
    @Override
    public List<QuerySpec<?>> getQueries() {
        return Collections.singletonList(
            new QuerySpec<>(MyCustomQuery.NAME, MyCustomQuery::new, p -> MyCustomQuery.fromXContent(p))
        );
    }
}

This simple declaration tells OpenSearch: "I provide a custom query type called my_custom_query."

Step 2: Building the Plugin Artifact

When you run gradle build, you produce a .zip file containing:

my-plugin-1.0.0.zip
├── opensearch-plugin-descriptor.properties
├── lib/
│   ├── my-plugin-1.0.0.jar
│   └── my-dependencies.jar (if any third-party libs needed)
├── bin/ (optional: scripts)
└── config/ (optional: default settings)

The opensearch-plugin-descriptor.properties file is the plugin manifest:

name=my-custom-plugin
description=My custom query plugin
version=1.0.0
opensearch.version=2.13.0
java.version=11
classname=com.example.MyCustomPlugin

This manifest declares: which OpenSearch version the plugin targets, what Java version it needs, and crucially, the entry point class name.

Step 3: Installation via the opensearch-plugin Tool

You install via CLI:

./bin/opensearch-plugin install file:///path/to/my-plugin-1.0.0.zip

The tool does several things:

Verifies the manifest - reads opensearch-plugin-descriptor.properties
Version checks - ensures plugin targets the installed OpenSearch version
Extracts - unpacks to plugins/my-custom-plugin/
Loads classes - prepares the plugin for JVM loading
Restarts the node - required to load the plugin code

After restart, your plugin code is live.

Class Loader Isolation and Bootstrap

Here's where it gets interesting. Your plugin code runs in the same JVM as OpenSearch core. How does OpenSearch prevent your plugin from accidentally (or maliciously) breaking core?

Class Loader Isolation:

OpenSearch uses a custom PluginClassLoader for each plugin. This loader is a child of the core class loader, but has its own namespace:

Core classes (org.opensearch.*) resolve from the main class loader
Plugin classes resolve from the plugin's class loader first
If a class isn't found in the plugin loader, it falls back to core

This prevents version conflicts. If your plugin wants to use a specific version of a library, it can bundle it, and its class loader will find that version first without conflicting with core.

Bootstrap Contract:

When OpenSearch starts, it:

Discovers all plugins in plugins/ directory
Reads each plugin's descriptor
Creates a PluginClassLoader for each
Instantiates each plugin's entry point class via reflection
Calls lifecycle methods: onIndexModule(), onNodeStarted(), etc.

If a plugin fails to load, OpenSearch will refuse to start. This is intentional: it's safer to fail loudly than to silently omit a plugin that applications might depend on.

Extension Points: How Plugins Hook Into OpenSearch

A plugin doesn't have direct access to internal OpenSearch code. Instead, it implements well-defined extension point interfaces. OpenSearch discovers these implementations and calls them at the right moments.

SearchPlugin: Custom Query Types and Aggregations

The most common extension point for search-focused plugins:

public class MySearchPlugin extends Plugin implements SearchPlugin {
    @Override
    public List<QuerySpec<?>> getQueries() {
        // Register custom query types
        return Collections.singletonList(
            new QuerySpec<>(MyQuery.NAME, MyQuery::new, p -> MyQuery.fromXContent(p))
        );
    }

    @Override
    public List<AggregationSpec> getAggregations() {
        // Register custom aggregations
        return Collections.singletonList(
            new AggregationSpec(MyAggregation.NAME, MyAggregation::new, p -> MyAggregation.parse(p))
        );
    }

    @Override
    public List<ScoreFunctionSpec<?>> getScoreFunctions() {
        // Register custom scoring functions
        return Collections.singletonList(
            new ScoreFunctionSpec<>(MyScoreFunction.NAME, MyScoreFunction::new, p -> MyScoreFunction.parse(p))
        );
    }
}

Once registered, your custom query is available via the REST API:

GET /my-index/_search
{
  "query": {
    "my_custom_query": {
      "field": "title",
      "boost": 2.0
    }
  }
}

ActionPlugin: Custom REST and Transport Actions

For plugins that need custom REST endpoints or transport operations:

public class MyActionPlugin extends Plugin implements ActionPlugin {
    @Override
    public List<ActionHandler<?, ?>> getActions() {
        return Collections.singletonList(
            new ActionHandler<>(MyAction.INSTANCE, TransportMyAction.class)
        );
    }

    @Override
    public List<RestHandler> getRestHandlers(Settings settings, RestController restController, 
            ClusterSettings clusterSettings, IndexScopedSettings indexScopedSettings,
            SettingsFilter settingsFilter, List<NamedWriteableRegistry> namedWriteableRegistries,
            List<NamedXContentRegistry> namedXContentRegistries, Supplier<DiscoveryNodes> nodesInCluster,
            Supplier<ClusterState> clusterStateSupplier) {
        return Collections.singletonList(
            new RestMyHandler()
        );
    }
}

Now you can hit a custom endpoint:

POST /_plugin/my-action
{
  "param1": "value"
}

MapperPlugin: Custom Field Types

If you need a new field type (beyond standard text, keyword, numeric, etc.):

public class MyMapperPlugin extends Plugin implements MapperPlugin {
    @Override
    public Map<String, Mapper.TypeParser> getMappers() {
        return Collections.singletonMap(
            "my_custom_field",
            (name, node, parserContext) -> new MyCustomFieldMapper(name, parserContext)
        );
    }
}

Now you can use it in mappings:

PUT /my-index
{
  "mappings": {
    "properties": {
      "custom_field": {
        "type": "my_custom_field",
        "analyzer": "standard"
      }
    }
  }
}

EnginePlugin: Custom Lucene Behavior

For advanced use cases, you can hook into the Lucene engine itself:

public class MyEnginePlugin extends Plugin implements EnginePlugin {
    @Override
    public Optional<EngineFactory> getEngineFactory(IndexSettings indexSettings) {
        return Optional.of(config -> new MyCustomEngine(config));
    }
}

IngestPlugin: Custom Processors

For plugins that process documents during ingestion:

public class MyIngestPlugin extends Plugin implements IngestPlugin {
    @Override
    public Map<String, Processor.Factory> getProcessors(Processor.Parameters parameters) {
        return Collections.singletonMap(
            "my_processor",
            (factories, tag, config) -> new MyIngestProcessor(tag, config)
        );
    }
}

Use in pipeline:

PUT /_ingest/pipeline/my_pipeline
{
  "processors": [
    {
      "my_processor": {
        "field": "content"
      }
    }
  ]
}

Real-World Example: The Search Relevance Plugin

OpenSearch's own search-relevance plugin demonstrates these concepts in action. It provides:

Custom query types for A/B testing search relevance
Custom aggregations for metrics collection
REST endpoints to manage experiments
System indexes (prefixed with .plugins-search-rel-) to store experiment state
Concurrent search request deciders (OpenSearch 2.17+) for custom query execution strategies

The plugin is battle-tested in production, used by teams optimizing ranking and relevance across massive datasets.

System Indexes: How Plugins Store Their Own State

Most non-trivial plugins need to persist data. Rather than requiring external storage, they use system indexes within OpenSearch itself.

System indexes are prefixed with .plugins- or .opendistro-:

.plugins-search-rel-<version>-experiments
.plugins-search-rel-<version>-notes
.plugins-ml-config
.opendistro-job-scheduler-lock

The challenge: how do you evolve the schema without breaking existing deployments?

OpenSearch plugins use a schema versioning pattern:

public static final String SCHEMA_VERSION = "1";

private void ensureIndexInitialized() {
    if (!indexExists()) {
        createIndex();
        return;
    }

    Map<String, Object> indexMeta = getIndexMeta();
    String currentVersion = (String) indexMeta.getOrDefault("schema_version", "0");

    if (!currentVersion.equals(SCHEMA_VERSION)) {
        migrateSchema(currentVersion, SCHEMA_VERSION);
    }
}

private void migrateSchema(String fromVersion, String toVersion) {
    // Use Put Mapping API to add new fields (additive only)
    // Never remove or change existing field types
    putMapping(newFields);
}

This ensures:

Old documents coexist with new schema
Upgrades are backwards compatible
No downtime required for schema evolution

Performance and Reliability Considerations

Startup Time

Each plugin adds to startup time. Large plugins or plugins that do heavy initialization can slow cluster startup. Monitor this in production.

Class Loader Memory

Each plugin gets its own class loader, holding copies of loaded classes in memory. Many plugins = higher memory footprint. Keep plugin count reasonable.

API Stability

OpenSearch's plugin APIs are versioned with OpenSearch itself. When OpenSearch releases a major version, plugins must recompile and test. This is by design: it ensures plugins stay compatible with core.

Security

Plugins run in the same JVM as OpenSearch core. A malicious or buggy plugin can crash the entire node. Only install plugins from trusted sources. In multi-tenant environments, consider network isolation or separate clusters.

Building Your Own Plugin: Where to Start

Clone the plugin template: OpenSearch provides plugin-template repository
Implement your extension point (SearchPlugin, ActionPlugin, etc.)
Write tests - use OpenSearch's testing framework
Build the .zip - gradle build produces the artifact
Install locally - ./bin/opensearch-plugin install file://...
Test end-to-end - verify your REST endpoint/query/aggregation works
Publish - host on artifact repository or GitHub Releases

Conclusion

OpenSearch plugins are not magic. They're well-structured Java code that hooks into OpenSearch via extension points. Understanding this architecture demystifies plugin behavior, helps you troubleshoot issues, and opens the door to building custom extensions.

Whether you're optimizing search relevance, integrating with custom systems, or building observability tooling, the plugin architecture gives you the hooks you need without compromising core stability.

The next time a plugin breaks after an upgrade, you'll know exactly where to look. And when you need to build one, you'll have a mental model of how the pieces fit together.

Want to explore further?

OpenSearch Plugin Developer Guide: https://opensearch.org/docs/latest/plugins/intro/
Plugin Template Repository: https://github.com/opensearch-project/plugin-template
Search Relevance Plugin (Prithvi's contribution): https://github.com/opensearch-project/dashboards-search-relevance

I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. Follow my work on GitHub: https://github.com/iprithv

Inverted Index Explained: How Elasticsearch Achieves Sub-Millisecond Search on Billions of Documents

Prithvi S — Tue, 14 Apr 2026 00:32:45 +0000

Imagine you're building a search feature for your product catalog. You have 10 million products, and you need to return relevant results in under 100 milliseconds. You decide to use PostgreSQL's full-text search, so you write:

SELECT * FROM products WHERE to_tsvector('english', title) @@ plainto_tsquery('english', 'wireless headphones');

It works. But then you get 100 million products. Then a billion. The queries crawl from 100ms to 5 seconds. Your users leave. Your boss asks why.

The answer isn't "use a bigger database." The answer is "use a different data structure."

Elasticsearch doesn't store data the way PostgreSQL does. It uses something called an inverted index, and that one difference is why Elasticsearch can search a billion documents in 2-5 milliseconds while traditional databases take seconds.

This post dives into how that magic works.

What Is an Inverted Index?

Think of a book. At the back, there's an index:

Elasticsearch ... pages 45, 78, 120, 156
Performance ... pages 45, 89, 203
Database ... pages 12, 78, 200

The index maps words to page numbers. When you want to find information about "Performance," you look it up once and jump directly to those pages. You don't read every single page.

That's the core idea of an inverted index.

Now imagine instead of a book, you have documents. Your "index" maps terms to document IDs:

elasticsearch -> [doc1, doc3, doc5, doc8, ...]
performance -> [doc1, doc2, doc7, ...]
database -> [doc3, doc4, doc9, ...]

It's "inverted" because it flips the relationship. A forward index says "doc1 contains terms: elasticsearch, performance, scalability." An inverted index says "term elasticsearch is in documents: 1, 3, 5, 8."

Why does this matter? Because searching becomes trivially fast.

When someone searches for "elasticsearch," Elasticsearch doesn't scan all documents. It looks up "elasticsearch" in the index once and gets back a list of document IDs. Done. O(1) lookup plus a single postings list traversal.

Under the Hood: How Elasticsearch Builds and Uses Inverted Indices

Step 1: Text Analysis (Before Indexing)

Before a document gets indexed, its text goes through an analyzer pipeline:

{
  "settings": {
    "analysis": {
      "analyzer": {
        "custom_analyzer": {
          "type": "custom",
          "char_filter": ["html_strip"],
          "tokenizer": "standard",
          "filter": ["lowercase", "stop", "porter_stem"]
        }
      }
    }
  }
}

This pipeline:

Removes HTML tags (character filter)
Splits text into tokens (tokenizer): "Elasticsearch is powerful" becomes ["Elasticsearch", "is", "powerful"]
Lowercases tokens (filter): ["elasticsearch", "is", "powerful"]
Removes stop words (filter): ["elasticsearch", "powerful"]
Stems words (filter): ["elasticsearch", "power"]

Now "powerful" and "powers" both map to the same root "power," so a search for "power" finds both.

The analyzer is completely customizable. For medical documents, you might preserve technical terms. For e-commerce, you might add synonym expansion (so "laptop" matches "notebook").

Step 2: Segment Creation (The Immutable Index)

Here's where Elasticsearch gets clever. Instead of maintaining one large mutable index, it creates immutable segments.

When documents arrive:

They sit in an in-memory buffer
Every ~1 second (refresh interval), the buffer flushes to disk as a new segment
Each segment is an inverted index, but immutable
Multiple segments are searched in parallel

Why immutable? Because it's fast. You never have to lock or rebalance. You just append new segments. If a crash happens mid-write, you have the translog to recover from.

Here's what a tiny two-document segment looks like:

INVERTED INDEX (Segment 1):

Term          | Postings List (Doc IDs)
--------------|----------------------
elasticsearch | [1, 2]
powerful      | [1]
scales        | [2]
horizontally  | [2]

DOCUMENT STORE:
Doc 1: "elasticsearch is powerful"
Doc 2: "elasticsearch scales horizontally"

Step 3: Term Lookup (Lightning Fast)

When you search for "elasticsearch," here's what happens:

The query arrives at a coordinator node
It broadcasts the query to all relevant shards
Each shard performs a binary search on the sorted terms in its segments
Found "elasticsearch"? Return the postings list: [1, 2]
Fetch those documents from the document store
Return to coordinator, which merges results from all shards

The magic: binary search on sorted terms is O(log N). On a million terms, that's ~20 comparisons. Then you get the postings list and you're done.

Step 4: Segment Merging (Background Optimization)

Over time, you accumulate many small segments. Searching 100 segments is slower than searching 1 large segment. So Elasticsearch periodically merges them:

Segment 1 (1000 docs) + Segment 2 (1000 docs) -> Merged Segment (2000 docs)

The merge is invisible to you. It happens in the background. Old segments are deleted. The new merged segment is searched going forward.

Why Inverted Index Beats Traditional Databases

Let's compare searching 1 billion documents for "elasticsearch":

PostgreSQL with Full-Text Search

SELECT id, title FROM products 
WHERE to_tsvector('english', title) @@ plainto_tsquery('english', 'elasticsearch')
LIMIT 10;

Internally, PostgreSQL uses a B-tree index. Here's the problem:

B-trees are designed for point lookups and range scans
Full-text search requires traversing multiple branches of the tree
For "elasticsearch," the database must:
- Find all occurrences of the term (multiple tree lookups)
- Reconstruct which documents contain them
- Filter by relevance

On a billion products, this takes 3-10 seconds.

Elasticsearch

GET /products/_search
{
  "query": {
    "match": {
      "title": "elasticsearch"
    }
  }
}

Elasticsearch:

Looks up "elasticsearch" in the inverted index (binary search, ~30 comparisons)
Gets back a postings list
Fetches the top 10 documents
Returns results

Time: 2-5 milliseconds on a properly sized cluster.

The difference: inverted indices are designed specifically for text search. B-trees are not.

The Trade-Off: Updates vs Queries

But inverted indices have a cost: updates are expensive.

When you update a document in Elasticsearch:

The old document is marked for deletion
A new document is indexed (goes through analysis, creates new segment)
A merge eventually removes the deleted document

This takes milliseconds to seconds, not microseconds. Elasticsearch is eventually consistent.

In PostgreSQL, you just UPDATE a row. Done immediately.

So when do you use each?

PostgreSQL: Transactional workloads, frequent updates, complex joins
Elasticsearch: Text search, logs, analytics, observability

Real-World Performance Numbers

Here are actual numbers from production Elasticsearch clusters:

Scenario	Elasticsearch	PostgreSQL
Search 1M docs	2-3ms	100-200ms
Search 1B docs	5-10ms	3-10s
Aggregation (cardinality)	5-20ms	500ms-2s
Index throughput	100K docs/sec	10K-50K docs/sec
Memory per 50GB data	8-16GB (compressed)	50GB+ (uncompressed)

The compression factor is huge: Elasticsearch's inverted index compresses 3-5x tighter than raw JSON because terms are deduplicated and encoded efficiently.

Relevance Scoring with BM25

Now that we can find documents fast, the next question is: which results should be first?

Elasticsearch uses BM25, a probabilistic relevance framework:

score = BM25(term_frequency, document_length, inverse_document_frequency)

In plain English:

Term frequency: how many times does "elasticsearch" appear in the document? (more = higher score)
Inverse document frequency: how rare is "elasticsearch"? (rare terms like "llama-index" rank higher than common terms like "the")
Document length normalization: prevent long documents from always ranking highest

So if you search "elasticsearch performance," a document mentioning "elasticsearch" 5 times and "performance" 3 times ranks higher than a document mentioning each once.

You can customize this with field boosting:

{
  "query": {
    "bool": {
      "must": [
        { "match": { "title": { "query": "elasticsearch", "boost": 2 } } },
        { "match": { "body": "elasticsearch" } }
      ]
    }
  }
}

Now matches in the title count twice as much. Perfect for building relevant search experiences.

Common Mistakes (And How to Avoid Them)

Mistake 1: Too Many Shards

You have 1GB of data and create 100 shards. Each shard has 10MB.

Problem: search latency goes through the roof because you're coordinating across 100 shards, and overhead dominates.

Rule of thumb: aim for 10-50GB per shard. If you have 1TB of data, 20-100 shards is reasonable.

Mistake 2: Ignoring Refresh Interval

You index a document and try to search it immediately. Nothing.

That's because the default refresh interval is 1 second. Your data sits in the buffer for up to 1 second before becoming searchable.

For near-real-time search, you might lower this to 100ms. But each refresh creates a new segment, and merging costs CPU.

Balance: 500ms-1s for most use cases. Only lower for critical real-time systems.

Mistake 3: Bad Analyzer Configuration

You don't configure an analyzer, so Elasticsearch uses the default standard analyzer.

Now when users search "AWS S3", they get no results because:

"AWS" tokenizes to "aws" (fine)
"S3" becomes "s" and "3" (terrible)

Custom analyzer with synonym expansion fixes this:

"filter": {
  "synonyms": {
    "type": "synonym",
    "synonyms": ["AWS S3 => s3", "machine learning => ml"]
  }
}

Conclusion: Why Inverted Index Matters

The inverted index is deceptively simple: a mapping from terms to document IDs. But this simple data structure enables Elasticsearch to do what traditional databases struggle with: search billions of documents in milliseconds.

The key insights:

Inverted index is designed for text search, not general-purpose queries
Immutable segments enable fast, lockless indexing
Binary search on terms makes lookup blazing fast
BM25 scoring automatically ranks results by relevance
The trade-off: fast reads, slower updates. Worth it for search workloads

If you're building a search feature, a logging system, or an observability platform, understanding how Elasticsearch works under the hood will save you from common mistakes and help you build systems that scale.

Next step? Learn how to scale Elasticsearch horizontally with sharding, tune refresh and flush intervals for your workload, and customize analyzers for your domain.

Happy searching.

Key Resources

I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. Follow my work on GitHub: https://github.com/iprithv

How Apache Iceberg's Metadata Architecture Enables ACID at Scale

Prithvi S — Mon, 13 Apr 2026 13:08:56 +0000

When you have a petabyte of data across millions of files in cloud storage, how do you ensure that reads are consistent, writes don't collide, and schema changes don't break everything? Traditional data lakes punt on this problem. Apache Iceberg solves it with an elegant metadata architecture that brings SQL-table reliability to distributed storage without needing a centralized database.

Let me walk you through how it works, why each layer matters, and what makes it fundamentally different from older table formats.

The Problem: Why Traditional Data Lakes Are Unreliable

Before Iceberg, data lakes operated like this:

Data engineers wrote Parquet files to S3
A Hive metastore tracked table schemas and partition locations
Queries discovered files by scanning directories or querying the metastore
Updates meant either rewriting entire partitions or leaving data inconsistent

This worked for append-only workflows. But the moment you needed:

Schema evolution (add a column) without rewriting data
Atomic deletes without breaking other writers
Time travel (query historical snapshots)
Concurrent writes without conflicts ...you hit a wall. The metastore was a single point of contention, and there was no reliable way to track which files belonged to which version of the table.

Iceberg fixes this by building a versioned metadata system directly into the table format. No metastore required (though one can help). Just immutable snapshots and a pointer to the current state.

The Metadata Hierarchy: Five Layers of Metadata

Iceberg organizes metadata in a clean bottom-to-top hierarchy. Each layer is immutable, and each layer is built from the layer below it. Here's how it works:

Layer 1: Data Files

At the bottom are your actual data files: Parquet, ORC, or Avro files stored in cloud storage (S3, GCS, Azure Blob, or local filesystems). These contain the raw table data, partitioned and compressed.

s3://my-bucket/warehouse/db/table/data/
  00001-abc123.parquet    <- 50MB, partition year=2024, month=01
  00002-def456.parquet    <- 60MB, partition year=2024, month=01
  00003-ghi789.parquet    <- 45MB, partition year=2024, month=02

From Iceberg's perspective, these files are opaque. It doesn't care about their internal structure. What matters is that each file has:

A file path
File format (parquet/orc/avro)
Partition values (year=2024, month=01)
Column-level statistics (min/max/null counts per column)
File size and record count

Layer 2: Manifest Files

Above data files sit manifest files. A manifest file is a Parquet file that lists the data files belonging to a snapshot, along with metadata about each one.

Think of a manifest like a file listing with extra info:

{
  "status": 1,
  "snapshot_id": 9223372036854775807,
  "sequence_number": 0,
  "file_path": "s3://my-bucket/warehouse/db/table/data/00001-abc123.parquet",
  "file_format": "PARQUET",
  "spec_id": 0,
  "partition": {
    "year": 2024,
    "month": 1
  },
  "record_count": 1234567,
  "file_size_in_bytes": 52428800,
  "column_sizes": {
    "1": 10485760,
    "2": 15728640,
    "3": 26214400
  },
  "value_counts": {
    "2": 1234567
  },
  "null_value_counts": {
    "2": 5
  },
  "lower_bounds": {
    "1": "2024-01-15",
    "2": 100
  },
  "upper_bounds": {
    "1": "2024-01-31",
    "2": 9999
  }
}

The manifest records:

Which data files are live vs deleted
Partition values for each file
Column-level statistics (min/max/null counts) for pruning
File sizes and record counts

This is where the magic starts. Because every data file's statistics are recorded in a manifest, query engines can:

Read a single manifest file (much smaller than scanning all data files)
Prune files based on partition values or column statistics
Skip reading files that don't match the query filter

On a table with a million data files, you might read one manifest file and determine that only 500 files match your query. No metadata service needed; it's all in the manifest.

Layer 3: Manifest List

A manifest list is a Parquet file that references all the manifest files for a given snapshot.

{
  "manifest_path": "s3://my-bucket/warehouse/db/table/metadata/10001-abc.avro",
  "manifest_length": 1048576,
  "partition_spec_id": 0,
  "added_snapshot_id": 9223372036854775807,
  "added_files_count": 150,
  "existing_files_count": 2500,
  "deleted_files_count": 10,
  "added_rows_count": 18750000,
  "existing_rows_count": 312500000,
  "deleted_rows_count": 125000,
  "partitions": [
    {
      "contains_null": false,
      "lower_bound": 2024,
      "upper_bound": 2024
    }
  ]
}

The manifest list aggregates statistics from all manifests for that snapshot:

How many files were added/existing/deleted?
How many rows were added/deleted?
What partition ranges does this snapshot contain?

Why? Because query engines need to know if a snapshot is even relevant before reading all its manifests. The manifest list answers that in one read.

Layer 4: Metadata File (JSON)

Above the manifest list sits the metadata file. This is a JSON file that contains:

Table schema and column definitions
Partition spec (how to partition the table)
Current snapshot ID (pointer to the "live" snapshot)
Snapshot history (all past snapshots)
Table properties and settings
Sorted order definitions
Schema evolution history (field IDs and renames)

Example metadata file structure:

{
  "format-version": 2,
  "table-uuid": "abc-123-def-456",
  "location": "s3://my-bucket/warehouse/db/table",
  "last-sequence-number": 3,
  "last-updated-ms": 1712973955000,
  "last-column-id": 3,
  "schema": {
    "type": "struct",
    "fields": [
      {"id": 1, "name": "event_id", "type": "string"},
      {"id": 2, "name": "user_id", "type": "long"},
      {"id": 3, "name": "event_timestamp", "type": "timestamp"}
    ]
  },
  "partition-spec": {
    "fields": [
      {"source-id": 3, "transform": "year", "name": "year"}
    ]
  },
  "current-snapshot-id": 9223372036854775807,
  "snapshots": [
    {
      "snapshot-id": 9223372036854775807,
      "timestamp-ms": 1712973955000,
      "summary": {
        "operation": "append",
        "spark.app.id": "app-20240412-123456"
      },
      "manifest-list": "s3://my-bucket/warehouse/db/table/metadata/v1.manifest.list"
    }
  ]
}

This single JSON file is the table's source of truth. It tells you:

What columns exist (with field IDs, not just names)
How the table is partitioned
Which snapshot is current
The full history of all snapshots ever created

Layer 5: The Catalog Pointer (The Only Mutable Piece)

Finally, at the very top sits the catalog. The catalog's job is simple: store a pointer to the current metadata file location.

table_identifier (db.table) -> s3://bucket/warehouse/db/table/metadata/v123.json

This is the only mutable piece in the entire system. When you commit a write, you:

Create a new metadata file (immutable)
Create new manifest files (immutable)
Create new data files (immutable)
Update the catalog pointer to point to the new metadata file (atomic CAS operation)

If the pointer update fails (because another writer already updated it), you retry with conflict detection. This gives you serializable isolation without a database.

Why This Matters: Three Critical Properties

This five-layer hierarchy gives you three things that traditional data lakes don't have:

1. Atomicity Without a Database

Traditional approach: Write files to S3, update the metastore database. If the database update fails, you have orphaned files. If the application crashes mid-write, the metastore is inconsistent.

Iceberg approach: Write everything (metadata files, manifests, data files) to immutable storage. The only atomic operation is the catalog pointer update (compare-and-swap on a key-value store, or a file rename on S3). If that fails, nothing changed. No orphaned files.

2. Schema Evolution Without Rewrites

With Hive tables, columns are identified by position or name. Want to add a column? Rewrite the schema. Want to rename a column? Rewrite all existing data files to update the column reference.

Iceberg uses field IDs instead of names or positions. Column 1 is always "event_id" internally, even if you rename the external column or reorder columns. Old data files don't change. New data files use the new schema. Queries automatically reconcile both.

Example: You have a table with schema (id, user_id, event_time). You want to add a source column:

Old data files don't have source (column ID 4)
New data files do have source
Iceberg handles missing columns transparently (NULL fill-down)
No rewrites

3. Time Travel and Snapshot Isolation

Every write creates a new immutable snapshot. Snapshots are never mutated or deleted (unless explicitly expired). This means:

You can query the table as it existed 30 days ago
Concurrent reads aren't blocked by concurrent writes
Snapshot expiration is manual (you decide when old snapshots are garbage)

Write Modes: Copy-on-Write vs Merge-on-Read

When you delete or update rows, Iceberg gives you two options:

Copy-on-Write (CoW)

When you update/delete rows in file X, rewrite the entire file
Pros: Readers always see clean data files, no performance penalty
Cons: Updates are expensive (rewrite entire files)
Best for: Read-heavy workloads

Merge-on-Read (MoR)

When you update/delete rows, write a separate delete file (position delete or equality delete)
Pros: Updates are fast (just write a small delete file)
Cons: Readers must merge data files + delete files, slight read penalty
Best for: Write-heavy workloads

Iceberg v2 spec introduced position deletes (deleted by row position) and equality deletes (deleted by column value). This gives engines flexibility in how they reconcile deletes.

Multi-Engine Interoperability

Here's what's remarkable: Spark, Trino, Flink, Presto, Hive, and Impala all read the same metadata format. The spec is engine-agnostic. A data engineer can:

Write data with Spark
Query it with Trino
Delete rows with Flink
Time travel with Presto ...all on the same table, without format conversions.

This is possible because Iceberg separates the format spec from the execution engine. The metadata hierarchy is standardized. Query engines just implement readers for that standard.

Optimistic Concurrency Control

With multiple writers, how does Iceberg prevent conflicts? Via optimistic concurrency:

Writer A reads the current metadata file
Writer B reads the current metadata file
Writer A finishes its changes, creates a new metadata file, tries to update the catalog pointer
Update succeeds (CAS operation)
Writer B finishes its changes, creates a new metadata file, tries to update the catalog pointer
Update fails (pointer no longer points to the metadata file Writer B read)
Writer B detects the conflict, re-reads the current metadata, recomputes its changes, and retries

This gives you serializable isolation. Writers don't block; they just retry on conflicts. For most workloads (few concurrent writers), conflicts are rare. For high-concurrency scenarios, you might need more sophisticated conflict resolution, but the default retry mechanism is sound.

Partition Evolution: Change Partitioning Without Rewriting

Suppose you initially partitioned by month(event_time), and now you want to partition by day(event_time) for better file pruning.

Traditional approach: Rewrite the entire table with the new partition scheme.

Iceberg approach:

Old snapshots keep their old partition scheme
New writes use the new partition scheme
Manifest files track partition spec ID per file
Queries automatically handle mixed partition layouts

No rewrites needed. This is huge for large tables where rewriting would take hours.

Conclusion: Why This Architecture Wins

Iceberg's metadata hierarchy achieves something remarkable: ACID guarantees on immutable cloud storage, without sacrificing performance or requiring a centralized database.

The design principles are:

Correctness over performance - atomic commits matter more than throughput
Immutability - makes caching, parallelism, and disaster recovery trivial
Versioning - every snapshot is preserved, enabling time travel and rollback
Engine-agnostic - the spec is open, allowing diverse tools to interoperate
File-level granularity - statistics are recorded per file, enabling efficient pruning

If you're building a data platform or migrating from Hive/Delta, understand this architecture. It's not just a file format; it's a rethinking of how to manage massive datasets reliably.

Want to dive deeper?

GitHub: https://github.com/apache/iceberg
Specification: https://iceberg.apache.org/spec/
Java API: org.apache.iceberg package in the repo

I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. Follow my work on GitHub: https://github.com/iprithv

Stop Guessing on Search Tuning: Using OpenSearch Search Relevance to Improve Results

Prithvi S — Mon, 13 Apr 2026 13:05:53 +0000

TL;DR: Search quality problems cost users and revenue. OpenSearch Search Relevance lets you measure exactly what's broken, iterate on fixes, and prove improvement with metrics. This guide walks you through a real-world search tuning workflow.

The Problem: Search That Works for You, Not Your Users

You've built a solid search system. Elasticsearch. OpenSearch. Full-text search running smoothly. But then you hear it:

"Why can't I find what I'm looking for?"
"Your search results are terrible."
"I have to refine my query five times to get what I need."

This is the gap between search that works and search that matters. Your system might be technically sound, but the relevance—the quality of what you return—is broken. And here's the problem: you can't measure what you can't improve.

Most teams guess. They tweak analyzers, adjust boost factors, shuffle query logic, deploy, and hope. Sometimes it helps. Sometimes it makes things worse. Nobody knows because they're not measuring.

This is where OpenSearch Search Relevance comes in.

What is OpenSearch Search Relevance?

OpenSearch Search Relevance is a plugin ecosystem that turns search tuning from guesswork into science. It does three key things:

Captures ground truth: You build a query set—representative questions your users actually ask
Runs experiments: You configure multiple search strategies and compare them side by side
Computes metrics: You get nDCG, precision, recall, and MRR scores—the same metrics information retrieval researchers use

The result: you know exactly what's working, what isn't, and by how much.

A Real-World Scenario: E-commerce Search Gone Wrong

Let's say you're an e-commerce platform. Your search is powered by OpenSearch. Basic setup: BM25 scoring, standard analyzer, some field boosting. Reasonable. But your metrics show a problem:

Users searching "comfortable running shoes" get back:

Hiking boots (nope)
Dress shoes on sale (nope)
A few running shoes at the bottom of page 2 (finally!)

Your nDCG score at position 10 is 0.42. That's bad. Users are leaving frustrated.

The question: what's broken? The analyzer? The query type? The field weights? Without measurement, you're shooting in the dark.

Step 1: Build Your Query Set

First, you collect representative queries. Not made-up queries—real questions from your search logs, support tickets, user research. For the e-commerce example:

"comfortable running shoes"
"women's waterproof winter boots"
"lightweight hiking shoes under $100"
"best cross-training footwear"
"slip-on shoes for office"

You grade the relevance of top results for each query. The grading is simple:

Grade 3: Perfect match (what you wanted to buy)
Grade 2: Good match (acceptable alternative)
Grade 1: Poor match (tangentially related)
Grade 0: Irrelevant (why is this here?)

This becomes your ground truth. This is what good search looks like for your domain.

Step 2: Set Up Your Baseline Search Configuration

You define how search works today. For our e-commerce example:

Query: {
  bool: {
    must: [
      { multi_match: {
          query: "<user_query>",
          fields: ["title^2", "description", "category"]
        }
      }
    ],
    filter: [
      { term: { "is_active": true } }
    ]
  }
}

Analyzer: "standard" (default tokenization, lowercase)

This is your baseline. We'll measure it, then try to beat it.

Step 3: Run an Experiment

Now you hypothesize: "The standard analyzer is losing words. If we use a synonym-aware analyzer and boost title matches more, we'll rank better."

You create a second search configuration:

Query: {
  bool: {
    must: [
      { multi_match: {
          query: "<user_query>",
          fields: ["title^3", "tags^2", "description"]
        }
      }
    ],
    filter: [
      { term: { "is_active": true } }
    ]
  }
}

Analyzer: "custom_with_synonyms"
  - Tokenizer: standard
  - Filters: lowercase, stop_words, synonym (running/jogging/athletic)

You run a PAIRWISE_COMPARISON experiment:

Take your query set
Execute each query with both configurations
Present results side by side to human evaluators (or use implicit signals like CTR)
Compute metrics for each

The output:

Baseline (standard analyzer):

nDCG@10: 0.42
Precision@10: 0.35
Recall: 0.48

Variant (synonym-aware, title boost):

nDCG@10: 0.68
Precision@10: 0.58
Recall: 0.71

That's a 62% improvement in nDCG. The variant wins decisively.

Step 4: Iterate

One win doesn't mean you're done. You run another experiment:

"What if we add a custom BM25 parameter tuning? Default is k1=1.2, b=0.75. We have short product titles—maybe b=0.5 would work better (less impact from field length)."

You create variant #3, measure it, and compare all three. Now you have data-driven evidence, not hunches.

Why This Matters

You stop guessing. Every change is measured against your ground truth.

You build confidence. When nDCG goes from 0.42 to 0.68, you're not wondering if you broke something—you know you improved it.

You compound gains. Each iteration is small (sometimes), but over months, small improvements stack. A series of 10% wins is a 2.6x total improvement.

You communicate value. When leadership asks "Did the search redesign help?", you show metrics, not opinions.

You catch regressions. New feature breaks relevance? Your metrics catch it before users do.

Technical Implementation: Getting Started

Prerequisites

OpenSearch cluster (7.10+) with search-relevance plugin installed
OpenSearch Dashboards with dashboards-search-relevance plugin

Workflow

Create a query set via the UI or API
- POST to search-relevance query set endpoint
- Provide queries + graded judgments
Define search configurations
- Store in index templates or as JSON documents
- Configurations are just OpenSearch queries + analyzer settings
Create an experiment
- Specify baseline vs variant(s)
- Set experiment type (PAIRWISE_COMPARISON, etc.)
- Run via API or UI
Evaluate results
- Dashboard shows nDCG, precision, recall, MRR
- Human evaluators refine judgments (optional)
- Export results for reporting

Example: Creating a Query Set via API

curl -X POST "localhost:9200/.search-relevance-queries/_doc" \
  -H 'Content-Type: application/json' \
  -d'{
    "query_set_name": "ecommerce_footwear_q2_2026",
    "queries": [
      {
        "query_text": "comfortable running shoes",
        "judgments": [
          {"doc_id": "shoe_123", "grade": 3},
          {"doc_id": "shoe_456", "grade": 3},
          {"doc_id": "boot_789", "grade": 0}
        ]
      }
    ]
  }'

Best Practices for Search Quality Evaluation

1. Represent your domain. Query sets should reflect real user behavior. If 40% of your queries are brand-specific ("Nike Air Max"), weight them accordingly.

2. Grade consistently. Define grade rubrics upfront. Grade 3 should mean the same thing across all evaluators. Consider inter-rater agreement checks (Kappa scores).

3. Start with quick wins. Don't boil the ocean. Fix analyzer issues, obvious field weight problems, missing synonyms. You'll see 20-30% gains fast.

4. Measure multiple metrics. nDCG is great, but also track precision (false positives matter) and recall (missed results matter). Together they tell the full story.

5. A/B test in production. Once confident in experiments, shadow your baseline for a week. Real user behavior > offline metrics.

6. Monitor over time. As your catalog changes, re-evaluate. New product types? Seasonal shifts? Your query set may need updates.

Common Pitfalls

Overfitting to your query set. If you tune only to 20 queries, you might break search for the other 980 query types you haven't measured.
Fix: Expand your query set regularly. Aim for 100+ representative queries.

Ignoring search latency. You improved nDCG by 10%, but query time went from 50ms to 500ms. Users see slower search as worse search.
Fix: Track latency alongside relevance metrics.

Forgetting about cold starts. Your new analyzer is great, but what about rare queries with few matches? Relevance breaks down at the edges.
Fix: Define fallback strategies. What happens when your perfect query gets zero results?

Not evaluating at scale. Your query set works, but only 30% of real user queries are covered. The other 70% are long-tail.
Fix: Use implicit signals (CTR, dwell time) to sample long-tail queries.

Next Steps

Start small. Pick 50 queries. Grade top-10 results. Measure baseline nDCG.
Hypothesize. What analyzer issue, field weight problem, or query logic gap would improve things?
Experiment. Create one variant. Run the experiment. Compare metrics.
Iterate. Keep the winner. Try the next improvement.
Scale. Once confident, expand your query set and refine your configuration.

Wrapping Up

Search quality problems are hidden until you measure them. OpenSearch Search Relevance gives you the tools to turn search tuning from guesswork into data-driven iteration.

Your users asked for better search. Now you can prove you delivered.

I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. Follow my work on GitHub: https://github.com/iprithv

Beyond Keywords: Unpacking Elasticsearch's Inverted Index for Sub-Millisecond Search

Prithvi S — Fri, 10 Apr 2026 07:06:20 +0000

Introduction: The Need for Speed in Search

In today's data-driven world, users expect instant access to information. Whether it's finding a product on an e-commerce site, searching through legal documents, or sifting through vast logs for an anomaly, every millisecond counts. Traditional relational databases, while excellent for structured queries and transactional integrity, often fall short when it comes to the complex, full-text search requirements of modern applications.

This is where Elasticsearch shines. At its core, Elasticsearch is a distributed, RESTful search and analytics engine capable of tackling billions of documents and returning results in sub-millisecond times. But what's the magic behind this speed? The answer lies in its foundational data structure: the Inverted Index.

In this deep dive, we'll peel back the layers of Elasticsearch to understand the inverted index-what it is, how it works, and why it's the cornerstone of Elasticsearch's remarkable performance. We'll also cover how it differs from traditional database indexing and why it's crucial for achieving lightning-fast, highly relevant search experiences.

The Problem with Traditional Database Search

Consider a typical relational database table, perhaps storing articles:

id	title	content
1	The Quick Brown Fox	The quick brown fox jumps over the lazy dog.
2	Lazy Dog's Adventures	The lazy dog enjoys chasing squirrels in the park.
3	Squirrels and Foxes	Foxes and squirrels often cross paths in the wild.

If you wanted to find articles containing the word "fox," you might write a SQL query like:

SELECT * FROM articles WHERE content LIKE '%fox%';

This query, while simple, is problematic for large datasets. The LIKE '%fox%' clause with a leading wildcard prevents the database from using a standard B-tree index efficiently. The database would likely resort to a full table scan, reading every row and checking the content column, which is incredibly slow for millions of records. Even with full-text search extensions built into some databases, they often struggle with the scale and feature set that dedicated search engines offer.

Enter the Inverted Index: A Paradigm Shift

Instead of mapping records to the words they contain (like a traditional index), an inverted index flips this relationship. It maps words to the documents (or articles, in our example) that contain them.

Let's take our example sentences and build a simple inverted index. First, we break down each document into individual terms, a process called tokenization, and normalize them (e.g., lowercase).

Document 1: "The quick brown fox jumps over the lazy dog."
Tokens: the, quick, brown, fox, jumps, over, lazy, dog

Document 2: "The lazy dog enjoys chasing squirrels in the park."
Tokens: the, lazy, dog, enjoys, chasing, squirrels, in, park

Document 3: "Foxes and squirrels often cross paths in the wild."
Tokens: foxes, and, squirrels, often, cross, paths, in, wild

Now, let's construct the inverted index:

Term	Document List (Posting List)
and	3
brown	1
chasing	2
cross	3
dog	1, 2
enjoys	2
fox	1
foxes	3
in	2, 3
jumps	1
lazy	1, 2
often	3
over	1
park	2
paths	3
quick	1
squirrels	2, 3
the	1, 2
wild	3

Notice a few things:

Each unique term is listed once.
Next to each term is a "posting list" – a list of document IDs where that term appears.
Terms like "fox" and "foxes" are treated separately here, but in a real-world Elasticsearch setup, an analyzer could stem them to a common root (fox) if desired.

How Elasticsearch Uses the Inverted Index for Speed

When you query Elasticsearch for "fox," it doesn't scan documents. Instead, it performs an O(1) lookup in the inverted index for the term "fox." This immediately returns Document 1. If you searched for "lazy dog," it would:

Look up "lazy" → [1, 2]
Look up "dog" → [1, 2]
Perform an intersection of these two posting lists: [1, 2] INTERSECT [1, 2] = [1, 2]
The result is Document 1 and Document 2.

This process is incredibly fast because it's operating on sorted lists of document IDs, making intersections highly efficient.

Beyond Simple Lookups: Position and Frequency

Real-world search is more complex than just knowing if a word exists in a document. We need to know:

How often a word appears (term frequency)
Where it appears (position)

To achieve this, the inverted index stores more than just document IDs in its posting lists. It also includes:

Term Frequency (TF): How many times a term appears in a document. This is crucial for relevance scoring (e.g., a document mentioning "Elasticsearch" ten times is likely more relevant than one mentioning it once).
Term Position: The positions within the document where the term occurs. This is vital for phrase queries ("quick brown fox") and proximity searches (words appearing near each other).

Let's enhance our inverted index fragment:

Term	Document (ID, TF, Positions)
fox	1: (TF=1, Positions: [3]), 3: (TF=1, Positions: [0])
lazy	1: (TF=1, Positions: [6]), 2: (TF=1, Positions: [1])
dog	1: (TF=1, Positions: [7]), 2: (TF=1, Positions: [2])
quick	1: (TF=1, Positions: [1])
brown	1: (TF=1, Positions: [2])

Now, if we search for the phrase "quick brown fox":

Elasticsearch finds documents containing "quick" at position X, "brown" at X+1, and "fox" at X+2.
It quickly identifies Document 1 as a match because "quick" is at 1, "brown" at 2, and "fox" at 3.

This additional metadata, while increasing the storage footprint of the index itself, drastically improves the speed and accuracy of complex full-text searches.

The Indexing Pipeline: From Raw Text to Inverted Index

How does raw text transform into this optimized inverted index? Elasticsearch uses a sophisticated indexing pipeline:

Document Arrival: A new document (e.g., a JSON object) is sent to an Elasticsearch coordinating node.
Shard Routing: The coordinating node determines which primary shard the document belongs to. This is typically done using a hash of the document's ID. For example, shard = hash(document_id) % num_primary_shards.
Write Buffer: The document is temporarily stored in an in-memory buffer on the data node hosting the target primary shard.
Analysis: The document's text fields undergo an "analysis" process. This is where text is transformed:
- Character Filters: Clean up text (e.g., remove HTML tags, convert special characters).
- Tokenizer: Breaks the text into individual terms (tokens). The "standard" tokenizer splits on whitespace and punctuation.
- Token Filters: Process tokens (e.g., lowercase them, remove stop words like "the", apply stemming to reduce words to their root form like "fishing" -> "fish", handle synonyms, etc.). This analyzed output is what forms the terms in the inverted index.
Segment Creation: Periodically (typically every 1 second by default), the terms in the in-memory write buffer are flushed to disk, creating a new "segment." A segment is a mini-inverted index, immutable once written. This is when the data becomes searchable. This process is called "refresh."
```
POST /my-index/_doc
{ "title": "Elasticsearch is fast" }
```
This single document hits the write buffer, gets analyzed (e.g., "elasticsearch", "fast"), and then after a refresh, these terms appear in a new segment.
Replication: For high availability and read scalability, Elasticsearch replicates the document to its replica shards.
Commit: To ensure durability, segments are periodically "flushed." This involves writing all in-memory segments to a new commit point on disk and creating a commit file that lists all known segments. The transaction log (translog) is also flushed, ensuring that even if the system crashes, no data is lost between flushes.
Merging: Over time, many small segments accumulate. Searching across numerous small segments can be less efficient. Elasticsearch intelligently merges these smaller segments into larger ones in the background. This process is resource-intensive but crucial for maintaining search performance.

Importance of Immutability

Each segment in Elasticsearch is immutable. Once written to disk, it cannot be changed. When a document is updated, Elasticsearch doesn't modify the existing segment. Instead, it marks the old document as deleted (logically, not physically) and indexes the new version of the document into a new segment. Deleted documents are eventually removed during segment merging.

This immutability offers significant advantages:

Concurrency: No need for complex locking mechanisms when reading segments, as they never change.
Caching: Segments can be aggressively cached by the operating system, improving read performance.
Reliability: Simpler to manage and recover from failures.

Relevance Scoring: Beyond Just Existence

The inverted index provides the raw material for finding documents, but how does Elasticsearch rank them? This is where relevance scoring comes in, and the inverted index's detailed term information is invaluable.

Elasticsearch uses a scoring algorithm called BM25 (Best Match 25) by default. While a full explanation of BM25 is beyond this post, it primarily considers:

Term Frequency (TF): How often a term appears in a document. More occurrences generally mean higher relevance.
Inverse Document Frequency (IDF): How rare a term is across all documents. Rarer terms are more significant.
Field Length: Documents where a term appears in a shorter field (e.g., title) are often considered more relevant than if it appears in a very long field.

The inverted index provides the Term Frequency and Term Position directly. The IDF can be calculated across all segments. By combining these factors, BM25 assigns a score to each matching document, allowing Elasticsearch to return the most relevant results first.

Conclusion: The Unsung Hero of Fast Search

The inverted index is more than just an internal detail; it's fundamental to understanding why Elasticsearch excels at search. It's a clever reversal of traditional indexing, transforming the challenge of full-text search into an efficient lookup and set intersection problem.

From its role in fast lookups and phrase queries to providing the necessary statistics for sophisticated relevance scoring, the inverted index empowers Elasticsearch to deliver the sub-millisecond search experiences that users now demand. By understanding this core component, you gain valuable insight into how to design, optimize, and troubleshoot your Elasticsearch deployments for peak performance and accuracy.

So next time you perform a lightning-fast search, take a moment to appreciate the unsung hero working tirelessly beneath the surface: the mighty inverted index.

About the author: I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. Follow my work on GitHub.

How Apache Polaris Vends Credentials: Securing Data Access Without Sharing Keys

Prithvi S — Thu, 09 Apr 2026 09:10:24 +0000

The modern data warehouse demands a fundamental shift in how we think about access control. When you build multi-tenant systems at scale, the traditional approach - distributing long-lived API keys or database credentials - becomes a security nightmare. Apache Polaris solves this elegantly: vend temporary, scoped credentials on demand, revoke instantly, audit everything.

The Problem: Why Long-Lived Credentials Don't Scale

At Netflix, Cloudera, or any major data platform, you're managing access across hundreds of users, services, and applications. If you hand out permanent API keys:

Revocation is impossible - a compromised key stays valid until you manually rotate it
Audit trails are fuzzy - you don't know which key accessed which data when
Compliance is painful - SOC2, HIPAA, PCI-DSS demand temporal, traceable access
Key rotation is a nightmare - updating thousands of clients, coordinating across teams
Scope is too broad - a key that works today still works tomorrow, even if access should have expired

This is why cloud providers moved away from permanent credentials. AWS uses temporary STS tokens. GCP uses short-lived access tokens. Azure has managed identities. The pattern is clear: trust should be ephemeral, scoped, and revocable.

Polaris applies this principle to data catalogs and table access.

How Polaris Vends Credentials

Polaris is an open-source, REST-first Iceberg catalog that implements the Iceberg REST API. Unlike traditional catalogs (which require direct database access or assume long-lived credentials), Polaris mints temporary credentials on every access.

Here's the flow:

1. Authorization Check (Who Are You? What Can You Do?)

When a client requests data access, Polaris first checks:

Is this principal (user/service) authenticated?
Do they have a role with permission to access this table?
Is the access read-only or read-write?

This uses Polaris's two-tier RBAC model:

Principal Roles - assigned to service principals (identities)
Catalog Roles - define actual permissions (TABLE_READ_DATA, TABLE_WRITE_DATA, etc.)

Example: A data analyst gets role analyst_prod, which is granted TABLE_READ_DATA on catalog.sales.transactions. A service account gets role etl_writer, which gets TABLE_WRITE_DATA on catalog.etl.staging.

2. Storage Configuration Lookup

Polaris queries its configuration:

Which cloud provider hosts this table? (AWS S3, GCS, Azure Blob)
What credentials should be used for minting? (Polaris's service role)
Are there any table-specific overrides?

3. Credential Minting

Here's where the magic happens. Polaris calls the cloud provider's API to mint temporary, scoped credentials:

For AWS (S3):

assume-role(
  role_arn=polaris-service-role,
  session_name=client-session-xyz,
  session_duration=15m,
  policy=restrict-to-s3://bucket/table-path/
)

Returns: temporary AWS credentials (access key + secret key) valid for 15 minutes, scoped to the specific table path.

For GCS:

create-service-account-key(
  service_account=polaris-sa@project.iam.gserviceaccount.com,
  lifetime=15m,
  custom-claims={ "resource": "gs://bucket/table-path" }
)

Returns: short-lived JWT valid for 15 minutes, scoped to the table path.

For Azure:

get-managed-identity-token(
  resource=https://storage.azure.com,
  lifetime=15m,
  scope=/subscriptions/xxx/resourceGroups/yyy/providers/Microsoft.Storage/storageAccounts/zzz/blobServices/default/containers/table-path
)

Returns: short-lived bearer token (OAuth2) valid for 15 minutes, scoped to the container path.

4. Scope Restriction

The credentials are scoped to:

Path - exact table location (e.g., s3://data/catalog/table/)
Operations - read-only (GET) vs read-write (GET, PUT, DELETE)
Duration - typically 15 minutes (configurable)

A client can't use these credentials to:

Access other tables
Write data to a read-only table
Perform actions after expiration

5. Return to Client

Polaris returns the temporary credentials to the client. The client's query engine (Spark, Trino, Presto, DuckDB, etc.) receives these credentials and uses them to read/write data directly to object storage.

No long-lived secrets are distributed. The client never sees Polaris's service credentials.

Why This Matters

Security Benefits

Instant Revocation - Delete a principal's role, all future requests are denied instantly
Fine-Grained Access - per-table, per-operation, per-principal permissions
Auditability - every credential mint event is logged (who, when, which table, read/write)
Compliance-Ready - temporal credentials, immutable audit trails, no shared secrets
Blast Radius - if a credential leaks, it's only valid for 15 minutes and only for one table

Operational Benefits

No Credential Rotation - credentials are automatically rotated every request
No Key Distribution - no need to distribute, store, or rotate permanent keys
Multi-Cloud Ready - same API works with S3, GCS, Azure, MinIO
Client Simplicity - clients just receive credentials and query - they don't manage them

Business Benefits

Compliance Aligned - meets SOC2, HIPAA, PCI-DSS, FedRAMP requirements
Cost Control - audit who accessed what, charge accordingly
Governance - enforce data mesh principles (teams own their data, Polaris mediates access)

Real-World Example: Data Mesh at Scale

Imagine you're running a data mesh with 50 teams, each owning their own datasets. Without Polaris:

Each team issues permanent API keys to consumers
Keys spread across configuration files, CI/CD pipelines, notebooks
A leaked key compromises an entire dataset
Revocation requires manual updates across dozens of systems
Audit trails are incomplete (keys used by multiple systems)

With Polaris:

Each consumer requests access via Polaris API (authenticated via OIDC, OAuth2, mTLS)
Polaris checks if consumer's identity has permission
Polaris mints a 15-minute credential scoped to the specific table and operation
Consumer queries the data with the temporary credential
On next request, the credential is already expired - a new one is minted
Revoke a consumer's role, and all future requests fail instantly
Audit logs show exactly which identity accessed which table at what time

Version 1.3.0 Features (January 2026)

Recent Polaris releases added:

Federated Credential Vending - Polaris can mint credentials for external catalogs (Snowflake, AWS Glue) instead of clients using their own credentials
OPA Integration - externalize authorization logic to Open Policy Agent for complex policies
Generic Tables - support Delta Lake and Hudi alongside Iceberg
Metrics Reporting - pluggable framework to report table metrics (row/byte counts, commits)

Getting Started

Polaris is on Apache Foundation:

GitHub: https://github.com/apache/polaris
Docs: https://polaris.apache.org/
REST API: Implements Iceberg REST Catalog spec

If you're building a data platform, data mesh, or multi-tenant system, Polaris's credential vending model is worth studying. It's a pattern that applies beyond Iceberg - any system managing shared resource access can benefit from temporal, scoped credentials.

About the author: I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. Follow my work on GitHub.