Forem: Erik Hatcher

Enriching the Search Experience

Erik Hatcher — Tue, 11 Nov 2025 13:56:51 +0000

Welcome to a series on enriching the search experience by using Views with MongoDB Search!

MongoDB lexical and vector indexes are built directly from the data in the associated collection. Every document is mapped through an index configuration into one, or more if using embeddedDocuments, Lucene documents. A mapping determines what fields are indexed and, primarily for string fields, how they are indexed. A mapping only can map what it sees: the fields on each available document.

There are situations where filtering what documents are indexed is necessary, perhaps when archived is true. Rather than indexing all documents and filtering them out at $search time, we can simply avoid indexing them altogether. In this case, our index size will only be based on non-archived documents.

And there's situations where enriching a document before it is indexed can enhance searches, such as indexing the size of an array rather than using a full collection scanning query-time expression, or transforming a boolean into a string to support faceting.

Index it like you want to search it—check out the recipes in this series to learn more.

First view: Indexing array size

Our first enrichment recipe fits when your application needs to query over the size of an array. Here's a couple of documents:

[  
  {  
    _id: 1,  
    values: [1,2,3,4]  
  },  
  {  
    _id: 2,  
    values: [1,2,3,4,5]  
  }  
]

We would like to find all documents that have exactly four values. We could use a brute force $match aggregation pipeline in this way:

[  
  {  
    $match: {  
      values: { $size: 4 }  
    }  
  }  
]

That aggregation returns the right results—however, at the expense of a COLLSCAN (collection scan) to evaluate that expression for every document in the collection. Scale matters—this will be fine to a point, but then it'll be way too slow.

Rather than compute repeatedly at query time visiting documents that do not match the criteria, it would be way more efficient to have the size of the array as a separate field and index that instead. Sure, we could use the Computed Pattern, though that would require our application to compute the size, store it alongside the array, and ensure it is kept in sync. There's an alternative: to compute the size during indexing.

Creating an enriched view

First, create a standard view on the docs collection, called docs_with_sizes:

db.createView(  
  "docs_with_sizes",  
  "docs",  
  [  
    {  
      $addFields: {  
        num_values: { $size: "$values" }  
      }  
    }  
  ]  
)

This is using JavaScript in mongosh, though you can create a view from other environments as well.

This view acts as a collection such that db.docs_with_sizes.find({}) returns:

[  
  { _id: 1, values: [1, 2, 3, 4], num_values: 4 },  
  { _id: 2, values: [1, 2, 3, 4, 5], num_values: 5 }  
]

The num_values field isn't stored in the database—it's computed on the fly.

Here's where the powerful part comes in—creating a search index on a view:

db.docs_with_sizes.createSearchIndex(  
  "view_index",  
  {  
    "mappings": {  
      "dynamic": true  
    }  
  }  
)

Our view_index has indexed the view, incorporating the added num_values field as an indexed, and thus queryable, value.

The final result

With this enriched index, we can straightforwardly query on num_values...

[  
  {  
    $search: {  
      index: "view_index",  
      equals: {  
        path: "num_values",  
        value: 4  
      }  
    }  
  }  
]

...yielding these results:

[ { _id: 1, values: [1, 2, 3, 4], num_values: 4 } ]

Conclusion

Index what you want to search. Using Views with MongoDB Search enhances and enables a number of interesting use cases that would otherwise be difficult or burdensome to tackle. Add these techniques to your search toolkit. Stay tuned for additional recipes in this series!

Playing With Search: Which Clause(s) Matched?

Erik Hatcher — Mon, 21 Jul 2025 15:33:34 +0000

This article is part of a series. Please see the complete series above.

Here's our playground of the post.

With this custom scoring trick, clauses that match a query can be determined with a little binary math.

Given these documents:

[
  {
    _id: 1,
    name: "Red"
  },
  {
    _id: 2,
    name: "Red Green"
  },
  {
    _id: 3,
    name: "Red Green Blue"
  }
]

Let's search for documents that match any of the terms "red", "green", or "blue" in a way we can determine which ones matched:

[
  {
    $search: {
      index: "default",
      compound: {
        should: [
          {
            text: {
              path: "name",
              query: "red",
              score: {
                constant: { value: 1 }
              }
            }
          },
          {
            text: {
              path: "name",
              query: "green",
              score: {
                constant: { value: 2 }
              }
            }
          },
                    {
            text: {
              path: "name",
              query: "blue",
              score: {
                constant: { value: 4 }
              }
            }
          }
        ]
      }
    }
  },
  {
    $addFields: {
      score: { $meta: "searchScore" }
    }
  }
]

This query overrides the score for each clause, setting the score for "red" matches to 1.0, matches of "green" are assigned the score 2.0, and matches for "blue" are scored 4.0. These values are the first values of the sequence of decimal values of binary digits. In binary, 1.0 is represented as 001, 2.0 is 010, and 4.0 is 100. Because each match has a unique digit position, we can reverse engineer the resulting relevancy searchScore to determine which clauses matched.

Let's analyze the search results:

[
  {
    "_id": 3,
    "name": "Red Green Blue",
    "score": 7
  },
  {
    "_id": 2,
    "name": "Red Green",
    "score": 3
  },
  {
    "_id": 1,
    "name": "Red",
    "score": 1
  }
]

A score of 7.0 is binary 111, meaning every clause matched. The score of 3.0 is binary 011, meaning that the first two ("red" and "green") clauses matched, but not the third ("blue").

See more about adjusting the scores.

Playing With Search: Query Term Highlighting

Erik Hatcher — Tue, 08 Jul 2025 18:05:03 +0000

This article is part of a series. Please see the complete series above.

Visit the query term highlighting Playground.

The collection contains this document:

[
  {
    "_id": 1,
    "name": "Introduction to Atlas Search"
  }
]

Enable highlighting and project the searchHighlights:

[
  {
    $search: {
      index: "default",
      text: {
        query: "atlas",
        path: "name"
      },
      highlight: {
        path: "name"
      }
    }
  },
  {
    $project: {
      highlights: {
        $meta: "searchHighlights"
      }
    }
  }
]

The query searched for the word "atlas", and is shown in context of the full field value in the response:

[
  {
    "_id": 1,
    "highlights": [
      {
        "score": 1.3466932773590088,
        "path": "name",
        "texts": [
          {
            "value": "Introduction to ",
            "type": "text"
          },
          {
            "value": "Atlas",
            "type": "hit"
          },
          {
            "value": " Search",
            "type": "text"
          }
        ]
      }
    ]
  }
]

The application code, when presenting the search results, can concatenate all the values, accentuating the hits.

Introduction to Atlas Search

Learn more search fundamentals!

Playing With Search: geoWithin

Erik Hatcher — Thu, 03 Jul 2025 11:30:06 +0000

This article is part of a series. Please see the complete series above.

Here's the geoWithin playground for this post.

Geographic points are annotated on documents (longitude first, then latitude):

{
  _id: 1,
  city: "Charlottesville",
  location: {
  type: "Point",
    coordinates: [-78.47668,38.02931]
  }
}

GeoJSON objects are not automatically indexed with "dynamic": true mappings and must be explicitly mapped, like this:

{
  "mappings": {
    "dynamic": true,
    "fields": {
      "location": {
        "type": "geo"
      }
    }
  }
}

Now, the geoWithin operator can be used to locate points within a shape—in this case, a 100m radius circle centered near the location of our one document:

{
  $search: {
    index: "default",
    "geoWithin": {
      "circle": {
        "center": {
          "type": "Point",
          "coordinates": [-78.476,38.029]
        },
        "radius": 100
      },
      "path": "location"
    }
  }
}

Press Run on the Playground (linked near the top of this post) and you'll see instead of an empty array [] that it returns the, one, matching document.

Learn more about search fundamentals.

Playing With Search: Stemming

Erik Hatcher — Tue, 01 Jul 2025 21:44:57 +0000

This article is part of a series. Please see the complete series above.

Let's start with this "text": {"path": "name", "query": "searches"} playground.

Playground "unit test" technique

Before we get into the textual analysis stemming topic, I want to share a different way of displaying search results in a playground. When playing with textual analysis, only a single document with a string field is needed. Rather than returning an empty array or an array containing only that one document from $search, I'd rather see literally 0 (false) or 1 (true) indicating whether the document matched the query or not.

The $searchMeta stage performs the query the same as $search, but instead of returning an array of matching documents, it returns a single document of search result metadata. In this case, only the total count is included. No actual collection documents are returned from $searchMeta.

[{
  "$searchMeta": {
    "text": {
      "path": "name",
      "query": "searches"
    },
    "count": {
      "type": "total"
    }
  }
}]

Stemming example

Will the query above match this document (given the default MongoDB Atlas Search configuration)?

  {
    "_id": 1,
    "name": "The Atlas Search Playground"
  }

No, it doesn't (count of 0):

[
  {
    "count": {
      "total": 0
    }
  }
]

The default mapping for string fields uses the lucene.standard analyzer. The standard analyzer is language-agnostic, but Unicode "word" aware, and tokenizes the name field into these lowercased terms: "the", "atlas", "search", "playground". The query of "searches" does not exactly match the indexed term "search", so the document does not match the query.

It would be better if this query would match that document, given the words "search" and "searches" are the same root word in English. Setting the analyzer to lucene.english brings in English-specific handling including stemming words to their "word stem". In this case, both "search" and "searches" (the content and the query) are analyzed to "search":

{
  "analyzer": "lucene.english",
  "mappings": {
    "dynamic": true
  }
}

And the document matches (count of 1):

[
  {
    "count": {
      "total": 1
    }
  }
]

Here's the successful lucene.english playground.

Head to the next article in this series.

Learn more about search fundamentals.

Introducing the "Playing With Search" Series

Erik Hatcher — Mon, 30 Jun 2025 17:50:23 +0000

The inherent underlying complexity in building software necessitates techniques to peel away layers so that we can focus on a specific capability. Search-based applications, in particular, demand a special kind of understanding and experience, with nuanced challenges such as fuzzy and sub-string matching, synonym handling, custom text tokenization, and lexical relevancy scoring computations. Let's ignore, for now, how and where things will be deployed. Let's not, yet, write application code. Let's peel away all of surrounding stuff above and below "search" and we'll just search and that's it.

And while we're peeling away the software code and infrastructure layers, we'll pare down the quantity and structure of the data to just the smallest we need to experiment with a configuration change, or query adjustment.

This series of concise, focused posts will demonstrate a wide variety of learning about, configuring, diagnosing, and tuning MongoDB Atlas Search.

Disclaimer: My job is to advocate for developers leveraging MongoDB, but search is search, so if you're not using MongoDB currently, that's cool. The data, configuration, and queries will look familiar to those who have experience with other search systems. The concepts from all search systems apply, especially anything built on Lucene (documents, fields, analyzers, tokenizers, etc).

Most of the examples in this series will be publicly accessible without requiring any sign-up, and easily editable by you once you click into them.

Our first Playground

Borrowing from the article I wrote last year introducing the Atlas Search Playground—Atlas Search Playground: Easy Experimentation—here's a case-insensitive search example using the default configuration and a simple lowercase query.

First a screenshot:

This ought to look the same for you clicking into the Playground and pressing Run.

When we need to add case-insensitive, language-agnostic, word-level search to an application, we only need one document with one effective field to figure out how to make it work. Here's the data:

[
  {
    "_id": 1,
    "name": "The Atlas Search Playground"
  }
]

A user should find this document when searching for the silly-cased query of pLaYgRoUnD. In MongoDB Atlas Search, that sort of simple "text" query is this:

{
  $search: {
    text: {
      query: "pLaYgRoUnD",
      path: "name"
    }
  }
}

Welcome to this new series! Stay tuned for more.

Learn more about search fundamentals.

When NOT to use Atlas Search

Erik Hatcher — Fri, 06 Dec 2024 22:07:19 +0000

Design reviews are one-on-one meetings where MongoDB experts deliver advice on data modeling best practices and application design challenges. In this series, we are going to explore common real-life scenarios where design reviews helped developers achieve meaningful success with MongoDB. - How to Align Your Data Model With Your Application Needs When Migrating From RDBMS to MongoDB | by Néstor Daza

Note: I’m taking the opportunity to link liberally, sometimes loony-ily. I love the serendipity of following interesting links. I had fun researching and reminding myself of oldies but goodies. Here’s to at least some of the shiny paths followed being entertaining and educational to you too.

We’ve got to start with a couple of assumptions for this article to best fit:

You’ve got documents in MongoDB Atlas.
The documents need to be findable.

If your documents aren’t in Atlas, then Atlas Search doesn’t (yet) apply, and thus the rest of this write-up is moot.

If anything, I’m pragmatic and agile. Duct tape, twine, or a card catalog—use what works for the job. Whether you use MongoDB or not, the document model is a good way to think about data challenges and worth having handy when the time is right. Do consider Atlas for your future data needs, as it’s a platform that provides a lot of necessary and powerful capabilities. Just sayin’.

Findability is one such necessary database capability. If you can’t find your content, it may as well not exist.

Atlas Search enables powerful, scalable, and relevant search features. Its strength primarily stems from one little, old Java library. There’s a potent elixir in that .jar! And it has been The Solution to All The Challenges for the bulk of my career. One rather fun aspect of my life at MongoDB is tackling Design Reviews that involve some aspect of search. I excel at, and enjoy, solving concrete search problems. These reviews typically are with folks using Atlas Search and wanting to dig in deeper to get a bit more nuance to relevancy tuning, or folks using MongoDB $match and $regex and exploring if and how to leverage Atlas Search instead. Here’s a story about a recent Design Review with a customer already well versed in Atlas Search and using it effectively… to a point.

Atlas Search matching as expected

Here’s the use case presented to me by a customer during a design review session:

We have a service built on MongoDB Atlas that needs to rapidly match identity requests using only a few fields of exact (though case-insensitive) values, such as an ID, e-mail address, and phone number.

Case-insensitive matching over a few fields? Definitely a problem that Atlas Search can solve handily! Take a few compound.should clauses and call us in the morning.

And unsurprisingly, the customer reports that:

Atlas Search matching works as expected...

But not so fast

Literally, and unfortunately,

… However, the time to “eventual consistency” in order to match recently updated documents is too long for the required SLA.

And, to work around that,

A [third-party key-value] caching mechanism was implemented for a first pass lookup.

Both of these topics warrant a bit of a deeper dive, so that we can understand how best to help this customer.

Eventual consistency
Key lookup using indexes

Eventual consistency

Yes, Atlas Search is awesome! It can slice, dice, and do all sorts of groovy things, yet it obediently stays within the laws of physics. Data, being what it is, always will be adding, updating, and deleting from the database and its replica set. The Atlas Search process (mongot) handles the database change stream and updates the underlying Lucene index. This process, by default, runs co-located with the database processes themselves on the same hardware, though ideally should run on its own hardware nearby.

Coupled Architecture

Dedicated Search Nodes

Atlas Search is eventually consistent. These machinations involve shards, replicas, CPU, disk, memory, network, and a bit of time. Changes to the database will, eventually, be reflected in associated search indexes. But it isn’t instantaneous, and there are many variables that affect the lag between a database change and search requests finding documents by the modified criteria: rate of data changes, complexity of index mapping configuration, deployment architecture/capabilities, resource contentions, size of the index, query load, and maybe even solar flares.

Depending on the nature of the application, the eventually consistent lag time may be irrelevant or a critical aspect of consideration. An update to a book record in a library can get reindexed overnight without affecting operations. However, this identity request for a record that just got updated failing to match the latest value in the database is unacceptable.

The trade-off of a search index being eventually consistent is to not delay, or interfere, with database-level updates and transactions. A search index update has so many variables involved and can change over time in complexity; a change to the index configuration could cause vastly more terms or documents to be indexed. An Atlas Search index is an index configuration and its corresponding Lucene index. This word “index” is a great one, but actually a Lucene index is really a collection of special purpose data structures, one for each field (and multi) defined. Each field “type” has its own optimized index data structure. Lexicographically ordered inverted indexes with posting lists complete with term, document, and corpus-level statistics power string mapped fields and queries. This is the heart of relevancy computations.

Key value lookup using indexes

Finding by _id (every document's unique key) is a given (so we throw that one in for free!). What about finding your data by other exact match types of criteria, such as all products in a specific category? Or all documents modified by a particular username? No doubt this type of findability is crucial too. MongoDB is really good at looking up documents by a value, provided the value is indexed.

This particular application needs exact, case-insensitive value lookup over a few fields. Let’s push the case-insensitivity issue to the application, and simply have it lowercase the field value any time it is being written or queried, so now it’s fully an exact match situation on the MongoDB side of things. Following indexing best practices such as the ESR rule, a few single-field B-Tree index definitions are all the customer needs to satisfy their performance SLA. These indexes don’t come for free, either, but are managed in the database process quickly and handled synchronously with every document update, so consistency is guaranteed.

And to be sure, key/value lookup in Lucene (via Atlas Search) is very fast. It’s the eventual consistency lag that drives the design recommendation here. If the use case had been querying across dozens of fields in any combination for exact values and eventual consistency was an acceptable trade-off, Atlas Search would be the better approach here. With a lot of fields to intersect, B-Tree index configuration would be arduous and resource-intensive, whereas an Atlas Search index configured for multiple-field intersection would be quite efficient and performant.

Design recommendation: B-Tree pragmatism

For this case of a few fields of exact value matching, with no full-text fuzzy search needed, the clear winner is leveraging the in-process, consistent, and quick B-Tree index capabilities.

When queries are exact field matches and the eventual consistency time lag is a critical blocker, consider using classic MongoDB B-Tree indexes rather than Atlas Search. Atlas Search indexes are updated in a separate process, maybe even on separate hardware via a network hop, whereas B-Tree index updates happen within the scope of database update transactions and are immediately usable after an update completes. Note that _id is implicitly indexed in this fashion and can be used for domain values if appropriate. With B-Tree-based lookups, a front-end cache is not needed as this is already a fast key/value lookup from a RAM-based index.

Be sure to learn about data modeling and schema design for Atlas Search so that you’re ready for the problems for which it shines!

Lucene and I

Erik Hatcher — Thu, 05 Dec 2024 19:55:43 +0000

There is one tool that has found its way into the toolbox (or let’s say “stack”) of, dare I said, a large part of the world's computer systems in one way or another. It’s a tool that has intrigued, educated, and (give thanks) employed me for, now, nearly 25 years. You know what I’m talking about…

Lucene and I

I’ve been a Lucene-ite since Doug Cutting’s creation migrated from Sourceforge to Jakarta.”I Love[d] Lucene” so much.that I volunteered, a large part of time over 14 months of my life (and shout out to Otis and Mike for sharing the arduous grinds) to co-author the first edition of Lucene in Action. The process of discovering a really cool, super powerful, and easy to use full text search library, realizing the word about it needed to get out widely, had me dig deep into the community and codebase with my Apache hat on, tinkering, contributing, committing, and generally loitering on the shoulders of folks way smarter than me.

I thank the bulk of my professional life to Lucene. I’d be kinda lost without it. A lot of us would.

It just be this easy with a little Java elbow grease. And because it’s fairly straightforward to send data into Lucene and then query it powerfully, and because Mr. Cutting nurtured such a benevolent, inviting yet demanding, open source environment, an entire ecosystem of add-ons, forks, ports, wrappers, and companies, and ... And ... AND!

When my homie Marcus “so-real-he-is” Eagan implored me to join him at MongoDB to work on Lucene integrated capabilities, called Atlas Search, I was intrigued and jumped at the opportunity to start a fresh challenge advocating for developers baking search into “document data”-based applications.

All of this said is to set the stage here - Lucene is the answer to All of The Problems! I’m an obeyer of the Law of the instrument, and when life gives me data, I make Lucene-ade.