Forem: MongoDB Guests

Async PyMongo in FastAPI

MongoDB Guests — Tue, 14 Apr 2026 14:53:07 +0000

This tutorial was written by Gourav Bhabesh.

1. Introduction

Why modern APIs need asynchronous I/O

Modern APIs need to be robust and be able to serve calls on the go, and minimize the instances of timeouts and server overloads while providing a good user experience during long-running jobs. Asynchronous I/O can solve one of the key issues making modern APIs: staying fast and reliable. It lets the user know the progress of their task while the API handles the requests in the background and is available for other requests. One of the key challenges in the API calls to the database is that some queries can be returned in a few milliseconds but others can take seconds (even with well indexed queries). Without asynchronous I/O in the API, this can lead to long request queues.

You can imagine this as a bakery storefront serving different goods at the same time to multiple customers. We want the orders to be filled asap without the queue becoming too big. In our example, we have multiple servers (scalable infrastructure) who take the orders which can take varying time to process and each customer with longer order times is given a status bar to know where their order’s current status. So a customer with an order that can be instantly filled (picking up a cookie) does not have to wait behind such a customer with a long processing order (baking a cheese cake).

FastAPI’s design philosophy around async

The building block of the FastAPI is ASGI (Asynchronous Server Gateway Interface) standard, unlike the older WSGI (Web Server Gateway Interface) which was synchronous in nature by default. In short, it uses Python’s async/await syntax, yielding control to the event loop, allowing servers to process other requests in the meantime, rather than blocking an entire thread. Using an event loop lets FastAPI handle thousands of concurrent requests with fewer resources. This leads to higher throughput per thread and lower latency. This automatically encourages the developer to use the async ecosystem to fully leverage the performance benefits of async programming.

Database bottlenecks & why async DB drivers matter

One of the key challenges that any database faces is the uncertainty in response times, in which the queries will be answered by the database server. Some queries can take milli-seconds and some can take seconds. If the architecture is not well built to handle this uncertainty, it can cause severe database bottlenecks. To handle this challenge, async database drivers ensure that longer requests do not become a bottleneck for other incoming requests. Initially, the async MongoDB driver was built in the library Motor. But in late 2023, Pymongo team officially decided to build the native async solution, with the goal to replace Motor as the long-term solution.

Introducing the new Async API in PyMongo

The Pymongo team has been working on the goal to replace the Motor library to give native async support in the pymongo library itself. This gives confidence on the client side to use the official library with async support. The async support was first provided in the version 4.9 in Pymongo and it has been further stabilised to the current version of 4.15.0. The new API capabilities bring the asyncio support directly to the official MongoDB driver, allowing developers to use awaitable database operations without external libraries like Motor. Combined with the FastAPI framework, which already supports await/async capability, Pymongo can now be natively used to build high-throughput, non-blocking services.

2. What is PyMongo?

Overview of the official MongoDB Python driver

MongoDB has provided Pymongo as the official Python driver to allow the applications to communicate with MongoDB clusters, providing production-tested API to perform CRUD operations, manage indexes, run aggregation pipelines and handle advanced features such as transactions and change streams. Pymongo was traditionally built on a synchronous I/O model, meaning every database call made will block the executing thread until the MongoDB server does not return a response leading to database bottlenecks.

Traditional synchronous PyMongo behavior

The key behaviours of the traditional Pymongo library, which we should be aware of, to appreciate the addition of the native support of asynchronous, are that synchronous operations block Python threads until the operation completes or a network error or timeout occurs.. This leads to another behaviour - sequential execution, blocking all requests until and unless the previous request is not complete. This made the MongoClient “Thread-Safe” allowing for multi-threading applications to be written, but each thread still performs blocking database operations.

In our example, of the baking shop, it would be like each server is concurrently serving the customers but each server is waiting for the order to complete rather than handing a token to wait, while serving the next customer request which might be served in much less time.

Blocking vs non-blocking I/O explained simply

Blocking I/O means the application must wait for the database operation to finish or network timeout before it will allow anything else to continue. On the other hand, non-blocking I/O allows requests to continue flowing to the database while previous requests are still being processed. In addition, the async drivers provide a way to receive the status of the request, giving power to the developer to handle long operations and create a better user-experience.

Limitations of synchronous DB operations in async frameworks

Using synchronous DB operations in the async framework is one of the key mistakes made by many clients of MongoDB, part of the reason was the non-availability of official Python drivers that actually supported asynchronous behavior. If the synchronous Pymongo driver is used inside an async framework such as FastAPI, these blocking calls freeze the event loop, preventing the server from processing other incoming requests and significantly reducing concurrency. As a result, applications that rely on synchronous database drivers inside async frameworks often experience increased latency, request queuing, and poor scalability under load, even if the rest of the code is written using async and await.

Imagine our baking shop upgraded its entire framework to be non-synchronous. The cooks can cook multiple dishes at the same time, but the servers are still the same, processing sequential orders and not taking new orders until the previous order has been delivered. The cook is having an easy day!!

3. What is Async PyMongo?

Overview of PyMongo’s new async API (`pymongo.async_`)

The official MongoDB team started working on the native-async support in the Pymongo driver in version4.9, where it was first released as beta version in late 2023. As of writing this article, the pymongo version 4.15 has been released and has been stable since version 4.13. With the pymongo.With async support, we can create an AsyncMongoClient object similar to the MongoClient object we created in the previous synchronous call. The Client provides fully awaitable database operations such as insert_one, find_one, update_one, and aggregation pipelines, allowing MongoDB I/O to integrate natively with Python’s event loop.

Key advantages over synchronous API

Compared to the traditional synchronous API, async PyMongo significantly improves scalability and responsiveness under concurrent workloads. Because database calls no longer block the event loop, asynchronous framework applications can continue handling new requests while waiting for MongoDB responses, resulting in lower latency, higher throughput, and better CPU utilization—especially in high-traffic microservices

Relationship to Motor (Motor’s role before async PyMongo existed)

Before async Pymongo, motor was the de facto library to be used when developers needed asynchronous behavior when communicating with the MongoDB clusters. Motor did this by exposing a coroutine-based API, allowing applications to switch context and perform other tasks while waiting for I/O operations. However, Motor was not a separate driver from scratch; it was designed as a wrapper around the core synchronous PyMongo library. It used Tornado’s or asyncio’s event loop, wrapping Pymongo’s operations asynchronously, effectively allowing the main event loop to remain unblocked. Motor was specifically created to support asynchronous web frameworks, initially Tornado and later adding support for asyncio. Performance comparison between motor and async pymongo can be found on the official documentation here.

How to migrate from motor to async-pymongo and sync-pymongo to async-pymongo

Migrating from the motor-based asynchronous library to the async-pymongo library is simple and is well documented by MongoDB here. In simple terms, one needs to change the driver call to the database using the AsyncMongoClient and add the keyword await in front of the database operations.

When migrating from sync Pymongo to async Pymongo, keep the following important points from the official MongoDB documentation in mind:
· To convert an AsyncCursor to a list, you must use the asynchronous cursor.to_list() method.
· The AsyncCollection.find() method in the PyMongo Async API is synchronous, but returns an AsyncCursor.
To iterate through the cursor, you must use an async for loop.
· The AsyncMongoClient object does not support the connect keyword argument.
· You cannot share AsyncMongoClient objects across threads or event loops.
· To access a property or method of a result returned by an asynchronous call, you must properly wrap the call in parentheses, as shown in the following example:

id = (await posts.insert_one(doc)).inserted_id

4. Key Features of Async PyMongo

True async interface with awaitable operations

The biggest advantage of the async Pymongo library for developers is its awaitable operations. Functions such as insert_one(), find_one(), update_one() and aggregate() return objects that can wait rather than blocking the results. This fundamentally decouples the event loop from the operation, meaning the database operation running on the database side yields the control back to the event loop, allowing FastAPI to process the other incoming requests without blocking the thread.

In our baking shop example, the server has been decoupled from the chef, while the previous order is getting cooked in the kitchen, the server takes in a new order without ceasing the operation and blocking the thread and allowing for incoming requests to be fulfilled !! This model of working is fundamentally different from the traditional synchronous PyMongo, where every database call ties up a worker thread until completion, blocking the thread from serving the next call when under load.

Connection pooling behavior

The connection pool behaviour is quite an interesting one here. On the surface it appears both the synchronous and the asynchronous Pymongo connection-poling implementation look the same (parameters available - maxPoolSize, minPoolSize, waitQueueTimeoutMS etc.) and indeed rely on the same MongoDB C driver. However, async applications are actually able to spawn thousands of concurrent operations and grab many connections from the pool at once, whereas, in the synchronous application each request occupies a python worker thread, holding a connection from the pool waiting for MongoDB. This naturally limits concurrency as the application is limited by the number of workers, effectively meaning the full pool size can be never used optimally.

Imagining in our world of baking shop, we have an army of chefs to process the orders, the async model allows the server to grab hold of any available chef to accept a new order and move on to the next one, whereas in the synchronous setup the server will wait for the chef to finish the previous order before moving to the new order.

Async cursors and non-blocking iteration

One of the primary challenges that any database faces is returning results to query with a large result set. The traditional PyMongo driver already returns a cursor object rather than loading all the documents into the memory at once. In async PyMongo, these cursors are fully asynchronous in nature as well, resulting in iteration without blocking the event loop. Developers now have the choice of using the await cursor.to_list() or iterating over the cursor object using the async for. This will allow data to be streamed in batches while the application is handling incoming requests. This feature is of significant importance when we have endpoints that return large datasets, background processing jobs and real-time analytics workloads.

5. What is FastAPI?

FastAPI’s async-first architecture

FastAPI is a high-performance web framework that is production-ready and designed for building APIs in python, with roots embedded in asynchronous programming. It embraces Python’s asyncio model as its core principle rather than treating it as optional. This design choice gives FastAPI leverage for I/O bound workloads such as db access, external API calls and real-time data streaming.

Other traditional web frameworks default to synchronous request handling, whereas FastAPI encourages using asynchronous endpoints (async def) wherever possible. When an endpoint is declared using the async keyword, FastAPI uses the event loop execution rather than a worker thread. Now as long as those requests involve the non-blocking I/O operations, a single worker process can handle many requests concurrently. This is where FastAPI’s async-first architecture shines. When the entire request path - including the database access - is non blocking, each request becomes super efficient. Any stack in the request that blocks the event loop will make the FastAPI’s concurrency model diminish significantly.

Event loop and concurrency model

At the heart of FastAPI’s concurrency model is Python’s asyncio event loop. The event loop manages the execution of many coroutines, switching between them whenever one is waiting for I/O, such as a network response from MongoDB. This cooperative multitasking enables a single process to efficiently handle thousands of simultaneous requests without spawning thousands of threads. However, this model only works as intended when all long-running operations are awaitable. If a synchronous operation blocks the event loop — for example, a traditional synchronous PyMongo database call — the entire server becomes unresponsive to other requests until that operation completes.

Why FastAPI pairs naturally with non-blocking database drivers

As explained above the core principle of FastAPI is to yield the control back to the event loop and any blocking I/O operation in the chain of the operations sent by the request will be like throwing a wrench on the gears. With the pairing of the non-blocking db driver such as the async pymongo, when a FastAPI endpoint awaits a MongoDB operation, control is yielded back to the event loop, allowing for the next set of requests to be processed in parallel. With the alignment between the db driver and the base principle of FastAPI to be async, we eliminate the friction and make the full request truly asynchronous in nature and hence enable FastAPI to scale efficiently under high concurrency.

Surprising performance differences when mixing sync DB code into async routes

A common pitfall for developers writing code in FastAPI is mixing the synchronous PyMongo calls inside the async def endpoints. On the surface it may appear to work, but it breaks the core principle of the FastAPI to be truly asynchronous. One single blocking db call and the entire event loop gets stalled as other requests will have to queue behind it - even when the server has many workers available.

In practice, this means that an API written with FastAPI but using synchronous PyMongo can perform worse under load than a fully synchronous application with multiple worker threads. By contrast, pairing FastAPI with Async PyMongo preserves the framework’s intended concurrency benefits, leading to better throughput, lower latency for typical requests, and more predictable scaling behavior. In large applications, having the capability to predict the scalability of infrastructure and have a control on the concurrency is of paramount importance.

6. Why Use Async PyMongo with FastAPI?

Avoid event-loop blocking

Using FastAPI with Asynchronous Pymongo correctly is essential if we want to avoid event-loop blocking. The requests from the async-first framework like FastAPI, share the same event-loop within each worker process. Meaning any operation in this request if becomes synchronized will lead to blocking of the entire event-loop, which high-concurrency systems want to avoid at all cost.

By using Async Pymongo, each db operation becomes awaitable rather than blocking in nature, ensuring a slow db operation does not freeze the entire application, allowing other requests to get processed.

Handle thousands of concurrent requests

FastAPI is capable of handling thousands of concurrent requests per worker process, but designing the entire chain of operations in an asynchronous manner is a must to achieve this. One of the primary mistakes that are made in the coding side is missing the async and await keywords in the db operation call, this breaks the core principle of FastAPI leading to slowness. With correct implementation we can make the infrastructure more cost-effective and efficient as compared to a purely synchronous system which would not be scalable in the long run as more threads are required to handle the huge volume of requests.

Reduce latency under load

With modern tech stacks, both horizontal and vertical scaling is a must and systems built with such capabilities can withstand the test of time. And today's system should be capable enough to handle varying loads. With FastAPI - async nature is built in-core principle of the library itself and when combined with Async Pymongo it allows for the potential blocking db operations to be handled asynchronously. This leads to lower median latency for requests, especially I/O intensive ones. In section 8, we will see how the sync/async jobs specifically prove this point, when we compare the median latency of thousands of concurrent requests.

Improve throughput for high-traffic microservices

In the world of microservices where responsiveness and scalability are critical Async Pymongo dramatically improves the performance by removing the blocker of such requests to the db layer. Combined with async frameworks such as FastAPI, Async Pymongo is superbly placed to serve real-time applications, analytics dashboards or user-facing services.

Better CPU utilization and scaling on limited infrastructure

One of the vital nerves of any IT infrastructure is cost-efficiency and with Devops team’s mantra - “doing more with less”, fits perfectly with the Async Pymongo. Making the most of the available resources, async architecture uses the same thread without blocking incoming requests by handing over the control back to the single event loop. Instead of maintaining hundreds or thousands of idle threads waiting on the I/O, FastAPI with Async Pymongo exploits this single event loop allowing it to switch between tasks as needed. In cloud based infrastructure, where compute resources are directly tied to cost, it can lead to meaningful savings without sacrificing performance.

7. Async CRUD Operations with Examples

This section demonstrates the common MongoDB operations asynchronously using the pymongo.async_ inside FastAPI.

All examples assume:

from pymongo.async_ import AsyncMongoClient 
from bson import ObjectId 
client = AsyncMongoClient (MONGO_URI) db = client["appdb"] 
users = db["users"]

Insert one / insert many (async versions)

Insert one document example:

@app.post("/users") 
async def create_user(user: dict):
  result = await users.insert_one (user) 
  return {"inserted_id": str(result.inserted_id)}

Insert many documents example:

@app.post("/users/bulk") 
async def create_many(users_in: list[dict]): 
  result = await users.insert_many (users_in) 
  return {"inserted_ids": [str(i) for i in result.inserted_ids]}

Find one / find many with async cursors

Find one document example:

@app.get("/users/{email}")
async def get_user(email: str): 
 return await users.find_one ({"email": email})

Find many documents example:

@app.get("/users") 
async def list_users():
  cursor = users.find ({}) 
  return await cursor.to_list(length=100)

Async pagination using cursors

@app.get("/users/page") 
async def paginated_users(last_id: str | None = None, limit: int = 20): 
  query = {"_id": {"$gt": ObjectId(last_id)}} if last_id else {} 
  cursor = users.find(query).sort("_id", 1).limit (limit) 
  results = await cursor.to_list(length=limit) 

  return { 
    "results": results,
    "next_cursor": str(results[-1]["_id"]) if results else None, 
  }

(Avoid skip() for large collections; use _id cursor pagination instead.)

Update and replace operations

Update one document example:

@app.patch("/users/{email}")
async def update_user(email: str, changes: dict):
  result = await users.update_one(
    {"email": email},
    {"$set": changes},
  ) 
  return {"matched": result.matched_count, "modified":
result.modified_count}

Replace one document example:

@app.put("/users/{email}")
async def replace_user(email: str, new_doc: dict):
  result = await users.replace_one({"email": email}, new_doc) 
  return {"matched": result.matched_count}

Delete one/many operations

Delete one document example:

@app.delete("/users/{email}") 
async def delete_user(email: str):
  result = await users.delete_one ({"email": email}) 
  return {"deleted": result.deleted_count}

Delete many documents example:

@app.delete("/users/inactive")
async def delete_inactive():
  result = await users.delete_many ({"active": False}) 
  return {"deleted": result.deleted_count}

Async aggregation pipelines

@app.get("/users/stats")
async def user_stats():
  pipeline = [
    {"$group": {"_id": "$country", ""count"": {"$sum": 1}}},
    {"$sort": {"count": -1}},
  ] 
cursor = await users.aggregate (pipeline)
return await cursor.to_list(length=100)

Error handling (duplicate keys, timeouts, network issues)

Duplicate keys:

from pymongo.errors import DuplicateKeyError
from fastapi import HTTPException

try: 
  await users.insert_one({"email": "a@b.com"})
except DuplicateKeyError:
  raise HTTPException(status_code=400, detail="User already exists")

Timeout/network issues:

from pymongo.errors import ExecutionTimeout, ConnectionFailure

try:
  await users.find_one({"email": "a@b.com"})
except ExecutionTimeout:
  raise HTTPException(status_code=504, detail="DB timeout")
except ConnectionFailure:
  raise HTTPException(status_code=503, detail="DB unreachable")

8. Performance Comparison: Sync PyMongo vs Async PyMongo

In this section, we break down the high-stakes battle between the traditional Synchronous (Sync) driver and the modern Asynchronous (Async) driver using the large-scale sample_airbnb dataset. This isn't just a syntax preference; it's the difference between a server that remains responsive under fire and one that collapses under its own weight.

Example endpoints and profiling results

To push these drivers to the limit, we implemented a Complex Aggregation task. Instead of a simple "Ping," we forced each request to filter high-rated Airbnb listings, sort them by price, and project complex nested fields.

Sync Logic: Utilizes list(users.aggregate(pipeline)), which occupies a physical thread for the duration of the database round-trip and data serialization.

Async Logic: Utilizes await (await users.aggregate(pipeline)).to_list().
The server yields control during the database "wait time," allowing the single-threaded event loop to process other incoming users.

Locust Benchmark setup

We subjected both implementations to a "Trial by Fire" using Locust:

User Load: Scaled up to 500 concurrent users.

Hardware Constraint: We intentionally limited the Sync thread pool to 1 thread to simulate extreme resource exhaustion and observe the "Queueing Effect."

Target: listingsAndReviews collection in MongoDB Atlas.

Understanding event-loop blocking

The results highlight a fundamental architectural divide. In the Sync model, when the thread pool is exhausted (in our case, limited to 1), every new user must wait for the person in front of them to finish their entire database operation. This creates a "Single Lane Bridge" bottleneck. In the Async model, even with thousands of users, the event loop acts as a high-speed scheduler. It sends the request to MongoDB and immediately moves to the next user, effectively "stacking" thousands of concurrent database operations without needing a thread for each one.

Graphs and metrics discussion (Locust latency plots)

Table: Sync stats showing the performance on one thread
Graph: Sync response times

Table: Async stats showing the performance

Graph: Async response times

Looking at our Locust metrics, the disparity is stark:
| Metric | Sync (1 Thread) | Async (Event Loop) |
|-------------------|-----------------|--------------------|
| Max RPS | ~48.8 | ~113.4 |
| Median Latency | ~8,700 ms | ~2,800 ms |
| 95th-Percentile | ~9,400 ms | ~4,100 ms |
| Throughput Driver | Thread-bound | DB / Network-bound |

Sync Graphs: Notice the staircase effect in response times.
As users increase, the latency climbs linearly because users are waiting in a queue. The RPS is "hard-capped" because the single thread can only process so many documents per second.

Async Graphs: The response times remain significantly lower and more stable.
Even as we ramped to 500 users, the 95th percentile hovered around 4 seconds compared to nearly 10 seconds for Sync. This proves that Async manages high-concurrency significantly better, even when the individual database query is complex.

Full working snippet for readers to run

Ready to see these results on your own machine?
We have provided a complete repository containing the FastAPI implementation, the AnyIO thread limiter configuration, and the Locust scripts used for these benchmarks.

Demo work - https://github.com/gouravmongodb/fastapi_pymongo_demo

9. Common Pitfalls & How to Avoid Them

Even with Async PyMongo, it is easy to introduce subtle performance or correctness issues if you are not careful. Below are the most common pitfalls developers encounter and how to avoid them.

Forgetting await on Database Calls

Because Async PyMongo methods are awaitable, forgetting await will not raise an immediate syntax error but will return a coroutine object instead of executing the operation. This can lead to silent failures or unexpected behavior in your API responses. Always double-check that every database operation inside an async def route is properly awaited.

Creating a New Client for Every Request

Instantiating AsyncMongoClient inside each request handler is a major anti-pattern. It creates unnecessary TCP connections, increases latency, and quickly exhausts resources under load. Instead, create a single shared client at application startup (using FastAPI lifespan or startup events) and reuse it across all requests.

Returning Non-JSON-Serializable Types

MongoDB types like ObjectId, datetime, or Decimal128 are not directly JSON serializable. Returning raw MongoDB documents from FastAPI will cause serialization errors. The recommended approach is to either convert these types to strings or map your documents to Pydantic models that handle serialization correctly.

Exhausting Connection Pools Under Heavy Load

Async applications can issue far more concurrent database operations than synchronous ones, which makes connection pool tuning critical. If maxPoolSize is too low, requests will start queuing inside the driver, increasing latency. Monitor Atlas metrics and adjust maxPoolSize, minPoolSize, and maxConnecting based on your workload.

Mixing Synchronous and Asynchronous Code

Using synchronous PyMongo calls inside async def routes (or vice versa) undermines FastAPI’s concurrency model and can block the event loop. Choose one model—preferably Async PyMongo for FastAPI—and stick to it consistently throughout your application.

10. Conclusion

Asynchronous database access is a cornerstone of modern, scalable API design. In an async-first framework like FastAPI, using a non-blocking database driver is not a luxury—it is a necessity for handling high concurrency, reducing latency, and making efficient use of resources.

With the stabilization of pymongo.async_ in PyMongo 4.13, Async PyMongo has become the recommended and officially supported way to use MongoDB with FastAPI. It provides a native, awaitable interface that aligns with FastAPI’s event loop, simplifies dependencies, and brings the future of the MongoDB Python ecosystem under a single driver.

Synchronous PyMongo still has a place in the ecosystem — particularly for simple scripts, batch jobs, or applications that are not built around async frameworks. However, for production web services, real-time applications, and microservices that must scale, Async PyMongo is the better choice.

Ultimately, performance depends not just on “sync vs async,” but on how you design your workload, tune your connection pool, index your collections, and scale your MongoDB cluster. When used thoughtfully, Async PyMongo with FastAPI offers a powerful, maintainable, and future-proof foundation for building high-performance Python APIs.

Research work

· https://pymongo.readthedocs.io/en/stable/changelog.html#changes-in-version-4-9-2024-09-18
· From Pymongo driver v4.9.0 introduced the beta version of async function.
· From Pymongo driver v4.13.0 - asynchronous API is now stable and no longer in beta.
· https://www.mongodb.com/docs/languages/python/pymongo-driver/current/reference/migration/#migrate-to-pymongo-async
· https://www.mongodb.com/docs/languages/python/pymongo-driver/current/reference/migration/#performance-benchmarks

Comparing Python ODMs for MongoDB

MongoDB Guests — Mon, 13 Apr 2026 15:45:57 +0000

This article was written by Fabio Cionini.

ORM and ODM: Relational vs. document

You might already be familiar with object-relational mapping (ORM) techniques that convert relational database data into objects suitable for use in a programming language. Libraries like Sequelize (for Node.js) or, sticking with Python, SQLAlchemy, allow us to easily interact with data structures that are very different from the object metaphor we use in modern software development. They save us from writing boilerplate code and help encapsulate business rules and data validation.

When working with a document-oriented, non-relational database like MongoDB — whose data structure more closely resembles the objects we manipulate with our code — these techniques and libraries are often called object-document mapping (ODM).

The MongoDB document model

The MongoDB data structure closely resembles the objects we usually manipulate with our code. It uses JSON (in practice, BSON under the hood), similar to JavaScript objects, as the preferred human-readable data interchange format.

Because we can embed objects and arrays as values in documents, it becomes easier to integrate the data layer with the application layer. MongoDB’s flexible schema also allows rapid prototyping and fast iteration without the need to manage migrations and database schema versions as strictly as in traditional relational databases.

From this perspective, you might think that ODMs are less necessary than ORMs. Although the official MongoDB drivers already “speak” the language of objects, ODMs can be very useful to:

Speed up development by generating boilerplate code for CRUD operations and object mapping.
Make data access code more abstract and portable.
Integrate data validation in a consistent way.

When coupled with a data validation library like Pydantic, ODMs can help guarantee data consistency by enforcing validation rules at the application level.

In this article, we’ll explore the main features offered by some of the most used and loved object-document mapping libraries for Python and see how they complement the native capabilities of the official MongoDB drivers.

Python ODMs

Beanie

Beanie is an asynchronous ODM whose data models are based on Pydantic, the most widely used data validation library for Python. It is lightweight, very intuitive, and has been around for several years, making it one of the most used and loved ODMs available.

You can use modern Python language features such as type hints, decorators, and of course asynchronous calls (using non-blocking access to MongoDB provided by Async PyMongo, which is built on top of Python’s asyncio library).

Installation is easily accomplished using PIP:

py
pip install beanie

or Poetry:

py
poetry add beanie

Beanie uses Async PyMongo as an async database engine. You provide an AsyncMongoClient instance and a list of your document models to the init_beanie(...) function.

In Beanie, each database collection is represented by a class that extends the Document base class. By extending that class, you define the schema and perform all the read and write operations on the corresponding collection.

You can also use pydantic.BaseModel directly by extending its class for embedded documents. For example:

py
import asyncio
from typing import Optional
from pydantic import BaseModel
from beanie import init_beanie, Document, Indexed
from pymongo import AsyncMongoClient

class Category(BaseModel):
name: str
description: str

class Product(Document):
name: str
description: Optional[str] = None
price: Indexed(float)
category: Category

async def main():
client = AsyncMongoClient("mongodb://root:root@localhost:27017/")
await init_beanie(database=client.shop, document_models=[Product])

coffee = Category(
    name="Coffee",
    description="A preparation of roasted and ground coffee seeds.",
)
espresso = Product(
    name="Italian Espresso",
    price=2.95,
    category=coffee,
)

await espresso.insert()
if name == "main":
asyncio.run(main())

This defines two models:

Category, based on Pydantic, used as an embedded document inside Product.
Product, the actual document that will be inserted into the corresponding MongoDB collection.

Notice that price is defined as an indexed field via Indexed(float): this will create an index on price in the underlying collection. The insert() method stores the Product instance in MongoDB.

The Beanie website provides a clear, easy-to-follow tutorial on defining documents along with comprehensive documentation for more advanced use cases.

MongoEngine

MongoEngine is a mature and widely adopted ODM. It has been around for many years and is considered one of the most stable options in the Python ecosystem.

MongoEngine follows a synchronous programming model and offers a very Django-like developer experience, with:

Explicit field definitions.
Built-in validation.
Rich query capabilities.

Similarly to Beanie, each collection is represented by a class that extends the Document base class. Fields are declared explicitly using typed field classes, and embedded documents are modeled by extending EmbeddedDocument.

Installation is done with the usual pip command: pip install mongoengine

A simple example looks like this:

py
from mongoengine import (
connect,
Document,
EmbeddedDocument,
StringField,
FloatField,
EmbeddedDocumentField,
)

class Category(EmbeddedDocument):
name = StringField(required=True)
description = StringField()

class Product(Document):
name = StringField(required=True)
description = StringField()
price = FloatField(required=True)
category = EmbeddedDocumentField(Category)

connect(db="shop", host="mongodb://localhost:27017/shop")

coffee = Category(
name="Coffee",
description="A preparation of roasted and ground coffee seeds.",
)
espresso = Product(
name="Italian Espresso",
price=2.95,
category=coffee,
)
espresso.save()

Calling save() persists the document to MongoDB, automatically handling inserts or updates as needed.

The synchronous nature of MongoEngine offers a concise syntax and fine-grained control over the sequence of operations. For workloads where you don’t need concurrency at scale, this can make application logic easier to reason about, even if asynchronous ODMs may use resources more efficiently under heavy I/O load.

MongoEngine’s online documentation provides both a comprehensive API reference and a getting started tutorial.

ODMantic

ODMantic is a modern asynchronous ODM built on top of Pydantic and Motor, designed to provide strong typing, data validation, and native asyncio support.

Its design philosophy is quite close to Beanie: both libraries offer “Pythonic” ways to interact with MongoDB. ODMantic, however, is somewhat less focused on data migration and more on ergonomic models and async workflows.

Here is a basic example:

py
import asyncio
from typing import Optional
from odmantic import Model, Field, EmbeddedModel, AIOEngine
from motor.motor_asyncio import AsyncIOMotorClient

class Category(EmbeddedModel):
name: str
description: str

class Product(Model):
name: str
description: Optional[str] = None
price: float = Field(index=True)
category: Category

async def main():
client = AsyncIOMotorClient("mongodb://localhost:27017")
engine = AIOEngine(client=client, database="shop")

coffee = Category(
    name="Coffee",
    description="A preparation of roasted and ground coffee seeds.",
)
espresso = Product(
    name="Italian Espresso",
    price=2.95,
    category=coffee,
)

await engine.save(espresso)
if name == "main":
asyncio.run(main())

The model definitions and asynchronous syntax strongly resemble Beanie, but the way you persist data (engine.save(...)) feels similar to MongoEngine and many other ODMs/ORMs.

ODMantic can also interact with MongoDB synchronously in specific scenarios, but its primary strength lies in asynchronous code paths.

Documentation is available on the ODMantic website, including a concise quick-start example.

Django

Django is not an ODM by design, but through third-party integrations it can be used to model and interact with MongoDB using a familiar ORM-like interface.

Libraries such as Djongo and the MongoEngine–Django integration allow developers to reuse Django’s model layer while targeting MongoDB as the backing datastore. This approach is often chosen when MongoDB is introduced into an existing Django project and you want to minimize changes to the application layer.

Inside an existing Django backend, you can use a MongoDB-backed Django model like this (using Djongo):

py
from djongo import models
class Category(models.Model):
name = models.CharField(max_length=200)
description = models.TextField()

class Meta:
    abstract = True
class Product(models.Model):
name = models.CharField(max_length=200)
description = models.TextField(null=True, blank=True)
price = models.FloatField(db_index=True)
category = models.EmbeddedField(
model_container=Category,
)

coffee = Category(
name="Coffee",
description="A preparation of roasted and ground coffee seeds.",
)
espresso = Product.objects.create(
name="Italian Espresso",
price=2.95,
category=coffee,
)

The Meta class definition with abstract = True tells Django not to create a separate collection for Category. It is just an abstract class that will be used for embedded objects. The objects.create(...) call then inserts the document into MongoDB using Django’s familiar query API.

Documentation for Djongo is available on its website, while the Django–MongoEngine integration can be found on GitHub.

Asynchronous vs. synchronous

In the overview above, we’ve seen both synchronous and asynchronous ODM libraries. So what is the difference, and which one is best for your use case?

At a high level:

Synchronous code executes operations one after another. Each I/O operation (like a database call) blocks the current thread until it finishes.
Asynchronous code is non-blocking. When an I/O operation is waiting on the network or disk, the event loop can switch to other tasks instead of blocking.

In Python, a mechanism called the Global Interpreter Lock (GIL) prevents true parallel execution of Python bytecode in multiple threads at the same time, ensuring memory consistency and state predictability. This is a separate concern from async I/O, but it’s worth keeping in mind when considering concurrency strategies.

Python’s asyncio module is designed for managing asynchronous operations using coroutines and an event loop. Python also supports concurrency via the threading module and can leverage multiple CPUs via multiprocessing, which bypasses the GIL by using separate processes.

As mentioned earlier, ODM libraries like Beanie and ODMantic provide asynchronous capabilities using asyncio, while Django typically relies on the ASGI gateway and its ecosystem for async support.

Another important piece in the async story is Motor, the official asynchronous driver for MongoDB in Python. Motor has historically been the go-to choice for async access in Python, but it was deprecated in May 2025 in favor of the PyMongo Async API in the pymongo library (in GA at the time of writing). MongoDB will not add new features to Motor and will provide only bug fixes until its end of life on May 14, 2026.

From a performance standpoint:

Asynchronous libraries tend to shine when tasks are frequently blocked by I/O and do not use much CPU — for example, web applications that handle a high volume of concurrent client requests. When a task is waiting for input/output, it can yield control so other tasks can proceed.
Synchronous programming can still be a great fit when there are relatively few concurrent requests and tasks are CPU-bound. In these cases, the overhead of managing an event loop might not pay off, and a straightforward synchronous approach can be simpler and efficient enough.

Conclusion

We have explored different approaches to ODMs in Python:

Beanie and ODMantic are modern, lightweight, and support asynchronous programming, making them attractive for new projects, microservices architectures, and applications that need to handle many concurrent requests efficiently.
MongoEngine offers a solid foundation for synchronous software and a mature ecosystem, with a familiar, Django-like style.
Given the popularity and historical importance of Django in the Python web ecosystem, we also looked at how to integrate MongoDB via Django-friendly tools like Djongo and the Django–MongoEngine integration, which can be particularly useful when you are adding MongoDB to an existing Django project.

Choosing the “best” Python ODM for MongoDB depends on your context:

If you are building a new async-first API or microservice, Beanie or ODMantic are strong candidates.
If you prefer a synchronous style and value maturity and stability, MongoEngine is a solid choice.
If your project already relies heavily on Django, leveraging Djongo or the Django–MongoEngine integration lets you take advantage of MongoDB while preserving familiar patterns.

Ultimately, all of these tools help you align Python’s object-oriented programming model with MongoDB’s flexible, document-oriented data model — so you can focus more on your application’s logic and less on boilerplate data-access code.

Best Practices Cheat Sheet for Flask-PyMongo

MongoDB Guests — Wed, 08 Apr 2026 11:17:13 +0000

This tutorial was written by João Araújo.

Introduction

Starting a project is always a challenge. Naming the first things, deciding on a project structure, what stack to use, and even where we want that project to go. While it is important to have most of it planned, if we are unsure or lack visibility into the project's direction, we should use the right tools to navigate any challenges that may arise. The ability to build applications that evolve over time is a strength of both MongoDB and Flask. If you are starting a project today and are not sure how fast it will grow, or if you just need quick prototyping, MongoDB performance aligned with Flask should be considered. If you are here just for the cheat sheet, it is at the end - but consider reading through to understand the “whys” behind it.

What is MongoDB?

MongoDB is an open-source NoSQL database that stores and manages data using flexible document-oriented structures rather than the traditional tables and rows of relational databases. In MongoDB, data is stored in JSON-like documents (BSON) organized into databases and collections, allowing schemas to be dynamic and easily changed over time, making it highly adaptable for handling diverse, structured, and unstructured data. It’s designed for scalability, flexibility, and ease of development, and is widely used in modern web and mobile applications where rapid iteration and large data volumes are common.

What is Flask?

Flask is a micro web framework written in Python that provides the core essentials for building web applications without imposing a lot of structure or extra components, making it lightweight and easy to use. Flask’s biggest strengths are its simplicity and flexibility. Because it doesn’t enforce a specific project layout or include unnecessary features by default, developers can structure applications as they see fit and choose the best tools for tasks such as database access, authentication, and form handling. Additionally, Flask has a large ecosystem of add-ons that provide features such as authentication, authorization, login, sessions, and more.

Why Choose MongoDB with Flask?

Choosing MongoDB with Flask is a great combination for Python web projects, as both tools are built around flexibility and simplicity. MongoDB’s document-oriented, schema-less storage means you don’t have to define rigid table structures ahead of time. Data can reside in flexible JSON-like documents, making it easy to get started and adapt as your app evolves. Another big plus of using MongoDB with Flask is that JSON-like data structures (e.g., Python dictionaries) map naturally to BSON documents in the database, so developers can work with familiar Python types without complex transformations. We could go on, but let's get this cheat sheet started.

Setup and Project Structure

Configure MongoDB connection parameters (URI, host, port, credentials) via environment variables or configuration files.

Always use configuration files or environment variables to define database connections and overall credentials. This ensures there is no hard-coded information across production and other environments.

In the environment file, there isn't much to see here. I added information on how and what access we have to our database, as well as the environment we run on.

.env
FLASK_ENV=development
FLASK_APP=app.py

MONGODB_URI=mongodb://localhost:27017
MONGODB_DB=my_database

Use a single, globally shared `MongoClient` (or an appropriately scoped one) instead of creating a new connection per request.

Avoid using multiple database clients within the same application. You should create a single connection and reuse it. MongoDB Drivers manage a connection pool and can be configured to allow more or fewer active connections. When a new connection is required, the driver will use an available connection or establish a new one. This ensures authentication occurs only once, so the time spent resolving the request is dedicated to the actual query rather than to authentication and operation execution. With Flask's built-in extension management, we can instantiate it once and reuse it later.

python
import os
from pymongo import MongoClient
from flask import current_app
from dotenv import load_dotenv

# load environment variables
load_dotenv()

# defining my mongo connection
class Mongo:
    def __init__(self):
        self.client = None
        self.db = None

    def init_app(self, app):
        uri = os.getenv("MONGODB_URI")
        db_name = os.getenv("MONGODB_DB")

        if not uri or not db_name:
            raise RuntimeError("MongoDB environment variables are not set")

        self.client = MongoClient(uri)
        self.db = self.client[db_name]

        # store reference inside Flask app as extension
        app.extensions["mongo"] = self

    @staticmethod
    def get_db():
        mongo = current_app.extensions.get("mongo")
        if not mongo:
            raise RuntimeError("Mongo extension not initialized")
        return mongo.db


mongo = Mongo()

Keep your database access logic, application logic (Flask routes), and configuration separate. Helps with upgrades, testing, and debugging.

By keeping your database access logic, configuration, and business rules separate, you can support each point of view independently without having to search your project to identify the root cause. This is especially useful when you need to understand the application workload and determine which index to use on MongoDB.

The example below shows an example project:

.md
project/
├── .env
├── app.py
├── extensions.py        # Mongo connection
├── routes/
│   └── example.py      # Route logic
├── repositories/
│   └── example_repo.py # Mongo collection queries
└── requirements.txt

extensions.py

In the following file, I am instantiating a MongoDB client. I will instantiate it later in my Flask app by calling the init_app function and passing the Flask instance. This way, whenever my “main” is called, it will create my connection.

python
import os
from pymongo import MongoClient
from flask import current_app
from dotenv import load_dotenv

# load environment variables
load_dotenv()

# defining my mongo connection
class Mongo:
    def __init__(self):
        self.client = None
        self.db = None

    def init_app(self, app):
        uri = os.getenv("MONGODB_URI")
        db_name = os.getenv("MONGODB_DB")

        if not uri or not db_name:
            raise RuntimeError("MongoDB environment variables are not set")

        self.client = MongoClient(uri)
        self.db = self.client[db_name]

        # store reference inside Flask app as extension
        app.extensions["mongo"] = self

    @staticmethod
    def get_db():
        mongo = current_app.extensions.get("mongo")
        if not mongo:
            raise RuntimeError("Mongo extension not initialized")
        return mongo.db


mongo = Mongo()

repositories/example_repo.py

Here, I am simply defining how I interact with my database. If required, I would also define indexes here (not creating them). Having both the indexes and your interactions with that collection in the same folder makes it easier to evaluate new indexes, remove unnecessary ones, or upgrade indexes as new requirements arise.

python
from extensions import Mongo

COLLECTION_NAME = "example"

def get_all():
    db = Mongo.get_db()
    return list(db[COLLECTION_NAME].find({}, {"_id": 0}))

def insert_message(message: str):
    db = Mongo.get_db()
    result = db[COLLECTION_NAME].insert_one({"message": message})
    return str(result.inserted_id)

routes/example.py

Finally, I'm using Flask Blueprints to define my routes. Blueprints are a powerful, built-in mechanism for structuring applications into modular, reusable components rather than placing all code in one file.

python
from flask import Blueprint, jsonify, request
from repositories.example_repo import get_all, insert_message

example_bp = Blueprint("example", __name__, url_prefix="/example")

@example_bp.route("/", methods=["GET"])
def list_messages():
    return jsonify(get_all())

@example_bp.route("/", methods=["POST"])
def create_message():
    payload = request.get_json(silent=True) or {}
    message = payload.get("message")

    if not message:
        return jsonify({"error": "message is required"}), 400

    inserted_id = insert_message(message)
    return jsonify({"inserted_id": inserted_id}), 201

app.py

python
from flask import Flask
from routes.example import example_bp
from extensions import mongo

def create_app():
    app = Flask(__name__)

    # init extensions
    mongo.init_app(app)

    # register routes
    app.register_blueprint(example_bp)

    return app

app = create_app()

if __name__ == "__main__":
    app.run(debug=True)

Data Modeling & Schema Design

Maintain consistent field names and types in the application.

Maintaining consistent field names and data types across the application is critical to avoid subtle bugs, simplify queries, and keep your data model predictable over time. When field names or types change inconsistently (for example, userId versus user_id, or a date stored sometimes as a string and sometimes as a Date), it increases cognitive load, breaks indexes, complicates aggregations, and can cause runtime errors that are hard to trace. Data consistency makes your application easier to maintain, safer to refactor, and more reliable when multiple services, developers, or analytics tools interact with the same data. If you need to have multiple versions of the same document, consider using Document Versioning pattern.

You can use the MongoDB schema validator to define and validate documents inserted into a collection after the schema definition.

javascript
db.createCollection("users", {
  validator: {
    $jsonSchema: {
      bsonType: "object",
      required: ["userId", "email", "isActive", "createdAt"],
      properties: {
        userId: {
          bsonType: "string",
          description: "Must be a string and is required"
        },
        email: {
          bsonType: "string",
          pattern: "^.+@.+$",
          description: "Must be a valid email string."
        },
        isActive: {
          bsonType: "bool",
          description: "Must be a boolean."
        },
        createdAt: {
          bsonType: "date",
          description: "Must be a BSON date."
        },
        optionalField: {
         bsonType: “string”,
         description: “This field is optional.”
       }
      }
    }
  },
  validationLevel: "strict",
  validationAction: "error"
})

Design your document model according to query and access patterns.

Designing your document model based on query and access patterns means shaping your MongoDB documents around how the application actually reads and writes data, not just how the data appears conceptually. By embedding frequently accessed data together and avoiding unnecessary joins, you reduce query complexity, minimize network round-trips, and improve performance. A model optimized for real access patterns is faster, simpler to query, and scales more predictably than one designed only from a relational mindset.

In MongoDB, data that is read together should usually live together. That’s the core idea behind modeling for access patterns.

Model not aligned with access patterns(reference-heavy):

javascript
// orders
{ "_id": 1, "customer_id": 10 }

// order_items
{ "order_id": 1, "product": "Book", "qty": 2 }

// customers
{ "_id": 10, "name": "Alice" }

Query (multiple round trips / $lookup):

db.orders.aggregate([
  { $match: { _id: 1 } },
  { $lookup: {
      from: "order_items",
      localField: "_id",
      foreignField: "order_id",
      as: "items"
  }},
  { $lookup: {
      from: "customers",
      localField: "customer_id",
      foreignField: "_id",
      as: "customer"
  }}
])

Model aligned with access patterns (embedded):

javascript
# orders
{
  "_id": 1,
  "customer": {
    "id": 10,
    "name": "Alice"
  },
  "items": [
    { "product": "Book", "qty": 2 },
    { "product": "Pen", "qty": 1 }
  ]
}

Query (single, simple read):

python
db.orders.findOne({ _id: 1 })

Performance & Query Optimization

Use indexes for fields that you frequently query, filter, or sort by.

Using indexes on frequently queried, filtered, or sorted fields is essential for efficient MongoDB queries. A well-designed compound index can support multiple query patterns when it follows the ESR rule (Equality → Sort → Range), meaning equality filters should come first, followed by sort fields, and then range conditions. By ordering fields correctly, a single index can satisfy multiple query patterns without creating multiple indexes:

“user”
“user” and “age”
“user”, “age”, and “workedHours”

Index creation:

python
db.example.createIndex({ user: 1, age: 1, workedHours: 1 })

Query examples:

python
db.example.find({ user: "joao" })

db.example.find({ user: "joao", age: "28" })
# or
db.example.find({ user: "joao" }).sort({age: 1})

db.example.find({
  user: "joao",
  workedHours: { $gte: 100, $lte: 200 }
}).sort({age: 1})

For potentially large result sets, use pagination (_id skip/limit).

For potentially large result sets, always use pagination to avoid loading too much data into memory and to keep response times predictable. In MongoDB, we can rely on the _id field for pagination over large skip values because it scales efficiently as collections grow and avoids unnecessary data scans. By filtering on _id and combining it with the $limit stage, each page starts where the previous one ended, providing consistent performance and avoiding the cost of scanning and skipping documents. This is possible because ObjectIds are unique and grow monotonically by default.

python
# First page
db.logs.find({})
  .sort({ _id: 1 })
  .limit(10)

# Next page (use last _id from previous page)
db.logs.find({ _id: { $gt: ObjectId("65a123...") } }) // change to $lt if using descending
  .sort({ _id: 1 })
  .limit(10)

Security

Always validate and sanitize user input before inserting it into MongoDB.

Always validate and sanitize user input before inserting it into MongoDB, especially when the input comes from a public endpoint, since HTTP requests should be treated as untrusted by default. Proper validation ensures required fields are present and have the correct data types, while sanitization prevents malformed data, unexpected structures, or NoSQL injection (e.g. $gt, $ne) from reaching the database.

Additionally, if you store sensitive data, such as passwords, never store it in plaintext. Always hash them with an algorithm, such as bcrypt. This protects data integrity, reduces security risks, and keeps your MongoDB collections consistent and secure, even when accessed.

Another protection layer is using Binary subtype 8. This subtype will replace its content in log files for ###. Making sure that even if someone gets access to your MongoDB logs, they will be redacted on that specific field.

python
from flask import request, jsonify
import bcrypt
from bson import Binary

@app.route("/users", methods=["POST"])
def create_user():
    payload = request.get_json(silent=True) or {}

    # Validation
    if not isinstance(payload.get("email"), str):
        return jsonify({"error": "email must be a string"}), 400
    if not isinstance(payload.get("password"), str):
        return jsonify({"error": "password must be a string"}), 400

    salt = bcrypt.gensalt()
    hashed_pwd = bcrypt.hashpw(payload["password"].encode('utf-8'), salt)

    # Sanitization
    user = {
        "email": payload["email"].strip().lower(),
        # Hash password before storing
        "password": Binary(hashed_pwd, 8),
        "is_active": bool(payload.get("is_active", True))
    }

    db.users.insert_one(user)
    return jsonify({"status": "created"}), 201

Conclusion

This article outlines best practices for using MongoDB with Flask. This is not an exhaustive list, but a few common mistakes I've seen after many consulting sessions. Hopefully, this has shed some light and will help you avoid those common mistakes. The best way to learn MongoDB is through its extensive documentation, which is comprehensive and always up to date. If you are unsure or couldn't find anything, there are always the forums and community to help out!

And as promised, here's the Cheat sheet!

Area	Best Practice	Key Action / Code Example
Setup & Structure	Configure connection via `environment variables`.	Use environment variables for credentials.
Setup & Structure	Use a single, globally shared `MongoClient`.	Instantiate `MongoClient` once in an extension and reuse it.
Setup & Structure	Separate logic (routes, repo, config).	Use Flask Blueprints for routes and dedicated repository files for database interactions.
Data Modeling	Maintain consistent field names and types.	Enforce consistency with MongoDB's `$jsonSchema` validator when required.
Data Modeling	Design model for query & access patterns.	Embed data that is read together (e.g., customer info inside the order document).
Performance	Use indexes for frequent query, filter, or sort fields.	Follow the ESR Rule for compound indexes: Equality, Sort, Range.
Performance	Use _id field for pagination.	Use `find({ _id: { $gt: last_id } }).sort({ _id: 1 }).limit(N)` to avoid large `skip` values.
Security	Validate and sanitize all user input.	Check data types and strip/lower strings before insertion.
Security	Never store sensitive data in plaintext.	Always hash passwords before saving them to the database.

Create a workspace scheduler using Bryntum Scheduler Pro and MongoDB

MongoDB Guests — Wed, 01 Apr 2026 11:40:43 +0000

This Tutorial was written by Arsalan Khattak.

Bryntum Scheduler Pro is a scheduling UI component for the web. With features such as a scheduling engine, constraints, and a resource utilization view, it simplifies managing complex schedules.

In this tutorial, we'll use Bryntum Scheduler Pro and MongoDB, the popular document database, to build a workspace booking app for meeting rooms, desk banks, and coworking lounges. We'll use MongoDB Atlas, the fully managed MongoDB cloud service.

We'll do the following:

Set up MongoDB Atlas and get the connection string.
Create an npm workspaces monorepo for the backend and frontend code.
Seed the MongoDB database.
Create the backend server and the Bryntum load and sync endpoints.
Create a Vite vanilla TypeScript client.
Add Bryntum Scheduler Pro to the client.

Here's what we'll build:

You can find the code for the completed tutorial in our GitHub repository: Workspace scheduler using Bryntum Scheduler Pro and MongoDB.

Prerequisites

To follow along, you need Node.js version 20.19+, installed on your system. You'll also need a MongoDB Atlas account, you can register for an Atlas account using your GitHub account, your Google account, or your email address.

Setting Up MongoDB Atlas

We'll use the MongoDB Atlas CLI to create the organization, project, and cluster from the terminal. Using the CLI instead of the Atlas Cloud UI also works well if you're following along with an AI coding agent. It provides a better agent experience.

First, install the MongoDB Atlas CLI. You can install it on macOS or Linux with Homebrew:

brew install mongodb-atlas

atlas auth login

Select UserAccount when the CLI asks how you want to authenticate.

Create a new Atlas organization for this tutorial:

atlas organizations create bryntum --output json

Copy the returned organization ID, then create a new Atlas project:

atlas projects create schedulerpro --orgId <orgId> --output json

Copy the returned project ID, and use it to set the default org_id for Atlas:

atlas config set org_id <orgId

Set the default project_id to your project ID as well:

atlas config set project_id <projectId>

Create an Atlas cluster using the following command:

atlas setup --projectId <projectId> --currentIp --connectWith skip --skipSampleData

Take note of the database username, database password, and connection string.

To find the allowed IP list in your MongoDB Atlas dashboard, go to Security in the left navigation menu and select Database & Network Access. Under Network Access, select IP Access List.

Make sure your IP address is on the allowed list; otherwise, your Express server won't be able to connect to the database. Click the + Add IP Address button to add your IP address.

Setting up the monorepo

Create an empty project folder:

mkdir bryntumschedulerpro-mongodb
cd bryntumschedulerpro-mongodb

Create a package.json file in the project root folder and add the following JSON object:

{
  "name": "bryntumschedulerpro-mongodb",
  "version": "1.0.0",
  "private": true,
  "workspaces": [
    "client",
    "server"
  ],
  "scripts": {
    "dev": "concurrently 'npm run dev --workspace server' 'npm run dev --workspace client'",
    "build": "npm run build --workspace server && npm run build --workspace client",
    "seed": "npm run seed --workspace server",
    "start": "npm run start --workspace server"
  },
  "devDependencies": {
    "@types/node": "22.13.10",
    "concurrently": "9.2.1",
    "nodemon": "3.1.14",
    "typescript": "5.9.3",
    "vite": "7.3.1"
  }
}

This root package defines an npm workspaces TypeScript monorepo with a server and a client. The dev script starts both the Express server and the Vite client together. We'll use the seed script to populate the MongoDB database with example scheduler data.

Install the shared root dependencies:

npm install

Create a tsconfig.base.json file in the root folder:

{
  "compilerOptions": {
    "target": "ES2022",
    "strict": true,
    "skipLibCheck": true,
    "esModuleInterop": true,
    "allowSyntheticDefaultImports": true,
    "forceConsistentCasingInFileNames": true,
    "resolveJsonModule": true
  }
}

This base TypeScript config keeps the shared compiler settings in one place, and both workspaces extend it.

Create a .gitignore file in the root folder and add the following lines to it:

# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*
lerna-debug.log*

# Dependencies
node_modules
client/node_modules
server/node_modules

# Build output
**/dist
.vite

# Local config
*.local
.env

# Editor directories and files
.vscode/*
!.vscode/extensions.json
.idea
.DS_Store
*.suo
*.ntvs*
*.njsproj
*.sln
*.sw?

Create a .env file in the root folder, add the following environmental variables to it:

PORT=1339
MONGODB_URI=mongodb+srv://<dbUsername>:<dbPassword>@<clusterName>.<uniqueId>.mongodb.net/?appName=devrel-tutorial-javascript-bryntum
MONGODB_DB=bryntum-schedulerpro

Add your MongoDB connection string to the MONGODB_URI and add your database username and password. You can get the clusterName and uniqueId by running the following command:

atlas clusters connectionStrings describe <clusterName>

Creating the backend Express Server

Create the server folders:

mkdir -p server/src/lib server/src/routes server/src/scripts

Create a package.json file in the server folder and add the following JSON object to it:

{
  "name": "server",
  "private": true,
  "type": "module",
  "scripts": {
    "dev": "nodemon --watch src --ext ts --exec \"npm run serve\"",
    "serve": "npm run build && node dist/server.js",
    "build": "tsc -p tsconfig.json",
    "start": "node dist/server.js",
    "seed": "npm run build && node dist/scripts/seed.js"
  }
}

This workspace compiles the TypeScript server to dist and reruns the built server with nodemon during development.

Install the server dependencies:

npm install express@4.21.2 dotenv@16.4.7 mongodb@6.15.0 --workspace=server
npm install --save-dev @types/express@5.0.1 --workspace=server

Create a tsconfig.json file in the server folder and add the following configuration object:

{
  "extends": "../tsconfig.base.json",
  "compilerOptions": {
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "rootDir": "src",
    "outDir": "dist",
    "lib": ["ES2022"],
    "types": ["node"]
  },
  "include": ["src/**/*.ts"]
}

This config compiles the server as Node.js ES modules. Because of this, the TypeScript source files will use .js import paths that match the compiled output.

Create a loadEnv.ts file in the server/src/lib folder and add the following lines of code to it:

import dotenv from 'dotenv';
import { existsSync } from 'node:fs';
import path from 'node:path';

export function loadEnv(): void {
    const candidates = [
        path.resolve(process.cwd(), '.env'),
        path.resolve(process.cwd(), '../.env')
    ];

    const envPath = candidates.find(existsSync);

    if (envPath) {
        dotenv.config({ path : envPath });
        return;
    }

    dotenv.config();
}

This helper loads environment variables whether we run the server scripts from the monorepo root or from the server workspace.

Now create a mongo.ts file in the server/src/lib folder and add the following code to it:

import { MongoClient, type MongoClientOptions, ServerApiVersion } from 'mongodb';

export function createMongoClient(uri: string): MongoClient {
    const isAtlasUri = uri.startsWith('mongodb+srv://') || uri.includes('.mongodb.net');
    const options: MongoClientOptions = isAtlasUri
        ? {
            serverApi : {
                version           : ServerApiVersion.v1,
                strict            : true,
                deprecationErrors : true
            }
        }
        : {};

    return new MongoClient(uri, options);
}

This helper uses the Atlas Server API options when the connection string points at MongoDB Atlas. Local MongoDB connections keep the default client options.

Create a cors.ts file in the server/src/lib folder and add the following code to it:

import type { RequestHandler } from 'express';

type CorsConfig = {
    origin : string
};

export default function cors(config: CorsConfig): RequestHandler {
    return (req, res, next) => {
        res.setHeader('Access-Control-Allow-Origin', config.origin);
        res.setHeader('Access-Control-Allow-Methods', 'GET,POST,PUT,PATCH,DELETE,OPTIONS');
        res.setHeader('Access-Control-Allow-Headers', 'Content-Type, Authorization');

        if (req.method === 'OPTIONS') {
            res.sendStatus(204);
            return;
        }

        next();
    };
}

This middleware allows the Vite client to call the API directly during local development.

Next, create a schedulerCrud.ts file in the server/src/lib/ folder to define the structure for the load and sync API routes. In this file, add the following imports, collection names, and TypeScript types that describe the request and response shapes:

import { randomUUID } from 'crypto';
import { Collection } from 'mongodb';

export const COLLECTIONS = {
    resources    : 'resources',
    events       : 'events',
    assignments  : 'assignments',
    calendars    : 'calendars',
    dependencies : 'dependencies'
} as const;

export type CrudRequestId = string | number;
export type GenericRecord = { _id?: string; [key: string]: any };
export type StoreKey = keyof typeof COLLECTIONS;
export type IdMap = Map<string, string>;
export type IdMaps = Record<StoreKey, IdMap>;
export type Normalizer = (record: GenericRecord) => GenericRecord;

export type StoreChanges = {
    added?: GenericRecord[];
    updated?: GenericRecord[];
    removed?: Array<{ id?: CrudRequestId }>;
};

export type SyncRequestBody = {
    requestId?: CrudRequestId | null;
    resources?: StoreChanges;
    events?: StoreChanges;
    assignments?: StoreChanges;
    calendars?: StoreChanges;
    dependencies?: StoreChanges;
};

export type StoreResponseSection = {
    rows: GenericRecord[];
    removed?: GenericRecord[];
};

export type SyncResponseBody = {
    success: boolean;
    requestId: CrudRequestId | null;
    resources?: StoreResponseSection;
    events?: StoreResponseSection;
    assignments?: StoreResponseSection;
    calendars?: StoreResponseSection;
    dependencies?: StoreResponseSection;
    message?: string;
};

The collections persist in the Bryntum data stores.

We'll use the Bryntum Crud manager to load the MongoDB data into the Bryntum Scheduler Pro and sync data changes to it. The Crud Manager expects a specific sync request structure and sync response structure, which you can see in the SyncRequestBody and SyncResponseBody types.

Add the following functions to the bottom of the file:

export function createIdMaps(): IdMaps {
    return {
        resources    : new Map(),
        events       : new Map(),
        assignments  : new Map(),
        calendars    : new Map(),
        dependencies : new Map()
    };
}

export function normalizeDate(value: unknown): unknown {
    if (value == null) {
        return value;
    }

    if (value instanceof Date) {
        return value.toISOString();
    }

    return value;
}

export function normalizeEventFields(record: GenericRecord): GenericRecord {
    if (!record) {
        return record;
    }

    const normalized = { ...record };

    if ('startDate' in record) {
        normalized.startDate = normalizeDate(record.startDate);
    }

    if ('endDate' in record) {
        normalized.endDate = normalizeDate(record.endDate);
    }

    return normalized;
}

In this code:

The createIdMaps function builds a phantom-to-permanent ID map for each store.
- When Bryntum creates a new record, it sends a temporary phantom ID that's generated on the client. The sync response then replaces the phantom ID with a permanent ID from the database. When the sync request sends multiple new records at once, one record might reference another that was also just created.
- Consider the example of a new dependency referencing a new event: The server inserts the event first, and maps its phantom ID to the new MongoDB _id. Then, when processing the dependency, resolveId looks up the event's phantom ID in the map to get the database-assigned _id.
The normalizeEventFields function converts Date objects to ISO strings before the server returns them.

Add the following ID resolver and normalizer functions to the bottom of the file:

export function resolveId(id: CrudRequestId, map: IdMap): string {
    const asString = String(id);

    return map.get(asString) || asString;
}

export function normalizeAssignmentFields(record: GenericRecord, idMaps: IdMaps): GenericRecord {
    if (!record) {
        return record;
    }

    const eventId = record.eventId ?? record.event;
    const resourceId = record.resourceId ?? record.resource;

    const normalized = {
        ...record
    };

    if (eventId != null) {
        normalized.eventId = resolveId(eventId, idMaps.events);
    }

    if (resourceId != null) {
        normalized.resourceId = resolveId(resourceId, idMaps.resources);
    }

    delete normalized.event;
    delete normalized.resource;

    return normalized;
}

export function normalizeDependencyFields(record: GenericRecord, idMaps: IdMaps): GenericRecord {
    if (!record) {
        return record;
    }

    const fromEvent = record.fromEvent ?? record.from;
    const toEvent = record.toEvent ?? record.to;

    const normalized = {
        ...record
    };

    if (fromEvent != null) {
        normalized.fromEvent = resolveId(fromEvent, idMaps.events);
    }

    if (toEvent != null) {
        normalized.toEvent = resolveId(toEvent, idMaps.events);
    }

    delete normalized.from;
    delete normalized.to;

    return normalized;
}

These two normalizers map Bryntum's event and resource references to permanent IDs using the ID maps built earlier in the same sync request.

Add the following helper functions that remove MongoDB internal fields:

export function stripInternalFields(record: GenericRecord): GenericRecord {
    if (!record) {
        return record;
    }

    const { _id, ...clean } = record;

    return { ...clean, id : String(_id) };
}

export function collectStoreRows(rows: GenericRecord[]): GenericRecord[] {
    return rows.map(row => stripInternalFields(row));
}

The stripInternalFields function maps MongoDB's _id field to Bryntum's id field before sending data to the client. The collectStoreRows function applies this cleanup to every row in a collection result.

Finally, add the applyStoreChanges() function that handles the add, update, and remove sync requests for every store:

export async function applyStoreChanges({
    collection,
    changes,
    idMap,
    normalizeAdded = value => value,
    normalizeUpdated = value => value
}: {
    collection: Collection<GenericRecord>;
    changes: StoreChanges;
    idMap: IdMap;
    normalizeAdded?: Normalizer;
    normalizeUpdated?: Normalizer;
}): Promise<GenericRecord[]> {
    const rows: GenericRecord[] = [];

    if (Array.isArray(changes.added)) {
        for (const rawRecord of changes.added) {
            const {
                $PhantomId,
                ...data
            } = rawRecord;

            const normalized = normalizeAdded(data);
            const id = String(normalized.id || randomUUID());
            delete normalized.id;
            const recordToStore = {
                ...normalized,
                _id : id
            };

            await collection.insertOne(recordToStore);

            if ($PhantomId) {
                idMap.set(String($PhantomId), id);
                rows.push({ $PhantomId, id });
            }
            else {
                rows.push({ id });
            }
        }
    }

    if (Array.isArray(changes.updated)) {
        for (const rawRecord of changes.updated) {
            const {
                id,
                ...updatedFields
            } = rawRecord;

            if (!id) {
                continue;
            }

            const normalized = normalizeUpdated(updatedFields);

            await collection.updateOne(
                { _id : String(id) },
                { $set : normalized }
            );
        }
    }

    if (Array.isArray(changes.removed) && changes.removed.length) {
        const ids = changes.removed
            .map(record => record.id)
            .filter(Boolean)
            .map(id => String(id));

        if (ids.length) {
            await collection.deleteMany({
                _id : {
                    $in : ids
                }
            });
        }
    }

    return rows;
}

For added records, the function strips the Bryntum $PhantomId, normalizes the data, and maps the phantom to the real ID so that dependent stores can resolve their references in the same sync request.

In MongoDB, every document needs a unique _id field as its primary key. If omitted, the MongoDB driver auto-generates an ObjectId, which consists of a timestamp, machine and process identifier, and counter. However, _id can be any unique value. We use a randomUUID() from Node.js's crypto module instead of ObjectId for consistency with our seed data. The seed script uses human-readable string IDs like 'booking-1' and 'room-terrace' so that cross-references between stores are easy to follow. For example, an assignment's eventId: 'booking-1' pointing to the event with an _id of 'booking-1'.

Creating a database seed script

Now let's create a seed script to populate the MongoDB Atlas database with example data. Create a seed.ts file in the server/src/scripts folder. Add the following imports, database connection setup, and the resources array:

import { loadEnv } from '../lib/loadEnv.js';
import { createMongoClient } from '../lib/mongo.js';

loadEnv();

const MONGODB_URI = process.env.MONGODB_URI || 'mongodb://127.0.0.1:27017';
const MONGODB_DB = process.env.MONGODB_DB || 'bryntum-schedulerpro';

const client = createMongoClient(MONGODB_URI);

type GenericRecord = Record<string, unknown>;

const resources: GenericRecord[] = [
    {
        _id           : 'room-terrace',
        name          : 'Terrace Briefing Room',
        workspaceType : 'Meeting room',
        neighborhood  : 'Docklands',
        floor         : 'Level 2',
        capacity      : 8,
        maxUnits      : 100,
        city          : 'Cape Town',
        amenities     : 'Display wall, VC bar'
    },
    {
        _id           : 'room-harbor',
        name          : 'Harbor Strategy Suite',
        workspaceType : 'Meeting room',
        neighborhood  : 'Harbor Wing',
        floor         : 'Level 5',
        capacity      : 16,
        maxUnits      : 100,
        city          : 'Cape Town',
        amenities     : 'Board table, dual screens'
    },
    {
        _id           : 'room-sprint',
        name          : 'Sprint Garage',
        workspaceType : 'Meeting room',
        neighborhood  : 'Atrium North',
        floor         : 'Level 3',
        capacity      : 6,
        maxUnits      : 100,
        city          : 'Cape Town',
        amenities     : 'Whiteboards, pin-up wall'
    },
    {
        _id           : 'desk-orbit-1',
        name          : 'Orbit Desk Bank A',
        workspaceType : 'Desk bank',
        neighborhood  : 'Docklands',
        floor         : 'Level 4',
        capacity      : 6,
        maxUnits      : 180,
        city          : 'Cape Town',
        amenities     : 'Sit-stand desks, lockers'
    },
    {
        _id           : 'desk-orbit-2',
        name          : 'Orbit Desk Bank B',
        workspaceType : 'Desk bank',
        neighborhood  : 'Docklands',
        floor         : 'Level 4',
        capacity      : 6,
        maxUnits      : 180,
        city          : 'Cape Town',
        amenities     : 'Phone booths, lockers'
    },
    {
        _id           : 'desk-focus',
        name          : 'Focus Pods',
        workspaceType : 'Desk bank',
        neighborhood  : 'Atrium North',
        floor         : 'Level 2',
        capacity      : 4,
        maxUnits      : 140,
        city          : 'Cape Town',
        amenities     : 'Quiet zone, task lights'
    },
    {
        _id           : 'coworking-canvas',
        name          : 'Canvas Coworking Lounge',
        workspaceType : 'Coworking lounge',
        neighborhood  : 'Harbor Wing',
        floor         : 'Ground',
        capacity      : 24,
        maxUnits      : 300,
        city          : 'Cape Town',
        amenities     : 'Cafe seating, soft booths'
    },
    {
        _id           : 'coworking-sky',
        name          : 'Skyline Lounge',
        workspaceType : 'Coworking Lounge',
        neighborhood  : 'Atrium North',
        floor         : 'Rooftop',
        capacity      : 18,
        maxUnits      : 260,
        city          : 'Cape Town',
        amenities     : 'Outdoor tables, power rails'
    }
];

Each resource represents a bookable workspace with a type, location, capacity, and amenities. The maxUnits field sets the resource's percentage capacity. It works together with the assignment units field: if the sum of all assignment units for a resource exceeds its maxUnits, the utilization panel marks that resource as overallocated.

Meeting rooms use 100 by default, so that a single 100% booking fills the room.
Desk banks use 180, so two overlapping bookings at 80% each would still be within capacity.
Coworking lounges use 300, allowing several concurrent partial bookings before hitting the limit.

Now add the events array. Each event represents a booking with a category, host, team, and time window:

const events: GenericRecord[] = [
    {
        _id             : 'booking-1',
        name            : 'Weekly launch review',
        bookingCategory : 'Meeting',
        host            : 'Ava',
        team            : 'Product',
        attendees       : 7,
        notes           : 'Mirror a client-facing property review flow with no overlaps.',
        startDate       : '2026-03-16T08:30:00',
        endDate         : '2026-03-16T11:30:00'
    },
    {
        _id             : 'booking-2',
        name            : 'Quiet desk block',
        bookingCategory : 'Desk',
        host            : 'Jules',
        team            : 'Growth',
        attendees       : 5,
        notes           : 'Focus time for a temporary landing team.',
        startDate       : '2026-03-16T09:00:00',
        endDate         : '2026-03-16T17:00:00'
    },
    {
        _id             : 'booking-3',
        name            : 'Sales deal room',
        bookingCategory : 'Meeting',
        host            : 'Maya',
        team            : 'Sales',
        attendees       : 12,
        notes           : 'Large suite booking for end-of-quarter negotiations.',
        startDate       : '2026-03-16T11:00:00',
        endDate         : '2026-03-16T14:00:00'
    },
    {
        _id             : 'booking-4',
        name            : 'Design sprint desks',
        bookingCategory : 'Desk',
        host            : 'Niko',
        team            : 'Design',
        attendees       : 6,
        notes           : 'Desk bank reserved as one inventory block.',
        startDate       : '2026-03-17T08:00:00',
        endDate         : '2026-03-17T18:00:00'
    },
    {
        _id             : 'booking-5',
        name            : 'Founder coffee session',
        bookingCategory : 'Coworking',
        host            : 'Lebo',
        team            : 'Community',
        attendees       : 14,
        notes           : 'Shared-capacity lounge booking with flexible units.',
        startDate       : '2026-03-17T10:00:00',
        endDate         : '2026-03-17T13:00:00'
    },
    {
        _id             : 'booking-6',
        name            : 'Board prep workshop',
        bookingCategory : 'Meeting',
        host            : 'Dylan',
        team            : 'Operations',
        attendees       : 5,
        notes           : 'Smaller room blocked for a prep workshop.',
        startDate       : '2026-03-17T12:30:00',
        endDate         : '2026-03-17T14:30:00'
    },
    {
        _id             : 'booking-7',
        name            : 'Product all-hands overflow',
        bookingCategory : 'Coworking',
        host            : 'Rae',
        team            : 'Product',
        attendees       : 20,
        notes           : 'Lounge used as shared overflow for team gathering.',
        startDate       : '2026-03-18T09:00:00',
        endDate         : '2026-03-18T12:00:00'
    },
    {
        _id             : 'booking-8',
        name            : 'Engineering hot-desk rotation',
        bookingCategory : 'Desk',
        host            : 'Theo',
        team            : 'Engineering',
        attendees       : 4,
        notes           : 'Desk cluster reserved to mirror a rotating occupancy block.',
        startDate       : '2026-03-18T08:00:00',
        endDate         : '2026-03-18T16:00:00'
    }
];

The bookings span three days and cover all three workspace types.

Finally, add the assignments, calendar, dependencies, and the seed function:

const assignments: GenericRecord[] = [
    { _id : 'assign-1', eventId : 'booking-1', resourceId : 'room-terrace', units : 100 },
    { _id : 'assign-2', eventId : 'booking-2', resourceId : 'desk-orbit-1', units : 80 },
    { _id : 'assign-3', eventId : 'booking-3', resourceId : 'room-harbor', units : 100 },
    { _id : 'assign-4', eventId : 'booking-4', resourceId : 'desk-orbit-2', units : 100 },
    { _id : 'assign-5', eventId : 'booking-5', resourceId : 'coworking-canvas', units : 55 },
    { _id : 'assign-6', eventId : 'booking-6', resourceId : 'room-sprint', units : 100 },
    { _id : 'assign-7', eventId : 'booking-7', resourceId : 'coworking-sky', units : 70 },
    { _id : 'assign-8', eventId : 'booking-8', resourceId : 'desk-focus', units : 65 }
];

const calendars: GenericRecord[] = [
    {
        _id       : 'workspace-hours',
        name      : 'Workspace Hours',
        intervals : [
            {
                recurrentStartDate : 'every weekday at 07:00',
                recurrentEndDate   : 'every weekday at 20:00',
                isWorking          : true
            }
        ]
    }
];

const dependencies: GenericRecord[] = [
    { _id : 'dep-1', fromEvent : 'booking-1', toEvent : 'booking-6' },
    { _id : 'dep-2', fromEvent : 'booking-4', toEvent : 'booking-8' },
    { _id : 'dep-3', fromEvent : 'booking-5', toEvent : 'booking-7' }
];

async function seed(): Promise<void> {
    await client.connect();

    const db = client.db(MONGODB_DB);
    const collectionNames = ['resources', 'events', 'assignments', 'calendars', 'dependencies'];

    await Promise.all(collectionNames.map(name => db.collection(name).deleteMany({})));
    await db.collection('resources').insertMany(resources);
    await db.collection('events').insertMany(events);
    await db.collection('assignments').insertMany(assignments);
    await db.collection('calendars').insertMany(calendars);
    await db.collection('dependencies').insertMany(dependencies);

    console.log(`Seeded ${MONGODB_DB} with workspace scheduler data.`);

    await client.close();
}

void seed().catch(error => {
    console.error(error);
    process.exit(1);
});

Assignments connect each booking to a workspace. The units field sets the utilization percentage for each assignment, which the Bryntum Scheduler Pro utilization widget will display.

The seed script clears all five collections before inserting the example data, so every run resets the database to the same state.

Now seed the database:

npm run seed

You can see the created collections and documents in your MongoDB Atlas dashboard. In the left navigation menu, under Database, select Clusters. Click on the Browse Collections to open the Data Explorer:

Creating the load and sync routes

Now let's create the load and sync routes that will load data into the Bryntum Scheduler Pro and save data changes to the MongoDB Atlas database.

Create a scheduler.ts file in the server/src/routes folder. Add the following imports, router factory, and the /load request route:

import { type Request, type Response, Router } from 'express';
import type { Db } from 'mongodb';
import {
    COLLECTIONS,
    applyStoreChanges,
    collectStoreRows,
    createIdMaps,
    normalizeAssignmentFields,
    normalizeDependencyFields,
    normalizeEventFields,
    type GenericRecord,
    type SyncRequestBody,
    type SyncResponseBody
} from '../lib/schedulerCrud.js';

type GetDb = () => Db;

export function createSchedulerRouter(getDb: GetDb): Router {
    const router = Router();

    router.get('/load', async(_req: Request, res: Response) => {
        try {
            const database = getDb();
            const [resources, events, assignments, calendars, dependencies] = await Promise.all([
                database.collection<GenericRecord>(COLLECTIONS.resources).find({}).sort({ neighborhood : 1, floor : 1, name : 1 }).toArray(),
                database.collection<GenericRecord>(COLLECTIONS.events).find({}).sort({ startDate : 1 }).toArray(),
                database.collection<GenericRecord>(COLLECTIONS.assignments).find({}).toArray(),
                database.collection<GenericRecord>(COLLECTIONS.calendars).find({}).toArray(),
                database.collection<GenericRecord>(COLLECTIONS.dependencies).find({}).toArray()
            ]);

            res.json({
                success      : true,
                resources    : { rows : collectStoreRows(resources) },
                events       : { rows : collectStoreRows(events).map(normalizeEventFields) },
                assignments  : { rows : collectStoreRows(assignments) },
                calendars    : { rows : collectStoreRows(calendars) },
                dependencies : { rows : collectStoreRows(dependencies) }
            });
        }
        catch (error) {
            console.error(error);

            res.status(500).json({
                success : false,
                message : 'Could not load workspace scheduler data'
            });
        }
    });

The /load route queries all five MongoDB collections in parallel and returns them in the structure that Bryntum's ProjectModel expects. Resources are sorted by neighborhood, floor, and name so the grid rows appear in a logical order.

Add the /sync request route below the /load route:

router.post('/sync', async(req: Request, res: Response) => {
        const {
            requestId,
            resources,
            events,
            assignments,
            calendars,
            dependencies
        } = req.body as SyncRequestBody;

        const idMaps = createIdMaps();

        try {
            const database = getDb();
            const response: SyncResponseBody = {
                success   : true,
                requestId : requestId || null
            };

            if (resources) {
                const rows = await applyStoreChanges({
                    collection : database.collection<GenericRecord>(COLLECTIONS.resources),
                    changes    : resources,
                    idMap      : idMaps.resources
                });

                if (rows.length) {
                    response.resources = { rows };
                }
            }

            if (events) {
                const rows = await applyStoreChanges({
                    collection       : database.collection<GenericRecord>(COLLECTIONS.events),
                    changes          : events,
                    idMap            : idMaps.events,
                    normalizeAdded   : normalizeEventFields,
                    normalizeUpdated : normalizeEventFields
                });

                if (rows.length) {
                    response.events = { rows };
                }
            }

            if (assignments) {
                const rows = await applyStoreChanges({
                    collection : database.collection<GenericRecord>(COLLECTIONS.assignments),
                    changes    : assignments,
                    idMap      : idMaps.assignments,
                    normalizeAdded(record) {
                        return normalizeAssignmentFields(record, idMaps);
                    },
                    normalizeUpdated(record) {
                        return normalizeAssignmentFields(record, idMaps);
                    }
                });

                if (rows.length) {
                    response.assignments = { rows };
                }
            }

            if (calendars) {
                const rows = await applyStoreChanges({
                    collection : database.collection<GenericRecord>(COLLECTIONS.calendars),
                    changes    : calendars,
                    idMap      : idMaps.calendars
                });

                if (rows.length) {
                    response.calendars = { rows };
                }
            }

            if (dependencies) {
                const rows = await applyStoreChanges({
                    collection : database.collection<GenericRecord>(COLLECTIONS.dependencies),
                    changes    : dependencies,
                    idMap      : idMaps.dependencies,
                    normalizeAdded(record) {
                        return normalizeDependencyFields(record, idMaps);
                    },
                    normalizeUpdated(record) {
                        return normalizeDependencyFields(record, idMaps);
                    }
                });

                if (rows.length) {
                    response.dependencies = { rows };
                }
            }

            res.json(response);
        }
        catch (error) {
            console.error(error);

            res.status(500).json({
                success   : false,
                requestId : requestId || null,
                message   : 'Could not sync workspace scheduler data'
            } satisfies SyncResponseBody);
        }
    });

    return router;
}

The /sync route processes each store in order: resources and events first, then assignments and dependencies. This order matters because assignments reference event and resource IDs, so those ID maps need to be populated before the assignment normalizer runs. For added rows, the response includes the database-assigned IDs so the Bryntum Scheduler Pro can replace its temporary phantom IDs with the database IDs.

Creating the Express server entry point

Create a server.ts in the server/srcfolder and add the following code to it:

import express from 'express';
import { type Db } from 'mongodb';
import cors from './lib/cors.js';
import { loadEnv } from './lib/loadEnv.js';
import { createMongoClient } from './lib/mongo.js';
import { createSchedulerRouter } from './routes/scheduler.js';

loadEnv();

const PORT = Number(process.env.PORT || 1339);
const FRONTEND_URL = process.env.FRONTEND_URL || 'http://localhost:5173';
const MONGODB_URI = process.env.MONGODB_URI || 'mongodb://127.0.0.1:27017';
const MONGODB_DB = process.env.MONGODB_DB || 'workspace_scheduler2';

const app = express();
app.use(cors({
    origin : FRONTEND_URL
}));
app.use(express.json({ limit : '2mb' }));

const client = createMongoClient(MONGODB_URI);
let db: Db | undefined;

function getDb(): Db {
    if (!db) {
        throw new Error('Database connection is not ready yet');
    }

    return db;
}

app.use('/api', createSchedulerRouter(getDb));

async function start(): Promise<void> {
    await client.connect();
    db = client.db(MONGODB_DB);
    console.log(`MongoDB ready for database "${MONGODB_DB}"`);

    app.listen(PORT, () => {
        console.log(`Workspace Scheduler server running on http://localhost:${PORT}`);
    });
}

start().catch((error) => {
    console.error('Failed to start server:', error);
    process.exit(1);
});

This server loads the environment variables, connects to MongoDB, and mounts the scheduler routes under /api.

Creating the Vite client app

Generate the Vite vanilla TypeScript client app using the following command:

npm create vite@latest client -- --template vanilla-ts

Select Install with npm and start now in the CLI options.

Replace the "scripts" object in the client/package.json with the following scripts:

"scripts": {
  "dev": "vite",
  "build": "tsc -p tsconfig.json && vite build"
}

Replace the configuration in client/tsconfig.json with the following code:
{
  "extends": "../tsconfig.base.json",
  "compilerOptions": {
    "module": "ESNext",
    "moduleResolution": "Bundler",
    "lib": ["ES2022", "DOM", "DOM.Iterable"],
    "types": ["vite/client"],
    "noEmit": true
  },
  "include": ["src/**/*.ts", "vite.config.ts"]
}

This config uses the Vite bundler module resolution strategy and does not emit JavaScript during the TypeScript check.

Create a vite.config.ts file in the client folder and add the following lines of code to it:

import { defineConfig } from 'vite';

export default defineConfig({
    server : {
        port  : 5173,
        proxy : {
            '/api' : 'http://localhost:1339'
        }
    },
    build : {
        outDir : 'dist'
    }
});

This config proxies /api requests to the Express server running on port 1339, so the client can use relative API URLs during local development.

Replace client/index.html with the following code:

<!doctype html>
<html lang="en">
<head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Workspace Scheduler</title>
</head>
<body>
    <main id="app">
        <div id="scheduler" class="scheduler-host"></div>
        <div id="utilization" class="utilization-host"></div>
    </main>
    <script type="module" src="/src/main.ts"></script>
</body>
</html>

This HTML creates two mount points: one for the Scheduler Pro, and one for the linked resource utilization view.

Delete the Vite starter files that we do not use:

rm -r client/public/icons.svg client/src/assets client/src/counter.ts

Installing the Bryntum Scheduler Pro

If you’re using the free trial, install the public Bryntum trial package:

npm install @bryntum/schedulerpro@npm:@bryntum/schedulerpro-trial  --workspace=client

If you have a Bryntum license, refer to our npm Repository Guide and install the licensed package:

npm install @bryntum/schedulerpro --workspace=client

Configuring the Bryntum Scheduler Pro

Create a schedulerProConfig.ts file in the client/src folder. Add the following imports, custom BookingModel, color map, filter dropdown options, and options objects:

import {
    DateHelper,
    EventModel,
    ProjectModel,
    StringHelper,
    type ProjectModelConfig,
    type ResourceUtilizationConfig,
    type SchedulerEventModel,
    type SchedulerProConfig
} from '@bryntum/schedulerpro';

class BookingModel extends EventModel {
    static override fields = [
        { name : 'bookingCategory', type : 'string' },
        { name : 'host', type : 'string' },
        { name : 'team', type : 'string' },
        { name : 'attendees', type : 'number' },
        { name : 'notes', type : 'string' }
    ];
}

const bookingColors: Record<string, string> = {
    Meeting   : '#ea6d4f',
    Desk      : '#267b8a',
    Coworking : '#d69a27'
};

export const workspaceTypeOptions = [
    { value : 'all', text : 'All types' },
    { value : 'Meeting room', text : 'Meeting room' },
    { value : 'Desk bank', text : 'Desk bank' },
    { value : 'Coworking lounge', text : 'Coworking lounge' }
] as const;

export const neighborhoodOptions = [
    { value : 'all', text : 'All neighborhoods' },
    { value : 'Docklands', text : 'Docklands' },
    { value : 'Atrium North', text : 'Atrium North' },
    { value : 'Harbor Wing', text : 'Harbor Wing' }
] as const;

export const filterState = {
    workspaceType : 'all',
    neighborhood  : 'all'
};

The BookingModel adds custom fields to the Bryntum EventModel. The bookingColors map gives each booking category a distinct color on the timeline.

Add the project configuration and instantiation:

export const projectConfig: Partial<ProjectModelConfig> = {
    eventModelClass : BookingModel,
    calendar        : 'workspace-hours',
    transport : {
        load : {
            url : '/api/load'
        },
        sync : {
            url : '/api/sync'
        }
    },
    autoLoad         : true,
    autoSync         : true,
    validateResponse : true
};

export const project = new ProjectModel(projectConfig);

The projectConfig transport property configures the load and sync URLs that the Bryntum Scheduler Pro Crud Manager uses. These URLs are the Express API endpoints that we created. Because autoLoad and autoSync are both enabled, the scheduler fetches data on page load and posts changes back to the API whenever data is modified in the scheduler.

Add the following scheduler configuration:

export const schedulerConfig: Partial<SchedulerProConfig> = {
    appendTo   : 'scheduler',
    project,
    startDate  : '2026-03-16T07:00:00',
    endDate    : '2026-03-18T20:00:00',
    viewPreset : 'hourAndDay',
    rowHeight  : 62,
    barMargin  : 7,
    multiEventSelect : true,
    columns          : [
        {
            type  : 'resourceInfo',
            text  : 'Workspace',
            field : 'name',
            width : 240
        },
        {
            text  : 'Type',
            field : 'workspaceType',
            width : 145
        },
        {
            text  : 'Neighborhood',
            field : 'neighborhood',
            width : 150
        },
        {
            text  : 'Floor',
            field : 'floor',
            width : 90
        },
        {
            text   : 'Capacity',
            field  : 'capacity',
            width  : 100,
            align  : 'center',
            editor : false
        }
    ],
    features : {
        dependencies : true,
        eventMenu    : true,
        taskEdit     : {
            editorConfig : {
                title : 'Reservation',
                width : '36em'
            },
            items : {
                generalTab : {
                    title : 'Booking details',
                    items : {
                        resourcesField : {
                            label : 'Workspace'
                        },
                        bookingCategory : {
                            type     : 'combo',
                            name     : 'bookingCategory',
                            label    : 'Booking type',
                            editable : false,
                            items    : ['Meeting', 'Desk', 'Coworking'],
                            weight   : 210
                        },
                        host : {
                            type   : 'text',
                            name   : 'host',
                            label  : 'Host',
                            weight : 220
                        },
                        team : {
                            type   : 'text',
                            name   : 'team',
                            label  : 'Team',
                            weight : 230
                        },
                        attendees : {
                            type   : 'number',
                            name   : 'attendees',
                            label  : 'Attendees',
                            min    : 1,
                            weight : 240
                        },
                        notes : {
                            type   : 'text',
                            name   : 'notes',
                            label  : 'Notes',
                            weight : 250
                        }
                    }
                }
            }
        },
    },
    eventRenderer({ eventRecord, renderData }) {
        renderData.eventColor = bookingColors[eventRecord.get('bookingCategory')] || '#1d5b52';

        return {
            children : [
                {
                    className : 'booking-title',
                    text      : eventRecord.name
                },
                {
                    className : 'booking-meta',
                    text      : `${eventRecord.get('host') || 'TBD'} | ${eventRecord.get('attendees') || 0} seats`
                }
            ]
        };
    }
};

Here, we configure the timeline settings, task editor, and event renderer. The columns array defines the locked grid on the left side of the scheduler. The taskEdit feature customizes the reservation editor popup with the booking-specific fields. The eventRenderer sets the event bar color based on booking category and shows the host name and seat count inside each bar.

Add the utilization panel configuration at the end of the file:

export const utilizationConfig: Partial<ResourceUtilizationConfig> = {
    appendTo    : 'utilization',
    project,
    hideHeaders : true,
    rowHeight   : 44,
    showBarTip  : true,
    showBarText : true,
    features    : {
        nonWorkingTime : false
    },
    columns : [
        {
            type  : 'tree',
            text  : 'Resource / booking',
            field : 'name',
            flex  : 1
        }
    ]
};

This configures the resource utilization widget that we'll display below the scheduler. It shares the same project instance as the scheduler, so both views stay in sync. The tree column lets you expand each resource to see its individual bookings. The showBarTip and showBarText options display the utilization percentage on the bars so you can spot overallocated workspaces at a glance.

Replace the styles in the client/src/style.css file with the following:

@import url('https://fonts.googleapis.com/css2?family=Poppins:wght@400;500;600;700&display=swap');
@import '@bryntum/schedulerpro/fontawesome/css/fontawesome.css';
@import '@bryntum/schedulerpro/fontawesome/css/solid.css';
@import '@bryntum/schedulerpro/schedulerpro.css';
@import '@bryntum/schedulerpro/svalbard-light.css';

* {
    margin: 0;
}

body {
    font-family: 'Poppins', 'Segoe UI', Arial, sans-serif;
}

#app {
    display        : flex;
    flex-direction : column;
    height         : 100vh;
    font-size      : 14px;
}

#scheduler {
    flex : 3;
}

#utilization {
    flex       : 2;
    border-top : 2px solid #e0e0e0;
}

This stylesheet imports the Bryntum Scheduler Pro structural CSS, the Svalbard theme, and gives the page a two-panel layout, with the scheduler above the utilization view. Bryntum provides five themes with light and dark variants.

Replace the code in lient/src/main.ts with the following:

import {
    Combo,
    ResourceUtilization,
    SchedulerPro,
} from '@bryntum/schedulerpro';
import './style.css';
import {
    filterState,
    neighborhoodOptions,
    project,
    schedulerConfig,
    utilizationConfig,
    workspaceTypeOptions
} from './schedulerProConfig';

const scheduler = new SchedulerPro({
    ...schedulerConfig,
    tbar : [
        {
            type         : 'combo',
            ref          : 'workspaceTypeFilter',
            width        : 190,
            editable     : false,
            value        : filterState.workspaceType,
            valueField   : 'value',
            displayField : 'text',
            items        : workspaceTypeOptions,
            placeholder  : 'Workspace type',
            listeners    : {
                change({ value }: { value: string }) {
                    filterState.workspaceType = value || 'all';
                    applyResourceFilters();
                }
            }
        },
        {
            type         : 'combo',
            ref          : 'neighborhoodFilter',
            width        : 205,
            editable     : false,
            value        : filterState.neighborhood,
            valueField   : 'value',
            displayField : 'text',
            items        : neighborhoodOptions,
            placeholder  : 'Neighborhood',
            listeners    : {
                change({ value }: { value: string }) {
                    filterState.neighborhood = value || 'all';
                    applyResourceFilters();
                }
            }
        },
        {
            type     : 'button',
            icon     : 'b-icon-undo',
            text     : 'Reset filters',
            onAction : () => {
                filterState.workspaceType = 'all';
                filterState.neighborhood = 'all';
                (scheduler.widgetMap.workspaceTypeFilter as Combo).value = 'all';
                (scheduler.widgetMap.neighborhoodFilter as Combo).value = 'all';
                applyResourceFilters();
            }
        },
        {
            type     : 'button',
            icon     : 'b-icon-search-plus',
            text     : 'Zoom in',
            onAction : () => scheduler.zoomIn()
        },
        {
            type     : 'button',
            icon     : 'b-icon-search-minus',
            text     : 'Zoom out',
            onAction : () => scheduler.zoomOut()
        }
    ]
});

This instantiates a Scheduler Pro instance, passes in the schedulerConfig, and adds a toolbar with filter dropdowns and zoom buttons. The two combo dropdowns filter the resource grid by workspace type and neighborhood. The reset button clears both filters, and the zoom buttons let you adjust the timeline scale.

Create the partnered utilization view and the filter function below the scheduler:

new ResourceUtilization({
    ...utilizationConfig,
    partner : scheduler
});

function applyResourceFilters(): void {
    const selectedType = filterState.workspaceType;
    const selectedNeighborhood = filterState.neighborhood;

    scheduler.resourceStore.filter({
        filters : (resource: Record<string, unknown>) => {
            const matchesType = selectedType === 'all' || resource.workspaceType === selectedType;
            const matchesNeighborhood = selectedNeighborhood === 'all' || resource.neighborhood === selectedNeighborhood;

            return matchesType && matchesNeighborhood;
        },
        replace : true
    });
}

project.on({
    load() {
        applyResourceFilters();
    }
});

The partner property links the utilization panel to the scheduler so they share the same time axis and scroll position. The applyResourceFilters function runs a combined filter on the resource store whenever a dropdown changes. The project.on('load') listener reapplies the filters after the initial data load completes.

Running the application

Run the client and server concurrently using the following command:

npm run dev

Open http://localhost:5173/. You’ll see the Bryntum Scheduler Pro populated with the example data:

[VIDEO]

You now have a Bryntum Scheduler Pro workspace booking CRUD app powered by MongoDB Atlas. The linked resource utilization view lets you spot overallocated resources.

Next Steps

Now that you have a working Bryntum Scheduler Pro app with MongoDB, explore the additional features you can add on the examples page, such as:

Check out the Bryntum AI feature that lets users interact with scheduling data using natural language through a chat panel.

Both Bryntum and MongoDB have MCP servers to assist with agentic workflows:

The Bryntum MCP server returns API references, configuration options, working source code examples, and migration guides for specific Bryntum versions.
The MongoDB MCP server helps with code generation, database optimization, and data exploration.

How MongoDB Handles High Traffic and Large Datasets

MongoDB Guests — Mon, 30 Mar 2026 11:44:41 +0000

This article was written by Karen Zhang.

MongoDB is built for scale. Its distributed architecture combines replica sets, sharding, and intelligent query routing to handle high traffic and large datasets without requiring you to endlessly upgrade a single server.

This article covers how each of these mechanisms works, when to apply them, and what you need to get right to avoid performance problems in production.

Horizontal Scaling in MongoDB

1. The limits of vertical scaling

When a database starts slowing down, the instinct is to add more hardware: more CPU cores, more RAM, faster storage. This is vertical scaling, and it works up to a point.

The problem is that a single server can only get so large. Beyond a certain threshold, the cost grows exponentially while the performance gains diminish. More importantly, a single powerful server is still a single point of failure. If it goes down, your entire application goes with it.

Horizontal scaling solves this differently. Instead of making one machine larger, you distribute the work across multiple machines. MongoDB is designed for this from the ground up.

2. Replica sets for availability and read scalability

The foundation of MongoDB’s horizontal architecture is the replica set — a group of MongoDB instances that all hold the same data, kept in sync through continuous replication.

A replica set consists of:

Primary: The only member that accepts write operations. All writes go here.
Secondaries: Members that replicate from the primary’s oplog (operation log) and maintain a copy of the data.
Arbiter (optional): Participates in elections but holds no data. Used to break ties in smaller deployments.

Primary  ──oplog──►  Secondary 1
         ──oplog──►  Secondary 2
         ──oplog──►  Secondary 3 (hidden, used for backups)

When the primary becomes unavailable, the remaining members automatically elect a new primary — typically within a few seconds — with no manual intervention required. For deployment architecture options, refer to replica set deployment architectures.

3. Handling concurrent reads and writes at scale

By default, all reads and writes go through the primary. This keeps things simple and guarantees you’re always reading the latest data. For many applications, this is the right choice.

For read-heavy workloads where some replication lag is acceptable — analytics dashboards, reporting queries, batch exports — you can offload reads to secondaries. This distributes read traffic across members and reduces pressure on the primary.

Please note that secondary reads are not appropriate for anything requiring read-your-own-writes consistency. If a user writes a record and immediately reads it back, routing that read to a secondary may return stale data.

4. Traffic distribution using read preferences

MongoDB gives you five read preference modes to control where read operations are sent:

Mode	Behavior
primary	All reads from primary (default).
primaryPreferred	Primary when available, fallback to secondary.
secondary	Always read from a secondary.
secondaryPreferred	Secondary, when available, fallback to primary.
nearest	Member with the lowest network latency, regardless of role. Does not guarantee reads stay within a region — latency measurements can change and may cross region boundaries. For strict regional routing, use tags with `readPref('secondary', [{ region: 'us-east' }])` rather than relying on `nearest` alone.

For multi-region deployments, nearest is particularly useful. Users in Europe read from European replicas, users in Asia read from Asian replicas — MongoDB’s server selection algorithm handles this automatically based on measured network latency.

// Route reads to a secondary tagged for a specific region
db.orders.find({ status: "completed" }).readPref("secondary", [{ region: "eu-west" }])

For a detailed breakdown of when each mode is appropriate, refer to read preference use cases.

Sharding for Large Datasets

1. When and why sharding is required

Replica sets solve availability and read scalability, but they do not solve storage capacity or write throughput. Every replica holds a complete copy of your data, so storage scales with the number of nodes rather than across them. Write throughput is still bounded by what the primary can handle alone.

Sharding addresses this by distributing data across multiple replica sets, each holding a subset of the data. You should consider sharding when:

Your dataset is too large for a single server to store efficiently
Write throughput has saturated your primary node
Your working set no longer fits in memory, causing excessive disk I/O

Do not shard prematurely. A well-indexed, well-resourced replica set handles far more load than most applications will ever generate. Sharding adds operational complexity, and reversing it is painful. Only shard when you have clear evidence you need it. Review operational restrictions in sharded clusters before committing.

2. Sharded cluster components

A sharded MongoDB cluster is made up of three types of components:

Shards: Each shard is a replica set that holds a subset of your sharded data. With three shards, each holds roughly one-third of the collection. Because each shard is itself a replica set, you retain fault tolerance at the shard level.

Config servers: A replica set that stores the cluster’s metadata — specifically, the mapping of which chunk ranges live on which shard. Config servers must always be available; if they go down, the cluster cannot route new operations.

mongos: A lightweight query router that sits between your application and the shards. Your application connects to mongos exactly as it would a standalone MongoDB instance — the sharding complexity is invisible. mongos caches routing metadata from the config servers and forwards operations to the appropriate shard or shards.

Application
    │
    ▼
 mongos (query router)
    │
    ├──► Shard 1 (replica set: primary + 2 secondaries)
    ├──► Shard 2 (replica set: primary + 2 secondaries)
    └──► Shard 3 (replica set: primary + 2 secondaries)

Config Servers (replica set)

3. How data is distributed across shards

MongoDB distributes data at the collection level using data partitioning with chunks. When you shard a collection, you specify a shard key — a field or combination of fields that MongoDB uses to divide data into chunks and assign those chunks to shards.

Each chunk covers a contiguous range of shard key values. As data grows, MongoDB’s balancer process monitors chunk distribution and migrates chunks between shards automatically to maintain an even distribution.

Shard Key Design and Performance

1. Importance of shard key selection

Shard key selection is the single most consequential architectural decision in a sharded deployment. The shard key determines how data is distributed across shards, which queries can be routed efficiently, and whether write traffic is balanced or concentrated on a single shard.

Please note that changing a shard key after the fact is extremely disruptive. MongoDB added support for resharding in recent versions, but it is a costly operation. Get this right before you shard.

2. Range vs. hashed shard keys

Range sharding partitions data into contiguous ranges of shard key values. This makes range queries efficient — a query filtering on a date range, for example, can be routed to one or two shards instead of all of them.

The downside is that monotonically increasing shard keys (timestamps, auto-incrementing IDs) cause all new writes to land on the same shard — the one holding the highest range. This creates a write hotspot and defeats the purpose of sharding.

Hashed sharding applies a hash function to the shard key value before distribution. This scatters writes evenly across shards, eliminating hotspots. The trade-off is range queries — because the hash destroys value locality, a range query must be sent to all shards. Note that hashed sharding distributes writes evenly only when the shard key has high cardinality. A hashed boolean field, for example, still concentrates data on two shards at most.

// Range shard key — efficient range queries, risk of write hotspots on monotonic fields
sh.shardCollection("mydb.events", { timestamp: 1 })

// Hashed shard key — even write distribution, no range query efficiency
sh.shardCollection("mydb.events", { _id: "hashed" })

3. Avoiding write hotspots and uneven distribution

A good shard key satisfies three criteria:

High cardinality: Enough distinct values to distribute data across many chunks. A boolean field, for example, can only ever produce two chunks regardless of how many documents you have.
Low frequency: No single value that dominates the dataset. If 40% of your documents share the same shard key value, 40% of your data lands on one shard.
Non-monotonically increasing (for write-heavy workloads): Avoid timestamps or sequential IDs as standalone shard keys. They route all new writes to the maximum chunk, creating a hotspot.

When no single field satisfies all three criteria, a compound shard key can help. Pairing a high-cardinality field with a monotonically increasing field — like { user_id: 1, timestamp: 1 } — can give you even distribution while preserving some query locality.

Query Routing and Scalability

1. Targeted vs. scatter-gather queries

When mongos receives a query, it inspects the filter to determine whether the shard key is included.

Targeted query: The filter includes the shard key. mongos routes the query directly to the one (or occasionally two) shards that hold the relevant data. This is fast and efficient.
Scatter-gather query: The filter does not include the shard key. mongos must broadcast the query to every shard, collect all results, and merge them. Latency is bounded by the slowest shard, and cost scales with the number of shards.

// Targeted — shard key (user_id) is included in the filter
db.orders.find({ user_id: "user_123", status: "pending" })

// Scatter-gather — no shard key, every shard is queried
db.orders.find({ status: "pending" })

This is why your access patterns should drive shard key selection. Before sharding, audit your most frequent and most latency-critical queries. The shard key should align with what those queries filter on.

2. Indexing considerations in sharded environments

Each shard maintains its own independent set of indexes. When you create an index on a sharded collection, MongoDB creates it on every shard. A few rules apply:

Unique indexes must include the shard key as a prefix. MongoDB cannot enforce global uniqueness on a field that isn’t the shard key, because there is no global index spanning all shards. If you need global uniqueness across all shards, the shard key must be a prefix of the unique index. Non-unique indexes have no such restriction.
The shard key is automatically indexed. You do not need to create an index on it separately.
Compound indexes that lead with the shard key field work well for targeted queries and can serve as covered indexes, eliminating document fetches entirely.

3. Impact on latency and throughput

Targeted queries in a sharded cluster can be faster than the equivalent query on a single large server, because each shard operates on a smaller dataset. Scatter-gather queries, however, are slower than on a single server — the overhead of fan-out, parallel execution, and result merging adds latency that doesn’t exist when querying one node.

The practical implication: sharding improves throughput and capacity, but it does not automatically improve query latency. Latency improves only when your queries are targeted. Design your shard key accordingly.

Elasticity and Fault Tolerance

1. Automatic balancing and chunk migration

As data grows and new shards are added, chunks become unevenly distributed. MongoDB’s background balancer detects this and migrates chunks between shards automatically to restore balance.

Chunk migration is a live operation. MongoDB copies data to the destination shard, then atomically updates the routing table in the config servers. Reads and writes continue normally throughout — your application sees no interruption.

For write-heavy workloads, migration traffic can compete with application traffic. You can restrict balancing to a scheduled window during off-peak hours:

use config
db.settings.updateOne(
  { _id: "balancer" },
  { $set: { activeWindow: { start: "02:00", stop: "06:00" } } },
  { upsert: true }
)

2. Scaling out with minimal downtime

Adding a new shard to a running cluster is an online operation. You provision the new replica set, register it with sh.addShard(), and the balancer begins migrating chunks to it automatically. No application configuration changes are needed — mongos picks up the updated routing table from the config servers.

sh.addShard("rs3/mongodb-shard3-1:27017,mongodb-shard3-2:27017,mongodb-shard3-3:27017")

If your access patterns change significantly and your current shard key is no longer appropriate, MongoDB’s resharding feature lets you migrate to a new shard key without rebuilding the cluster from scratch — a significant improvement over earlier versions of MongoDB, where the shard key was effectively permanent.

3. High availability through replica sets

Every layer of a sharded cluster is replicated:

Each shard is a replica set. If a shard’s primary fails, its replica set holds an automatic election and promotes a secondary within seconds.
The config servers are a replica set. Losing one config server does not take the cluster offline.
mongos instances are stateless. You can run multiple mongos instances behind a load balancer, and losing one affects only the connections it was actively handling.

There is no single point of failure at any layer of the architecture. For a detailed look at failure scenarios and how MongoDB handles each one, refer to sharded cluster high availability.

Summary and Next Steps

MongoDB’s scaling approach is layered. Replica sets provide read scalability and fault tolerance as a baseline. Sharding extends this to write throughput and storage capacity. mongos keeps the distributed complexity invisible to application code, and automatic balancing ensures the cluster adapts to growth without manual intervention.

Key takeaways for designing scalable MongoDB workloads:

Do not shard until you have evidence you need to. A well-tuned replica set handles more than most applications require. Shard only when you’ve identified a clear bottleneck that sharding solves.
Let your access patterns drive shard key selection. The queries you run most frequently should determine the shard key — not your schema or your data model in isolation.
Use hashed sharding for write-heavy workloads, range sharding when range queries matter. Know which trade-off your application can live with before you commit.
Every scatter-gather query is a cost that scales with your cluster size. Design for targeted queries wherever possible.
Use read preferences deliberately. Secondary reads are a useful tool for the right workloads, but they introduce replication lag — make sure that trade-off is acceptable for the feature using them.

For hands-on practice, MongoDB University offers free courses on sharding and replica set administration. The official sharding documentation is also one of the more thorough references available for production deployment considerations.

MongoDB Schema Design: Do’s and Don’ts for Real Projects

MongoDB Guests — Wed, 25 Mar 2026 09:50:36 +0000

This tutorial was written by Nancy Agarwal.

Understanding MongoDB Schema Design

If you have worked with MongoDB, you have probably heard someone say, "MongoDB is schema-less — just store whatever JSON you want." This is one of the biggest misconceptions.

MongoDB is not schema-less — it is schema-flexible. That flexibility is powerful, but it also means developers must take responsibility for good schema design.

When schema design is done correctly:

Queries become extremely fast
APIs stay simple
Applications scale smoothly

When schema design is ignored:

Queries become slow
Documents grow bloated
Updates become difficult
Systems become harder to maintain

MongoDB Schema Design Do's

Model for Access Patterns, Not Entities

Relational databases typically start with entities and relationships. MongoDB flips this approach. Instead of starting with tables, you should start with how your application accesses data.

Ask yourself:

What are the most common queries?
What data needs to be fetched together?
What operations happen most frequently?

Example

In one HR system, employee data was always fetched along with department and manager information. Instead of performing joins with $lookup, we used the Extended Reference Pattern.

Key manager fields were embedded inside the employee document, while full manager details were stored separately.

This allowed most reads to happen in a single document query, keeping performance extremely fast.

Patterns to Consider

Extended Reference Pattern

Duplicate important parent fields in child documents to avoid joins.

Pre-Compute Pattern

Store computed values such as totals, aggregates, or derived metrics so they do not need to be recalculated repeatedly.

Pro Tip: Before finalizing your schema, write down the top three queries your application performs. If your schema cannot serve them efficiently, redesign it.

Use the Pre-Compute Pattern for Expensive Aggregations

MongoDB’s aggregation framework is powerful, but running heavy aggregations repeatedly can become expensive at scale.

Example

In one analytics system, every dashboard load executed several $group aggregations on a 50 million record collection.

The result: query latency of over 3 seconds.

We solved it using the Pre-Compute Pattern. Aggregations were computed nightly and stored in a collection like dailyUserMetrics.
Dashboards then simply read the pre-aggregated results, reducing latency to around 80 milliseconds.

Use this pattern for:

Dashboards
Leaderboards
Analytics reports

Use the Polymorphic Pattern for Flexible Data Types

Sometimes documents share a common structure but contain different fields depending on their type.

Example CRM activities documents:
{ type: "email", subject: "...", recipients: [...] }
{ type: "call", duration: 180, recordingUrl: "..." }
{ type: "meeting", participants: [...], location: "..." }

Each document shares common metadata like timestamps or userId, but contains type-specific attributes.

Use the Bucket Pattern for Time-Series Data

Applications storing logs, telemetry, or IoT events often generate millions of small records.

The Bucket Pattern groups multiple readings into a single document.

Example:
{
deviceId: "D123",
startTime: ISODate("2025-10-29T10:00:00Z"),
readings: [
{ t: 0, value: 1.02 },
{ t: 1, value: 1.05 }
]
}

This dramatically reduces document count and improves query performance.

Version and Evolve Your Schema

Schemas evolve as products grow. Ignoring schema evolution leads to inconsistent documents and fragile code.

Example older invoice:
{ amount: 100, currency: "USD" }

Example newer invoice:
{ amount: 100, currency: "USD", tax: 10, discount: 5, schemaVersion: 2 }

MongoDB Schema Design Don’ts

Don’t Let Arrays Grow Unbounded

Unbounded arrays are a common MongoDB anti-pattern. When arrays grow indefinitely, documents become large and slow.

In one system, all user sessions were stored inside a user document. Some users accumulated over 50,000 sessions, causing slow queries.

A better approach was to store sessions in a separate collection referenced by userId.

Don’t Treat MongoDB Like a Relational Database

Over-normalizing data leads to excessive joins using $lookup.

Recommended approach:

Embed when relationships are tight
Reference when data is large or independent
Denormalize intentionally

Don’t Over-Index or Over-Engineer

Indexes improve reads but increase memory usage and slow writes.

Best practices:

Only index fields used by queries
Periodically review unused indexes
Avoid excessive collections

Final Takeaways

Design around access patterns, not tables.
Use the right schema patterns - Extended Reference, Pre-Compute, 3. Polymorphic, Bucket - to shape data for performance.
Never let arrays grow unbounded.
Monitor document size and index efficiency.
Version your schema - evolution is inevitable.

Conclusion

MongoDB’s schema flexibility is powerful but requires thoughtful design. By applying these patterns and avoiding common anti-patterns, teams can build MongoDB systems that remain performant as data grows.

If you found this useful, I'd love to hear how you're modeling your MongoDB schemas in production - or what anti-patterns you've encountered. Let's trade war stories in the comments.

How to Use MongoDB’s Text Search

MongoDB Guests — Fri, 20 Mar 2026 10:27:56 +0000

This tutorial was written by Damilola Oladele.

How to Use MongoDB's Text Search

MongoDB Search is an embedded full-text search engine that lives inside your MongoDB deployment. You don't need to run a separate search system alongside your database. You get fine-grained text indexing, a rich query language for complex search logic, customizable relevance scoring, and advanced features like autocomplete and faceted search.

This tutorial shows you how to insert a sample book catalog into MongoDB, create a search index, and run six types of text search queries in Python.

You'll learn the following in this tutorial:

What search queries, search indexes, and search analyzers are in MongoDB
How to insert sample data and create a search index
How to run a basic text search
How to search across multiple fields
How to search with fuzzy matching
How to search for a term while excluding another word
How to combine search conditions with the compound operator
How to include relevance scores in your results

You can find all the code samples for this tutorial in the GitHub repository.

What You Need to Know Before You Start

Three concepts are central to text search in MongoDB. They'll help you understand why each search operation behaves the way it does.

Search queries

A search query asks MongoDB to find documents that are relevant to a term or phrase. Search queries are different from traditional database queries. A traditional database query follows a strict syntax and returns exact matches. On the other hand, a search query can match similar phrases, tolerate typos, support wildcards, and rank results by how relevant they are.

MongoDB Search queries take the form of an aggregation pipeline stage. The $search stage returns the actual search results. The $searchMeta stage returns metadata about the results, such as counts or facet data. Both stages must come first in your aggregation pipeline. You then chain additional stages, like $project or $limit, after them.

Search indexes

A search index is a data structure that maps terms to the documents that contain them. Think of it like the index at the back of a book. Instead of scanning every page to find a topic, you look it up in the index and go straight to the right pages. A search index works the same way. MongoDB consults it instead of scanning every document in your collection, thereby improving search speed.

You must create a search index before you can run any $search queries.

Search analyzers and tokens

An analyzer controls how MongoDB transforms your text into searchable units called tokens. This transformation happens both when MongoDB builds the index and when it processes your query. The analyzer you choose determines which tokens end up in the index and which tokens your query produces.

MongoDB Search ships with five built-in analyzers. The one you choose shapes what your search can and can't find.

Analyzer	How it splits text	Lowercase	Stop words
Standard	Whitespace and punctuation	Yes	Yes
Simple	Any non-letter character	Yes	No
Whitespace	Whitespace only	No	No
Keyword	Treats the whole field as one token	No	No
Language	Language-specific rules	Varies	Varies

If you don't specify an analyzer, MongoDB uses the Standard analyzer. Later in this tutorial, you'll see how different analyzers change what a query finds and why.

Now let's get started.

Prerequisites

Before you start, make sure you have the following ready:

Python 3.8 or later installed
A MongoDB Atlas account with an M0 (Free tier) cluster set up
pymongo 4.0 or later installed
Basic familiarity with Python and MongoDB collections
Your Atlas connection string, available in the Atlas UI under Database > Connect > Drivers

Make sure your IP address is allowlisted in Atlas before you run any scripts. Go to Security > Network Access in the Atlas UI and add your current IP address. Your scripts won't connect to your cluster without this step.

Set Up Your Project

Create a folder for your project and navigate into it in your terminal:

mkdir mongodb-text-search
cd mongodb-text-search

Create and activate a virtual environment:

python -m venv venv
source venv/bin/activate

On Windows, activate it with:

venv\Scripts\activate

Install pymongo:

python -m pip install pymongo

You'll create a separate Python file for each step of this tutorial. All files go in the mongodb-text-search folder.

Insert Your Data and Create a Search Index

Create a file named insert_data.py. This file does two things:

It inserts a sample book catalog into your cluster
It creates the search index you'll use for most of the tutorial.

You only need to run this file once. The other files in this tutorial just connect and query.

from pymongo import MongoClient
import time

client = MongoClient(
    "mongodb+srv://<USERNAME>:<PASSWORD>@<HOST>/",
    appname="devrel-tutorial-python-search"
)
db = client["bookstore"]
books = db["books"]

# Drop the collection to start clean on each run:
books.drop()

books.insert_many([
    {
        "title": "The Midnight Library",
        "author": "Matt Haig",
        "genre": "fiction",
        "description": "A woman discovers a library between life and death that contains books representing all the lives she could have lived.",
        "year": 2020
    },
    {
        "title": "Atomic Habits",
        "author": "James Clear",
        "genre": "self-help",
        "description": "A practical guide to building good habits and breaking bad ones through small, incremental changes.",
        "year": 2018
    },
    {
        "title": "Project Hail Mary",
        "author": "Andy Weir",
        "genre": "science fiction",
        "description": "An astronaut wakes up alone in space with no memory and must figure out how to save Earth from an extinction-level threat.",
        "year": 2021
    },
    {
        "title": "Educated",
        "author": "Tara Westover",
        "genre": "memoir",
        "description": "A woman reflects on her isolated rural childhood and her journey to educate herself and earn a PhD from Cambridge.",
        "year": 2018
    },
    {
        "title": "The Hitchhiker's Guide to the Galaxy",
        "author": "Douglas Adams",
        "genre": "science fiction",
        "description": "An ordinary man is swept across the universe after Earth is demolished to make way for a hyperspace expressway.",
        "year": 1979
    }
])

print("Sample data inserted.")

# Drop any existing search index named 'default' before creating a new one:
try:
    books.drop_search_index("default")
    time.sleep(5)
except Exception:
    pass

# Create a search index with dynamic mappings:
books.create_search_index({
    "name": "default",
    "definition": {
        "mappings": {
            "dynamic": True
        }
    }
})

print("Search index created. Waiting for it to go active...")

# Poll until the index is ready:
while True:
    indexes = list(books.list_search_indexes())
    if indexes and indexes[0].get("status") == "READY":
        print("Search index is active. You're ready to run queries.")
        break
    time.sleep(3)

Replace <USERNAME>, <PASSWORD>, and <HOST> with your actual Atlas credentials.

The dynamic: True setting tells MongoDB to automatically index all supported string fields in your documents. The while loop polls the index status and pauses the script until the index is ready.

Run the script:

python insert_data.py

You'll get the following output in your terminal:

Sample data inserted.
Search index created. Waiting for it to go active...
Search index is active. You're ready to run queries.

Run a Basic Text Search

A basic text search finds documents where a specific field contains your query term. The text operator handles this. You set query to the term you want to find and path to the field you want to search.

Create a file named basic_search.py:

from pymongo import MongoClient

# Replace the placeholder values with your Atlas credentials and cluster hostname:
client = MongoClient(
    "mongodb+srv://<USERNAME>:<PASSWORD>@<HOST>/",
    appname="devrel-tutorial-python-search"
)
db = client["bookstore"]
books = db["books"]

# Search the description field for the term "space":
results = books.aggregate([
    {
        "$search": {
            "text": {
                "query": "space",
                "path": "description"
            }
        }
    },
    {
        # Return title, author, and description — suppress _id:
        "$project": {
            "_id": 0,
            "title": 1,
            "author": 1,
            "description": 1
        }
    }
])

for doc in results:
    print(doc)

Run the script:

python basic_search.py

You'll get the following output:

{
  "title": "Project Hail Mary",
  "author": "Andy Weir",
  "description": "An astronaut wakes up alone in space with no memory and must figure out how to save Earth from an extinction-level threat."
}

Only "Project Hail Mary" matches because "space" appears there as a standalone word. "The Hitchhiker's Guide to the Galaxy" has the word "hyperspace" in its description, but the Standard analyzer splits text on whitespace and punctuation only. “hyperspace” is tokenized as a single word, not broken into “hyper” and “space”. For “space” to match “hyperspace”, you'd need a different approach, like using the wildcard operator or a custom analyzer with edge n-grams.

Search Across Multiple Fields

You can pass a list of field names to path. MongoDB searches all listed fields and returns any document where your query term appears in at least one of them.

Create a file named multi_field_search.py:

from pymongo import MongoClient

# Replace the placeholder values with your Atlas credentials and cluster hostname:
client = MongoClient(
    "mongodb+srv://<USERNAME>:<PASSWORD>@<HOST>/",
    appname="devrel-tutorial-python-search"
)
db = client["bookstore"]
books = db["books"]

# Search title, description, and genre for the term "fiction":
results = books.aggregate([
    {
        "$search": {
            "text": {
                "query": "fiction",
                "path": ["title", "description", "genre"]
            }
        }
    },
    {
        # Return title, genre, and description — suppress _id:
        "$project": {
            "_id": 0,
            "title": 1,
            "genre": 1,
            "description": 1
        }
    }
])

for doc in results:
    print(doc)

Run the script:

python multi_field_search.py

You'll get the following output:

{
  "title": "The Midnight Library",
  "genre": "fiction",
  "description": "A woman discovers a library between life and death that contains books representing all the lives she could have lived."
},
{
  "title": "The Hitchhiker's Guide to the Galaxy",
  "genre": "science fiction",
  "description": "An ordinary man is swept across the universe after Earth is demolished to make way for a hyperspace expressway."
},
{
  "title": "Project Hail Mary",
  "genre": "science fiction",
  "description": "An astronaut wakes up alone in space with no memory and must figure out how to save Earth from an extinction-level threat."
}

All three books match because "fiction" appears as a token in each of their genre fields. The Standard analyzer tokenizes "science fiction" into two separate tokens: "science" and "fiction". So the two science fiction books match on the token "fiction", and "The Midnight Library" matches because its genre is exactly "fiction".

Search with Fuzzy Matching

A fuzzy search finds documents that contain terms similar to your query. This is useful when users make typos or spelling mistakes.

Create a file named fuzzy_search.py:

from pymongo import MongoClient

# Replace the placeholder values with your Atlas credentials and cluster hostname:
client = MongoClient(
    "mongodb+srv://<USERNAME>:<PASSWORD>@<HOST>/",
    appname="devrel-tutorial-python-search"
)
db = client["bookstore"]
books = db["books"]

# Search for terms similar to "librery" — allow up to one character edit:
results = books.aggregate([
    {
        "$search": {
            "text": {
                "query": "librery",
                "path": "description",
                "fuzzy": {
                    # maxEdits controls how many character changes MongoDB tolerates:
                    "maxEdits": 1
                }
            }
        }
    },
    {
        # Return title and description — suppress _id:
        "$project": {
            "_id": 0,
            "title": 1,
            "description": 1
        }
    }
])

for doc in results:
    print(doc)

Run the script:

python fuzzy_search.py

You'll get the following output:

{
  "title": "The Midnight Library",
  "description": "A woman discovers a library between life and death that contains books representing all the lives she could have lived."
}

The maxEdits value controls how many character changes MongoDB tolerates. A value of 1 means MongoDB matches terms that differ from your query by one insertion, deletion, or substitution. The misspelled "librery" is one character away from "library", so the match succeeds.

Search for a Term While Excluding Another Word

Sometimes you want to find documents that match one term but don't contain another. This is different from a basic text search because you need two conditions to work together:

a positive match
an exclusion

The compound operator handles this through its must and mustNot clauses.

must requires the document to match a condition to appear in the results. mustNot requires the document not to match a condition to appear in the results. A document only returns if it satisfies both at once.

Create a file named exclude_term_search.py:

from pymongo import MongoClient

# Replace the placeholder values with your Atlas credentials and cluster hostname:
client = MongoClient(
    "mongodb+srv://<USERNAME>:<PASSWORD>@<HOST>/",
    appname="devrel-tutorial-python-search"
)
db = client["bookstore"]
books = db["books"]

results = books.aggregate([
    {
        "$search": {
            "compound": {
                # Must: the description field must contain "woman":
                "must": [
                    {
                        "text": {
                            "query": "woman",
                            "path": "description"
                        }
                    }
                ],
                # MustNot: exclude any book whose description contains "death":
                "mustNot": [
                    {
                        "text": {
                            "query": "death",
                            "path": "description"
                        }
                    }
                ]
            }
        }
    },
    {
        # Return title and description — suppress _id:
        "$project": {
            "_id": 0,
            "title": 1,
            "description": 1
        }
    }
])

for doc in results:
    print(doc)

Run the script:

python exclude_term_search.py

You'll get the following output:

{
  "title": "Educated",
  "description": "A woman reflects on her isolated rural childhood and her journey to educate herself and earn a PhD from Cambridge."
}

Two books in the collection have "woman" in their description: "The Midnight Library" and "Educated". The query returns only "Educated" because "The Midnight Library" contains the word "death", which the mustNot clause blocks.

A mustNot clause doesn't contribute to the relevance score of any document. It acts purely as a filter that removes documents from the result set. A document either passes or fails the exclusion check, and that outcome doesn't change how MongoDB ranks the documents that do pass.

Combine Conditions with the Compound Operator

The compound operator lets you combine multiple search conditions in a single query. It supports four clauses:

must: the document must match for it to appear in the results. Maps to AND.
mustNot: the document must not match for it to appear in the results. Maps to AND NOT.
should: documents that match rank higher in the results. Maps to OR.
filter: the document must match, but matching doesn't affect the relevance score.

Create a file named compound_search.py:

from pymongo import MongoClient

client = MongoClient(
    "mongodb+srv://<USERNAME>:<PASSWORD>@<HOST>/",
    appname="devrel-tutorial-python-search"
)
db = client["bookstore"]
books = db["books"]

results = books.aggregate([
    {
        "$search": {
            "compound": {
                # Must: the genre field must contain the token "fiction"
                "must": [
                    {
                        "text": {
                            "query": "fiction",
                            "path": "genre"
                        }
                    }
                ],
                # MustNot: exclude any book whose description mentions "astronaut"
                "mustNot": [
                    {
                        "text": {
                            "query": "astronaut",
                            "path": "description"
                        }
                    }
                ],
                # Should: boost books whose description mentions "library"
                "should": [
                    {
                        "text": {
                            "query": "library",
                            "path": "description"
                        }
                    }
                ],
                # Filter: only include books published in 1979 or later (no score impact)
                "filter": [
                    {
                        "range": {
                            "path": "year",
                            "gte": 1979
                        }
                    }
                ]
            }
        }
    },
    {
        "$project": {
            "_id": 0,
            "title": 1,
            "genre": 1,
            "description": 1,
            "year": 1,
        }
    }
])

for doc in results:
    print(doc)

Run it:

python compound_search.py

You'll get the following output:

{
  "title": "The Midnight Library",
  "genre": "fiction",
  "description": "A woman discovers a library between life and death that contains books representing all the lives she could have lived.",
  "year": 2020
},
{
  "title": "The Hitchhiker's Guide to the Galaxy",
  "genre": "science fiction",
  "description": "An ordinary man is swept across the universe after Earth is demolished to make way for a hyperspace expressway.",
  "year": 1979
}

The query works in four steps. The must clause narrows the candidate set to all books whose genre contains the token "fiction": "The Midnight Library" (genre: "fiction"), "Project Hail Mary" (genre: "science fiction"), and "The Hitchhiker's Guide to the Galaxy" (genre: "science fiction"). The mustNot clause then removes "Project Hail Mary" because its description contains "astronaut". The filter clause confirms that both remaining books were published in 1979 or later without adding any score. The should clause boosts "The Midnight Library" because its description contains "library", which pushes it to the top of the results.

Include Relevance Scores in Your Results

Every document in a $search result gets a relevance score. A higher score means the document is a closer match to your query. Documents score higher when the query term appears often in that document, and lower when the same term appears across many documents in the collection.

You include scores in results by adding a $project stage with the {"$meta": "searchScore"} expression. To see ranking in action, you need a query that returns more than one document. Searching for "woman" matches two books and lets you compare how MongoDB scores them against each other.

Create a file named search_with_scores.py:

from pymongo import MongoClient

# Replace the placeholder values with your Atlas credentials and cluster hostname:
client = MongoClient(
    "mongodb+srv://<USERNAME>:<PASSWORD>@<HOST>/",
    appname="devrel-tutorial-python-search"
)
db = client["bookstore"]
books = db["books"]

# Search the description field for the term "woman":
results = books.aggregate([
    {
        "$search": {
            "text": {
                "query": "woman",
                "path": "description"
            }
        }
    },
    {
        # Include the relevance score alongside each document:
        "$project": {
            "_id": 0,
            "title": 1,
            "description": 1,
            "score": {"$meta": "searchScore"}
        }
    },
    {
        # Return results from highest to lowest score:
        "$sort": {"score": -1}
    }
])

for doc in results:
    print(doc)

Run it:

python search_with_scores.py

You'll get the following output:

{
  "title": "The Midnight Library",
  "description": "A woman discovers a library between life and death that contains books representing all the lives she could have lived.",
  "score": 0.39296838641166687
}
{
  "title": "Educated",
  "description": "A woman reflects on her isolated rural childhood and her journey to educate herself and earn a PhD from Cambridge.",
  "score": 0.39296838641166687
}

Both books contain the word "woman" in their descriptions, so both return. The scores are equal because "woman" appears exactly once in each description and both descriptions are similar in length. This is the expected behavior when term frequency and document length are similar across matching documents.

The exact score values you see may differ from another cluster's output. Relevance scores depend on term frequency and document count at index time, which can vary between deployments. What matters is the relative ranking across results, not the absolute numbers.

The $sort stage makes the ranking explicit. Without it, MongoDB Search still returns results from highest to lowest score by default, but the score field lets you confirm and reason about the order.

Key Takeaways

MongoDB Search is embedded in your cluster. You don't need a separate search system.
You must create a search index before running any $search query. If the index doesn't exist, MongoDB returns an empty result set without an error.
The $search stage must come first in your aggregation pipeline. You chain other stages, like $project, $sort, and $limit, after it.
The Standard analyzer tokenizes multi-word strings into separate tokens. "science fiction" becomes two tokens: "science" and "fiction". This affects what matches your queries' return.
Compound words like "hyperspace" stay as a single token. A search for "space" doesn't match "hyperspace".
The compound operator lets you combine must, mustNot, should, and filter clauses in a single query. A mustNot clause acts as a pure filter and doesn't affect relevance scores.
Every $search result gets a relevance score. Documents score higher when the query term appears often in that document and lower when the same term appears across many documents in the collection.

All the code samples from this tutorial are available in the GitHub repository.

Using C# Record types with MongoDB

MongoDB Guests — Thu, 12 Feb 2026 16:42:28 +0000

This guest author article was written by Markus Wildgruber - Managing Director/Cloud Architect, sevantage software GmbH

When C# 9.0 was introduced a while back, the main focus was better support for immutable types. This programming pattern strives to enhance support for concurrent execution and make code easier to understand due to reduced chance of side effects. Though you could define immutable types before manually, C# 9.0 introduced the record keyword that allows for easy definition of immutable types. This piece of "syntactic sugar" also adds the following capabilities to a type:

record types add an implementation of IEquatable<T> and various other methods that handle value equality - so no more manual implementation of these methods is necessary. Even if you seldom compare MongoDB documents by all of their properties in application code, unit tests are much easier to read if you can compare objects as a whole instead of checking each property value individually.
Also, record types provide an implementation of GetHashCode based on the property values. As most record types use only positional parameters that are immutable, the hash code also cannot change after initialization of the object. This is especially important if you use the object as a key in a dictionary or add it to a hash set. This rules out a kind of bug that is very hard to track down: not finding items in a collection anymore due to mismatches between hash code and object values.
The ToString method of a record returns a meaningful display of the object properties - a small change that helps a lot when debugging!

For a full overview of the characteristics of a record type, please read the documentation on the Microsoft website.

Defining a record POCO

One additional upside of the definition of a record over a traditional class type is that it is so much shorter:

using MongoDB.Bson;

public record Movie(
    ObjectId Id, 
    string Title, 
    string Plot
);

As shown in the sample above, most record types use a primary constructor that lists the properties directly in the parentheses. The compiler will change this definition to a class similar to the following:

using MongoDB.Bson;

public class Movie : IEquatable<Movie>
{
    public Movie(ObjectId id, string title, string plot) 
    {
        Id = id;
        Title = title;
        Plot = plot;
    }

    public ObjectId Id { get; init; }
    public string Title { get; init; }
    public string Plot { get; init; }

    // ... 
}

Declarative Mapping

When following this approach, be aware that you have to adapt the declarative mapping of MongoDB attributes so that they are applied to the properties after the compiler has changed the structure.

For instance, if you want to use string as the type of the Id property (but have an ObjectId in MongoDB after mapping) and change the properties in MongoDB, you need to apply the property attribute target specifier:

using MongoDB.Bson;
using MongoDB.Bson.Serialization.Attributes;

[BsonIgnoreExtraElements]
public record Movie(
    [property: BsonRepresentation(BsonType.ObjectId)] string Id,
    [property: BsonElement("title")] string Title,
    [property: BsonElement("plot")] string Plot
);

Manual Mapping

If you prefer to create the class map manually (see the MongoDB C# Driver documentation for details), this is also supported:

BsonClassMap.RegisterClassMap<Movie>(classMap =>
{
    classMap.SetIgnoreExtraElements(true);
    classMap.MapMember(p => p.Id).SetSerializer(new StringSerializer(MongoDB.Bson.BsonType.ObjectId));
    classMap.MapMember(p => p.Title).SetElementName("title");
    classMap.MapMember(p => p.Plot).SetElementName("plot");
});

Putting it all together

The following file-based C# app shows that no changes are needed when accessing data from MongoDB that is based upon a record type:

#:package MongoDB.Driver@*
using MongoDB.Bson;
using MongoDB.Bson.Serialization.Attributes;
using MongoDB.Driver;

string connStr = "mongodb://localhost:27017";
string databaseName = "sample_mflix";
int limit = 5;
IMongoClient client = new MongoClient(connStr);
IMongoDatabase database = client.GetDatabase(databaseName);
IMongoCollection<Movie> moviesCollection = database.GetCollection<Movie>("movies");

foreach (Movie movie in moviesCollection.Find(FilterDefinition<Movie>.Empty).Limit(limit).ToEnumerable())
{
    Console.WriteLine(movie);
}

[BsonIgnoreExtraElements]
public record Movie(
    [property: BsonRepresentation(BsonType.ObjectId)] string Id,
    [property: BsonElement("title")] string Title,
    [property: BsonElement("plot")] string Plot
);

Bottom line & next steps

Records make the model layer of a MongoDB‑based C# application both leaner and safer. They give you immutable value semantics for free, remove boilerplate code for equality checking and add a ToString implementation. Be sure to use the property attribute target to apply MongoDB serialization attributes to properties.

Give it a try – replace a few of your class‑based POCOs with records. We love it when you share your experience – if you hit a pitfall or discover a pattern that worked well, drop a comment so we can all learn.

Happy coding!

Optimizing the MongoDB Java Driver: How minor optimizations led to macro gains

MongoDB Guests — Wed, 11 Feb 2026 20:21:08 +0000

This tutorial was written by Slav Babanin & Nasir Qureshi.

Donald Knuth, widely recognized as the ‘father of the analysis of algorithms,’ warned against premature optimization—spending effort on code that appears inefficient but is not on the critical path. He observed that programmers often focus on the wrong 97% of the codebase. Real performance gains come from identifying and optimizing the critical 3%. But, how can you identify the critical 3%? Well, that’s where the philosophy of ‘never guess, always measure’ comes in.

In this blog, we share how the Java developer experience team optimized the MongoDB Java Driver by strictly adhering to this principle. We discovered that performance issues were rarely where we thought they were. This post explains how we achieved throughput improvements between 20% to over 90% in specific workloads. We’ll cover specific techniques, including using SWAR (SIMD Within A Register) for null-terminator detection, caching BSON array indexes, and eliminating redundant invariant checks.

These are the lessons we learned turning micro-optimizations into macro-gains. Our findings might surprise you — they certainly surprised us — so we encourage you to read until the end.

Getting the metrics right

Development teams often assume they know where bottlenecks are, but intuition is rarely dependable. During this exercise, the MongoDB Java team discovered that performance problems were often not where the team expected them to be.

Donald Knuth emphasizes this concept in his paper on Premature Optimization:

Programmers waste enormous amounts of time thinking about the speed of noncritical parts of their programs, and these attempts to improve efficiency have a strong negative impact on debugging and maintenance. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.

To avoid ‘premature optimization’—that is, improving code that appears slow but isn't on the critical path—we follow a strict rule: never guess, always measure.

We applied the Pareto principle (also known as the 80/20 rule) to target the specific code paths responsible for the majority of execution time. For this analysis, we used async-profiler. Its low-overhead, sampling-based approach allowed us to capture actionable CPU and memory profiles with negligible performance impact.

How we measured performance

We standardized performance tests based on throughput (MB/s), simplifying comparisons across all scenarios. Our methodology focused on minimizing the influence of external variables and ensuring practical relevance.

Testing Methodology:

Tested representative workloads: Testing focused on representative driver operations (for example, small, large, and bulk document inserts) executed via the MongoClient API, not isolated method microbenchmarks.
Isolated the testing environment: We conducted performance tests across multiple isolated cloud machines to minimize external variability and prevent performance degradation from resource contention (i.e., 'noisy neighbors'). Each test was run multiple times on each machine, and the median throughput score was used as the final result for that machine.
Statistical verification: Next, we aggregated the median results and directly compared the optimized branch mean throughput with the stable mainline mean. We verified statistical significance through percentage improvement and z-score analysis.

From this exercise, we realized that what truly mattered was that the performance improvements appear at the MongoClient API level. Internal microbenchmarks may show significant gains, but since users interact solely through the API, any gains that do not manifest at the API level will not translate into noticeable improvements in application performance.

Refer to the Performance Benchmarking drivers specification for a more detailed description of these tests.

Below, we will explain the six techniques we used to optimize the MongoDB Java driver, while staying true to our guiding principle: ‘never guess, always measure’.

1. Caching BSON array indexes

In BSON, array indexes are not merely integers; they are encoded as null-terminated strings, or CStrings. For example, the index 0 becomes the UTF-8 byte sequence for '0' (U+0030) followed by the UTF-8 byte sequence for NULL (U+0000). Encoding an array involves converting each numeric index to a string, encoding it into UTF-8 bytes, and then appending a null terminator:

for (int i = 0; i < arraySize; i++) {
    encodeCString(Integer.toString(i));;
   }

Calling toString() and encoding the result for every index was clearly suboptimal, because it repeats the same conversion work for the same indexes over and over again: each call rebuilds the String representation of i and then turns that String into a byte[]. This involves unnecessary copying, even though the result remains the same.

Our first improvement was to precompute and cache these immutable byte arrays for reuse in tight loops.

private static final byte[][] PRE_ENCODED_INDICES = new byte[1000][];

for (int i = 0; i < 1000; i++) {
    PRE_ENCODED_INDICES[i] = (Integer.toString(i) + '\u0000').getBytes("UTF-8");
}

for (int i = 0; i < arraySize; i++) {
    if (i < PRE_ENCODED_INDICES.length) {
        buffer.put(PRE_ENCODED_INDICES[i]);
    } else {
        encodeCString(Integer.toString(i));
    }
}

This caching step was already effective. We also changed the cache layout from a jagged byte[] to a flat byte[] representation.

private static final byte[] PRE_ENCODED_INDICES;

The flat byte[] layout remains the preferred choice because it uses fewer heap objects and scales more efficiently as the cache grows due to spatial locality. Our benchmarks showed no significant throughput difference compared to jagged byte[][] structures in smaller caches; this parity stems from the HotSpot’s Thread-Local Allocation Buffer (TLAB) allocator, which places small rows close together in memory. Even with garbage collection (GC) settings that forced frequent promotion into the old generation, we often observed the same effect. Because that behaviour is JVM- and GC-specific rather than guaranteed, we use the flat array as the more robust solution. To quantify the impact of caching itself, we adapted the “Small Doc insertOne” workload from our performance specification for array-heavy documents. Instead of the original shape, each document now contained A arrays of B length (that is, 100×100, and 100×1000), so the total number of encoded array indexes per document was A × B. The larger the arrays we use per document, the more significant the difference is, as the array encoding fraction in the “insertOne” operation is larger for larger arrays.

Figure 1. The figure below shows the before-and-after results of performance tests on 100x100 and 100x1000 array documents. The larger arrays saw the greatest improvement in performance.

2. Java Virtual Machine (JVM) intrinsics

As Java developers, we write code with many abstractions, which makes the code easier to maintain and understand; however, these abstractions can also cause significant performance issues. What is easy for humans to read isn’t always what the machine prefers to execute, which is why having mechanical sympathy may be beneficial.

For example, in BSON, numbers must be encoded with little-endian order. In our original code, when encoding an int to BSON, we wrote each byte to a ByteBuffer separately, doing manual shifting and masking to produce little-endian byte order.

write(value >> 0);

write(value >> 8);

write(value >> 16);

write(value >> 24);

However, this approach wasn’t efficient. It required individual bounds checks and manual byte shuffling for every byte written, which showed up as a hotspot in profiles. We chose to adopt ByteBuffer’s methods—such as putInt, putLong, and putDouble. This method collapses four separate byte operations into a single call (putInt) that handles byte order automatically.

Under the hood, modern JITs (e.g., in HotSpot) can compile these methods using intrinsics—such as Unsafe.putInt and Integer.reverseBytes—often mapping them to efficient machine instructions. For more context, see Intrinsic functions.

The JVM can replace these helper methods with a very small sequence of machine instructions, sometimes even a single instruction. For example, on x86, the Integer.reverseBytes(int i) method may use the BSWAP instruction; on ARM, the JVM might use the REV instruction.

Bonus tip: Repeated invariant checks in the hot path are computationally expensive. In the original code of the BsonOutput serializer, every single-byte write also re-checks invariants, such as “is this BsonOutput still open?” If you’ve already validated invariants elsewhere, you can safely skip repeated invariant checks inside the loop.

After implementing our changes and testing them, we realized that a simple change affecting less than 0.03% of the codebase resulted in a nearly 39% improvement in throughput for large document bulk inserts.

Figure 2. The figure below shows throughput improvements for each insert command. ‘Large Doc Bulk Insert’ saw the most significant gain because processing larger payloads maximizes the impact of optimizing the hottest execution paths.

3. Check and check again

As mentioned earlier, size checks on ByteBuffer in the critical path are expensive. However, we also performed similar checks on invariants in many other places. When encoding BSON, we retrieved the current buffer from a list by index on every write:

ByteBuffer currentBuffer = bufferList.get(currentBufferIndex);

//other code
currentBuffer.put(value);

This get() call is fast, but performing it many times adds up—especially since each call includes range checks and method indirection (as the path is too deep; the JVM might not always inline it).

Aha moment: If the same buffer will be used for thousands of operations before switching, why should we keep re-checking?

By caching the buffer in a field and updating it only when switching buffers, we eliminated at least three redundant range checks. Here is how:

private ByteBuffer currentByteBuffer;// Only update when changing bufferscurrentByteBuffer.put(value);

This minor change led to a 16% increase in throughput for bulk inserts. This wasn’t the only area where redundant checks could be eliminated; when we tested removing similar invariant checks from other abstractions, we observed an additional 15% improvement in throughput for bulk inserts.

The lesson: Remove unnecessary checks from the hottest paths. Because these checks run so frequently, they quickly become bottlenecks that drag down performance.

4. BSON null terminator detection with SWAR

Every BSON document is structured as a list of triplets: a type byte, a field name, and a value. Crucially, each field name is a null-terminated string—CString—not a length-prefixed string. While this design saves four bytes per field, it introduces a performance trade-off: extracting a CString now requires a linear scan rather than a constant-time lookup.

Our original implementation processed the buffer byte-by-byte, searching for the terminating zero:

boolean checkNext = true;

while (checkNext) {
    if (!buffer.hasRemaining()) {
        throw new BsonSerializationException("Found a BSON string that is not null-terminated");
    }
    checkNext = buffer.get() != 0;
}

The primary issue with this approach is that it requires more comparisons for the same amount of work. For large documents, the process calls buffer.get() billions of times. Processing each byte individually requires a load, check, and conditional jump each time, which rapidly increases the total instruction count.

To improve performance, we used a classic optimization technique: SWAR (SIMD Within A Register Vectorization). Instead of checking each byte separately, SWAR lets us examine eight bytes simultaneously with a single 64-bit load and some bitwise operations. Here’s what that looks like:

long chunk = buffer.getLong(pos);
long mask = chunk - 0x0101010101010101L;
mask &= ~chunk;
mask &= 0x8080808080808080L;
if (mask != 0) {
    int offset = Long.numberOfTrailingZeros(mask) >>> 3;
    return (pos - prevPos) + offset + 1;
}

These 'magic numbers' aren’t arbitrary: 0x0101010101010101L repeats the byte 1, while 0x8080808080808080L repeats the byte 128. By subtracting 1 from each byte, ANDing with the inverted value, and applying the high-bit mask, you can instantly detect if a zero exists. Then, simply counting the trailing zeros allows you to calculate the precise byte offset. This method is highly effective with CPU pipelining.

Let’s take a simple example. Suppose we use an int (4 bytes) for simplicity. The value: 0x7a000aFF contains a zero byte. We will demonstrate how the SWAR technique detects it:

Step                    | Value (Hex)   | Value (Binary, per byte)

------------------------|---------------|-----------------------------

chunk                   | 0x7a000aFF    | 01111010 00000000 00001010 11111111

ones                    | 0x01010101    | 00000001 00000001 00000001 00000001

mask (high bit mask)    | 0x80808080    | 10000000 10000000 10000000 10000000

Subtraction:

chunk      = 01111010 00000000 00001010 11111111

- ones     = 00000001 00000001 00000001 00000001

------------------------------------------------

            01111000 11111111 00001001 11111110

                          ↑
                      underflow

                       (0-1=FF)

Bitwise AND with inverted chunk:

prevResult = 01111001 11111111 00001001 11111110

& ~chunk   = 10000101 11111111 11110101 00000000

------------------------------------------------

            00000001 11111111 00000001 00000000

Bitwise AND with mask (high bits):

prevResult = 00000001 11111111 00000001 00000000

& mask     = 10000000 10000000 10000000 10000000

------------------------------------------------

             00000000 10000000 00000000 00000000

The final result:

The result has a high bit set (10000000) in Byte 2, showing there’s a zero byte at that position.
After isolating one byte, we can use the Integer.numberOfTrailingZeros(mask) >>> 3 to get the offset of the zero byte. This method is often an intrinsic function, built into the JVM, producing efficient single instructions.

Because the loop body now consists of a small, predictable set of arithmetic instructions, it integrates seamlessly with modern CPU pipelines. The efficiency of SWAR stems from its reduced instruction count, the absence of per-byte branches, and one memory load for every eight bytes.

5. Avoiding redundant copies and allocations

While optimizing CString detection with SWAR, we also identified an opportunity to reduce allocations and copies on the string decoding path.

Earlier versions of the driver wrapped the underlying ByteBuffer in a read-only view to guard against accidental writes, but that choice forced every CString decode to perform two copies:

From the buffer into a temporary ‘byte[]’.
From that ‘byte[]’ into the internal ‘byte[]’ that backs the ‘String’.

By verifying that the buffer contents remain immutable during decoding, we were able to safely remove the restrictive read-only wrapper. This allows us to access the underlying array directly and decode the string without intermediate copies.

if (buffer.isBackedByArray()) {
    int position = buffer.position();
    int arrayOffset = buffer.arrayOffset();
    return new String(array, arrayOffset + position, bsonStringSize - 1, StandardCharsets.UTF_8);
}

For direct buffers (which are not backed by a Java heap array), we cannot hand a backing array to the String constructor. We still need to copy bytes from the buffer, but we can avoid allocating a new byte[] for every string.

To achieve this, the decoder maintains a reusable byte[] buffer. The first call allocates it (or grows it if a larger string is encountered), and subsequent calls reuse the same memory region. That has two benefits:

Fewer allocations, less GC pressure, and memory zeroing: We no longer create a fresh temporary byte[] for every CString, which reduces the amount of work the allocator and garbage collector must do per document.
Better cache behavior: The JVM repeatedly reads and writes the same small piece of memory, which tends to remain hot in the CPU cache. We examined CPU cache behavior on our “FindMany and empty cursor” workload using async-profiler’s cache-misses event. Async-profiler samples hardware performance counters exposed by the CPU’s Performance Monitoring Unit (PMU), the hardware block that tracks events such as cache misses, branch misses, and cycles. For readString(), cache-miss samples dropped by roughly 13–28% between the old and new implementation, as we touch fewer cache lines per CString. We still treat the PMU data as directional rather than definitive — counters and sampling semantics vary by CPU and kernel — so the primary signal remains the end-to-end throughput gains (MB/s) that users actually observe.

On our “FindMany and empty cursor” workload, eliminating the redundant intermediate copy in readString improved throughput by approximately 22.5%. Introducing the reusable buffer contributed a ~5% improvement in cases where the internal array is not available.

6. String Encoding, removing method indirection and redundant checks

While benchmarking our code, we observed that encoding Strings to UTF-8, the format used by BSON, consumed a significant amount of time. BSON documents contain many strings, including attribute names as CStrings and various string values of different lengths and Unicode code points. The process of encoding strings to UTF-8 was identified as a hot path, prompting us to investigate it and suggest potential improvements. Our initial implementation used custom UTF-8 encoding to avoid creating additional arrays with the standard JDK libraries.

But inside the loop, every character involved several inner checks:

Verifying ByteBuffer capacity
Branching for different Unicode ranges
Repeatedly calling abstractions and methods

If the buffer was full, we’d fetch another one from a buffer pool (we pool ByteBuffers to reduce garbage collection (GC) pressure:

for (int i = 0; i < len;) {
    int c = Character.codePointAt(str, i);
    if (checkForNullCharacters && c == 0x0) {
        throw new BsonSerializationException(...);
    }

    if (c < 0x80) {
        //check if ByteBuffer has capacity and write one byt
    } else if (c < 0x800) {
        //check if ByteBuffer has capacity and write two bytes
    } else if (c < 0x10000) {
       //check if ByteBuffer has capacity and write three bytes
       total += 3;
    } else 
      //check if ByteBuffer has capacity and write fourfoure bytes
  }
  i += Character.charCount(c);
}

In practice, modern JVMs can unroll tight, counted loops, reducing back branches and enhancing instruction pipeline efficiency under suitable conditions. However, when examining the assembly generated by the JIT for this method, we observed that loop unrolling did not occur in this instance. This underscores the importance of keeping the hot path as straight as possible, minimizing branches, checks, and method indirection, especially for large workloads.

Our first optimization was based on the hypothesis that most workloads mainly use ASCII characters. Using this assumption, we developed a new loop that was much more JIT-friendly.

for (; sp < str.length(); sp++, pos++) {
    char c = str.charAt(sp);
    if (checkNullTermination && c == 0) {
        //throw exception
    }

    if (c >= 0x80) {
        break;
    }
    dst[pos] = (byte) c;
}

Before entering the loop, we verified that String.length() < ByteBuffer’s capacity and got the underlying array from the ByteBuffer (our ByteBuffer is a wrapper over the JDK or Netty buffers)

By verifying this invariant upfront, we eliminated the need for repeated capacity checks or method indirection inside the loop. Instead, we worked directly with the internal array of a ByteBuffer.

We also added a safeguard: if the character to encode is greater than 0x80, we’ve encountered a non-ASCII character and must fall back to a slower, more general loop with additional branching.

With this setup, the JIT usually unrolls the loop body, replacing it with several consecutive copies. This modification decreases the number of back branches and improves pipeline performance efficiency. If we zoom in on the compiled assembly, we can see that C2 has unrolled the loop by a factor of 4. Instead of processing one character per iteration, the hot path processes four consecutive characters (sp, sp+1, sp+2, sp+3) and then increments sp by 4. For example, HotSpot C2 on AArch64 might produce the following assembly, with some bookkeeping removed and only 2 of the 4 unrolled copies for brevity:

loop body:

; ----- char 0: value[sp], dst[pos] -----

    ldrsb   w5,  [x12,#16]          ; load value[sp]

    and     w11, w5, #0xff          ; w11 = (unsigned) c0

    cbz     w11, null_trap          ; if (c0 == 0) -> slow path (null terminator check)

    cmp     w11, #0x80              ; if (c0 >= 0x80) -> slow path (non-ASCII)

    b.ge    non_ascii1_path

    strb    w5,  [x10,#16]          ; dst[pos] = (byte)c0

    ; ----- char 1: value[sp+1], dst[pos+1] -----

    ldrsb   w4,  [x12,#17]          ; load value[sp+1]

    and     w11, w4, #0xff          ; w11 = (unsigned) c1

    cbz     w11, null_trap          ; if (c1 == 0) -> slow path (null terminator check)

    cmp     w11, #0x80              ; if (c1 >= 0x80) -> slow path (non-ASCII)

    b.ge    non_ascii1_path

    strb    w4,  [x10,#17]          ; dst[pos+1] = (byte)c1

What we did:

Removed internal method indirection (like write() wrappers) that introduced extra bound checks.
When writing ASCII, we wrote directly to the underlying ByteBuffer array if the capacity allowed, skipping extra bounds and range checks.
For multi-byte code points, we avoided repeated calls to ensureOpen(), hasRemaining(), and related checks by caching position and limit values inside hot paths.

This optimization improved insert throughput across all related benchmarks. For example:

Bulk write insert throughput for UTF-8 multi-byte characters improved by nearly 31%.
Bulk write insert throughput for ASCII improved by nearly 33%.

You can see the particular test conditions in the Performance Benchmarking specification.

Lessons learned

Cache immutable data on the hot path: In our case, pre-encoding BSON index CStrings once into a compact flat byte[] removed repeated int to byte[] conversions and cut heap overhead from thousands of tiny byte[] objects.
The JVM can surprise you: Use intrinsics and hardware features whenever possible. After implementing our changes and testing, we found that a simple modification affecting less than 0.03% of the codebase increased throughput for large document bulk inserts by nearly 39%.
Thoroughly profile your code: Optimize where it matters. Small, smart changes in the hot path can yield more benefits than rewriting cold code. By caching the buffer in a field and updating it only when switching to a new buffer, we improved bulk insert throughput by 16%.
Even cheap checks can add up: Bounds checks and branches in the hot path can be surprisingly costly - multiply a few cycles by billions, and you’ll notice a real impact. Move checks outside inner loops where possible, and don’t repeat invariants that have already been verified.
Vectorization (SIMD): Rewriting critical code paths with vectorized methods (e.g., SWAR) can significantly increase throughput by enabling you to process multiple data elements simultaneously per instruction.
Removing method indirection and redundant checks: Optimizing low-level operations required removing write(...)/put(...) wrappers to eliminate an extra layer of method indirection and the repeated invariant checks. By writing directly to the underlying ByteBuffer array for ASCII and caching position values in hot paths for multi-byte code points, we bypassed repeated validation calls, such as ensureOpen() and hasRemaining(). This resulted in a 33% improvement in bulk write insert throughput for ASCII.

Figure 3. The figure below shows the final results for throughput improvements (measured in MB/s) after optimizing the driver, as explained above, based on this performance benchmarking specification. The ‘Large doc Collection BulkWrite insert’ saw the highest performance improvement +96.46%.

Check out our developer blog to learn how we are solving different engineering problems, or consider joining our engineering team.

Building a Semantic Search Engine with Hugging Face Transformers and MongoDB Atlas Vector Search

MongoDB Guests — Tue, 10 Feb 2026 15:30:24 +0000

This tutorial was written by Arek Borucki.

Search is one of the most critical components of applications that work with text data, yet traditional keyword-based search has a major limitation: it only works when users type the exact words stored in the database, usually in the same language.

Imagine an online fashion store with products described in English. A customer searches in French for "chaussures de course confortables pour marathon" (comfortable marathon running shoes) and then in Polish for "wygodne buty do biegania na maraton". Even though both queries describe the same concept, a keyword-based search will return no results because the wording and language are different from the English product descriptions.

Semantic search addresses this limitation by focusing on meaning rather than exact words or language. Instead of comparing text strings, it maps text to numerical representations that capture semantic intent, allowing it to recognize that all of these queries refer to the same concept.

In this blog post, you will build a semantic search engine using Transformers and a real dataset from Hugging Face, a multilingual language model, and MongoDB Atlas Vector Search.

Understand Semantic Search Mechanisms

Before moving to the implementation, it is important to understand how the individual components work together.

Semantic search relies on three core elements:

A language model that converts text into numerical representations called embeddings.
A vector database capable of storing and searching these representations.
An application layer that connects everything into a working system.

Explore the Hugging Face Platform

Training language models from scratch requires enormous computational resources and training data. A single model can cost millions of dollars in infrastructure. Hugging Face solves this problem by providing a repository of ready-to-use models and datasets.

Often described as the “GitHub for AI”, Hugging Face is an open platform and community for machine learning, focused on natural language processing and multimodal AI. It hosts thousands of pre-trained models and public datasets, along with developer tools such as the Transformers library and hosted inference APIs, enabling teams to build AI-powered applications without training models from scratch. Companies like Google, Meta, Microsoft, and MongoDB publish their models there, which anyone can download and use.

In this tutorial, you will download two resources from Hugging Face: a multilingual language model that enables cross-language semantic search and a fashion product dataset that will be used as the searchable data source.

Understand the Transformers Architecture

Hugging Face Transformers is an open-source Python library that provides a unified API for working with Transformer-based models. The library handles model loading, tokenization, and inference, while pre-trained model checkpoints are hosted on the Hugging Face Hub and automatically downloaded when used.

Transformers 5 introduces a more modular architecture with a strong focus on interoperability, making quantization a first-class feature for smaller models and faster inference.

This tutorial uses sentence-transformers, a library built on top of Hugging Face Transformers that specializes in generating sentence and text embeddings for semantic similarity tasks.

Discover Vector Search

Vector search is a database search technique that powers semantic search. MongoDB Atlas provides this capability through Atlas Vector Search, where instead of matching keywords, the database compares numerical vectors.

This approach works by storing embeddings generated by Hugging Face language models directly in the database. Using Hugging Face Transformers, text is converted into embeddings, which are numerical representations of semantic meaning.

These embeddings are stored in MongoDB Atlas Vector Search and indexed for efficient similarity search. Each Hugging Face model produces embeddings of a fixed size. For example, the paraphrase-multilingual-MiniLM-L12-v2 model generates vectors with 384 dimensions, and MongoDB Atlas Vector Search must be configured with the same dimensionality.

When a user submits a query, Hugging Face generates an embedding for the query text. MongoDB Atlas Vector Search compares this vector with stored embeddings and returns the most semantically similar documents using Approximate Nearest Neighbor algorithms.

In short, Hugging Face generates embeddings, and MongoDB Atlas Vector Search stores and searches them. Together, they enable fast, meaning-based search.

Configure the Development Environment

The development environment requires several components: a Python interpreter with machine learning libraries, a local MongoDB Atlas deployment with Vector Search enabled, and a web framework to expose the API. Each of these components can be run locally.

Create the Project Structure

The project structure separates source code from configuration to keep the application organized and secure. Configuration files such as .env contain sensitive information, including connection strings, and should never be committed to the repository. The src directory isolates the application logic, making the project easier to test, maintain, and containerize later. Create the project directories and configuration files as shown below:

mkdir semantic-search-demo
cd semantic-search-demo
mkdir src
touch requirements.txt .env

Install Dependencies

Python does not isolate dependencies between projects by default, which can lead to version conflicts. To avoid this, project dependencies and their exact versions are defined in a requirements.txt file.

This project uses:

Sentence-transformers to generate embeddings with Hugging Face models
Datasets library to download datasets from Hugging Face
Pymongo to communicate with MongoDB
FastAPI and Uvicorn to serve the API
Torch as the underlying machine learning framework.

Create the requirements.txt file using the following command:

cat <<EOF > requirements.txt
fastapi>=0.109.0
uvicorn>=0.27.0
pymongo>=4.6.1
sentence-transformers>=2.3.0
python-dotenv>=1.0.0
datasets>=2.16.0
Pillow>=10.0.0
torch
EOF

Now, create a Python virtual environment and install the project dependencies. A virtual environment isolates the project from the system Python and prevents version conflicts, which is standard practice in production environments.

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Run MongoDB Atlas with Vector Search

To use MongoDB Atlas Vector Search without deploying a cloud cluster, you can run a local Atlas deployment using Atlas CLI. This approach provides the same Vector Search and Atlas Search capabilities as the cloud version, but runs locally using Docker, making it ideal for development and learning. Atlas CLI manages the entire setup and ensures your local environment behaves the same way as Atlas in the cloud. This helps avoid configuration mismatches later. First, install Atlas CLI if you do not have it already:

brew install mongodb-atlas-cli

Create a local Atlas deployment using atlas deployments setup command:

atlas deployments setup

? What type of deployment would you like to work with? local
[Default Settings]
Deployment Name         local5783
MongoDB Major Version   8 (latest minor version)
? How do you want to set up your local Atlas deployment? default

Creating your cluster local9328
1/3: Starting your local environment...
2/3: Downloading the latest MongoDB image to your local environment...
3/3: Creating your deployment local9328...
Deployment created!
Connection string: "mongodb://localhost:51213/?directConnection=true"
? How would you like to connect to local9328? mongosh

When prompted, choose local and accept the default settings. Atlas CLI will download the required MongoDB image and create a local Atlas cluster. Once the deployment is ready, Atlas CLI displays a connection string similar to:

mongodb://localhost:51213/?directConnection=true

When prompted to connect using mongosh, selecting yes will automatically open a shell session connected to the local Atlas cluster, allowing you to verify that the database is running correctly.

AtlasLocalDev local9328 [direct: primary] test> show dbs
admin   224.00 KiB
config  224.00 KiB
local   392.00 KiB
AtlasLocalDev local9328 [direct: primary] test>

After creating the cluster, update your .env file with the connection string from Atlas CLI.

cat <<EOF > .env
MONGO_URI=mongodb://localhost:51213/?directConnection=true
DB_NAME=fashion_db
COLLECTION_NAME=products
# Multilingual embedding model (50+ languages, including French, Polish, and English)
MODEL_NAME=sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
EOF

Implement System Components

The system is composed of three main modules: a database module, a language model module, and an indexing module. Together, they form the core of the semantic search pipeline.

Create the Database Module

The database module is responsible for managing the MongoDB connection and configuring the vector search index. The vector index enables efficient similarity search by organizing embeddings in a structure optimized for nearest-neighbor retrieval in high-dimensional space. Create the database module:

cat <<'EOF' > src/database.py
import os
from pymongo import MongoClient
from pymongo.operations import SearchIndexModel
from dotenv import load_dotenv

load_dotenv()


class MongoManager:
    def __init__(self):
        self.client = MongoClient(os.getenv("MONGO_URI"))
        self.db = self.client[os.getenv("DB_NAME")]
        self.collection = self.db[os.getenv("COLLECTION_NAME")]

    def get_collection(self):
        return self.collection

    def create_vector_index(self):
        """Creates a Vector Search index"""
        search_index_model = SearchIndexModel(
            definition={
                "fields": [
                    {
                        "type": "vector",
                        "path": "embedding",
                        "numDimensions": 384,
                        "similarity": "cosine"
                    }
                ]
            },
            name="vector_index",
            type="vectorSearch"
        )
        try:
            self.collection.create_search_index(model=search_index_model)
            print("Vector index 'vector_index' has been initialized.")
        except Exception as e:
            print(f"Index status: {e}")
EOF

Create the Language Model Module

The AI module encapsulates the embedding generation logic. The paraphrase-multilingual-MiniLM-L12-v2 model was trained on text from more than 50 languages, allowing it to capture semantic similarity across languages without explicit translation.
On the first run, the sentence-transformers library downloads the model weights from the Hugging Face Hub and stores them in a local cache.

cat <<'EOF' > src/ai.py
import os
from sentence_transformers import SentenceTransformer
from dotenv import load_dotenv

load_dotenv()


class AIModel:
    def __init__(self):
        model_name = os.getenv("MODEL_NAME")
        print(f"Loading Transformer model: {model_name}...")
        self.model = SentenceTransformer(model_name)
        print("Model ready.")

    def generate_embedding(self, text: str):
        if not text:
            return []
        return self.model.encode(text).tolist()
EOF

The initial download requires an Internet connection and may take several minutes depending on network speed. Subsequent runs reuse the cached model and start almost instantly.

Create the Data Indexing Module

The indexing module downloads a dataset from Hugging Face, generates embeddings for each document, and stores the enriched data in MongoDB.

cat <<'EOF' > src/indexer.py
from datasets import load_dataset
from src.database import MongoManager
from src.ai import AIModel


def run_indexing():
    print("=" * 50)
    print("Starting indexing process...")
    print("=" * 50)

    print("\n[1/6] Initializing MongoDB connection...")
    db = MongoManager()
    print("✓ MongoDB connected")

    print("\n[2/6] Loading AI model...")
    ai = AIModel()
    print("✓ AI model loaded")

    print("\n[3/6] Getting MongoDB collection...")
    collection = db.get_collection()
    print("✓ Collection ready")

    print("\n[4/6] Downloading fashion dataset from Hugging Face...")
    dataset = load_dataset(
        "ashraq/fashion-product-images-small",
        split="train"
    )
    print(f"✓ Dataset downloaded: {len(dataset)} total products")

    sample_data = dataset.shuffle(seed=42).select(range(200))
    print(f"\n[5/6] Processing {len(sample_data)} products (randomly sampled)...")

    collection.delete_many({})
    print("✓ Cleared existing data")

    docs_to_save = []

    for idx, item in enumerate(sample_data):
        if idx % 20 == 0:
            print(f"  Processing item {idx + 1}/{len(sample_data)}...")

        product = {
            "name": item["productDisplayName"],
            "category": item["masterCategory"],
            "subCategory": item["subCategory"],
            "gender": item["gender"],
            "price": 99.99,
        }

        text_to_embed = (
            f"{item['productDisplayName']} "
            f"{item['usage']} "
            f"{item['baseColour']}"
        )
        product["embedding"] = ai.generate_embedding(text_to_embed)

        docs_to_save.append(product)

    if docs_to_save:
        print(f"\n  Saving {len(docs_to_save)} products to MongoDB...")
        collection.insert_many(docs_to_save)
        print(f"✓ Saved {len(docs_to_save)} products")

    print("\n[6/6] Creating vector search index...")
    db.create_vector_index()

    print("\n" + "=" * 50)
    print("✓ Indexing completed successfully!")
    print("=" * 50)


if __name__ == "__main__":
    run_indexing()
EOF

Create the Search API

The API layer exposes semantic search functionality over HTTP. The /search endpoint accepts a text query, converts it into an embedding using the language model, and performs vector search in MongoDB.

MongoDB executes a $vectorSearch aggregation stage, which finds documents whose embedding vectors are most similar to the query vector. The results are returned together with a similarity score.

cat <<'EOF' > src/main.py
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, field_validator
from src.database import MongoManager
from src.ai import AIModel

app = FastAPI(title="Fashion Semantic Search")
db = MongoManager()
ai = AIModel()
collection = db.get_collection()


class SearchRequest(BaseModel):
    query: str
    limit: int = 5

    @field_validator('query')
    @classmethod
    def query_must_not_be_empty(cls, v):
        if not v or not v.strip():
            raise ValueError('Query cannot be empty')
        return v.strip()

    @field_validator('limit')
    @classmethod
    def limit_must_be_positive(cls, v):
        if v < 1:
            raise ValueError('Limit must be at least 1')
        if v > 100:
            raise ValueError('Limit cannot exceed 100')
        return v


@app.post("/search")
def search_products(payload: SearchRequest):
    query_vector = ai.generate_embedding(payload.query)

    if not query_vector:
        raise HTTPException(status_code=400, detail="Failed to generate embedding for query")

    pipeline = [
        {
            "$vectorSearch": {
                "index": "vector_index",
                "path": "embedding",
                "queryVector": query_vector,
                "numCandidates": 100,
                "limit": payload.limit
            }
        },
        {
            "$project": {
                "_id": 0,
                "name": 1,
                "category": 1,
                "subCategory": 1,
                "score": {"$meta": "vectorSearchScore"}
            }
        }
    ]

    try:
        results = list(collection.aggregate(pipeline))
        return {"query": payload.query, "results": results}
    except Exception as e:
        raise HTTPException(status_code=500, detail=f"Search failed: {str(e)}")


@app.get("/")
def read_root():
    return {"status": "ok", "service": "Fashion Semantic Search API"}
EOF

Test Multilingual Search

The database contains products with descriptions in English. A key advantage of semantic search is the ability to handle queries written in a different language than the stored data. A multilingual model recognizes that a query in French or Polish can express the same intent as an English product description.

Run the Indexing Process

Before running the indexer, mark the src directory as a Python package:

touch src/__init__.py

At this point, the project structure should look like this:

project-root/
├── src/
│   ├── __init__.py
│   ├── ai.py
│   ├── database.py
│   ├── indexer.py
│   └── main.py
├── requirements.txt
├── .env
└── venv/

Start by indexing the data. This step downloads the dataset from Hugging Face, generates embeddings for each product, and stores the documents together with their vectors in MongoDB. Run the indexer:

python -m src.indexer

The terminal displays progress information for each step of the process, including database initialization, model loading, dataset download, document processing, and vector index creation:

[1/6] Initializing MongoDB connection...
✓ MongoDB connected

[2/6] Loading AI model...
Model ready.
✓ AI model loaded

[3/6] Getting MongoDB collection...
✓ Collection ready

[4/6] Downloading fashion dataset from Hugging Face...
✓ Dataset downloaded: 44072 total products

[5/6] Processing 200 products (randomly sampled)...
✓ Cleared existing data
  Processing item 1/200...
  Processing item 21/200...
  Processing item 41/200...
  Processing item 61/200...
  Processing item 81/200...
  Processing item 101/200...
  Processing item 121/200...
  Processing item 141/200...
  Processing item 161/200...
  Processing item 181/200...

  Saving 200 products to MongoDB...
✓ Saved 200 products

[6/6] Creating vector search index...
Vector index 'vector_index' has been initialized.

==================================================
✓ Indexing completed successfully!
==================================================

Run the API Server

Activate the virtual environment and start the FastAPI server:

source venv/bin/activate
python -m uvicorn src.main:app --reload

The API is now available at http://localhost:8000

Execute Test Queries

The first test checks cross-language semantic search using a French query. The database contains product descriptions written in English.

curl -X POST "http://localhost:8000/search" \
     -H "Content-Type: application/json" \
     -d '{"query": "chaussures de course confortables pour marathon"}'

Output:

{                                                                                                                                                                                                                 
    "query": "chaussures de course confortables pour marathon",                                                                                                                                                     
    "results": [                                                                                                                                                                                                    
      {                                                                                                                                                                                                             
        "name": "Nike Men Transform III White Sports Shoes",                                                                                                                                                        
        "category": "Footwear",                                                                                                                                                                                     
        "subCategory": "Shoes",                                                                                                                                                                                     
        "score": 0.7942747473716736                                                                                                                                                                                 
      },                                                                                                                                                                                                            
      {                                                                                                                                                                                                             
        "name": "Fila Men's Basic Low Red Shoe",                                                                                                                                                                    
        "category": "Footwear",                                                                                                                                                                                     
        "subCategory": "Shoes",                                                                                                                                                                                     
        "score": 0.7913200855255127                                                                                                                                                                                 
      },                                                                                                                                                                                                            
      {                                                                                                                                                                                                             
        "name": "ADIDAS Unisex Brown Casual Shoes",                                                                                                                                                                 
        "category": "Footwear",                                                                                                                                                                                     
        "subCategory": "Shoes",                                                                                                                                                                                     
        "score": 0.789846658706665                                                                                                                                                                                  
      },                                                                                                                                                                                                            
      {                                                                                                                                                                                                             
        "name": "Numero Uno Men's Red Canvas Shoe",                                                                                                                                                                 
        "category": "Footwear",                                                                                                                                                                                     
        "subCategory": "Shoes",                                                                                                                                                                                     
        "score": 0.7863043546676636                                                                                                                                                                                 
      },                                                                                                                                                                                                            
      {                                                                                                                                                                                                             
        "name": "Lotto Men Sprint Black Sports Shoes",                                                                                                                                                              
        "category": "Footwear",                                                                                                                                                                                     
        "subCategory": "Shoes",                                                                                                                                                                                     
        "score": 0.7819452285766602                                                                                                                                                                                 
      }                                                                                                                                                                                                             
    ]                                                                                                                                                                                                               
  }

The output shows that MongoDB Atlas Vector Search returns a ranked list of products based on vector similarity scores. Although the query is written in French and the product data is in English, the multilingual embedding model correctly identifies that the query refers to athletic footwear. The top results include Nike and Lotto sports shoes from well-known athletic brands, which are semantically close to "marathon running shoes".

The second test verifies cross-language understanding using a Polish query. "Wygodne buty do biegania na maraton" translates to "comfortable running shoes for marathon". This is the same concept as the French query above.

curl -X POST "http://localhost:8000/search" \                                                                                                                                                                     
       -H "Content-Type: application/json" \                                                                                                                                                                        
       -d '{"query": "wygodne buty do biegania na maraton"}'

Output:

{                                                                                                                                                                                                                 
    "query": "wygodne buty do biegania na maraton",                                                                                                                                                                 
    "results": [                                                                                                                                                                                                    
      {                                                                                                                                                                                                             
        "name": "Lotto Men Sprint Black Sports Shoes",                                                                                                                                                              
        "category": "Footwear",                                                                                                                                                                                     
        "subCategory": "Shoes",                                                                                                                                                                                     
        "score": 0.7927727699279785                                                                                                                                                                                 
      },                                                                                                                                                                                                            
      {                                                                                                                                                                                                             
        "name": "Nike Men Transform III White Sports Shoes",                                                                                                                                                        
        "category": "Footwear",                                                                                                                                                                                     
        "subCategory": "Shoes",                                                                                                                                                                                     
        "score": 0.787016749382019                                                                                                                                                                                  
      },                                                                                                                                                                                                            
      {                                                                                                                                                                                                             
        "name": "Fila Men's Basic Low Red Shoe",                                                                                                                                                                    
        "category": "Footwear",                                                                                                                                                                                     
        "subCategory": "Shoes",                                                                                                                                                                                     
        "score": 0.7850795984268188                                                                                                                                                                                 
      },                                                                                                                                                                                                            
      {                                                                                                                                                                                                             
        "name": "ADIDAS Unisex Brown Casual Shoes",                                                                                                                                                                 
        "category": "Footwear",                                                                                                                                                                                     
        "subCategory": "Shoes",                                                                                                                                                                                     
        "score": 0.7824510335922241                                                                                                                                                                                 
      },                                                                                                                                                                                                            
      {                                                                                                                                                                                                             
        "name": "Nike Men Downshifter White Sports Shoes",                                                                                                                                                          
        "category": "Footwear",                                                                                                                                                                                     
        "subCategory": "Shoes",                                                                                                                                                                                     
        "score": 0.7783344388008118                                                                                                                                                                                 
      }                                                                                                                                                                                                             
    ]                                                                                                                                                                                                               
  }

The Polish query returns the same category of products as the French query — athletic footwear from Nike, Lotto, Fila, and Adidas. This confirms that the multilingual model correctly maps semantically equivalent queries across languages to the same product space.

The third test explores a different product category. This Polish query translates to "elegant evening handbag" and tests whether the model can identify accessories rather than footwear.

curl -X POST "http://localhost:8000/search" \
     -H "Content-Type: application/json" \
     -d '{"query": "elegancka wieczorowa torebka"}'

Output:

{
  "query": "elegancka wieczorowa torebka",
  "results": [
    {
      "name": "SDL by Sweet Dreams Women Green Printed Night Suit S11-3124",
      "category": "Apparel",
      "subCategory": "Loungewear and Nightwear",
      "score": 0.7503366470336914
    },
    {
      "name": "Lucera Women Silver Pendant",
      "category": "Accessories",
      "subCategory": "Jewellery",
      "score": 0.7264891862869263
    },
    {
      "name": "Giorgio Armani Women Idole Perfume",
      "category": "Personal Care",
      "subCategory": "Fragrance",
      "score": 0.7015564441680908
    },
    {
      "name": "Levis Men Comfort Style Grey Innerwear Vest",
      "category": "Apparel",
      "subCategory": "Innerwear",
      "score": 0.6910820007324219
    },
    {
      "name": "Kiara Women Teal Handbag",
      "category": "Accessories",
      "subCategory": "Bags",
      "score": 0.6863116025924683
    }
  ]
}

The results show that the semantic search engine understands the Polish query, even though products are described in English. However, the handbag appears only at position 5. The word "wieczorowa" (evening) led the model to nightwear, and "elegancka" (elegant) to jewelry and perfume. These connections make sense linguistically, but they are not what the user was looking for. In a real application, adding category filters would help return more relevant results.

Summary

This semantic search system is built on Hugging Face language models, which transform text into dense vector embeddings that capture meaning across languages. Using a multilingual model from the Hugging Face Hub eliminates the need to train custom models and enables cross-language similarity out of the box.

These embeddings are stored and searched with MongoDB Atlas Vector Search, while the application layer connects user queries to relevant results. Together, Hugging Face and vector search enable multilingual, meaning-based retrieval that goes beyond traditional keyword matching.

As shown in the test results, semantic search correctly identifies broad categories across languages. The less precise results for the handbag query are primarily due to the limited sample size of 200 products used in this demo—with a full catalog, the system would have more relevant items to match. For production applications, consider indexing the complete dataset, combining vector search with metadata filters, implementing re-ranking, or using domain-specific models to improve precision.

Getting Started with MongoDB BulkWrite in PHP

MongoDB Guests — Thu, 29 Jan 2026 22:38:41 +0000

This tutorial was written by James Shisiah.

MongoDB is a document-oriented NoSQL database with a flexible schema designed to store, query, and manipulate large volumes of data. For PHP development, we have the official MongoDB PHP driver, which consists of a PHP extension as well as a high-level library (mongodb/mongodb) that exposes a clean API for interacting with collections, documents, and advanced database features, including full-text and semantic search on MongoDB Atlas.

With the library, you get tools for standard database operations like inserts, updates, and deletes. Additionally, it supports bulk write operations via the MongoDB BulkWrite API. MongoDB BulkWrite is a command that allows an application to send a mixed list of write operations to a MongoDB server in a single network request. Write operations include insert, update, replace, and delete commands.

In this guide, we will explore the MongoDB BulkWrite API and its performance improvements, and we’ll demonstrate how to handle mixed bulk operations with sample code using the PHP library.

Prerequisites

PHP installed (>= 8.1 required)
MongoDB database server (Atlas for easy setup)—or locally if preferred, via the provided docker-compose.yml file
PIE (a modern PHP installer for extensions)—used to install the PHP MongoDB extension
Composer for managing PHP dependencies, used to install mongodb/mongodb

Why use BulkWrite for batch operations?

Many real-world applications require inserting, updating, or deleting large numbers of documents at once. Examples include:

Bulk importing thousands of records from a CSV file, e.g., log files.
Updating product prices in bulk.
Syncing customer data from external systems.
Cleaning old or inactive entries.
Data migrations and mass updates.

Using individual write operations (insertOne, updateOne, etc.) for each record incurs performance penalties because every write is sent separately over the network.

How MongoDB’s BulkWrite API solves this problem

Improved performance: Multiple operations are grouped into a single batch and sent together, reducing the time spent on the network round-trip.
Fewer network calls: Instead of 1,000 operations, which would make 1,000 requests, BulkWrite can send them all in a single request.
Ideal for imports, updates, and data cleanups: Whether you're inserting new data, updating existing records, or performing deletions, BulkWrite can handle all operations efficiently and atomically at the document level.

Comparing old vs. new BulkWrite APIs in PHP

Before proceeding with the tutorial, let’s look into the different APIs that MongoDB provides for working with large bulk write operations. The BulkWrite API performs bulk write operations at two levels. The old API performs write operations at the collection level, whereas the new API performs them at the MongoDB cluster (client) level.

Old BulkWrite API (collection level)

At the collection level, you can perform bulk write operations on a single collection. The old MongoDB bulk write implementation provides two related APIs:

MongoDB\Driver\Manager::executeBulkWrite()
MongoDB\Collection::bulkWrite()

Collection::bulkWrite() is a convenience wrapper provided by the high-level PHP library. Internally, it prepares a MongoDB\Driver\BulkWrite object and executes it via Manager::executeBulkWrite(), automatically supplying the correct namespace. In other words, Collection::bulkWrite() object provides a namespace plus write operations to Manager::executeBulkWrite(), making your code more readable.

The Collection::bulkwrite method makes a database call for each type of write operation. It can perform multiple insert operations in one call, but it makes two separate calls to the database for an insert operation and a replace operation. While this is efficient when performing a bulk operation for one type of write operation—e.g., inserting thousands of rows from a CSV file—if you combine inserts and updates in one ops array, it will make separate calls to the database for each type (insert and update). Let’s explore sample code snippets with the old bulk write APIs.

Sample code with Manager::executeBulkWrite()

$bulk = new MongoDB\Driver\BulkWrite;

$bulk->insert(['name' => 'Alice']);
$bulk->update(['email' => 'john@example.com'], ['$set' => ['active' => true]]);
$bulk->delete(['status' => 'inactive']);

$manager = new MongoDB\Driver\Manager($mongoDBuri);
$result = $manager->executeBulkWrite('bulkwritedb.users', $bulk);

Sample code with Collection::bulkwrite()

$client = new MongoDB\Client($mongoDBuri);

 $database = $client->selectDatabase('bulkwritedb');
 $collection = $database->selectCollection('users');

 $operations = [
        ['insertOne' => [['name' => 'Alice', 'age' => 25]]],
        ['insertOne' => [['name' => 'Bob', 'age' => 30]]],
        ['insertOne' => [['name' => 'Charlie', 'age' => 28]]],
    ];

 $result = $collection->bulkWrite($operations);

In this example, Collection::bulkWrite() internally provides the namespace bulkwritedb.users to Manager::executeBulkWrite(), which then makes the database call.

Keep in mind the following points when working with the old bulk write APIs:

Operates on a single namespace
Returns a single write result object
Can fail if the server’s response exceeds the maximum BSON document size
Well-suited for small to medium batch operations

New BulkWrite API (cluster/client level)

The new MongoDB BulkWrite API makes calls at the client level, allowing you to perform operations on multiple collections and databases in the same cluster. The new bulkwrite API is exposed via the MongoDB\Client::bulkWrite() method and works with MongoDB Server version 8.0 or later. If you are still running an older MongoDB version, this is your sign to upgrade. For instance, bulkWrite operations on MongoDB 8.0 can run up to 56% faster than bulkWrite operations on MongoDB 7.0. The new API performs all write operations (inserts, updates, replace, and delete) in one database call, unlike the old API, which makes different calls for each type of write operation. It follows the following execution path:

Creates a MongoDB\Driver\BulkWriteCommand instance
Executes it via Manager::executeBulkWriteCommand()
Returns results using a cursor (the old API does not return results using a cursor)

By returning results through a cursor, MongoDB can stream responses incrementally rather than sending a single, potentially large BSON document. This allows us to work around the limitations of the old BulkWrite API and hence can handle much larger bulk operations safely. Your application will have better resilience during large imports and migrations.

Sample code with MongoDB\Client::bulkWrite()

$client = new MongoDB\Client($mongoDBuri);

// Create the ClientBulkWrite for your 'users' collection
$usersCollection = $client->bulkwritedb->users;
$bulkWrite = MongoDB\ClientBulkWrite::createWithCollection($usersCollection);

// Add your operations
$bulkWrite->insertOne(['name' => 'Alice', 'age' => 25]);
$bulkWrite->insertOne(['name' => 'Bob', 'age' => 30]);
$bulkWrite->insertOne(['name' => 'Charlie', 'age' => 28]);

// Perform the bulk write operation
$result = $client->bulkWrite($bulkWrite);

The sample code above demonstrates how to create a ClientBulkWrite instance from a MongoDB\Collection instance by using the createWithCollection() method.

In this guide, we will be working with the new MongoDB bulkWrite API. You will learn how to write to multiple collections by chaining calls to the createWithCollection() method on your MongoDB\Client::bulkWrite() instance. For modern, scalable applications, especially writing to multiple collections, we recommend using MongoDB\Client::bulkWrite() since it allows you to iterate through its results reliably.

1. Project setup

Before we start working with MongoDB’s BulkWrite operations in PHP, let’s set up a sample project environment. We have avoided working with frameworks so you can focus purely on understanding how MongoDB’s PHP extension and the high-level library work. The code for this project can be retrieved from this GitHub repository, which you can clone with the command:

git clone git@github.com:jaymoh/mongodb-bulkwrite-php.git

Create the project folder

Choose a location on your machine and create a new directory, e.g.:

mongodb-bulkwrite-php/

Inside it, we will have a few files, such as:

mongodb-bulkwrite-php/
├── composer.json
├── bulk_write_demo.php
└── customers_10000.csv

You will add more demo files as needed during the tutorial.

Install the MongoDB PHP extension if it is not already installed

The MongoDB PHP extension (mongodb.so) must be installed before the library can work.

Check if it is enabled using the command:

php -m | grep mongodb

Running the command should display mongodb in your terminal. If nothing appears, install the extension by following the official MongoDB PHP setup documentation or the PHP manual. For a quick setup, you can install and enable it using PIE with the following command:

pie install mongodb/mongodb-extension

Install the MongoDB high-level PHP library via Composer

The preferred method for installing MongoDB’s PHP library is via Composer. Install it with the command:

composer require mongodb/mongodb

Since we will be working with a MongoDB connection string, and we do not want it pushed to the git repositories, we will save it in an environment variable (.env) file. This file is then added to .gitignore so it’s ignored when pushing your code. Any dotenv library (symfony/dotenv, vlucas/phpdotenv, etc.) can be used to read environment variables. For this guide, we will be using the vlucas/phpdotenv library. You can install it into your project directory with the command:

composer require vlucas/phpdotenv

That is it for this step. Let’s establish a connection to MongoDB in the next step.

2. Connect to MongoDB

Now that the project is set up, the next step is to connect your PHP script to a MongoDB database. For the simplicity of this tutorial, we will work with a free MongoDB Atlas cluster, since it works online, requires no local installation, and gives you a clean connection URI.

However, if you prefer running MongoDB locally (e.g., via mongod on your machine) or via the MongoDB Atlas Docker image, you can also use a local URI; the steps are the same. We have included a docker-compose.yml file that you can easily use to run the MongoDB Atlas Docker image locally.

Get your connection string

Visit MongoDB Atlas and register for a free Atlas account. The Get Started With MongoDB Atlas docs outline clear steps you can follow to create a free cluster, set up user credentials, and add allowed IP addresses (here, you should include your current IP or add 0.0.0.0/0 to allow access from anywhere, for development purposes only).

When you create your cluster for the first time, you get this setup wizard that you can follow for a quick configuration:

We named our cluster BulkWrite for the sake of this tutorial. In the next step of the connect flow, under Choose a connection method, a connection string is displayed. You can choose your programming language (PHP) to view the code sample:

A connection string looks like this:

mongodb+srv://jamesshisiah_db_user:<db_password>@bulkwrite.vcdi5mk.mongodb.net/?appName=BulkWrite

Add the connection string to your project’s .env file

Copy your connection string, then create an .env file in your project folder. Add the following variables, replacing the MONGODB_URI value with your connection string:

MONGODB_URI=mongodb+srv://jamesshisiah_db_user:<db_password>@bulkwrite.vcdi5mk.mongodb.net/?appName=BulkWrite
MONGODB_DB=bulkwritedb

Next, create a PHP file to test the connection. We can call it bulk_write_connect.php. Add the following code snippet:

<?php

require 'vendor/autoload.php';

use MongoDB\Client;
use Dotenv\Dotenv;

$dotenv = Dotenv::createImmutable(__DIR__);
$dotenv->load();

$uri = $_ENV['MONGODB_URI'] ?: null; // '<your_mongodb_connection_string_here>'

// Testing Connect to MongoDB running on MongoDB Atlas
$client = new Client($uri);

// Test connection by listing databases
$databases = $client->listDatabases();
// If the above line does not throw an exception, the connection is successful
echo "Successfully connected to MongoDB!\n";

Run the script:

php bulk_write_connect.php

If everything is set up correctly, you should see:

Successfully connected to MongoDB!

If, for some reason, you did not see the success message or there was an exception thrown in your code, consider checking out the Connection Troubleshooting page for potential solutions to issues that you might encounter when working with the MongoDB PHP library.

With your connection set up, you're now ready to start working with MongoDB’s new BulkWrite API operations.

3. Prepare BulkWrite operations

After confirming that you can connect to a MongoDB database, let's write some PHP code to see how MongoDB’s BulkWrite feature works. BulkWrite allows you to combine multiple write operations, such as inserts, updates, and deletes, into a single batch request that can be executed on multiple collections or multiple databases in the same cluster.

Create BulkWriteCommand instance using MongoDB\ClientBulkWrite

Given that it works at the client level, the first step in your code is to create a client connection instance to the MongoDB server:

use MongoDB\Client;
$client = new Client("your-mongodb-uri-here");

Next, use the MongoDB\ClientBulkWrite builder class to create a BulkWriteCommand instance that specifies the write operations to perform. To do this, call the createWithCollection() method and pass in the first collection you wish to target. For example:

$usersCollection = $client->bulkwritedb->users;
$bulkWrite = MongoDB\ClientBulkWrite::createWithCollection($usersCollection);

Then, call one or more of the following write methods on the ClientBulkWrite instance to construct the bulk write operation:

deleteOne()
deleteMany()
insertOne()
replaceOne()
updateOne()
updateMany()

For example, here is how you can insert into the users’ collection using the created ClientBulkWrite instance:

$bulkWrite->insertOne(['name' => 'Bob', 'age' => 30]);

You can use the same $bulkWrite instance to perform an update operation:

$bulkWrite->updateOne(
    ['name' => 'John Who'],
    ['$set' => ['age' => 35]],
    ['upsert' => true],
);

Switch to a different Collection using withCollection()

What is more crucial to note here is that you can change the target collection for subsequent bulk operations by calling the withCollection() method on an existing ClientBulkWrite instance before making a database call. For example, you can insert into a customer’s collection in another database (e.g., my_shop_db) using the code:

$customersCollection = $client->my_shop_db->customers;

$bulkWrite = $bulkWrite->withCollection($customersCollection);
$bulkWrite->insertOne(['name' => 'Noella Gray', 'orders' => 10]);

You can still switch to another collection and perform deletion. For example, one of your users (user_id_10) wants all their data scraped from your app’s database:

$logsCollection = $client->my_app->user_logs;

$bulkWrite = $bulkWrite->withCollection($logsCollection);
$bulkWrite->deleteMany(['user_id' => 'user_id_10']);

When we finally execute the database call, the above line will remove all documents from the user_logs collection where the user_id field is equal to “user_id_10.”

Perform the bulk write operation

Once you have chained all your write operations (inserts, updates, deletes), you can then perform the bulk write operation with the code:

$result = $client->bulkWrite($bulkWrite);

MongoDB bulk write API sends all the different operations to different collections and databases in a single call, which is significantly faster than having to send the operations separately. Additionally, the fluent API is cleaner, offering a good developer experience.

Example script with all bulkWrite operations

Let’s say you have two application databases: one for your restaurant app and another for your e-commerce website. Both databases are in one cluster, so you can connect to them using a single client instance, and each database has different collections. In the GitHub repository, you will find a bulk_write_demo.php script. Here is a code snippet extracted from the script:

// Start with e-commerce database
$ecommerceDb = $client->ecommerce;
$customersCollection = $ecommerceDb->customers;
$ordersCollection = $ecommerceDb->orders;

// Restaurant database
$restaurantDb = $client->restaurant;
$menusCollection = $restaurantDb->menus;

// Create bulk write starting with customers
$bulkWrite = ClientBulkWrite::createWithCollection($customersCollection);

// Add customer operations
$bulkWrite->insertOne(['name' => 'Alice', 'email' => 'alice@example.com']);
$bulkWrite->updateOne(['name' => 'Alice'], ['$set' => ['status' => 'premium']]);

// Switch to orders collection (same database)
$bulkWrite = $bulkWrite->withCollection($ordersCollection);
$bulkWrite->insertOne(['customer' => 'Alice', 'total' => 99.99]);

// Switch to menus collection (different database!)
$bulkWrite = $bulkWrite->withCollection($menusCollection);
$bulkWrite->insertOne(['name' => 'Pizza', 'price' => 12.99]);
$bulkWrite->deleteMany(['available' => false]);

// Execute ALL operations in a single request and get a result object
$result = $client->bulkWrite($bulkWrite);

The script puts together the concepts we have covered in this section, including;

Cross-database operations: E-commerce DB and Restaurant DB in a single batch database call.
Collection switching: Using withCollection() to switch between collections.
All write operations: insertOne, updateOne, updateMany, replaceOne, deleteOne, and deleteMany.
Options: Demonstrates upsert option with replaceOne(). Upsert means "update or insert." When you run an update/replace with ['upsert' => true], MongoDB will update matching documents; if none match, it will insert a new document instead.
ID capture: Shows how to capture generated _id values from insertOne(). The second parameter of the insertOne() method takes a variable name by reference, and sets the insertId to the variable. For example, $bobId will hold the generated _id after the database insertion:

 $bulkWrite->insertOne(['name' => 'Bob Smith'], $bobId)

Result reporting: Displays counts for all operation types (more on this in the next step).

4. Handle BulkWrite results

After running a bulk operation, MongoDB returns a MongoDB\BulkWriteCommandResult object. A significant improvement in the new BulkWrite API (Client::bulkWrite()) is its ability to return results using a cursor, rather than a single large BSON document as in the previous API. This change allows you to handle much larger bulk write responses, removing the risk of failure if the response exceeds the maximum BSON document size (16MB), an issue that could occur with the older API (Collection::bulkWrite()).

Understanding how to read these values is essential for logging, debugging, and validating large data-processing operations. The new BulkWrite API returns two types of results: summary counts and cursor-based verbose results.

Summary counts (always available)

Summary counts are always returned and can be accessed from the result object using the following methods:

getInsertedCount(): total documents inserted
getMatchedCount(): total documents matched by filters
getModifiedCount(): total documents modified
getUpsertedCount(): total documents upserted
getDeletedCount(): total documents deleted
isAcknowledged(): whether the write was acknowledged
- isAcknowledged() returns true when using write concern w: 1 or higher (including majority). With w: 0, MongoDB does not wait for confirmation; the above result summary counts methods will return NULL. Default majority ensures the write is durable (can survive primary failover) across replica set failovers.

Sample code showing how to get summary counts

$result = $client->bulkWrite($bulkWrite);

echo "=== Summary Counts ===\n";
echo "Inserted:  " . $result->getInsertedCount() . "\n";
echo "Matched:   " . $result->getMatchedCount() . "\n";
echo "Modified:  " . $result->getModifiedCount() . "\n";
echo "Upserted:  " . $result->getUpsertedCount() . "\n";
echo "Deleted:   " . $result->getDeletedCount() . "\n";
echo "Acknowledged: " . ($result->isAcknowledged() ? 'Yes' : 'No') . "\n";

Cursor-based verbose results (new API advantage)

The new BulkWrite API supports returning results in batches using a cursor rather than only summary counts, so very large responses don’t hit BSON size limits. To receive detailed, per-operation results from the new BulkWrite API, you should set the verboseResults option to true when calling bulkWrite. You can enable verboseResults in the $options array parameter in the following ways.

When creating the ClientBulkWrite instance:

$bulkWrite = ClientBulkWrite::createWithCollection($collection, [
    'ordered' => true,
    'verboseResults' => true  // Enable detailed per-operation results
]);

When calling the bulkWrite command with your bulk write operations on your ClientBulkWrite instance:

$result = $client->bulkWrite(
    $bulkWrite,
    ['verboseResults' => true]
);

With verboseResults enabled, the $result object will now expose $cursor-based results in the following methods:

getInsertResults(): returns an array of detailed results for each insert operation (including insertedId)
getUpdateResults(): returns an array of detailed results for each update (including matchedCount, modifiedCount, upsertedId)
getDeleteResults(): returns an array of detailed results for each delete (including deletedCount)

Sample code showing how to loop and get the detailed results for each operation in your bulk write

$result = $client->bulkWrite($bulkWrite);

$insertResults = $result->getInsertResults();
    if ($insertResults) {
        foreach ($insertResults as $index => $insertResult) {
            echo "Insert operation #$index:\n";
            echo "  Inserted ID: " . $insertResult->insertedId . "\n";
        }
    }

$updateResults = $result->getUpdateResults();
    if ($updateResults) {
        foreach ($updateResults as $index => $updateResult) {
            echo "Update operation #$index:\n";
            echo "  Matched:  " . $updateResult->matchedCount . "\n";
            echo "  Modified: " . $updateResult->modifiedCount . "\n";
            if (isset($updateResult->upsertedId)) {
                echo "  Upserted ID: " . $updateResult->upsertedId . "\n";
            }
        }
    }

$deleteResults = $result->getDeleteResults();
    if ($deleteResults) {
        foreach ($deleteResults as $index => $deleteResult) {
            echo "Delete operation #$index:\n";
            echo "  Deleted: " . $deleteResult->deletedCount . "\n";
        }
    }

Understanding WriteConcerns

WriteConcern controls how strictly MongoDB should confirm a write.

For example:

"majority"—Wait for most nodes to acknowledge the write. This is the default WriteConcern. It ensures that write operations are acknowledged only after a majority of replica set members have written the data to their journals, providing strong guarantees.
"1"—Only the primary node needs to confirm the write.
"0"—Don’t wait for confirmation. Hence, this is faster. Your application is only notified that the primary server received the call, but it does not wait to get confirmation whether the write was successful. With (w: 0), the isAcknowledged() result method returns FALSE. The other summary count methods: getInsertedCount(), getModifiedCount(), and getDeletedCount() return NULL. Use only when performance is critical, and data loss is acceptable.

You can specify the WriteConcern option in the $options array parameter (just like verboseResults) of the bulkWrite command.

Example acknowledged write (w: ‘majority’)

use MongoDB\Driver\WriteConcern;

$writeConcern = new WriteConcern(WriteConcern::MAJORITY, 1000); // majority acknowledgement, 1 second timeout
$result = $client->bulkWrite(
    $bulkWrite,
    [
        'writeConcern' => $writeConcern,
        // You can also include other options, like 'verboseResults'
    ]
);

Note: When performing bulk imports or updates, use a higher write concern (majority) for critical data, and use lower write concerns for performance in non-critical writes. You can also use the wtimeout option (1000ms in the code snippet above) to specify how long to wait for acknowledgement before timing out.

Example unacknowledged write (w: 0)

$unacknowledgedWriteConcern = new WriteConcern(0); // w: 0

    $unacknowledgedBulkWrite = ClientBulkWrite::createWithCollection($collection, [
        'writeConcern' => $unacknowledgedWriteConcern,
    ]);

    $unacknowledgedBulkWrite->insertOne(['name' => 'Unacknowledged Product', 'price' => 5.99]);

    $unackResult = $client->bulkWrite($unacknowledgedBulkWrite);

    echo "Acknowledged: " . ($unackResult->isAcknowledged() ? 'Yes' : 'No') . "\n";
    echo "Inserted Count: " . ($unackResult->getInsertedCount() ?? 'NULL (not available)') . "\n";
    echo "\nNote: With w:0, result counts are unavailable - the server doesn't confirm the write.\n";

Error handling with try...catch

Bulk operations may fail due to various reasons, including:

Invalid queries.
Duplicate keys.
Network timeouts.
Schema validation rules.
Write concern failures.

You should always wrap your bulk operations in a try...catch. For optimal debugging, specifically catch the MongoDB\Driver\Exception\BulkWriteCommandException. This tells you what went wrong while still providing a partial result via getPartialResult(). You can check specific failures using getWriteErrors() and identify any write concern issues using getWriteConcernErrors().

This code snippet shows how to read the bulkWrite error results. Please find the full source code from the GitHub repository, in the file called bulk_write_results.php:

<?php

use MongoDB\Client;
use MongoDB\ClientBulkWrite;
use Dotenv\Dotenv;
use MongoDB\Driver\Exception\BulkWriteCommandException;
use MongoDB\Driver\WriteConcern;

$dotenv = Dotenv::createImmutable(__DIR__);
$dotenv->load();

$uri = $_ENV['MONGODB_URI'] ?: null;

$writeConcern = new WriteConcern(WriteConcern::MAJORITY, 1000); // majority acknowledgement, 1 second timeout

try {
    $client = new Client($uri);

    $database = $client->tutorial;
    $collection = $database->bulk_results_demo;

    // Create ClientBulkWrite with verboseResults and custom WriteConcern
    $bulkWrite = ClientBulkWrite::createWithCollection($collection, [
        'ordered' => true,
        'verboseResults' => true,  // Enable detailed per-operation results
        'writeConcern' => $writeConcern,
    ]);

    // Add as many operations as needed...
    $bulkWrite->insertOne(['name' => 'Product A', 'price' => 29.99, 'category' => 'electronics'], $id1);

    $bulkWrite->updateOne(
        ['name' => 'Product F'],
        ['$set' => ['name' => 'Product F', 'price' => 39.99, 'category' => 'clothing']],
        ['upsert' => true]
    );

    $bulkWrite->deleteOne(['name' => 'Product C']);

    // Execute the bulk write
    $result = $client->bulkWrite($bulkWrite);
} catch (BulkWriteCommandException $e) {
    // Get partial results (operations that succeeded before the error)
    $partialResult = $e->getPartialResult();
    if ($partialResult) {
        echo "\nPartial Results (before failure):\n";
        echo "  Inserted: " . $partialResult->getInsertedCount() . "\n";
        echo "  Modified: " . $partialResult->getModifiedCount() . "\n";
        echo "  Deleted: " . $partialResult->getDeletedCount() . "\n";
    }

    // Get write errors for specific operations that failed
    $writeErrors = $e->getWriteErrors();
    if ($writeErrors) {
        echo "\nWrite Errors:\n";
        foreach ($writeErrors as $index => $error) {
            echo "  Operation #$index: " . $error->getMessage() . "\n";
        }
    }

    // Get write concern errors if any
    $writeConcernErrors = $e->getWriteConcernErrors();
    if ($writeConcernErrors) {
        echo "\nWrite Concern Errors:\n";
        foreach ($writeConcernErrors as $index => $wcError) {
            echo "  Error #$index: " . $wcError->getMessage() . "\n";
        }
    }
    exit(1);
} catch (\Exception $e) {
    echo "Error: " . $e->getMessage() . "\n";
    exit(1);
}

You can run the script with the command:

php bulk_write_results.php

5. Realistic CSV import with BulkWrite

CSV imports are one of the most common real-world use cases for BulkWrite. Instead of inserting documents one at a time, you can use batches of 5,000 rows, for example, into a single BulkWrite operation. While the new API can handle very large operations, batching is still recommended for:

Memory efficiency (don't load entire CSV into memory).
Network reliability (smaller retries on failure).
Progress tracking and resumability.
Server resource management.

Depending on your server resources and error tolerance of your application, you can use higher batch sizes since we are not limited by the 16MB BSON response document size. The bulk write logic also splits outgoing commands into multiple batches to stay below the maximum size for BSON messages. Essentially, you don’t have to worry about sizes as long as an individual document is less than 16 MB. Our code example uses batch sizes of 5,000 rows.

In this example, you will learn how to use the new Client BulkWrite API to import data from multiple CSV files into different collections in a single bulk operation. We are using two CSV files of 10,000 rows each, both downloaded from Datablist. The CSV files are included in the repository.

The customers’ CSV has the following columns:

Index,Customer Id,First Name,Last Name,Company,City,Country,Phone 1,Phone 2,Email,Subscription Date,Website

The organizations’ CSV has the following columns:

Index,Organization Id,Name,Website,Country,Description,Founded,Industry,Number of employees

CSV row → MongoDB document mapping

A CSV row like…

…will become:

[
    'customer_id' => 1001,
    'first_name' => 'Alice',
    'last_name'  => 'Johnson',
    'email'      => 'alice@example.com',
    'company'    => '...',
    'city'       => '...',
    'country'    => '...',
    'phone_1'    => '...',
    'phone_2'    => '...',
    'subscription_date' => new MongoDB\BSON\UTCDateTime(...),
    'website'    => '...',
]

You can either map all fields or only the ones your app needs. You can even add more fields, e.g., imported_at.

The organisations’ CSV is mapped like this:

[
        'organization_id' =>'',
        'name' =>'',
        'website' =>'',
        'country' =>'',
        'description' =>'',
        'founded' => '',
        'industry' =>'',
        'number_of_employees' => '',
        'imported_at' => new UTCDateTime(),
    ]

Full CSV import script

For brevity, we have not included the full import script in this guide, but you can access it from the repository, bulk_write_csv_import.php. Below is a code snippet from the script:

// Batch size configuration
$batchSize = 5000; // Much larger than legacy API's 500-1000 // You can still go higher/lower based on memory/network

// Open CSV files
$customersHandle = fopen('customers.csv', 'r');
$orgsHandle = fopen('organizations.csv', 'r');

// Skip headers
fgetcsv($customersHandle);
fgetcsv($orgsHandle);

$bulkWrite = null;
$operationCount = 0;

while (!feof($customersHandle) || !feof($orgsHandle)) {
    // Read and add customer
    if (!feof($customersHandle)) {
        $row = fgetcsv($customersHandle);
        if ($row) {
            $bulkWrite = $bulkWrite ?? ClientBulkWrite::createWithCollection($customersCollection, [
                'ordered' => false,
                'verboseResults' => false
            ]);
            $bulkWrite = $bulkWrite->withCollection($customersCollection);
            $bulkWrite->updateOne(
                ['customer_id' => $row[0]],
                ['$set' => ['name' => $row[1], 'email' => $row[2], 'imported_at' => new UTCDateTime()]],
                ['upsert' => true]
            );
            $operationCount++;
        }
    }

    // Read and add organization
    if (!feof($orgsHandle)) {
        $row = fgetcsv($orgsHandle);
        if ($row) {
            $bulkWrite = $bulkWrite->withCollection($organizationsCollection);
            $bulkWrite->updateOne(
                ['org_id' => $row[0]],
                ['$set' => ['name' => $row[1], 'industry' => $row[2], 'imported_at' => new UTCDateTime()]],
                ['upsert' => true]
            );
            $operationCount++;
        }
    }

    // Execute batch when threshold reached
    if ($operationCount >= $batchSize) {
        $client->bulkWrite($bulkWrite);
        $bulkWrite = null;
        $operationCount = 0;
    }
}

// Execute remaining operations
if ($bulkWrite && $operationCount > 0) {
    $client->bulkWrite($bulkWrite);
}

You can run the script from the repository with the command:

php bulk_write_csv_import.php

You should see a success message, “✓ Multi-collection CSV import completed successfully!”, in your terminal if everything went well.

You should confirm the records in your MongoDB Atlas dashboard:

Why BulkWrite is ideal for CSV imports

Faster imports: Instead of sending 10,000 inserts one by one, you only send two batches of 5000.
Fewer network calls: Each bulk write = one network round-trip.
Order is preserved: Documents are inserted in the order you read them, unless you use the unordered flow.
Consistent error handling: You can log failures per batch without stopping the entire import.
Works well with validation rules: You still get detailed error messages from BulkWriteCommandException.

6. Best practices when working with MongoDB BulkWrite

To get optimum performance, reliability, and clarity from your bulk operations, follow these recommended best practices. These guidelines are commonly used in production systems that handle imports, data migrations, and large update operations.

1. Use unordered writes when order does not matter

MongoDB supports two bulk modes:

a. Ordered (default)

Operations run in sequence.
If one operation fails, the rest stop.
Ideal when order is important (e.g., dependent updates).

b. Unordered

MongoDB executes operations in parallel, in any order.
Failures do not stop the entire batch.
Much faster for large writes.

For example, we use the unordered writes in the CSV import code sample.

2. Verbose results

In the CSV import code, verboseResults is disabled. You can enable it if you are experiencing errors in your import and want to perform some debugging/auditing. Otherwise, for memory efficiency, keep it disabled for large imports.

3. Manage large batches (chunking strategy)

While BulkWrite can theoretically process thousands of operations, large batches can:

Increase memory usage.
Risk timeouts.
Make error reporting harder.

Chunking helps you:

Control memory usage.
Handle partial successes.
Ensure predictable performance.
Build resumable import systems—in case of a failure, you can resume from the batch number that failed, skipping others.

Example $batchSize = 5000;

4. Log errors for debugging failed operations

BulkWrite exceptions provide detailed information about:

Duplicate key errors.
Validation failures.
Network issues.
WriteConcern failures.
Operations that failed.

You should always wrap BulkWrite in a try...catch block and log failure details as seen in the example code blocks above, because:

BulkWrite may succeed partially, and you need to know where it failed.
Logging helps diagnose invalid data, duplicate keys, or corrupted CSV entries.
In production pipelines, logs help users retry or fix only the failed items.

5. Working with transactions for atomicity

When you need atomic guarantees, where either all operations succeed, or none are applied, then wrap your bulk write in a transaction. The MongoDB\with_transaction() helper simplifies this by handling retries for transient errors automatically. Here is a sample code snippet for working with transactions:

use function MongoDB\with_transaction;

$client = new Client($uri);

// Start a session for the transaction
$session = $client->startSession();

$customersCollection = $client->selectCollection('shop', 'customers');
$ordersCollection = $client->selectCollection('shop', 'orders');

// Use MongoDB\with_transaction() helper for automatic retry handling
with_transaction($session, function (Session $session) use ($client, $customersCollection, $ordersCollection) {
    // Create bulk write with the session
    $bulkWrite = ClientBulkWrite::createWithCollection($customersCollection, [
        'session' => $session,
        'ordered' => true
    ]);

    // Add operations that must all succeed together
    $bulkWrite->insertOne(['name' => 'Alice', 'email' => 'alice@example.com', 'balance' => 1000]);
    $bulkWrite->updateOne(
        ['name' => 'Bob'],
        ['$inc' => ['balance' => -500]]
    );

    // Switch to orders collection (same transaction)
    $bulkWrite = $bulkWrite->withCollection($ordersCollection);
    $bulkWrite->insertOne([
        'customer' => 'Bob',
        'recipient' => 'Alice',
        'amount' => 500,
        'type' => 'transfer'
    ]);

    // Execute all operations atomically
    $client->bulkWrite($bulkWrite);
});

Why use transactions with bulk writes?

Atomicity: All operations commit together or roll back on failure.
Automatic retries: The with_transaction() helper retries on transient errors (e.g., network issues), avoiding the need for manually implementing retry logic.
Cross-collection consistency: Maintain data integrity across multiple collections.

When to use transactions

Financial operations (transfers, payments)
Related data that must stay consistent (e.g., inventory + orders)
Multi-collection updates that depend on each other
Any scenario where partial writes would leave data in an invalid state

Note: For standalone deployments, transactions are only supported for single-document writes. Multi-document transactions require a replica set or sharded cluster; they are not available on standalone MongoDB deployments. For bulk writes, if a single operation cannot be successfully executed, any operations from the same bulk write that had already completed will only be rolled back if you used a transaction.

6. Additional best practices

Optimize indexing: When performing bulk updates or deletes, an indexed filter greatly speeds up execution. Additionally, avoid over-indexing. While indexes improve query speed, they can hinder write operations and consume additional disk space. Regularly review and remove unused or unnecessary indexes.
Validate data before adding to the batch: Catch “bad rows” early during CSV import rather than letting MongoDB reject them.
Use upserts carefully: They help merge data, but ensure your match filter is accurate.
Avoid mixing too many different operation types: Multiple inserts + updates + deletes are fine, but mixing extremely complex update operations may complicate debugging.
Monitor your MongoDB logs: MongoDB will log slow bulk operations or write concern issues.

Conclusion

BulkWrite is one of the most powerful additions to the MongoDB PHP library features. It proves useful when your application requires high-volume inserts, updates, and deletes. By combining multiple operations, you significantly reduce network round-trips and improve performance, especially when importing CSV data, huge system logs, cleaning up large datasets, or syncing records.

In this tutorial, we explored:

How to connect to MongoDB (Atlas or local).
How to perform bulk inserts, updates, and deletes.
How to combine multiple operations in a single batch.
How to handle results and errors.
How to build a realistic CSV importer using BulkWrite.
Best practices for performance and reliability.

With this knowledge, you’re ready to build fast, scalable data-processing scripts in PHP using MongoDB.

Next steps

Here are helpful resources to deepen your understanding:

MongoDB client bulk write docs (official)
MongoDB PHP extension documentation
MongoDB error handling/WriteConcern docs

A Simple MongoDB MCP Server Agent

MongoDB Guests — Thu, 29 Jan 2026 16:34:53 +0000

Written by Jack Woehr - SoftWoehr LLC - Seiden Group - ProCern Technology Solutions

After reading this article and exploring the associated code repository, you should be able to write your own agent which leverages the MongoDB MCP server.

The code and this article are illustrative only. This is not production code, but it is a complete working example. In contrast to our working example, in production:

You will do more for security.
You will conform to an advanced protocol such as:
- Agent Client Protocol (ACP).
- Agent2Agent (A2A).
You may use a gateway such as IBM MCP Context Forge.
You will probably containerize your entire project.

Chat with MongoDB in natural language

The MongoDB MCP server allows you to write an agent that you can use:

In conjunction with a web application.
In a multi-agent application.
To be called from a simple chat interface as implemented in our code.

We wrote the code ("we" being "me" and various agentic tasking code assistants, mostly Roo Code, some Kilo, some Cline, and some IBM Project Bob) as part of a design-code-play-repeat cycle on a larger web dashboard project (not yet released).

Architecture

All communication in this simple example is via HTTP.

Chat

The chat sends to the agent the:

Query.
Context (i.e., chat history).
Prompt.

Agent

The agent sends to the model (the AI service) the content relayed from the chat:

query
context
prompt

…and also the tools list which the agent loaded from the MongoDB MCP server when the agent started.

Model

The model examines the content from the agent and makes an MCP tool call to the agent.
The agent makes the tool call to the MongoDB MCP server.
The agent sends back to the model the result of the tool call.
The model analyzes the result and formats a text response to the query which is then returned to the agent, which relays it back to the chat.

The code

The code is on GitHub.

The agent and chat interface

After starting the MongoDB MCP server (see below), the README.md file explains how to set up and run the agent and the chat interface.

Install the required Node packages.
- npm i
Copy the .env.example to .env and edit it.
Start the agent in a terminal window.
- npm start
Start the chat in another terminal window
- npm run chat

Trying out the chat

Data model

In the illustration above, we see a portion of the validation for the pottery collection of the herwheel3_test database that we query using Google Gemini as our model provider. herwheel3_test is not a public database and is not part of the code and is merely used for illustration. Use your own database!

Gemini is smart enough to figure out to search the tags array when queried about "pots tagged as mugs," as shown in this second illustration.

Starting the MongoDB MCP server

There is a script, start_mongodb-mcp-server.py, to start the MongoDB MCP server. You do not need to use it. You may start the mongodb-mcp-server instance as you wish, but make sure it comes up on the TCP port expected by your configuration for the agent.

That Python script depends on python-dotenv, so you should create a Python virtual environment, activate it, and install that package if you want to use the script. There's even a requirements.txt file.

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Run that script with the -h flag to see the arguments.

$ ./start_mongodb-mcp-server.py  -h
usage: start_mongodb-mcp-server.py [-h] [--log-path LOG_PATH] DOTENV

Load a dotenv file into the environment and launch mongodb-mcp-server in the background.

positional arguments:
  DOTENV               Path to a dotenv-style file to load (e.g. .env)

options:
  -h, --help           show this help message and exit
  --log-path LOG_PATH  Fully qualified filepath of the log file. Default: same directory as dotenv file with name 'mongodb-mcp-server.log'

Of course, you will need to have installed and configured the MongoDB MCP server. The instructions for doing that are here, and I have provided a .env.mongodb-mcp-server.example with the code. Copy it to some other name, edit it, and provide it as the argument to start_mongodb-mcp-server.py.

We used our MongoDB Atlas installation as the target, so the server configuration we provide with the sample needs three factors:

MDB_MCP_API_CLIENT_ID=API_CLIENT_ID
MDB_MCP_API_CLIENT_SECRET=API_CLIENT_SECRET
MDB_MCP_CONNECTION_STRING=CONNECTION_STRING

The connection string is of the form MongoDB URI that we are all familiar with.
API_CLIENT_ID and API_CLIENT_SECRET are Atlas API factors. Please see MongoDB document Manage Programmatic Access to a Project to learn how to create these artifacts.

Conclusion

You should now be able to write your own agent which leverages the MongoDB MCP server, allowing you to incorporate MongoDB in your agentic mix.

A final thought about code assistants

Those developers best equipped to use code assistants are those developers who code well already. Code assistants are good, very good, and are getting better, but you are the ultimate authority on suitability, sustainability, and correctness. With the advent of these tools, our profession has changed forever and irrevocably, and will continue to change in the coming years, but our mission is still the same. Our job will not get easier, but we will be more productive.

Forem: MongoDB Guests

Async PyMongo in FastAPI

1. Introduction

Why modern APIs need asynchronous I/O

FastAPI’s design philosophy around async

Database bottlenecks & why async DB drivers matter

Introducing the new Async API in PyMongo

2. What is PyMongo?

Overview of the official MongoDB Python driver

Traditional synchronous PyMongo behavior

Blocking vs non-blocking I/O explained simply

Limitations of synchronous DB operations in async frameworks

3. What is Async PyMongo?

Overview of PyMongo’s new async API (pymongo.async_)

Key advantages over synchronous API

Relationship to Motor (Motor’s role before async PyMongo existed)

How to migrate from motor to async-pymongo and sync-pymongo to async-pymongo

4. Key Features of Async PyMongo

True async interface with awaitable operations

Connection pooling behavior

Async cursors and non-blocking iteration

5. What is FastAPI?

FastAPI’s async-first architecture

Event loop and concurrency model

Why FastAPI pairs naturally with non-blocking database drivers

Surprising performance differences when mixing sync DB code into async routes

6. Why Use Async PyMongo with FastAPI?

Avoid event-loop blocking

Handle thousands of concurrent requests

Reduce latency under load

Improve throughput for high-traffic microservices

Better CPU utilization and scaling on limited infrastructure

7. Async CRUD Operations with Examples

Insert one / insert many (async versions)

Find one / find many with async cursors

Async pagination using cursors

Update and replace operations

Delete one/many operations

Async aggregation pipelines

Error handling (duplicate keys, timeouts, network issues)

8. Performance Comparison: Sync PyMongo vs Async PyMongo

Example endpoints and profiling results

Locust Benchmark setup

Understanding event-loop blocking

Graphs and metrics discussion (Locust latency plots)

Full working snippet for readers to run

9. Common Pitfalls & How to Avoid Them

Forgetting await on Database Calls

Creating a New Client for Every Request

Returning Non-JSON-Serializable Types

Exhausting Connection Pools Under Heavy Load

Mixing Synchronous and Asynchronous Code

10. Conclusion

Research work

Comparing Python ODMs for MongoDB

ORM and ODM: Relational vs. document

The MongoDB document model

Python ODMs

Beanie

MongoEngine

ODMantic

Django

Asynchronous vs. synchronous

Conclusion

Best Practices Cheat Sheet for Flask-PyMongo

Introduction

What is MongoDB?

What is Flask?

Why Choose MongoDB with Flask?

Setup and Project Structure

Configure MongoDB connection parameters (URI, host, port, credentials) via environment variables or configuration files.

Use a single, globally shared MongoClient (or an appropriately scoped one) instead of creating a new connection per request.

Keep your database access logic, application logic (Flask routes), and configuration separate. Helps with upgrades, testing, and debugging.

extensions.py

repositories/example_repo.py

routes/example.py

app.py

Data Modeling & Schema Design

Maintain consistent field names and types in the application.

Design your document model according to query and access patterns.

Overview of PyMongo’s new async API (`pymongo.async_`)

Use a single, globally shared `MongoClient` (or an appropriately scoped one) instead of creating a new connection per request.