Forem: MongoDB

Async PyMongo in FastAPI

MongoDB Guests — Tue, 14 Apr 2026 14:53:07 +0000

This tutorial was written by Gourav Bhabesh.

1. Introduction

Why modern APIs need asynchronous I/O

Modern APIs need to be robust and be able to serve calls on the go, and minimize the instances of timeouts and server overloads while providing a good user experience during long-running jobs. Asynchronous I/O can solve one of the key issues making modern APIs: staying fast and reliable. It lets the user know the progress of their task while the API handles the requests in the background and is available for other requests. One of the key challenges in the API calls to the database is that some queries can be returned in a few milliseconds but others can take seconds (even with well indexed queries). Without asynchronous I/O in the API, this can lead to long request queues.

You can imagine this as a bakery storefront serving different goods at the same time to multiple customers. We want the orders to be filled asap without the queue becoming too big. In our example, we have multiple servers (scalable infrastructure) who take the orders which can take varying time to process and each customer with longer order times is given a status bar to know where their order’s current status. So a customer with an order that can be instantly filled (picking up a cookie) does not have to wait behind such a customer with a long processing order (baking a cheese cake).

FastAPI’s design philosophy around async

The building block of the FastAPI is ASGI (Asynchronous Server Gateway Interface) standard, unlike the older WSGI (Web Server Gateway Interface) which was synchronous in nature by default. In short, it uses Python’s async/await syntax, yielding control to the event loop, allowing servers to process other requests in the meantime, rather than blocking an entire thread. Using an event loop lets FastAPI handle thousands of concurrent requests with fewer resources. This leads to higher throughput per thread and lower latency. This automatically encourages the developer to use the async ecosystem to fully leverage the performance benefits of async programming.

Database bottlenecks & why async DB drivers matter

One of the key challenges that any database faces is the uncertainty in response times, in which the queries will be answered by the database server. Some queries can take milli-seconds and some can take seconds. If the architecture is not well built to handle this uncertainty, it can cause severe database bottlenecks. To handle this challenge, async database drivers ensure that longer requests do not become a bottleneck for other incoming requests. Initially, the async MongoDB driver was built in the library Motor. But in late 2023, Pymongo team officially decided to build the native async solution, with the goal to replace Motor as the long-term solution.

Introducing the new Async API in PyMongo

The Pymongo team has been working on the goal to replace the Motor library to give native async support in the pymongo library itself. This gives confidence on the client side to use the official library with async support. The async support was first provided in the version 4.9 in Pymongo and it has been further stabilised to the current version of 4.15.0. The new API capabilities bring the asyncio support directly to the official MongoDB driver, allowing developers to use awaitable database operations without external libraries like Motor. Combined with the FastAPI framework, which already supports await/async capability, Pymongo can now be natively used to build high-throughput, non-blocking services.

2. What is PyMongo?

Overview of the official MongoDB Python driver

MongoDB has provided Pymongo as the official Python driver to allow the applications to communicate with MongoDB clusters, providing production-tested API to perform CRUD operations, manage indexes, run aggregation pipelines and handle advanced features such as transactions and change streams. Pymongo was traditionally built on a synchronous I/O model, meaning every database call made will block the executing thread until the MongoDB server does not return a response leading to database bottlenecks.

Traditional synchronous PyMongo behavior

The key behaviours of the traditional Pymongo library, which we should be aware of, to appreciate the addition of the native support of asynchronous, are that synchronous operations block Python threads until the operation completes or a network error or timeout occurs.. This leads to another behaviour - sequential execution, blocking all requests until and unless the previous request is not complete. This made the MongoClient “Thread-Safe” allowing for multi-threading applications to be written, but each thread still performs blocking database operations.

In our example, of the baking shop, it would be like each server is concurrently serving the customers but each server is waiting for the order to complete rather than handing a token to wait, while serving the next customer request which might be served in much less time.

Blocking vs non-blocking I/O explained simply

Blocking I/O means the application must wait for the database operation to finish or network timeout before it will allow anything else to continue. On the other hand, non-blocking I/O allows requests to continue flowing to the database while previous requests are still being processed. In addition, the async drivers provide a way to receive the status of the request, giving power to the developer to handle long operations and create a better user-experience.

Limitations of synchronous DB operations in async frameworks

Using synchronous DB operations in the async framework is one of the key mistakes made by many clients of MongoDB, part of the reason was the non-availability of official Python drivers that actually supported asynchronous behavior. If the synchronous Pymongo driver is used inside an async framework such as FastAPI, these blocking calls freeze the event loop, preventing the server from processing other incoming requests and significantly reducing concurrency. As a result, applications that rely on synchronous database drivers inside async frameworks often experience increased latency, request queuing, and poor scalability under load, even if the rest of the code is written using async and await.

Imagine our baking shop upgraded its entire framework to be non-synchronous. The cooks can cook multiple dishes at the same time, but the servers are still the same, processing sequential orders and not taking new orders until the previous order has been delivered. The cook is having an easy day!!

3. What is Async PyMongo?

Overview of PyMongo’s new async API (`pymongo.async_`)

The official MongoDB team started working on the native-async support in the Pymongo driver in version4.9, where it was first released as beta version in late 2023. As of writing this article, the pymongo version 4.15 has been released and has been stable since version 4.13. With the pymongo.With async support, we can create an AsyncMongoClient object similar to the MongoClient object we created in the previous synchronous call. The Client provides fully awaitable database operations such as insert_one, find_one, update_one, and aggregation pipelines, allowing MongoDB I/O to integrate natively with Python’s event loop.

Key advantages over synchronous API

Compared to the traditional synchronous API, async PyMongo significantly improves scalability and responsiveness under concurrent workloads. Because database calls no longer block the event loop, asynchronous framework applications can continue handling new requests while waiting for MongoDB responses, resulting in lower latency, higher throughput, and better CPU utilization—especially in high-traffic microservices

Relationship to Motor (Motor’s role before async PyMongo existed)

Before async Pymongo, motor was the de facto library to be used when developers needed asynchronous behavior when communicating with the MongoDB clusters. Motor did this by exposing a coroutine-based API, allowing applications to switch context and perform other tasks while waiting for I/O operations. However, Motor was not a separate driver from scratch; it was designed as a wrapper around the core synchronous PyMongo library. It used Tornado’s or asyncio’s event loop, wrapping Pymongo’s operations asynchronously, effectively allowing the main event loop to remain unblocked. Motor was specifically created to support asynchronous web frameworks, initially Tornado and later adding support for asyncio. Performance comparison between motor and async pymongo can be found on the official documentation here.

How to migrate from motor to async-pymongo and sync-pymongo to async-pymongo

Migrating from the motor-based asynchronous library to the async-pymongo library is simple and is well documented by MongoDB here. In simple terms, one needs to change the driver call to the database using the AsyncMongoClient and add the keyword await in front of the database operations.

When migrating from sync Pymongo to async Pymongo, keep the following important points from the official MongoDB documentation in mind:
· To convert an AsyncCursor to a list, you must use the asynchronous cursor.to_list() method.
· The AsyncCollection.find() method in the PyMongo Async API is synchronous, but returns an AsyncCursor.
To iterate through the cursor, you must use an async for loop.
· The AsyncMongoClient object does not support the connect keyword argument.
· You cannot share AsyncMongoClient objects across threads or event loops.
· To access a property or method of a result returned by an asynchronous call, you must properly wrap the call in parentheses, as shown in the following example:

id = (await posts.insert_one(doc)).inserted_id

4. Key Features of Async PyMongo

True async interface with awaitable operations

The biggest advantage of the async Pymongo library for developers is its awaitable operations. Functions such as insert_one(), find_one(), update_one() and aggregate() return objects that can wait rather than blocking the results. This fundamentally decouples the event loop from the operation, meaning the database operation running on the database side yields the control back to the event loop, allowing FastAPI to process the other incoming requests without blocking the thread.

In our baking shop example, the server has been decoupled from the chef, while the previous order is getting cooked in the kitchen, the server takes in a new order without ceasing the operation and blocking the thread and allowing for incoming requests to be fulfilled !! This model of working is fundamentally different from the traditional synchronous PyMongo, where every database call ties up a worker thread until completion, blocking the thread from serving the next call when under load.

Connection pooling behavior

The connection pool behaviour is quite an interesting one here. On the surface it appears both the synchronous and the asynchronous Pymongo connection-poling implementation look the same (parameters available - maxPoolSize, minPoolSize, waitQueueTimeoutMS etc.) and indeed rely on the same MongoDB C driver. However, async applications are actually able to spawn thousands of concurrent operations and grab many connections from the pool at once, whereas, in the synchronous application each request occupies a python worker thread, holding a connection from the pool waiting for MongoDB. This naturally limits concurrency as the application is limited by the number of workers, effectively meaning the full pool size can be never used optimally.

Imagining in our world of baking shop, we have an army of chefs to process the orders, the async model allows the server to grab hold of any available chef to accept a new order and move on to the next one, whereas in the synchronous setup the server will wait for the chef to finish the previous order before moving to the new order.

Async cursors and non-blocking iteration

One of the primary challenges that any database faces is returning results to query with a large result set. The traditional PyMongo driver already returns a cursor object rather than loading all the documents into the memory at once. In async PyMongo, these cursors are fully asynchronous in nature as well, resulting in iteration without blocking the event loop. Developers now have the choice of using the await cursor.to_list() or iterating over the cursor object using the async for. This will allow data to be streamed in batches while the application is handling incoming requests. This feature is of significant importance when we have endpoints that return large datasets, background processing jobs and real-time analytics workloads.

5. What is FastAPI?

FastAPI’s async-first architecture

FastAPI is a high-performance web framework that is production-ready and designed for building APIs in python, with roots embedded in asynchronous programming. It embraces Python’s asyncio model as its core principle rather than treating it as optional. This design choice gives FastAPI leverage for I/O bound workloads such as db access, external API calls and real-time data streaming.

Other traditional web frameworks default to synchronous request handling, whereas FastAPI encourages using asynchronous endpoints (async def) wherever possible. When an endpoint is declared using the async keyword, FastAPI uses the event loop execution rather than a worker thread. Now as long as those requests involve the non-blocking I/O operations, a single worker process can handle many requests concurrently. This is where FastAPI’s async-first architecture shines. When the entire request path - including the database access - is non blocking, each request becomes super efficient. Any stack in the request that blocks the event loop will make the FastAPI’s concurrency model diminish significantly.

Event loop and concurrency model

At the heart of FastAPI’s concurrency model is Python’s asyncio event loop. The event loop manages the execution of many coroutines, switching between them whenever one is waiting for I/O, such as a network response from MongoDB. This cooperative multitasking enables a single process to efficiently handle thousands of simultaneous requests without spawning thousands of threads. However, this model only works as intended when all long-running operations are awaitable. If a synchronous operation blocks the event loop — for example, a traditional synchronous PyMongo database call — the entire server becomes unresponsive to other requests until that operation completes.

Why FastAPI pairs naturally with non-blocking database drivers

As explained above the core principle of FastAPI is to yield the control back to the event loop and any blocking I/O operation in the chain of the operations sent by the request will be like throwing a wrench on the gears. With the pairing of the non-blocking db driver such as the async pymongo, when a FastAPI endpoint awaits a MongoDB operation, control is yielded back to the event loop, allowing for the next set of requests to be processed in parallel. With the alignment between the db driver and the base principle of FastAPI to be async, we eliminate the friction and make the full request truly asynchronous in nature and hence enable FastAPI to scale efficiently under high concurrency.

Surprising performance differences when mixing sync DB code into async routes

A common pitfall for developers writing code in FastAPI is mixing the synchronous PyMongo calls inside the async def endpoints. On the surface it may appear to work, but it breaks the core principle of the FastAPI to be truly asynchronous. One single blocking db call and the entire event loop gets stalled as other requests will have to queue behind it - even when the server has many workers available.

In practice, this means that an API written with FastAPI but using synchronous PyMongo can perform worse under load than a fully synchronous application with multiple worker threads. By contrast, pairing FastAPI with Async PyMongo preserves the framework’s intended concurrency benefits, leading to better throughput, lower latency for typical requests, and more predictable scaling behavior. In large applications, having the capability to predict the scalability of infrastructure and have a control on the concurrency is of paramount importance.

6. Why Use Async PyMongo with FastAPI?

Avoid event-loop blocking

Using FastAPI with Asynchronous Pymongo correctly is essential if we want to avoid event-loop blocking. The requests from the async-first framework like FastAPI, share the same event-loop within each worker process. Meaning any operation in this request if becomes synchronized will lead to blocking of the entire event-loop, which high-concurrency systems want to avoid at all cost.

By using Async Pymongo, each db operation becomes awaitable rather than blocking in nature, ensuring a slow db operation does not freeze the entire application, allowing other requests to get processed.

Handle thousands of concurrent requests

FastAPI is capable of handling thousands of concurrent requests per worker process, but designing the entire chain of operations in an asynchronous manner is a must to achieve this. One of the primary mistakes that are made in the coding side is missing the async and await keywords in the db operation call, this breaks the core principle of FastAPI leading to slowness. With correct implementation we can make the infrastructure more cost-effective and efficient as compared to a purely synchronous system which would not be scalable in the long run as more threads are required to handle the huge volume of requests.

Reduce latency under load

With modern tech stacks, both horizontal and vertical scaling is a must and systems built with such capabilities can withstand the test of time. And today's system should be capable enough to handle varying loads. With FastAPI - async nature is built in-core principle of the library itself and when combined with Async Pymongo it allows for the potential blocking db operations to be handled asynchronously. This leads to lower median latency for requests, especially I/O intensive ones. In section 8, we will see how the sync/async jobs specifically prove this point, when we compare the median latency of thousands of concurrent requests.

Improve throughput for high-traffic microservices

In the world of microservices where responsiveness and scalability are critical Async Pymongo dramatically improves the performance by removing the blocker of such requests to the db layer. Combined with async frameworks such as FastAPI, Async Pymongo is superbly placed to serve real-time applications, analytics dashboards or user-facing services.

Better CPU utilization and scaling on limited infrastructure

One of the vital nerves of any IT infrastructure is cost-efficiency and with Devops team’s mantra - “doing more with less”, fits perfectly with the Async Pymongo. Making the most of the available resources, async architecture uses the same thread without blocking incoming requests by handing over the control back to the single event loop. Instead of maintaining hundreds or thousands of idle threads waiting on the I/O, FastAPI with Async Pymongo exploits this single event loop allowing it to switch between tasks as needed. In cloud based infrastructure, where compute resources are directly tied to cost, it can lead to meaningful savings without sacrificing performance.

7. Async CRUD Operations with Examples

This section demonstrates the common MongoDB operations asynchronously using the pymongo.async_ inside FastAPI.

All examples assume:

from pymongo.async_ import AsyncMongoClient 
from bson import ObjectId 
client = AsyncMongoClient (MONGO_URI) db = client["appdb"] 
users = db["users"]

Insert one / insert many (async versions)

Insert one document example:

@app.post("/users") 
async def create_user(user: dict):
  result = await users.insert_one (user) 
  return {"inserted_id": str(result.inserted_id)}

Insert many documents example:

@app.post("/users/bulk") 
async def create_many(users_in: list[dict]): 
  result = await users.insert_many (users_in) 
  return {"inserted_ids": [str(i) for i in result.inserted_ids]}

Find one / find many with async cursors

Find one document example:

@app.get("/users/{email}")
async def get_user(email: str): 
 return await users.find_one ({"email": email})

Find many documents example:

@app.get("/users") 
async def list_users():
  cursor = users.find ({}) 
  return await cursor.to_list(length=100)

Async pagination using cursors

@app.get("/users/page") 
async def paginated_users(last_id: str | None = None, limit: int = 20): 
  query = {"_id": {"$gt": ObjectId(last_id)}} if last_id else {} 
  cursor = users.find(query).sort("_id", 1).limit (limit) 
  results = await cursor.to_list(length=limit) 

  return { 
    "results": results,
    "next_cursor": str(results[-1]["_id"]) if results else None, 
  }

(Avoid skip() for large collections; use _id cursor pagination instead.)

Update and replace operations

Update one document example:

@app.patch("/users/{email}")
async def update_user(email: str, changes: dict):
  result = await users.update_one(
    {"email": email},
    {"$set": changes},
  ) 
  return {"matched": result.matched_count, "modified":
result.modified_count}

Replace one document example:

@app.put("/users/{email}")
async def replace_user(email: str, new_doc: dict):
  result = await users.replace_one({"email": email}, new_doc) 
  return {"matched": result.matched_count}

Delete one/many operations

Delete one document example:

@app.delete("/users/{email}") 
async def delete_user(email: str):
  result = await users.delete_one ({"email": email}) 
  return {"deleted": result.deleted_count}

Delete many documents example:

@app.delete("/users/inactive")
async def delete_inactive():
  result = await users.delete_many ({"active": False}) 
  return {"deleted": result.deleted_count}

Async aggregation pipelines

@app.get("/users/stats")
async def user_stats():
  pipeline = [
    {"$group": {"_id": "$country", ""count"": {"$sum": 1}}},
    {"$sort": {"count": -1}},
  ] 
cursor = await users.aggregate (pipeline)
return await cursor.to_list(length=100)

Error handling (duplicate keys, timeouts, network issues)

Duplicate keys:

from pymongo.errors import DuplicateKeyError
from fastapi import HTTPException

try: 
  await users.insert_one({"email": "a@b.com"})
except DuplicateKeyError:
  raise HTTPException(status_code=400, detail="User already exists")

Timeout/network issues:

from pymongo.errors import ExecutionTimeout, ConnectionFailure

try:
  await users.find_one({"email": "a@b.com"})
except ExecutionTimeout:
  raise HTTPException(status_code=504, detail="DB timeout")
except ConnectionFailure:
  raise HTTPException(status_code=503, detail="DB unreachable")

8. Performance Comparison: Sync PyMongo vs Async PyMongo

In this section, we break down the high-stakes battle between the traditional Synchronous (Sync) driver and the modern Asynchronous (Async) driver using the large-scale sample_airbnb dataset. This isn't just a syntax preference; it's the difference between a server that remains responsive under fire and one that collapses under its own weight.

Example endpoints and profiling results

To push these drivers to the limit, we implemented a Complex Aggregation task. Instead of a simple "Ping," we forced each request to filter high-rated Airbnb listings, sort them by price, and project complex nested fields.

Sync Logic: Utilizes list(users.aggregate(pipeline)), which occupies a physical thread for the duration of the database round-trip and data serialization.

Async Logic: Utilizes await (await users.aggregate(pipeline)).to_list().
The server yields control during the database "wait time," allowing the single-threaded event loop to process other incoming users.

Locust Benchmark setup

We subjected both implementations to a "Trial by Fire" using Locust:

User Load: Scaled up to 500 concurrent users.

Hardware Constraint: We intentionally limited the Sync thread pool to 1 thread to simulate extreme resource exhaustion and observe the "Queueing Effect."

Target: listingsAndReviews collection in MongoDB Atlas.

Understanding event-loop blocking

The results highlight a fundamental architectural divide. In the Sync model, when the thread pool is exhausted (in our case, limited to 1), every new user must wait for the person in front of them to finish their entire database operation. This creates a "Single Lane Bridge" bottleneck. In the Async model, even with thousands of users, the event loop acts as a high-speed scheduler. It sends the request to MongoDB and immediately moves to the next user, effectively "stacking" thousands of concurrent database operations without needing a thread for each one.

Graphs and metrics discussion (Locust latency plots)

Table: Sync stats showing the performance on one thread
Graph: Sync response times

Table: Async stats showing the performance

Graph: Async response times

Looking at our Locust metrics, the disparity is stark:
| Metric | Sync (1 Thread) | Async (Event Loop) |
|-------------------|-----------------|--------------------|
| Max RPS | ~48.8 | ~113.4 |
| Median Latency | ~8,700 ms | ~2,800 ms |
| 95th-Percentile | ~9,400 ms | ~4,100 ms |
| Throughput Driver | Thread-bound | DB / Network-bound |

Sync Graphs: Notice the staircase effect in response times.
As users increase, the latency climbs linearly because users are waiting in a queue. The RPS is "hard-capped" because the single thread can only process so many documents per second.

Async Graphs: The response times remain significantly lower and more stable.
Even as we ramped to 500 users, the 95th percentile hovered around 4 seconds compared to nearly 10 seconds for Sync. This proves that Async manages high-concurrency significantly better, even when the individual database query is complex.

Full working snippet for readers to run

Ready to see these results on your own machine?
We have provided a complete repository containing the FastAPI implementation, the AnyIO thread limiter configuration, and the Locust scripts used for these benchmarks.

Demo work - https://github.com/gouravmongodb/fastapi_pymongo_demo

9. Common Pitfalls & How to Avoid Them

Even with Async PyMongo, it is easy to introduce subtle performance or correctness issues if you are not careful. Below are the most common pitfalls developers encounter and how to avoid them.

Forgetting await on Database Calls

Because Async PyMongo methods are awaitable, forgetting await will not raise an immediate syntax error but will return a coroutine object instead of executing the operation. This can lead to silent failures or unexpected behavior in your API responses. Always double-check that every database operation inside an async def route is properly awaited.

Creating a New Client for Every Request

Instantiating AsyncMongoClient inside each request handler is a major anti-pattern. It creates unnecessary TCP connections, increases latency, and quickly exhausts resources under load. Instead, create a single shared client at application startup (using FastAPI lifespan or startup events) and reuse it across all requests.

Returning Non-JSON-Serializable Types

MongoDB types like ObjectId, datetime, or Decimal128 are not directly JSON serializable. Returning raw MongoDB documents from FastAPI will cause serialization errors. The recommended approach is to either convert these types to strings or map your documents to Pydantic models that handle serialization correctly.

Exhausting Connection Pools Under Heavy Load

Async applications can issue far more concurrent database operations than synchronous ones, which makes connection pool tuning critical. If maxPoolSize is too low, requests will start queuing inside the driver, increasing latency. Monitor Atlas metrics and adjust maxPoolSize, minPoolSize, and maxConnecting based on your workload.

Mixing Synchronous and Asynchronous Code

Using synchronous PyMongo calls inside async def routes (or vice versa) undermines FastAPI’s concurrency model and can block the event loop. Choose one model—preferably Async PyMongo for FastAPI—and stick to it consistently throughout your application.

10. Conclusion

Asynchronous database access is a cornerstone of modern, scalable API design. In an async-first framework like FastAPI, using a non-blocking database driver is not a luxury—it is a necessity for handling high concurrency, reducing latency, and making efficient use of resources.

With the stabilization of pymongo.async_ in PyMongo 4.13, Async PyMongo has become the recommended and officially supported way to use MongoDB with FastAPI. It provides a native, awaitable interface that aligns with FastAPI’s event loop, simplifies dependencies, and brings the future of the MongoDB Python ecosystem under a single driver.

Synchronous PyMongo still has a place in the ecosystem — particularly for simple scripts, batch jobs, or applications that are not built around async frameworks. However, for production web services, real-time applications, and microservices that must scale, Async PyMongo is the better choice.

Ultimately, performance depends not just on “sync vs async,” but on how you design your workload, tune your connection pool, index your collections, and scale your MongoDB cluster. When used thoughtfully, Async PyMongo with FastAPI offers a powerful, maintainable, and future-proof foundation for building high-performance Python APIs.

Research work

· https://pymongo.readthedocs.io/en/stable/changelog.html#changes-in-version-4-9-2024-09-18
· From Pymongo driver v4.9.0 introduced the beta version of async function.
· From Pymongo driver v4.13.0 - asynchronous API is now stable and no longer in beta.
· https://www.mongodb.com/docs/languages/python/pymongo-driver/current/reference/migration/#migrate-to-pymongo-async
· https://www.mongodb.com/docs/languages/python/pymongo-driver/current/reference/migration/#performance-benchmarks

Build a Movie Watchlist with Node.js, TypeScript, and MongoDB

Nic Raboy — Mon, 13 Apr 2026 15:19:10 +0000

Almost every modern web application will need a REST API for a client to talk to, and in almost every scenario, that client is going to expect JSON. The best developer experience is a stack where you can stay in JSON-shaped data end to end, without awkward transformations in the middle.

Take MongoDB, Express Framework, and Node.js as an example.

Express receives HTTP requests and sends responses. MongoDB sits in the middle and stores documents. The client can send JSON to your routes, your routes can send documents to MongoDB, and MongoDB can hand BSON back that maps naturally to what you serialize in the response. That works well because MongoDB is a document database. When you also want text search over fields like title and plot, MongoDB Search gives you a $search stage in an aggregation pipeline on the same cluster, so you are not bolting on a separate search system just to power a search box.

In this tutorial, we'll see how to build a small movie watchlist API using TypeScript and MongoDB. We'll explore a few different schema design opportunities and make use of MongoDB Search for full-text search.

The Prerequisites

Prior to starting this tutorial, you'll need a few things in place:

A MongoDB Atlas instance (M0 is fine) with the sample_mflix sample dataset loaded and available.
Node.js 22+

The expectation is that your cluster is already provisioned with credentials and network access rules, and that the sample dataset is loaded. If you need help deploying or configuring MongoDB Atlas, check out the MongoDB documentation for getting started.

We'll be working with an empty project, so to kick things off, you might want to run the following commands:

mkdir movie-watchlist-api
cd movie-watchlist-api
npm init -y

The above commands will create a new project directory and initialize it for Node.js by creating a package.json file.

There are a few dependencies that we need to install. They can be installed by executing the following commands:

npm install express mongodb dotenv

In the above, we're installing Express Framework and the MongoDB Node.js Driver, plus a package that will allow us to read from a .env file for configuration variables.

For TypeScript, the tsx runner (useful in development with watch mode), and the Express and Node type packages, run:

npm install --save-dev typescript tsx @types/express @types/node

To run this project in development mode, you'd execute tsx watch src/app.ts from your command line. However, that won't be possible until later in the tutorial.

Add a TypeScript Configuration File

Because we're using TypeScript, we need a tsconfig.json at the root of the project. Create the file with something along the following lines:

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "CommonJS",
    "moduleResolution": "node",
    "outDir": "dist",
    "rootDir": "src",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true,
    "resolveJsonModule": true,
    "declaration": true,
    "declarationMap": true,
    "sourceMap": true
  },
  "include": ["src/**/*"]
}

The goal here is to emit CommonJS into dist/ from everything under src/, which keeps production as simple as node dist/app.js after tsc.

Understanding the Source Tree of the Project

With the tooling in place, it helps to know where things will live. This project will use the following foundation:

src/config.ts
src/db.ts
src/types/watchlist.ts
src/routes/movies.ts
src/routes/watchlist.ts
src/app.ts

Each file will serve a different purpose that we'll explore throughout the remainder of the tutorial. Go ahead and create the src, src/routes, and src/types directories so you're ready to paste code as we go.

Define and Configure the Environment Variables for the Project

We're going to need connection details before the Express Framework application can talk to MongoDB Atlas. Add the following to a .env file at the project's root:

MONGODB_URI=mongodb+srv://<USERNAME>:<PASSWORD>@<HOST>/
PORT=3000
SEARCH_INDEX_NAME=default

Make sure to add this file to your project's .gitignore if you're using version control, since the MONGODB_URI is considered sensitive information.

Rather than read process.env in every route file of the project, we'll define a small configuration module that validates what is required and exports plain values.

Open src/config.ts and include the following code:

import "dotenv/config";

function requiredEnv(name: string): string {
    const value = process.env[name];
    if (!value?.trim()) {
        throw new Error(`Missing required environment variable: ${name}`);
    }
    return value.trim();
}

export const config = {
    mongoUri: requiredEnv("MONGODB_URI"),
    port: Number(process.env.PORT) || 3000,
    dbName: "sample_mflix",
    moviesCollection: "movies",
    watchlistCollection: "mflix_watchlist",
    searchIndexName: process.env.SEARCH_INDEX_NAME?.trim() || "default",
};

The import "dotenv/config" line means that as soon as something imports config.ts, your .env values are loaded. Database and collection names are hard-coded to sample_mflix, movies, and mflix_watchlist, so this tutorial stays aligned with the sample dataset.

Connecting to MongoDB with a Singleton Database Class

You could open a new MongoClient on every request, but that usually is not what you want. It adds latency, wastes resources, and makes graceful shutdown harder. The pattern we'll use instead is a singleton where we'll connect once when the application boots, reuse that client for every request, and close it when the process exits.

With the environment variables already in config.ts, open src/db.ts and include the following code:

import { MongoClient, type Db } from "mongodb";
import { config } from "./config";

let singleton: Database | null = null;

export class Database {
    private constructor(
        private readonly mongoClient: MongoClient,
        readonly db: Db
    ) {}

    static async connect(): Promise<Database> {
        if (singleton) return singleton;
        const mongoClient = new MongoClient(config.mongoUri, {
            appName: "devrel-tutorial-typescript-moviewatchlist",
        });
        await mongoClient.connect();
        const db = mongoClient.db(config.dbName);
        singleton = new Database(mongoClient, db);
        await singleton.ensureWatchlistIndexes();
        return singleton;
    }

    static get instance(): Database {
        if (!singleton) {
            throw new Error("Database not connected");
        }
        return singleton;
    }

    async close(): Promise<void> {
        await this.mongoClient.close();
        singleton = null;
    }

    private async ensureWatchlistIndexes(): Promise<void> {
        const watchlistCollection = this.db.collection(
            config.watchlistCollection
        );
        await watchlistCollection.createIndex({ movieId: 1 }, { unique: true });
    }
}

In the connect method, we have the following logic:

static async connect(): Promise<Database> {
    if (singleton) return singleton;
    const mongoClient = new MongoClient(config.mongoUri, {
        appName: "devrel-tutorial-typescript-moviewatchlist",
    });
    await mongoClient.connect();
    const db = mongoClient.db(config.dbName);
    singleton = new Database(mongoClient, db);
    await singleton.ensureWatchlistIndexes();
    return singleton;
}

If singleton is already set, we return immediately and avoid opening another connection. Otherwise we create MongoClient with config.mongoUri and an appName that shows up in MongoDB Atlas, await connect(), take a Db reference for config.dbName (here sample_mflix), assign singleton, and call ensureWatchlistIndexes before returning. Doing the index step before the HTTP server starts listening means every route can assume the watchlist collection already has a unique index on movieId.

The ensureWatchlistIndexes function creates a unique index on movieId in mflix_watchlist, so MongoDB rejects a second insert for the same movie and the watchlist POST handler can reason about "no row", "inactive row", and "active row" without duplicate documents.

Define the Watchlist Document Type for TypeScript

Before we get too far into the routes, it's a good idea to describe the document shape we plan to store in mflix_watchlist. TypeScript will not infer that for you.

In src/types/watchlist.ts, add the following:

import type { ObjectId } from "mongodb";

export interface WatchlistDoc {
    _id: ObjectId;
    movieId: ObjectId;
    title: string;
    poster: string | null;
    rating?: number;
    active: boolean;
    createdAt: Date;
    updatedAt: Date;
}

We're denormalizing title and poster onto the watchlist document when the user adds or reactivates a movie, so a client can render the list without joining back to movies on every GET. The advantage of this approach is that we're making our queries faster by not doing a $lookup (JOIN) operation unless we absolutely have to. When listing all items in a watchlist, it is probably not common to need details such as the actor cast list, so we're only including the fields we absolutely need alongside the id value in case we want to expand the details later with a single fetch operation.

Searching for Movies within the Catalog with MongoDB Search

To search for movies in our application, it makes sense to leverage MongoDB Search for its full-text search capabilities. Before we can do that, we need to apply a search index on the collection and fields we want to search through.

Take the following MongoDB Search index:

{
    "mappings": {
        "dynamic": false,
        "fields": {
            "title": {
                "type": "string"
            },
            "plot": {
                "type": "string"
            }
        }
    }
}

In the above index, which is different from the unique index we created earlier, we are saying we only want to be able to use full-text search on the title and plot fields of a document.

To apply the index, navigate to MongoDB Atlas and make use of the Search & Vector Search navigation item. When prompted in the index creation process, make sure to choose sample_mflix as the database and movies as the collection. You can name the index "default" to match what we have in our configuration, or name it whatever makes sense to you. Just make sure to update the environment variables configuration if you change the name.

Open the project's src/routes/movies.ts file and include the following:

import { Router, type IRouter } from "express";
import { config } from "../config";
import { Database } from "../db";

export function moviesRouter(): IRouter {
    const router = Router();

    router.get("/search", async (req, res, next) => {
        try {
            const searchQuery = String(req.query.q ?? "").trim();
            if (!searchQuery) {
                throw new Error(
                    'Query parameter "q" is required and cannot be empty'
                );
            }

            const pipeline = [
                {
                    $search: {
                        index: config.searchIndexName,
                        text: {
                            query: searchQuery,
                            path: ["title", "plot"],
                        },
                    },
                },
                { $limit: 10 },
                {
                    $project: {
                        _id: 1,
                        title: 1,
                        poster: 1,
                        year: 1,
                    },
                },
            ];

            const database = Database.instance.db;
            const moviesCollection = database.collection(
                config.moviesCollection
            );
            const results = await moviesCollection
                .aggregate(pipeline)
                .toArray();

            res.json({ results });
        } catch (err) {
            next(err);
        }
    });

    return router;
}

The router above has a single GET endpoint that we'll use for searching. It expects a q query parameter to exist in our requests, where this parameter represents a search query.

With a MongoDB aggregation pipeline, we can use $search as our first stage, specifying the index name, the search query, and the fields we wish to search against. For the sake of this example, the second pipeline stage will limit our results to no more than 10 results. Finally, we end our pipeline with a $project stage that allows us to define what fields are actually important to us for returning to the client. In this example, we plan to return the _id, title, poster, and year fields for every document matched.

We are only using MongoDB Search for finding movies we want to add to our watchlist. We need to create a route and a series of endpoints for managing our watchlist next.

Building a Movie Watchlist with Ratings

The watchlist will make up most of our application logic for this tutorial. We want to accomplish the following:

Add movies based on id to the watchlist, with certain fields, as long as the movie doesn't already exist in the list.
Add ratings to movies in the watch list.
Soft-delete (deactivate) movies in the watchlist, but not actually removing the document from the collection.
Remove ratings from movies in the watchlist.

In the src/routes/watchlist.ts file, let's use the following as a starting point:

import { Router, type IRouter } from "express";
import { ObjectId } from "mongodb";
import { config } from "../config";
import { Database } from "../db";
import type { WatchlistDoc } from "../types/watchlist";

export function watchlistRouter(): IRouter {
    const router = Router();

    return router;
}

There aren't any endpoints for now, but we're going to add and make sense of each one at a time.

Our most complex endpoint will be the POST endpoint responsible for adding movies to the watchlist. You can find this endpoint below.

router.post("/", async (req, res, next) => {
    try {
        const database = Database.instance.db;
        const movieId = new ObjectId(String(req.body?.movieId));
        const watchlistCollection = database.collection(config.watchlistCollection);
        const moviesCollection = database.collection(config.moviesCollection);

        const existingWatchlistDoc = await watchlistCollection.findOne({
            movieId,
        });

        if (existingWatchlistDoc) {
            const activeEntry = existingWatchlistDoc as WatchlistDoc;
            if (activeEntry.active) {
                res.status(200).json({
                    document: activeEntry,
                    alreadyActive: true,
                    reactivated: false,
                });
                return;
            }

            const now = new Date();
            await watchlistCollection.updateOne(
                { movieId },
                { $set: { active: true, updatedAt: now } }
            );
            const reactivated: WatchlistDoc = {
                ...activeEntry,
                active: true,
                updatedAt: now,
            };
            res.status(200).json({
                document: reactivated,
                alreadyActive: false,
                reactivated: true,
            });
            return;
        }

        const movieFromCatalog = await moviesCollection.findOne(
            { _id: movieId },
            { projection: { title: 1, poster: 1 } }
        );
        if (!movieFromCatalog) {
            throw new Error("Movie not found in sample_mflix");
        }
        const { title, poster } = movieFromCatalog;
        const now = new Date();
        const newWatchlistDocument = {
            movieId,
            title,
            poster,
            active: true,
            createdAt: now,
            updatedAt: now,
        };
        const insertResult = await watchlistCollection.insertOne(
            newWatchlistDocument
        );
        const document: WatchlistDoc = {
            _id: insertResult.insertedId,
            ...newWatchlistDocument,
        };
        res.status(201).json({
            document,
            alreadyActive: false,
            reactivated: false,
        });
    } catch (err) {
        next(err);
    }
});

To break down each stage of the above endpoint, we're going to start with the initial query:

const existingWatchlistDoc = await watchlistCollection.findOne({
    movieId,
});

The movieId should have been passed in with the request payload. It should represent a valid movie id from the movies collection. After executing the query, we check to see if results came back, and if they did, it means this particular movie is already in the watchlist. If the movie is already in the watchlist and it is in the active state, we can just return that information now. If it is in the watchlist, but inactive, we want to reactivate it and update the timestamp before returning it.

Finally, we have this code within the endpoint:

const movieFromCatalog = await moviesCollection.findOne(
    { _id: movieId },
    { projection: { title: 1, poster: 1 } }
);
if (!movieFromCatalog) {
    throw new Error("Movie not found in sample_mflix");
}
const { title, poster } = movieFromCatalog;
const now = new Date();
const newWatchlistDocument = {
    movieId,
    title,
    poster,
    active: true,
    createdAt: now,
    updatedAt: now,
};
const insertResult = await watchlistCollection.insertOne(
    newWatchlistDocument
);
const document: WatchlistDoc = {
    _id: insertResult.insertedId,
    ...newWatchlistDocument,
};
res.status(201).json({
    document,
    alreadyActive: false,
    reactivated: false,
});

If the movie is not in the watchlist already, we first need to query the movies collection for information about the movie based on the provided movieId value. With the title and poster information, we can construct a new watchlist document and then insert it into our collection.

For this particular project, we won't worry about keeping our movie posters up to date. We're just going to assume they never change.

The next endpoint we'll look at is a GET endpoint for listing all the movies in our watchlist:

router.get("/", async (req, res, next) => {
    try {
        const database = Database.instance.db;
        const raw = req.query.active;
        const activeOnly =
            raw === undefined
                ? true
                : !["false", "0"].includes(String(raw).toLowerCase());

        const watchlistCollection = database.collection(
            config.watchlistCollection
        );
        const items = await watchlistCollection
            .find({ active: activeOnly })
            .sort({ updatedAt: -1 })
            .toArray();
        res.json({ active: activeOnly, items });
    } catch (err) {
        next(err);
    }
});

This particular endpoint will accept an active query parameter, allowing the client to request watchlist items that are either active or have been soft-deleted (inactive).

When we run our MongoDB query, we are sorting by the updatedAt timestamp so the newest additions to the watchlist appear first. We aren't doing a $project here because our documents are small, but you could easily define which fields are important to you for returning.

The next endpoint is our PATCH endpoint, which is responsible for assigning a user rating to a movie in the watchlist:

router.patch("/:movieId/rating", async (req, res, next) => {
    try {
        const database = Database.instance.db;
        const movieId = new ObjectId(req.params.movieId);
        const stars = Number(req.body?.stars);
        if (!Number.isInteger(stars) || stars < 1 || stars > 5) {
            throw new Error("Rating must be an integer from 1 to 5");
        }

        const watchlistCollection = database.collection(
            config.watchlistCollection
        );
        const now = new Date();
        const updateResult = await watchlistCollection.updateOne(
            { movieId },
            { $set: { rating: stars, updatedAt: now } }
        );
        if (updateResult.matchedCount === 0) {
            throw new Error(
                "Movie is not on the watchlist; only watchlist movies can be rated"
            );
        }
        const updatedWatchlistDoc = await watchlistCollection.findOne({
            movieId,
        });
        if (!updatedWatchlistDoc) {
            throw new Error(
                "Failed to load watchlist item after rating update"
            );
        }
        res.json({ document: updatedWatchlistDoc });
    } catch (err) {
        next(err);
    }
});

Here, we accept a movie id as a route parameter and a star rating within the request payload.

Using the updateOne operator, we can attempt to update the document. If the matchCount after the operation is zero, it means that there wasn't a document to update. This is likely because the movie id was incorrect, or it wasn't in our watchlist. When adding a rating, we could just respond with a simple message stating the update was complete, but in this example, we are querying for the watchlist item and returning what appears in the document.

The two final endpoints for our watchlist include removing a rating and soft-deleting a watchlist item.

Starting with the rating removal, we have the following DELETE request:

router.delete("/:movieId/rating", async (req, res, next) => {
    try {
        const database = Database.instance.db;
        const movieId = new ObjectId(req.params.movieId);
        const watchlistCollection = database.collection(config.watchlistCollection);
        const now = new Date();
        const updateResult = await watchlistCollection.updateOne(
            { movieId },
            { $unset: { rating: "" }, $set: { updatedAt: now } }
        );
        if (updateResult.matchedCount === 0) {
            throw new Error(
                "Movie is not on the watchlist; only watchlist movies can be rated"
            );
        }
        const updatedWatchlistDoc = await watchlistCollection.findOne({
            movieId,
        });
        if (!updatedWatchlistDoc) {
            throw new Error(
                "Failed to load watchlist item after rating removal"
            );
        }
        res.json({ document: updatedWatchlistDoc });
    } catch (err) {
        next(err);
    }
});

The above code will take a movieId and make use of the $unset operator within an updateOne operation. This will remove the rating. Just like with the previous endpoint, we could have opted to return a message about the operation’s success, but here we are querying for the full document and returning it to the client instead.

The final endpoint in our route looks like the following:

router.delete("/:movieId", async (req, res, next) => {
    try {
        const database = Database.instance.db;
        const movieId = new ObjectId(req.params.movieId);
        const watchlistCollection = database.collection(config.watchlistCollection);
        const existingWatchlistDoc = await watchlistCollection.findOne({
            movieId,
        });
        if (!existingWatchlistDoc) {
            throw new Error("Watchlist item not found");
        }
        if (!(existingWatchlistDoc as WatchlistDoc).active) {
            res.status(204).send();
            return;
        }
        const now = new Date();
        await watchlistCollection.updateOne(
            { movieId },
            { $set: { active: false, updatedAt: now } }
        );
        res.status(204).send();
    } catch (err) {
        next(err);
    }
});

Similar logic applies here. If the watchlist item exists and it is not already inactive, we do an updateOne operation and $set the active status to inactive. In this example, we are just doing an empty return with an appropriate status to show it was deactivated.

Configuring Express Framework to Connect to MongoDB and Serve the Application

At this point, our two routes are in place, and the appropriate indexes have been configured. We need to be able to serve our application so clients of any shape can interact with it.

Open the project's src/app.ts file and include the following:

import express, { type ErrorRequestHandler } from "express";
import { moviesRouter } from "./routes/movies";
import { watchlistRouter } from "./routes/watchlist";
import { config } from "./config";
import { Database } from "./db";

async function main() {
    await Database.connect();

    const app = express();
    app.use(express.json());

    app.get("/health", (_req, res) => {
        res.json({ ok: true });
    });

    app.use("/movies", moviesRouter());
    app.use("/watchlist", watchlistRouter());

    const onError: ErrorRequestHandler = (err, _req, res, next) => {
        if (res.headersSent) {
            next(err);
            return;
        }
        console.error(err);
        res.status(500).json({
            error: err instanceof Error ? err.message : String(err),
        });
    };
    app.use(onError);

    const server = app.listen(config.port, () => {
        console.log(`Listening on http://localhost:${config.port}`);
    });

    const shutdown = async () => {
        server.close();
        await Database.instance.close();
        process.exit(0);
    };
    process.on("SIGINT", () => void shutdown());
    process.on("SIGTERM", () => void shutdown());
}

main().catch((err) => {
    console.error(err);
    process.exit(1);
});

In the above code, we are configuring Express Framework to accept JSON payloads and to recognize our two route files. Skipping beyond the error handling and listening of our backend, you'll find the shutdown logic.

When a shutdown is triggered, we close the server from listening for connections, and we also close the connection to our MongoDB database. This is good practice for making sure everything is gracefully terminated.

Running the API to Test the Endpoints

For this particular tutorial, we are not exploring a frontend. However, we'll run and test our API endpoints so you can get an idea of what it looks like.

From a command line, execute the following to run the application in development mode:

tsx watch src/app.ts

With the application running, you can first test the health check endpoint:

curl -s http://localhost:3000/health

If that looks good, try searching for a movie:

curl -s "http://localhost:3000/movies/search?q=Terminator"

Replace the q with whatever value you'd like to search for. Make note of one of the movie id values in the response because you'll need it for populating your watchlist with the following command:

curl -s -X POST http://localhost:3000/watchlist \
  -H "Content-Type: application/json" \
  -d '{"movieId":"<PASTE_OBJECT_ID_HERE>"}'

Go ahead and give the other endpoints a test!

It's worth mentioning that what we built is good for a demo, but it will need some work for production. We didn't do much data validation, our error handling needs work, and we have no cross-origin resource sharing (CORS) policy in place to define who is allowed to access our application. However, it should get you started when working with MongoDB in unique ways.

Conclusion

You just saw how to use various features of MongoDB, such as MongoDB Search and CRUD, to build a movie watchlist application with ratings. We made use of MongoDB Search for full-text search functionality, and we used CRUD to process documents in our watchlist. We also saw that it is alright to duplicate data between the movies collection and the mflix_watchlist collection to boost performance and avoid JOIN operations.

To take this project to the next level, try adding a frontend or a few more watchlist features. Maybe you can include MongoDB Vector Search to come up with movie recommendations based on your watchlist items and ratings.

Want to try this project for yourself? Clone the repository on GitHub!

Build a Task Manager with Django MongoDB Backend

Abigail Afi Gbadago — Fri, 10 Apr 2026 01:24:45 +0000

Whilst life can work out beautifully when things are spontaneous, sometimes, a little planning and management help put things into perspective. Today, we are going to build a task manager using the MongoDB Atlas Free Tier and Django MongoDB Backend to help plan out tasks and activities. This task manager, which we will call “Get Going,” features a drag-and-drop Kanban board (To Do, In Progress, Done sections) built with HTMX.

Create a MongoDB Atlas cluster & get connection string

To kick off things, we will create a MongoDB Atlas cluster by creating an account; if you don’t already have one (link to register.

If you already have an account, go ahead and sign in. In the left pane, click Project Overview to display a page with Clusters, Application Development, and a Toolbar.

Next, click on Create cluster.

Select the MongoDB Atlas Free Tier and enter your preferred cluster name. Then select your preferred provider and other settings.
Click create to deploy your new cluster.

After your cluster has been created, click Connect and add a database username and password.

After successfully adding credentials, click on Connect to Cluster, select Drivers, select Python, and install the Python driver via your command line interface.

Copy the Connection URI to use as the HOST setting in settings.py.

Next, we’ll create a virtual env and install dependencies.

Prerequisites

Platform used: MacOS Tahoe 26.3
Python 3.12+
MongoDB Atlas account (free tier)

Project setup

Create a project directory, set up a virtual environment, and install dependencies:
Create a requirements.txt file with the following:

#requirements.txt

Django>=6.0
django-mongodb-backend
python-dotenv

After, run pip install -r requirements.txt

Next, create a Django project using the MongoDB starter template, then create the tasks app:

django-admin startproject taskmanager --template https://github.com/mongodb-labs/django-mongodb-project/archive/refs/heads/6.0.x.zip
cd taskmanager
python manage.py startapp tasks
Add the task app to settings.py by adding it to INSTALLED_APPS.

#settings.py

INSTALLED_APPS = [
    ...
    'tasks',
]

Environment Variables

Store sensitive credentials in a .env file at the project root (next to manage.py) to keep them out of source control. Then replace , , and with your MongoDB Atlas credentials.

#.env

SECRET_KEY=your-secret-key
DEBUG=True
ALLOWED_HOSTS=localhost,127.0.0.1
MONGODB_HOST=mongodb+srv://<username>:<password>@<cluster>.mongodb.net/?retryWrites=true&w=majority&appName=devrel-tutorial-django-taskmanager-devto
MONGODB_NAME=taskmanager

Task Model

To define the Task model, we'll include fields for title, description, status, priority, owner, an optional due date, and auto-managed timestamps. Status choices map to three columns on the Kanban board: To Do, In Progress, and Done.

#models.py

from django.conf import settings
from django.db import models

class Task(models.Model):
    class Status(models.TextChoices):
        TODO = 'todo', 'To Do'
        IN_PROGRESS = 'in_progress', 'In Progress'
        DONE = 'done', 'Done'

    class Priority(models.TextChoices):
        LOW = 'low', 'Low'
        MEDIUM = 'medium', 'Medium'
        HIGH = 'high', 'High'

    title = models.CharField(max_length=255)
    description = models.TextField(blank=True, default='')
    status = models.CharField(
        max_length=20,
        choices=Status.choices,
        default=Status.TODO,
    )
    priority = models.CharField(
        max_length=10,
        choices=Priority.choices,
        default=Priority.MEDIUM,
    )
    owner = models.ForeignKey(
        settings.AUTH_USER_MODEL,
        on_delete=models.CASCADE,
        related_name='tasks',
    )
    due_date = models.DateTimeField(null=True, blank=True)
    created_at = models.DateTimeField(auto_now_add=True)
    updated_at = models.DateTimeField(auto_now=True)

    class Meta:
        ordering = ['-created_at']

    def __str__(self):
        return self.title

After creating our models, we will create and run migrations:
python manage.py makemigrations tasks
python manage.py migrate

Forms

Before we create views, create a TaskForm using Django's ModelForm, which will handle validation and rendering the creation and editing tasks, with fields for title, description, status, priority, and due date.

#forms.py 

from django import forms
from .models import Task

class TaskForm(forms.ModelForm):
    class Meta:
        model = Task
        fields = ['title', 'description', 'status', 'priority', 'due_date']
        widgets = {
            'due_date': forms.DateTimeInput(
                attrs={'type': 'date'},
                format='%Y-%m-%d',
            ),
        }

Authentication

For this project, we will use Django’s built-in authentication system, which provides user accounts, login, logout, and password handling out of the box.

Login/Logout Views

Regarding authentication when logging in and out, Django provides a LoginView and LogoutView right out of the box. After this, we will register the views in urls.py and point LOGIN_URL and LOGIN_REDIRECT_URL in settings.py so unauthenticated users are sent to the login page, and successful logins land on the task manager board.

To protect the view, we use the @login_required decorator on every view to ensure only authenticated users can access the board, create tasks, or modify them. Each task is also scoped to its owner — users can only see and manage their own tasks.

We will also add a login page at templates/registration/login.html, which is a form that requires username and password fields.

Next, to get into the system and access the task manager, we need to create a superuser.

Run python manage.py createsuperuser

Follow the prompts and type out your name, email, and password, as we will use them to access the task manager board.

Add urls.py at the project and app level

Finally, we wire everything up in taskmanager/urls.py, which maps the root URL to the board view, /login/ and /logout/ to Django's built-in auth views, and task operations to their respective views. Each task URL uses the task's ID in the path for edit, delete, and move actions.

#taskmanager/urls.py

from django.contrib import admin
from django.contrib.auth import views as auth_views
from django.urls import include, path


urlpatterns = [
    path('', include('tasks.urls')),
    path('login/', auth_views.LoginView.as_view(), name='login'),
    path('logout/', auth_views.LogoutView.as_view(), name='logout'),
    path('admin/', admin.site.urls),
]

#tasks/urls.py

from django.urls import path

from . import views

urlpatterns = [
    path('', views.board_view, name='board'),
    path('tasks/new/', views.task_create_view, name='task-create'),
    path(
        'tasks/<str:pk>/edit/',
        views.task_edit_view,
        name='task-edit',
    ),
    path(
        'tasks/<str:pk>/delete/',
        views.task_delete_view,
        name='task-delete',
    ),
    path(
        'tasks/<str:pk>/move/',
        views.task_move_view,
        name='task-move',
    ),
]

Building the Task Manager Board (Kanban Board)

To build the task manager board, we will need:
Logic for the board view — a view that queries the current user's tasks and groups them into three columns by status (To Do, In Progress, Done)
Templates — a base layout, the board page with its three-column grid, a reusable card partial for each task, and a form modal for creating and editing tasks
HTMX for interactivity — create, edit, and delete tasks without full page reloads using hx-get, hx-post, and hx-delete attributes

Next, let’s create the views.py, which has a board_view function that filters tasks belonging to the logged-in user and splits them into columns. We also add a COLUMNS constant that defines the three statuses(TO-DO, IN PROGRESS, and DONE) so we can loop through them.

Also, we will add a @login_required decorator that redirects unauthenticated users to /login/. Each column dict holds the status value, a display label, a CSS dot class for the colored indicator, and the filtered queryset of tasks for that status.

Create views

Now, we can go ahead and create views for our taskapp. The board_view queries the user's tasks, groups them by status into three columns, and displays the task manager board. In addition, we will create views for creating, editing, deleting, and moving tasks between columns — each protected with @login_required and scoped to the current user.

#views.py 

import json

from django.contrib.auth.decorators import login_required
from django.http import HttpResponse, JsonResponse
from django.shortcuts import get_object_or_404, render
from django.views.decorators.http import require_POST

from .forms import TaskForm
from .models import Task

COLUMNS = [
    {'status': 'todo', 'label': 'To Do', 'dot': 'todo'},
    {'status': 'in_progress', 'label': 'In Progress', 'dot': 'progress'},
   {'status': 'done', 'label': 'Done', 'dot': 'done'},
]

@login_required
def board_view(request):
    tasks = Task.objects.filter(owner=request.user)
    columns = []
    for col in COLUMNS:
        columns.append({
            **col,
            'tasks': tasks.filter(status=col['status']),
        })
    return render(request, 'tasks/board.html', {'columns': columns})


@login_required
def task_create_view(request):
    if request.method == 'POST':
        form = TaskForm(request.POST)
        if form.is_valid():
            task = form.save(commit=False)
            task.owner = request.user
            task.save()
            return HttpResponse(headers={'HX-Redirect': '/'})
    else:
        form = TaskForm()
    return render(request, 'tasks/_form_modal.html', {
        'form': form,
        'form_title': 'New Task',
        'form_action': request.path,
        'submit_label': 'Create',
    })
@login_required
def task_edit_view(request, pk):
    task = get_object_or_404(Task, pk=pk, owner=request.user)
    if request.method == 'POST':
        form = TaskForm(request.POST, instance=task)
        if form.is_valid():
            form.save()
            return HttpResponse(headers={'HX-Redirect': '/'})
    else:
        form = TaskForm(instance=task)
    return render(request, 'tasks/_form_modal.html', {
        'form': form,
        'form_title': 'Edit Task',
        'form_action': request.path,
        'submit_label': 'Save',
    })
@login_required
def task_delete_view(request, pk):
    task = get_object_or_404(Task, pk=pk, owner=request.user)
    task.delete()
    return HttpResponse(headers={'HX-Redirect': '/'})


@login_required
@require_POST
def task_move_view(request, pk):
    """Update a task's status when dragged to a new column."""
    task = get_object_or_404(Task, pk=pk, owner=request.user)
    data = json.loads(request.body)
    new_status = data.get('status')
    valid_statuses = [c['status'] for c in COLUMNS]
    if new_status not in valid_statuses:
        return JsonResponse({'error': 'Invalid status'}, status=400)
    task.status = new_status
    task.save()
    return JsonResponse({'ok': True})

Templates

Now, we move on to templates, which form how our UI is going to look.

tasks/base.html — this loads the stylesheet, the HTMX and SortableJS CDN scripts, and sets a global CSRF token on the tag so every HTMX request is authenticated.

{% load static %}
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>{% block title %}Get Going{% endblock %}</title>
  <link rel="stylesheet" href="{% static 'css/style.css' %}">
  <script src="https://unpkg.com/htmx.org@2.0.4"></script>
  <script src="https://cdn.jsdelivr.net/npm/sortablejs@1.15.6/Sortable.min.js"></script>
</head>
<body hx-headers='{"X-CSRFToken": "{{ csrf_token }}"}'>
  {% block body %}{% endblock %}
</body>
</html>

registration/login.html - for signing into the task manager board

{% extends "base.html" %}
{% block title %}Login — Get Going{% endblock %}

{% block body %}
<div class="login-page">
  <div class="login-card">
    <h1>Get <span style="color:var(--green)">Going</span></h1>
    <p class="subtitle">Sign in to manage your tasks</p>

    {% if form.errors %}
    <div class="error-msg">Invalid username or password.</div>
    {% endif %}

    <form method="post">
      {% csrf_token %}
      <div class="form-group">
        <label for="id_username">Username</label>
        <input type="text" name="username" id="id_username" autofocus required>
      </div>
      <div class="form-group">
        <label for="id_password">Password</label>
        <input type="password" name="password" id="id_password" required>
      </div>
      <button type="submit" class="btn btn-primary">Sign In</button>
    </form>
  </div>
  <footer class="app-footer" style="color:var(--sidebar-muted)">Get Going 2026 &trade;</footer>
</div>
{% endblock %}

tasks/board.html — extends base.html and contains the sidebar with navigation and the "+ New Task" button, a three-column grid that loops over columns, and an empty #modal-container div where modals get injected. Each column renders its header (colored dot, label, task count badge) and loops through its tasks using the card partial.

{% extends "base.html" %}
{% block title %}Board — Get Going{% endblock %}

{% block body %}
<div class="app-layout">

  <!-- Sidebar -->
  <aside class="sidebar">
    <div class="logo">Get <span>Going</span></div>

    <button class="new-task-btn"
            hx-get="{% url 'task-create' %}"
            hx-target="#modal-container"
            hx-swap="innerHTML">
      + New Task
    </button>

    <div class="sidebar-section">Menu</div>
    <nav class="sidebar-nav">
      <a href="{% url 'board' %}" class="active">📋 My Tasks</a>
    </nav>
 <div class="sidebar-bottom">
      <div class="user-info">
        <span>{{ request.user.username }}</span>
        <a href="{% url 'logout' %}" class="btn btn-ghost btn-small">Sign Out</a>
      </div>
    </div>
  </aside>

  <!-- Main -->
  <main class="main-content">
    <div class="main-header">
      <h2>My Tasks</h2>
    </div>

    <div class="board">
      {% for col in columns %}
      <div class="column">
        <div class="column-header">
          <span class="column-title">
            <span class="dot dot-{{ col.dot }}"></span>
            {{ col.label }}
          </span>
          <span class="column-count">{{ col.tasks|length }}</span>
        </div>
        <div class="card-list" id="col-{{ col.status }}" data-status="{{ col.status }}">
          {% for task in col.tasks %}
            {% include "tasks/_card.html" %}
          {% endfor %}
        </div>
      </div>
      {% endfor %}
    </div>

    <footer class="app-footer">Get Going 2026 &trade;</footer>
  </main>

</div>
<div id="modal-container"></div>

<script>
document.addEventListener('DOMContentLoaded', function() {
  const csrfToken = document.querySelector('[name=csrfmiddlewaretoken]')?.value
    || document.querySelector('body').getAttribute('hx-headers')
    && JSON.parse(document.querySelector('body').getAttribute('hx-headers'))['X-CSRFToken'];

  document.querySelectorAll('.card-list').forEach(function(list) {
    new Sortable(list, {
      group: 'board',
      animation: 150,
      ghostClass: 'drag-ghost',
      dragClass: 'drag-active',
      onEnd: function(evt) {
        const taskId = evt.item.dataset.id;
        const newStatus = evt.to.dataset.status;
        // Update column counts
        updateCounts();
        // Persist to server
        fetch('/tasks/' + taskId + '/move/', {
          method: 'POST',
          headers: {
            'Content-Type': 'application/json',
            'X-CSRFToken': csrfToken,
          },
          body: JSON.stringify({ status: newStatus }),
        }).then(function(res) {
          if (!res.ok) {
            evt.from.insertBefore(evt.item, evt.from.children[evt.oldIndex]);
            updateCounts();
          }
        });
      }
    });
  });

  function updateCounts() {
    document.querySelectorAll('.column').forEach(function(col) {
      const count = col.querySelector('.card-list').children.length;
      col.querySelector('.column-count').textContent = count;
    });
  }
});
</script>
{% endblock %}

tasks/_card.html — a partial template for a single task card. It displays the title, description (clamped to two lines), due date, a color-coded priority badge, and edit/delete action buttons. Each card has a data-id="{{ task.id }}" attribute that SortableJS uses later for drag-and-drop.

<div class="task-card" id="task-{{ task.id }}" data-id="{{ task.id }}">
  <div class="task-card-title">{{ task.title }}</div>
  {% if task.description %}
  <div class="task-card-desc">{{ task.description }}</div>
  {% endif %}
  {% if task.due_date %}
  <div class="task-card-due">{{ task.due_date|date:'M d, Y' }}</div>
  {% endif %}
  <div class="task-card-footer">
    <span class="priority-badge priority-{{ task.priority }}">{{ task.get_priority_display }}</span>
    <div class="task-actions">
      <button title="Edit"
              hx-get="{% url 'task-edit' task.id %}"
              hx-target="#modal-container"
              hx-swap="innerHTML">✎</button>
      <button title="Delete" class="delete-btn"
              hx-delete="{% url 'task-delete' task.id %}"
              hx-target="#task-{{ task.id }}"
              hx-swap="outerHTML"
              hx-confirm="Delete this task?">✕</button>
    </div>
  </div>
</div>

tasks/_form_modal.html — a reusable modal for both creating and editing tasks. It receives form_title, form_action, and submit_label through the template context, so the same template can be rendered as "New Task" with a "Create" button or "Edit Task" with a "Save" button. Clicking the backdrop dismisses the modal.

<div class="modal-backdrop" onclick="if(event.target===this)this.remove()">
  <div class="modal">
    <h3>{{ form_title }}</h3>
    <form hx-post="{{ form_action }}"
          hx-target="body"
          hx-swap="innerHTML"
          hx-push-url="false">
      {% csrf_token %}
      <div class="form-group">
        <label for="id_title">Title</label>
        <input type="text" name="title" id="id_title"
               value="{{ form.title.value|default:'' }}" required>
      </div>
      <div class="form-group">
        <label for="id_description">Description</label>
        <textarea name="description" id="id_description">{{ form.description.value|default:'' }}</textarea>
      </div>
      <div class="form-group">
        <label for="id_status">Status</label>
        <select name="status" id="id_status">
          {% for value, label in form.fields.status.choices %}
          <option value="{{ value }}" {% if form.status.value == value %}selected{% endif %}>{{ label }}</option>
          {% endfor %}
        </select>
      </div>
      <div class="form-group">
        <label for="id_priority">Priority</label>
        <select name="priority" id="id_priority">
          {% for value, label in form.fields.priority.choices %}
          <option value="{{ value }}" {% if form.priority.value == value %}selected{% endif %}>{{ label }}</option>
          {% endfor %}
        </select>
      </div>
      <div class="form-group">
        <label for="id_due_date">Due Date</label>
        <input type="date" name="due_date" id="id_due_date"
               value="{{ form.due_date.value|date:'Y-m-d'|default:'' }}">
      </div>
      <div class="modal-actions">
        <button type="button" class="btn btn-ghost" onclick="this.closest('.modal-backdrop').remove()">Cancel</button>
        <button type="submit" class="btn btn-primary">{{ submit_label }}</button>
      </div>
    </form>
  </div>
</div>

NB: for the CRUD methods:

Create method — The "+ New Task" button in the sidebar uses hx-get to fetch the empty form modal from the server and inject it into #modal-container. When the form is submitted, hx-post sends the data to the create endpoint.
Edit method — The edit button on each card uses hx-get to fetch the form modal pre-filled with that task's data.
Delete method— The delete button (✕) uses hx-delete with hx-confirm="Delete this task?" for browser-native confirmation. It targets only the specific card element using hx-target="#task-{{ task.id }}" and hx-swap="outerHTML" to remove just that card from the DOM.

Drag & Drop

To bring some flexibility into the task manager, we will add drag and drop functionality. To achieve this, we will use SortableJS to handle the drag interaction in the browser and a dedicated endpoint to persist the status change to MongoDB.

SortableJS Setup

CDN load in base.html: SortableJS is loaded as a global script from the jsDelivr CDN, making the Sortable constructor available on every page that extends the base template (with this line <script src="https://cdn.jsdelivr.net/npm/sortablejs@1.15.6/Sortable.min.js"></script> )
Initialization + drag handler in board.html: On DOMContentLoaded, each .card-list element gets a Sortable instance with group: 'board' so cards can cross columns. The onEnd callback reads data-id and data-status to POST the new status to the move endpoint, with rollback on failure.
Drag feedback styles in style.css: .drag-ghost renders the drop placeholder with a dashed green border and reduced opacity, while .drag-active gives the card being dragged an elevated shadow and slight rotation for depth.

Styling

For styling, we keep it simple and use a MongoDB-inspired color palette defined as CSS custom properties; a dark sidebar layout, and a responsive breakpoint for mobile. Colors used for status indicator dots: gray for To Do, amber for In Progress, forest green for Done; priority badges: teal for Low, gold for Medium, red for High, and interactive elements like buttons and card hover states.

Responsiveness - CSS

To keep the board usable on smaller screens, we specify a single breakpoint at 768px to restructure the layout:

@media (max-width: 768px) {
    .sidebar {
        width: 100%;
        position: relative;
        flex-direction: row;
        padding: .75rem 1rem;
    }
    .sidebar-nav, .sidebar-section { display: none; }
    .main-content { margin-left: 0; }
    .board { grid-template-columns: 1fr; }
}

On mobile, the fixed sidebar collapses into a horizontal top bar with the logo and "+ New Task" button inline, hiding the nav links to save space. The main content drops its left margin, and the board switches from a three-column grid to a single stacked column.

Final Look at Task Manager

Adding tasks for display

We can create, read, update, and delete tasks via the website or Python Shell; however, we will add tasks via the Add Task button on the Task Manager board.

After adding sample tasks,

The same data is displayed under the task collection of the MongoDB cluster tier that was initially created. Example shown below.

This brings us to the end of this tutorial using Django MongoDB Backend to create a Task Manager with MongoDB Atlas Free Tier. To download the project, kindly access this link to project. If you liked this tutorial, kindly like this tutorial and share your thoughts and/or contributions in the comments section.

Thank you, and see you in the next one.

References

Best Practices Cheat Sheet for Flask-PyMongo

MongoDB Guests — Wed, 08 Apr 2026 11:17:13 +0000

This tutorial was written by João Araújo.

Introduction

Starting a project is always a challenge. Naming the first things, deciding on a project structure, what stack to use, and even where we want that project to go. While it is important to have most of it planned, if we are unsure or lack visibility into the project's direction, we should use the right tools to navigate any challenges that may arise. The ability to build applications that evolve over time is a strength of both MongoDB and Flask. If you are starting a project today and are not sure how fast it will grow, or if you just need quick prototyping, MongoDB performance aligned with Flask should be considered. If you are here just for the cheat sheet, it is at the end - but consider reading through to understand the “whys” behind it.

What is MongoDB?

MongoDB is an open-source NoSQL database that stores and manages data using flexible document-oriented structures rather than the traditional tables and rows of relational databases. In MongoDB, data is stored in JSON-like documents (BSON) organized into databases and collections, allowing schemas to be dynamic and easily changed over time, making it highly adaptable for handling diverse, structured, and unstructured data. It’s designed for scalability, flexibility, and ease of development, and is widely used in modern web and mobile applications where rapid iteration and large data volumes are common.

What is Flask?

Flask is a micro web framework written in Python that provides the core essentials for building web applications without imposing a lot of structure or extra components, making it lightweight and easy to use. Flask’s biggest strengths are its simplicity and flexibility. Because it doesn’t enforce a specific project layout or include unnecessary features by default, developers can structure applications as they see fit and choose the best tools for tasks such as database access, authentication, and form handling. Additionally, Flask has a large ecosystem of add-ons that provide features such as authentication, authorization, login, sessions, and more.

Why Choose MongoDB with Flask?

Choosing MongoDB with Flask is a great combination for Python web projects, as both tools are built around flexibility and simplicity. MongoDB’s document-oriented, schema-less storage means you don’t have to define rigid table structures ahead of time. Data can reside in flexible JSON-like documents, making it easy to get started and adapt as your app evolves. Another big plus of using MongoDB with Flask is that JSON-like data structures (e.g., Python dictionaries) map naturally to BSON documents in the database, so developers can work with familiar Python types without complex transformations. We could go on, but let's get this cheat sheet started.

Setup and Project Structure

Configure MongoDB connection parameters (URI, host, port, credentials) via environment variables or configuration files.

Always use configuration files or environment variables to define database connections and overall credentials. This ensures there is no hard-coded information across production and other environments.

In the environment file, there isn't much to see here. I added information on how and what access we have to our database, as well as the environment we run on.

.env
FLASK_ENV=development
FLASK_APP=app.py

MONGODB_URI=mongodb://localhost:27017
MONGODB_DB=my_database

Use a single, globally shared `MongoClient` (or an appropriately scoped one) instead of creating a new connection per request.

Avoid using multiple database clients within the same application. You should create a single connection and reuse it. MongoDB Drivers manage a connection pool and can be configured to allow more or fewer active connections. When a new connection is required, the driver will use an available connection or establish a new one. This ensures authentication occurs only once, so the time spent resolving the request is dedicated to the actual query rather than to authentication and operation execution. With Flask's built-in extension management, we can instantiate it once and reuse it later.

python
import os
from pymongo import MongoClient
from flask import current_app
from dotenv import load_dotenv

# load environment variables
load_dotenv()

# defining my mongo connection
class Mongo:
    def __init__(self):
        self.client = None
        self.db = None

    def init_app(self, app):
        uri = os.getenv("MONGODB_URI")
        db_name = os.getenv("MONGODB_DB")

        if not uri or not db_name:
            raise RuntimeError("MongoDB environment variables are not set")

        self.client = MongoClient(uri)
        self.db = self.client[db_name]

        # store reference inside Flask app as extension
        app.extensions["mongo"] = self

    @staticmethod
    def get_db():
        mongo = current_app.extensions.get("mongo")
        if not mongo:
            raise RuntimeError("Mongo extension not initialized")
        return mongo.db


mongo = Mongo()

Keep your database access logic, application logic (Flask routes), and configuration separate. Helps with upgrades, testing, and debugging.

By keeping your database access logic, configuration, and business rules separate, you can support each point of view independently without having to search your project to identify the root cause. This is especially useful when you need to understand the application workload and determine which index to use on MongoDB.

The example below shows an example project:

.md
project/
├── .env
├── app.py
├── extensions.py        # Mongo connection
├── routes/
│   └── example.py      # Route logic
├── repositories/
│   └── example_repo.py # Mongo collection queries
└── requirements.txt

extensions.py

In the following file, I am instantiating a MongoDB client. I will instantiate it later in my Flask app by calling the init_app function and passing the Flask instance. This way, whenever my “main” is called, it will create my connection.

python
import os
from pymongo import MongoClient
from flask import current_app
from dotenv import load_dotenv

# load environment variables
load_dotenv()

# defining my mongo connection
class Mongo:
    def __init__(self):
        self.client = None
        self.db = None

    def init_app(self, app):
        uri = os.getenv("MONGODB_URI")
        db_name = os.getenv("MONGODB_DB")

        if not uri or not db_name:
            raise RuntimeError("MongoDB environment variables are not set")

        self.client = MongoClient(uri)
        self.db = self.client[db_name]

        # store reference inside Flask app as extension
        app.extensions["mongo"] = self

    @staticmethod
    def get_db():
        mongo = current_app.extensions.get("mongo")
        if not mongo:
            raise RuntimeError("Mongo extension not initialized")
        return mongo.db


mongo = Mongo()

repositories/example_repo.py

Here, I am simply defining how I interact with my database. If required, I would also define indexes here (not creating them). Having both the indexes and your interactions with that collection in the same folder makes it easier to evaluate new indexes, remove unnecessary ones, or upgrade indexes as new requirements arise.

python
from extensions import Mongo

COLLECTION_NAME = "example"

def get_all():
    db = Mongo.get_db()
    return list(db[COLLECTION_NAME].find({}, {"_id": 0}))

def insert_message(message: str):
    db = Mongo.get_db()
    result = db[COLLECTION_NAME].insert_one({"message": message})
    return str(result.inserted_id)

routes/example.py

Finally, I'm using Flask Blueprints to define my routes. Blueprints are a powerful, built-in mechanism for structuring applications into modular, reusable components rather than placing all code in one file.

python
from flask import Blueprint, jsonify, request
from repositories.example_repo import get_all, insert_message

example_bp = Blueprint("example", __name__, url_prefix="/example")

@example_bp.route("/", methods=["GET"])
def list_messages():
    return jsonify(get_all())

@example_bp.route("/", methods=["POST"])
def create_message():
    payload = request.get_json(silent=True) or {}
    message = payload.get("message")

    if not message:
        return jsonify({"error": "message is required"}), 400

    inserted_id = insert_message(message)
    return jsonify({"inserted_id": inserted_id}), 201

app.py

python
from flask import Flask
from routes.example import example_bp
from extensions import mongo

def create_app():
    app = Flask(__name__)

    # init extensions
    mongo.init_app(app)

    # register routes
    app.register_blueprint(example_bp)

    return app

app = create_app()

if __name__ == "__main__":
    app.run(debug=True)

Data Modeling & Schema Design

Maintain consistent field names and types in the application.

Maintaining consistent field names and data types across the application is critical to avoid subtle bugs, simplify queries, and keep your data model predictable over time. When field names or types change inconsistently (for example, userId versus user_id, or a date stored sometimes as a string and sometimes as a Date), it increases cognitive load, breaks indexes, complicates aggregations, and can cause runtime errors that are hard to trace. Data consistency makes your application easier to maintain, safer to refactor, and more reliable when multiple services, developers, or analytics tools interact with the same data. If you need to have multiple versions of the same document, consider using Document Versioning pattern.

You can use the MongoDB schema validator to define and validate documents inserted into a collection after the schema definition.

javascript
db.createCollection("users", {
  validator: {
    $jsonSchema: {
      bsonType: "object",
      required: ["userId", "email", "isActive", "createdAt"],
      properties: {
        userId: {
          bsonType: "string",
          description: "Must be a string and is required"
        },
        email: {
          bsonType: "string",
          pattern: "^.+@.+$",
          description: "Must be a valid email string."
        },
        isActive: {
          bsonType: "bool",
          description: "Must be a boolean."
        },
        createdAt: {
          bsonType: "date",
          description: "Must be a BSON date."
        },
        optionalField: {
         bsonType: “string”,
         description: “This field is optional.”
       }
      }
    }
  },
  validationLevel: "strict",
  validationAction: "error"
})

Design your document model according to query and access patterns.

Designing your document model based on query and access patterns means shaping your MongoDB documents around how the application actually reads and writes data, not just how the data appears conceptually. By embedding frequently accessed data together and avoiding unnecessary joins, you reduce query complexity, minimize network round-trips, and improve performance. A model optimized for real access patterns is faster, simpler to query, and scales more predictably than one designed only from a relational mindset.

In MongoDB, data that is read together should usually live together. That’s the core idea behind modeling for access patterns.

Model not aligned with access patterns(reference-heavy):

javascript
// orders
{ "_id": 1, "customer_id": 10 }

// order_items
{ "order_id": 1, "product": "Book", "qty": 2 }

// customers
{ "_id": 10, "name": "Alice" }

Query (multiple round trips / $lookup):

db.orders.aggregate([
  { $match: { _id: 1 } },
  { $lookup: {
      from: "order_items",
      localField: "_id",
      foreignField: "order_id",
      as: "items"
  }},
  { $lookup: {
      from: "customers",
      localField: "customer_id",
      foreignField: "_id",
      as: "customer"
  }}
])

Model aligned with access patterns (embedded):

javascript
# orders
{
  "_id": 1,
  "customer": {
    "id": 10,
    "name": "Alice"
  },
  "items": [
    { "product": "Book", "qty": 2 },
    { "product": "Pen", "qty": 1 }
  ]
}

Query (single, simple read):

python
db.orders.findOne({ _id: 1 })

Performance & Query Optimization

Use indexes for fields that you frequently query, filter, or sort by.

Using indexes on frequently queried, filtered, or sorted fields is essential for efficient MongoDB queries. A well-designed compound index can support multiple query patterns when it follows the ESR rule (Equality → Sort → Range), meaning equality filters should come first, followed by sort fields, and then range conditions. By ordering fields correctly, a single index can satisfy multiple query patterns without creating multiple indexes:

“user”
“user” and “age”
“user”, “age”, and “workedHours”

Index creation:

python
db.example.createIndex({ user: 1, age: 1, workedHours: 1 })

Query examples:

python
db.example.find({ user: "joao" })

db.example.find({ user: "joao", age: "28" })
# or
db.example.find({ user: "joao" }).sort({age: 1})

db.example.find({
  user: "joao",
  workedHours: { $gte: 100, $lte: 200 }
}).sort({age: 1})

For potentially large result sets, use pagination (_id skip/limit).

For potentially large result sets, always use pagination to avoid loading too much data into memory and to keep response times predictable. In MongoDB, we can rely on the _id field for pagination over large skip values because it scales efficiently as collections grow and avoids unnecessary data scans. By filtering on _id and combining it with the $limit stage, each page starts where the previous one ended, providing consistent performance and avoiding the cost of scanning and skipping documents. This is possible because ObjectIds are unique and grow monotonically by default.

python
# First page
db.logs.find({})
  .sort({ _id: 1 })
  .limit(10)

# Next page (use last _id from previous page)
db.logs.find({ _id: { $gt: ObjectId("65a123...") } }) // change to $lt if using descending
  .sort({ _id: 1 })
  .limit(10)

Security

Always validate and sanitize user input before inserting it into MongoDB.

Always validate and sanitize user input before inserting it into MongoDB, especially when the input comes from a public endpoint, since HTTP requests should be treated as untrusted by default. Proper validation ensures required fields are present and have the correct data types, while sanitization prevents malformed data, unexpected structures, or NoSQL injection (e.g. $gt, $ne) from reaching the database.

Additionally, if you store sensitive data, such as passwords, never store it in plaintext. Always hash them with an algorithm, such as bcrypt. This protects data integrity, reduces security risks, and keeps your MongoDB collections consistent and secure, even when accessed.

Another protection layer is using Binary subtype 8. This subtype will replace its content in log files for ###. Making sure that even if someone gets access to your MongoDB logs, they will be redacted on that specific field.

python
from flask import request, jsonify
import bcrypt
from bson import Binary

@app.route("/users", methods=["POST"])
def create_user():
    payload = request.get_json(silent=True) or {}

    # Validation
    if not isinstance(payload.get("email"), str):
        return jsonify({"error": "email must be a string"}), 400
    if not isinstance(payload.get("password"), str):
        return jsonify({"error": "password must be a string"}), 400

    salt = bcrypt.gensalt()
    hashed_pwd = bcrypt.hashpw(payload["password"].encode('utf-8'), salt)

    # Sanitization
    user = {
        "email": payload["email"].strip().lower(),
        # Hash password before storing
        "password": Binary(hashed_pwd, 8),
        "is_active": bool(payload.get("is_active", True))
    }

    db.users.insert_one(user)
    return jsonify({"status": "created"}), 201

Conclusion

This article outlines best practices for using MongoDB with Flask. This is not an exhaustive list, but a few common mistakes I've seen after many consulting sessions. Hopefully, this has shed some light and will help you avoid those common mistakes. The best way to learn MongoDB is through its extensive documentation, which is comprehensive and always up to date. If you are unsure or couldn't find anything, there are always the forums and community to help out!

And as promised, here's the Cheat sheet!

Area	Best Practice	Key Action / Code Example
Setup & Structure	Configure connection via `environment variables`.	Use environment variables for credentials.
Setup & Structure	Use a single, globally shared `MongoClient`.	Instantiate `MongoClient` once in an extension and reuse it.
Setup & Structure	Separate logic (routes, repo, config).	Use Flask Blueprints for routes and dedicated repository files for database interactions.
Data Modeling	Maintain consistent field names and types.	Enforce consistency with MongoDB's `$jsonSchema` validator when required.
Data Modeling	Design model for query & access patterns.	Embed data that is read together (e.g., customer info inside the order document).
Performance	Use indexes for frequent query, filter, or sort fields.	Follow the ESR Rule for compound indexes: Equality, Sort, Range.
Performance	Use _id field for pagination.	Use `find({ _id: { $gt: last_id } }).sort({ _id: 1 }).limit(N)` to avoid large `skip` values.
Security	Validate and sanitize all user input.	Check data types and strip/lower strings before insertion.
Security	Never store sensitive data in plaintext.	Always hash passwords before saving them to the database.

Create a workspace scheduler using Bryntum Scheduler Pro and MongoDB

MongoDB Guests — Wed, 01 Apr 2026 11:40:43 +0000

This Tutorial was written by Arsalan Khattak.

Bryntum Scheduler Pro is a scheduling UI component for the web. With features such as a scheduling engine, constraints, and a resource utilization view, it simplifies managing complex schedules.

In this tutorial, we'll use Bryntum Scheduler Pro and MongoDB, the popular document database, to build a workspace booking app for meeting rooms, desk banks, and coworking lounges. We'll use MongoDB Atlas, the fully managed MongoDB cloud service.

We'll do the following:

Set up MongoDB Atlas and get the connection string.
Create an npm workspaces monorepo for the backend and frontend code.
Seed the MongoDB database.
Create the backend server and the Bryntum load and sync endpoints.
Create a Vite vanilla TypeScript client.
Add Bryntum Scheduler Pro to the client.

Here's what we'll build:

You can find the code for the completed tutorial in our GitHub repository: Workspace scheduler using Bryntum Scheduler Pro and MongoDB.

Prerequisites

To follow along, you need Node.js version 20.19+, installed on your system. You'll also need a MongoDB Atlas account, you can register for an Atlas account using your GitHub account, your Google account, or your email address.

Setting Up MongoDB Atlas

We'll use the MongoDB Atlas CLI to create the organization, project, and cluster from the terminal. Using the CLI instead of the Atlas Cloud UI also works well if you're following along with an AI coding agent. It provides a better agent experience.

First, install the MongoDB Atlas CLI. You can install it on macOS or Linux with Homebrew:

brew install mongodb-atlas

atlas auth login

Select UserAccount when the CLI asks how you want to authenticate.

Create a new Atlas organization for this tutorial:

atlas organizations create bryntum --output json

Copy the returned organization ID, then create a new Atlas project:

atlas projects create schedulerpro --orgId <orgId> --output json

Copy the returned project ID, and use it to set the default org_id for Atlas:

atlas config set org_id <orgId

Set the default project_id to your project ID as well:

atlas config set project_id <projectId>

Create an Atlas cluster using the following command:

atlas setup --projectId <projectId> --currentIp --connectWith skip --skipSampleData

Take note of the database username, database password, and connection string.

To find the allowed IP list in your MongoDB Atlas dashboard, go to Security in the left navigation menu and select Database & Network Access. Under Network Access, select IP Access List.

Make sure your IP address is on the allowed list; otherwise, your Express server won't be able to connect to the database. Click the + Add IP Address button to add your IP address.

Setting up the monorepo

Create an empty project folder:

mkdir bryntumschedulerpro-mongodb
cd bryntumschedulerpro-mongodb

Create a package.json file in the project root folder and add the following JSON object:

{
  "name": "bryntumschedulerpro-mongodb",
  "version": "1.0.0",
  "private": true,
  "workspaces": [
    "client",
    "server"
  ],
  "scripts": {
    "dev": "concurrently 'npm run dev --workspace server' 'npm run dev --workspace client'",
    "build": "npm run build --workspace server && npm run build --workspace client",
    "seed": "npm run seed --workspace server",
    "start": "npm run start --workspace server"
  },
  "devDependencies": {
    "@types/node": "22.13.10",
    "concurrently": "9.2.1",
    "nodemon": "3.1.14",
    "typescript": "5.9.3",
    "vite": "7.3.1"
  }
}

This root package defines an npm workspaces TypeScript monorepo with a server and a client. The dev script starts both the Express server and the Vite client together. We'll use the seed script to populate the MongoDB database with example scheduler data.

Install the shared root dependencies:

npm install

Create a tsconfig.base.json file in the root folder:

{
  "compilerOptions": {
    "target": "ES2022",
    "strict": true,
    "skipLibCheck": true,
    "esModuleInterop": true,
    "allowSyntheticDefaultImports": true,
    "forceConsistentCasingInFileNames": true,
    "resolveJsonModule": true
  }
}

This base TypeScript config keeps the shared compiler settings in one place, and both workspaces extend it.

Create a .gitignore file in the root folder and add the following lines to it:

# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*
lerna-debug.log*

# Dependencies
node_modules
client/node_modules
server/node_modules

# Build output
**/dist
.vite

# Local config
*.local
.env

# Editor directories and files
.vscode/*
!.vscode/extensions.json
.idea
.DS_Store
*.suo
*.ntvs*
*.njsproj
*.sln
*.sw?

Create a .env file in the root folder, add the following environmental variables to it:

PORT=1339
MONGODB_URI=mongodb+srv://<dbUsername>:<dbPassword>@<clusterName>.<uniqueId>.mongodb.net/?appName=devrel-tutorial-javascript-bryntum
MONGODB_DB=bryntum-schedulerpro

Add your MongoDB connection string to the MONGODB_URI and add your database username and password. You can get the clusterName and uniqueId by running the following command:

atlas clusters connectionStrings describe <clusterName>

Creating the backend Express Server

Create the server folders:

mkdir -p server/src/lib server/src/routes server/src/scripts

Create a package.json file in the server folder and add the following JSON object to it:

{
  "name": "server",
  "private": true,
  "type": "module",
  "scripts": {
    "dev": "nodemon --watch src --ext ts --exec \"npm run serve\"",
    "serve": "npm run build && node dist/server.js",
    "build": "tsc -p tsconfig.json",
    "start": "node dist/server.js",
    "seed": "npm run build && node dist/scripts/seed.js"
  }
}

This workspace compiles the TypeScript server to dist and reruns the built server with nodemon during development.

Install the server dependencies:

npm install express@4.21.2 dotenv@16.4.7 mongodb@6.15.0 --workspace=server
npm install --save-dev @types/express@5.0.1 --workspace=server

Create a tsconfig.json file in the server folder and add the following configuration object:

{
  "extends": "../tsconfig.base.json",
  "compilerOptions": {
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "rootDir": "src",
    "outDir": "dist",
    "lib": ["ES2022"],
    "types": ["node"]
  },
  "include": ["src/**/*.ts"]
}

This config compiles the server as Node.js ES modules. Because of this, the TypeScript source files will use .js import paths that match the compiled output.

Create a loadEnv.ts file in the server/src/lib folder and add the following lines of code to it:

import dotenv from 'dotenv';
import { existsSync } from 'node:fs';
import path from 'node:path';

export function loadEnv(): void {
    const candidates = [
        path.resolve(process.cwd(), '.env'),
        path.resolve(process.cwd(), '../.env')
    ];

    const envPath = candidates.find(existsSync);

    if (envPath) {
        dotenv.config({ path : envPath });
        return;
    }

    dotenv.config();
}

This helper loads environment variables whether we run the server scripts from the monorepo root or from the server workspace.

Now create a mongo.ts file in the server/src/lib folder and add the following code to it:

import { MongoClient, type MongoClientOptions, ServerApiVersion } from 'mongodb';

export function createMongoClient(uri: string): MongoClient {
    const isAtlasUri = uri.startsWith('mongodb+srv://') || uri.includes('.mongodb.net');
    const options: MongoClientOptions = isAtlasUri
        ? {
            serverApi : {
                version           : ServerApiVersion.v1,
                strict            : true,
                deprecationErrors : true
            }
        }
        : {};

    return new MongoClient(uri, options);
}

This helper uses the Atlas Server API options when the connection string points at MongoDB Atlas. Local MongoDB connections keep the default client options.

Create a cors.ts file in the server/src/lib folder and add the following code to it:

import type { RequestHandler } from 'express';

type CorsConfig = {
    origin : string
};

export default function cors(config: CorsConfig): RequestHandler {
    return (req, res, next) => {
        res.setHeader('Access-Control-Allow-Origin', config.origin);
        res.setHeader('Access-Control-Allow-Methods', 'GET,POST,PUT,PATCH,DELETE,OPTIONS');
        res.setHeader('Access-Control-Allow-Headers', 'Content-Type, Authorization');

        if (req.method === 'OPTIONS') {
            res.sendStatus(204);
            return;
        }

        next();
    };
}

This middleware allows the Vite client to call the API directly during local development.

Next, create a schedulerCrud.ts file in the server/src/lib/ folder to define the structure for the load and sync API routes. In this file, add the following imports, collection names, and TypeScript types that describe the request and response shapes:

import { randomUUID } from 'crypto';
import { Collection } from 'mongodb';

export const COLLECTIONS = {
    resources    : 'resources',
    events       : 'events',
    assignments  : 'assignments',
    calendars    : 'calendars',
    dependencies : 'dependencies'
} as const;

export type CrudRequestId = string | number;
export type GenericRecord = { _id?: string; [key: string]: any };
export type StoreKey = keyof typeof COLLECTIONS;
export type IdMap = Map<string, string>;
export type IdMaps = Record<StoreKey, IdMap>;
export type Normalizer = (record: GenericRecord) => GenericRecord;

export type StoreChanges = {
    added?: GenericRecord[];
    updated?: GenericRecord[];
    removed?: Array<{ id?: CrudRequestId }>;
};

export type SyncRequestBody = {
    requestId?: CrudRequestId | null;
    resources?: StoreChanges;
    events?: StoreChanges;
    assignments?: StoreChanges;
    calendars?: StoreChanges;
    dependencies?: StoreChanges;
};

export type StoreResponseSection = {
    rows: GenericRecord[];
    removed?: GenericRecord[];
};

export type SyncResponseBody = {
    success: boolean;
    requestId: CrudRequestId | null;
    resources?: StoreResponseSection;
    events?: StoreResponseSection;
    assignments?: StoreResponseSection;
    calendars?: StoreResponseSection;
    dependencies?: StoreResponseSection;
    message?: string;
};

The collections persist in the Bryntum data stores.

We'll use the Bryntum Crud manager to load the MongoDB data into the Bryntum Scheduler Pro and sync data changes to it. The Crud Manager expects a specific sync request structure and sync response structure, which you can see in the SyncRequestBody and SyncResponseBody types.

Add the following functions to the bottom of the file:

export function createIdMaps(): IdMaps {
    return {
        resources    : new Map(),
        events       : new Map(),
        assignments  : new Map(),
        calendars    : new Map(),
        dependencies : new Map()
    };
}

export function normalizeDate(value: unknown): unknown {
    if (value == null) {
        return value;
    }

    if (value instanceof Date) {
        return value.toISOString();
    }

    return value;
}

export function normalizeEventFields(record: GenericRecord): GenericRecord {
    if (!record) {
        return record;
    }

    const normalized = { ...record };

    if ('startDate' in record) {
        normalized.startDate = normalizeDate(record.startDate);
    }

    if ('endDate' in record) {
        normalized.endDate = normalizeDate(record.endDate);
    }

    return normalized;
}

In this code:

The createIdMaps function builds a phantom-to-permanent ID map for each store.
- When Bryntum creates a new record, it sends a temporary phantom ID that's generated on the client. The sync response then replaces the phantom ID with a permanent ID from the database. When the sync request sends multiple new records at once, one record might reference another that was also just created.
- Consider the example of a new dependency referencing a new event: The server inserts the event first, and maps its phantom ID to the new MongoDB _id. Then, when processing the dependency, resolveId looks up the event's phantom ID in the map to get the database-assigned _id.
The normalizeEventFields function converts Date objects to ISO strings before the server returns them.

Add the following ID resolver and normalizer functions to the bottom of the file:

export function resolveId(id: CrudRequestId, map: IdMap): string {
    const asString = String(id);

    return map.get(asString) || asString;
}

export function normalizeAssignmentFields(record: GenericRecord, idMaps: IdMaps): GenericRecord {
    if (!record) {
        return record;
    }

    const eventId = record.eventId ?? record.event;
    const resourceId = record.resourceId ?? record.resource;

    const normalized = {
        ...record
    };

    if (eventId != null) {
        normalized.eventId = resolveId(eventId, idMaps.events);
    }

    if (resourceId != null) {
        normalized.resourceId = resolveId(resourceId, idMaps.resources);
    }

    delete normalized.event;
    delete normalized.resource;

    return normalized;
}

export function normalizeDependencyFields(record: GenericRecord, idMaps: IdMaps): GenericRecord {
    if (!record) {
        return record;
    }

    const fromEvent = record.fromEvent ?? record.from;
    const toEvent = record.toEvent ?? record.to;

    const normalized = {
        ...record
    };

    if (fromEvent != null) {
        normalized.fromEvent = resolveId(fromEvent, idMaps.events);
    }

    if (toEvent != null) {
        normalized.toEvent = resolveId(toEvent, idMaps.events);
    }

    delete normalized.from;
    delete normalized.to;

    return normalized;
}

These two normalizers map Bryntum's event and resource references to permanent IDs using the ID maps built earlier in the same sync request.

Add the following helper functions that remove MongoDB internal fields:

export function stripInternalFields(record: GenericRecord): GenericRecord {
    if (!record) {
        return record;
    }

    const { _id, ...clean } = record;

    return { ...clean, id : String(_id) };
}

export function collectStoreRows(rows: GenericRecord[]): GenericRecord[] {
    return rows.map(row => stripInternalFields(row));
}

The stripInternalFields function maps MongoDB's _id field to Bryntum's id field before sending data to the client. The collectStoreRows function applies this cleanup to every row in a collection result.

Finally, add the applyStoreChanges() function that handles the add, update, and remove sync requests for every store:

export async function applyStoreChanges({
    collection,
    changes,
    idMap,
    normalizeAdded = value => value,
    normalizeUpdated = value => value
}: {
    collection: Collection<GenericRecord>;
    changes: StoreChanges;
    idMap: IdMap;
    normalizeAdded?: Normalizer;
    normalizeUpdated?: Normalizer;
}): Promise<GenericRecord[]> {
    const rows: GenericRecord[] = [];

    if (Array.isArray(changes.added)) {
        for (const rawRecord of changes.added) {
            const {
                $PhantomId,
                ...data
            } = rawRecord;

            const normalized = normalizeAdded(data);
            const id = String(normalized.id || randomUUID());
            delete normalized.id;
            const recordToStore = {
                ...normalized,
                _id : id
            };

            await collection.insertOne(recordToStore);

            if ($PhantomId) {
                idMap.set(String($PhantomId), id);
                rows.push({ $PhantomId, id });
            }
            else {
                rows.push({ id });
            }
        }
    }

    if (Array.isArray(changes.updated)) {
        for (const rawRecord of changes.updated) {
            const {
                id,
                ...updatedFields
            } = rawRecord;

            if (!id) {
                continue;
            }

            const normalized = normalizeUpdated(updatedFields);

            await collection.updateOne(
                { _id : String(id) },
                { $set : normalized }
            );
        }
    }

    if (Array.isArray(changes.removed) && changes.removed.length) {
        const ids = changes.removed
            .map(record => record.id)
            .filter(Boolean)
            .map(id => String(id));

        if (ids.length) {
            await collection.deleteMany({
                _id : {
                    $in : ids
                }
            });
        }
    }

    return rows;
}

For added records, the function strips the Bryntum $PhantomId, normalizes the data, and maps the phantom to the real ID so that dependent stores can resolve their references in the same sync request.

In MongoDB, every document needs a unique _id field as its primary key. If omitted, the MongoDB driver auto-generates an ObjectId, which consists of a timestamp, machine and process identifier, and counter. However, _id can be any unique value. We use a randomUUID() from Node.js's crypto module instead of ObjectId for consistency with our seed data. The seed script uses human-readable string IDs like 'booking-1' and 'room-terrace' so that cross-references between stores are easy to follow. For example, an assignment's eventId: 'booking-1' pointing to the event with an _id of 'booking-1'.

Creating a database seed script

Now let's create a seed script to populate the MongoDB Atlas database with example data. Create a seed.ts file in the server/src/scripts folder. Add the following imports, database connection setup, and the resources array:

import { loadEnv } from '../lib/loadEnv.js';
import { createMongoClient } from '../lib/mongo.js';

loadEnv();

const MONGODB_URI = process.env.MONGODB_URI || 'mongodb://127.0.0.1:27017';
const MONGODB_DB = process.env.MONGODB_DB || 'bryntum-schedulerpro';

const client = createMongoClient(MONGODB_URI);

type GenericRecord = Record<string, unknown>;

const resources: GenericRecord[] = [
    {
        _id           : 'room-terrace',
        name          : 'Terrace Briefing Room',
        workspaceType : 'Meeting room',
        neighborhood  : 'Docklands',
        floor         : 'Level 2',
        capacity      : 8,
        maxUnits      : 100,
        city          : 'Cape Town',
        amenities     : 'Display wall, VC bar'
    },
    {
        _id           : 'room-harbor',
        name          : 'Harbor Strategy Suite',
        workspaceType : 'Meeting room',
        neighborhood  : 'Harbor Wing',
        floor         : 'Level 5',
        capacity      : 16,
        maxUnits      : 100,
        city          : 'Cape Town',
        amenities     : 'Board table, dual screens'
    },
    {
        _id           : 'room-sprint',
        name          : 'Sprint Garage',
        workspaceType : 'Meeting room',
        neighborhood  : 'Atrium North',
        floor         : 'Level 3',
        capacity      : 6,
        maxUnits      : 100,
        city          : 'Cape Town',
        amenities     : 'Whiteboards, pin-up wall'
    },
    {
        _id           : 'desk-orbit-1',
        name          : 'Orbit Desk Bank A',
        workspaceType : 'Desk bank',
        neighborhood  : 'Docklands',
        floor         : 'Level 4',
        capacity      : 6,
        maxUnits      : 180,
        city          : 'Cape Town',
        amenities     : 'Sit-stand desks, lockers'
    },
    {
        _id           : 'desk-orbit-2',
        name          : 'Orbit Desk Bank B',
        workspaceType : 'Desk bank',
        neighborhood  : 'Docklands',
        floor         : 'Level 4',
        capacity      : 6,
        maxUnits      : 180,
        city          : 'Cape Town',
        amenities     : 'Phone booths, lockers'
    },
    {
        _id           : 'desk-focus',
        name          : 'Focus Pods',
        workspaceType : 'Desk bank',
        neighborhood  : 'Atrium North',
        floor         : 'Level 2',
        capacity      : 4,
        maxUnits      : 140,
        city          : 'Cape Town',
        amenities     : 'Quiet zone, task lights'
    },
    {
        _id           : 'coworking-canvas',
        name          : 'Canvas Coworking Lounge',
        workspaceType : 'Coworking lounge',
        neighborhood  : 'Harbor Wing',
        floor         : 'Ground',
        capacity      : 24,
        maxUnits      : 300,
        city          : 'Cape Town',
        amenities     : 'Cafe seating, soft booths'
    },
    {
        _id           : 'coworking-sky',
        name          : 'Skyline Lounge',
        workspaceType : 'Coworking Lounge',
        neighborhood  : 'Atrium North',
        floor         : 'Rooftop',
        capacity      : 18,
        maxUnits      : 260,
        city          : 'Cape Town',
        amenities     : 'Outdoor tables, power rails'
    }
];

Each resource represents a bookable workspace with a type, location, capacity, and amenities. The maxUnits field sets the resource's percentage capacity. It works together with the assignment units field: if the sum of all assignment units for a resource exceeds its maxUnits, the utilization panel marks that resource as overallocated.

Meeting rooms use 100 by default, so that a single 100% booking fills the room.
Desk banks use 180, so two overlapping bookings at 80% each would still be within capacity.
Coworking lounges use 300, allowing several concurrent partial bookings before hitting the limit.

Now add the events array. Each event represents a booking with a category, host, team, and time window:

const events: GenericRecord[] = [
    {
        _id             : 'booking-1',
        name            : 'Weekly launch review',
        bookingCategory : 'Meeting',
        host            : 'Ava',
        team            : 'Product',
        attendees       : 7,
        notes           : 'Mirror a client-facing property review flow with no overlaps.',
        startDate       : '2026-03-16T08:30:00',
        endDate         : '2026-03-16T11:30:00'
    },
    {
        _id             : 'booking-2',
        name            : 'Quiet desk block',
        bookingCategory : 'Desk',
        host            : 'Jules',
        team            : 'Growth',
        attendees       : 5,
        notes           : 'Focus time for a temporary landing team.',
        startDate       : '2026-03-16T09:00:00',
        endDate         : '2026-03-16T17:00:00'
    },
    {
        _id             : 'booking-3',
        name            : 'Sales deal room',
        bookingCategory : 'Meeting',
        host            : 'Maya',
        team            : 'Sales',
        attendees       : 12,
        notes           : 'Large suite booking for end-of-quarter negotiations.',
        startDate       : '2026-03-16T11:00:00',
        endDate         : '2026-03-16T14:00:00'
    },
    {
        _id             : 'booking-4',
        name            : 'Design sprint desks',
        bookingCategory : 'Desk',
        host            : 'Niko',
        team            : 'Design',
        attendees       : 6,
        notes           : 'Desk bank reserved as one inventory block.',
        startDate       : '2026-03-17T08:00:00',
        endDate         : '2026-03-17T18:00:00'
    },
    {
        _id             : 'booking-5',
        name            : 'Founder coffee session',
        bookingCategory : 'Coworking',
        host            : 'Lebo',
        team            : 'Community',
        attendees       : 14,
        notes           : 'Shared-capacity lounge booking with flexible units.',
        startDate       : '2026-03-17T10:00:00',
        endDate         : '2026-03-17T13:00:00'
    },
    {
        _id             : 'booking-6',
        name            : 'Board prep workshop',
        bookingCategory : 'Meeting',
        host            : 'Dylan',
        team            : 'Operations',
        attendees       : 5,
        notes           : 'Smaller room blocked for a prep workshop.',
        startDate       : '2026-03-17T12:30:00',
        endDate         : '2026-03-17T14:30:00'
    },
    {
        _id             : 'booking-7',
        name            : 'Product all-hands overflow',
        bookingCategory : 'Coworking',
        host            : 'Rae',
        team            : 'Product',
        attendees       : 20,
        notes           : 'Lounge used as shared overflow for team gathering.',
        startDate       : '2026-03-18T09:00:00',
        endDate         : '2026-03-18T12:00:00'
    },
    {
        _id             : 'booking-8',
        name            : 'Engineering hot-desk rotation',
        bookingCategory : 'Desk',
        host            : 'Theo',
        team            : 'Engineering',
        attendees       : 4,
        notes           : 'Desk cluster reserved to mirror a rotating occupancy block.',
        startDate       : '2026-03-18T08:00:00',
        endDate         : '2026-03-18T16:00:00'
    }
];

The bookings span three days and cover all three workspace types.

Finally, add the assignments, calendar, dependencies, and the seed function:

const assignments: GenericRecord[] = [
    { _id : 'assign-1', eventId : 'booking-1', resourceId : 'room-terrace', units : 100 },
    { _id : 'assign-2', eventId : 'booking-2', resourceId : 'desk-orbit-1', units : 80 },
    { _id : 'assign-3', eventId : 'booking-3', resourceId : 'room-harbor', units : 100 },
    { _id : 'assign-4', eventId : 'booking-4', resourceId : 'desk-orbit-2', units : 100 },
    { _id : 'assign-5', eventId : 'booking-5', resourceId : 'coworking-canvas', units : 55 },
    { _id : 'assign-6', eventId : 'booking-6', resourceId : 'room-sprint', units : 100 },
    { _id : 'assign-7', eventId : 'booking-7', resourceId : 'coworking-sky', units : 70 },
    { _id : 'assign-8', eventId : 'booking-8', resourceId : 'desk-focus', units : 65 }
];

const calendars: GenericRecord[] = [
    {
        _id       : 'workspace-hours',
        name      : 'Workspace Hours',
        intervals : [
            {
                recurrentStartDate : 'every weekday at 07:00',
                recurrentEndDate   : 'every weekday at 20:00',
                isWorking          : true
            }
        ]
    }
];

const dependencies: GenericRecord[] = [
    { _id : 'dep-1', fromEvent : 'booking-1', toEvent : 'booking-6' },
    { _id : 'dep-2', fromEvent : 'booking-4', toEvent : 'booking-8' },
    { _id : 'dep-3', fromEvent : 'booking-5', toEvent : 'booking-7' }
];

async function seed(): Promise<void> {
    await client.connect();

    const db = client.db(MONGODB_DB);
    const collectionNames = ['resources', 'events', 'assignments', 'calendars', 'dependencies'];

    await Promise.all(collectionNames.map(name => db.collection(name).deleteMany({})));
    await db.collection('resources').insertMany(resources);
    await db.collection('events').insertMany(events);
    await db.collection('assignments').insertMany(assignments);
    await db.collection('calendars').insertMany(calendars);
    await db.collection('dependencies').insertMany(dependencies);

    console.log(`Seeded ${MONGODB_DB} with workspace scheduler data.`);

    await client.close();
}

void seed().catch(error => {
    console.error(error);
    process.exit(1);
});

Assignments connect each booking to a workspace. The units field sets the utilization percentage for each assignment, which the Bryntum Scheduler Pro utilization widget will display.

The seed script clears all five collections before inserting the example data, so every run resets the database to the same state.

Now seed the database:

npm run seed

You can see the created collections and documents in your MongoDB Atlas dashboard. In the left navigation menu, under Database, select Clusters. Click on the Browse Collections to open the Data Explorer:

Creating the load and sync routes

Now let's create the load and sync routes that will load data into the Bryntum Scheduler Pro and save data changes to the MongoDB Atlas database.

Create a scheduler.ts file in the server/src/routes folder. Add the following imports, router factory, and the /load request route:

import { type Request, type Response, Router } from 'express';
import type { Db } from 'mongodb';
import {
    COLLECTIONS,
    applyStoreChanges,
    collectStoreRows,
    createIdMaps,
    normalizeAssignmentFields,
    normalizeDependencyFields,
    normalizeEventFields,
    type GenericRecord,
    type SyncRequestBody,
    type SyncResponseBody
} from '../lib/schedulerCrud.js';

type GetDb = () => Db;

export function createSchedulerRouter(getDb: GetDb): Router {
    const router = Router();

    router.get('/load', async(_req: Request, res: Response) => {
        try {
            const database = getDb();
            const [resources, events, assignments, calendars, dependencies] = await Promise.all([
                database.collection<GenericRecord>(COLLECTIONS.resources).find({}).sort({ neighborhood : 1, floor : 1, name : 1 }).toArray(),
                database.collection<GenericRecord>(COLLECTIONS.events).find({}).sort({ startDate : 1 }).toArray(),
                database.collection<GenericRecord>(COLLECTIONS.assignments).find({}).toArray(),
                database.collection<GenericRecord>(COLLECTIONS.calendars).find({}).toArray(),
                database.collection<GenericRecord>(COLLECTIONS.dependencies).find({}).toArray()
            ]);

            res.json({
                success      : true,
                resources    : { rows : collectStoreRows(resources) },
                events       : { rows : collectStoreRows(events).map(normalizeEventFields) },
                assignments  : { rows : collectStoreRows(assignments) },
                calendars    : { rows : collectStoreRows(calendars) },
                dependencies : { rows : collectStoreRows(dependencies) }
            });
        }
        catch (error) {
            console.error(error);

            res.status(500).json({
                success : false,
                message : 'Could not load workspace scheduler data'
            });
        }
    });

The /load route queries all five MongoDB collections in parallel and returns them in the structure that Bryntum's ProjectModel expects. Resources are sorted by neighborhood, floor, and name so the grid rows appear in a logical order.

Add the /sync request route below the /load route:

router.post('/sync', async(req: Request, res: Response) => {
        const {
            requestId,
            resources,
            events,
            assignments,
            calendars,
            dependencies
        } = req.body as SyncRequestBody;

        const idMaps = createIdMaps();

        try {
            const database = getDb();
            const response: SyncResponseBody = {
                success   : true,
                requestId : requestId || null
            };

            if (resources) {
                const rows = await applyStoreChanges({
                    collection : database.collection<GenericRecord>(COLLECTIONS.resources),
                    changes    : resources,
                    idMap      : idMaps.resources
                });

                if (rows.length) {
                    response.resources = { rows };
                }
            }

            if (events) {
                const rows = await applyStoreChanges({
                    collection       : database.collection<GenericRecord>(COLLECTIONS.events),
                    changes          : events,
                    idMap            : idMaps.events,
                    normalizeAdded   : normalizeEventFields,
                    normalizeUpdated : normalizeEventFields
                });

                if (rows.length) {
                    response.events = { rows };
                }
            }

            if (assignments) {
                const rows = await applyStoreChanges({
                    collection : database.collection<GenericRecord>(COLLECTIONS.assignments),
                    changes    : assignments,
                    idMap      : idMaps.assignments,
                    normalizeAdded(record) {
                        return normalizeAssignmentFields(record, idMaps);
                    },
                    normalizeUpdated(record) {
                        return normalizeAssignmentFields(record, idMaps);
                    }
                });

                if (rows.length) {
                    response.assignments = { rows };
                }
            }

            if (calendars) {
                const rows = await applyStoreChanges({
                    collection : database.collection<GenericRecord>(COLLECTIONS.calendars),
                    changes    : calendars,
                    idMap      : idMaps.calendars
                });

                if (rows.length) {
                    response.calendars = { rows };
                }
            }

            if (dependencies) {
                const rows = await applyStoreChanges({
                    collection : database.collection<GenericRecord>(COLLECTIONS.dependencies),
                    changes    : dependencies,
                    idMap      : idMaps.dependencies,
                    normalizeAdded(record) {
                        return normalizeDependencyFields(record, idMaps);
                    },
                    normalizeUpdated(record) {
                        return normalizeDependencyFields(record, idMaps);
                    }
                });

                if (rows.length) {
                    response.dependencies = { rows };
                }
            }

            res.json(response);
        }
        catch (error) {
            console.error(error);

            res.status(500).json({
                success   : false,
                requestId : requestId || null,
                message   : 'Could not sync workspace scheduler data'
            } satisfies SyncResponseBody);
        }
    });

    return router;
}

The /sync route processes each store in order: resources and events first, then assignments and dependencies. This order matters because assignments reference event and resource IDs, so those ID maps need to be populated before the assignment normalizer runs. For added rows, the response includes the database-assigned IDs so the Bryntum Scheduler Pro can replace its temporary phantom IDs with the database IDs.

Creating the Express server entry point

Create a server.ts in the server/srcfolder and add the following code to it:

import express from 'express';
import { type Db } from 'mongodb';
import cors from './lib/cors.js';
import { loadEnv } from './lib/loadEnv.js';
import { createMongoClient } from './lib/mongo.js';
import { createSchedulerRouter } from './routes/scheduler.js';

loadEnv();

const PORT = Number(process.env.PORT || 1339);
const FRONTEND_URL = process.env.FRONTEND_URL || 'http://localhost:5173';
const MONGODB_URI = process.env.MONGODB_URI || 'mongodb://127.0.0.1:27017';
const MONGODB_DB = process.env.MONGODB_DB || 'workspace_scheduler2';

const app = express();
app.use(cors({
    origin : FRONTEND_URL
}));
app.use(express.json({ limit : '2mb' }));

const client = createMongoClient(MONGODB_URI);
let db: Db | undefined;

function getDb(): Db {
    if (!db) {
        throw new Error('Database connection is not ready yet');
    }

    return db;
}

app.use('/api', createSchedulerRouter(getDb));

async function start(): Promise<void> {
    await client.connect();
    db = client.db(MONGODB_DB);
    console.log(`MongoDB ready for database "${MONGODB_DB}"`);

    app.listen(PORT, () => {
        console.log(`Workspace Scheduler server running on http://localhost:${PORT}`);
    });
}

start().catch((error) => {
    console.error('Failed to start server:', error);
    process.exit(1);
});

This server loads the environment variables, connects to MongoDB, and mounts the scheduler routes under /api.

Creating the Vite client app

Generate the Vite vanilla TypeScript client app using the following command:

npm create vite@latest client -- --template vanilla-ts

Select Install with npm and start now in the CLI options.

Replace the "scripts" object in the client/package.json with the following scripts:

"scripts": {
  "dev": "vite",
  "build": "tsc -p tsconfig.json && vite build"
}

Replace the configuration in client/tsconfig.json with the following code:
{
  "extends": "../tsconfig.base.json",
  "compilerOptions": {
    "module": "ESNext",
    "moduleResolution": "Bundler",
    "lib": ["ES2022", "DOM", "DOM.Iterable"],
    "types": ["vite/client"],
    "noEmit": true
  },
  "include": ["src/**/*.ts", "vite.config.ts"]
}

This config uses the Vite bundler module resolution strategy and does not emit JavaScript during the TypeScript check.

Create a vite.config.ts file in the client folder and add the following lines of code to it:

import { defineConfig } from 'vite';

export default defineConfig({
    server : {
        port  : 5173,
        proxy : {
            '/api' : 'http://localhost:1339'
        }
    },
    build : {
        outDir : 'dist'
    }
});

This config proxies /api requests to the Express server running on port 1339, so the client can use relative API URLs during local development.

Replace client/index.html with the following code:

<!doctype html>
<html lang="en">
<head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Workspace Scheduler</title>
</head>
<body>
    <main id="app">
        <div id="scheduler" class="scheduler-host"></div>
        <div id="utilization" class="utilization-host"></div>
    </main>
    <script type="module" src="/src/main.ts"></script>
</body>
</html>

This HTML creates two mount points: one for the Scheduler Pro, and one for the linked resource utilization view.

Delete the Vite starter files that we do not use:

rm -r client/public/icons.svg client/src/assets client/src/counter.ts

Installing the Bryntum Scheduler Pro

If you’re using the free trial, install the public Bryntum trial package:

npm install @bryntum/schedulerpro@npm:@bryntum/schedulerpro-trial  --workspace=client

If you have a Bryntum license, refer to our npm Repository Guide and install the licensed package:

npm install @bryntum/schedulerpro --workspace=client

Configuring the Bryntum Scheduler Pro

Create a schedulerProConfig.ts file in the client/src folder. Add the following imports, custom BookingModel, color map, filter dropdown options, and options objects:

import {
    DateHelper,
    EventModel,
    ProjectModel,
    StringHelper,
    type ProjectModelConfig,
    type ResourceUtilizationConfig,
    type SchedulerEventModel,
    type SchedulerProConfig
} from '@bryntum/schedulerpro';

class BookingModel extends EventModel {
    static override fields = [
        { name : 'bookingCategory', type : 'string' },
        { name : 'host', type : 'string' },
        { name : 'team', type : 'string' },
        { name : 'attendees', type : 'number' },
        { name : 'notes', type : 'string' }
    ];
}

const bookingColors: Record<string, string> = {
    Meeting   : '#ea6d4f',
    Desk      : '#267b8a',
    Coworking : '#d69a27'
};

export const workspaceTypeOptions = [
    { value : 'all', text : 'All types' },
    { value : 'Meeting room', text : 'Meeting room' },
    { value : 'Desk bank', text : 'Desk bank' },
    { value : 'Coworking lounge', text : 'Coworking lounge' }
] as const;

export const neighborhoodOptions = [
    { value : 'all', text : 'All neighborhoods' },
    { value : 'Docklands', text : 'Docklands' },
    { value : 'Atrium North', text : 'Atrium North' },
    { value : 'Harbor Wing', text : 'Harbor Wing' }
] as const;

export const filterState = {
    workspaceType : 'all',
    neighborhood  : 'all'
};

The BookingModel adds custom fields to the Bryntum EventModel. The bookingColors map gives each booking category a distinct color on the timeline.

Add the project configuration and instantiation:

export const projectConfig: Partial<ProjectModelConfig> = {
    eventModelClass : BookingModel,
    calendar        : 'workspace-hours',
    transport : {
        load : {
            url : '/api/load'
        },
        sync : {
            url : '/api/sync'
        }
    },
    autoLoad         : true,
    autoSync         : true,
    validateResponse : true
};

export const project = new ProjectModel(projectConfig);

The projectConfig transport property configures the load and sync URLs that the Bryntum Scheduler Pro Crud Manager uses. These URLs are the Express API endpoints that we created. Because autoLoad and autoSync are both enabled, the scheduler fetches data on page load and posts changes back to the API whenever data is modified in the scheduler.

Add the following scheduler configuration:

export const schedulerConfig: Partial<SchedulerProConfig> = {
    appendTo   : 'scheduler',
    project,
    startDate  : '2026-03-16T07:00:00',
    endDate    : '2026-03-18T20:00:00',
    viewPreset : 'hourAndDay',
    rowHeight  : 62,
    barMargin  : 7,
    multiEventSelect : true,
    columns          : [
        {
            type  : 'resourceInfo',
            text  : 'Workspace',
            field : 'name',
            width : 240
        },
        {
            text  : 'Type',
            field : 'workspaceType',
            width : 145
        },
        {
            text  : 'Neighborhood',
            field : 'neighborhood',
            width : 150
        },
        {
            text  : 'Floor',
            field : 'floor',
            width : 90
        },
        {
            text   : 'Capacity',
            field  : 'capacity',
            width  : 100,
            align  : 'center',
            editor : false
        }
    ],
    features : {
        dependencies : true,
        eventMenu    : true,
        taskEdit     : {
            editorConfig : {
                title : 'Reservation',
                width : '36em'
            },
            items : {
                generalTab : {
                    title : 'Booking details',
                    items : {
                        resourcesField : {
                            label : 'Workspace'
                        },
                        bookingCategory : {
                            type     : 'combo',
                            name     : 'bookingCategory',
                            label    : 'Booking type',
                            editable : false,
                            items    : ['Meeting', 'Desk', 'Coworking'],
                            weight   : 210
                        },
                        host : {
                            type   : 'text',
                            name   : 'host',
                            label  : 'Host',
                            weight : 220
                        },
                        team : {
                            type   : 'text',
                            name   : 'team',
                            label  : 'Team',
                            weight : 230
                        },
                        attendees : {
                            type   : 'number',
                            name   : 'attendees',
                            label  : 'Attendees',
                            min    : 1,
                            weight : 240
                        },
                        notes : {
                            type   : 'text',
                            name   : 'notes',
                            label  : 'Notes',
                            weight : 250
                        }
                    }
                }
            }
        },
    },
    eventRenderer({ eventRecord, renderData }) {
        renderData.eventColor = bookingColors[eventRecord.get('bookingCategory')] || '#1d5b52';

        return {
            children : [
                {
                    className : 'booking-title',
                    text      : eventRecord.name
                },
                {
                    className : 'booking-meta',
                    text      : `${eventRecord.get('host') || 'TBD'} | ${eventRecord.get('attendees') || 0} seats`
                }
            ]
        };
    }
};

Here, we configure the timeline settings, task editor, and event renderer. The columns array defines the locked grid on the left side of the scheduler. The taskEdit feature customizes the reservation editor popup with the booking-specific fields. The eventRenderer sets the event bar color based on booking category and shows the host name and seat count inside each bar.

Add the utilization panel configuration at the end of the file:

export const utilizationConfig: Partial<ResourceUtilizationConfig> = {
    appendTo    : 'utilization',
    project,
    hideHeaders : true,
    rowHeight   : 44,
    showBarTip  : true,
    showBarText : true,
    features    : {
        nonWorkingTime : false
    },
    columns : [
        {
            type  : 'tree',
            text  : 'Resource / booking',
            field : 'name',
            flex  : 1
        }
    ]
};

This configures the resource utilization widget that we'll display below the scheduler. It shares the same project instance as the scheduler, so both views stay in sync. The tree column lets you expand each resource to see its individual bookings. The showBarTip and showBarText options display the utilization percentage on the bars so you can spot overallocated workspaces at a glance.

Replace the styles in the client/src/style.css file with the following:

@import url('https://fonts.googleapis.com/css2?family=Poppins:wght@400;500;600;700&display=swap');
@import '@bryntum/schedulerpro/fontawesome/css/fontawesome.css';
@import '@bryntum/schedulerpro/fontawesome/css/solid.css';
@import '@bryntum/schedulerpro/schedulerpro.css';
@import '@bryntum/schedulerpro/svalbard-light.css';

* {
    margin: 0;
}

body {
    font-family: 'Poppins', 'Segoe UI', Arial, sans-serif;
}

#app {
    display        : flex;
    flex-direction : column;
    height         : 100vh;
    font-size      : 14px;
}

#scheduler {
    flex : 3;
}

#utilization {
    flex       : 2;
    border-top : 2px solid #e0e0e0;
}

This stylesheet imports the Bryntum Scheduler Pro structural CSS, the Svalbard theme, and gives the page a two-panel layout, with the scheduler above the utilization view. Bryntum provides five themes with light and dark variants.

Replace the code in lient/src/main.ts with the following:

import {
    Combo,
    ResourceUtilization,
    SchedulerPro,
} from '@bryntum/schedulerpro';
import './style.css';
import {
    filterState,
    neighborhoodOptions,
    project,
    schedulerConfig,
    utilizationConfig,
    workspaceTypeOptions
} from './schedulerProConfig';

const scheduler = new SchedulerPro({
    ...schedulerConfig,
    tbar : [
        {
            type         : 'combo',
            ref          : 'workspaceTypeFilter',
            width        : 190,
            editable     : false,
            value        : filterState.workspaceType,
            valueField   : 'value',
            displayField : 'text',
            items        : workspaceTypeOptions,
            placeholder  : 'Workspace type',
            listeners    : {
                change({ value }: { value: string }) {
                    filterState.workspaceType = value || 'all';
                    applyResourceFilters();
                }
            }
        },
        {
            type         : 'combo',
            ref          : 'neighborhoodFilter',
            width        : 205,
            editable     : false,
            value        : filterState.neighborhood,
            valueField   : 'value',
            displayField : 'text',
            items        : neighborhoodOptions,
            placeholder  : 'Neighborhood',
            listeners    : {
                change({ value }: { value: string }) {
                    filterState.neighborhood = value || 'all';
                    applyResourceFilters();
                }
            }
        },
        {
            type     : 'button',
            icon     : 'b-icon-undo',
            text     : 'Reset filters',
            onAction : () => {
                filterState.workspaceType = 'all';
                filterState.neighborhood = 'all';
                (scheduler.widgetMap.workspaceTypeFilter as Combo).value = 'all';
                (scheduler.widgetMap.neighborhoodFilter as Combo).value = 'all';
                applyResourceFilters();
            }
        },
        {
            type     : 'button',
            icon     : 'b-icon-search-plus',
            text     : 'Zoom in',
            onAction : () => scheduler.zoomIn()
        },
        {
            type     : 'button',
            icon     : 'b-icon-search-minus',
            text     : 'Zoom out',
            onAction : () => scheduler.zoomOut()
        }
    ]
});

This instantiates a Scheduler Pro instance, passes in the schedulerConfig, and adds a toolbar with filter dropdowns and zoom buttons. The two combo dropdowns filter the resource grid by workspace type and neighborhood. The reset button clears both filters, and the zoom buttons let you adjust the timeline scale.

Create the partnered utilization view and the filter function below the scheduler:

new ResourceUtilization({
    ...utilizationConfig,
    partner : scheduler
});

function applyResourceFilters(): void {
    const selectedType = filterState.workspaceType;
    const selectedNeighborhood = filterState.neighborhood;

    scheduler.resourceStore.filter({
        filters : (resource: Record<string, unknown>) => {
            const matchesType = selectedType === 'all' || resource.workspaceType === selectedType;
            const matchesNeighborhood = selectedNeighborhood === 'all' || resource.neighborhood === selectedNeighborhood;

            return matchesType && matchesNeighborhood;
        },
        replace : true
    });
}

project.on({
    load() {
        applyResourceFilters();
    }
});

The partner property links the utilization panel to the scheduler so they share the same time axis and scroll position. The applyResourceFilters function runs a combined filter on the resource store whenever a dropdown changes. The project.on('load') listener reapplies the filters after the initial data load completes.

Running the application

Run the client and server concurrently using the following command:

npm run dev

Open http://localhost:5173/. You’ll see the Bryntum Scheduler Pro populated with the example data:

[VIDEO]

You now have a Bryntum Scheduler Pro workspace booking CRUD app powered by MongoDB Atlas. The linked resource utilization view lets you spot overallocated resources.

Next Steps

Now that you have a working Bryntum Scheduler Pro app with MongoDB, explore the additional features you can add on the examples page, such as:

Check out the Bryntum AI feature that lets users interact with scheduling data using natural language through a chat panel.

Both Bryntum and MongoDB have MCP servers to assist with agentic workflows:

The Bryntum MCP server returns API references, configuration options, working source code examples, and migration guides for specific Bryntum versions.
The MongoDB MCP server helps with code generation, database optimization, and data exploration.

How MongoDB Handles High Traffic and Large Datasets

MongoDB Guests — Mon, 30 Mar 2026 11:44:41 +0000

This article was written by Karen Zhang.

MongoDB is built for scale. Its distributed architecture combines replica sets, sharding, and intelligent query routing to handle high traffic and large datasets without requiring you to endlessly upgrade a single server.

This article covers how each of these mechanisms works, when to apply them, and what you need to get right to avoid performance problems in production.

Horizontal Scaling in MongoDB

1. The limits of vertical scaling

When a database starts slowing down, the instinct is to add more hardware: more CPU cores, more RAM, faster storage. This is vertical scaling, and it works up to a point.

The problem is that a single server can only get so large. Beyond a certain threshold, the cost grows exponentially while the performance gains diminish. More importantly, a single powerful server is still a single point of failure. If it goes down, your entire application goes with it.

Horizontal scaling solves this differently. Instead of making one machine larger, you distribute the work across multiple machines. MongoDB is designed for this from the ground up.

2. Replica sets for availability and read scalability

The foundation of MongoDB’s horizontal architecture is the replica set — a group of MongoDB instances that all hold the same data, kept in sync through continuous replication.

A replica set consists of:

Primary: The only member that accepts write operations. All writes go here.
Secondaries: Members that replicate from the primary’s oplog (operation log) and maintain a copy of the data.
Arbiter (optional): Participates in elections but holds no data. Used to break ties in smaller deployments.

Primary  ──oplog──►  Secondary 1
         ──oplog──►  Secondary 2
         ──oplog──►  Secondary 3 (hidden, used for backups)

When the primary becomes unavailable, the remaining members automatically elect a new primary — typically within a few seconds — with no manual intervention required. For deployment architecture options, refer to replica set deployment architectures.

3. Handling concurrent reads and writes at scale

By default, all reads and writes go through the primary. This keeps things simple and guarantees you’re always reading the latest data. For many applications, this is the right choice.

For read-heavy workloads where some replication lag is acceptable — analytics dashboards, reporting queries, batch exports — you can offload reads to secondaries. This distributes read traffic across members and reduces pressure on the primary.

Please note that secondary reads are not appropriate for anything requiring read-your-own-writes consistency. If a user writes a record and immediately reads it back, routing that read to a secondary may return stale data.

4. Traffic distribution using read preferences

MongoDB gives you five read preference modes to control where read operations are sent:

Mode	Behavior
primary	All reads from primary (default).
primaryPreferred	Primary when available, fallback to secondary.
secondary	Always read from a secondary.
secondaryPreferred	Secondary, when available, fallback to primary.
nearest	Member with the lowest network latency, regardless of role. Does not guarantee reads stay within a region — latency measurements can change and may cross region boundaries. For strict regional routing, use tags with `readPref('secondary', [{ region: 'us-east' }])` rather than relying on `nearest` alone.

For multi-region deployments, nearest is particularly useful. Users in Europe read from European replicas, users in Asia read from Asian replicas — MongoDB’s server selection algorithm handles this automatically based on measured network latency.

// Route reads to a secondary tagged for a specific region
db.orders.find({ status: "completed" }).readPref("secondary", [{ region: "eu-west" }])

For a detailed breakdown of when each mode is appropriate, refer to read preference use cases.

Sharding for Large Datasets

1. When and why sharding is required

Replica sets solve availability and read scalability, but they do not solve storage capacity or write throughput. Every replica holds a complete copy of your data, so storage scales with the number of nodes rather than across them. Write throughput is still bounded by what the primary can handle alone.

Sharding addresses this by distributing data across multiple replica sets, each holding a subset of the data. You should consider sharding when:

Your dataset is too large for a single server to store efficiently
Write throughput has saturated your primary node
Your working set no longer fits in memory, causing excessive disk I/O

Do not shard prematurely. A well-indexed, well-resourced replica set handles far more load than most applications will ever generate. Sharding adds operational complexity, and reversing it is painful. Only shard when you have clear evidence you need it. Review operational restrictions in sharded clusters before committing.

2. Sharded cluster components

A sharded MongoDB cluster is made up of three types of components:

Shards: Each shard is a replica set that holds a subset of your sharded data. With three shards, each holds roughly one-third of the collection. Because each shard is itself a replica set, you retain fault tolerance at the shard level.

Config servers: A replica set that stores the cluster’s metadata — specifically, the mapping of which chunk ranges live on which shard. Config servers must always be available; if they go down, the cluster cannot route new operations.

mongos: A lightweight query router that sits between your application and the shards. Your application connects to mongos exactly as it would a standalone MongoDB instance — the sharding complexity is invisible. mongos caches routing metadata from the config servers and forwards operations to the appropriate shard or shards.

Application
    │
    ▼
 mongos (query router)
    │
    ├──► Shard 1 (replica set: primary + 2 secondaries)
    ├──► Shard 2 (replica set: primary + 2 secondaries)
    └──► Shard 3 (replica set: primary + 2 secondaries)

Config Servers (replica set)

3. How data is distributed across shards

MongoDB distributes data at the collection level using data partitioning with chunks. When you shard a collection, you specify a shard key — a field or combination of fields that MongoDB uses to divide data into chunks and assign those chunks to shards.

Each chunk covers a contiguous range of shard key values. As data grows, MongoDB’s balancer process monitors chunk distribution and migrates chunks between shards automatically to maintain an even distribution.

Shard Key Design and Performance

1. Importance of shard key selection

Shard key selection is the single most consequential architectural decision in a sharded deployment. The shard key determines how data is distributed across shards, which queries can be routed efficiently, and whether write traffic is balanced or concentrated on a single shard.

Please note that changing a shard key after the fact is extremely disruptive. MongoDB added support for resharding in recent versions, but it is a costly operation. Get this right before you shard.

2. Range vs. hashed shard keys

Range sharding partitions data into contiguous ranges of shard key values. This makes range queries efficient — a query filtering on a date range, for example, can be routed to one or two shards instead of all of them.

The downside is that monotonically increasing shard keys (timestamps, auto-incrementing IDs) cause all new writes to land on the same shard — the one holding the highest range. This creates a write hotspot and defeats the purpose of sharding.

Hashed sharding applies a hash function to the shard key value before distribution. This scatters writes evenly across shards, eliminating hotspots. The trade-off is range queries — because the hash destroys value locality, a range query must be sent to all shards. Note that hashed sharding distributes writes evenly only when the shard key has high cardinality. A hashed boolean field, for example, still concentrates data on two shards at most.

// Range shard key — efficient range queries, risk of write hotspots on monotonic fields
sh.shardCollection("mydb.events", { timestamp: 1 })

// Hashed shard key — even write distribution, no range query efficiency
sh.shardCollection("mydb.events", { _id: "hashed" })

3. Avoiding write hotspots and uneven distribution

A good shard key satisfies three criteria:

High cardinality: Enough distinct values to distribute data across many chunks. A boolean field, for example, can only ever produce two chunks regardless of how many documents you have.
Low frequency: No single value that dominates the dataset. If 40% of your documents share the same shard key value, 40% of your data lands on one shard.
Non-monotonically increasing (for write-heavy workloads): Avoid timestamps or sequential IDs as standalone shard keys. They route all new writes to the maximum chunk, creating a hotspot.

When no single field satisfies all three criteria, a compound shard key can help. Pairing a high-cardinality field with a monotonically increasing field — like { user_id: 1, timestamp: 1 } — can give you even distribution while preserving some query locality.

Query Routing and Scalability

1. Targeted vs. scatter-gather queries

When mongos receives a query, it inspects the filter to determine whether the shard key is included.

Targeted query: The filter includes the shard key. mongos routes the query directly to the one (or occasionally two) shards that hold the relevant data. This is fast and efficient.
Scatter-gather query: The filter does not include the shard key. mongos must broadcast the query to every shard, collect all results, and merge them. Latency is bounded by the slowest shard, and cost scales with the number of shards.

// Targeted — shard key (user_id) is included in the filter
db.orders.find({ user_id: "user_123", status: "pending" })

// Scatter-gather — no shard key, every shard is queried
db.orders.find({ status: "pending" })

This is why your access patterns should drive shard key selection. Before sharding, audit your most frequent and most latency-critical queries. The shard key should align with what those queries filter on.

2. Indexing considerations in sharded environments

Each shard maintains its own independent set of indexes. When you create an index on a sharded collection, MongoDB creates it on every shard. A few rules apply:

Unique indexes must include the shard key as a prefix. MongoDB cannot enforce global uniqueness on a field that isn’t the shard key, because there is no global index spanning all shards. If you need global uniqueness across all shards, the shard key must be a prefix of the unique index. Non-unique indexes have no such restriction.
The shard key is automatically indexed. You do not need to create an index on it separately.
Compound indexes that lead with the shard key field work well for targeted queries and can serve as covered indexes, eliminating document fetches entirely.

3. Impact on latency and throughput

Targeted queries in a sharded cluster can be faster than the equivalent query on a single large server, because each shard operates on a smaller dataset. Scatter-gather queries, however, are slower than on a single server — the overhead of fan-out, parallel execution, and result merging adds latency that doesn’t exist when querying one node.

The practical implication: sharding improves throughput and capacity, but it does not automatically improve query latency. Latency improves only when your queries are targeted. Design your shard key accordingly.

Elasticity and Fault Tolerance

1. Automatic balancing and chunk migration

As data grows and new shards are added, chunks become unevenly distributed. MongoDB’s background balancer detects this and migrates chunks between shards automatically to restore balance.

Chunk migration is a live operation. MongoDB copies data to the destination shard, then atomically updates the routing table in the config servers. Reads and writes continue normally throughout — your application sees no interruption.

For write-heavy workloads, migration traffic can compete with application traffic. You can restrict balancing to a scheduled window during off-peak hours:

use config
db.settings.updateOne(
  { _id: "balancer" },
  { $set: { activeWindow: { start: "02:00", stop: "06:00" } } },
  { upsert: true }
)

2. Scaling out with minimal downtime

Adding a new shard to a running cluster is an online operation. You provision the new replica set, register it with sh.addShard(), and the balancer begins migrating chunks to it automatically. No application configuration changes are needed — mongos picks up the updated routing table from the config servers.

sh.addShard("rs3/mongodb-shard3-1:27017,mongodb-shard3-2:27017,mongodb-shard3-3:27017")

If your access patterns change significantly and your current shard key is no longer appropriate, MongoDB’s resharding feature lets you migrate to a new shard key without rebuilding the cluster from scratch — a significant improvement over earlier versions of MongoDB, where the shard key was effectively permanent.

3. High availability through replica sets

Every layer of a sharded cluster is replicated:

Each shard is a replica set. If a shard’s primary fails, its replica set holds an automatic election and promotes a secondary within seconds.
The config servers are a replica set. Losing one config server does not take the cluster offline.
mongos instances are stateless. You can run multiple mongos instances behind a load balancer, and losing one affects only the connections it was actively handling.

There is no single point of failure at any layer of the architecture. For a detailed look at failure scenarios and how MongoDB handles each one, refer to sharded cluster high availability.

Summary and Next Steps

MongoDB’s scaling approach is layered. Replica sets provide read scalability and fault tolerance as a baseline. Sharding extends this to write throughput and storage capacity. mongos keeps the distributed complexity invisible to application code, and automatic balancing ensures the cluster adapts to growth without manual intervention.

Key takeaways for designing scalable MongoDB workloads:

Do not shard until you have evidence you need to. A well-tuned replica set handles more than most applications require. Shard only when you’ve identified a clear bottleneck that sharding solves.
Let your access patterns drive shard key selection. The queries you run most frequently should determine the shard key — not your schema or your data model in isolation.
Use hashed sharding for write-heavy workloads, range sharding when range queries matter. Know which trade-off your application can live with before you commit.
Every scatter-gather query is a cost that scales with your cluster size. Design for targeted queries wherever possible.
Use read preferences deliberately. Secondary reads are a useful tool for the right workloads, but they introduce replication lag — make sure that trade-off is acceptable for the feature using them.

For hands-on practice, MongoDB University offers free courses on sharding and replica set administration. The official sharding documentation is also one of the more thorough references available for production deployment considerations.

MongoDB Schema Design: Do’s and Don’ts for Real Projects

MongoDB Guests — Wed, 25 Mar 2026 09:50:36 +0000

This tutorial was written by Nancy Agarwal.

Understanding MongoDB Schema Design

If you have worked with MongoDB, you have probably heard someone say, "MongoDB is schema-less — just store whatever JSON you want." This is one of the biggest misconceptions.

MongoDB is not schema-less — it is schema-flexible. That flexibility is powerful, but it also means developers must take responsibility for good schema design.

When schema design is done correctly:

Queries become extremely fast
APIs stay simple
Applications scale smoothly

When schema design is ignored:

Queries become slow
Documents grow bloated
Updates become difficult
Systems become harder to maintain

MongoDB Schema Design Do's

Model for Access Patterns, Not Entities

Relational databases typically start with entities and relationships. MongoDB flips this approach. Instead of starting with tables, you should start with how your application accesses data.

Ask yourself:

What are the most common queries?
What data needs to be fetched together?
What operations happen most frequently?

Example

In one HR system, employee data was always fetched along with department and manager information. Instead of performing joins with $lookup, we used the Extended Reference Pattern.

Key manager fields were embedded inside the employee document, while full manager details were stored separately.

This allowed most reads to happen in a single document query, keeping performance extremely fast.

Patterns to Consider

Extended Reference Pattern

Duplicate important parent fields in child documents to avoid joins.

Pre-Compute Pattern

Store computed values such as totals, aggregates, or derived metrics so they do not need to be recalculated repeatedly.

Pro Tip: Before finalizing your schema, write down the top three queries your application performs. If your schema cannot serve them efficiently, redesign it.

Use the Pre-Compute Pattern for Expensive Aggregations

MongoDB’s aggregation framework is powerful, but running heavy aggregations repeatedly can become expensive at scale.

Example

In one analytics system, every dashboard load executed several $group aggregations on a 50 million record collection.

The result: query latency of over 3 seconds.

We solved it using the Pre-Compute Pattern. Aggregations were computed nightly and stored in a collection like dailyUserMetrics.
Dashboards then simply read the pre-aggregated results, reducing latency to around 80 milliseconds.

Use this pattern for:

Dashboards
Leaderboards
Analytics reports

Use the Polymorphic Pattern for Flexible Data Types

Sometimes documents share a common structure but contain different fields depending on their type.

Example CRM activities documents:
{ type: "email", subject: "...", recipients: [...] }
{ type: "call", duration: 180, recordingUrl: "..." }
{ type: "meeting", participants: [...], location: "..." }

Each document shares common metadata like timestamps or userId, but contains type-specific attributes.

Use the Bucket Pattern for Time-Series Data

Applications storing logs, telemetry, or IoT events often generate millions of small records.

The Bucket Pattern groups multiple readings into a single document.

Example:
{
deviceId: "D123",
startTime: ISODate("2025-10-29T10:00:00Z"),
readings: [
{ t: 0, value: 1.02 },
{ t: 1, value: 1.05 }
]
}

This dramatically reduces document count and improves query performance.

Version and Evolve Your Schema

Schemas evolve as products grow. Ignoring schema evolution leads to inconsistent documents and fragile code.

Example older invoice:
{ amount: 100, currency: "USD" }

Example newer invoice:
{ amount: 100, currency: "USD", tax: 10, discount: 5, schemaVersion: 2 }

MongoDB Schema Design Don’ts

Don’t Let Arrays Grow Unbounded

Unbounded arrays are a common MongoDB anti-pattern. When arrays grow indefinitely, documents become large and slow.

In one system, all user sessions were stored inside a user document. Some users accumulated over 50,000 sessions, causing slow queries.

A better approach was to store sessions in a separate collection referenced by userId.

Don’t Treat MongoDB Like a Relational Database

Over-normalizing data leads to excessive joins using $lookup.

Recommended approach:

Embed when relationships are tight
Reference when data is large or independent
Denormalize intentionally

Don’t Over-Index or Over-Engineer

Indexes improve reads but increase memory usage and slow writes.

Best practices:

Only index fields used by queries
Periodically review unused indexes
Avoid excessive collections

Final Takeaways

Design around access patterns, not tables.
Use the right schema patterns - Extended Reference, Pre-Compute, 3. Polymorphic, Bucket - to shape data for performance.
Never let arrays grow unbounded.
Monitor document size and index efficiency.
Version your schema - evolution is inevitable.

Conclusion

MongoDB’s schema flexibility is powerful but requires thoughtful design. By applying these patterns and avoiding common anti-patterns, teams can build MongoDB systems that remain performant as data grows.

If you found this useful, I'd love to hear how you're modeling your MongoDB schemas in production - or what anti-patterns you've encountered. Let's trade war stories in the comments.

How to Use MongoDB’s Text Search

MongoDB Guests — Fri, 20 Mar 2026 10:27:56 +0000

This tutorial was written by Damilola Oladele.

How to Use MongoDB's Text Search

MongoDB Search is an embedded full-text search engine that lives inside your MongoDB deployment. You don't need to run a separate search system alongside your database. You get fine-grained text indexing, a rich query language for complex search logic, customizable relevance scoring, and advanced features like autocomplete and faceted search.

This tutorial shows you how to insert a sample book catalog into MongoDB, create a search index, and run six types of text search queries in Python.

You'll learn the following in this tutorial:

What search queries, search indexes, and search analyzers are in MongoDB
How to insert sample data and create a search index
How to run a basic text search
How to search across multiple fields
How to search with fuzzy matching
How to search for a term while excluding another word
How to combine search conditions with the compound operator
How to include relevance scores in your results

You can find all the code samples for this tutorial in the GitHub repository.

What You Need to Know Before You Start

Three concepts are central to text search in MongoDB. They'll help you understand why each search operation behaves the way it does.

Search queries

A search query asks MongoDB to find documents that are relevant to a term or phrase. Search queries are different from traditional database queries. A traditional database query follows a strict syntax and returns exact matches. On the other hand, a search query can match similar phrases, tolerate typos, support wildcards, and rank results by how relevant they are.

MongoDB Search queries take the form of an aggregation pipeline stage. The $search stage returns the actual search results. The $searchMeta stage returns metadata about the results, such as counts or facet data. Both stages must come first in your aggregation pipeline. You then chain additional stages, like $project or $limit, after them.

Search indexes

A search index is a data structure that maps terms to the documents that contain them. Think of it like the index at the back of a book. Instead of scanning every page to find a topic, you look it up in the index and go straight to the right pages. A search index works the same way. MongoDB consults it instead of scanning every document in your collection, thereby improving search speed.

You must create a search index before you can run any $search queries.

Search analyzers and tokens

An analyzer controls how MongoDB transforms your text into searchable units called tokens. This transformation happens both when MongoDB builds the index and when it processes your query. The analyzer you choose determines which tokens end up in the index and which tokens your query produces.

MongoDB Search ships with five built-in analyzers. The one you choose shapes what your search can and can't find.

Analyzer	How it splits text	Lowercase	Stop words
Standard	Whitespace and punctuation	Yes	Yes
Simple	Any non-letter character	Yes	No
Whitespace	Whitespace only	No	No
Keyword	Treats the whole field as one token	No	No
Language	Language-specific rules	Varies	Varies

If you don't specify an analyzer, MongoDB uses the Standard analyzer. Later in this tutorial, you'll see how different analyzers change what a query finds and why.

Now let's get started.

Prerequisites

Before you start, make sure you have the following ready:

Python 3.8 or later installed
A MongoDB Atlas account with an M0 (Free tier) cluster set up
pymongo 4.0 or later installed
Basic familiarity with Python and MongoDB collections
Your Atlas connection string, available in the Atlas UI under Database > Connect > Drivers

Make sure your IP address is allowlisted in Atlas before you run any scripts. Go to Security > Network Access in the Atlas UI and add your current IP address. Your scripts won't connect to your cluster without this step.

Set Up Your Project

Create a folder for your project and navigate into it in your terminal:

mkdir mongodb-text-search
cd mongodb-text-search

Create and activate a virtual environment:

python -m venv venv
source venv/bin/activate

On Windows, activate it with:

venv\Scripts\activate

Install pymongo:

python -m pip install pymongo

You'll create a separate Python file for each step of this tutorial. All files go in the mongodb-text-search folder.

Insert Your Data and Create a Search Index

Create a file named insert_data.py. This file does two things:

It inserts a sample book catalog into your cluster
It creates the search index you'll use for most of the tutorial.

You only need to run this file once. The other files in this tutorial just connect and query.

from pymongo import MongoClient
import time

client = MongoClient(
    "mongodb+srv://<USERNAME>:<PASSWORD>@<HOST>/",
    appname="devrel-tutorial-python-search"
)
db = client["bookstore"]
books = db["books"]

# Drop the collection to start clean on each run:
books.drop()

books.insert_many([
    {
        "title": "The Midnight Library",
        "author": "Matt Haig",
        "genre": "fiction",
        "description": "A woman discovers a library between life and death that contains books representing all the lives she could have lived.",
        "year": 2020
    },
    {
        "title": "Atomic Habits",
        "author": "James Clear",
        "genre": "self-help",
        "description": "A practical guide to building good habits and breaking bad ones through small, incremental changes.",
        "year": 2018
    },
    {
        "title": "Project Hail Mary",
        "author": "Andy Weir",
        "genre": "science fiction",
        "description": "An astronaut wakes up alone in space with no memory and must figure out how to save Earth from an extinction-level threat.",
        "year": 2021
    },
    {
        "title": "Educated",
        "author": "Tara Westover",
        "genre": "memoir",
        "description": "A woman reflects on her isolated rural childhood and her journey to educate herself and earn a PhD from Cambridge.",
        "year": 2018
    },
    {
        "title": "The Hitchhiker's Guide to the Galaxy",
        "author": "Douglas Adams",
        "genre": "science fiction",
        "description": "An ordinary man is swept across the universe after Earth is demolished to make way for a hyperspace expressway.",
        "year": 1979
    }
])

print("Sample data inserted.")

# Drop any existing search index named 'default' before creating a new one:
try:
    books.drop_search_index("default")
    time.sleep(5)
except Exception:
    pass

# Create a search index with dynamic mappings:
books.create_search_index({
    "name": "default",
    "definition": {
        "mappings": {
            "dynamic": True
        }
    }
})

print("Search index created. Waiting for it to go active...")

# Poll until the index is ready:
while True:
    indexes = list(books.list_search_indexes())
    if indexes and indexes[0].get("status") == "READY":
        print("Search index is active. You're ready to run queries.")
        break
    time.sleep(3)

Replace <USERNAME>, <PASSWORD>, and <HOST> with your actual Atlas credentials.

The dynamic: True setting tells MongoDB to automatically index all supported string fields in your documents. The while loop polls the index status and pauses the script until the index is ready.

Run the script:

python insert_data.py

You'll get the following output in your terminal:

Sample data inserted.
Search index created. Waiting for it to go active...
Search index is active. You're ready to run queries.

Run a Basic Text Search

A basic text search finds documents where a specific field contains your query term. The text operator handles this. You set query to the term you want to find and path to the field you want to search.

Create a file named basic_search.py:

from pymongo import MongoClient

# Replace the placeholder values with your Atlas credentials and cluster hostname:
client = MongoClient(
    "mongodb+srv://<USERNAME>:<PASSWORD>@<HOST>/",
    appname="devrel-tutorial-python-search"
)
db = client["bookstore"]
books = db["books"]

# Search the description field for the term "space":
results = books.aggregate([
    {
        "$search": {
            "text": {
                "query": "space",
                "path": "description"
            }
        }
    },
    {
        # Return title, author, and description — suppress _id:
        "$project": {
            "_id": 0,
            "title": 1,
            "author": 1,
            "description": 1
        }
    }
])

for doc in results:
    print(doc)

Run the script:

python basic_search.py

You'll get the following output:

{
  "title": "Project Hail Mary",
  "author": "Andy Weir",
  "description": "An astronaut wakes up alone in space with no memory and must figure out how to save Earth from an extinction-level threat."
}

Only "Project Hail Mary" matches because "space" appears there as a standalone word. "The Hitchhiker's Guide to the Galaxy" has the word "hyperspace" in its description, but the Standard analyzer splits text on whitespace and punctuation only. “hyperspace” is tokenized as a single word, not broken into “hyper” and “space”. For “space” to match “hyperspace”, you'd need a different approach, like using the wildcard operator or a custom analyzer with edge n-grams.

Search Across Multiple Fields

You can pass a list of field names to path. MongoDB searches all listed fields and returns any document where your query term appears in at least one of them.

Create a file named multi_field_search.py:

from pymongo import MongoClient

# Replace the placeholder values with your Atlas credentials and cluster hostname:
client = MongoClient(
    "mongodb+srv://<USERNAME>:<PASSWORD>@<HOST>/",
    appname="devrel-tutorial-python-search"
)
db = client["bookstore"]
books = db["books"]

# Search title, description, and genre for the term "fiction":
results = books.aggregate([
    {
        "$search": {
            "text": {
                "query": "fiction",
                "path": ["title", "description", "genre"]
            }
        }
    },
    {
        # Return title, genre, and description — suppress _id:
        "$project": {
            "_id": 0,
            "title": 1,
            "genre": 1,
            "description": 1
        }
    }
])

for doc in results:
    print(doc)

Run the script:

python multi_field_search.py

You'll get the following output:

{
  "title": "The Midnight Library",
  "genre": "fiction",
  "description": "A woman discovers a library between life and death that contains books representing all the lives she could have lived."
},
{
  "title": "The Hitchhiker's Guide to the Galaxy",
  "genre": "science fiction",
  "description": "An ordinary man is swept across the universe after Earth is demolished to make way for a hyperspace expressway."
},
{
  "title": "Project Hail Mary",
  "genre": "science fiction",
  "description": "An astronaut wakes up alone in space with no memory and must figure out how to save Earth from an extinction-level threat."
}

All three books match because "fiction" appears as a token in each of their genre fields. The Standard analyzer tokenizes "science fiction" into two separate tokens: "science" and "fiction". So the two science fiction books match on the token "fiction", and "The Midnight Library" matches because its genre is exactly "fiction".

Search with Fuzzy Matching

A fuzzy search finds documents that contain terms similar to your query. This is useful when users make typos or spelling mistakes.

Create a file named fuzzy_search.py:

from pymongo import MongoClient

# Replace the placeholder values with your Atlas credentials and cluster hostname:
client = MongoClient(
    "mongodb+srv://<USERNAME>:<PASSWORD>@<HOST>/",
    appname="devrel-tutorial-python-search"
)
db = client["bookstore"]
books = db["books"]

# Search for terms similar to "librery" — allow up to one character edit:
results = books.aggregate([
    {
        "$search": {
            "text": {
                "query": "librery",
                "path": "description",
                "fuzzy": {
                    # maxEdits controls how many character changes MongoDB tolerates:
                    "maxEdits": 1
                }
            }
        }
    },
    {
        # Return title and description — suppress _id:
        "$project": {
            "_id": 0,
            "title": 1,
            "description": 1
        }
    }
])

for doc in results:
    print(doc)

Run the script:

python fuzzy_search.py

You'll get the following output:

{
  "title": "The Midnight Library",
  "description": "A woman discovers a library between life and death that contains books representing all the lives she could have lived."
}

The maxEdits value controls how many character changes MongoDB tolerates. A value of 1 means MongoDB matches terms that differ from your query by one insertion, deletion, or substitution. The misspelled "librery" is one character away from "library", so the match succeeds.

Search for a Term While Excluding Another Word

Sometimes you want to find documents that match one term but don't contain another. This is different from a basic text search because you need two conditions to work together:

a positive match
an exclusion

The compound operator handles this through its must and mustNot clauses.

must requires the document to match a condition to appear in the results. mustNot requires the document not to match a condition to appear in the results. A document only returns if it satisfies both at once.

Create a file named exclude_term_search.py:

from pymongo import MongoClient

# Replace the placeholder values with your Atlas credentials and cluster hostname:
client = MongoClient(
    "mongodb+srv://<USERNAME>:<PASSWORD>@<HOST>/",
    appname="devrel-tutorial-python-search"
)
db = client["bookstore"]
books = db["books"]

results = books.aggregate([
    {
        "$search": {
            "compound": {
                # Must: the description field must contain "woman":
                "must": [
                    {
                        "text": {
                            "query": "woman",
                            "path": "description"
                        }
                    }
                ],
                # MustNot: exclude any book whose description contains "death":
                "mustNot": [
                    {
                        "text": {
                            "query": "death",
                            "path": "description"
                        }
                    }
                ]
            }
        }
    },
    {
        # Return title and description — suppress _id:
        "$project": {
            "_id": 0,
            "title": 1,
            "description": 1
        }
    }
])

for doc in results:
    print(doc)

Run the script:

python exclude_term_search.py

You'll get the following output:

{
  "title": "Educated",
  "description": "A woman reflects on her isolated rural childhood and her journey to educate herself and earn a PhD from Cambridge."
}

Two books in the collection have "woman" in their description: "The Midnight Library" and "Educated". The query returns only "Educated" because "The Midnight Library" contains the word "death", which the mustNot clause blocks.

A mustNot clause doesn't contribute to the relevance score of any document. It acts purely as a filter that removes documents from the result set. A document either passes or fails the exclusion check, and that outcome doesn't change how MongoDB ranks the documents that do pass.

Combine Conditions with the Compound Operator

The compound operator lets you combine multiple search conditions in a single query. It supports four clauses:

must: the document must match for it to appear in the results. Maps to AND.
mustNot: the document must not match for it to appear in the results. Maps to AND NOT.
should: documents that match rank higher in the results. Maps to OR.
filter: the document must match, but matching doesn't affect the relevance score.

Create a file named compound_search.py:

from pymongo import MongoClient

client = MongoClient(
    "mongodb+srv://<USERNAME>:<PASSWORD>@<HOST>/",
    appname="devrel-tutorial-python-search"
)
db = client["bookstore"]
books = db["books"]

results = books.aggregate([
    {
        "$search": {
            "compound": {
                # Must: the genre field must contain the token "fiction"
                "must": [
                    {
                        "text": {
                            "query": "fiction",
                            "path": "genre"
                        }
                    }
                ],
                # MustNot: exclude any book whose description mentions "astronaut"
                "mustNot": [
                    {
                        "text": {
                            "query": "astronaut",
                            "path": "description"
                        }
                    }
                ],
                # Should: boost books whose description mentions "library"
                "should": [
                    {
                        "text": {
                            "query": "library",
                            "path": "description"
                        }
                    }
                ],
                # Filter: only include books published in 1979 or later (no score impact)
                "filter": [
                    {
                        "range": {
                            "path": "year",
                            "gte": 1979
                        }
                    }
                ]
            }
        }
    },
    {
        "$project": {
            "_id": 0,
            "title": 1,
            "genre": 1,
            "description": 1,
            "year": 1,
        }
    }
])

for doc in results:
    print(doc)

Run it:

python compound_search.py

You'll get the following output:

{
  "title": "The Midnight Library",
  "genre": "fiction",
  "description": "A woman discovers a library between life and death that contains books representing all the lives she could have lived.",
  "year": 2020
},
{
  "title": "The Hitchhiker's Guide to the Galaxy",
  "genre": "science fiction",
  "description": "An ordinary man is swept across the universe after Earth is demolished to make way for a hyperspace expressway.",
  "year": 1979
}

The query works in four steps. The must clause narrows the candidate set to all books whose genre contains the token "fiction": "The Midnight Library" (genre: "fiction"), "Project Hail Mary" (genre: "science fiction"), and "The Hitchhiker's Guide to the Galaxy" (genre: "science fiction"). The mustNot clause then removes "Project Hail Mary" because its description contains "astronaut". The filter clause confirms that both remaining books were published in 1979 or later without adding any score. The should clause boosts "The Midnight Library" because its description contains "library", which pushes it to the top of the results.

Include Relevance Scores in Your Results

Every document in a $search result gets a relevance score. A higher score means the document is a closer match to your query. Documents score higher when the query term appears often in that document, and lower when the same term appears across many documents in the collection.

You include scores in results by adding a $project stage with the {"$meta": "searchScore"} expression. To see ranking in action, you need a query that returns more than one document. Searching for "woman" matches two books and lets you compare how MongoDB scores them against each other.

Create a file named search_with_scores.py:

from pymongo import MongoClient

# Replace the placeholder values with your Atlas credentials and cluster hostname:
client = MongoClient(
    "mongodb+srv://<USERNAME>:<PASSWORD>@<HOST>/",
    appname="devrel-tutorial-python-search"
)
db = client["bookstore"]
books = db["books"]

# Search the description field for the term "woman":
results = books.aggregate([
    {
        "$search": {
            "text": {
                "query": "woman",
                "path": "description"
            }
        }
    },
    {
        # Include the relevance score alongside each document:
        "$project": {
            "_id": 0,
            "title": 1,
            "description": 1,
            "score": {"$meta": "searchScore"}
        }
    },
    {
        # Return results from highest to lowest score:
        "$sort": {"score": -1}
    }
])

for doc in results:
    print(doc)

Run it:

python search_with_scores.py

You'll get the following output:

{
  "title": "The Midnight Library",
  "description": "A woman discovers a library between life and death that contains books representing all the lives she could have lived.",
  "score": 0.39296838641166687
}
{
  "title": "Educated",
  "description": "A woman reflects on her isolated rural childhood and her journey to educate herself and earn a PhD from Cambridge.",
  "score": 0.39296838641166687
}

Both books contain the word "woman" in their descriptions, so both return. The scores are equal because "woman" appears exactly once in each description and both descriptions are similar in length. This is the expected behavior when term frequency and document length are similar across matching documents.

The exact score values you see may differ from another cluster's output. Relevance scores depend on term frequency and document count at index time, which can vary between deployments. What matters is the relative ranking across results, not the absolute numbers.

The $sort stage makes the ranking explicit. Without it, MongoDB Search still returns results from highest to lowest score by default, but the score field lets you confirm and reason about the order.

Key Takeaways

MongoDB Search is embedded in your cluster. You don't need a separate search system.
You must create a search index before running any $search query. If the index doesn't exist, MongoDB returns an empty result set without an error.
The $search stage must come first in your aggregation pipeline. You chain other stages, like $project, $sort, and $limit, after it.
The Standard analyzer tokenizes multi-word strings into separate tokens. "science fiction" becomes two tokens: "science" and "fiction". This affects what matches your queries' return.
Compound words like "hyperspace" stay as a single token. A search for "space" doesn't match "hyperspace".
The compound operator lets you combine must, mustNot, should, and filter clauses in a single query. A mustNot clause acts as a pure filter and doesn't affect relevance scores.
Every $search result gets a relevance score. Documents score higher when the query term appears often in that document and lower when the same term appears across many documents in the collection.

All the code samples from this tutorial are available in the GitHub repository.

How to Set Up Queryable Encryption in a Django + MongoDB Application

Abigail Afi Gbadago — Thu, 19 Mar 2026 18:08:58 +0000

Build a Secure Risk Assessment API with MongoDB Queryable Encryption in Django

In this tutorial, we will build a Risk Assessment API with encrypted fields using the Django MongoDB Backend. This tutorial targets anyone working with sensitive data that needs to be protected. In this case, we will focus on assessing risk, which is fundamental in sectors such as finance or wealth asset management.

The Problem

Encryption plays a crucial role in mitigating potential threats to investments and ensuring regulatory compliance and long-term profitability of organizations. The major problem arises when sensitive data, such as Personal Identifiable Information (PII), financial, and health records, is stored in plain text in databases, which are vulnerable to exposure in the case of a database breach.

Under the EU’s Digital Operational Resilience Act (DORA), which became effective in January 2025, financial institutions are explicitly mandated to implement robust Information and Communications Technology (ICT) risk management frameworks, which include strict requirements for encryption of data—both at rest, in transit, and in use for operating in European markets, with similar requirements emerging globally.

Why MongoDB Queryable Encryption

MongoDB's Queryable Encryption solves this by encrypting data on the client side before it reaches the server, meaning the database can process queries on encrypted fields without ever seeing the plaintext. With Django MongoDB Backend's Queryable Encryption support, Django developers can protect their data. The fields are encrypted on write, decrypted on read, and the MongoDB server never has access to the raw data, even if compromised.

Queryable Encryption is supported on MongoDB versions as early as MongoDB 7.0 with new query operators for range, prefix, suffix, and substring queries that come with each versioned release. MongoDB 8.0 adds support for range queries on encrypted fields. More advanced text-style queries are being introduced in preview releases prior to general availability and are expected to become fully supported in a future release. To see the full range of supported operations, check here.

Design the Data Model

When deciding the data model for the risk assessment, let’s take a minute to think about the fields in our model.

Things we will encrypt: names, titles, descriptions that contain PII, and sensitive text.
Things we won’t encrypt: Scores, dates, Foreign keys, and anything you need to perform a sort or index on.

The relationship is visualized in the ERD below:

Overview of the Process

When a user makes a request, for example, a write request (POST /api/categories/) with "name": "Financial"} in the request body, Django receives the plaintext "Financial", the EncryptedCharField uses Queryable Encryption to encrypt sensitive data before storing or retrieving it as binary.

Prerequisites

Platform used: MacOS Tahoe 26.3
Python 3.13+
MongoDB Atlas account (free tier works)
Download mongo_crypt_shared library
- You can choose from MongoDB Enterprise Downloads or choose the current release:
- Select your OS; macOS ARM64 / Linux / Windows here
- Extract and note the path to mongo_crypt_v1.dylib (macOS) or mongo_crypt_v1.so (Linux)
- Reference the path in crypt_shared_lib_path in settings.py

Next, create a requirements.txt file with the following content:

Django>=6.0
djangorestframework==3.16.1

# For encryption(depends on corresponding Django version, >=6.0)
django-mongodb-backend[encryption]

# For the API Documentation(Swagger)
drf-spectacular

# For environment variables
Django-environ

Then run the following command in your terminal:
pip install -r requirements.txt

Project Setup

Create your Django project and app:

mkdir querysafe && cd querysafe
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
django-admin startproject querysafe --template https://github.com/mongodb-labs/django-mongodb-project/archive/refs/heads/6.0.x.zip
python manage.py startapp riskapp

Add apps to settings.py

It is ideal to use [MONGODB_USERNAME], [PASSWORD] in an .env file

INSTALLED_APPS = [
    ...
    "rest_framework",
    "drf_spectacular",
    "riskapp",
]

Configure encrypted database

For the (encrypted) database, let's set up a few things first.

Download mongo_crypt_shared

Download from MongoDB Enterprise Downloads or choose the current release here
Extract and note the path to the library file
Reference the path in crypt_shared_lib_path in settings.py (usually looks like Crypt_shared_lib_path: /Users/path/mongo_crypt_shared_v1-macos-arm64-enterprise-8.2.5/lib/mongo_crypt_v1.dylib)
For more information, check here

Now add the encrypted database to your settings.py file under the default database:

# (default)
`
"default": {
    "ENGINE": "django_mongodb_backend",
    "HOST": "MONGO_URI", # "mongodb+srv://[username]:[password]@cluster0.hpljptt.mongodb.net/?appName=devrel-tutorial-python-qe-medium",
    "NAME": "querysafe",
}

# (encrypted)

"encrypted": {
        "ENGINE": "django_mongodb_backend",
        "HOST": "mongodb+srv://[username]:[password]@cluster0.hpljptt.mongodb.net/?appName=devrel-tutorial-python-qe-medium",
        "NAME": "encrypted_db",

        "OPTIONS": {
            "auto_encryption_opts": AutoEncryptionOpts(
                key_vault_namespace="encrypted_db.__keyVault",
                kms_providers={
                    "local": {
                        # Generated by os.urandom(96)
                        "key": (
                            b'-\xc3\x0c\xe3\x93\xc3\x8b\xc0\xf8\x12\xc5#b'
                            b'\x19\xf3\xbc\xccR\xc8\xedI\xda\\ \xfb\x9cB'
                            b'\x7f\xab5\xe7\xb5\xc9x\xb8\xd4d\xba\xdc\x9c'
                            b'\x9a\xdb9J]\xe6\xce\x104p\x079q.=\xeb\x9dK*'
                            b'\x97\xea\xf8\x1e\xc3\xd49K\x18\x81\xc3\x1a"'
                            b'\xdc\x00U\xc4u"X\xe7xy\xa5\xb2\x0e\xbc\xd6+-'
                            b'\x80\x03\xef\xc2\xc4\x9bU'
                        )
                    },
                },
                crypt_shared_lib_path="[PATH/TO/mongo_crypt_v1.dylib]", #for macOS
                crypt_shared_lib_required=True,
            )
        },
    },

}

Now, to visualize the encryption process for our risk data, we will first route operations to the encrypted database by creating a router, define our models, perform migrations, seed the db and then take a look at the encrypted fields in the db.

Create a router

# riskapp/routers.py

class EncryptedRouter:
    def allow_migrate(self, db, app_label, model_name=None, **hints):

        if app_label == "riskapp":
            return db == "encrypted"

        if db == "encrypted":
            return False
        return None

    def db_for_read(self, model, **hints):

        if model._meta.app_label == "riskapp":
            return "encrypted"
        return None

    db_for_write = db_for_read

Define models

from django.db import models
from django.conf import settings
from django_mongodb_backend.models import EmbeddedModel
from django_mongodb_backend.fields import (
    EncryptedCharField,
    EmbeddedModelField,
)

class RiskCategory(EmbeddedModel):
    name = EncryptedCharField(max_length=100)
    description = EncryptedCharField(max_length=500, blank=True)

    def __str__(self):
        return str(self.name)

class RiskOwner(EmbeddedModel):
    username = EncryptedCharField(max_length=150)
    email = EncryptedCharField(max_length=254, blank=True)
    role = EncryptedCharField(max_length=100, blank=True)

    def __str__(self):
        return f"{self.username} ({self.role})"


class Risk(EmbeddedModel):
    title = EncryptedCharField(max_length=200)
    description = EncryptedCharField(max_length=500, blank=True, default="")
    category = EmbeddedModelField(RiskCategory)

    def __str__(self):
        return str(self.title)


class RiskAssessment(models.Model):
    RISK_LEVEL_CHOICES = [
        ("Low", "Low"),
        ("Medium", "Medium"),
        ("High", "High"),
    ]

    risk = EmbeddedModelField(Risk)
    owner = EmbeddedModelField(RiskOwner, blank=True, null=True)
    likelihood = models.CharField(max_length=10, choices=RISK_LEVEL_CHOICES)
    impact = models.CharField(max_length=10, choices=RISK_LEVEL_CHOICES)
    score = models.PositiveSmallIntegerField()
    assessed_on = models.DateField(auto_now_add=True)

    class Meta:
        db_table = "risk_assessments"
        ordering = ["-assessed_on"]
        indexes = [
            models.Index(fields=["likelihood"]),
            models.Index(fields=["impact"]),
            models.Index(fields=["score"]),
        ]

    def __str__(self):
        return f"{self.risk.title} ({self.likelihood}/{self.impact})"

Run migrations

python manage.py makemigrations riskapp
python manage.py migrate --database encrypted

Create serializers for request and response

#serializers.py
from rest_framework import serializers

RISK_LEVEL_CHOICES = ["Low", "Medium", "High"]


class RiskCategorySerializer(serializers.Serializer):

    name = serializers.CharField(max_length=100)
    description = serializers.CharField(
        max_length=500, required=False, default=""
    )


class RiskSerializer(serializers.Serializer):

    title = serializers.CharField(max_length=200)
    description = serializers.CharField(
        max_length=500, required=False, default=""
    )
    category = RiskCategorySerializer()


class RiskOwnerSerializer(serializers.Serializer):

    username = serializers.CharField(max_length=150)
    email = serializers.CharField(max_length=254, required=False, default="")
    role = serializers.CharField(max_length=100, required=False, default="")


class AssessmentSerializer(serializers.Serializer):

    risk = RiskSerializer()
    owner = RiskOwnerSerializer(required=False)
    likelihood = serializers.ChoiceField(choices=RISK_LEVEL_CHOICES)
    impact = serializers.ChoiceField(choices=RISK_LEVEL_CHOICES)
    score = serializers.IntegerField(min_value=1, max_value=10)


class AssessmentResponseSerializer(serializers.Serializer):

    id = serializers.CharField(source='pk', read_only=True)
    risk = RiskSerializer()
    owner = RiskOwnerSerializer()
    likelihood = serializers.CharField()
    impact = serializers.CharField()
    score = serializers.IntegerField()
    assessed_on = serializers.DateField()

Create class-based views with APIView

We will use @extend_schema to provide detailed, accurate documentation for each endpoint.

#views.py

from rest_framework import status
from rest_framework.response import Response
from rest_framework.views import APIView
from drf_spectacular.utils import extend_schema

from riskapp.models import (
    RiskAssessment,
    Risk,
    RiskCategory,
    RiskOwner,
)
from riskapp.serializers import (
    AssessmentSerializer,
    AssessmentResponseSerializer,
)

DB = 'encrypted_db'


class AssessmentView(APIView):

    @extend_schema(
        responses=AssessmentResponseSerializer(many=True),
        summary="List all assessments",
        tags=["Assessments"],
    )
    def get(self, request):
        assessments = RiskAssessment.objects.using(DB).all()
        serializer = AssessmentResponseSerializer(assessments, many=True)
        return Response(serializer.data)

    @extend_schema(
        request=AssessmentSerializer,
        responses=AssessmentResponseSerializer,
        summary="Create an assessment",
        tags=["Assessments"],
    )
    def post(self, request):
        serializer = AssessmentSerializer(data=request.data)
        serializer.is_valid(raise_exception=True)
        data = serializer.validated_data

        risk_data = data['risk']
        category_data = risk_data['category']

        assessment = RiskAssessment(
            risk=Risk(
                title=risk_data['title'],
                description=risk_data.get('description', ''),
                category=RiskCategory(
                    name=category_data['name'],
                    description=category_data.get('description', ''),
                ),
            ),
            likelihood=data['likelihood'],
            impact=data['impact'],
            score=data['score'],
        )

        if data.get('owner'):
            owner_data = data['owner']
            assessment.owner = RiskOwner(
                username=owner_data['username'],
                email=owner_data.get('email', ''),
                role=owner_data.get('role', ''),
            )

        assessment.save(using=DB)

        return Response(
            AssessmentResponseSerializer(assessment).data,
            status=status.HTTP_201_CREATED,
        )

Add urls.py at the project and app level.

#querysafe/urls.py

from django.contrib import admin
from django.urls import path, include
from drf_spectacular.views import SpectacularAPIView, SpectacularSwaggerView

urlpatterns = [
    path('admin/', admin.site.urls),
    path('api/', include('riskapp.urls')),
    path('api/schema/', SpectacularAPIView.as_view(), name='schema'), #for swagger-ui
    path('api/docs/', SpectacularSwaggerView.as_view(url_name='schema'), name='swagger-ui'), #for swagger-ui
]

#riskapp/urls.py

from django.urls import path
from riskapp.views import AssessmentView

urlpatterns = [
    path('assessments/', AssessmentView.as_view(), name='assessments'),
]

Next, we will create a management command structure for custom management commands by:

Creating a folder called management in the riskapp and a folder called commands, which is where we will add a Python script to seed the data. This can be achieved either via the GUI or by running mkdir -p riskapp/management/commands.
Then creating init.py(empty) files in each directory to make them Python packages by touch riskapp/management/__init__.py and touch riskapp/management/commands/__init__.py or by creating them via GUI.

Seed data with the seed_data.py, which contains mock data below.

NB: Because encrypted fields can't be queried, the seed script uses create() with in-memory references and a --target flush flag to stay rerunnable.

#seed_data.py

from django.core.management.base import BaseCommand
from django.db import transaction

from riskapp.models import (
    RiskAssessment,
    Risk,
    RiskCategory,
    RiskOwner,
)
ASSESSMENTS = [
    {
        "risk_title": "NoSQL Injection",
        "risk_description": "Potential NoSQL injection via unsanitised operators",
        "category": "Security",
        "cat_desc": "Security and authentication data",
        "likelihood": "High",
        "impact": "High",
        "score": 9,
    },
    {
        "risk_title": "Data Exposure",
        "risk_description": "Sensitive data returned without field projection",
        "category": "Personal",
        "cat_desc": "Personal identifiable information (PII)",
        "likelihood": "Medium",
        "impact": "High",
        "score": 7,
    },
    {
        "risk_title": "Privilege Escalation",
        "risk_description": "Attempt to modify roles or permissions",
        "category": "Security",
        "cat_desc": "Security and authentication data",
        "likelihood": "Low",
        "impact": "High",
        "score": 6,
    },
    {
        "risk_title": "GDPR Violation",
        "risk_description": "Query accesses EU citizen data without consent check",
        "category": "Compliance",
        "cat_desc": "Regulatory and compliance data",
        "likelihood": "Medium",
        "impact": "Medium",
        "score": 5,
    },
    {
        "risk_title": "PHI Exposure",
        "risk_description": "Protected health information returned in query",
        "category": "Healthcare",
        "cat_desc": "Medical and health records (PHI)",
        "likelihood": "High",
        "impact": "High",
        "score": 9,
    },
    {
        "risk_title": "Credit Card Leak",
        "risk_description": "Payment card data queried without masking",
        "category": "Financial",
        "cat_desc": "Financial data and transactions",
        "likelihood": "Medium",
        "impact": "High",
        "score": 8,
    },
    {
        "risk_title": "Mass Data Export",
        "risk_description": "Query fetches excessive number of documents",
        "category": "Compliance",
        "cat_desc": "Regulatory and compliance data",
        "likelihood": "Low",
        "impact": "Medium",
        "score": 4,
    },
    {
        "risk_title": "Unindexed Scan",
        "risk_description": "Collection scan on sensitive collection",
        "category": "Financial",
        "cat_desc": "Financial data and transactions",
        "likelihood": "High",
        "impact": "Low",
        "score": 3,
    },
]

class Command(BaseCommand):
    """
    Seed the database with sample risk assessment data.
    Use --target flush to clear data before reseeding.
    """

    help = "Seed the database with sample risk assessment data"

    def add_arguments(self, parser):
        parser.add_argument(
            '--target',
            default='all',
            choices=['all', 'default', 'encrypted', 'flush'],
            help="'flush' to clear data, 'all' to seed both, or specify a database",
        )

    def handle(self, *args, **options):
        target = options['target']

        if target == 'flush':
            self._flush('default')
            self._flush('encrypted')
            return

        if target == 'all':
            databases = ['default', 'encrypted']
        else:
            databases = [target]

        try:
            for db in databases:
                self.db = db
                self.stdout.write(f"\nSeeding into: {self.db}")
                with transaction.atomic(using=self.db):
                    self._seed_assessments()
            self.stdout.write(self.style.SUCCESS("\nAll data seeded successfully!"))
        except Exception as e:
            self.stdout.write(self.style.ERROR(f"\nSeeding failed: {e}"))

    def _flush(self, db):
        RiskAssessment.objects.using(db).all().delete()
        self.stdout.write(self.style.WARNING(f"Flushed data from: {db}"))

    def _seed_assessments(self):
        self.stdout.write("\n--- Seeding Risk Assessments ---")

        owner = RiskOwner(
            username="assessor",
            email="assessor@querysafe.com",
            role="Analyst",
        )

        for item in ASSESSMENTS:
            assessment = RiskAssessment(
                risk=Risk(
                    title=item["risk_title"],
                    description=item["risk_description"],
                    category=RiskCategory(
                        name=item["category"],
                        description=item["cat_desc"],
                    ),
                ),
                owner=owner,
                likelihood=item["likelihood"],
                impact=item["impact"],
                score=item["score"],
            )
            assessment.save(using=self.db)
            self.stdout.write(
                f"  Created: {item['risk_title']} ({item['likelihood']}/{item['impact']})"
            )

Run the script file in the terminal
python manage.py seed_data

Swagger Documentation

Next, we add Swagger UI to create a presentation layer for our API.

Install drf-spectacular
pip install drf-spectacular

Next, add it to your INSTALLED_APPS and configure it in settings.py:

INSTALLED_APPS = [
    'querysafe.apps.MongoAdminConfig',
    'querysafe.apps.MongoAuthConfig',
    'querysafe.apps.MongoContentTypesConfig',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django_mongodb_backend',
    'drf_spectacular',
    'riskapp',
]

REST_FRAMEWORK = {
    "DEFAULT_SCHEMA_CLASS": "drf_spectacular.openapi.AutoSchema",
}

SPECTACULAR_SETTINGS = {
    "TITLE": "QuerySafe API",
    "DESCRIPTION": "API for Risk Assessment using Queryable Encryption",
    "VERSION": "1.0.0",
    "SERVE_INCLUDE_SCHEMA": False,
}

Next, run the server to access the Swagger UI at http://localhost:8000/api/docs/

Results

Encrypted Data in MongoDB Atlas

The following screenshots show how sensitive data appears as encrypted binary data when viewed directly in MongoDB Atlas, while the Swagger interface displays the decrypted plaintext for authorized users.

Swagger UI (Auto-Decrypted)

The Swagger UI at http://127.0.0.1:8000/api/docs/ automatically decrypts and displays the data in plaintext for API consumers.

Terminal Query

When querying the database directly from the terminal using find_one(), the data appears as encrypted binary, demonstrating that the database server never has access to the plaintext.

Limitations

With Django MongoDB Backend’s encrypted model fields, you cannot use get_or_create() or filter() — each encryption produces a different ciphertext, so WHERE lookups never match.
Performance cost of client-side encryption/decryption - A read and write to an encrypted field requires client-side encryption/decryption on raw values directly. After fetching the data, your application server must individually decrypt each field for every record. This means that listing 1000 records requires 1000 decryption operations, which adds CPU overhead and slows down response times.
Why you can't encrypt everything:
- Dates — auto_now_add, ordering, and date filtering won't work on encrypted fields, else you can’t auto-populate timestamps (auto_now_add), order results (order_by('-created_at')) or filter by date ranges (filter(created_at__gte=last_week)).
- Scores/numbers — you can't sort or aggregate encrypted values / The database can't perform ORDER BY, SUM(), AVG(), MIN(), MAX(), or range queries (filter(risk_score__gte=80)) on encrypted values because it just sees random ciphertext.
- Indexed fields — CharField with choices like likelihood and impact need to remain queryable, so they can’t be encrypted.

Key Takeaways

Only encrypt fields that hold sensitive data and that you never need to search, sort, or filter by at the database level.
In the event of a database breach, the binary data won't be of any use to attackers, which keeps the integrity of your data safe (MongoDB never sees your plaintext data).

If you liked this tutorial, kindly comment and give me an emoji.

Thank you.

References

How to Scale to a Billion Documents in Symfony - Part II

Rishabh Bisht — Fri, 27 Feb 2026 13:08:44 +0000

This article was written by Andreas Braun.

Building an Application

In the first part of this series, we designed a schema to store fuel prices in MongoDB and imported a large amount of data into the database. We also looked at how to efficiently aggregate this data using MongoDB's aggregation framework. With the database prepared, it is now time to build a Symfony application that allows us to interact with this data. In this part of the tutorial, we'll use the Doctrine MongoDB ODM to map our documents to PHP classes, and Symfony UX to build a modern frontend.

To recap, this is what we want our application to do:

Show a dashboard with some general information
Show a list of stations
Allow searching for a station by name, brand, or location
Show the details of a station, including:
- A map showing the station's location
- The latest prices for each fuel type
- A chart showing the price history for the last day

Setting Up the Application

First, we set up our Symfony application using the Symfony CLI. If you don't have it installed yet, you can find instructions on the Symfony website. Once the CLI is installed, create a new Symfony project:

symfony new --webapp fuel-price-app

This creates a new web application with the latest version of Symfony. Since we don't need the Doctrine ORM, we can remove a number of packages from composer.json:

composer remove doctrine/dbal doctrine/doctrine-bundle doctrine/doctrine-migrations-bundle doctrine/orm symfony/doctrine-messenger

Now, make sure that you have the MongoDB extension installed and enabled. If you don't have it installed yet, install it using pie:

pie install mongodb/mongodb-extension:^2.1

Now we're ready to install the Doctrine MongoDB ODM bundle. This bundle allows easy configuration of the Doctrine MongoDB ODM and provides a set of tools to work with MongoDB in Symfony applications. To install the bundle, run the following command in your project directory:

composer require doctrine/mongodb-odm-bundle

Using Symfony Flex, this command sets up everything you need to connect to MongoDB. You can check the default configuration in config/packages/doctrine_mongodb.yaml. To connect to your database, update the MONGODB_URL environment variable in your .env file.

Mapping Documents

The first document we'll map is the station data. Create a new Document\Station class and add fields for everything we need to store:

<?php

declare(strict_types=1);

namespace App\Document;

use App\Document\Partial\AbstractStation;
use App\Repository\StationRepository;
use Doctrine\Common\Collections\ArrayCollection;
use Doctrine\Common\Collections\Collection;
use Doctrine\ODM\MongoDB\Mapping\Annotations\Document;
use Doctrine\ODM\MongoDB\Mapping\Annotations\EmbedOne;
use Doctrine\ODM\MongoDB\Mapping\Annotations\Field;
use Doctrine\ODM\MongoDB\Mapping\Annotations\Id;
use Doctrine\ODM\MongoDB\Mapping\Annotations\Index;
use GeoJson\Geometry\Point;
use Symfony\Component\Uid\Uuid;
use Symfony\Component\Uid\UuidV4;

#[Document(repositoryClass: StationRepository::class)]
#[Index(keys: ['location' => '2dsphere'])]
class Station
{
    #[Id(type: 'uuid', strategy: 'none')]
    public readonly Uuid $id;

    #[Field]
    public string $name;

    #[Field]
    public string $brand;

    #[Field(type: 'point')]
    public Point $location;

    #[EmbedOne(targetDocument: Address::class)]
    public Address $address;

    public function __construct(string|UuidV4|null $id = null)
    {
        $this->id = $id instanceof UuidV4 ? $id : new UuidV4($id);
    }
}

Note that we're not yet mapping the latestPrice and latestPrices fields. We'll do that later when we have the DailyPrice document ready. While we're mostly relying on imported data, we still want fully functionable documents, so we make sure that we can create a new station both with and without an ID. We store the ID as a uuid type, which was recently added to the Doctrine MongoDB ODM. This allows us to use the Uuid class from Symfony, mapping to a MongoDB Binary UUID. The location is stored as a Point type, which I also created specifically for this project.

The Address class is a simple value object that we map as an embedded document. Since we're only handling German addresses, we can keep it simple and implement the Stringable interface for easy string conversion. Here's the complete Address class:

<?php

declare(strict_types=1);

namespace App\Document;

use Doctrine\ODM\MongoDB\Mapping\Annotations\EmbeddedDocument;
use Doctrine\ODM\MongoDB\Mapping\Annotations\Field;
use Doctrine\ODM\MongoDB\Mapping\Annotations\Index;
use Stringable;

#[EmbeddedDocument]
class Address implements Stringable
{
    public function __construct(
        #[Field]
        public string $street,
        #[Field]
        public string $houseNumber,
        #[Field]
        #[Index]
        public string $postCode,
        #[Field]
        public string $city,
    ) {
    }

    public function __toString(): string
    {
        return <<<EOT
$this->street $this->houseNumber
$this->postCode $this->city
EOT;
    }
}

The mapping also shows the indexes we'll want to create. For the stations themselves, we're creating a 2dsphere index on the location field to allow for geospatial queries. The address.postCode field is indexed to allow for quick lookups by postal code. We'll talk about more indexes later, but for now this will suffice.

Before we can continue with the DailyPrice document, we need to think about the station mapping. We want to store stations as a standalone document, but we also want to be able to embed station data into the DailyPrice document. Embedding certain station data allows us to query prices without having to join two collections. Since documents and embedded documents require different mapping information, we'll have to introduce an abstraction in the form of a mapped superclass:

use Doctrine\ODM\MongoDB\Mapping\Annotations\EmbedOne;
use Doctrine\ODM\MongoDB\Mapping\Annotations\Field;
use Doctrine\ODM\MongoDB\Mapping\Annotations\Index;
use Doctrine\ODM\MongoDB\Mapping\Annotations\MappedSuperclass;
use GeoJson\Geometry\Point;

#[MappedSuperclass]
#[Index(keys: ['location' => '2dsphere'])]
abstract class AbstractStation
{
    #[Field]
    public string $name;

    #[Field]
    public string $brand;

    #[Field(type: 'point')]
    public Point $location;

    #[EmbedOne(targetDocument: Address::class)]
    public Address $address;
}

We then create one station document that is used as a standalone document:

class Station extends AbstractStation
{
    #[Id(type: 'binaryUuid', strategy: 'none')]
    public readonly Uuid $id;

    public function __construct(string|UuidV4|null $id = null)
    {
        $this->id = $id instanceof UuidV4 ? $id : new UuidV4($id);
        $this->latestPrices = new ArrayCollection();
    }
}

The embedded station document is then mapped as follows:

#[EmbeddedDocument]
class EmbeddedStation extends AbstractStation
{
    #[ReferenceOne(name: '_id', storeAs: 'id', targetDocument: Station::class)]
    private Station $referencedStation;
}

In the embedded document, we include a reference to the Station collection, but we only store its identifier in the _id field. This is a common strategy to deal with denormalized data in MongoDB, as it allows us to resolve the reference using a $lookup operator or the ODM’s reference mechanisms when we need to access data that we haven’t duplicated into the embedded document. We'll be using the exact same strategy for the DailyPrice document, since it needs to be embedded into the Station document as well. So, let's map that next:

#[MappedSuperclass]
class AbstractDailyPrice
{
    #[Id]
    public readonly string $id;

    #[Field(type: Type::DATE_IMMUTABLE)]
    public readonly DateTimeImmutable $day;

    #[Field(enumType: Fuel::class)]
    public readonly Fuel $fuel;

    #[Field(nullable: true)]
    public readonly ?float $openingPrice;

    #[Field]
    public readonly float $closingPrice;

    #[EmbedOne(targetDocument: Price::class)]
    public readonly Price $lowestPrice;

    #[EmbedOne(targetDocument: Price::class)]
    public readonly Price $highestPrice;

    #[EmbedMany(targetDocument: Price::class)]
    public readonly Collection $prices;

    #[Field]
    public readonly float $weightedAveragePrice;
}

This is the base class for all DailyPrice documents, and it comes with a small embedded document:

#[EmbeddedDocument]
class Price
{
    #[Field(name: '_id', type: Type::OBJECTID)]
    public readonly string $id;

    #[Field(type: Type::DATE_IMMUTABLE)]
    public readonly DateTimeImmutable $date;

    #[Field()]
    public readonly float $price;

    #[Field()]
    public readonly ?float $previousPrice;
}

The DailyPrice document we embed in the Station document does not need to embed a station, so it can be empty:

#[EmbeddedDocument]
class EmbeddedDailyPrice extends AbstractDailyPrice
{
}

The actual DailyPrice document that we store in its own collection adds an embedded relationship to a station:

#[Index(keys: ['fuel' => 1, 'day' => -1, 'station._id' => 1], unique: true)]
#[Index(keys: ['station._id' => 1, 'day' => -1])]
class DailyPrice extends AbstractDailyPrice
{
    #[EmbedOne(targetDocument: EmbeddedStation::class)]
    public readonly EmbeddedStation $station;
}

That's the most important mappings completed. You can also see that we create two separate indexes: one for fuel, day, and station that we also require to be unique, and another for the station and day. These indexes will cover most of our use cases. When building indexes, I usually apply the ESR (Equality, Sort, Range) guideline. The first fields in an index should be those used to do equality checks (e.g. checking for a specific fuel type), followed by fields used for sorting, and lastly fields used for range queries. There may be reasons to slightly deviate from this guideline, but in most cases it works well. Compound indexes in MongoDB also come with prefix indexes, meaning that if you query for a subset of fields in an index, MongoDB can still use that index for the query. Our first index on fuel, day, and station._id can also be used to query for just fuel and day, or just for fuel. For queries that don't filter by fuel type, we can use the second index on station._id and day. Note that the order of the station ID and day are flipped here, as we may want to sort by date when querying for a station's prices, or might be looking for a range of days for a station. Speaking of sorting, you'll notice that the index order is reversed on the day field, and that data is sorted in descending order. We are more likely to be interested in newer price data than in older price data, so a reverse sort on the index makes our queries more efficient.

Showing Station Details

Now that we have our documents mapped, let’s show some data in our application. I’ll skip over the details as the controllers for the station list and details page are essentially CRUD controllers. If you are interested in those details, you can check out the full source code of the controller. In the templates, I created several Symfony UX components to have reusable cards showing important station information, like the name. One example would be the one for station details that includes a map of where to find this station. By using Symfony UX components, we can encapsulate the logic for generating the map and adding all the information inside a dedicated PHP class:

#[AsTwigComponent('Station:Map')]
class Map
{
    public AbstractStation $station;

    public function getMap(): UxMap
    {
        [$longitude, $latitude] = $this->station->location->getCoordinates();
        $location = new Point($latitude, $longitude);

        $map = new UxMap();
        $map
            ->center($location)
            ->zoom(13)
            ->addMarker(new Marker(
                position: $location,
                title: $this->station->name,
                infoWindow: new InfoWindow(
                    headerContent: $this->station->name,
                    content: nl2br((string) $this->station->address),
                ),
            ));

        return $map;
    }
}

We can then include a map for a station wherever we want by including our handy little component:

<twig:Station:Map station="{{ station }}" />

We’re now showing a list of stations and some details about that station, but we’re not displaying prices yet. So, let’s see how we can efficiently show prices for the station.

Showing Latest Prices

When we imported data, we stored the latest price for each fuel type in the Station document, as well as a list of the last 30 days. To remind ourselves, this is the latestPrice field in the Station document:

{
  "latestPrice": {
    "diesel": {
      "_id": {
        "$oid": "673f8d1f3de33b548fc21a9f"
      },
      "closingPrice": 1.599,
      "day": {
        "$date": "2024-11-19T00:00:00.000Z"
      },
      "fuel": "diesel",
      "highestPrice": {
        "_id": {
          "$oid": "673f29d86526447058420b3d"
        },
        "date": {
          "$date": "2024-11-19T11:30:38.000Z"
        },
        "price": 1.649
      },
      "lowestPrice": {
        "_id": {
          "$oid": "673f29df65264470584ab598"
        },
        "date": {
          "$date": "2024-11-19T21:07:55.000Z"
        },
        "price": 1.559
      },
      "openingPrice": 1.599,
      "prices": [],
      "weightedAveragePrice": 1.6
    },
    "e10": {},
    "e5": {}
  }
}

To map this data, we'll need to create a new DailyPriceReport class that represents a daily price report for all fuel types at a station:

#[EmbeddedDocument]
class LatestPrice
{
    #[EmbedOne(targetDocument: EmbeddedDailyPrice::class)]
    public readonly EmbeddedDailyPrice $diesel;

    #[EmbedOne(targetDocument: EmbeddedDailyPrice::class)]
    public readonly EmbeddedDailyPrice $e5;

    #[EmbedOne(targetDocument: EmbeddedDailyPrice::class)]
    public readonly EmbeddedDailyPrice $e10;
}

We can now embed this document in the Station class. We can build a similar embedded document with an EmbedMany relationship for the last 30 days of prices, which we'll call LatestPrices and map it here as well:

class Station extends AbstractStation
{
    #[EmbedOne(targetDocument: LatestPrice::class)]
    public ?LatestPrice $latestPrice = null;

    #[EmbedOne(targetDocument: LatestPrices::class)]
    public ?LatestPrices $latestPrices;
}

Once we have our documents mapped, we can create more UX components to display them. For example, this is the component to show the latest price data:

{# @var App\Document\EmbeddedDailyPrice #}
{% props dailyPrice = null %}

{% if dailyPrice is not null %}
<dt>{{ dailyPrice.fuel.displayValue }} <span class="text-muted">({{ dailyPrice.latestPrice.date|date }})</span></dt>
<dd>
    <twig:Price price="{{ dailyPrice.latestPrice.price }}" />
    (Average: <twig:Price price="{{ dailyPrice.weightedAveragePrice }}" />)
</dd>
{% endif %}

I want to point out one small detail: accessing the latest price in a DailyPrice object. While the latest price is also stored as the closingPrice, we don't record the time or any other detail for that. So, in our templates, we'd always have to call dailyPrice.prices.last() to access this. With PHP 8.4 and property accessors, we can also create a virtual property that returns the latest price:

public ?Price $latestPrice {
    get {
        return $this->prices->last() ?: null;
    }
}

This way, our template doesn't need to know where the latest price comes from, and if we later change our schema to store all price information in the closingPrice field, we can change the implementation of this property without affecting the template code. It's these small details that sometimes make code easier to understand and maintain.

Our application now serves a paginated list of stations, and we can click on a station to see it on a map and see its latest prices. If you remember, we also wanted to show a chart with the price history. In the next part of this tutorial, we'll do exactly that. We'll be using Symfony UX's Chart.js integration to show charts on the station page, and we'll also embed charts from Atlas Charts.

Visualizing Data with Charts

Welcome to the third part of this series. In the previous page, we ended up with a list of stations and a details page that shows the latest prices for each fuel type. In this part, we want to add some charts to better visualize the price data that we have. To do this, we can start by using Symfony UX's Chart.js integration. First, we install the package as well as an adapter for Chart.js to handle dates:

composer require symfony/ux-chartjs
./bin/console importmap:require chartjs-adapter-date-fns

Once it's installed, we can create a small component that creates and renders our chart. The template only contains a call to render_chart(), all the logic is contained in the component class:

#[AsTwigComponent('Chart:Station:DayPriceOverview')]
class DayPriceOverview
{
    public LatestPrice $latestPrice;

    public function __construct(protected readonly ChartBuilderInterface $chartBuilder)
    {
    }

    public function getChart(): Chart
    {
        $chart = $this->chartBuilder->createChart(Chart::TYPE_LINE);

        $chart->setData([
            'datasets' => $this->createDatasets(),
        ]);

        $chart->setOptions([
            'parsing' => false,
            'datasets' => [
                'line' => ['stepped' => 'before'],
            ],
            'scales' => [
                'x' => ['type' => 'time'],
            ],
        ]);

        return $chart;
    }

    private function createDatasets(): array
    {
        $datasets = [];

        foreach (Fuel::cases() as $fuel) {
            $fuelType = $fuel->value;
            if (! isset($this->latestPrice->$fuelType)) {
                continue;
            }

            $datasets[] = $this->createDataset($this->latestPrice->$fuelType, $fuel);
        }

        return $datasets;
    }

    private function createDataset(EmbeddedDailyPrice $dailyPrice, Fuel $fuel): array
    {
        return [
            'label' => $fuel->value,
            'data' => array_map(
                static fn (Price $price) => [
                    'x' => $price->date->getTimestamp() * 1000,
                    'y' => $price->price,
                ],
                $dailyPrice->prices->toArray(),
            ),
        ];
    }
}

The logic basically revolves around massaging the data so that Chart.js can display it without having to parse it. We use millisecond timestamps for the X axis, and prices for the Y axis, showing each fuel type as its own dataset. We can now render this chart in our template. We can do pretty much the same for the price history chart, where we want to show the lowest, average, and highest prices for each of the last 30 days. We already have all the necessary data in the latestPrices station field, so we can create an equivalent component for that data. With that, we're already showing a good chunk of information about the station's prices. And, since we have pre-aggregated this data, the view page is lightning fast and requires a single query to the station collection. If we had to join to a collection storing all prices, loading this data would take significantly longer.

This approach works great for the data that we have denormalized into the Station document, and we can also leverage it for other data that we can fetch from the database. Since we might have to aggregate the DailyPrice collection directly for other charts, we should also consider caching data for charts. This is especially important for charts that show historical data, as we don't want to run expensive aggregation queries every time a user visits a page. There is another alternative for this so we don't have to do this ourselves: MongoDB Atlas Charts. Atlas Charts allows us to create charts based on our MongoDB data, all without having to write any code. We can then embed single charts or an entire dashboard into our Symfony application using the Atlas Charts Embed API. Atlas Charts automatically caches chart data, so we don't have to worry too much about the performance of our charts. A full deep dive into Charts would be a bit too much for this blog post. Feel free to check out Atlas Charts yourself if you want to go further!

We're now in a good place: we have a working application that allows us to view stations, their latest prices, and some charts to visualize the data. The data we have was entirely imported from a CSV dump, and we haven't yet considered adding new data to the database. In the next part of this series, we'll look at how we can maintain consistency in this denormalized database and how MongoDB's query framework makes this easier.

Optimizing the MongoDB Java Driver: How minor optimizations led to macro gains

MongoDB Guests — Wed, 11 Feb 2026 20:21:08 +0000

This tutorial was written by Slav Babanin & Nasir Qureshi.

Donald Knuth, widely recognized as the ‘father of the analysis of algorithms,’ warned against premature optimization—spending effort on code that appears inefficient but is not on the critical path. He observed that programmers often focus on the wrong 97% of the codebase. Real performance gains come from identifying and optimizing the critical 3%. But, how can you identify the critical 3%? Well, that’s where the philosophy of ‘never guess, always measure’ comes in.

In this blog, we share how the Java developer experience team optimized the MongoDB Java Driver by strictly adhering to this principle. We discovered that performance issues were rarely where we thought they were. This post explains how we achieved throughput improvements between 20% to over 90% in specific workloads. We’ll cover specific techniques, including using SWAR (SIMD Within A Register) for null-terminator detection, caching BSON array indexes, and eliminating redundant invariant checks.

These are the lessons we learned turning micro-optimizations into macro-gains. Our findings might surprise you — they certainly surprised us — so we encourage you to read until the end.

Getting the metrics right

Development teams often assume they know where bottlenecks are, but intuition is rarely dependable. During this exercise, the MongoDB Java team discovered that performance problems were often not where the team expected them to be.

Donald Knuth emphasizes this concept in his paper on Premature Optimization:

Programmers waste enormous amounts of time thinking about the speed of noncritical parts of their programs, and these attempts to improve efficiency have a strong negative impact on debugging and maintenance. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.

To avoid ‘premature optimization’—that is, improving code that appears slow but isn't on the critical path—we follow a strict rule: never guess, always measure.

We applied the Pareto principle (also known as the 80/20 rule) to target the specific code paths responsible for the majority of execution time. For this analysis, we used async-profiler. Its low-overhead, sampling-based approach allowed us to capture actionable CPU and memory profiles with negligible performance impact.

How we measured performance

We standardized performance tests based on throughput (MB/s), simplifying comparisons across all scenarios. Our methodology focused on minimizing the influence of external variables and ensuring practical relevance.

Testing Methodology:

Tested representative workloads: Testing focused on representative driver operations (for example, small, large, and bulk document inserts) executed via the MongoClient API, not isolated method microbenchmarks.
Isolated the testing environment: We conducted performance tests across multiple isolated cloud machines to minimize external variability and prevent performance degradation from resource contention (i.e., 'noisy neighbors'). Each test was run multiple times on each machine, and the median throughput score was used as the final result for that machine.
Statistical verification: Next, we aggregated the median results and directly compared the optimized branch mean throughput with the stable mainline mean. We verified statistical significance through percentage improvement and z-score analysis.

From this exercise, we realized that what truly mattered was that the performance improvements appear at the MongoClient API level. Internal microbenchmarks may show significant gains, but since users interact solely through the API, any gains that do not manifest at the API level will not translate into noticeable improvements in application performance.

Refer to the Performance Benchmarking drivers specification for a more detailed description of these tests.

Below, we will explain the six techniques we used to optimize the MongoDB Java driver, while staying true to our guiding principle: ‘never guess, always measure’.

1. Caching BSON array indexes

In BSON, array indexes are not merely integers; they are encoded as null-terminated strings, or CStrings. For example, the index 0 becomes the UTF-8 byte sequence for '0' (U+0030) followed by the UTF-8 byte sequence for NULL (U+0000). Encoding an array involves converting each numeric index to a string, encoding it into UTF-8 bytes, and then appending a null terminator:

for (int i = 0; i < arraySize; i++) {
    encodeCString(Integer.toString(i));;
   }

Calling toString() and encoding the result for every index was clearly suboptimal, because it repeats the same conversion work for the same indexes over and over again: each call rebuilds the String representation of i and then turns that String into a byte[]. This involves unnecessary copying, even though the result remains the same.

Our first improvement was to precompute and cache these immutable byte arrays for reuse in tight loops.

private static final byte[][] PRE_ENCODED_INDICES = new byte[1000][];

for (int i = 0; i < 1000; i++) {
    PRE_ENCODED_INDICES[i] = (Integer.toString(i) + '\u0000').getBytes("UTF-8");
}

for (int i = 0; i < arraySize; i++) {
    if (i < PRE_ENCODED_INDICES.length) {
        buffer.put(PRE_ENCODED_INDICES[i]);
    } else {
        encodeCString(Integer.toString(i));
    }
}

This caching step was already effective. We also changed the cache layout from a jagged byte[] to a flat byte[] representation.

private static final byte[] PRE_ENCODED_INDICES;

The flat byte[] layout remains the preferred choice because it uses fewer heap objects and scales more efficiently as the cache grows due to spatial locality. Our benchmarks showed no significant throughput difference compared to jagged byte[][] structures in smaller caches; this parity stems from the HotSpot’s Thread-Local Allocation Buffer (TLAB) allocator, which places small rows close together in memory. Even with garbage collection (GC) settings that forced frequent promotion into the old generation, we often observed the same effect. Because that behaviour is JVM- and GC-specific rather than guaranteed, we use the flat array as the more robust solution. To quantify the impact of caching itself, we adapted the “Small Doc insertOne” workload from our performance specification for array-heavy documents. Instead of the original shape, each document now contained A arrays of B length (that is, 100×100, and 100×1000), so the total number of encoded array indexes per document was A × B. The larger the arrays we use per document, the more significant the difference is, as the array encoding fraction in the “insertOne” operation is larger for larger arrays.

Figure 1. The figure below shows the before-and-after results of performance tests on 100x100 and 100x1000 array documents. The larger arrays saw the greatest improvement in performance.

2. Java Virtual Machine (JVM) intrinsics

As Java developers, we write code with many abstractions, which makes the code easier to maintain and understand; however, these abstractions can also cause significant performance issues. What is easy for humans to read isn’t always what the machine prefers to execute, which is why having mechanical sympathy may be beneficial.

For example, in BSON, numbers must be encoded with little-endian order. In our original code, when encoding an int to BSON, we wrote each byte to a ByteBuffer separately, doing manual shifting and masking to produce little-endian byte order.

write(value >> 0);

write(value >> 8);

write(value >> 16);

write(value >> 24);

However, this approach wasn’t efficient. It required individual bounds checks and manual byte shuffling for every byte written, which showed up as a hotspot in profiles. We chose to adopt ByteBuffer’s methods—such as putInt, putLong, and putDouble. This method collapses four separate byte operations into a single call (putInt) that handles byte order automatically.

Under the hood, modern JITs (e.g., in HotSpot) can compile these methods using intrinsics—such as Unsafe.putInt and Integer.reverseBytes—often mapping them to efficient machine instructions. For more context, see Intrinsic functions.

The JVM can replace these helper methods with a very small sequence of machine instructions, sometimes even a single instruction. For example, on x86, the Integer.reverseBytes(int i) method may use the BSWAP instruction; on ARM, the JVM might use the REV instruction.

Bonus tip: Repeated invariant checks in the hot path are computationally expensive. In the original code of the BsonOutput serializer, every single-byte write also re-checks invariants, such as “is this BsonOutput still open?” If you’ve already validated invariants elsewhere, you can safely skip repeated invariant checks inside the loop.

After implementing our changes and testing them, we realized that a simple change affecting less than 0.03% of the codebase resulted in a nearly 39% improvement in throughput for large document bulk inserts.

Figure 2. The figure below shows throughput improvements for each insert command. ‘Large Doc Bulk Insert’ saw the most significant gain because processing larger payloads maximizes the impact of optimizing the hottest execution paths.

3. Check and check again

As mentioned earlier, size checks on ByteBuffer in the critical path are expensive. However, we also performed similar checks on invariants in many other places. When encoding BSON, we retrieved the current buffer from a list by index on every write:

ByteBuffer currentBuffer = bufferList.get(currentBufferIndex);

//other code
currentBuffer.put(value);

This get() call is fast, but performing it many times adds up—especially since each call includes range checks and method indirection (as the path is too deep; the JVM might not always inline it).

Aha moment: If the same buffer will be used for thousands of operations before switching, why should we keep re-checking?

By caching the buffer in a field and updating it only when switching buffers, we eliminated at least three redundant range checks. Here is how:

private ByteBuffer currentByteBuffer;// Only update when changing bufferscurrentByteBuffer.put(value);

This minor change led to a 16% increase in throughput for bulk inserts. This wasn’t the only area where redundant checks could be eliminated; when we tested removing similar invariant checks from other abstractions, we observed an additional 15% improvement in throughput for bulk inserts.

The lesson: Remove unnecessary checks from the hottest paths. Because these checks run so frequently, they quickly become bottlenecks that drag down performance.

4. BSON null terminator detection with SWAR

Every BSON document is structured as a list of triplets: a type byte, a field name, and a value. Crucially, each field name is a null-terminated string—CString—not a length-prefixed string. While this design saves four bytes per field, it introduces a performance trade-off: extracting a CString now requires a linear scan rather than a constant-time lookup.

Our original implementation processed the buffer byte-by-byte, searching for the terminating zero:

boolean checkNext = true;

while (checkNext) {
    if (!buffer.hasRemaining()) {
        throw new BsonSerializationException("Found a BSON string that is not null-terminated");
    }
    checkNext = buffer.get() != 0;
}

The primary issue with this approach is that it requires more comparisons for the same amount of work. For large documents, the process calls buffer.get() billions of times. Processing each byte individually requires a load, check, and conditional jump each time, which rapidly increases the total instruction count.

To improve performance, we used a classic optimization technique: SWAR (SIMD Within A Register Vectorization). Instead of checking each byte separately, SWAR lets us examine eight bytes simultaneously with a single 64-bit load and some bitwise operations. Here’s what that looks like:

long chunk = buffer.getLong(pos);
long mask = chunk - 0x0101010101010101L;
mask &= ~chunk;
mask &= 0x8080808080808080L;
if (mask != 0) {
    int offset = Long.numberOfTrailingZeros(mask) >>> 3;
    return (pos - prevPos) + offset + 1;
}

These 'magic numbers' aren’t arbitrary: 0x0101010101010101L repeats the byte 1, while 0x8080808080808080L repeats the byte 128. By subtracting 1 from each byte, ANDing with the inverted value, and applying the high-bit mask, you can instantly detect if a zero exists. Then, simply counting the trailing zeros allows you to calculate the precise byte offset. This method is highly effective with CPU pipelining.

Let’s take a simple example. Suppose we use an int (4 bytes) for simplicity. The value: 0x7a000aFF contains a zero byte. We will demonstrate how the SWAR technique detects it:

Step                    | Value (Hex)   | Value (Binary, per byte)

------------------------|---------------|-----------------------------

chunk                   | 0x7a000aFF    | 01111010 00000000 00001010 11111111

ones                    | 0x01010101    | 00000001 00000001 00000001 00000001

mask (high bit mask)    | 0x80808080    | 10000000 10000000 10000000 10000000

Subtraction:

chunk      = 01111010 00000000 00001010 11111111

- ones     = 00000001 00000001 00000001 00000001

------------------------------------------------

            01111000 11111111 00001001 11111110

                          ↑
                      underflow

                       (0-1=FF)

Bitwise AND with inverted chunk:

prevResult = 01111001 11111111 00001001 11111110

& ~chunk   = 10000101 11111111 11110101 00000000

------------------------------------------------

            00000001 11111111 00000001 00000000

Bitwise AND with mask (high bits):

prevResult = 00000001 11111111 00000001 00000000

& mask     = 10000000 10000000 10000000 10000000

------------------------------------------------

             00000000 10000000 00000000 00000000

The final result:

The result has a high bit set (10000000) in Byte 2, showing there’s a zero byte at that position.
After isolating one byte, we can use the Integer.numberOfTrailingZeros(mask) >>> 3 to get the offset of the zero byte. This method is often an intrinsic function, built into the JVM, producing efficient single instructions.

Because the loop body now consists of a small, predictable set of arithmetic instructions, it integrates seamlessly with modern CPU pipelines. The efficiency of SWAR stems from its reduced instruction count, the absence of per-byte branches, and one memory load for every eight bytes.

5. Avoiding redundant copies and allocations

While optimizing CString detection with SWAR, we also identified an opportunity to reduce allocations and copies on the string decoding path.

Earlier versions of the driver wrapped the underlying ByteBuffer in a read-only view to guard against accidental writes, but that choice forced every CString decode to perform two copies:

From the buffer into a temporary ‘byte[]’.
From that ‘byte[]’ into the internal ‘byte[]’ that backs the ‘String’.

By verifying that the buffer contents remain immutable during decoding, we were able to safely remove the restrictive read-only wrapper. This allows us to access the underlying array directly and decode the string without intermediate copies.

if (buffer.isBackedByArray()) {
    int position = buffer.position();
    int arrayOffset = buffer.arrayOffset();
    return new String(array, arrayOffset + position, bsonStringSize - 1, StandardCharsets.UTF_8);
}

For direct buffers (which are not backed by a Java heap array), we cannot hand a backing array to the String constructor. We still need to copy bytes from the buffer, but we can avoid allocating a new byte[] for every string.

To achieve this, the decoder maintains a reusable byte[] buffer. The first call allocates it (or grows it if a larger string is encountered), and subsequent calls reuse the same memory region. That has two benefits:

Fewer allocations, less GC pressure, and memory zeroing: We no longer create a fresh temporary byte[] for every CString, which reduces the amount of work the allocator and garbage collector must do per document.
Better cache behavior: The JVM repeatedly reads and writes the same small piece of memory, which tends to remain hot in the CPU cache. We examined CPU cache behavior on our “FindMany and empty cursor” workload using async-profiler’s cache-misses event. Async-profiler samples hardware performance counters exposed by the CPU’s Performance Monitoring Unit (PMU), the hardware block that tracks events such as cache misses, branch misses, and cycles. For readString(), cache-miss samples dropped by roughly 13–28% between the old and new implementation, as we touch fewer cache lines per CString. We still treat the PMU data as directional rather than definitive — counters and sampling semantics vary by CPU and kernel — so the primary signal remains the end-to-end throughput gains (MB/s) that users actually observe.

On our “FindMany and empty cursor” workload, eliminating the redundant intermediate copy in readString improved throughput by approximately 22.5%. Introducing the reusable buffer contributed a ~5% improvement in cases where the internal array is not available.

6. String Encoding, removing method indirection and redundant checks

While benchmarking our code, we observed that encoding Strings to UTF-8, the format used by BSON, consumed a significant amount of time. BSON documents contain many strings, including attribute names as CStrings and various string values of different lengths and Unicode code points. The process of encoding strings to UTF-8 was identified as a hot path, prompting us to investigate it and suggest potential improvements. Our initial implementation used custom UTF-8 encoding to avoid creating additional arrays with the standard JDK libraries.

But inside the loop, every character involved several inner checks:

Verifying ByteBuffer capacity
Branching for different Unicode ranges
Repeatedly calling abstractions and methods

If the buffer was full, we’d fetch another one from a buffer pool (we pool ByteBuffers to reduce garbage collection (GC) pressure:

for (int i = 0; i < len;) {
    int c = Character.codePointAt(str, i);
    if (checkForNullCharacters && c == 0x0) {
        throw new BsonSerializationException(...);
    }

    if (c < 0x80) {
        //check if ByteBuffer has capacity and write one byt
    } else if (c < 0x800) {
        //check if ByteBuffer has capacity and write two bytes
    } else if (c < 0x10000) {
       //check if ByteBuffer has capacity and write three bytes
       total += 3;
    } else 
      //check if ByteBuffer has capacity and write fourfoure bytes
  }
  i += Character.charCount(c);
}

In practice, modern JVMs can unroll tight, counted loops, reducing back branches and enhancing instruction pipeline efficiency under suitable conditions. However, when examining the assembly generated by the JIT for this method, we observed that loop unrolling did not occur in this instance. This underscores the importance of keeping the hot path as straight as possible, minimizing branches, checks, and method indirection, especially for large workloads.

Our first optimization was based on the hypothesis that most workloads mainly use ASCII characters. Using this assumption, we developed a new loop that was much more JIT-friendly.

for (; sp < str.length(); sp++, pos++) {
    char c = str.charAt(sp);
    if (checkNullTermination && c == 0) {
        //throw exception
    }

    if (c >= 0x80) {
        break;
    }
    dst[pos] = (byte) c;
}

Before entering the loop, we verified that String.length() < ByteBuffer’s capacity and got the underlying array from the ByteBuffer (our ByteBuffer is a wrapper over the JDK or Netty buffers)

By verifying this invariant upfront, we eliminated the need for repeated capacity checks or method indirection inside the loop. Instead, we worked directly with the internal array of a ByteBuffer.

We also added a safeguard: if the character to encode is greater than 0x80, we’ve encountered a non-ASCII character and must fall back to a slower, more general loop with additional branching.

With this setup, the JIT usually unrolls the loop body, replacing it with several consecutive copies. This modification decreases the number of back branches and improves pipeline performance efficiency. If we zoom in on the compiled assembly, we can see that C2 has unrolled the loop by a factor of 4. Instead of processing one character per iteration, the hot path processes four consecutive characters (sp, sp+1, sp+2, sp+3) and then increments sp by 4. For example, HotSpot C2 on AArch64 might produce the following assembly, with some bookkeeping removed and only 2 of the 4 unrolled copies for brevity:

loop body:

; ----- char 0: value[sp], dst[pos] -----

    ldrsb   w5,  [x12,#16]          ; load value[sp]

    and     w11, w5, #0xff          ; w11 = (unsigned) c0

    cbz     w11, null_trap          ; if (c0 == 0) -> slow path (null terminator check)

    cmp     w11, #0x80              ; if (c0 >= 0x80) -> slow path (non-ASCII)

    b.ge    non_ascii1_path

    strb    w5,  [x10,#16]          ; dst[pos] = (byte)c0

    ; ----- char 1: value[sp+1], dst[pos+1] -----

    ldrsb   w4,  [x12,#17]          ; load value[sp+1]

    and     w11, w4, #0xff          ; w11 = (unsigned) c1

    cbz     w11, null_trap          ; if (c1 == 0) -> slow path (null terminator check)

    cmp     w11, #0x80              ; if (c1 >= 0x80) -> slow path (non-ASCII)

    b.ge    non_ascii1_path

    strb    w4,  [x10,#17]          ; dst[pos+1] = (byte)c1

What we did:

Removed internal method indirection (like write() wrappers) that introduced extra bound checks.
When writing ASCII, we wrote directly to the underlying ByteBuffer array if the capacity allowed, skipping extra bounds and range checks.
For multi-byte code points, we avoided repeated calls to ensureOpen(), hasRemaining(), and related checks by caching position and limit values inside hot paths.

This optimization improved insert throughput across all related benchmarks. For example:

Bulk write insert throughput for UTF-8 multi-byte characters improved by nearly 31%.
Bulk write insert throughput for ASCII improved by nearly 33%.

You can see the particular test conditions in the Performance Benchmarking specification.

Lessons learned

Cache immutable data on the hot path: In our case, pre-encoding BSON index CStrings once into a compact flat byte[] removed repeated int to byte[] conversions and cut heap overhead from thousands of tiny byte[] objects.
The JVM can surprise you: Use intrinsics and hardware features whenever possible. After implementing our changes and testing, we found that a simple modification affecting less than 0.03% of the codebase increased throughput for large document bulk inserts by nearly 39%.
Thoroughly profile your code: Optimize where it matters. Small, smart changes in the hot path can yield more benefits than rewriting cold code. By caching the buffer in a field and updating it only when switching to a new buffer, we improved bulk insert throughput by 16%.
Even cheap checks can add up: Bounds checks and branches in the hot path can be surprisingly costly - multiply a few cycles by billions, and you’ll notice a real impact. Move checks outside inner loops where possible, and don’t repeat invariants that have already been verified.
Vectorization (SIMD): Rewriting critical code paths with vectorized methods (e.g., SWAR) can significantly increase throughput by enabling you to process multiple data elements simultaneously per instruction.
Removing method indirection and redundant checks: Optimizing low-level operations required removing write(...)/put(...) wrappers to eliminate an extra layer of method indirection and the repeated invariant checks. By writing directly to the underlying ByteBuffer array for ASCII and caching position values in hot paths for multi-byte code points, we bypassed repeated validation calls, such as ensureOpen() and hasRemaining(). This resulted in a 33% improvement in bulk write insert throughput for ASCII.

Figure 3. The figure below shows the final results for throughput improvements (measured in MB/s) after optimizing the driver, as explained above, based on this performance benchmarking specification. The ‘Large doc Collection BulkWrite insert’ saw the highest performance improvement +96.46%.

Check out our developer blog to learn how we are solving different engineering problems, or consider joining our engineering team.

Building a Semantic Search Engine with Hugging Face Transformers and MongoDB Atlas Vector Search

MongoDB Guests — Tue, 10 Feb 2026 15:30:24 +0000

This tutorial was written by Arek Borucki.

Search is one of the most critical components of applications that work with text data, yet traditional keyword-based search has a major limitation: it only works when users type the exact words stored in the database, usually in the same language.

Imagine an online fashion store with products described in English. A customer searches in French for "chaussures de course confortables pour marathon" (comfortable marathon running shoes) and then in Polish for "wygodne buty do biegania na maraton". Even though both queries describe the same concept, a keyword-based search will return no results because the wording and language are different from the English product descriptions.

Semantic search addresses this limitation by focusing on meaning rather than exact words or language. Instead of comparing text strings, it maps text to numerical representations that capture semantic intent, allowing it to recognize that all of these queries refer to the same concept.

In this blog post, you will build a semantic search engine using Transformers and a real dataset from Hugging Face, a multilingual language model, and MongoDB Atlas Vector Search.

Understand Semantic Search Mechanisms

Before moving to the implementation, it is important to understand how the individual components work together.

Semantic search relies on three core elements:

A language model that converts text into numerical representations called embeddings.
A vector database capable of storing and searching these representations.
An application layer that connects everything into a working system.

Explore the Hugging Face Platform

Training language models from scratch requires enormous computational resources and training data. A single model can cost millions of dollars in infrastructure. Hugging Face solves this problem by providing a repository of ready-to-use models and datasets.

Often described as the “GitHub for AI”, Hugging Face is an open platform and community for machine learning, focused on natural language processing and multimodal AI. It hosts thousands of pre-trained models and public datasets, along with developer tools such as the Transformers library and hosted inference APIs, enabling teams to build AI-powered applications without training models from scratch. Companies like Google, Meta, Microsoft, and MongoDB publish their models there, which anyone can download and use.

In this tutorial, you will download two resources from Hugging Face: a multilingual language model that enables cross-language semantic search and a fashion product dataset that will be used as the searchable data source.

Understand the Transformers Architecture

Hugging Face Transformers is an open-source Python library that provides a unified API for working with Transformer-based models. The library handles model loading, tokenization, and inference, while pre-trained model checkpoints are hosted on the Hugging Face Hub and automatically downloaded when used.

Transformers 5 introduces a more modular architecture with a strong focus on interoperability, making quantization a first-class feature for smaller models and faster inference.

This tutorial uses sentence-transformers, a library built on top of Hugging Face Transformers that specializes in generating sentence and text embeddings for semantic similarity tasks.

Discover Vector Search

Vector search is a database search technique that powers semantic search. MongoDB Atlas provides this capability through Atlas Vector Search, where instead of matching keywords, the database compares numerical vectors.

This approach works by storing embeddings generated by Hugging Face language models directly in the database. Using Hugging Face Transformers, text is converted into embeddings, which are numerical representations of semantic meaning.

These embeddings are stored in MongoDB Atlas Vector Search and indexed for efficient similarity search. Each Hugging Face model produces embeddings of a fixed size. For example, the paraphrase-multilingual-MiniLM-L12-v2 model generates vectors with 384 dimensions, and MongoDB Atlas Vector Search must be configured with the same dimensionality.

When a user submits a query, Hugging Face generates an embedding for the query text. MongoDB Atlas Vector Search compares this vector with stored embeddings and returns the most semantically similar documents using Approximate Nearest Neighbor algorithms.

In short, Hugging Face generates embeddings, and MongoDB Atlas Vector Search stores and searches them. Together, they enable fast, meaning-based search.

Configure the Development Environment

The development environment requires several components: a Python interpreter with machine learning libraries, a local MongoDB Atlas deployment with Vector Search enabled, and a web framework to expose the API. Each of these components can be run locally.

Create the Project Structure

The project structure separates source code from configuration to keep the application organized and secure. Configuration files such as .env contain sensitive information, including connection strings, and should never be committed to the repository. The src directory isolates the application logic, making the project easier to test, maintain, and containerize later. Create the project directories and configuration files as shown below:

mkdir semantic-search-demo
cd semantic-search-demo
mkdir src
touch requirements.txt .env

Install Dependencies

Python does not isolate dependencies between projects by default, which can lead to version conflicts. To avoid this, project dependencies and their exact versions are defined in a requirements.txt file.

This project uses:

Sentence-transformers to generate embeddings with Hugging Face models
Datasets library to download datasets from Hugging Face
Pymongo to communicate with MongoDB
FastAPI and Uvicorn to serve the API
Torch as the underlying machine learning framework.

Create the requirements.txt file using the following command:

cat <<EOF > requirements.txt
fastapi>=0.109.0
uvicorn>=0.27.0
pymongo>=4.6.1
sentence-transformers>=2.3.0
python-dotenv>=1.0.0
datasets>=2.16.0
Pillow>=10.0.0
torch
EOF

Now, create a Python virtual environment and install the project dependencies. A virtual environment isolates the project from the system Python and prevents version conflicts, which is standard practice in production environments.

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Run MongoDB Atlas with Vector Search

To use MongoDB Atlas Vector Search without deploying a cloud cluster, you can run a local Atlas deployment using Atlas CLI. This approach provides the same Vector Search and Atlas Search capabilities as the cloud version, but runs locally using Docker, making it ideal for development and learning. Atlas CLI manages the entire setup and ensures your local environment behaves the same way as Atlas in the cloud. This helps avoid configuration mismatches later. First, install Atlas CLI if you do not have it already:

brew install mongodb-atlas-cli

Create a local Atlas deployment using atlas deployments setup command:

atlas deployments setup

? What type of deployment would you like to work with? local
[Default Settings]
Deployment Name         local5783
MongoDB Major Version   8 (latest minor version)
? How do you want to set up your local Atlas deployment? default

Creating your cluster local9328
1/3: Starting your local environment...
2/3: Downloading the latest MongoDB image to your local environment...
3/3: Creating your deployment local9328...
Deployment created!
Connection string: "mongodb://localhost:51213/?directConnection=true"
? How would you like to connect to local9328? mongosh

When prompted, choose local and accept the default settings. Atlas CLI will download the required MongoDB image and create a local Atlas cluster. Once the deployment is ready, Atlas CLI displays a connection string similar to:

mongodb://localhost:51213/?directConnection=true

When prompted to connect using mongosh, selecting yes will automatically open a shell session connected to the local Atlas cluster, allowing you to verify that the database is running correctly.

AtlasLocalDev local9328 [direct: primary] test> show dbs
admin   224.00 KiB
config  224.00 KiB
local   392.00 KiB
AtlasLocalDev local9328 [direct: primary] test>

After creating the cluster, update your .env file with the connection string from Atlas CLI.

cat <<EOF > .env
MONGO_URI=mongodb://localhost:51213/?directConnection=true
DB_NAME=fashion_db
COLLECTION_NAME=products
# Multilingual embedding model (50+ languages, including French, Polish, and English)
MODEL_NAME=sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
EOF

Implement System Components

The system is composed of three main modules: a database module, a language model module, and an indexing module. Together, they form the core of the semantic search pipeline.

Create the Database Module

The database module is responsible for managing the MongoDB connection and configuring the vector search index. The vector index enables efficient similarity search by organizing embeddings in a structure optimized for nearest-neighbor retrieval in high-dimensional space. Create the database module:

cat <<'EOF' > src/database.py
import os
from pymongo import MongoClient
from pymongo.operations import SearchIndexModel
from dotenv import load_dotenv

load_dotenv()


class MongoManager:
    def __init__(self):
        self.client = MongoClient(os.getenv("MONGO_URI"))
        self.db = self.client[os.getenv("DB_NAME")]
        self.collection = self.db[os.getenv("COLLECTION_NAME")]

    def get_collection(self):
        return self.collection

    def create_vector_index(self):
        """Creates a Vector Search index"""
        search_index_model = SearchIndexModel(
            definition={
                "fields": [
                    {
                        "type": "vector",
                        "path": "embedding",
                        "numDimensions": 384,
                        "similarity": "cosine"
                    }
                ]
            },
            name="vector_index",
            type="vectorSearch"
        )
        try:
            self.collection.create_search_index(model=search_index_model)
            print("Vector index 'vector_index' has been initialized.")
        except Exception as e:
            print(f"Index status: {e}")
EOF

Create the Language Model Module

The AI module encapsulates the embedding generation logic. The paraphrase-multilingual-MiniLM-L12-v2 model was trained on text from more than 50 languages, allowing it to capture semantic similarity across languages without explicit translation.
On the first run, the sentence-transformers library downloads the model weights from the Hugging Face Hub and stores them in a local cache.

cat <<'EOF' > src/ai.py
import os
from sentence_transformers import SentenceTransformer
from dotenv import load_dotenv

load_dotenv()


class AIModel:
    def __init__(self):
        model_name = os.getenv("MODEL_NAME")
        print(f"Loading Transformer model: {model_name}...")
        self.model = SentenceTransformer(model_name)
        print("Model ready.")

    def generate_embedding(self, text: str):
        if not text:
            return []
        return self.model.encode(text).tolist()
EOF

The initial download requires an Internet connection and may take several minutes depending on network speed. Subsequent runs reuse the cached model and start almost instantly.

Create the Data Indexing Module

The indexing module downloads a dataset from Hugging Face, generates embeddings for each document, and stores the enriched data in MongoDB.

cat <<'EOF' > src/indexer.py
from datasets import load_dataset
from src.database import MongoManager
from src.ai import AIModel


def run_indexing():
    print("=" * 50)
    print("Starting indexing process...")
    print("=" * 50)

    print("\n[1/6] Initializing MongoDB connection...")
    db = MongoManager()
    print("✓ MongoDB connected")

    print("\n[2/6] Loading AI model...")
    ai = AIModel()
    print("✓ AI model loaded")

    print("\n[3/6] Getting MongoDB collection...")
    collection = db.get_collection()
    print("✓ Collection ready")

    print("\n[4/6] Downloading fashion dataset from Hugging Face...")
    dataset = load_dataset(
        "ashraq/fashion-product-images-small",
        split="train"
    )
    print(f"✓ Dataset downloaded: {len(dataset)} total products")

    sample_data = dataset.shuffle(seed=42).select(range(200))
    print(f"\n[5/6] Processing {len(sample_data)} products (randomly sampled)...")

    collection.delete_many({})
    print("✓ Cleared existing data")

    docs_to_save = []

    for idx, item in enumerate(sample_data):
        if idx % 20 == 0:
            print(f"  Processing item {idx + 1}/{len(sample_data)}...")

        product = {
            "name": item["productDisplayName"],
            "category": item["masterCategory"],
            "subCategory": item["subCategory"],
            "gender": item["gender"],
            "price": 99.99,
        }

        text_to_embed = (
            f"{item['productDisplayName']} "
            f"{item['usage']} "
            f"{item['baseColour']}"
        )
        product["embedding"] = ai.generate_embedding(text_to_embed)

        docs_to_save.append(product)

    if docs_to_save:
        print(f"\n  Saving {len(docs_to_save)} products to MongoDB...")
        collection.insert_many(docs_to_save)
        print(f"✓ Saved {len(docs_to_save)} products")

    print("\n[6/6] Creating vector search index...")
    db.create_vector_index()

    print("\n" + "=" * 50)
    print("✓ Indexing completed successfully!")
    print("=" * 50)


if __name__ == "__main__":
    run_indexing()
EOF

Create the Search API

The API layer exposes semantic search functionality over HTTP. The /search endpoint accepts a text query, converts it into an embedding using the language model, and performs vector search in MongoDB.

MongoDB executes a $vectorSearch aggregation stage, which finds documents whose embedding vectors are most similar to the query vector. The results are returned together with a similarity score.

cat <<'EOF' > src/main.py
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, field_validator
from src.database import MongoManager
from src.ai import AIModel

app = FastAPI(title="Fashion Semantic Search")
db = MongoManager()
ai = AIModel()
collection = db.get_collection()


class SearchRequest(BaseModel):
    query: str
    limit: int = 5

    @field_validator('query')
    @classmethod
    def query_must_not_be_empty(cls, v):
        if not v or not v.strip():
            raise ValueError('Query cannot be empty')
        return v.strip()

    @field_validator('limit')
    @classmethod
    def limit_must_be_positive(cls, v):
        if v < 1:
            raise ValueError('Limit must be at least 1')
        if v > 100:
            raise ValueError('Limit cannot exceed 100')
        return v


@app.post("/search")
def search_products(payload: SearchRequest):
    query_vector = ai.generate_embedding(payload.query)

    if not query_vector:
        raise HTTPException(status_code=400, detail="Failed to generate embedding for query")

    pipeline = [
        {
            "$vectorSearch": {
                "index": "vector_index",
                "path": "embedding",
                "queryVector": query_vector,
                "numCandidates": 100,
                "limit": payload.limit
            }
        },
        {
            "$project": {
                "_id": 0,
                "name": 1,
                "category": 1,
                "subCategory": 1,
                "score": {"$meta": "vectorSearchScore"}
            }
        }
    ]

    try:
        results = list(collection.aggregate(pipeline))
        return {"query": payload.query, "results": results}
    except Exception as e:
        raise HTTPException(status_code=500, detail=f"Search failed: {str(e)}")


@app.get("/")
def read_root():
    return {"status": "ok", "service": "Fashion Semantic Search API"}
EOF

Test Multilingual Search

The database contains products with descriptions in English. A key advantage of semantic search is the ability to handle queries written in a different language than the stored data. A multilingual model recognizes that a query in French or Polish can express the same intent as an English product description.

Run the Indexing Process

Before running the indexer, mark the src directory as a Python package:

touch src/__init__.py

At this point, the project structure should look like this:

project-root/
├── src/
│   ├── __init__.py
│   ├── ai.py
│   ├── database.py
│   ├── indexer.py
│   └── main.py
├── requirements.txt
├── .env
└── venv/

Start by indexing the data. This step downloads the dataset from Hugging Face, generates embeddings for each product, and stores the documents together with their vectors in MongoDB. Run the indexer:

python -m src.indexer

The terminal displays progress information for each step of the process, including database initialization, model loading, dataset download, document processing, and vector index creation:

[1/6] Initializing MongoDB connection...
✓ MongoDB connected

[2/6] Loading AI model...
Model ready.
✓ AI model loaded

[3/6] Getting MongoDB collection...
✓ Collection ready

[4/6] Downloading fashion dataset from Hugging Face...
✓ Dataset downloaded: 44072 total products

[5/6] Processing 200 products (randomly sampled)...
✓ Cleared existing data
  Processing item 1/200...
  Processing item 21/200...
  Processing item 41/200...
  Processing item 61/200...
  Processing item 81/200...
  Processing item 101/200...
  Processing item 121/200...
  Processing item 141/200...
  Processing item 161/200...
  Processing item 181/200...

  Saving 200 products to MongoDB...
✓ Saved 200 products

[6/6] Creating vector search index...
Vector index 'vector_index' has been initialized.

==================================================
✓ Indexing completed successfully!
==================================================

Run the API Server

Activate the virtual environment and start the FastAPI server:

source venv/bin/activate
python -m uvicorn src.main:app --reload

The API is now available at http://localhost:8000

Execute Test Queries

The first test checks cross-language semantic search using a French query. The database contains product descriptions written in English.

curl -X POST "http://localhost:8000/search" \
     -H "Content-Type: application/json" \
     -d '{"query": "chaussures de course confortables pour marathon"}'

Output:

{                                                                                                                                                                                                                 
    "query": "chaussures de course confortables pour marathon",                                                                                                                                                     
    "results": [                                                                                                                                                                                                    
      {                                                                                                                                                                                                             
        "name": "Nike Men Transform III White Sports Shoes",                                                                                                                                                        
        "category": "Footwear",                                                                                                                                                                                     
        "subCategory": "Shoes",                                                                                                                                                                                     
        "score": 0.7942747473716736                                                                                                                                                                                 
      },                                                                                                                                                                                                            
      {                                                                                                                                                                                                             
        "name": "Fila Men's Basic Low Red Shoe",                                                                                                                                                                    
        "category": "Footwear",                                                                                                                                                                                     
        "subCategory": "Shoes",                                                                                                                                                                                     
        "score": 0.7913200855255127                                                                                                                                                                                 
      },                                                                                                                                                                                                            
      {                                                                                                                                                                                                             
        "name": "ADIDAS Unisex Brown Casual Shoes",                                                                                                                                                                 
        "category": "Footwear",                                                                                                                                                                                     
        "subCategory": "Shoes",                                                                                                                                                                                     
        "score": 0.789846658706665                                                                                                                                                                                  
      },                                                                                                                                                                                                            
      {                                                                                                                                                                                                             
        "name": "Numero Uno Men's Red Canvas Shoe",                                                                                                                                                                 
        "category": "Footwear",                                                                                                                                                                                     
        "subCategory": "Shoes",                                                                                                                                                                                     
        "score": 0.7863043546676636                                                                                                                                                                                 
      },                                                                                                                                                                                                            
      {                                                                                                                                                                                                             
        "name": "Lotto Men Sprint Black Sports Shoes",                                                                                                                                                              
        "category": "Footwear",                                                                                                                                                                                     
        "subCategory": "Shoes",                                                                                                                                                                                     
        "score": 0.7819452285766602                                                                                                                                                                                 
      }                                                                                                                                                                                                             
    ]                                                                                                                                                                                                               
  }

The output shows that MongoDB Atlas Vector Search returns a ranked list of products based on vector similarity scores. Although the query is written in French and the product data is in English, the multilingual embedding model correctly identifies that the query refers to athletic footwear. The top results include Nike and Lotto sports shoes from well-known athletic brands, which are semantically close to "marathon running shoes".

The second test verifies cross-language understanding using a Polish query. "Wygodne buty do biegania na maraton" translates to "comfortable running shoes for marathon". This is the same concept as the French query above.

curl -X POST "http://localhost:8000/search" \                                                                                                                                                                     
       -H "Content-Type: application/json" \                                                                                                                                                                        
       -d '{"query": "wygodne buty do biegania na maraton"}'

Output:

{                                                                                                                                                                                                                 
    "query": "wygodne buty do biegania na maraton",                                                                                                                                                                 
    "results": [                                                                                                                                                                                                    
      {                                                                                                                                                                                                             
        "name": "Lotto Men Sprint Black Sports Shoes",                                                                                                                                                              
        "category": "Footwear",                                                                                                                                                                                     
        "subCategory": "Shoes",                                                                                                                                                                                     
        "score": 0.7927727699279785                                                                                                                                                                                 
      },                                                                                                                                                                                                            
      {                                                                                                                                                                                                             
        "name": "Nike Men Transform III White Sports Shoes",                                                                                                                                                        
        "category": "Footwear",                                                                                                                                                                                     
        "subCategory": "Shoes",                                                                                                                                                                                     
        "score": 0.787016749382019                                                                                                                                                                                  
      },                                                                                                                                                                                                            
      {                                                                                                                                                                                                             
        "name": "Fila Men's Basic Low Red Shoe",                                                                                                                                                                    
        "category": "Footwear",                                                                                                                                                                                     
        "subCategory": "Shoes",                                                                                                                                                                                     
        "score": 0.7850795984268188                                                                                                                                                                                 
      },                                                                                                                                                                                                            
      {                                                                                                                                                                                                             
        "name": "ADIDAS Unisex Brown Casual Shoes",                                                                                                                                                                 
        "category": "Footwear",                                                                                                                                                                                     
        "subCategory": "Shoes",                                                                                                                                                                                     
        "score": 0.7824510335922241                                                                                                                                                                                 
      },                                                                                                                                                                                                            
      {                                                                                                                                                                                                             
        "name": "Nike Men Downshifter White Sports Shoes",                                                                                                                                                          
        "category": "Footwear",                                                                                                                                                                                     
        "subCategory": "Shoes",                                                                                                                                                                                     
        "score": 0.7783344388008118                                                                                                                                                                                 
      }                                                                                                                                                                                                             
    ]                                                                                                                                                                                                               
  }

The Polish query returns the same category of products as the French query — athletic footwear from Nike, Lotto, Fila, and Adidas. This confirms that the multilingual model correctly maps semantically equivalent queries across languages to the same product space.

The third test explores a different product category. This Polish query translates to "elegant evening handbag" and tests whether the model can identify accessories rather than footwear.

curl -X POST "http://localhost:8000/search" \
     -H "Content-Type: application/json" \
     -d '{"query": "elegancka wieczorowa torebka"}'

Output:

{
  "query": "elegancka wieczorowa torebka",
  "results": [
    {
      "name": "SDL by Sweet Dreams Women Green Printed Night Suit S11-3124",
      "category": "Apparel",
      "subCategory": "Loungewear and Nightwear",
      "score": 0.7503366470336914
    },
    {
      "name": "Lucera Women Silver Pendant",
      "category": "Accessories",
      "subCategory": "Jewellery",
      "score": 0.7264891862869263
    },
    {
      "name": "Giorgio Armani Women Idole Perfume",
      "category": "Personal Care",
      "subCategory": "Fragrance",
      "score": 0.7015564441680908
    },
    {
      "name": "Levis Men Comfort Style Grey Innerwear Vest",
      "category": "Apparel",
      "subCategory": "Innerwear",
      "score": 0.6910820007324219
    },
    {
      "name": "Kiara Women Teal Handbag",
      "category": "Accessories",
      "subCategory": "Bags",
      "score": 0.6863116025924683
    }
  ]
}

The results show that the semantic search engine understands the Polish query, even though products are described in English. However, the handbag appears only at position 5. The word "wieczorowa" (evening) led the model to nightwear, and "elegancka" (elegant) to jewelry and perfume. These connections make sense linguistically, but they are not what the user was looking for. In a real application, adding category filters would help return more relevant results.

Summary

This semantic search system is built on Hugging Face language models, which transform text into dense vector embeddings that capture meaning across languages. Using a multilingual model from the Hugging Face Hub eliminates the need to train custom models and enables cross-language similarity out of the box.

These embeddings are stored and searched with MongoDB Atlas Vector Search, while the application layer connects user queries to relevant results. Together, Hugging Face and vector search enable multilingual, meaning-based retrieval that goes beyond traditional keyword matching.

As shown in the test results, semantic search correctly identifies broad categories across languages. The less precise results for the handbag query are primarily due to the limited sample size of 200 products used in this demo—with a full catalog, the system would have more relevant items to match. For production applications, consider indexing the complete dataset, combining vector search with metadata filters, implementing re-ranking, or using domain-specific models to improve precision.

Forem: MongoDB

Async PyMongo in FastAPI

1. Introduction

Why modern APIs need asynchronous I/O

FastAPI’s design philosophy around async

Database bottlenecks & why async DB drivers matter

Introducing the new Async API in PyMongo

2. What is PyMongo?

Overview of the official MongoDB Python driver

Traditional synchronous PyMongo behavior

Blocking vs non-blocking I/O explained simply

Limitations of synchronous DB operations in async frameworks

3. What is Async PyMongo?

Overview of PyMongo’s new async API (pymongo.async_)

Key advantages over synchronous API

Relationship to Motor (Motor’s role before async PyMongo existed)

How to migrate from motor to async-pymongo and sync-pymongo to async-pymongo

4. Key Features of Async PyMongo

True async interface with awaitable operations

Connection pooling behavior

Async cursors and non-blocking iteration

5. What is FastAPI?

FastAPI’s async-first architecture

Event loop and concurrency model

Why FastAPI pairs naturally with non-blocking database drivers

Surprising performance differences when mixing sync DB code into async routes

6. Why Use Async PyMongo with FastAPI?

Avoid event-loop blocking

Handle thousands of concurrent requests

Reduce latency under load

Improve throughput for high-traffic microservices

Better CPU utilization and scaling on limited infrastructure

7. Async CRUD Operations with Examples

Insert one / insert many (async versions)

Find one / find many with async cursors

Async pagination using cursors

Update and replace operations

Delete one/many operations

Async aggregation pipelines

Error handling (duplicate keys, timeouts, network issues)

8. Performance Comparison: Sync PyMongo vs Async PyMongo

Example endpoints and profiling results

Locust Benchmark setup

Understanding event-loop blocking

Graphs and metrics discussion (Locust latency plots)

Full working snippet for readers to run

9. Common Pitfalls & How to Avoid Them

Forgetting await on Database Calls

Creating a New Client for Every Request

Returning Non-JSON-Serializable Types

Exhausting Connection Pools Under Heavy Load

Mixing Synchronous and Asynchronous Code

10. Conclusion

Research work

Build a Movie Watchlist with Node.js, TypeScript, and MongoDB

The Prerequisites

Add a TypeScript Configuration File

Understanding the Source Tree of the Project

Define and Configure the Environment Variables for the Project

Connecting to MongoDB with a Singleton Database Class

Define the Watchlist Document Type for TypeScript

Searching for Movies within the Catalog with MongoDB Search

Building a Movie Watchlist with Ratings

Configuring Express Framework to Connect to MongoDB and Serve the Application

Running the API to Test the Endpoints

Conclusion

Build a Task Manager with Django MongoDB Backend

Create a MongoDB Atlas cluster & get connection string

Prerequisites

Project setup

Environment Variables

Task Model

Forms

Authentication

Login/Logout Views

Add urls.py at the project and app level

Building the Task Manager Board (Kanban Board)

Create views

Templates

Drag & Drop

Overview of PyMongo’s new async API (`pymongo.async_`)

Use a single, globally shared `MongoClient` (or an appropriately scoped one) instead of creating a new connection per request.