Forem: MongoDB Guests

Comparing Replication and Failover in PostgreSQL and MongoDB

MongoDB Guests — Thu, 21 May 2026 09:00:00 +0000

This article was written by Arek Borucki.

Every production database eventually faces the same question: what happens when the machine running it dies? An EC2 instance gets retired with three minutes’ warning. A data center loses connectivity to its peers. The database keeps the company’s data, and the company cannot wait for someone to drive to a colocation facility. So databases learned to replicate themselves. To keep more than one copy of the data, on more than one machine, ready to take over when something breaks.

PostgreSQL and MongoDB both solve this problem. They both replicate data. They both promote a new primary when the current one fails. But they took radically different paths to get there, and those paths reflect when each system was born and what its designers believed about how databases should work. PostgreSQL grew out of an academic project in the 1990s, in an era when high availability meant a tape backup and a phone call to a DBA. MongoDB was conceived in 2009, after a decade of internet companies discovering that machines fail constantly and recovery cannot involve humans. Their replication systems carry that history in their bones.

This article carefully walks through both systems. You will learn how PostgreSQL streams write-ahead logs to standby servers and how MongoDB tails its operation log on every secondary. You will see why PostgreSQL needs Patroni or repmgr to fail over automatically, while MongoDB has Raft consensus built into its replica set protocol. You will understand why a transaction in flight on a PostgreSQL primary simply dies during failover, while a prepared transaction on a MongoDB primary survives and resumes on the new primary. By the end, you will have a clear framework for choosing between them. Not based on marketing pages, but on which trade-offs actually fit your system.

Understanding the philosophical divide

The most important thing to know about PostgreSQL replication is that it does not include automatic failover. Read that sentence twice. PostgreSQL can stream every byte of its write-ahead log to a standby server in real time, keep that standby in lockstep with the primary, and let you read from it. But if the primary dies at three in the morning, PostgreSQL itself does not promote the standby. A human, or a piece of software written by humans, must notice the failure, decide which standby is the most up-to-date, and run the promotion command.

This is not an oversight. It is a design choice that goes back to PostgreSQL’s origins as the Berkeley POSTGRES project in the mid-1980s and its long incubation through the 1990s. Database research at the time treated replication as an engineering problem with many possible solutions and no single right answer. Should the primary wait for the standby to acknowledge writes, or should it commit and replicate asynchronously? Should the standby be a hot spare, ready to serve reads, or a cold backup? Should failover require human judgment, or should it be automatic? PostgreSQL refused to choose. It built the primitive, write-ahead logs, replication slots, hot standbys, and left the high-level policy to the operator.

MongoDB started from the opposite assumption. It was designed in 2009, in the wreckage of the early-2000s internet, where companies like Google, Amazon, and Facebook had spent a decade learning that machines fail constantly. A typical Google datacenter sees thousands of disk failures, hundreds of machine failures, and dozens of rack failures per year, as documented in Jeff Dean’s famous 2009 talk on cluster reliability. By 2009, the operational lesson was clear: if your database requires a human to recover from a single machine failure, your database is broken. MongoDB’s designers chose to bake high availability into the database itself. A MongoDB replica set is not a primary plus a few read replicas; it is a small cluster that elects its own leader, detects its own failures, and recovers from them without human intervention.

A useful analogy is the difference between a car that comes with a spare tire and a car that comes with run-flat tires. PostgreSQL hands you a spare tire and a jack: it tells you exactly how to replace the wheel, but you have to be the one who notices the flat and pulls over. MongoDB ships run-flat tires that detect a puncture and let the car keep driving. Both approaches keep you on the road. The first gives you more control over what happens when something goes wrong; the second works at three in the morning when nobody is paying attention.

This difference shapes everything else in this chapter. When PostgreSQL operators want automatic failover, they install Patroni, repmgr, or pg_auto_failover. External systems that wrap PostgreSQL and add the missing logic. These tools watch the primary, detect when it stops responding, run an election among the surviving standbys, and promote the winner. Some of them use etcd or Consul to coordinate, treating PostgreSQL itself as a passive replicated state machine that responds to instructions from the outside. The PostgreSQL process knows nothing about its peers; it just streams WAL to whoever asks. When a MongoDB operator wants automatic failover, they do nothing. It is already there.

Why has PostgreSQL not absorbed this functionality? The community has debated it for years, and the answer is partly philosophical and partly practical.

Philosophically, PostgreSQL favors small, composable primitives over integrated systems. Adding automatic failover to core PostgreSQL would mean adding consensus algorithms, leader election, and cluster membership management. Substantial complexity that the community has been reluctant to take on, when external tools already do the job. Practically, automatic failover requires answering questions that depend on the deployment: how many failures can the cluster tolerate, what counts as the primary being down, and how should clients discover the new primary. PostgreSQL’s answer has been to expose enough hooks that external tools can implement any policy, rather than commit to one.

The trade-off is real. With PostgreSQL, you get flexibility. You can build exactly the HA system your environment needs, or pick from several mature options. You also get complexity. Patroni alone is a substantial system, and getting it right requires understanding etcd, distributed consensus, network partitions, and split-brain scenarios. A misconfigured Patroni cluster can cheerfully promote two primaries during a network partition, and only careful fencing prevents the resulting data divergence. With MongoDB, you get simplicity. The rs.initiate() command and three nodes give you a working HA cluster, and you give up flexibility. The MongoDB replica set protocol is what it is; you cannot swap out its election algorithm or its heartbeat interval beyond the configuration knobs the database exposes.

Warning: the apparent simplicity of MongoDB replication can mislead operators into deploying minimum configurations that fail in surprising ways. A two-node replica set looks reasonable until you realize that any partition leaves both nodes unable to form a majority, so neither can serve writes. An odd number of voting members is not optional advice; it is a structural requirement of the protocol. PostgreSQL operators face the equivalent trap when they configure synchronous replication with a single standby and discover that losing the standby blocks all writes on the primary. Both systems have failure modes that follow from their consistency guarantees, and both reward operators who understand those guarantees before deploying.

Throughout this article, you will see the philosophical divide play out at every level. PostgreSQL has two replication mechanisms because operators wanted both physical and logical replication. MongoDB has one. PostgreSQL’s synchronous replication is configured through cryptic strings in synchronous_standby_names; MongoDB exposes write concerns at the query level. PostgreSQL clients connect to a hostname, and MongoDB drivers maintain a topology view of the cluster. Each system is internally consistent, but they answer different questions, and the answers reflect the era in which each was built.

Tracing how PostgreSQL replicates data

PostgreSQL replication starts with the write-ahead log. Every change to the database, every INSERT, every UPDATE, every CREATE INDEX, is written to the WAL before it touches the actual data files. The WAL exists for crash recovery: if the database process dies between writing a row and flushing it to disk, replaying the WAL on restart reconstructs the lost change. This is how every modern relational database achieves durability. What PostgreSQL noticed, sometime in the early 2000s, is that the same WAL stream that lets a single server recover from a crash can be sent to a second server to keep it in sync. Replication, in PostgreSQL, is fundamentally a side effect of crash recovery extended across the network.

PostgreSQL exposes two replication mechanisms built on top of the WAL: physical replication and logical replication. They operate concurrently, on the same WAL stream, and serve different use cases. Understanding the difference between them clarifies why PostgreSQL has accumulated the configuration knobs and external tools it has.

Physical replication: the WAL on the wire

Physical replication ships the WAL byte-for-byte from the primary to one or more standby servers. The standby is, in a literal sense, a replica of the primary at the binary level. Every page in the standby’s data files matches the corresponding page on the primary. This makes physical replication conceptually simple: the primary writes its WAL; a process called walsender streams those records to the standby; the standby’s walreceiver process receives them; and the recovery system on the standby applies them as if it were replaying a crash. The standby is always in recovery mode and only stops being a standby when it is promoted to primary.

Streaming replication is the modern way to ship WAL. The standby connects to the primary via the replication protocol, identifies itself, and asks for WAL records starting from a specific log sequence number (LSN). The primary streams records as they are generated. To prevent the primary from deleting WAL files that a standby still needs, PostgreSQL uses replication slots: small pieces of state on the primary that track how far each standby has consumed. As long as a slot exists, the primary will not delete WAL beyond the slot’s position. This is critical because a standby that falls too far behind can otherwise discover that the WAL it needs has been recycled, at which point it cannot catch up without a fresh base backup.

PostgreSQL also supports cascading replication, where a standby acts as a relay, streaming WAL it has received to other downstream standbys. This reduces network bandwidth on the primary and lets you build replication trees that span multiple data centers without each standby holding a connection back to the origin. Cascading replication carries a subtlety: the downstream standbys can never be more current than the relay standby, which means failover policies must account for the relay sometimes lagging behind the primary.

Synchronous replication adds durability guarantees on top of streaming. By default, the primary acknowledges a commit as soon as the WAL is flushed to its own disk; the standby may receive and apply the change later. Synchronous replication makes the primary wait for one or more standbys to acknowledge the write before reporting success to the client. This is configured through synchronous_standby_names, a string that lists which standbys count as synchronous and how many of them must acknowledge. PostgreSQL supports two methods. FIRST n (s1, s2, s3) requires the first n standbys in the list, in priority order, to confirm. ANY n (s1, s2, s3) requires any n of the listed standbys to confirm. A quorum-style policy that tolerates individual standby failures better. The synchronous_commit parameter further refines what acknowledgment means: remote_write requires the standby to write the WAL to the filesystem cache (surviving a PostgreSQL crash, but not necessarily an OS crash), while on requires the WAL to be flushed to disk, and remote_apply requires it to actually apply the change so subsequent reads on the standby see it.

A common mistake is to enable synchronous replication with a single standby and a strict synchronous_commit value, then panic when writes hang during a standby restart. The primary, by design, waits for the synchronous standby to confirm the commit before acknowledging success to the client. If the standby is gone, the primary blocks. Production deployments either configure ANY-style synchrony with multiple standbys or use synchronous_commit = remote_write to balance durability with availability. The right setting depends on whether you would rather lose a recent transaction during a primary crash or stop accepting writes during a standby outage.

Logical replication: the row-level alternative

Physical replication is powerful, but it has constraints that follow from its byte-level nature. The standby must run the same major version of PostgreSQL as the primary; you cannot replicate from version 14 to version 16 because the on-disk page formats might differ. The standby replicates the entire PostgreSQL cluster (that is, the entire database instance, including all databases in it); you cannot replicate just one database or table. The standby is read-only; you cannot write to it without breaking replication. These constraints are fine for high availability, where you want a hot spare identical to the primary, but they make physical replication unsuitable for use cases like cross-version upgrades, partial replication for analytics, or feeding data into systems that are not PostgreSQL.

Built-in logical replication, introduced in PostgreSQL 10 and refined heavily since, solves these problems by replicating at the row level instead of the page level. Where physical replication says, “here is the byte change at offset 8192 in file 16384,” logical replication says, “here is an INSERT into table users with these column values.” This higher-level representation lets logical replication cross PostgreSQL major versions, replicate selected tables, and lets subscribers process the changes through plugins or external systems.

The architecture is publish-subscribe. On the publisher node, you create a publication that defines which tables to replicate. On the subscriber node, you create a subscription that connects to the publisher and asks for one or more publications. The publisher records each change in its WAL as usual, and a separate logical decoding process extracts the high-level row changes from the WAL and sends them to the subscriber, which applies them to its own tables.

-- On the publisher
CREATE PUBLICATION mypub FOR TABLE users, departments;

-- On the subscriber
CREATE SUBSCRIPTION mysub
  CONNECTION 'dbname=foo host=publisher.local user=repuser'
  PUBLICATION mypub;

Logical replication uses several worker processes coordinated by the logical replication launcher (ApplyLauncherMain in the source). The Apply Worker, one per subscription, receives changes from the publisher and applies them. The Table Sync Worker handles the initial copy of each table’s existing data when the subscription is first created. The Sequence Sync Worker synchronizes sequence values. The Parallel Apply Worker can apply streaming transactions in parallel, which matters for large transactions that would otherwise serialize on the subscriber.

Behind the scenes, logical decoding does the heavy lifting. The reorder buffer assembles individual WAL records into complete transactions, spooling large transactions to disk if they exceed memory. The pgoutput plugin transforms the assembled changes into the logical replication protocol that subscribers understand. Snapshot-build (snapbuild.c) maintains historical catalog snapshots so that a row decoded from old WAL is interpreted using the schema that existed when the row was written, not the current schema. These mechanisms make logical decoding work across schema changes, but they also make it more expensive than physical replication.

Logical replication has an important limitation that confuses operators on first encounter: it does not replicate DDL. A CREATE TABLE on the publisher does not propagate to the subscriber. Schema changes must be applied separately on each node, and they must be applied carefully. If a column is added to the publisher and not yet to the subscriber, replication of new rows including that column will fail until the subscriber catches up. The same applies to TRUNCATE behavior, sequence values for newly created sequences, and other DDL-adjacent operations. Logical replication is therefore well-suited for replicating data between known schemas but poorly suited as a general-purpose schema mirror.

Replication origins, exposed through the pg_replication_origin catalog, track where each subscriber is in each publisher’s WAL stream. Each origin records both the remote LSN it has consumed and the local LSN where the change was applied. This metadata is what makes logical replication safe across crashes: if the subscriber dies mid-apply, it knows on restart exactly which transaction to resume. Replication origins also prevent infinite loops in bidirectional replication setups. A change that came in from origin A is tagged with that origin, so the subscriber knows not to send it back.

Setting up replication, in either form, requires explicit configuration. Physical replication needs wal_level = replica, max_wal_senders, and a base backup made with pg_basebackup. Logical replication needs wal_level = logical and one of the available output plugins (almost always pgoutput in modern deployments). PostgreSQL ships utilities like pg_basebackup and pg_createsubscriber to make these workflows easier, but the operator is still in charge of running them in the right order and pointing the standby or subscriber at the right primary. There is no equivalent to MongoDB’s rs.initiate() command that creates a working replicated cluster from scratch.

Tracing how MongoDB replicates data

MongoDB replication looks deceptively simple from the outside. You start three mongod processes, run rs.initiate() against one of them, add the other two with rs.add(), and wait a few seconds. The first node becomes primary, the other two become secondaries, and the cluster begins serving traffic. There is no separate replication daemon, no external coordination service, and no configuration file string that lists which nodes count as synchronous. Everything runs inside the same mongod process. Behind that simplicity sits a careful design that turns Raft consensus, an operation log, and pull-based replication into a system that genuinely needs no human intervention to recover from a primary failure.

The oplog: a log of operations, not bytes

The heart of MongoDB replication is the oplog, a capped collection named oplog.rs in the local database that exists on every replica set member. Where PostgreSQL’s WAL records byte-level changes to data pages, the oplog records logical operations: insert this document into this collection, update this document with these field changes, drop this index. Each oplog entry carries a timestamp called an OpTime that uniquely identifies the operation in the cluster’s history. Secondaries replicate by pulling entries from the primary’s oplog and applying them to their own copies of the data.

The oplog, being a capped collection, has consequences. It is a fixed-size circular buffer, sized at startup or via the replSetResizeOplog command. When it fills up, the oldest entries are overwritten. This means a secondary that falls too far behind can discover that the oplog entries it needs have been overwritten, at which point it cannot catch up incrementally and must perform a full initial sync. Sizing the oplog is therefore an operational decision: too small, and a brief secondary outage forces a multi-hour resync; too large, and you waste storage on history nobody will replay. The MongoDB documentation recommends sizing the oplog to cover at least 24 to 72 hours of write traffic in production, but the right number depends on your tolerance for resync events.

The mechanics of write replication are straightforward. When a write arrives at the primary, it goes through the same path as on a standalone mongod: the storage engine commits the change to disk, and an OpObserver inserts a corresponding document into the oplog. Secondaries pull oplog entries continuously through a long-running connection. The pull-based model differs from PostgreSQL’s push-based streaming and reflects MongoDB’s assumption that secondaries should be in charge of their own progress, rather than relying on the primary to keep them up to date. If a secondary lags or disconnects, the primary keeps writing without slowing down; the secondary catches up at its own pace when it reconnects.

Initial sync handles the case where a new node joins the replica set or an existing secondary has fallen too far behind to catch up incrementally. The new secondary connects to a sync source (usually a healthy secondary, but it can be the primary), reads every collection in every database, and tails the oplog from the time it started reading. Initial sync is expensive. It can take hours or days for a multi-terabyte cluster, but the protocol is robust to interruption: if the new secondary disconnects mid-sync, it can resume rather than start over.

The replication coordinator and the topology coordinator

Inside the mongod process, MongoDB replication runs through two components that are easier to understand once you know they were deliberately factored apart. The ReplicationCoordinatorImpl is the central orchestrator. It owns the public API, commands like replSetGetStatus, replSetReconfig, and the implicit machinery that runs when a write arrives. It manages the lifecycle of replication: starting it up at server boot, transitioning between member states (PRIMARY, SECONDARY, RECOVERING, ROLLBACK), and shutting down cleanly. It tracks states like _memberState, _myLastAppliedOpTime, and _myLastWrittenOpTime, which other components query to make decisions.

The TopologyCoordinator is the brain of the cluster. It owns the election algorithm, the heartbeat policy, the sync source selection logic, and the rules for deciding when to step down. Critically, it performs no I/O. It is a pure decision engine that takes inputs (heartbeat responses, configuration changes, election timeouts) and produces decisions (start an election, vote for this candidate, step down). The separation matters because it makes the consensus logic testable: you can drive the TopologyCoordinator with synthetic events and verify its decisions without spinning up a real network.

This split mirrors how Raft is described in academic literature, which is not an accident. MongoDB’s replica set protocol implements a variant of Raft, the consensus algorithm published by Diego Ongaro and John Ousterhout in 2014. Raft was designed specifically to be easier to understand than its predecessor, Paxos, and MongoDB adopted it after a few years of running its earlier protocol (sometimes called PV0), which revealed edge cases that Raft’s formalism made easier to reason about. The current protocol version, often referred to as PV1 in the source code, applies Raft-style rules: every member has a term, every election increments the term, and votes are tallied by the majority of the voting members.

Heartbeats: knowing who is alive

Replica set members maintain their view of the cluster through heartbeats. Each member sends periodic heartbeat messages to every other member, asking for status. The default heartbeat interval is two seconds, and members are considered unreachable if they fail to respond within ten seconds (the electionTimeoutMillis default, also configurable). Heartbeats carry more than liveliness information: they include each member’s OpTime, term, configuration version, and state. This means every node knows which member is primary, who is most up-to-date, and whether a configuration change is in flight, just by observing the heartbeats.

The TopologyCoordinator maintains a MemberData structure for each peer, recording ping times, last heartbeat response, and health status. When a heartbeat response arrives, _handleHeartbeatResponse updates this structure and, if necessary, triggers state changes. If the primary stops responding for long enough, secondaries notice through their MemberData and start the election process. The election timeout is critical here: too short and transient network blips trigger spurious failovers, too long and real failures take a long time to detect. The default of ten seconds reflects a compromise that works for most deployments but can be tuned for environments with stricter latency requirements.

Pull-based replication and sync source selection

Secondaries replicate by pulling oplog entries from a chosen sync source. The choice of sync source is not fixed: a secondary picks among available members based on factors like ping time, oplog freshness, replication chain depth, and configuration preferences (members can be tagged to influence selection). A secondary may sync from another secondary rather than from the primary, which is conceptually similar to PostgreSQL’s cascading replication and reduces load on the primary in large clusters.

The pull model has one important property: it makes the secondary responsible for its own progress. If the secondary slows down for any reason, like disk pressure or network congestion, the primary continues serving writes without blocking. Replication lag accumulates on the secondary side and is visible in metrics like replSetGetStatus output and the oplog window. PostgreSQL’s push model can be configured for similar non-blocking behavior, but its synchronous mode actively ties the primary’s commit latency to the standby’s acknowledgment, which the MongoDB write concern model handles differently (covered in the next section).

A subtlety worth noting: oplog entries are designed to be idempotent. The same operation can be applied twice without changing the final state. This matters for replication safety. If a secondary crashes mid-apply, it can replay the last few operations on restart without corrupting data. PostgreSQL’s WAL records have similar properties (they are designed to be replayed), but the bookkeeping is different: PostgreSQL tracks LSN positions, MongoDB tracks OpTime, and both protocols include enough information to resume after an unclean shutdown.

Comparing durability and synchronous semantics

A write is durable when, after the database tells the client the write succeeded, the database can recover that write through any single failure. The two systems mean roughly the same thing by durability, but they let you control it through completely different interfaces, and the differences reveal each system’s assumptions about who should make durability decisions.

PostgreSQL exposes durability through the synchronous_commit and synchronous_standby_names parameters. These are server-side settings: the database administrator chooses them once for the cluster (or per role, or per session via SET), and every transaction inherits the choice. The model is rigid by design. If your deployment policy says writes must reach two standbys, the database enforces that universally. The synchronous_commit values define the durability level: off means commits are not even written to disk before acknowledging (fast, lossy), local means flushed to the primary’s disk only, on means the synchronous standbys have written but not necessarily applied, remote_write means the standbys have received the data, and remote_apply means the standbys have actually applied the change. The synchronous_standby_names string then defines which standbys count and how many must confirm.

MongoDB takes the opposite approach. Durability is specified per write, through the writeConcern parameter. A client can issue an insert with { w: "majority", j: true } and the server will wait until a majority of voting members have written the change to disk before acknowledging. Another client, on the same database, can issue a write with { w: 1 } and get an acknowledgment as soon as the primary has applied it. This is more flexible than PostgreSQL’s server-wide setting, and it makes the durability decision a property of the operation rather than the deployment. A user-facing write that creates an account uses { w: "majority", j: true }; a background job that records click metrics might use { w: 1 } because losing a click is acceptable.

Both systems converge on the same durability primitive: a quorum of disks has flushed the WAL or oplog. PostgreSQL with synchronous_commit = on and synchronous_standby_names = 'ANY 1 (s1, s2, s3)' gives you a durability guarantee similar to MongoDB with w: 2 and j: true. The difference is who chose: the operator who set up the cluster, or the developer who wrote the query. There is no universally right answer. PostgreSQL’s model is easier to audit. You can read the configuration and know what the cluster guarantees, but harder to vary by operation. MongoDB’s model is more expressive but requires every developer to think about write concerns, and code that omits the write concern silently uses the default ({ w: "majority" }).

The read side has a similar pattern. PostgreSQL supports reads on hot standbys, with hot_standby_feedback to let the standby tell the primary not to vacuum rows the standby is still using. Reads on standbys may be stale due to some replication lag. MongoDB exposes read preferences (primary, primaryPreferred, secondary, secondaryPreferred, nearest) and read concerns (local, available, majority, snapshot, linearizable) on a per-query basis. Reading with readConcern: "majority" means the data has been committed to a majority of nodes, which is stronger than reading from a possibly lagging secondary in PostgreSQL terms. linearizable read concern is even stronger. It guarantees you see the latest write, at the cost of additional round-trips. Again, the philosophical pattern repeats: PostgreSQL gives you knobs at the deployment level, MongoDB exposes them at the query level.

Why the divergence? The answer is partly about how each system thinks about clients. PostgreSQL was designed when SQL was the universal interface, and applications rarely shipped with database-level configuration awareness; the DBA controlled durability because the application could not be expected to know. MongoDB was designed when applications were richer, drivers shipped with sophisticated configuration, and developers were comfortable specifying durability per operation. Neither approach is wrong. They reflect different ideas about where the contract between the database and application sits.

A practical warning applies to both systems. Operators sometimes configure synchronous replication or majority write concern and then deploy a cluster that cannot satisfy the requirement. Too few standbys, an arbiter that does not store data and therefore cannot acknowledge majority writes, and a synchronous standby that is offline. The result is the primary blocking on writes, which presents to applications as latency or timeouts. Test your durability configuration against the failure modes you actually expect: take down a standby in a staging cluster, see what happens, then decide whether the behavior is acceptable.

Detecting failure

Replication keeps copies of the data; failure detection decides when to use them. The contrast between the two systems is sharpest here because PostgreSQL has essentially no failure detection in core, and MongoDB has it built into every node.

In MongoDB, every member of a replica set sends periodic heartbeats to every other member. The default heartbeat interval is two seconds, configurable via the settings.heartbeatIntervalMillis configuration. Heartbeats are bidirectional: every member is both a sender and a receiver, and every member maintains its own view of the cluster’s health.

When a member fails to respond to heartbeats for longer than electionTimeoutMillis (default 10 seconds), peers consider it unreachable. If the unreachable member is the primary, eligible secondaries begin the election process to choose a new one.

The heartbeat data is rich. A heartbeat response carries the responder’s state (PRIMARY, SECONDARY, etc.), its term, its OpTime, its configuration version, and various health metrics. The TopologyCoordinator on each node uses this stream of information to maintain a consistent view of the cluster. Because every node sees the same heartbeat traffic, every node can independently decide what to do when something changes. There is no single coordinator that must itself stay alive for the cluster to make decisions. This is the essence of distributed consensus: each member runs the same algorithm on the same inputs and arrives at the same conclusion.

PostgreSQL has nothing equivalent in core. The walreceiver on a standby maintains a connection to the primary and detects when the connection breaks, but it does not act on that detection. It simply tries to reconnect. The primary does not actively monitor its standbys for liveness; it only knows about a standby that has connected and is consuming WAL. There is no protocol for a standby to vote, no protocol for primary handoff, and no protocol for cluster membership beyond the operator-managed standby connection. PostgreSQL is, in this sense, a single-node database that streams its WAL to bystanders.

External tools fill the gap. Patroni, the most popular HA solution for PostgreSQL, runs alongside each Postgres instance and uses a distributed key-value store (typically etcd or Consul) to maintain cluster state. Each Patroni agent registers its node, monitors the local Postgres process, and writes liveness updates to the key-value store. If the agent on the primary fails to update its leader key within a configured TTL, the key expires, and the Patroni agents on the standbys race to acquire it through a compare-and-swap operation. The winner promotes its local Postgres to primary; the losers reconfigure themselves to follow the new primary. The leader election logic lives entirely in Patroni and etcd. PostgreSQL itself only sees commands like pg_promote() or recovery configuration changes.

repmgr, a different tool, takes a different approach. It uses a daemon (repmgrd) on each node that monitors the primary and coordinates failover through repmgr metadata stored in the database itself. The trade-offs are subtle: Patroni has a robust external coordination layer (etcd) at the cost of running an additional service; repmgr has fewer moving parts but relies on the standby cluster reaching consensus through reads of the primary’s metadata, which is harder when the primary is the failed node. pg_auto_failover, a newer option from Microsoft, uses a separate monitor process that all nodes report to, again with its own trade-offs around the monitor itself becoming a single point of failure.

The result is that failure detection in PostgreSQL is a deployment decision, not a database decision. Two PostgreSQL deployments running the same software can have completely different failure detection semantics depending on which HA tool wraps them, what its timeout values are, and how its coordination layer is configured. A MongoDB deployment’s failure detection is essentially the same on every cluster of the same version, because it is part of the database. This makes MongoDB easier to reason about for an operator who is running it for the first time. It makes PostgreSQL more flexible for an operator who needs failure detection that integrates with broader infrastructure — a Kubernetes cluster, an existing etcd-based coordination plane, or a custom orchestrator.

A common failure mode in PostgreSQL HA setups is the network partition that splits the etcd cluster from the database cluster. Patroni cannot make decisions without a quorum in etcd, so the database cluster effectively freezes, not because the database itself failed, but because the coordination layer cannot reach a quorum. MongoDB has a similar pattern, but the coordination layer and the database layer are the same nodes, so a partition that affects coordination affects the database in identical ways. The integrated approach has fewer surprises in partition scenarios but cannot offload coordination to a separate, possibly more reliable, layer.

Electing a new leader

When the primary is gone, the cluster needs to pick a new one. The procedure has to satisfy two constraints. First, it must terminate. A cluster that endlessly debates who should be primary is unusable. Second, it must produce exactly one primary. A cluster with two primaries can accept conflicting writes that diverge silently, a condition called split-brain that is one of the worst failure modes in distributed systems. Both PostgreSQL ecosystems and MongoDB solve this through consensus algorithms, but they put the consensus in very different places.

How MongoDB elects a primary

MongoDB uses a Raft-derived election protocol, implemented in the TopologyCoordinator and exercised through the ReplicationCoordinatorImpl. When a secondary detects that the primary has been unreachable longer than the election timeout, it transitions through a series of states designed to avoid wasting votes on candidates that cannot win.

The first phase is the dry run. Before requesting actual votes, the candidate sends a vote request marked as dry_run = true to all voting members. Each member responds with whether it would have voted for this candidate based on the current state. Specifically, whether the candidate is at least as up-to-date as the responder, and whether the responder has not already voted in this term. A dry run that returns a majority of yes responses means the candidate has a reasonable chance of winning a real election. A dry run that fails (typically because some other node thinks it is more up-to-date) lets the candidate back off without burning a term number.

If the dry run succeeds, the candidate increments its term, votes for itself, and sends a real vote request, ReplSetRequestVotesArgs, in the wire protocol. Voting members evaluate three things: (1) is the candidate at least as up-to-date as I am, measured by OpTime; (2) has the candidate’s term advanced beyond my current term; and (3) have I already voted in this term. If all three conditions are met, the voter grants the vote. Once the candidate collects a majority of votes from voting members, it transitions to the PRIMARY state, and the cluster recognizes it through the next round of heartbeats.

A few details matter for understanding what can and cannot happen. Only voting members count toward the majority. A member configured with votes: 0 (often used for analytics secondaries) is not part of the quorum and cannot vote in elections. Arbiters are voting members who hold no data. They exist solely to break ties in even-numbered clusters and are themselves a source of operational confusion because they affect quorum calculations without contributing data redundancy. The terminology can mislead: a five-node replica set with three data-bearing voting members and two arbiters has the same fault tolerance as a three-node cluster, not a five-node cluster, because the arbiters cannot serve data.

The dry run phase is a refinement of the stock Raft, designed to mitigate a specific bad pattern. In plain Raft, a partitioned node that loses contact with the rest of the cluster will keep starting elections, incrementing its term each time. When the partition heals, this node’s high term number forces the surviving primary to step down even though that primary was happily serving traffic. The dry run prevents the partitioned node from incrementing its term unless it can plausibly win an election, which means the high-term churn never happens.

How PostgreSQL elects a primary

PostgreSQL itself does not elect anything. Promoting a standby to primary is a single operation: pg_promote() (since Postgres 12) or, equivalently, calling the trigger file mechanism. The promoted node stops being in recovery, accepts writes, and starts producing its own WAL. The election logic, deciding which standby to promote, ensuring no other standby is also being promoted, and redirecting clients, lives entirely outside PostgreSQL.

In a Patroni-managed cluster, the election is run by Patroni agents using the underlying distributed coordination service. The protocol typically goes like this. Each Patroni agent periodically renews a leader key in etcd with a TTL of, say, 30 seconds. As long as the primary’s Patroni agent is alive and the primary database is healthy (Patroni runs local health checks), the key is renewed, and the leader is stable. If the primary fails or its agent stops renewing, the leader key expires. The Patroni agents on the standbys race to acquire the now-empty key through a compare-and-swap operation. Whichever standby wins the race and is also healthy and sufficiently caught up, it promotes its local PostgreSQL and writes itself as the new leader. Critically, the compare-and-swap is atomic in etcd, so exactly one standby can win.

The fault tolerance of this scheme depends on the etcd cluster, not on PostgreSQL. An etcd cluster of three nodes tolerates one failure; an etcd cluster of five tolerates two. If the etcd cluster loses quorum, no leader change can happen, and the existing leader (if any) eventually loses its lock. Whether this is acceptable depends on whether you can keep etcd more reliable than PostgreSQL itself. In Kubernetes deployments, etcd is often a separate cluster used for many things; in dedicated Postgres deployments, you might run etcd on the same machines as Patroni, and a failure that takes out a database node also takes out the local etcd member.

There is a subtlety around catch-up. Patroni and similar tools must decide whether to promote a standby that has slightly stale WAL or wait for it to catch up first. Promoting a stale standby loses recent writes that did not make it across; waiting risks extending the outage. Patroni’s default behavior favors availability — it will promote a standby that is reasonably caught up rather than wait — but you can configure it to be stricter. MongoDB faces the same trade-off and resolves it by prioritizing and requiring that the elected primary be at least as up-to-date as a majority of voting members. The mechanisms differ, but the choice of availability versus consistency is the same.

Preventing two primaries

Both systems take split-brain seriously, and both rely on majority requirements to prevent it. In MongoDB, a candidate must receive votes from a majority of voting members. If the cluster is partitioned into a minority and a majority partition, only the majority partition can elect a new primary; the minority cannot reach quorum and remains read-only or unavailable. The old primary, if it survives in the minority partition, eventually steps down on its own (the protocol requires the primary to maintain heartbeat contact with a majority of voters; if that contact is lost for the heartbeat timeout, the primary steps down).

In a Patroni-managed PostgreSQL cluster, split-brain prevention has two layers. The etcd lock prevents two Patroni agents from both believing they are the leader. The optional fencing mechanism (Patroni’s rewind feature, plus tools like pg_rewind) handles the case where an old primary returns after a network partition heals. MongoDB has a similar mechanism called rollback: when a stepped-down primary rejoins the cluster and discovers its tail of oplog was not replicated to the new primary, the unreplicated operations are rolled back and saved to a separate file for human inspection. Both rollback mechanisms are last resorts and should not happen in correctly configured clusters.

Handling failover at the client layer

A failover that the database handles is only useful if applications notice and adapt. The two systems take different approaches to the client side, and the differences shape how applications must be written.

MongoDB drivers maintain a topology view of the cluster. When an application connects to a MongoDB replica set, the driver does not connect to a single hostname. It connects to a list of seed hosts and discovers the rest of the cluster through the isMaster command (renamed hello in newer protocol versions). The driver tracks which member is currently primary, and routes writes to it; reads go to whichever member matches the configured read preference. When the primary changes, the driver detects the change through subsequent isMaster responses and updates its routing. From the application’s perspective, this happens transparently. A write that arrived during the failover sees a brief error, but with retryable writes enabled, it is retried automatically against the new primary.

Retryable writes are an important safety net. The MongoDB protocol assigns each write a unique transaction number derived from the client’s session. When the driver retries a failed write, the server checks whether the operation has already been applied (using the transaction number to look up the operation in a special transactions collection). If yes, the server reconstructs the original response without re-executing the write; if no, it executes the write normally. This makes retries safe against double-application, even in edge cases where the original write succeeded but the response did not reach the client. PostgreSQL has nothing equivalent at the protocol level; retrying a failed INSERT in PostgreSQL is the application’s responsibility, and idempotency is a property of the SQL the application chooses to write.

PostgreSQL clients connect to a hostname and port. There is no protocol-level concept of a cluster. If the primary fails and a standby takes over, the client must somehow be redirected to the new primary, and PostgreSQL itself does nothing to make that happen. Production deployments solve this in one of three ways.

The classic approach is a virtual IP (VIP) that floats between the primary and standby. Clients connect to the VIP; whichever node currently holds the VIP is the active primary. VIPs can be managed through Pacemaker, keepalived, or a cloud load balancer. The pattern is simple but has subtle failure modes. The VIP must move atomically (no two nodes can hold it simultaneously), and clients connected to the VIP at the time of failover see their connections drop, which they must then reconnect.

A second approach uses a connection pooler with topology awareness. PgBouncer in transaction-pooling mode, deployed in front of the cluster, can be reconfigured to point at the new primary on failover. The application sees a stable PgBouncer endpoint while the actual primary changes behind the scenes. This decouples the application from the cluster topology and tends to work well with HA tools like Patroni, which can update the pooler’s configuration during promotion.

The third approach pushes topology awareness into the client driver. JDBC supports multi-host connection strings (jdbc:postgresql://host1,host2,host3/db?targetServerType=primary), and similar mechanisms exist in many language ecosystems. The driver tries each host until it finds one that is currently the primary, retrying as needed. This is closer in spirit to MongoDB’s driver-level topology, but it lacks the protocol support to detect role changes proactively. The driver only learns about the new topology when a query fails, not before.

The practical consequence is that writing failover-tolerant applications looks different on the two systems. On MongoDB, you connect to the cluster, enable retryable writes (on by default in modern drivers), and let the driver handle most of the rest. Your application code is mostly indistinguishable from code that talks to a single mongod. In PostgreSQL, choose your failover routing strategy (VIP, pooler, or multi-host), ensure your transactions are idempotent or wrap them in retry logic, and verify that your client library handles connection drops the way you expect. Neither approach is intrinsically better, but the work that has to happen in application code is meaningfully larger in PostgreSQL.

A worth-mentioning subtlety: read preference handling. Both systems let clients route reads to non-primary nodes, but they handle staleness differently. MongoDB drivers honor read preferences (primary, secondary, secondaryPreferred, nearest) and read concerns (local, majority, snapshot, linearizable) on a per-query basis, with the driver picking the right node. PostgreSQL clients connecting through a multi-host string can specify targetServerType=preferSecondary, but they cannot specify a read concern; staleness is whatever the standby happens to be at, with no protocol-level guarantee. PostgreSQL’s hot_standby_feedback parameter helps prevent vacuum-induced query cancellations on standbys, but the application has no way to ask for a strict freshness guarantee at query time.

Surveying the operational reality

Running these systems in production looks different. Both PostgreSQL and MongoDB demand operational competence, but the shape of the work is different enough that an operator skilled in one will need to relearn habits to be effective with the other.

A typical PostgreSQL HA stack includes more components than first-time operators expect. You have PostgreSQL itself. The primary and standbys, each with its own postgresql.conf, pg_hba.conf, and recovery configuration. You have Patroni or repmgr running alongside each Postgres instance, with its own configuration file and process supervision. You have etcd or Consul (often three or five nodes for fault tolerance), with its own deployment, security, and backup considerations. You have a connection-routing layer — PgBouncer, HAProxy, a cloud load balancer, or some combination. That needs to be told about topology changes. You have monitoring, e.g., pg_exporter for Prometheus, Patroni’s own metrics, and etcd’s metrics. Each producing its own view of cluster health. Configuration drift is a real risk: PostgreSQL configuration changes have to be coordinated with Patroni, which has its own opinions about which settings it owns versus which ones the operator can change directly.

A typical MongoDB HA deployment has fewer moving parts. You have mongod processes. Three or five of them. Each running a single configuration file. You have client drivers that handle topology discovery automatically. You have monitoring (Prometheus exporters, mongostat, the built-in $listSessions and serverStatus commands). That is, in many cases, it. There is no separate orchestration daemon, no external coordination service, no connection router. The simplicity is an asset for small operations teams and a liability for large ones, because it means you cannot mix-and-match components the way you can with PostgreSQL.

The flip side: PostgreSQL’s flexibility means each shop runs a slightly different stack, and skills are partially transferable. An operator who knows Patroni can move between teams and apply roughly the same playbook. An operator who has only used vanilla MongoDB has fewer options to deploy, but the operations are more uniform across deployments. The rs.status() output is the same on every replica set, the failover behavior is the same, the metrics names are the same. If you join a team running MongoDB and want to know how to fail over, the answer is in the documentation; if you join a team running PostgreSQL with Patroni, you need to read both the PostgreSQL and Patroni documentation, plus the team’s internal runbooks for how their etcd cluster is laid out.

Upgrades

Major version upgrades are a real pain point in PostgreSQL. Physical replication does not work across major versions, so the standard upgrade path involves either pg_upgrade (which upgrades a single instance in place, requiring downtime) or logical replication (set up a new version cluster, replicate data into it, switch over). Each approach has trade-offs. pg_upgrade is fast for small clusters and complex for large ones; logical replication is more robust for large clusters but requires schema compatibility and careful coordination. Tools like pg_easy_replicate automate parts of the logical replication path, but the operator still has to think about the cutover.

MongoDB upgrades are simpler in the common case. Replica sets support rolling upgrades: you upgrade one secondary, wait for it to rejoin, repeat for the other secondaries, then step down the primary and upgrade it last. Each version of MongoDB documents its compatibility window with the previous version, and feature compatibility flags let you upgrade the binaries before enabling new on-disk features. There are still pitfalls. Some upgrades require running specific scripts to update internal collections, and major version jumps may require intermediate stops, but the basic flow is straightforward and well-trodden.

Observability and debugging

Both systems expose detailed cluster state, but through different surfaces. PostgreSQL exposes pg_stat_replication on the primary (showing each connected standby and its lag) and pg_stat_wal_receiver on the standby (showing the receiver’s state). Patroni adds its own /cluster endpoint (or patronictl list output) that shows the topology, including which node is the leader. MongoDB exposes rs.status() and rs.printSecondaryReplicationInfo(), each producing a JSON document that captures the entire replica set’s state. The MongoDB output is denser and more uniform; the PostgreSQL output is more accurate to the actual database state, but requires you to know which catalogs to query.

Replication lag is monitored differently. In PostgreSQL, lag is measured in WAL bytes (replay_lsn vs current_lsn) or in time (now() - last_replay_timestamp). In MongoDB, lag is measured in OpTime difference, which is roughly equivalent to time since the OpTime is itself a Timestamp. Both systems can produce confusing numbers when a cluster is idle: a PostgreSQL standby that is fully caught up reports zero lag, but the timestamp-based metric drifts upward in idle periods because the primary has not generated new WAL. MongoDB’s OpTime gap behaves similarly. Operators learn to distinguish “real lag” from “cluster is idle” by checking whether traffic is flowing.

Choosing between them

After seven sections of comparison, the temptation is to declare one system the winner. The honest answer is that neither system wins on its own merits; they win for specific deployments. The right way to choose is to start with the workload and the team, then ask which system’s replication and failover model fits.

Where PostgreSQL’s model wins

PostgreSQL’s philosophy of primitives over policy plays to its strengths in environments that already have strong infrastructure conventions. If your platform team runs etcd for other purposes, integrating Patroni is straightforward and gives you HA that fits your existing operational mental model. If your application has heterogeneous replication needs, a synchronous standby in the same datacenter, an asynchronous standby for disaster recovery in another region, and a logical replica for analytics with a subset of tables. PostgreSQL’s independent replication mechanisms let you build the exact topology you need without compromise. MongoDB’s replica sets are uniform; you cannot mix synchronous and asynchronous nodes within a set the way PostgreSQL’s synchronous_standby_names allows.

Regulatory environments often favor PostgreSQL for similar reasons. Auditors like to see explicit configuration: this row in the postgresql.conf file says writes must be confirmed by two standbys, and you can read it. MongoDB’s per-query write concerns are auditable too, but the audit involves tracing through application code rather than reading server-side configuration. Both can satisfy compliance requirements, but PostgreSQL’s server-side approach is easier to centralize.

Workloads that benefit from logical replication — ETL pipelines, change data capture into Kafka or other streams, and gradual migrations between database versions — are often easier on PostgreSQL than on MongoDB. PostgreSQL’s logical replication is mature, supported by the database itself, and integrated with widely used tools (Debezium, pg_logical, pglogical). MongoDB has change streams, which solve the same problem with a different shape, but they live at a higher level and are tightly coupled to the MongoDB protocol; they are great for MongoDB-aware consumers and less useful when you need to feed data into systems that want SQL semantics.

Where MongoDB’s model wins

MongoDB’s built-in HA is a real asset for teams that do not have specialized database operations expertise. A four-person backend team can deploy a three-node MongoDB replica set in production and trust that automatic failover will work without ever reading about Raft. The same team running PostgreSQL with Patroni would need to learn etcd, distributed consensus failure modes, fencing, and a half-dozen Patroni configuration parameters before they could be confident the cluster would recover correctly from a primary failure. Both teams can succeed, but the MongoDB team starts higher up the learning curve.

Cloud-native workloads often fit MongoDB’s model well. A Kubernetes deployment that uses StatefulSets, headless services, and DNS-based discovery can run a MongoDB replica set with minimal extra infrastructure. The cluster discovers itself through DNS, drivers handle topology, and the operator does not need to maintain a separate coordination layer. PostgreSQL operators on Kubernetes typically run a Postgres operator (CloudNativePG, Stolon, Zalando’s Postgres Operator, Crunchy Data’s operator) that wraps Patroni or implements similar logic, which is not much extra work but is one more dependency to keep track of.

Workloads that want fine-grained durability per operation — some writes need majority acknowledgment, others can fire-and-forget — fit MongoDB’s write concerns naturally. PostgreSQL can simulate this with synchronous_commit overrides per session, but the model is more awkward. Conversely, an application that wants every write to have the same durability guarantee finds PostgreSQL’s server-side configuration easier to reason about.

A practical decision framework

When you cannot decide based on workload features alone, fall back to operational fit. Ask three questions. First, what does your team know? An ops team with deep PostgreSQL experience will run PostgreSQL well, even with the additional HA tooling overhead. A team with MongoDB experience will deliver more reliably on MongoDB. Skills compound; the database you are best at running is the database you should run, all else being equal.

Second, what does your platform look like? If your platform already provides Patroni-style coordination (etcd, Consul, Zookeeper), running PostgreSQL on top of it is straightforward. If your platform is opinionated about minimizing dependencies (small Kubernetes clusters, edge deployments, single-tenant workloads), MongoDB’s self-contained model is a better fit. The infrastructure you do not have is sometimes the deciding factor.

Third, what is your worst-case failure mode? PostgreSQL with Patroni has well-understood failure modes (etcd partition, fencing failure, pg_rewind divergence) that experienced operators can debug. MongoDB has its own (rollback files, network partitions that strand minorities, oplog window exhaustion) that experienced MongoDB operators can debug. The systems are roughly comparable in reliability when run by experienced teams. If your team is new to both, the simpler operational model of MongoDB usually wins, because there is less to misconfigure.

A final note: the choice is rarely permanent. Both systems are mature, both have migration tools, and both can be replaced if the original choice turns out wrong. The cost of switching is real but not catastrophic for most workloads. The cost of choosing badly and then refusing to revisit the decision is much higher. If you choose PostgreSQL and your team finds itself spending half its time tuning Patroni, that is a signal worth acting on. If you choose MongoDB and discover you really need cross-version logical replication, that is worth acting on too. The replication and failover models in this chapter are a starting point for the choice, not the end of it.

Summary

PostgreSQL exposes replication primitives (WAL streaming, replication slots, hot standbys) and leaves automatic failover to external tools such as Patroni, repmgr, pg_auto_failover, or Kubernetes-native operators like CloudNativePG.
MongoDB ships replication and automatic failover as part of the database itself, implemented through the ReplicationCoordinator, the TopologyCoordinator, and a Raft-based election protocol.
PostgreSQL supports two replication mechanisms: physical (byte-for-byte WAL streaming, used for HA) and logical (publish-subscribe row-level replication, used for cross-version upgrades and selective replication).
MongoDB has a single replication model based on the oplog, a capped collection that records logical operations and is tailed by secondaries through pull-based replication.
Synchronous durability is configured at the cluster level in PostgreSQL through synchronous_standby_names (with FIRST and ANY methods) and at the per-write level in MongoDB through write concerns.
Failure detection in MongoDB happens through a heartbeat protocol that every node runs internally; PostgreSQL has no equivalent in core and relies on external monitors for detection.
Leader election in MongoDB uses Raft consensus with a dry-run phase to prevent unnecessary term increments; PostgreSQL leader election runs in external tools (Patroni with etcd, repmgr, or pg_auto_failover).
Split-brain prevention in both systems requires a majority of voting members; without a majority, the cluster cannot elect a primary or accept new writes.
MongoDB drivers maintain a topology view of the cluster and handle failover transparently, with retryable writes providing automatic safe retry across primary changes.
PostgreSQL clients connect to a hostname and require external mechanisms (VIPs, connection poolers like PgBouncer, or multi-host JDBC strings) to route connections after a failover.
In-flight transactions on a failed PostgreSQL primary are lost and must be retried by the application; MongoDB unprepared transactions are explicitly aborted, while prepared transactions survive stepdown for sharded two-phase commit safety.
A typical PostgreSQL HA stack involves PostgreSQL plus Patroni plus etcd plus a connection-routing layer; a typical MongoDB HA deployment involves only the mongod processes and standard drivers.
Major-version upgrades are easier in MongoDB (rolling upgrades within a replica set) than in PostgreSQL (where physical replication does not span major versions and pg_upgrade or logical replication is required).
Choose PostgreSQL when you need replication flexibility, mature logical replication tooling, or fit with existing infrastructure conventions (etcd, Consul, Kubernetes operators).
Choose MongoDB when you want HA without a separate coordination layer, per-operation durability semantics, or operational simplicity for teams without deep database operations expertise.

Designing the Agent Memory Schema: Document Shapes for Short-Term, Episodic, and Semantic Memory in MongoDB

MongoDB Guests — Tue, 19 May 2026 09:00:00 +0000

This tutorial was written by Yunying Karen Zhang.

If you've been following the recent wave of writing on AI agent memory, you've probably read about the taxonomy of memory types, or how frameworks like LangGraph expose a Memory Store API. All of that is a valuable foundation. But there's a gap: none of it shows you what the documents actually look like.

The taxonomy tells you what the memory types are. The framework docs tell you how to call the API. This post fills the space in between — it's about how to design the collections when you are the one sitting in front of MongoDB Compass with a blank schema.

By the end, you'll have three concrete document shapes, the indexes that back them, and a clear picture of how they wire together. Let's build it from the ground up.

Short-Term Memory: The Conversation Window

What it stores

Short-term memory is the live message log for an active session. It answers the question: what has been said in this conversation so far? It also holds a rolling summary—a compressed version of older turns that gets substituted in when the raw message log would overflow the model's context window.

This is the most transient of the three stores. Once a session ends, this data is largely superseded by the episodic record we'll cover in Section 2. It's a natural candidate for TTL expiration.

Document shape

Below is a representative document for a live conversation session. We'll walk through each field afterward:

{
  _id: ObjectId("..."),
  thread_id: "thread_abc123",
  user_id: "user_9921",
  messages: [
    {
      role: "user",
      content: "What's the status of my refund?",
      timestamp: ISODate("2026-04-29T14:15:00Z"),
      token_count: 12
    },
    {
      role: "assistant",
      content: "Let me look that up for you.",
      timestamp: ISODate("2026-04-29T14:15:03Z"),
      token_count: 9
    }
  ],
  summary: "User initiated chat at 14:00 regarding general policy; now specifically asking for refund status on order #8842.",
  summary_updated_at: ISODate("2026-04-29T14:10:00Z"),
  created_at: ISODate("2026-04-29T14:00:00Z"),
  expires_at: ISODate("2026-04-30T14:00:00Z")
}

Field-by-field walkthrough

thread_id is the primary lookup key. Every read against this collection is "give me everything for this thread", it's always a point query on this field.
user_id is on every document in every collection in this schema. We scope all queries to a user first, always. Cross-user memory leakage is an easy mistake and a serious one, never retrieve a thread without a user_id filter unless you have an explicit reason to.
messages[] is an embedded array. Each element carries role, content, timestamp, and token_count. Tracking token count per message lets us calculate how close we are to context window limits without re-tokenizing the whole conversation on every turn.
summary and summary_updated_at work together. When the cumulative token count of messages[] exceeds a threshold — say, 80% of the model's context window, we compress the oldest turns into summary, drop those messages from the array, and update summary_updated_at. This timestamp tells us how stale the summary is: if several new messages have arrived since the last summary, we may want to regenerate before injecting it into the prompt.
expires_at pairs with a TTL index. Sessions shouldn't persist forever; 24 to 48 hours is a reasonable default for most applications.

Embed vs. reference

We embed messages inside the document rather than storing them in a separate collection because the read pattern always gives the whole thread. There is no use case for fetching a single message in isolation. Embedding keeps the read to a single document fetch and lets us atomically append a new message with a simple $push. The one trade-off is document size, but the summary-and-truncate mechanism keeps this bounded in practice.

Index

db.short_term_memory.createIndex({ thread_id: 1 }, { unique: true })
db.short_term_memory.createIndex({ expires_at: 1 }, { expireAfterSeconds: 0 })

That's all we need here. No vector index — we are never searching this collection by meaning. We always know the thread_id before we read. For more on createIndex options, see the MongoDB documentation.

Episodic Memory: What the Agent Actually Did

What it stores

Episodic memory is a record of a completed agent run. Not what was said — that's the chat log. What was done: which tools were called, what their inputs and outputs were, and what the overall outcome was.

Think of it as the agent's after-action report. When a future session asks "Did I already try to book that flight?" or "What happened the last time this user asked about a refund?", episodic memory is where we look.

Document shape

{
  _id: ObjectId("..."),
  episode_id: "ep_20260429_xk92",
  thread_id: "thread_abc123",
  user_id: "user_9921",
  summary: "Looked up refund status for order #8842. Confirmed refund is processing, ETA 3-5 business days.",
  tool_calls: [
    {
      tool: "lookup_order",
      inputs: { order_id: "8842" },
      output: { status: "refund_processing", eta_days: 5 },
      success: true,
      called_at: ISODate("2026-04-29T14:01:05Z")
    }
  ],
  outcome: "succeeded",
  tags: ["refund", "order-lookup"],
  started_at: ISODate("2026-04-29T14:01:00Z"),
  ended_at: ISODate("2026-04-29T14:01:08Z")
}

Field-by-field walkthrough

episode_id uniquely identifies this run. It becomes the source_episode_id on any semantic memories that get extracted from this episode, giving us a clear provenance trail.
thread_id + user_id links the episode back to the session it came from. thread_id lets us find all episodes that share a conversation context; user_id lets us find all episodes for a given user across all sessions.
summary is a human-readable description of what happened, written or generated at episode close. We keep it factual and outcome-oriented: what was the goal, what was done, what was the result. This is the field we search when a future agent needs to recall prior work.
tool_calls[] is the detailed trace. We store inputs and outputs here so we can diagnose failures, avoid repeating failed approaches, and give the agent evidence to reason from. For example, "last time I called lookup_order with this ID, it returned status not_found." Note that, to avoid hitting the 16MB BSON document limit, you should implement a truncation policy for large tool outputs or cap the maximum number of tool calls per episode. If your agents perform hundreds of steps with heavy payloads, consider moving these traces to a separate tool_call_logs collection referenced by episode_id. For this reference schema, we assume bounded episodes where the action history remains within the document limit.
outcome is a controlled vocabulary field: succeeded, failed, or abandoned. It's a fast filter; if a future agent is looking for precedent, it might specifically want the last successful episode of a given type.
tags[] are optional but valuable for retrieval. They let us filter episodic recall to a domain, "show me recent refund episodes for this user", without having to run a full-text search on summary.

Why this is different from the chat log

The distinction matters in practice. The chat log captures conversation, turns, tone, and clarifications. The episode captures action, what the agent committed to, and what happened. We need both, and they serve different retrieval purposes. We pull the chat log to reconstruct conversational context; we pull the episode to reconstruct operational history.

Index

db.episodic_memory.createIndex({ episode_id: 1 }, { unique: true })
db.episodic_memory.createIndex({ user_id: 1, started_at: -1 })
db.episodic_memory.createIndex({ user_id: 1, tags: 1, started_at: -1 })
db.episodic_memory.createIndex(
{ summary: "text" },
{ default_language: "english" }
)
)

The unique index on episode_id is our primary integrity guard, ensuring that provenance links from other collections remain stable. The compound index on user_id + started_at serves a double-duty: it powers the most common "most recent N episodes" queries while providing the necessary performance for background retention sweeps (e.g., deleting records older than 90 days).

Adding tags to the compound index supports filtered recall by domain, allowing the agent to retrieve history specific to a topic like "refunds" or "billing." However, since tags is an array, this becomes a Multikey Index. It is a critical MongoDB constraint that a compound index can only contain one array field; if you later extend this schema with another array (like tools_used[]), you cannot add it to this existing index without causing inserts to fail. Finally, the text index on summary handles the "fuzzy" keyword-based discovery that vector search often misses

Semantic Memory: What the Agent Knows About the User

What it stores

Semantic memory is the long-term knowledge store, persistent facts and preferences that should survive across sessions indefinitely. This is where the agent remembers that a user prefers metric units, dislikes upsells, or has a standing instruction to always confirm before booking.

Unlike short-term and episodic memory, we retrieve semantic memories by meaning, not by session or timestamp. This is the collection that genuinely needs a vector index.

Document shape

{
  _id: ObjectId("..."),
  memory_id: "mem_u9921_0047",
  user_id: "user_9921",
  type: "preference",
  content: "User prefers distances and weights in metric units.",
  embedding: [0.023, -0.117, 0.204, ...],
  source_episode_id: "ep_20260429_xk92",
  strength: 0.87,
  last_accessed_at: ISODate("2026-04-29T14:01:00Z"),
  created_at: ISODate("2026-04-01T09:15:00Z")
}

Field-by-field walkthrough

user_id is critical here. We should always filter by user_id before running vector search. MongoDB Atlas Vector Search supports pre-filters on indexed fields — we use them. Running a pure ANN search across all users and then filtering the results is both slower and a potential data isolation bug waiting to happen.
type is a controlled vocabulary: preference (the user wants something a certain way), fact (something true about the user or their context), or instruction (an explicit standing directive). This lets us selectively inject memories by type depending on what the current task needs.
content is the plain-text statement that gets injected into the prompt. We keep it short and declarative — i.e., one fact per document. Chunking multiple facts into a single document makes both retrieval and decay harder to reason about.
embedding is the vector representation of content, generated at write time and stored as a BSON array of doubles. While often prototyped as a BSON array of doubles, you should use BSON BinData (with the vector subtype) for production systems. Using BinData allows MongoDB to compress your embeddings, requiring roughly three times less disk space. More importantly, it enables Atlas Vector Search to leverage quantization (like int8 or binary), which can reduce RAM requirements by up to 24x—a critical optimization for 2026-scale agentic memory stores where high-dimensional vector storage costs can otherwise become prohibitive. On retrieval, we embed the current query and run a nearest-neighbor search filtered by user_id.
strength is a float between 0 and 1 that decays over time and resets toward 1 on access. We recommend an exponential decay with a half-life (e.g., 30 days) combined with a multiplicative reset on access. For example, every time a memory is retrieved, you might close 30% of the gap toward 1.0 ( $strength_{new} = strength_{old} + (1.0 - strength_{old}) \times 0.3$ ). This ensures that frequently used preferences stay near 1.0, while unreferenced "facts" naturally sink to the bottom of your search results over time, allowing the agent's "personality" to evolve with the user. Memories that haven't been relevant in a long time fade; memories that keep being retrieved stay strong. This field is our alternative to TTL expiration for long-term memory - we don't want to hard-delete a preference just because it hasn't been triggered in 90 days, but we do want to deprioritize it in retrieval ranking. To manage this, we use a periodic background sweep (e.g., a daily cron job) that targets the last_accessed_at field to apply decay, rather than calculating it at read-time, which would add unnecessary latency to every retrieval.
source_episode_id is the provenance link, it tells us which agent run produced this memory. This matters for auditability and for bulk-invalidating memories that came from a faulty episode (e.g., if a tool malfunctioned). We don't index this field by default to save on write overhead, as bulk invalidation is typically an infrequent administrative task. However, if your pipeline requires high-frequency memory rollbacks, you should add a standard index: db.semantic_memory.createIndex({ source_episode_id: 1 }).

Embed vs. reference

Each semantic memory is its own document. We retrieve these semantically and independently — there is no "give me all memories for this thread" read pattern. References would add indirection without benefit. The document-per-memory model also makes it straightforward to update strength and last_accessed_at atomically without touching sibling memories.

Index

// Primary lookup and uniqueness constraint
db.semantic_memory.createIndex({ memory_id: 1 }, { unique: true })

// Compound index for non-vector queries (retrieval ranking)
db.semantic_memory.createIndex({ user_id: 1, type: 1, strength: -1 })

// Supporting provenance lookups and bulk invalidation
db.semantic_memory.createIndex({ source_episode_id: 1 })

// Index for background decay sweeps
db.semantic_memory.createIndex({ user_id: 1, last_accessed_at: 1 })

// Atlas Vector Search Index Definition
{
  "fields": [
    {
      "type": "vector",
      "path": "embedding",
      "numDimensions": 1024, // Optimized for voyage-3-large
      "similarity": "cosine"
    },
    {
      "type": "filter",
      "path": "user_id"
    },
    {
      "type": "filter",
      "path": "type"
    }
  ]
}

The vector index definition includes user_id and type as filter fields; this is what enables the $vectorSearch pre-filter. Always filter by user_id before vector search to prevent cross-user leakage.

The Referential Model: How the Three Collections Wire Together

The three join keys are user_id, thread_id, and episode_id. Here is how they move through a session lifecycle:

Session starts
  → short_term_memory document created
      { thread_id: "thread_abc123", user_id: "user_9921", messages: [] }

Turns accumulate
  → messages[] grows via $push
  → summary updated when token budget is reached

Session ends / agent run completes
  → episodic_memory record written
      { episode_id: "ep_...", thread_id: "thread_abc123", user_id: "user_9921" }

Semantic extraction runs
  → facts and preferences identified from the episode
  → semantic_memory documents written
      { source_episode_id: "ep_...", user_id: "user_9921", ... }

The read pattern per memory type

Short-term: fetch by thread_id. This is a single point query on one document. We always know the thread before we read.
Episodic: query by user_id sorted by started_at descending for recency, or combine with a tags filter for domain-specific recall. Text search on summary for keyword-based retrieval.
Semantic: $vectorSearch with a user_id pre-filter, optionally narrowed by type. Rank results by vector similarity, break ties, or rerank by strength.

What we never do

Join across these collections at query time. Each retrieval path is independent by design. The thread_id and episode_id fields are provenance links, useful for audit, debugging, and bulk operations, not foreign keys that we join on in the hot path.

Index Strategy: What Actually Needs a Vector Index (and What Doesn't)

The most common mistake in early agent memory implementations is adding vector indexes everywhere. Here is where we actually need them.

Short-term memory: no vector index. We always retrieve by thread_id. We know the session before we read. A vector index here would never be used and would slow down writes for no benefit.
Episodic memory: maybe, later. A text index on summary covers the majority of episodic recall needs — keyword-based retrieval like "find episodes involving refunds for this user." A vector index is only justified when we need semantic episode recall: "when did I last help this user with X?" — and most applications don't need this at launch. Start without it, add it only when we have a concrete retrieval requirement that text search can't satisfy.
Semantic memory: yes, this is the one. Semantic memory is our knowledge retrieval layer. Retrieval by meaning is the whole point. This collection needs a vector index from day one.

TTL strategy

Short-term memory is the natural TTL candidate. We wire up the TTL index on expires_at and let MongoDB handle the cleanup. A 24 to 48-hour expiry is reasonable for most applications.
Semantic memory should not use TTL expiration. A preference learned six months ago and not recently triggered should deprioritize in retrieval, not disappear entirely. We use the strength field for soft decay and reserve hard deletion for explicit user requests or compliance requirements.

Summary

Here are the three schemas as a single reference:

Attribute	short_term_memory	episodic_memory	semantic_memory
Primary key	`thread_id`	`episode_id`	`memory_id`
Scope key	`user_id`	`user_id`	`user_id`
Retrieved by	`thread_id`	`user_id` + recency/tags	vector search + `user_id`
Vector index	No	Optional	Yes
TTL	TTL index	No	No
Expiry mechanism	`expires_at` TTL	`started_at` policy	strength decay

The taxonomy, short-term, episodic, semantic, maps cleanly onto three MongoDB collections with distinct retrieval patterns and distinct index strategies. Each collection is optimized for how it is actually read, not for a generalized "memory store" abstraction.

Short-term memory is optimized for high-speed session lookups and uses native TTL indexes for automatic cleanup.
Episodic memory serves as the immutable audit trail. We reuse the started_at index for both recency-based retrieval and manual retention sweeps, avoiding the overhead of a separate age-based index.
Semantic memory acts as the long-term knowledge base. It is the only collection requiring a vector index and uses a background "strength decay" sweep based on last_accessed_at to keep retrieval results fresh and relevant.

For the conceptual background on agent memory types, the LangGraph Memory Store documentation and Richmond's taxonomy post are the right starting points. This schema is what you build once you've read those and are ready to sit down with a database.

Deploying MongoDB MCP Server on Amazon Bedrock AgentCore

MongoDB Guests — Fri, 15 May 2026 09:00:00 +0000

Authors: Igor Alekseev (PSA AWS), Anuj Panchal (SA MongoDB), Vin Dahake (SA AWS)

AI agents are only as useful as the tools they can access. With the Model Context Protocol (MCP) becoming the standard for connecting AI models to external data sources, running MCP servers in managed cloud environments is the natural next step. In this post, we'll walk through deploying the MongoDB MCP Server on Amazon Bedrock AgentCore - giving your AI agents direct, structured access to MongoDB databases.

What You're Building

By the end of this guide, you'll have a containerized MongoDB MCP Server running as an AgentCore MCP runtime. Your AI agents will be able to query collections, inspect schemas, run aggregations, and interact with MongoDB Atlas - all through the MCP protocol, managed by AgentCore.
The architecture is straightforward:

AI Agent → Bedrock AgentCore → MongoDB MCP Server (container) → MongoDB / Atlas

AgentCore handles session management, scaling, and invocation routing. The MCP server handles the MongoDB-specific tooling.

Prerequisites

Before getting started, make sure you have:

AWS CLI v2 installed and configured (aws configure)
Docker (or Finch) for building container images
An Amazon ECR private repository for storing the container image
Access to Amazon Bedrock AgentCore in your target region
A MongoDB connection string or Atlas API credentials

Step 1: Create an ECR Repository

First, create a private ECR repository to store the MCP server image:

aws ecr create-repository \
  --repository-name mongodb-mcp-server \
  --region us-east-1

Take note of the repository URI in the output - you'll need it in the next step.

Step 2: Build and Push the Container Image

The MongoDB MCP Server repository includes an AWS-optimized Dockerfile at deploy/aws/Dockerfile, pre-configured for AgentCore compatibility. Start by cloning the repo:

git clone https://github.com/mongodb-js/mongodb-mcp-server.git
cd mongodb-mcp-server

Then authenticate with ECR, build the image, and push it:

# Authenticate Docker with ECR
aws ecr get-login-password --region us-east-1 | \
  docker login --username AWS --password-stdin \
  <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com

# Build for ARM64 (required by AgentCore)
docker build --platform linux/arm64 \
  -t <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/mongodb-mcp-server:latest \
  -f deploy/aws/Dockerfile .

# Push to ECR
docker push <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/mongodb-mcp-server:latest

Important: AgentCore runtimes only support linux/arm64 images. Always build with --platform linux/arm64.

What's in the Dockerfile

The container is intentionally minimal. Here's what it configures:

Setting	Value	Why
MDB_MCP_EXTERNALLY_MANAGED_SESSIONS	true	Lets AgentCore manage MCP session IDs instead of the server
MDB_MCP_HTTP_RESPONSE_TYPE	json	Returns JSON responses instead of Server-Sent Events
MDB_MCP_DISABLED_TOOLS	atlas-local	Disables local deployment tools that don't work in containers
MDB_MCP_TRANSPORT	http	Runs over HTTP instead of the default stdio transport
Port	8000	The HTTP listener port AgentCore connects to

The server runs as a non-root user (mcp) for security, and the image is based on node:24-alpine to keep it lightweight.

Step 3: Configure the AgentCore Runtime

When creating your AgentCore MCP runtime, point the container image URI to your ECR image. The key configuration happens through environment variables that you set in the AgentCore runtime configuration.

Passing MongoDB Credentials

At minimum, you need to provide a way for the server to connect to your database. Set these environment variables in your AgentCore runtime:
For a direct MongoDB connection:

MDB_MCP_CONNECTION_STRING=mongodb://username:password@host:port/database

Note: Use standard mongodb:// connection strings. The mongodb+srv:// format is not yet supported by AgentCore.
For MongoDB Atlas API access (enables Atlas management tools):

MDB_MCP_API_CLIENT_ID=your-atlas-service-account-client-id
MDB_MCP_API_CLIENT_SECRET=your-atlas-service-account-client-secret

You can provide both a connection string and Atlas API credentials to enable the full set of tools --- database operations plus Atlas cluster management.

Optional Configuration

The server supports a wide range of configuration options through environment variables. A few worth considering for production:

MDB_MCP_READ_ONLY=true - Restricts the server to read-only operations. Highly recommended if your agents only need to query data.
MDB_MCP_MAX_DOCUMENTS_PER_QUERY=100 - Caps the number of documents returned per query (default: 100).
MDB_MCP_TELEMETRY=disabled - Disables telemetry collection if needed.
MDB_MCP_INDEX_CHECK=true - Rejects queries that don't use an index, enforcing performance best practices.

See the full configuration reference for all available options.

Step 4: Invoke the Runtime

Once deployed, you can invoke the AgentCore runtime using the Bedrock AgentCore API. Here's how to construct the invocation URL and make a request:

# Set your runtime ARN
AGENT_ARN="arn:aws:bedrock-agentcore:<REGION>:<ACCOUNT_ID>:runtime/<RUNTIME_NAME>"

# URL-encode the ARN
ENCODED_ARN=$(python3 -c "import urllib.parse; print(urllib.parse.quote('$AGENT_ARN', safe=''))")

# List available tools
curl -X POST \
  "https://bedrock-agentcore.<REGION>.amazonaws.com/runtimes/${ENCODED_ARN}/invocations?qualifier=DEFAULT" \
  -H "Authorization: Bearer <TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"tools/list","id":1}'

This returns the full list of MCP tools the server exposes - database queries, schema inspection, aggregation pipelines, Atlas management, and more.

Available Tools

Once deployed, your agents get access to a comprehensive set of MongoDB tools:

Database operations - find, aggregate, insert-many, update-many, delete-many, count, and more. These cover the full range of CRUD operations plus aggregation pipelines.
Schema and metadata - collection-schema, collection-indexes, list-databases, list-collections, db-stats. Useful for agents that need to understand the data model before querying.
Atlas management - atlas-list-clusters, atlas-create-free-cluster, atlas-list-projects, atlas-get-performance-advisor, and others. These let agents manage Atlas infrastructure directly.
Knowledge search - search-knowledge and list-knowledge-sources provide access to MongoDB's documentation and expert guidance, helping agents answer MongoDB-related questions accurately.

Monitoring

Keep an eye on your deployment through Amazon

CloudWatch logs:
aws logs describe-log-groups \
  --region us-east-1 \
  --log-group-name-prefix "/aws/bedrock-agentcore"

The server logs to stderr by default in the container configuration, which AgentCore captures and forwards to CloudWatch.

Security Considerations

A few things to keep in mind for production deployments:

Use read-only mode (MDB_MCP_READ_ONLY=true) unless your agents genuinely need write access.
Scope Atlas API credentials to the minimum required permissions using MongoDB Atlas Service Accounts.
Use standard MongoDB authentication - create a dedicated database user for the MCP server with only the necessary privileges.
Monitor query patterns - use the MDB_MCP_INDEX_CHECK=true option to prevent unindexed collection scans that could impact database performance.

Wrapping Up

Deploying the MongoDB MCP Server on AgentCore gives your AI agents a reliable, managed path to MongoDB data. AgentCore handles the infrastructure concerns --- session management, scaling, health checks --- while the MCP server provides the structured tooling layer that agents need to work with databases effectively.

The setup is minimal: one Dockerfile, a few environment variables, and you're running. From there, it's about tuning the configuration to match your security and performance requirements.

For more details, check out:

Whistleblower for Database: Set up an internal informant who exposes performance

MongoDB Guests — Thu, 07 May 2026 10:44:20 +0000

This tutorial was written by Darshan Jayarama.

As DBAs, waking up at 2 AM because production is down or slow is a nightmare, and none of us like it. However, we all know that it is not instant karma; it is like a pressure cooker where the database has reached its breaking point. Database whistleblowers constantly give signals that we ignore before it explodes into a Production Issue.

Whistleblower 1: Performance Informant

High CPU utilization?

CPU utilization is not just an alert, it is a future prediction with regard to the database. When you get constant High CPU utilization alerts, think of it as a Whistleblower reviewing the database logs, metrics, and processes that are consuming CPU. If possible, shift those processes to a different server and let your Database breathe without pressure. Ignoring high CPU utilization can lead to high latency.

Query <5 seconds?

A query that was blazing fast is now causing a slow query alert since the last data load. It is a clear indication of a Missing or Unoptimized index. If an index is missing, create one. If unoptimized, then redesign a better index. Still facing the same? It needs a deeper and wider investigation.

Cache hit ratio <80%?

A lower cache hit ratio means higher disk reads, which tends to cause high IO Wait, and an Increased disk queue depth. It can happen if your query is unoptimized (we had addressed this earlier), or if lower memory/cache is assigned. Increase the memory if there is nothing pending in Database tuning.

Whistleblower 2: Cost Informant

Storage >80%:

Set your whistleblower when storage utilization hits >80% its your duty now to analyze storage to archive any old logs, old data, or remove unwanted processes that consume your space.

IOPS > Budget?

We usually get an alert for a high document-scanned-to-returned ratio, which indicates that queries are performing more IO than required. Optimize them. Reduce IOPS consumption by rewriting queries, redesigning the schema, and implementing a better Index strategy.

Unexpected auto scale?

MongoDB Atlas has a feature that monitors system load and automatically scales up or down as needed. Implement the solution. In case of a scale down, it does require 24-hour monitoring, but if you are sure there are no more surprises, you can scale down yourself.

Whistleblower 3: Reliability Informant

Replication lag >10 Seconds?

Investigate where it is lagging. Is it in primary? Must be loaded from writes, which lowers the read request priority. It usually triggers the built-in flow-control logic. If secondary is the issue, then is it at writing or reading? A detailed understanding of flow can help resolve this data durability. Ask developers to set writeConcern to match the application's requirements.

Oplog <24Hours?

A lower oplog size means lower retention of replication events. Set at least 24 hours, resize as per the application write load minOplogRetention.

Connection >80%?

Each connection can consume 1 MB of memory; the more connections, the more memory is consumed. Maintain connection hygiene, use a connection pool, and close connections when no longer needed.

Conclusion: Your production incident is one missed alert away; hire your whistleblower. Choose: Pressure cooker explosion → OR → Proactive DBA legend?

Vacuum Seal Your MongoDB: Cuts Cost Down 50% Today

MongoDB Guests — Tue, 05 May 2026 10:10:10 +0000

This article was written by Darshan Jayarama.

I was recently packing for a short trip, and I personally hate carrying 2–3 pieces of luggage. One for my laptop, another for clothes. So a person like me has definitely used a “vacuum sealer,” allowing me to pack as many clothes as I want.

During the trip, I had an idea. Which vacuum sealers do we have to help store efficiently and reduce billing? Here, I have jotted down some…

Online Archive

Does your dataset contain cold data? For a supply chain company, data older than three months is considered infrequently accessed. For an e-commerce website or a courier company, data that has already been delivered is accessed by users very rarely. We can group them using a query, such as lastAccess > 30, and create the Online archive store in cloud object storage. Worried about how to access this when needed? Relax — it’s all covered, there are no access restrictions. Storage costs drop from $4/GB/Month to $0.02/GB/Month.

Table Compact

In a write-intensive application where you frequently delete data, storage watermarks always stay high. You can run the below command to check:

db.COLLNAME.stats().freeStorageSize

The above command outputS how many bytes are available to be reused. Even though wiredTiger uses this space organically over time, if the storage utilization is effectively essential, you can run compact on the collection.

We can run compact with the following syntax:

use mydb // switch to the database 
db.runCommand( { compact: "myCollection" } ) // mention the collection name

As it also has many other options, I would recommend visiting the official documentation of compact.

Resync Secondary:

Running compact is fine if you’ve got just a handful of collections. But when you’re dealing with many collections with high watermarks, especially large ones, running compact isn't ideal.

You can resync a member in rolling fashion so that all members can release that space back to the operating system and utilize the storage efficiently.

TTL Index:

So far, we have discussed the reactive approach for storage utilization. We are preparing the plan once the storage hits the ceiling. But do we have any such proactive approach? Yes, here is where TTL comes into play. If you believe the data is no longer needed after a certain time, set the TTL index to expire, and the record will be deleted.

This approach will be useful for the application that generates large logs. Set the TTL index for 3 days, and the log entry will be removed from the collection.

Query Optimization:

Being a query enthusiast, I circle back to query optimization, as a single bad query can eat up your IOPS, Network, and Compute.

Tune the query.
Create a compound index covering the query and the projection
Reduce the network round-trip and IOPS
If possible, precompute the value and store it in a collection.

Don't know where to start? Use/Abuse the Atlas Performance Advisor for index recommendations, inefficient queries, and inefficient schema advice. Or visit my previous blogs:

Flex Cluster:

Planning to perform some dev/API test? Go with the Flex cluster rather than the M10+ cluster for non-prod deployment. Same API, essentially Pause/Resume when not needed.

Conclusion:

MongoDB gives tons of storage-saving options — but only smart DBAs use them. Your cluster is bleeding $$$ right now. Online Archive, TTL indexes, compaction, proper sizing = 60% instant savings.

Pick 3 techniques → Deploy → Watch bill shrink. Simple as that. 💰

My first “local” Vector Search: MongoDB community edition

MongoDB Guests — Thu, 30 Apr 2026 10:56:11 +0000

This article was written by Darshan Jayarama.

Ever since I received an email about the vector search auto-Embedding feature released in MongoDB community edition, my palms have been itching to test it out. And now.. I’m happily writing this blog after witnessing its power firsthand.

Vector Search is one of the most admired, powerful, and amazing products of MongoDB. I felt pity for MongoDB Community/On-prem customers, as they were unable to use this because Vector search and Atlas search were only available in MongoDB Atlas. But this is no longer the case!

How did I test this?

I am sure by now you might be thinking, “Enough of this product endorsement, show me the code!”

I’m getting there :). Initially, I followed what the MongoDB documentation suggests, but I struggled a lot as there is a disconnection when setting up the Docker containers:

As per the Installation documentation, select Docker as your operating system.
When the MongoDB community server starts, it starts with hostname as mongod.search-community:27017. Make a note of this.
When we asked to start the MongoDB Search container, in that yml file, it sets the syncsource as mongot-community.search-community:27017
- Don't just copy/paste the yml from there. Make sure you are setting the correct syncSource, which should be the Community server container name mongod.search-community:27017. To ensure your container hostname, run db.isMaster().me, it should print you the hostname.
Next, there is mongot username and password entry that should be made in the yml file. Whileyou are creating a user, the instructions specify mongot user as mongot, but in the yml file, it is mentioned as mongotUser. If you created it with another name, correct the section.

After these corrections, you can see the 2 containers running happily (I literally jumped out of my chair over joy). Next, connect to the mongod using mongosh to test the power.

I created an index on the favorite movies collection, plot field:

db.movies.createSearchIndex("vector_index", "vectorSearch", {
 "fields": [
   {
     "type": "autoEmbed",
     "modality": "text",
     "path": "plot",
     "model": "voyage-4"
   },
   {
     "type": "filter",
     "path": "year"
   }
 ]
})

You can use voyage-4-lite, but as per the voyage usage stats, both voyage-4 and voyage-4-lite both have the same 10000TPM 3RPM limitation. My preferred choice is voyage-4.

Then I wanted to see the results based on the context. So I ran the below query:

db.movies.aggregate([
 {
   "$vectorSearch": {
     "index": "vector_index",
     "path": "plot",
     "query": {
       "text": "bullied boy learns karate"
     },"model": "voyage-4-lite" ,
           "numCandidates": 10000,
     "limit": 10
   }
 },
 {
   "$project": {
     "_id": 0,
     "title": 1,
     "year":1,
     "plot": 1,
     "score": { $meta: "vectorSearchScore" }
   }
 }
])

The above query should retrieve movies with context of “bullied boy learns karate” (here we expect ‘karate-kid’):

The answers amused me:

[
 {
   plot: 'A love-struck weakling must pretend to be boxer in order to gain respect from the family of the girl he loves.',
   title: 'Battling Butler',
   year: 1926,
   score: 0.6800339221954346
 },
 {
   plot: 'Two young brothers become the leaders of a gang of kids in their neighborhood. Their father is an office clerk who tries for advancement by playing up his boss. When the boys visit the boss...',
   title: 'I Was Born, But...',
   year: 1932,
   score: 0.6692402362823486
 },
 {
   plot: 'A living puppet, with the help of a cricket as his conscience, must prove himself worthy to become a real boy.',
   title: 'Pinocchio',
   year: 1940,
   score: 0.6628834009170532
 },
 {
   plot: 'An idealistic adolescent, suffering under the thumb of a sadistic schoolmaster, falls in love with a loose girl who is bullied and tormented by another lover.',
   title: 'Torment',
   year: 1944,
   score: 0.6620646715164185
 },
 {
   plot: `Against all odds Father Flanagan starts "Boys' Town" after hearing a convict's story. Whitey Marsh comes there. He runs away but, hungry, returns. He runs away again but, when friend Pee ...`,
   title: 'Boys Town',
   year: 1938,
   score: 0.6570459604263306
 },
 {
   plot: 'When three thuggish men are responsible for the death of his father and the crippling of his brother, young David must choose between supporting his family or risking his life and exacting vengeance.',
   title: "Tol'able David",
   year: 1921,
   score: 0.6559526920318604
 },
 {
   plot: "Fight promoter Nick Donati grooms a bellhop as a future champ, but has second thoughts when the 'kid' falls for his sister.",
   title: 'Kid Galahad',
   year: 1937,
   score: 0.6525536179542542
 },
 {
   plot: 'In a repressive boarding school with rigid rules of behavior, four boys decide to rebel against the direction on a celebration day.',
   title: 'Zero for Conduct',
   year: 1933,
   score: 0.651709794998169
 },
 {
   plot: 'While at a ski lodge, Larry Blake sees instructor Karin Borg and decides to sign up for private lessons. The next thing he knows, she is Mrs. Blake. When he announces that he is going back ...',
   title: 'Two-Faced Woman',
   year: 1941,
   score: 0.6513179540634155
 },
 {
   plot: 'To reconcile with his girlfriend, a bookish college student tries to become an athlete.',
   title: 'College',
   year: 1927,
   score: 0.6459328532218933
 }
]

Even though the expected result was not there, all the results related to karate, boxing, or being bullied.

Below are the commands I have used to complete the setup of this.

Pulling docker images;

docker pull mongodb/mongodb-community-search:latest
docker pull mongodb/mongodb-community-server:latest

Create internal docker network to communicate

docker network create search-community

Starting community server container

echo '
net:
  port: 27017
  bindIpAll: true  # Equivalent to --bind_ip_all

replication:
  replSetName: rs0
systemLog:
  destination: file
  path: "/var/log/mongodb/mongod.log"
  logAppend: true

setParameter:
  searchIndexManagementHostAndPort: mongot-community.search-community:27028
  mongotHost: mongot-community.search-community:27028
  skipAuthenticationToSearchIndexManagementServer: false
  useGrpcForSearch: true

# Security configuration
security:
  authorization: enabled  # Equivalent to --auth
  keyFile: /keyfile' > mongod.conf

mkdir ./data/db

openssl rand -base64 756 > keyfile
chmod 400 keyfile

docker run -d --rm --name mongod -e MONGODB_INITDB_ROOT_USERNAME=root -e MONGODB_INITDB_ROOT_PASSWORD=rootpass -v ./mongod.conf:/etc/mongod.conf:ro -v ./data/db:/data/db -v ./keyfile:/keyfile -p 27017:27017 --network search-community mongodb/mongodb-community-server:latest --config /etc/mongod.conf

Initiate replication

mongosh -u root -p rootpass --eval 'rs.initiate(); sleep(10); rs.status()'
mongosh -u root -p rootpass --eval "db.getSiblingDB('admin').createUser(
 {
   user: 'mongot',
   pwd: 'mongotPassword',
   roles: ['searchCoordinator']
 }
)"

Load sample movies collection data
Downloaded movies from https://github.com/neelabalan/mongodb-sample-dataset/blob/main/sample_mflix/movies.json

mongoimport  -d sample_mflix -c movies /Users/darshanj/Downloads/movies.json -u root -p rootpass --authenticationDatabase admin

Prepare search process

mkdir mongot_data

Creating mongot config file

hostname=`mongosh -u root -p rootpass --eval "db.isMaster().me"`
cat << EOF > mongot.config
syncSource:
  replicaSet:
     hostAndPort: "$hostname" 
     username: "mongot"
     passwordFile: "/passwordFile"
     authSource: "admin"
     tls: false
     readPreference: primaryPreferred
storage:
  dataPath: "/data/mongot"
server:
  grpc:
     address: "mongot-community.search-community:27028"
     tls:
        mode: "disabled"
metrics:
  enabled: true
  address: "mongot-community.search-community:9946"
healthCheck:
  address: "mongot-community.search-community:8080"
logging:
  verbosity: INFO
embedding:
  queryKeyFile: /etc/mongot/voyage-api-query-key
  indexingKeyFile: /etc/mongot/voyage-api-indexing-key
  providerEndpoint: https://api.voyageai.com/v1/embeddings
  isAutoEmbeddingViewWriter: true

echo -n "mongotPassword" > passwordFile
chmod 400 passwordFile

Go to https://dashboard.voyageai.com/organization/api-keys API key, copy secret

printf "<your-voyage-api-query-key>" > voyage-api-query-key

Repeat one more time for index API. Go to https://dashboard.voyageai.com/organization/api-keys API key, copy secret

printf "<your-voyage-api-index-key>" > voyage-api-indexing-key

Starting MongoDB Search

docker run -d --rm --name mongot-community -v ./mongot_data:/data/mongot -v ./mongot.config:/mongot-community/config.default.yml -v ./passwordFile:/passwordFile:ro -v ./voyage-api-indexing-key:/etc/mongot/voyage-api-indexing-key:ro -v ./voyage-api-query-key:/etc/mongot/voyage-api-query-key:ro --network search-community -p 8080:8080 -p 9946:9946 mongodb/mongodb-community-search:latest --internalListAllIndexesForTesting=true

Check the docker status

docker ps

Now connect to the mongosh and create the index and run the test query

mongosh -u root -p rootpass

use sample_mflix
db.movies.createSearchIndex("vector_index", "vectorSearch", {
 "fields": [
   {
     "type": "autoEmbed",
     "modality": "text",
     "path": "plot",
     "model": "voyage-4"
   },
   {
     "type": "filter",
     "path": "year"
   }
 ]
})

db.movies.aggregate([
 {
   "$vectorSearch": {
     "index": "vector_index",
     "path": "plot",
     "query": {
       "text": "bullied boy learns karate"
     },"model": "voyage-4-lite" ,
           "numCandidates": 10000,
     "limit": 10
   }
 },
 {
   "$project": {
     "_id": 0,
     "title": 1,
     "year":1,
     "plot": 1,
     "score": { $meta: "vectorSearchScore" }
   }
 }
])

I know we can make one single Docker compose file to make it easier, but I prefer this way to make debugging easier, and the understanding of each step will be clearer.

Congratulations! You have successfully created your first local vector search.

PS: To be safe, delete the API keys when no longer needed, just like I did :).

How schema anti-patterns in MongoDB can cost you $$$$

MongoDB Guests — Tue, 28 Apr 2026 13:38:52 +0000

This article was written by Darshan Jayarama.

MongoDB is a flexible schema database. This means you have the flexibility to modify the structure of the data store. One collection can have 5 fields in one row(document), 10 in another row, 1000 in another row, and 1 in another row. It doesn't matter how the structure has been aligned in the collection.

If you are a developer with a SQL background, you may find this to be a fantasy, like a science fiction movie. However, that is the flexibility you can achieve with the help of MongoDB. A good percentage of people choose MongoDB for this flexible schema, but let it be known that abusing this feature could cost you big time.

I would like to share some of the story from when I was working with customers to tune the application's performance.

Massive boundless array

Problem: One customer had ping data from an IOT device stored in an array field. In the first few hours of the day, document writes were working fine, but as time passed, writes started to become expensive.

Reason: As the document gets inflated with the growing array field, the findOneAndUpdate() method gets heavy in the network for each document to fetch →update →Save them.

Solution: As the ping frequency is every minute for 1000s of IoT devices, we created a new collection that maps device and ping data to deviceID, hence throughout the day, write was stabilized.

Timeseries saved time

Problem: For the same customer, write was stabilized with a referential model, but each read started costing them time as it had to map the same device ping throughout the day while creating the report.

Reason: Each deviceID data was spread across storage in different pages, so for 100 records, it was technically fetching 100 pages, keeping the required data, and throwing away the rest. Which increased the pagein-pageout.

Solution: MongoDB offers a timeseries collection, which utilizes a bucketing pattern, it has been explained in detail in the video series.

We have created a timeseries collection to store the ping data with deviceID as the metaField and timestamp as the timeField.

When multitenant became the villain

Problem: A customer from a webhosting platform was using a different DB for a different customer with a customer-identification suffix like billing_cust1, data_cust2… Started growing a customer base, became a pain in their database.

Reason: Customer was using MongoDB ChangeStream, which increased the number of databases and caused a delay in the start of their application collections. As explained in the documentation, changestream uses oplog collection in which we cannot utilize the indexes, hence, opening a high number of targeting streams can impact the performance.

Solution: We converted multi-tenancy to a monolithic database structure, storing all customer data we stored in a single collection with customerID in each document. So whenever we query for a specific customer, we append another filter with customerID to perform a targeted query and ensure data safety. This reduced 10000s of collections into 5–10 collections.

$lookup is designed badly

Problem: The application slowed down during the delivery time while verifying the product to be delivered.

Reason: A famous e-commerce company was storing productID in the orders collection. When a scan was performed via the QR code, it was performing multiple lookups in the products collection to fetch the product details.

Solution: Because the number of products was limited, we remodeled the table’s schema and stored the product details in the order collection. As the order details would no longer be accessed regularly after delivery, we archive orders that are 3–4 months old.

Closing thoughts: Performance tuning is a never-ending task, analyze the queries → tune → repeat.

How MongoDB Executes a find() Query: A Complete Lifecycle Guide

MongoDB Guests — Thu, 23 Apr 2026 08:30:25 +0000

This article was written by Darshan Jayarama.

When you type something like db.orders.find({ status: "pending", customerId: 1042 }) and the results come back in milliseconds, it feels simple… almost instant.

But behind that one line, MongoDB is doing a lot more than just “searching a collection.”

During my time as a Senior TSE at MongoDB, I spent most of my days deep in query performance and indexing issues, working closely with both customers and the engineering team. It may look simple, but internally, a lot is happening when a find() executes.

Most engineers understand it at a surface level. But once you start digging into what actually happens under the hood, that’s when things change. You begin to see why some queries are fast, why others are painfully slow, and how indexes truly make or break performance.

So let’s break it down — step by step — what really happens inside MongoDB when a query runs. No shortcuts, nothing skipped.

The full journey of a `find()` query (quick overview)

Before we go step by step, here’s the big picture. Every find() query goes through a series of steps in the mentioned order.

There are also two “fast paths” (shown as green arrows):
• When MongoDB can reuse a plan from the plan cache
• When the data is already in memory (WiredTiger cache)

Getting your queries to use these fast paths is what good MongoDB performance tuning is all about.

Stage 1: The Client Sends a Query Over the Wire

Before MongoDB even sees your query, your driver prepares it.

Whether you’re using PyMongo, the Node.js driver, or mongosh, your query is converted into BSON (a binary version of JSON) and sent over a network connection to MongoDB.

MongoDB uses the Wire Protocol to communicate, which is basically a structured way of sending messages between your app and the database.

Since MongoDB 3.6, most operations use a format called OP_MSG.

This message includes:
• which database and collection you’re querying
• the actual filter (your query conditions)
• extra options like projection, sort, limit, skip, hints, and read preferences

# PyMongo serializes this to BSON OP_MSG internally

db.orders.find(
   { "status": "pending", "customerId": 1042 },
   projection={"_id": 0, "amount": 1},
   sort=[("createdAt", -1)],
   limit=10
)

The query lands at the mongod process. The network listener accepts the connection and hands it off to a worker thread from the connection pool.

Stage 2: Authentication Check

The first thing MongoDB does — even before looking at any data — is check who is making the request.

MongoDB supports different ways to authenticate users, but in most real-world cases, this step is already handled. Drivers usually keep connections open and authenticated, so this check happens almost instantly

MongoDB just verifies the session linked to that connection.

If authentication fails, the query stops right there.
You’ll see an error like: “Authentication failed”, and the query never even reaches the database engine.

Stage 3: Authorization — Can You Read This Collection?

Authentication tells MongoDB who you are. Authorization decides what you’re allowed to do. MongoDB uses role-based access control (RBAC). It checks whether your user has permission to run a find() on that specific database and collection.

This could come from built-in roles like read or readAnyDatabase, or from custom roles with specific permissions.

If you don’t have access, the query stops there. You’ll see an error like: “not authorized to execute command”. The good part? This check is extremely fast — it’s just a quick in-memory lookup, not a database scan.

Stage 4: Query Parsing and BSON Validation

At this point, your query is sitting in memory as a BSON document. Now MongoDB starts understanding it.

It does two main things:

1. Checks if the query is valid

MongoDB makes sure your query is written correctly — valid operators like $eq, $in, $gt, $elemMatch, proper structure, and no invalid combinations. If something is wrong, the query fails right here with an error.

2. Converts it into an internal format:

MongoDB rewrites your query into a standard internal structure (think of it like a tree). This is why the order of fields doesn’t matter.

For example, these two are treated exactly the same:
{ a: 1, b: 2 }
{ b: 2, a: 1 }

Internally, MongoDB sees it more like:

AND
├── status = "pending"
└── customerId = 1042

In short, MongoDB first checks that your query is valid, then rewrites it into a format it can efficiently work with.

Stage 5: The Query Planner Enumerates Candidate Plans

This is where MongoDB actually starts making smart decisions. The query planner takes your parsed query and determines the best way to get the data. It doesn’t just pick one way — it tries out multiple options.

For every useful index, MongoDB creates a possible plan. It also always keeps one backup option: scanning the whole collection (COLLSCAN).

Each plan is basically a step-by-step approach to fetch the data.

For example:

Using index on status:

 FETCH
  └─ scan index { status: 1 }
Using index on customerId:
 FETCH
  └─ scan index { customerId: 1 }
Using compound index (status + customerId):
 scan index { status:1, customerId:1 }
 (no FETCH needed — everything is in the index)
No index:
 scan entire collection

MongoDB also thinks about a few important things here:
• Can it avoid sorting by using an index?
• Can it return results directly from the index (covered query)?
• Can it combine multiple indexes if needed?

In short, MongoDB tries different ways to run your query and prepares multiple plans before choosing the best one.

Stage 6: Plan Cache Lookup — The Fast Path

Before MongoDB tries out different plans, it first checks something called the plan cache.

The plan cache stores the best plan from previous runs of similar queries. So if MongoDB has already seen a query like yours before, it can skip all the extra work and reuse the same plan.

What matters here is the query shape — basically the structure of the query, not the actual values.

For example, these two queries are treated the same:

db.orders.find({ status: "pending", customerId: 1042 })

db.orders.find({ status: "shipped", customerId: 9999 })

Because structurally, they are identical:
{ status: <eq>, customerId: <eq> }

Now two things can happen:
• Cache hit → MongoDB already knows the best plan, so it skips all the trial work and runs it directly (this is the fast path)
• Cache miss → MongoDB doesn’t have a saved plan, so it tries multiple options to find the best one

A few important things to know about the plan cache:
• It’s stored in memory (not on disk)
• It’s maintained per collection
• It gets cleared when MongoDB restarts
• It’s also reset if the indexes change or the collection changes a lot

You can even inspect or clear it manually:

// See cached plans
db.orders.getPlanCache().list()
// Clear cache (forces MongoDB to re-evaluate plans)
db.orders.getPlanCache().clear()

In short, if your query has been seen before, MongoDB can skip straight to execution — which is why repeated queries are usually much faster 🚀

Stage 7: Multi-Plan Trial — The Index Race

If MongoDB doesn’t find a plan in the cache, it tries something really interesting.

Instead of guessing the best plan, it tests all possible plans simultaneously. This is called the multi-plan stage.

Each plan is given a chance to run for a few steps, one after the other — kind of like a race.

For example:
Round 1:
Plan A → small progress
Plan B → small progress
Plan C → small progress

Round 2:
Plan A → more progress
Plan B → more progress
Plan C → more progress

This continues until one plan clearly outperforms the others.

The winner is the plan that:
• returns the first ~100 results fastest, or
• finishes scanning the data quickest

Once a plan wins, MongoDB:
→ uses it for the current query
→ saves it in the plan cache for next time

What about full collection scans (COLLSCAN)?

They usually lose… but not always.

• If there are no useful indexes → COLLSCAN wins
• If the collection is very small → COLLSCAN can actually be faster than using an index

If you want to see how this decision was made, you can run:

db.orders.find({ status: "pending", customerId: 1042 })
        .explain("allPlansExecution")

Stage 8: Index Scan (IXSCAN) or Collection Scan (COLLSCAN)

Now that MongoDB has picked the best plan, it finally executes the query.

If an index is used (IXSCAN)

MongoDB uses indexes that work like a tree structure. It quickly navigates this tree to find matching entries.

Once it finds a match, it uses a reference (called RecordId) to go and fetch the actual document from the collection.

Think of it like:
— find the entry in the index
— then go grab the full document

There’s also something called a covered query: If all the fields you need are already in the index, MongoDB doesn’t even need to look at the actual documents.

It returns results directly from the index
This is the fastest possible way to read data

If no index is used (COLLSCAN)

If there’s no useful index, MongoDB has no choice — it reads every document one by one.

So if your collection has 10 million documents, it will scan all 10 million.

That’s definitely slow.

How to spot a problem

When you run explain(), watch for this:

db.orders.find({ status: "pending" }).explain("executionStats")
// Red flags to look for:
// "stage": "COLLSCAN"          ← no index used
// "totalDocsExamined": 9847321  ← scanned 9.8M docs
// "nReturned": 142              ← returned only 142
// Ratio: 69,000:1               ← extremely inefficient

Red flags:

"stage": "COLLSCAN" → no index is being used
Very high "totalDocsExamined"
Very low "nReturned"
scanned ~9.8 million documents
returned only 142

That’s a huge gap, and a clear sign your query is inefficient

Simple rule
If MongoDB is reading way more documents than it returns, you probably need a better index.

Stage 9: The Storage Layer — WiredTiger Cache and Disk

Once MongoDB knows which documents it needs, the next question is: Where does it actually read the data from?

There are three possible sources, and the speed depends on where the data is found.

1. WiredTiger cache (in-memory — fastest)
MongoDB uses WiredTiger as its storage engine, which keeps frequently accessed data in memory.

By default, it uses about 50% of the available RAM.

If the required data is already in this cache, MongoDB can return it almost instantly. This is the ideal scenario.

2. OS page cache (still fast)
If the data is not in MongoDB’s own cache, it checks the operating system’s page cache.

The OS may already have the data in memory from recent reads. Since MongoDB memory-maps its data files, this check is efficient.

This is slightly slower than the WiredTiger cache, but still very fast.

3. Disk (slowest)
If the data is not present in either cache, MongoDB has to read it from disk.

The performance here depends on the type of storage:

NVMe SSD: fastest among disks
SATA SSD: moderate
HDD: significantly slower

At scale, disk access becomes the main bottleneck, especially when queries access data randomly across large datasets. If your working data set fits in memory, queries remain fast. If MongoDB frequently needs to read from disk, query performance drops significantly. This is why working set size is critical in MongoDB performance tuning — your frequently accessed data should ideally fit in RAM.

Checking cache performance

You can inspect cache behavior using:

db.serverStatus().wiredTiger.cache
// 'pages read into cache'       ← total cache misses (disk reads)
// 'pages requested from cache'  ← total requests
// Hit ratio = 1 - (read/requested)
// Target: > 95%

Key metrics:

pages read into cache → disk reads (cache misses)
pages requested from cache → total requests

A good system typically has a cache hit ratio above 95%.

Stage 10: Results Returned to the Client

Here’s what happens:

Applies projection: Removes any fields you didn’t ask for and keeps only what’s needed
Applies skip and limit: Skips the first N documents (if specified) and limits how many results are returned
Converts back to BSON: Turns the in-memory document into a format that can be sent over the network
Creates response batches: MongoDB doesn’t send everything at once. The first batch contains up to 101 documents or 16MB of data (whichever comes first)
Sends the response: The data is sent back to your application over the same connection

What if there’s more data?

If your query returns more than one batch, MongoDB doesn’t send everything in one go.

Instead, it returns a cursor ID.

Your driver then automatically requests the next batch using getMore commands. This happens behind the scenes, so you usually don’t notice it.

The Complete Picture — All 10 Stages

Closing notes: Key Takeaways for Production

Always check your explain() output - Use explain("executionStats") to see exactly how your query ran — which plan was used, how many documents were scanned vs returned, and whether the plan came from cache.
Design indexes based on query patterns, not just fields -Think about how your queries are written. For example, a compound index like { status: 1, customerId: 1 } is usually much more effective than having separate indexes on each field. It can even avoid extra steps like fetching documents if the query is covered.
Keep an eye on cache efficiency - Monitor your WiredTiger cache hit ratio. If it drops below ~90%, it usually means your frequently accessed data no longer fits in memory. At that point, you may need to add more RAM, shard the data, or rethink how your application accesses data.
Be aware of plan cache issues - If a query suddenly becomes slow after something like a bulk insert, the plan cache might be the cause. MongoDB may have picked a suboptimal plan after re-evaluating. In such cases, clear the cache and check the query again with explain().

// Your daily driver for query diagnostics
db.orders.find({ status: "pending" }).explain("executionStats")

// Key fields to read:
// executionStats.executionTimeMillis    ← total time
// executionStats.totalDocsExamined      ← docs touched
// executionStats.totalKeysExamined      ← index keys scanned
// executionStats.nReturned              ← docs returned
// queryPlanner.winningPlan.stage        ← IXSCAN or COLLSCAN

Introduction to Events using MongoDB

MongoDB Guests — Wed, 22 Apr 2026 21:52:27 +0000

This article was written by Otavio Santana.

Keeping software simple is one of the hardest challenges for any software architect or senior engineer. Not because of technology, but because of decisions we often overlook—especially coupling. Many systems start simple, but as new requirements emerge, the same code begins to orchestrate multiple responsibilities. What once felt clean becomes fragile, and small changes require touching multiple parts of the system. This is how architectures silently degrade. The good news is that there are ways to address this problem. In this tutorial, we will explore an architectural approach that helps reduce coupling and improve extensibility, and how MongoDB can effectively support this style.

In this tutorial, you’ll:

Model a simple payment system
Write events sync and async events
Explore how MongoDB can help you on archive Event-driven design.

You can find all the code presented in this tutorial in the GitHub repository:

git clone git@github.com:soujava/helidon-mongodb-event-driven.git

Prerequisites

For this tutorial, you’ll need:

Java 21.
Maven.
A MongoDB cluster.
MongoDB Atlas (Option 1)
Docker (Option 2)

You can use the following Docker command to start a standalone MongoDB instance:

docker run --rm -d --name mongodb-instance -p 27017:27017 mongo

When we discuss event-driven design, we change the approach or the question: Instead of asking “who should I call next?”, we start asking: “What just happened?”

This change allows modules to interact without direct dependencies. Different parts of the system can react independently—either synchronously or asynchronously—without modifying the original flow.
As a first step, we need to understand what an event is. An event represents a fact that has already happened in the system. It is immutable and carries enough information for other components to react. Instead of telling the system what to do, events describe what occurred, allowing behavior to emerge through reactions.

In Domain-Driven Design (DDD), events take on a more specific role. They represent meaningful events in the business domain, using the same language as the business itself. These domain events can also act as integration points, enabling different parts of the system, including across bounded contexts—or even external systems—to respond to business changes. Event-driven goes beyond DDD, allowing facts on both the technical and the domain side.

In this tutorial, we will create an over-simplified payment system where, given a payment for a product, we will issue a payment request to an external provider, which will start an async event. Based on that, we will generate an event depending on whether the payment succeeded or failed. The whole process starts at the JAX-RS endpoint, generating a sequence of events and states inside the payment.

In this tutorial, we will simplify using CDI events, but you can naturally enhance it or use another service to handle those events, such as Kafka or a JMS Broker. Thus, we will use MongoDB and Helidon. Creating a Helidon project is simple: use the Helidon Starter, and in the wizard progress flow, you will select Helidon MP, QuickStart, and JSON-B. Feel free to define the groupid, artifactId, version, and package name as you wish. After downloading, the next step is including the MongoDB integration, in this case, Eclipse JNoSQL, in the pom.xml file in the root of the downloaded project:

<dependency>
   <groupId>org.eclipse.jnosql.databases</groupId>
   <artifactId>jnosql-mongodb</artifactId>
   <version>1.1.13</version>
</dependency>

With the project defined, the next step is to set the database configuration. You can either use a local database or explore MongoDB Atlas; either is fine. We will use locally, thus, run this Docker command to start a MongoDB instance, include those new properties:

# configure the MongoDB client for a replica set of two nodes
jnosql.mongodb.url=mongodb://localhost:27017
# mandatory: define the database name
jnosql.document.database=ecommerce

Step 1: Create the entities

With the project defined, the next step is to create the entities and repositories that will support it.

In the src/main/java directory, create a Product class:

package com.acme;

import jakarta.nosql.Column;
import jakarta.nosql.Embeddable;

import java.util.UUID;

@Embeddable(Embeddable.EmbeddableType.GROUPING)
public record Product(@Column UUID code, @Column String name) {

}

Create a PaymentCounter class:

package com.acme.statistics;

import com.acme.Product;
import com.acme.infraestructure.JsonFieldStrategy;
import jakarta.json.bind.annotation.JsonbVisibility;
import jakarta.nosql.Column;
import jakarta.nosql.Entity;
import jakarta.nosql.Id;

import java.util.UUID;

@Entity
@JsonbVisibility(JsonFieldStrategy.class)
public class PaymentCounter {

    @Id
    private UUID productCode;

    @Column
    private Product product;

    @Column
    private int successfulPayments;

    @Column
    private int failedPayments;

    PaymentCounter(Product product) {
        this.productCode = product.code();
        this.product = product;
    }

    PaymentCounter() {
    }

    public void paymentFailed() {
        this.failedPayments++;
    }

    public void paymentSucceeded() {
        this.successfulPayments++;
    }

    @Override
    public String toString() {
        return "PaymentCounter{" +
                "productCode=" + productCode +
                ", successfulPayments=" + successfulPayments +
                ", failedPayments=" + failedPayments +
                '}';
    }
}

And the Payment and PaymentStatus classes:

package com.acme.payment;

import com.acme.Product;
import com.acme.infraestructure.JsonFieldStrategy;
import jakarta.json.bind.annotation.JsonbVisibility;
import jakarta.nosql.Column;
import jakarta.nosql.Entity;
import jakarta.nosql.Id;

import java.math.BigDecimal;
import java.util.Objects;
import java.util.UUID;

@Entity
@JsonbVisibility(JsonFieldStrategy.class)
public class Payment {

    @Id
    private UUID id;

    @Column
    private BigDecimal amount;

    @Column
    private PaymentStatus status;

    @Column
    private Product product;

    Payment() {
    }

    Payment(Product product, BigDecimal amount, PaymentStatus status) {
        this.id = UUID.randomUUID();
        this.amount = amount;
        this.status = status;
        this.product = product;
    }

    public UUID getId() {
        return id;
    }

    @Override
    public boolean equals(Object o) {
        if (!(o instanceof Payment payment)) {
            return false;
        }
        return Objects.equals(id, payment.id);
    }

    @Override
    public int hashCode() {
        return Objects.hashCode(id);
    }

    public Product getProduct() {
        return product;
    }

    @Override
    public String toString() {
        return "Payment{" +
                "id='" + id + '\'' +
                ", amount=" + amount +
                ", status=" + status +
                ", product=" + product +
                '}';
    }

    public void failed() {
        this.status = PaymentStatus.FAILED;
    }

    public void confirmed() {
        this.status = PaymentStatus.CONFIRMED;
    }
}


//different file

package com.acme.payment;

public enum PaymentStatus {
    PENDING,
    CONFIRMED,
    FAILED
}

Explanation of annotations:

@entity: Marks the Room class as a database entity for management by Jakarta NoSQL.
@id: Indicates the primary identifier for the entity, uniquely distinguishing each document in the MongoDB collection.
@column: Maps fields (roomNumber, type) for reading from or writing to MongoDB.

With the entities created, the next step is creating the repositories for each entity. In our tutorial, we simplified Product as a Value Object.

package com.acme.payment;

import jakarta.data.repository.BasicRepository;
import jakarta.data.repository.Repository;

import java.util.UUID;

@Repository
public interface PaymentRepository extends BasicRepository<Payment, UUID> {
}

//different file
package com.acme.statistics;

import jakarta.data.repository.BasicRepository;
import jakarta.data.repository.Repository;

import java.util.UUID;

@Repository
public interface PaymentCounterRepository extends BasicRepository<PaymentCounter, UUID> {
}

With both entities and repositories defined, the next step is defining the events that we will handle on this system.

Step 2: Create the events

In this sample, we will have three events to represent three states of the payment: when started (pending) ,because it requires operation on an external payment service; Payment failed; or confirmed based on the response from this external payment provider.

In the src/main/java directory, create a Product class:

package com.acme.payment;

public record PaymentFailedEvent(Payment payment) {
}
//different file
package com.acme.payment;

public record PaymentConfirmedEvent(Payment payment) {
}
//different file
package com.acme.payment;

public record PaymentRequestedEvent(Payment payment) {
}

Step 3: Generate the listeners

At this event, we will have listeners or classes that respond to some facts or events in our system. Using CDI that is pretty simple, we just need to have a parameter with the class to be observed and put either Observes or ObservesAsync to observe synchronous or asynchronous simultaneously.

We will start creating a service to update the status based on both confirmed or failed payment. This service will find the payment by id and then update the status.

In the src/main/java directory, create a PaymentStatusService class:

package com.acme.payment;

import jakarta.enterprise.context.ApplicationScoped;
import jakarta.enterprise.event.Observes;
import jakarta.inject.Inject;

import java.util.logging.Logger;

@ApplicationScoped
class PaymentStatusService {

    private static final Logger LOGGER = Logger.getLogger(PaymentStatusService.class.getName());

    private final PaymentRepository repository;

    @Inject
    PaymentStatusService(PaymentRepository repository) {
        this.repository = repository;
    }

    PaymentStatusService() {
        this.repository = null;
    }

    void payed(@Observes PaymentConfirmedEvent event) {
        LOGGER.info("Payment " + event.payment().getId() + " was payed");
        var payment = repository.findById(event.payment().getId()).orElseThrow();
        payment.confirmed();
        repository.save(payment);
        LOGGER.info("Payment " + payment.getId() + " was confirmed");
    }

    void errorOnPayment(@Observes PaymentFailedEvent event) {
        LOGGER.info("Payment " + event.payment().getId() + " failed");
        var payment = repository.findById(event.payment().getId()).orElseThrow();
        payment.failed();
        repository.save(payment);
        LOGGER.info("Payment " + payment.getId() + " was failed");
    }
}

The next step is the counter that will register success and failure based on the product code.

In the src/main/java directory, create a PaymentCounterService class:

package com.acme.statistics;

import com.acme.Product;
import com.acme.payment.Payment;
import com.acme.payment.PaymentFailedEvent;
import com.acme.payment.PaymentConfirmedEvent;
import jakarta.data.Order;
import jakarta.data.Sort;
import jakarta.data.page.Page;
import jakarta.data.page.PageRequest;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.enterprise.event.Observes;
import jakarta.inject.Inject;

import java.util.List;
import java.util.function.Consumer;
import java.util.logging.Logger;

@ApplicationScoped
public class PaymentCounterService {

    private static final Order<PaymentCounter> PRODUCT_CODE = Order.by(Sort.asc("productCode"));

    private static final Logger LOGGER = Logger.getLogger(PaymentCounterService.class.getName());

    private final PaymentCounterRepository repository;

    @Inject
    PaymentCounterService(PaymentCounterRepository repository) {
        this.repository = repository;
    }

    PaymentCounterService() {
        this.repository = null;
    }

    void success(@Observes PaymentConfirmedEvent event) {
        LOGGER.info("Payment successful, incrementing counter: " + event.payment().getId());
        execute(PaymentCounter::paymentSucceeded, event.payment().getProduct());
    }


    void success(@Observes PaymentFailedEvent event) {
        LOGGER.info("Payment failed, incrementing counter: " + event.payment().getId());
        execute(PaymentCounter::paymentFailed, event.payment().getProduct());
    }

    void execute(Consumer<PaymentCounter> action, Product product) {
        PaymentCounter counter = repository.findById(product.code()).orElseGet(() -> new PaymentCounter(product));
        LOGGER.info("Counter found: " + counter);
        action.accept(counter);
        LOGGER.info("Counter incremented: " + counter);
        repository.save(counter);
    }

    public List<PaymentCounter> findAll(PageRequest pageRequest) {
        LOGGER.info("Finding all counters, page: " + pageRequest.page());
        Page<PaymentCounter> counters = repository.findAll(pageRequest, PRODUCT_CODE);
        return counters.content();
    }
}

Finally, we will have the service that will handle an external provider. To simplify, we will set a wait of 2 seconds, and then based on the counter defined as either success or failure, this one uses an asynchronous event, as you can see in the annotation.

In the src/main/java directory, create a PaymentProvider class:

package com.acme.payment.provider;

import com.acme.payment.PaymentFailedEvent;
import com.acme.payment.PaymentRequestedEvent;
import com.acme.payment.PaymentConfirmedEvent;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.enterprise.event.Event;
import jakarta.enterprise.event.ObservesAsync;
import jakarta.inject.Inject;

import java.util.concurrent.atomic.AtomicLong;
import java.util.logging.Logger;

@ApplicationScoped
class PaymentProvider {

    private static final Logger LOGGER = Logger.getLogger(PaymentProvider.class.getName());

    private final AtomicLong counter = new AtomicLong();

    private final Event<PaymentConfirmedEvent> successEvent;

    private final Event<PaymentFailedEvent> errorEvent;

    @Inject
    PaymentProvider(Event<PaymentConfirmedEvent> successEvent, Event<PaymentFailedEvent> errorEvent) {
        this.successEvent = successEvent;
        this.errorEvent = errorEvent;
    }

    PaymentProvider() {
        this.successEvent = null;
        this.errorEvent = null;
    }

    void receivePayment(@ObservesAsync PaymentRequestedEvent event) {
        LOGGER.info("Processing payment: " + event.payment().getId());
        try {
            Thread.sleep(2000);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
        if(counter.get() % 2 == 0) {
            LOGGER.warning("Payment failed: " + event.payment().getId());
            errorEvent.fire(new PaymentFailedEvent(event.payment()));
        } else {
            LOGGER.info("Payment successful: " + event.payment().getId());
            successEvent.fire(new PaymentConfirmedEvent(event.payment()));
        }
        LOGGER.info("Payment processed: " + event.payment().getId() + " - Confirmation #" + counter.incrementAndGet());
    }
}

Step 4: Generating the rest endpoint

The observers are done, but we need one way to start the whole process. In this case, we will use a rest end-point. This resource will have two methods, one to create a payment and another to paginate the current payments.

At src/main/java, create the PaymentResource class.

package com.acme.payment;

import jakarta.data.page.PageRequest;
import jakarta.inject.Inject;
import jakarta.ws.rs.DefaultValue;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.POST;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.QueryParam;
import jakarta.ws.rs.core.Response;

import java.util.List;

@Path("/payments")
public class PaymentResource {


    private final PaymentService service;

    @Inject
    public PaymentResource(PaymentService service) {
        this.service = service;
    }

    PaymentResource() {
        this.service = null;
    }

    @POST
    public Response create(PaymentRequest request) {
        return Response.accepted(service.create(request)).build();
    }

    @GET
    public List<Payment> getAll(@QueryParam("page") @DefaultValue("1") int page) {
        var pageRequest = PageRequest.ofPage(page);
        return service.findAll(pageRequest);
    }
}

The resource itself needs the PaymentService that will create a payment as pending status, it will start an asynchronous event, and also, it will provide the payment status.

At src/main/java, create the PaymentService class.

package com.acme.payment;

import com.acme.Product;
import jakarta.data.Order;
import jakarta.data.Sort;
import jakarta.data.page.Page;
import jakarta.data.page.PageRequest;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.enterprise.event.Event;
import jakarta.inject.Inject;

import java.math.BigDecimal;
import java.util.List;
import java.util.logging.Logger;

@ApplicationScoped
public class PaymentService {

    private static final Logger LOGGER = Logger.getLogger(PaymentService.class.getName());
    private static final Order<Payment> PAYMENT_ORDER = Order.by(Sort.asc("id"));

    @Inject
    private PaymentRepository repository;

    @Inject
    private Event<PaymentRequestedEvent> event;

    public Payment create(PaymentRequest request) {

        LOGGER.info("Creating payment for " + request.productCode());
        Product product = new Product(
                request.productCode(),
                request.productName()
        );

        BigDecimal total = request.unitPrice()
                .multiply(BigDecimal.valueOf(request.quantity()));

        Payment payment = new Payment(
                product,
                total,
                PaymentStatus.PENDING
        );

        repository.save(payment);
        LOGGER.info("Payment created: " + payment);
        event.fireAsync(new PaymentRequestedEvent(payment));
        return payment;
    }

    public List<Payment> findAll(PageRequest pageRequest) {
        LOGGER.info("Finding all payments, page: " + pageRequest.page());
        Page<Payment> payments = repository.findAll(pageRequest, PAYMENT_ORDER);
        return payments.content();
    }
}

The last resource for today is the one to show the statistics of our payment based on the product code.

At src/main/java, create the PaymentCounterResource class.

package com.acme.statistics;

import com.acme.payment.Payment;
import jakarta.data.page.PageRequest;
import jakarta.inject.Inject;
import jakarta.ws.rs.DefaultValue;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.QueryParam;

import java.util.List;

@Path("/payment-counter")
public class PaymentCounterResource {

    private final PaymentCounterService service;

    @Inject
    public PaymentCounterResource(PaymentCounterService service) {
        this.service = service;
    }

    PaymentCounterResource() {
        this.service = null;
    }

    @GET
    public List<PaymentCounter> getAll(@QueryParam("page") @DefaultValue("1") int page) {
        var pageRequest = PageRequest.ofPage(page);
        return service.findAll(pageRequest);
    }

}

Conclusion

We got it! In this tutorial, we learned a new software architecture pattern and its motivation: the event-driven design, which changes our way of thinking about the system by focusing on the events that occur within it, rather than worrying about orchestrating multiple services and eventually handling coupling. We could explore Domain-Driven design concepts with a lightweight event-driven approach, separating responsibilities, handling asynchronous processes naturally, and keeping the core logic simple and extensible.

Ready to explore the benefits of MongoDB Atlas? Get started now by trying MongoDB Atlas.
Access the source code used in this tutorial.
Any questions? Come chat with us in the MongoDB Community Forum.

References:

Source code

Detective mode: Fixing the MongoDB Aggregation pipeline

MongoDB Guests — Tue, 21 Apr 2026 09:32:31 +0000

This article was written by Darshan Jayarama.

It was a Friday, and I was quickly wrapping up my week, finishing the tasks that were assigned. Suddenly, the board became red — another S1. I was like, not now!!!

The application team started complaining about the API responses from the database, going from 600ms to 17 seconds. Booooom!!! That’s insane. No other choice — since it was production, I had to jump in. The war room was ready. The monitoring dashboard was nothing but a heart patient’s ECG — spikes, timeouts, and the main app was choking.

What's the Mystery?

From the dashboard, we identified a query that was asked by the VP to add a breakdown by region to the dashboard. It was just one more field. But it was ONE.MORE.FIELD.

Innocent query, which was harmless for years:

db.orders.aggregate([
 {
   $match: {
     status: "completed",
     created_at: { $gte: new Date("2025-01-31") }
   }
 },
 {
   $group: {
     _id: "$customer_id",
     total_orders: { $sum: 1 },
     total_revenue: { $sum: "$amount" },
     avg_order_value: { $avg: "$amount" }
   }
 },
 {
   $sort: { total_revenue: -1 }
 },
 {
   $limit: 100
 }
])

Simple, right? top 100 customer revenue dashboard.

Execution time 500–700ms, Docs Examined ~2.5M.

We then added just one more field query:

db.orders.aggregate([
 {
   $match: {
     status: "completed",
     created_at: { $gte: new Date("2024-01-01") }
   }
 },
 {
   $lookup: { 
     from: "customers",
     localField: "customer_id",
     foreignField: "_id",
     as: "customer"
   }
 },
 {
   $unwind: "$customer"
 },
 {
   $group: {
     _id: {
       customer_id: "$customer_id",
       region: "$customer.region"
     },
     total_orders: { $sum: 1 },
     total_revenue: { $sum: "$amount" },
     avg_order_value: { $avg: "$amount" }
   }
 },
 {
   $sort: { total_revenue: -1 }
 },
 {
   $limit: 100
 }
])

Execution time >15000ms, Docs Examined 2.5M X 2.5M, memory exploded.

The Investigation: Agent Darshan reporting

Evidence #1:

MongoDB profiler, setting the status to 2, helps us collect all the queries that are being run on the given database. Set this level only when needed, else you’re gonna blow up the log file:

db.setProfilingLevel(2, { slowms: 100 })
db.system.profile.find().sort({ ts: -1 }).limit(1).pretty()

I could see no mercy in our query, brutally attacking the MongoDB cluster:

{
 "op": "command",
 "ns": "production.orders",
 "command": { /* our aggregation */ },
 "keysExamined": 2489234,      // ✅ Used index on status
 "docsExamined": 2489234,      // But still examined all matched docs
 "numYield": 18943,
 "nreturned": 100,
 "responseLength": 8234,
 "millis": 647,
 "planSummary": "IXSCAN { status: 1 }",  // ✅ Used status index
 "execStats": {
   "stage": "FETCH",
   "filter": {
     "created_at": { "$gte": ISODate("2025-01-31") }  // 👈 Filter applied AFTER index
   },
   "inputStage": {
     "stage": "IXSCAN",
     "keyPattern": { "status": 1 },
     "indexName": "status_1",
     "keysExamined": 2489234,
     "seeks": 1,
     "dupsTested": 0,
     "dupsDropped": 0,
     "direction": "forward"
   },
   "totalDocsExamined": 2489234,
   "totalKeysExamined": 2489234,
   "executionTimeMillisEstimate": 447
 }
}

Red Flags:

DocsExamined = ~2.5M, whereas nReturened 100
PlanSummary: Single key index on boolean status_1.
High yield.

Evidence #2:

Need to review and explain the plan to make the query confess to the brutal attack:

{
 "stages": [
   {
     "$cursor": {
       "queryPlanner": {
         "winningPlan": {
          "stage": "IXSCAN",
           "keyPattern": { "status": 1 },
           "filter": {
             "status": { "$eq": "completed" },
             "created_at": { "$gte": new Date("2025-01-31") }
           }
         }
       },
       "executionStats": {
         "executionTimeMillis": 647,  // $match stage: 647ms
         "totalDocsExamined": 2489234,
         "totalKeysExamined": 2489234,  //
         "nReturned": 2489234  // Passed 2.48M docs to next stage
       }
     }
   },
   {
     "$lookup": {  // 💀💀💀 THE SERIAL KILLER
       "from": "customers",
       "as": "customer",
       "executionTimeMillis": 13934,  // 13.9 seconds just for $lookup!
       "totalDocsExamined": 2489234,  // Looked up 2.48M customers
       "stage": "NESTED_LOOP_JOIN",  // No index on customers._id
       "indexesUsed": []  // 👈 DISASTER
     }
   },
   {
     "$unwind": {
       "executionTimeMillis": 823
     }
   },
   {
     "$group": {
       "executionTimeMillis": 1156,
       "usedDisk": true,  // 💀 Spilled to disk (>100MB memory)
       "spilledRecords": 847293
     }
   },
   {
     "$sort": {
       "executionTimeMillis": 87,
       "sortPattern": { "total_revenue": -1 },
       "usedDisk": false  // Lucky, only 100 docs to sort after $limit
     }
   }
 ],
 "executionTimeMillis": 16847
}

Time breakdown:

$match: 647ms → $lookup: 13.9 seconds → $unwind: 823ms → $group: 1156ms → $sort: 87ms → Total 16.9 seconds

Suspects list

Suspect #1:

Unoptimized index for $match. Even though we had a single index on status:1 and created_at:1, we didn't have a compound index on both.

Motive: Index inefficiency
Proof: was high nReturned to the next stage.

Suspect #2:

The $lookup brutalness, for each customer_id from the previous stage, finding the join on the customers collection with an absent element in indexUsed. Isn’t that brutal?

Motive: just one more field request
Proof: empty indexUsed and 13.9s execution time.

Suspect #3:

$group surrendered with memory hog, spilled ~847k documents to disk as it exceeded 100MB per stage MongoDB limit.

Motive: Grouping 2.4M records with useDisk true
Proof: 847k spilledRecords.

Suspect #4:

We were performing an expensive $lookup operation before we could filter down to 100 records.

Motive: Not thinking about pipeline optimization
Proof: Processing 2.48M docs when we only needed 100

Recapping all mistakes to fix!

We were doing $match: 647ms(inefficient index)→ $lookup: 13.9 seconds(massive 2.4M join) → $unwind: 823ms → $group: 1156ms(memory overflow) → $sort: 87ms → Total 16.9 seconds

We redesigned as,

$match with compound index for efficient filtering
$group by customer_id reduce the dataset
$sort and $limit top 100 customer and we have only 100 records now.
$lookup for the 100 customer data we wanted.
$project to add region information.

Does it fix the issues?

The Solution:

2.5M orders
↓ $match (INDEXED — fast!)
2.48M orders
↓ $group (by customer_id only)
~500K customer groups
↓ $sort + $limit
100 top customers
↓ $lookup (only 100 lookups!) ← ✅ FIXED
100 results with customer data
↓ $project (extract region)
100 final results

The best method of using $lookup is not using it!

db.orders.aggregate([
 {
   $match: {
     status: "completed",
     created_at: { $gte: new Date("2024-01-01") }
   }
 },
 {
   $group: {
     _id: "$customer_id",  // Group BEFORE lookup
     total_orders: { $sum: 1 },
     total_revenue: { $sum: "$amount" },
     avg_order_value: { $avg: "$amount" }
   }
 },
 {
   $sort: { total_revenue: -1 }
 },
 {
   $limit: 100  // Reduce to 100 BEFORE lookup
 },
 {
   $lookup: {  // Now only looking up 100 customers
     from: "customers",
     localField: "_id",
     foreignField: "_id",
     as: "customer"
   }
 },
 {
   $unwind: "$customer"
 },
 {
   $project: {
     customer_id: "$_id",
     region: "$customer.region",
     total_orders: 1,
     total_revenue: 1,
     avg_order_value: 1
   }
 }
])

Curious to know the metrics now? I am excited to display as well:

Execution time 847ms, docsExamined 2.5M, lookup performed 100 documents.

✅ Issue resolved

Root cause: $lookup before aggregation.
Fix: Aggregate first, then join the top 100
Performance: 17s → 0.8s (20x improvement)
Added indexes: (status, created_at)

The Costly Lessons Learned:

One field can kill you. That “simple” region addition cost us $24,000 in downtime.
$lookup is not free. It’s actually very expensive. Treat it like a loaded gun.
Order matters. Aggregate → Limit → Join. Not Join → Aggregate → Limit.
Indexes aren’t optional. They’re the difference between 1.8 seconds and 0.15 seconds.
Explain is your friend. If you’re not running, explain, you’re flying blind.
The 100MB limit is real. When $group says “usedDisk: true”, you’re already in trouble.
Test with production data volumes. 1,000 docs is not the same as 2,500,000 docs.

FastAPI With LangChain and MongoDB

MongoDB Guests — Fri, 17 Apr 2026 08:48:54 +0000

This article was written by Carlos Barboza.

I'm happy to welcome you to this tutorial! This demonstration provides a fully functional example of integrating FastAPI, LangChain, and MongoDB, and I'm eager to share it with all technology enthusiasts.

The value of this guide is that it provides a working example—you can copy and paste the code, and it will run seamlessly without any further configuration. It will also give you a clear, practical overview of the kinds of applications you can build using these powerful tools.

I am developing a suggestion application designed to assist customers globally in generating new ideas. This involves providing personalized recommendations and checking for existing companies that offer comparable services.

Prerequisites

Python, virtualenv basics
Basic understanding of REST APIs
MongoDB basics (collections, documents)
Accounts/keys
- MongoDB Atlas or local MongoDB
- LLM provider API key

Project Architecture

FastAPI is a Python framework for building APIs quickly, efficiently, and with very little code.

It’s known for being fast (as the name implies), easy to use, and perfect for modern applications like AI, microservices, and backend APIs. In this example, it will handle the API queries for the backend.

MongoDB is a NoSQL, document-based database that stores data in flexible JSON-like documents instead of rigid tables like SQL. This makes it fast to develop, easy to scale, and great for evolving data structures. For our case, it will store all the information.

LangChain is a framework for building applications powered by LLMs (like GPT, LLaMA, Ollama, and Mistral) that lets you connect models with tools, memory, external data, and multi-step reasoning. It provides the necessary tools in Python for utilizing AI models.

Project Setup

MongoDB Atlas

The simplest way to obtain a MongoDB instance for this guide is to register on the official site. This will provide you with an M0 instance, which is entirely sufficient for executing this guide.

Docker

Should you not have Docker installed, the official link provides installation instructions based on your operating system.

The docker-compose file is configured to set up a MongoDB container. This container uses the official version 7.0, is named mongodb, and uses the credentials root as the username and example as the password. Furthermore, a volume is defined to ensure data persistence across container reloads.

version: "3.3"
services:
  mongo:
    image: mongo:7.0
    container_name: mongodb
    restart: unless-stopped
    ports:
      - "27017:27017"
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: example
    volumes:
      # Named volume for persistent data
      - mongo_data:/data/db

volumes:
  mongo_data:

Execute the following commands to execute the docker:

docker compose up -d

Finally, you can connect to the MongoDB instance using the connection string. In your case, it’s mongodb://root:example@localhost:27017.

LLMs

Large language models (LLMs) are AI models trained on massive amounts of text that can understand language, generate text, write code, answer questions, and hold conversations like a human.

They don’t think—they predict the next word intelligently based on patterns they've learned. For our case, we are going to use Ollama so you can execute the example on your local machine.
Alternatively, you may choose to use a different LLM, such as OpenAI, though this may incur additional costs.

Ollama installation

Depending on your operating system, you must follow the official instructions. At the end of the day, you should be able to execute Ollama commands in your terminal.

ollama list # Shows all the models available in your terminal
ollama pull llama3.2 # Download the llama3.2 model

Development

Requirements.txt

These are all the libraries needed for the project:

aiohappyeyeballs==2.6.1
aiohttp==3.13.2
aiosignal==1.4.0
annotated-doc==0.0.4
annotated-types==0.7.0
anyio==4.12.0
attrs==25.4.0
bcrypt==3.2.2
certifi==2025.11.12
cffi==2.0.0
charset-normalizer==3.4.4
click==8.3.1
colorama==0.4.6
cryptography==46.0.3
dataclasses-json==0.6.7
distro==1.9.0
dnspython==2.8.0
ecdsa==0.19.1
email-validator==2.3.0
fastapi==0.123.0
frozenlist==1.8.0
greenlet==3.2.4
h11==0.16.0
httpcore==1.0.9
httpx==0.28.1
httpx-sse==0.4.3
idna==3.11
jiter==0.12.0
jsonpatch==1.33
jsonpointer==3.0.0
langchain==1.1.0
langchain-classic==1.0.0
langchain-community==0.4.1
langchain-core==1.1.0
langchain-openai==1.1.0
langchain-text-splitters==1.0.0
langgraph==1.0.4
langgraph-checkpoint==3.0.1
langgraph-prebuilt==1.0.5
langgraph-sdk==0.2.10
langsmith==0.4.49
marshmallow==3.26.1
motor==3.7.1
multidict==6.7.0
mypy_extensions==1.1.0
numpy==2.3.5
openai==2.8.1
orjson==3.11.4
ormsgpack==1.12.0
packaging==25.0
passlib==1.7.4
propcache==0.4.1
pyasn1==0.6.1
pycparser==2.23
pydantic==2.12.5
pydantic-settings==2.12.0
pydantic_core==2.41.5
pymongo==4.15.4
python-dotenv==1.2.1
python-jose==3.5.0
python-multipart==0.0.20
PyYAML==6.0.3
regex==2025.11.3
requests==2.32.5
requests-toolbelt==1.0.0
rsa==4.9.1
six==1.17.0
sniffio==1.3.1
SQLAlchemy==2.0.44
starlette==0.50.0
tenacity==9.1.2
tiktoken==0.12.0
tqdm==4.67.1
typing-inspect==0.9.0
typing-inspection==0.4.2
typing_extensions==4.15.0
urllib3==2.5.0
uvicorn==0.38.0
xxhash==3.6.0
yarl==1.22.0
zstandard==0.25.0

Environment file

It’s recommended to have an .env file with sensitive information like the MongoDB URI, JWT Secret Key, and other settings. This file should be excluded from the git.ignore and must be configured on the main location.

MONGO_URI=mongodb+srv://admin:admin@atlascluster.f3spasq.mongodb.net/
MONGO_DB_NAME=fastapi_demo
JWT_SECRET_KEY=super-secret-key
JWT_ALGORITHM=HS256
ACCESS_TOKEN_EXPIRE_MINUTES=60
OLLAMA_HOST=http://host.docker.internal:11434 # Only if fastAPI is running with docker

.gitignore

# ---------------------
# Environment variables
# ---------------------
.env
.env.local
.env.*.local

# ---------------------
# Python
# ---------------------
__pycache__/
*.pyc
*.pyo
*.pyd
*.pdb
*.pkl
*.db

# Virtual environments
venv/
.venv/

# Byte-compiled
*.py[cod]
*$py.class

# ---------------------
# FastAPI / Uvicorn logs
# ---------------------
*.log

# ---------------------
# IDE & editor files
# ---------------------
.vscode/
.idea/
*.swp

# ---------------------
# OS files
# ---------------------
.DS_Store
Thumbs.db

# ---------------------
# Docker
# ---------------------
/data/
docker-data/
*.pid

# ---------------------
# Optional cache folders
# ---------------------
cache/
logs/

Schema folder

chat.py
from pydantic import BaseModel

class ChatRequest(BaseModel):
    question: str

class ChatResponse(BaseModel):
    answer: str
    used_context: bool

item.py
from pydantic import BaseModel

class Item(BaseModel):
    id: str
    name: str
    price: float

user.py
from pydantic import BaseModel, EmailStr

class UserCreate(BaseModel):
    email: EmailStr
    password: str

class UserPublic(BaseModel):
    id: str
    email: EmailStr

class Token(BaseModel):
    access_token: str
    token_type: str = "bearer"

class UserInDB(BaseModel):
    id: str
    email: EmailStr
    hashed_password: str

notes.py
from pydantic import BaseModel

class NotesCreate(BaseModel):
    text: str

class NotesInDB(BaseModel):
    id: str
    user_id: str
    text: str

Core folder

config.py
import os

MONGO_URI = os.getenv("MONGO_URI", "mongodb+srv://admin:admin@atlascluster.f3spasq.mongodb.net/")
MONGO_DB_NAME = os.getenv("MONGO_DB_NAME", "fastapi_demo")

JWT_SECRET_KEY = os.getenv("JWT_SECRET_KEY", "change-me")
JWT_ALGORITHM = os.getenv("JWT_ALGORITHM", "HS256")
ACCESS_TOKEN_EXPIRE_MINUTES = int(os.getenv("ACCESS_TOKEN_EXPIRE_MINUTES", "60"))

llm.py
import os
from langchain_community.chat_models import ChatOllama

OLLAMA_HOST = os.getenv("OLLAMA_HOST", "http://localhost:11434")

llm = ChatOllama(
    model="llama3.2"
    ,base_url=OLLAMA_HOST,
)

security.py
from datetime import datetime, timedelta, timezone
from typing import Optional

from fastapi.security import OAuth2PasswordBearer
from jose import jwt
from passlib.context import CryptContext

from app.core.config import JWT_SECRET_KEY, JWT_ALGORITHM, ACCESS_TOKEN_EXPIRE_MINUTES

pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto")
oauth2_scheme = OAuth2PasswordBearer(tokenUrl="auth/login")

def hash_password(password: str) -> str:
    return pwd_context.hash(password)

def verify_password(plain_password: str, hashed_password: str) -> bool:
    return pwd_context.verify(plain_password, hashed_password)

def create_access_token(
    data: dict,
    expires_delta: Optional[timedelta] = None,
) -> str:
    to_encode = data.copy()
    expire = datetime.now(timezone.utc) + (
        expires_delta or timedelta(minutes=ACCESS_TOKEN_EXPIRE_MINUTES)
    )
    to_encode.update({"exp": expire})
    encoded_jwt = jwt.encode(to_encode, JWT_SECRET_KEY, algorithm=JWT_ALGORITHM)
    return encoded_jwt

db folder

from motor.motor_asyncio import AsyncIOMotorClient
from app.core import config as core_config

client = AsyncIOMotorClient(core_config.MONGO_URI)
db = client[core_config.MONGO_DB_NAME]

users_collection = db["users"]
notes_collection = db["notes"]
items_collection = db["items"]

depsFolder

auth.py
from typing import Optional, Annotated

from fastapi import Depends, HTTPException, status
from jose import JWTError, jwt

from app.core.config import JWT_SECRET_KEY, JWT_ALGORITHM
from app.core.security import oauth2_scheme, hash_password, verify_password
from app.db.mongo import users_collection
from app.schemas.user import UserInDB, UserCreate

async def get_user_by_email(email: str) -> Optional[UserInDB]:
    doc = await users_collection.find_one({"email": email})
    if not doc:
        return None
    return UserInDB(
        id=str(doc["_id"]),
        email=doc["email"],
        hashed_password=doc["hashed_password"],
    )

async def create_user(user: UserCreate) -> UserInDB:
    existing = await get_user_by_email(user.email)
    if existing:
        raise HTTPException(
            status_code=status.HTTP_400_BAD_REQUEST,
            detail="Email already registered",
        )
    hashed = hash_password(user.password)
    res = await users_collection.insert_one(
        {"email": user.email, "hashed_password": hashed}
    )
    return UserInDB(id=str(res.inserted_id), email=user.email, hashed_password=hashed)

async def get_current_user(
    token: Annotated[str, Depends(oauth2_scheme)]
) -> UserInDB:
    credentials_exception = HTTPException(
        status_code=status.HTTP_401_UNAUTHORIZED,
        detail="Could not validate credentials",
        headers={"WWW-Authenticate": "Bearer"},
    )
    try:
        payload = jwt.decode(token, JWT_SECRET_KEY, algorithms=[JWT_ALGORITHM])
        email: str = payload.get("sub")
        if email is None:
            raise credentials_exception
    except JWTError:
        raise credentials_exception

    user = await get_user_by_email(email)
    if user is None:
        raise credentials_exception
    return user

Routers folder

auth.py
from typing import Annotated

from fastapi import APIRouter, Depends, HTTPException, status
from fastapi.security import OAuth2PasswordRequestForm

from app.core.security import create_access_token, verify_password
from app.deps.auth import create_user, get_user_by_email, get_current_user
from app.schemas.user import UserCreate, UserPublic, Token, UserInDB

router = APIRouter(prefix="/auth", tags=["auth"])

@router.post("/register", response_model=UserPublic)
async def register(user: UserCreate):
    new_user = await create_user(user)
    return UserPublic(id=new_user.id, email=new_user.email)

@router.post("/login", response_model=Token)
async def login(form_data: Annotated[OAuth2PasswordRequestForm, Depends()]):
    # I am using email as the username field in the OAuth2 form.
    user = await get_user_by_email(form_data.username)
    if not user or not verify_password(form_data.password, user.hashed_password):
        raise HTTPException(
            status_code=status.HTTP_401_UNAUTHORIZED,
            detail="Incorrect email or password",
        )
    access_token = create_access_token(data={"sub": user.email})
    return Token(access_token=access_token)

@router.get("/me", response_model=UserPublic)
async def get_me(
    current_user: Annotated[UserInDB, Depends(get_current_user)]
):
    return UserPublic(id=current_user.id, email=current_user.email)

chat.py
from typing import Annotated

from fastapi import APIRouter, Depends
from langchain_core.messages import HumanMessage, SystemMessage

from app.core.llm import llm
from app.db.mongo import notes_collection
from app.deps.auth import get_current_user
from app.schemas.user import UserInDB
from app.schemas.chat import ChatRequest, ChatResponse

router = APIRouter(prefix="/chat", tags=["chat"])

@router.post("/", response_model=ChatResponse)
async def chat_with_ai(
    body: ChatRequest,
    current_user: Annotated[UserInDB, Depends(get_current_user)],
    # Uses LangChain + OpenAI + some context from MongoDB.
):
    print(current_user)

    notes_cursor = (
        notes_collection.find({"user_id": current_user.id})
        .sort("_id", -1)
        .limit(3)
    )
    notes = await notes_cursor.to_list(length=3)

    context_text = "\n".join(
        note.get("text", "") for note in notes if note.get("text")
    )
    used_context = bool(context_text)

    system_prompt = (
        "You are an assistant helping the user with their notes."
        " "
        "Use the following context if relevant; if not, just answer normally.\n\n"
        f"Context:\n{context_text or '(no context available)'}"
    )
    messages = [
        SystemMessage(content=system_prompt),
        HumanMessage(content=body.question),
    ]

    response = await llm.ainvoke(messages)

    return ChatResponse(answer=response.content, used_context=used_context)

items.py
from typing import List, Annotated

from fastapi import APIRouter, Depends

from app.db.mongo import items_collection
from app.schemas.item import Item
from app.schemas.user import UserInDB
from app.deps.auth import get_current_user

router = APIRouter(prefix="/items", tags=["items"])

@router.get("/", response_model=List[Item])
async def list_items(
    current_user: Annotated[UserInDB, Depends(get_current_user)]
):
    # Example async Mongo query using Motor.
    # Demo: seed items if collection is empty
    count = await items_collection.count_documents({})
    if count == 0:
        await items_collection.insert_many(
            [
                {"name": "Book", "price": 10.5},
                {"name": "Laptop", "price": 999.99},
            ]
        )

    docs = await items_collection.find().to_list(length=100)
    return [
        Item(id=str(doc["_id"]), name=doc["name"], price=float(doc["price"]))
        for doc in docs
    ]

notes.py
from typing import Annotated

from fastapi import APIRouter, Depends
from langchain_core.messages import HumanMessage, SystemMessage

from app.core.llm import llm
from app.db.mongo import notes_collection
from app.deps.auth import get_current_user
from app.schemas.user import UserInDB
from app.schemas.chat import ChatRequest, ChatResponse
from app.schemas.notes import NotesCreate, NotesInDB

router = APIRouter(prefix="/notes", tags=["notes"])

@router.post("/", response_model=NotesInDB)
async def create_note(body: NotesCreate,current_user: Annotated[UserInDB, Depends(get_current_user)]):
    print(current_user)

    note = await notes_collection.insert_one({"text": body.text, "user_id": current_user.id})
    return NotesInDB(id=str(note.inserted_id), text=body.text, user_id=current_user.id)

Main.py

from fastapi import FastAPI

from app.routers import auth, items, chat, notes

app = FastAPI(title="FastAPI + MongoDB + JWT + LangChain Example")

@app.get("/")
async def root():
    return {"message": "FastAPI + MongoDB + JWT + LangChain example"}

# Include routers
app.include_router(auth.router)
app.include_router(items.router)
app.include_router(chat.router)
app.include_router(notes.router)

Application execution

On the main folder, you can execute the following command:

uvicorn app.main:app --reload
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)

If you access the page http://127.0.0.1:8000/docs#, you can find the Swagger documentation created by default.

Users creation

You can execute the following command to create a new user:

curl -X 'POST' \
  'http://127.0.0.1:8000/auth/register' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "email": "user@example.com",
  "password": "string" 
}'

Or if you have vscode Rest Api Extension you can execute
POST http://127.0.0.1:8000/auth/register
Accept: application/json
Content-Type: application/json

{
  "email": "user@example.com",
  "password": "string"
}

The above command will create a new user with hashed password:

Users authentication

In order to authenticate the user and retrieve the bearer token, please execute the following command:

curl -X 'POST' \
  'http://127.0.0.1:8000/auth/login' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'grant_type=password&username=user%40example.com&password=string&scope=&client_id=string&client_secret=string'
Rest API extension
POST http://127.0.0.1:8000/auth/login
Content-Type: application/x-www-form-urlencoded
accept: application/json

grant_type=password&username=test@example.com&password=123&scope=&client_id=string&client_secret=string

After a successful authentication, you will receive the following output. The token is important to authenticate on the remaining routes.

HTTP/1.1 200 OK
date: Tue, 02 Dec 2025 17:40:11 GMT
server: uvicorn
content-length: 180
content-type: application/json
Connection: close

{
  "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ0ZXN0QGV4YW1wbGUuY29tIiwiZXhwIjoxNzY0NzAwODExfQ.riXj1vvLZSyzyU4T7kG3ygYaUU8sgvgCjK_Os_uggvo",
  "token_type": "bearer"
}

Idea creation

The main purpose of the application is to make suggestions based on ideas that the end users have. The following request will create those notes.

POST http://127.0.0.1:8000/notes/
Content-Type: application/json
Authorization: Bearer {{token}}

{
  "text": "testing123"
}

Chat interaction

The application now offers advice based on the notes stored by users in the database. To enable interaction with the LLM, we define the chat's behavior in the chat.py file. This definition is an example of prompt engineering, and this is the section where you can modify the prompt's language and content, if needed.

system_prompt = (
        "You are an assistant helping the user with their notes."
        " "
        "Use the following context if relevant; if not, just answer normally.\n\n"
        f"Context:\n{context_text or '(no context available)'}"
    )

Execution example:

POST http://127.0.0.1:8000/chat/
Content-Type: application/json
Authorization: Bearer {{token}}

{
  "question": "What i need to do"
}

Execution output:

{
  "answer": "To help you achieve your goals of saving more money and finishing your project, here are some actionable steps:\n\n**Saving More Money:**\n\n1. **Track your expenses**: Write down every single transaction, no matter how small, for a week or two to understand where your money is going.\n2. **Create a budget**: Based on your income and expenses, allocate a specific amount for savings each month.\n3. **Automate your savings**: Set up an automatic transfer from your checking account to your savings account.\n4. **Cut back on unnecessary expenses**: Identify areas where you can cut back on unnecessary spending, such as dining out or subscription services.\n5. **Consider a savings challenge**: Try a savings challenge like the \"52-week savings challenge\" where you save an amount equal to the number of the week (e.g., Week 1: Save $1, Week 2: Save $2 etc.).\n\n**Finishing Your Project:**\n\n1. **Break down your project into smaller tasks**: Divide your project into manageable tasks to make it feel less overwhelming.\n2. **Create a schedule**: Set a specific deadline for each task and stick to it.\n3. **Prioritize your tasks**: Focus on the most critical tasks first, and then move on to less important ones.\n4. **Remove distractions**: Identify potential distractions (e.g., social media, email, phone notifications) and eliminate them while you work.\n5. **Take regular breaks**: Take short breaks to recharge and maintain productivity.\n\nWhich of these steps do you want to start with?",
"used_context": true
}

Docker (optional)

Our application is currently functional. However, to execute it in a different environment, we must install the necessary dependencies. Docker can facilitate and streamline this deployment process.

Create Dockerfile and paste the following code:

# Dockerfile
FROM python:3.11-slim

# Set work directory inside the container
WORKDIR /app

# Install system dependencies (optional but useful for many libs)
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    && rm -rf /var/lib/apt/lists/*

# Copy only requirements first (for better Docker cache)
COPY requirements.txt .

# Install Python dependencies
RUN pip install --no-cache-dir -r requirements.txt

# Copy the rest of your project
COPY . .

# Expose the FastAPI port
EXPOSE 8000

# Default command: run uvicorn
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

Create the docker-compose.yml:

version: "3.8"
services:
  fastapi:
    build: .
    container_name: fastapi-langchain
    ports:
      - "8000:8000"
    env_file:
      - .env # optional: if you have one
    extra_hosts:
      - "host.docker.internal:host-gateway"
    restart: unless-stopped

Now, we can build and execute our docker:

docker-compose build
docker-compose up -d

Conclusion

This tutorial has provided a complete, practical demonstration of building a modern, data-driven application by seamlessly integrating FastAPI, LangChain, and MongoDB.

We have successfully covered:

FastAPI for creating a robust, fast, and documented REST API, including user registration and JWT-based authentication.
MongoDB (using Motor) for asynchronous and scalable document storage, handling user data and application-specific notes.
LangChain (with Ollama) for enabling powerful AI capabilities, using the stored user notes as dynamic context to generate personalized, helpful suggestions and answers.

The example application, an idea generation assistant, serves as a clear blueprint for how to leverage this stack for complex projects requiring data persistence, secure user access, and cutting-edge large language model integration. By using a local LLM via Ollama, we also ensured the entire stack is executable on a local machine, making it accessible for rapid prototyping and development.

This integration strategy—combining the speed of FastAPI, the flexibility of MongoDB, and the intelligence of LangChain—is a highly effective pattern for developing the next generation of intelligent web applications. We hope this guide serves as a solid foundation for your future AI-powered projects.

Async PyMongo in FastAPI

MongoDB Guests — Tue, 14 Apr 2026 14:53:07 +0000

This tutorial was written by Gourav Bhabesh.

1. Introduction

Why modern APIs need asynchronous I/O

Modern APIs need to be robust and be able to serve calls on the go, and minimize the instances of timeouts and server overloads while providing a good user experience during long-running jobs. Asynchronous I/O can solve one of the key issues making modern APIs: staying fast and reliable. It lets the user know the progress of their task while the API handles the requests in the background and is available for other requests. One of the key challenges in the API calls to the database is that some queries can be returned in a few milliseconds but others can take seconds (even with well indexed queries). Without asynchronous I/O in the API, this can lead to long request queues.

You can imagine this as a bakery storefront serving different goods at the same time to multiple customers. We want the orders to be filled asap without the queue becoming too big. In our example, we have multiple servers (scalable infrastructure) who take the orders which can take varying time to process and each customer with longer order times is given a status bar to know where their order’s current status. So a customer with an order that can be instantly filled (picking up a cookie) does not have to wait behind such a customer with a long processing order (baking a cheese cake).

FastAPI’s design philosophy around async

The building block of the FastAPI is ASGI (Asynchronous Server Gateway Interface) standard, unlike the older WSGI (Web Server Gateway Interface) which was synchronous in nature by default. In short, it uses Python’s async/await syntax, yielding control to the event loop, allowing servers to process other requests in the meantime, rather than blocking an entire thread. Using an event loop lets FastAPI handle thousands of concurrent requests with fewer resources. This leads to higher throughput per thread and lower latency. This automatically encourages the developer to use the async ecosystem to fully leverage the performance benefits of async programming.

Database bottlenecks & why async DB drivers matter

One of the key challenges that any database faces is the uncertainty in response times, in which the queries will be answered by the database server. Some queries can take milli-seconds and some can take seconds. If the architecture is not well built to handle this uncertainty, it can cause severe database bottlenecks. To handle this challenge, async database drivers ensure that longer requests do not become a bottleneck for other incoming requests. Initially, the async MongoDB driver was built in the library Motor. But in late 2023, Pymongo team officially decided to build the native async solution, with the goal to replace Motor as the long-term solution.

Introducing the new Async API in PyMongo

The Pymongo team has been working on the goal to replace the Motor library to give native async support in the pymongo library itself. This gives confidence on the client side to use the official library with async support. The async support was first provided in the version 4.9 in Pymongo and it has been further stabilised to the current version of 4.15.0. The new API capabilities bring the asyncio support directly to the official MongoDB driver, allowing developers to use awaitable database operations without external libraries like Motor. Combined with the FastAPI framework, which already supports await/async capability, Pymongo can now be natively used to build high-throughput, non-blocking services.

2. What is PyMongo?

Overview of the official MongoDB Python driver

MongoDB has provided Pymongo as the official Python driver to allow the applications to communicate with MongoDB clusters, providing production-tested API to perform CRUD operations, manage indexes, run aggregation pipelines and handle advanced features such as transactions and change streams. Pymongo was traditionally built on a synchronous I/O model, meaning every database call made will block the executing thread until the MongoDB server does not return a response leading to database bottlenecks.

Traditional synchronous PyMongo behavior

The key behaviours of the traditional Pymongo library, which we should be aware of, to appreciate the addition of the native support of asynchronous, are that synchronous operations block Python threads until the operation completes or a network error or timeout occurs.. This leads to another behaviour - sequential execution, blocking all requests until and unless the previous request is not complete. This made the MongoClient “Thread-Safe” allowing for multi-threading applications to be written, but each thread still performs blocking database operations.

In our example, of the baking shop, it would be like each server is concurrently serving the customers but each server is waiting for the order to complete rather than handing a token to wait, while serving the next customer request which might be served in much less time.

Blocking vs non-blocking I/O explained simply

Blocking I/O means the application must wait for the database operation to finish or network timeout before it will allow anything else to continue. On the other hand, non-blocking I/O allows requests to continue flowing to the database while previous requests are still being processed. In addition, the async drivers provide a way to receive the status of the request, giving power to the developer to handle long operations and create a better user-experience.

Limitations of synchronous DB operations in async frameworks

Using synchronous DB operations in the async framework is one of the key mistakes made by many clients of MongoDB, part of the reason was the non-availability of official Python drivers that actually supported asynchronous behavior. If the synchronous Pymongo driver is used inside an async framework such as FastAPI, these blocking calls freeze the event loop, preventing the server from processing other incoming requests and significantly reducing concurrency. As a result, applications that rely on synchronous database drivers inside async frameworks often experience increased latency, request queuing, and poor scalability under load, even if the rest of the code is written using async and await.

Imagine our baking shop upgraded its entire framework to be non-synchronous. The cooks can cook multiple dishes at the same time, but the servers are still the same, processing sequential orders and not taking new orders until the previous order has been delivered. The cook is having an easy day!!

3. What is Async PyMongo?

Overview of PyMongo’s new async API (`pymongo.async_`)

The official MongoDB team started working on the native-async support in the Pymongo driver in version4.9, where it was first released as beta version in late 2023. As of writing this article, the pymongo version 4.15 has been released and has been stable since version 4.13. With the pymongo.With async support, we can create an AsyncMongoClient object similar to the MongoClient object we created in the previous synchronous call. The Client provides fully awaitable database operations such as insert_one, find_one, update_one, and aggregation pipelines, allowing MongoDB I/O to integrate natively with Python’s event loop.

Key advantages over synchronous API

Compared to the traditional synchronous API, async PyMongo significantly improves scalability and responsiveness under concurrent workloads. Because database calls no longer block the event loop, asynchronous framework applications can continue handling new requests while waiting for MongoDB responses, resulting in lower latency, higher throughput, and better CPU utilization—especially in high-traffic microservices

Relationship to Motor (Motor’s role before async PyMongo existed)

Before async Pymongo, motor was the de facto library to be used when developers needed asynchronous behavior when communicating with the MongoDB clusters. Motor did this by exposing a coroutine-based API, allowing applications to switch context and perform other tasks while waiting for I/O operations. However, Motor was not a separate driver from scratch; it was designed as a wrapper around the core synchronous PyMongo library. It used Tornado’s or asyncio’s event loop, wrapping Pymongo’s operations asynchronously, effectively allowing the main event loop to remain unblocked. Motor was specifically created to support asynchronous web frameworks, initially Tornado and later adding support for asyncio. Performance comparison between motor and async pymongo can be found on the official documentation here.

How to migrate from motor to async-pymongo and sync-pymongo to async-pymongo

Migrating from the motor-based asynchronous library to the async-pymongo library is simple and is well documented by MongoDB here. In simple terms, one needs to change the driver call to the database using the AsyncMongoClient and add the keyword await in front of the database operations.

When migrating from sync Pymongo to async Pymongo, keep the following important points from the official MongoDB documentation in mind:
· To convert an AsyncCursor to a list, you must use the asynchronous cursor.to_list() method.
· The AsyncCollection.find() method in the PyMongo Async API is synchronous, but returns an AsyncCursor.
To iterate through the cursor, you must use an async for loop.
· The AsyncMongoClient object does not support the connect keyword argument.
· You cannot share AsyncMongoClient objects across threads or event loops.
· To access a property or method of a result returned by an asynchronous call, you must properly wrap the call in parentheses, as shown in the following example:

id = (await posts.insert_one(doc)).inserted_id

4. Key Features of Async PyMongo

True async interface with awaitable operations

The biggest advantage of the async Pymongo library for developers is its awaitable operations. Functions such as insert_one(), find_one(), update_one() and aggregate() return objects that can wait rather than blocking the results. This fundamentally decouples the event loop from the operation, meaning the database operation running on the database side yields the control back to the event loop, allowing FastAPI to process the other incoming requests without blocking the thread.

In our baking shop example, the server has been decoupled from the chef, while the previous order is getting cooked in the kitchen, the server takes in a new order without ceasing the operation and blocking the thread and allowing for incoming requests to be fulfilled !! This model of working is fundamentally different from the traditional synchronous PyMongo, where every database call ties up a worker thread until completion, blocking the thread from serving the next call when under load.

Connection pooling behavior

The connection pool behaviour is quite an interesting one here. On the surface it appears both the synchronous and the asynchronous Pymongo connection-poling implementation look the same (parameters available - maxPoolSize, minPoolSize, waitQueueTimeoutMS etc.) and indeed rely on the same MongoDB C driver. However, async applications are actually able to spawn thousands of concurrent operations and grab many connections from the pool at once, whereas, in the synchronous application each request occupies a python worker thread, holding a connection from the pool waiting for MongoDB. This naturally limits concurrency as the application is limited by the number of workers, effectively meaning the full pool size can be never used optimally.

Imagining in our world of baking shop, we have an army of chefs to process the orders, the async model allows the server to grab hold of any available chef to accept a new order and move on to the next one, whereas in the synchronous setup the server will wait for the chef to finish the previous order before moving to the new order.

Async cursors and non-blocking iteration

One of the primary challenges that any database faces is returning results to query with a large result set. The traditional PyMongo driver already returns a cursor object rather than loading all the documents into the memory at once. In async PyMongo, these cursors are fully asynchronous in nature as well, resulting in iteration without blocking the event loop. Developers now have the choice of using the await cursor.to_list() or iterating over the cursor object using the async for. This will allow data to be streamed in batches while the application is handling incoming requests. This feature is of significant importance when we have endpoints that return large datasets, background processing jobs and real-time analytics workloads.

5. What is FastAPI?

FastAPI’s async-first architecture

FastAPI is a high-performance web framework that is production-ready and designed for building APIs in python, with roots embedded in asynchronous programming. It embraces Python’s asyncio model as its core principle rather than treating it as optional. This design choice gives FastAPI leverage for I/O bound workloads such as db access, external API calls and real-time data streaming.

Other traditional web frameworks default to synchronous request handling, whereas FastAPI encourages using asynchronous endpoints (async def) wherever possible. When an endpoint is declared using the async keyword, FastAPI uses the event loop execution rather than a worker thread. Now as long as those requests involve the non-blocking I/O operations, a single worker process can handle many requests concurrently. This is where FastAPI’s async-first architecture shines. When the entire request path - including the database access - is non blocking, each request becomes super efficient. Any stack in the request that blocks the event loop will make the FastAPI’s concurrency model diminish significantly.

Event loop and concurrency model

At the heart of FastAPI’s concurrency model is Python’s asyncio event loop. The event loop manages the execution of many coroutines, switching between them whenever one is waiting for I/O, such as a network response from MongoDB. This cooperative multitasking enables a single process to efficiently handle thousands of simultaneous requests without spawning thousands of threads. However, this model only works as intended when all long-running operations are awaitable. If a synchronous operation blocks the event loop — for example, a traditional synchronous PyMongo database call — the entire server becomes unresponsive to other requests until that operation completes.

Why FastAPI pairs naturally with non-blocking database drivers

As explained above the core principle of FastAPI is to yield the control back to the event loop and any blocking I/O operation in the chain of the operations sent by the request will be like throwing a wrench on the gears. With the pairing of the non-blocking db driver such as the async pymongo, when a FastAPI endpoint awaits a MongoDB operation, control is yielded back to the event loop, allowing for the next set of requests to be processed in parallel. With the alignment between the db driver and the base principle of FastAPI to be async, we eliminate the friction and make the full request truly asynchronous in nature and hence enable FastAPI to scale efficiently under high concurrency.

Surprising performance differences when mixing sync DB code into async routes

A common pitfall for developers writing code in FastAPI is mixing the synchronous PyMongo calls inside the async def endpoints. On the surface it may appear to work, but it breaks the core principle of the FastAPI to be truly asynchronous. One single blocking db call and the entire event loop gets stalled as other requests will have to queue behind it - even when the server has many workers available.

In practice, this means that an API written with FastAPI but using synchronous PyMongo can perform worse under load than a fully synchronous application with multiple worker threads. By contrast, pairing FastAPI with Async PyMongo preserves the framework’s intended concurrency benefits, leading to better throughput, lower latency for typical requests, and more predictable scaling behavior. In large applications, having the capability to predict the scalability of infrastructure and have a control on the concurrency is of paramount importance.

6. Why Use Async PyMongo with FastAPI?

Avoid event-loop blocking

Using FastAPI with Asynchronous Pymongo correctly is essential if we want to avoid event-loop blocking. The requests from the async-first framework like FastAPI, share the same event-loop within each worker process. Meaning any operation in this request if becomes synchronized will lead to blocking of the entire event-loop, which high-concurrency systems want to avoid at all cost.

By using Async Pymongo, each db operation becomes awaitable rather than blocking in nature, ensuring a slow db operation does not freeze the entire application, allowing other requests to get processed.

Handle thousands of concurrent requests

FastAPI is capable of handling thousands of concurrent requests per worker process, but designing the entire chain of operations in an asynchronous manner is a must to achieve this. One of the primary mistakes that are made in the coding side is missing the async and await keywords in the db operation call, this breaks the core principle of FastAPI leading to slowness. With correct implementation we can make the infrastructure more cost-effective and efficient as compared to a purely synchronous system which would not be scalable in the long run as more threads are required to handle the huge volume of requests.

Reduce latency under load

With modern tech stacks, both horizontal and vertical scaling is a must and systems built with such capabilities can withstand the test of time. And today's system should be capable enough to handle varying loads. With FastAPI - async nature is built in-core principle of the library itself and when combined with Async Pymongo it allows for the potential blocking db operations to be handled asynchronously. This leads to lower median latency for requests, especially I/O intensive ones. In section 8, we will see how the sync/async jobs specifically prove this point, when we compare the median latency of thousands of concurrent requests.

Improve throughput for high-traffic microservices

In the world of microservices where responsiveness and scalability are critical Async Pymongo dramatically improves the performance by removing the blocker of such requests to the db layer. Combined with async frameworks such as FastAPI, Async Pymongo is superbly placed to serve real-time applications, analytics dashboards or user-facing services.

Better CPU utilization and scaling on limited infrastructure

One of the vital nerves of any IT infrastructure is cost-efficiency and with Devops team’s mantra - “doing more with less”, fits perfectly with the Async Pymongo. Making the most of the available resources, async architecture uses the same thread without blocking incoming requests by handing over the control back to the single event loop. Instead of maintaining hundreds or thousands of idle threads waiting on the I/O, FastAPI with Async Pymongo exploits this single event loop allowing it to switch between tasks as needed. In cloud based infrastructure, where compute resources are directly tied to cost, it can lead to meaningful savings without sacrificing performance.

7. Async CRUD Operations with Examples

This section demonstrates the common MongoDB operations asynchronously using the pymongo.async_ inside FastAPI.

All examples assume:

from pymongo.async_ import AsyncMongoClient 
from bson import ObjectId 
client = AsyncMongoClient (MONGO_URI) db = client["appdb"] 
users = db["users"]

Insert one / insert many (async versions)

Insert one document example:

@app.post("/users") 
async def create_user(user: dict):
  result = await users.insert_one (user) 
  return {"inserted_id": str(result.inserted_id)}

Insert many documents example:

@app.post("/users/bulk") 
async def create_many(users_in: list[dict]): 
  result = await users.insert_many (users_in) 
  return {"inserted_ids": [str(i) for i in result.inserted_ids]}

Find one / find many with async cursors

Find one document example:

@app.get("/users/{email}")
async def get_user(email: str): 
 return await users.find_one ({"email": email})

Find many documents example:

@app.get("/users") 
async def list_users():
  cursor = users.find ({}) 
  return await cursor.to_list(length=100)

Async pagination using cursors

@app.get("/users/page") 
async def paginated_users(last_id: str | None = None, limit: int = 20): 
  query = {"_id": {"$gt": ObjectId(last_id)}} if last_id else {} 
  cursor = users.find(query).sort("_id", 1).limit (limit) 
  results = await cursor.to_list(length=limit) 

  return { 
    "results": results,
    "next_cursor": str(results[-1]["_id"]) if results else None, 
  }

(Avoid skip() for large collections; use _id cursor pagination instead.)

Update and replace operations

Update one document example:

@app.patch("/users/{email}")
async def update_user(email: str, changes: dict):
  result = await users.update_one(
    {"email": email},
    {"$set": changes},
  ) 
  return {"matched": result.matched_count, "modified":
result.modified_count}

Replace one document example:

@app.put("/users/{email}")
async def replace_user(email: str, new_doc: dict):
  result = await users.replace_one({"email": email}, new_doc) 
  return {"matched": result.matched_count}

Delete one/many operations

Delete one document example:

@app.delete("/users/{email}") 
async def delete_user(email: str):
  result = await users.delete_one ({"email": email}) 
  return {"deleted": result.deleted_count}

Delete many documents example:

@app.delete("/users/inactive")
async def delete_inactive():
  result = await users.delete_many ({"active": False}) 
  return {"deleted": result.deleted_count}

Async aggregation pipelines

@app.get("/users/stats")
async def user_stats():
  pipeline = [
    {"$group": {"_id": "$country", ""count"": {"$sum": 1}}},
    {"$sort": {"count": -1}},
  ] 
cursor = await users.aggregate (pipeline)
return await cursor.to_list(length=100)

Error handling (duplicate keys, timeouts, network issues)

Duplicate keys:

from pymongo.errors import DuplicateKeyError
from fastapi import HTTPException

try: 
  await users.insert_one({"email": "a@b.com"})
except DuplicateKeyError:
  raise HTTPException(status_code=400, detail="User already exists")

Timeout/network issues:

from pymongo.errors import ExecutionTimeout, ConnectionFailure

try:
  await users.find_one({"email": "a@b.com"})
except ExecutionTimeout:
  raise HTTPException(status_code=504, detail="DB timeout")
except ConnectionFailure:
  raise HTTPException(status_code=503, detail="DB unreachable")

8. Performance Comparison: Sync PyMongo vs Async PyMongo

In this section, we break down the high-stakes battle between the traditional Synchronous (Sync) driver and the modern Asynchronous (Async) driver using the large-scale sample_airbnb dataset. This isn't just a syntax preference; it's the difference between a server that remains responsive under fire and one that collapses under its own weight.

Example endpoints and profiling results

To push these drivers to the limit, we implemented a Complex Aggregation task. Instead of a simple "Ping," we forced each request to filter high-rated Airbnb listings, sort them by price, and project complex nested fields.

Sync Logic: Utilizes list(users.aggregate(pipeline)), which occupies a physical thread for the duration of the database round-trip and data serialization.

Async Logic: Utilizes await (await users.aggregate(pipeline)).to_list().
The server yields control during the database "wait time," allowing the single-threaded event loop to process other incoming users.

Locust Benchmark setup

We subjected both implementations to a "Trial by Fire" using Locust:

User Load: Scaled up to 500 concurrent users.

Hardware Constraint: We intentionally limited the Sync thread pool to 1 thread to simulate extreme resource exhaustion and observe the "Queueing Effect."

Target: listingsAndReviews collection in MongoDB Atlas.

Understanding event-loop blocking

The results highlight a fundamental architectural divide. In the Sync model, when the thread pool is exhausted (in our case, limited to 1), every new user must wait for the person in front of them to finish their entire database operation. This creates a "Single Lane Bridge" bottleneck. In the Async model, even with thousands of users, the event loop acts as a high-speed scheduler. It sends the request to MongoDB and immediately moves to the next user, effectively "stacking" thousands of concurrent database operations without needing a thread for each one.

Graphs and metrics discussion (Locust latency plots)

Table: Sync stats showing the performance on one thread
Graph: Sync response times

Table: Async stats showing the performance

Graph: Async response times

Looking at our Locust metrics, the disparity is stark:
| Metric | Sync (1 Thread) | Async (Event Loop) |
|-------------------|-----------------|--------------------|
| Max RPS | ~48.8 | ~113.4 |
| Median Latency | ~8,700 ms | ~2,800 ms |
| 95th-Percentile | ~9,400 ms | ~4,100 ms |
| Throughput Driver | Thread-bound | DB / Network-bound |

Sync Graphs: Notice the staircase effect in response times.
As users increase, the latency climbs linearly because users are waiting in a queue. The RPS is "hard-capped" because the single thread can only process so many documents per second.

Async Graphs: The response times remain significantly lower and more stable.
Even as we ramped to 500 users, the 95th percentile hovered around 4 seconds compared to nearly 10 seconds for Sync. This proves that Async manages high-concurrency significantly better, even when the individual database query is complex.

Full working snippet for readers to run

Ready to see these results on your own machine?
We have provided a complete repository containing the FastAPI implementation, the AnyIO thread limiter configuration, and the Locust scripts used for these benchmarks.

Demo work - https://github.com/gouravmongodb/fastapi_pymongo_demo

9. Common Pitfalls & How to Avoid Them

Even with Async PyMongo, it is easy to introduce subtle performance or correctness issues if you are not careful. Below are the most common pitfalls developers encounter and how to avoid them.

Forgetting await on Database Calls

Because Async PyMongo methods are awaitable, forgetting await will not raise an immediate syntax error but will return a coroutine object instead of executing the operation. This can lead to silent failures or unexpected behavior in your API responses. Always double-check that every database operation inside an async def route is properly awaited.

Creating a New Client for Every Request

Instantiating AsyncMongoClient inside each request handler is a major anti-pattern. It creates unnecessary TCP connections, increases latency, and quickly exhausts resources under load. Instead, create a single shared client at application startup (using FastAPI lifespan or startup events) and reuse it across all requests.

Returning Non-JSON-Serializable Types

MongoDB types like ObjectId, datetime, or Decimal128 are not directly JSON serializable. Returning raw MongoDB documents from FastAPI will cause serialization errors. The recommended approach is to either convert these types to strings or map your documents to Pydantic models that handle serialization correctly.

Exhausting Connection Pools Under Heavy Load

Async applications can issue far more concurrent database operations than synchronous ones, which makes connection pool tuning critical. If maxPoolSize is too low, requests will start queuing inside the driver, increasing latency. Monitor Atlas metrics and adjust maxPoolSize, minPoolSize, and maxConnecting based on your workload.

Mixing Synchronous and Asynchronous Code

Using synchronous PyMongo calls inside async def routes (or vice versa) undermines FastAPI’s concurrency model and can block the event loop. Choose one model—preferably Async PyMongo for FastAPI—and stick to it consistently throughout your application.

10. Conclusion

Asynchronous database access is a cornerstone of modern, scalable API design. In an async-first framework like FastAPI, using a non-blocking database driver is not a luxury—it is a necessity for handling high concurrency, reducing latency, and making efficient use of resources.

With the stabilization of pymongo.async_ in PyMongo 4.13, Async PyMongo has become the recommended and officially supported way to use MongoDB with FastAPI. It provides a native, awaitable interface that aligns with FastAPI’s event loop, simplifies dependencies, and brings the future of the MongoDB Python ecosystem under a single driver.

Synchronous PyMongo still has a place in the ecosystem — particularly for simple scripts, batch jobs, or applications that are not built around async frameworks. However, for production web services, real-time applications, and microservices that must scale, Async PyMongo is the better choice.

Ultimately, performance depends not just on “sync vs async,” but on how you design your workload, tune your connection pool, index your collections, and scale your MongoDB cluster. When used thoughtfully, Async PyMongo with FastAPI offers a powerful, maintainable, and future-proof foundation for building high-performance Python APIs.

Research work

· https://pymongo.readthedocs.io/en/stable/changelog.html#changes-in-version-4-9-2024-09-18
· From Pymongo driver v4.9.0 introduced the beta version of async function.
· From Pymongo driver v4.13.0 - asynchronous API is now stable and no longer in beta.
· https://www.mongodb.com/docs/languages/python/pymongo-driver/current/reference/migration/#migrate-to-pymongo-async
· https://www.mongodb.com/docs/languages/python/pymongo-driver/current/reference/migration/#performance-benchmarks