Forem: Roman Right

Studio 3T Performance Suite: How to Identify and Resolve Performance Bottlenecks in MongoDB

Roman Right — Thu, 01 Feb 2024 11:01:00 +0000

Introduction

In today's data-driven landscape, the performance of databases is not just a technical concern but a pivotal factor in the success of any application. MongoDB, renowned for its scalability and flexibility, is no exception. However, as systems grow in complexity and data volume, performance bottlenecks can emerge, significantly impacting user experience and system reliability. This article delves into harnessing the power of Studio 3T, a comprehensive tool suite for MongoDB, to optimize database performance. We will explore how Studio 3T's features, like the Query Profiler, Index Manager and Visual Explain, can be used to identify and resolve common performance issues, ensuring that your MongoDB database runs efficiently and effectively.

Understanding the Query Profiler

The Heart of Performance Analysis

The Query Profiler in Studio 3T stands as a vital instrument for enhancing MongoDB's performance. Integrating the capabilities of MongoDB's Database Profiler, it offers an in-depth examination of your database operations. This tool is essential in illuminating both the execution time and patterns of your queries, providing a surprising degree of useful insight.

Spotting Slow Queries: A Practical Scenario

Consider a smart home system that processes a myriad of device events for a wide customer base. Each event, whether from a security camera or a smart thermostat, results in a data entry. Envision this system bogged down by a particular query:

{
  "customer_id": ObjectId('65b69ed094c014b3d11fa889'),
  "device_type": "security_camera",
  "event_details.alert_type": "sound_detected"
}

In the absence of efficient indexing, the execution time of this query stretches to an excruciating 2.5 seconds, an unacceptable delay in a real-time monitoring scenario.

In this instance, the Query Profiler adeptly identifies the tardy query. Its graphical display reveals vital statistics, such as execution time and frequency, which are instrumental in understanding the query's impact on the system's overall performance. This leads us to two crucial components of the Query Profiler:

Query Code Window: This section displays the query in question, aiding developers in pinpointing the problematic segment in the code base. It serves as a direct link between the observed performance issue and the specific part of the code that may require optimization.

Validating Improvements using Visual Explain

Demystifying Query Execution with Visual Insights

Visual Explain in Studio 3T is an invaluable tool for comprehensively understanding the execution process of MongoDB queries. It provides a detailed graphical representation of how queries are executed and the impact of different indexes on these processes. In addition to the visualisation, Visual Explain provides runtime stats for each step, showing how much data is being transferred and how long it has taken. It is available in the main query tools in Studio 3T and now also in its own window in the Query Profiler.

Explain Plan Window: Here, the profiler presents a graphical explanation of the query execution process. This visual breakdown includes various stages of the query execution, notably highlighting steps like 'collection scan.' A 'collection scan' indicates the absence of an index at a critical point, signaling a potential area for performance enhancement through indexing strategies.

Equipped with this comprehensive analysis, developers gain the necessary insights to make informed decisions, such as refining the query or implementing the required indexes, to optimize performance.

Optimizing with the Index Manager

Streamlining Performance through Strategic Indexing

The Index Manager in Studio 3T is a pivotal tool for optimizing query performance in MongoDB. It plays a critical role in identifying and creating indexes that can significantly reduce query execution times. By efficiently managing indexes, the Index Manager ensures that your database operations are streamlined and performant.

Case Study: Enhancing Query Speed with a Custom Index

To illustrate, let's revisit the slow query from our smart home system scenario:

{
  "customer_id": ObjectId('65b69ed094c014b3d11fa889'),
  "device_type": "security_camera",
  "event_details.alert_type": "sound_detected"
}

To address the 2.5-second delay in query execution, we can use the Index Manager to create a compound index. This index would be on the fields customer_id, device_type, and event_details.alert_type. By doing so, we are aligning the index directly with the query's structure, allowing MongoDB to retrieve the necessary data more efficiently.

In this screenshot, we see the Index Manager showcasing the creation of the new compound index. The fields customer_id, device_type, and event_details.alert_type are highlighted, demonstrating how the index aligns with the query structure.

Observing the Impact of the New Index on Execution Time

With the new index created via the Index Manager, we used Visual Explain to evaluate its effect on our query. This tool graphically displays the modified execution plan. The key alteration is the switch from a collection scan to an index scan, reflecting a more efficient data retrieval method.

The Visual Explain interface highlights this change, emphasizing the faster index scan. Remarkably, the total execution time of the query is now shown as 0ms, a significant decrease from the original 2.5 seconds, demonstrating the substantial impact of the optimization.

Checking Results in the Query Profiler

Confirming Optimization Success

The final step in our optimization journey involves revisiting the Query Profiler in Studio 3T. This tool, which initially helped us identify the slow query, now serves to confirm the effectiveness of our optimization strategies.

In the Query Profiler display, we observe a significant change. The previously sluggish query, after optimization, is conspicuously absent from recent query executions. We only see historical instances from before the optimization. This indicates that the query, now executing in under a millisecond, is no longer a bottleneck in performance. This marked improvement is a strong endorsement of our indexing strategy and the comprehensive optimization process.

Conclusion

Harnessing Studio 3T for Peak MongoDB Performance

In conclusion, this guide has demonstrated the power and utility of Studio 3T's tools in identifying and resolving performance issues within MongoDB databases. Starting with the Query Profiler, we identified a performance bottleneck. We then utilized the Index Manager to strategically create an index, drastically improving query execution times. The Visual Explain feature provided us with a clear, graphical representation of this improvement, showing a reduction in execution time to under a millisecond. Finally, our return to the Query Profiler confirmed the elimination of the slow query from recent executions, validating the effectiveness of our optimization efforts.

This journey underscores the importance of a systematic approach to database performance tuning. By leveraging Studio 3T’s comprehensive suite of tools, developers and database administrators can ensure their MongoDB databases are running efficiently and effectively. I encourage readers to apply these methodologies to their own databases, ensuring they remain high-performing and robust in handling data-intensive operations.

MongoDB Performance Tools

Roman Right — Wed, 10 Jan 2024 11:00:00 +0000

MongoDB is a powerful database, capable of handling a wide range of use cases. However, to extract the best performance from MongoDB, understanding and utilizing a range of supportive tools is key. This guide introduces a variety of tools that can enhance MongoDB's performance, irrespective of their origin.

Mongostat

Mongostat is an invaluable tool for real-time monitoring of MongoDB's performance. It offers snapshots of database activity, enabling developers to swiftly assess and address performance challenges. By examining the mongostat output from your MongoDB instance, we can glean crucial insights.

High Activity

The first screenshot displays a high level of database activity, with insert, query, and update operations at approximately 1300-1400 operations per second. This signifies a system under heavy load, managing a significant volume of data transactions. Notably, the dirty and used percentages remain low, suggesting efficient memory usage with the current working set.

Low Activity

In contrast, the second screenshot reveals a marked decrease in operation rates, with counts falling to 10-48 operations per second for insert, query, and update. This reduction is noted under consistent load test conditions, where both the number of connections and client-side requests remained unchanged. Despite a steady influx of client requests, the database appears to have become a bottleneck. Possible causes include resource saturation, inefficient query execution, or suboptimal indexing, all leading to prolonged response times and diminished throughput.

In this screenshot, the qrw (queued read and write operations) and arw (active read and write operations) columns are particularly telling. An increase in these values suggests a build-up of operations waiting to be processed, confirming the bottleneck scenario. The qrw metric alerts us to the number of operations in queue, highlighting potential delays in processing, while arw shows the number of actively processing operations, providing insight into the current load on the database.

Observations from mongostat are critical in real-world contexts, as they enable the identification of heavy load periods and sudden performance downturns. Such data is instrumental for developers in diagnosing issues, whether that involves delving into specific queries, reevaluating indexing strategies, or adjusting resource allocation for better load management.

The real-time feedback provided by mongostat is essential for maintaining optimal database performance, particularly in scenarios involving high traffic. Continuous monitoring of these metrics empowers developers to proactively manage their database environment, ensuring efficient and smooth operation.

MongoDB Compass

MongoDB Compass provides an intuitive graphical interface to visualize and understand the structure of your database collections. This is particularly useful in identifying inefficiencies and potential areas for optimization.

Collection Structure Overview

This screenshot presents an overview of the collection's structure. A notable aspect here is the address_historyfield. It's an array of documents, each containing 5 nested fields. The array length varies significantly across documents, with the minimum length being 0, an average of 7.56, and a maximum of 20.

Such variability in array length can be a critical factor in the collection's performance. Larger array sizes, especially when they exceed the average by a considerable margin, can lead to increased document size and potentially slower query performance. This insight can guide us in restructuring the data, perhaps by normalizing the collection or implementing more efficient indexing strategies.

Detailed View of Address History Field

The second screenshot offers a detailed view of the address_history field. It breaks down the frequency of various cities and other sub-fields within the address history. This level of detail can reveal trends or anomalies in the data distribution.

For example, if certain city names appear disproportionately often, it might indicate data skewness, which can impact the performance of queries filtered by city names. Similarly, understanding the distribution of other sub-fields helps in identifying redundant or rarely used data, paving the way for data cleanup and optimization.

Through these screenshots, MongoDB Compass effectively demonstrates its capability to provide deep insights into your data's structure and composition. This information is vital for making informed decisions about database optimization and restructuring, ensuring efficient data storage and retrieval.

Mongotop

Mongotop is an essential tool in the MongoDB suite, offering real-time insights into the read and write activities of database collections. It's instrumental in identifying resource-intensive collections and guiding optimization efforts.

Before Indexing

During a recent analysis with mongotop, the articles.users collection was closely monitored. The initial data presented a notable level of activity:

Total time: 4168 milliseconds
Read operations: 1994 milliseconds
Write operations: 2174 milliseconds
- After Indexing

However, after indexing and a subsequent observation period, there was a remarkable shift in the operational times:

Total time: 599 milliseconds
Read operations: 290 milliseconds
Write operations: 308 milliseconds

This marked decrease in both read and write times is a clear indicator of enhanced efficiency, likely due to the implementation of the index. The balanced distribution between read and write operations post-optimization suggests a more efficient handling of database tasks.

The ability of mongotop to provide such real-time, granular data is invaluable for database administrators and developers. It not only allows them to monitor the immediate impact of optimizations but also helps in identifying areas requiring further attention. This tool is crucial for maintaining and improving database performance, ensuring optimal operation and efficiency.

Studio 3T

For many MongoDB enthusiasts, Studio 3T is the toolbelt that feels like an extension of their own hands. A treasure trove of features, Studio 3T is crafted to augment MongoDB's native capabilities.

Query Profiler

Studio 3T's Query Profiler stands as a key instrument for optimizing MongoDB performance. Specializing in identifying and improving slow queries, this tool uses MongoDB's Database Profiler to gather detailed insights about query execution times and patterns, thus equipping developers with actionable information.

Spotting Slow Queries

The provided screenshot showcases the Query Profiler's adeptness at highlighting a slow query. For instance, a $lookup aggregation without an index initially results in a significant delay, taking about 10 seconds. Such extended execution times can drastically affect database performance, particularly in high-load environments.

The profiler efficiently displays vital statistics such as execution time and frequency, offering a clear view of a query's impact. In addition to providing the full runtime JSON script, it also separates out the query code itself. Developers can now not only understand the causer of the slowdown, but they can then remedy the problem there and then, either by adding or removing an index or by editing the query itself before rerunning and checking for improvement.

Optimization via Indexing

The Query Profiler is instrumental in the optimization process. By adding an index to relevant fields, like product_ids in the orders collection, the execution time for the same $lookup aggregation is notably reduced. The profiler confirms this enhancement, as the query no longer appears as a performance bottleneck.

This improvement, evident in the screenshot, illustrates the profiler's effectiveness in not just identifying but also in validating the impact of optimization strategies.

In conclusion, Studio 3T's Query Profiler is a vital asset for MongoDB developers, simplifying the often intricate task of database optimization. It facilitates a clear and efficient workflow for detecting performance issues and validating the efficacy of implemented solutions. Through its use, MongoDB instances can be fine-tuned for optimal performance, ensuring that they operate not only effectively but also at their peak efficiency.

Visual Explain

Every MongoDB query has an execution plan. It's like the route your GPS suggests when you're driving. Now, imagine if you could visually trace this route to identify traffic jams or shortcuts? Studio 3T's Visual Explain does exactly that—it offers a graphical representation of the MongoDB execution plan. By displaying how queries run, it makes it infinitely easier to spot bottlenecks.

You can find more information about this tool in the first article of the series: Setting the Stage: Unraveling MongoDB Query Performance

Aggregation Editor

Aggregation in MongoDB is an art and a science. Complex aggregation queries can, at times, be performance guzzlers. Studio 3T’s Aggregation Editor offers a respite. With a stage-by-stage interface, it simplifies the process of building, debugging, and most importantly, optimizing aggregation queries.

More information about this tool can be found in the second article of the series: Identifying MongoDB Performance Pitfalls

Conclusion

To maximize MongoDB's performance, a combination of various tools is essential. Tools like MongoDB Compass, Mongostat, Mongotop, and Studio 3T each contribute uniquely to understanding, monitoring, and enhancing database performance. By utilizing these tools effectively, developers and database administrators can ensure optimal operation and efficiency of their MongoDB instances.

Crafting Efficient MongoDB Queries

Roman Right — Thu, 30 Nov 2023 11:00:00 +0000

MongoDB, a leading database in the NoSQL domain, is not just about storing data—it's about harnessing it efficiently. Dive into the heart of its query system, understand its inner workings, and discover strategies to transform the way you interact with and retrieve your data.

MongoDB's Response Mechanism: The Power of the `explain()` Method.

Imagine sending a message out into the void, hoping for an answer but never quite knowing if your message was received, let alone understood. That's what querying without feedback feels like. With MongoDB's explain() method, we get to peek behind the curtain, understanding how our queries fare in the grand scheme of things.

What's this `explain()` magic?

At its core, explain() provides insights into the execution of MongoDB operations. By analyzing the output, developers can decode the database's decision-making process, getting a step-by-step walkthrough of how a query is executed. This doesn't just make you feel like a database wizard, but it also arms you with the knowledge to optimize queries for better performance.

Deciphering the Talkback

From the troves of information that explain() returns, a few fields stand out in their importance:

executionStats: This provides an overview of how your query was executed. Here you'll see details about the number of documents scanned, the number of documents returned, and how long the query took.
executionStages: A closer look into the mechanics of your query's execution. It can reveal whether your query did a full collection scan (COLLSCAN) or if it used an index (IXSCAN).
queryPlanner: Offers an insight into MongoDB's thought process. What strategy did it decide was best for your query? Did it consider multiple strategies before choosing one?

To bring these abstract concepts to life, let's consider a simple query where we wish to fetch all products in category "A" from our fictional collection:

collection.find({"category": "A"})

When we use the explain() method without an index on the "category" field, the output highlights that MongoDB had to scan every single document in the collection to find matches. This is evident in the executionStages field with the value COLLSCAN and totalDocsExamined being 1,000,000.

However, when an index on the category field is introduced and the same query is analyzed with explain(), the difference is night and day. Instead of COLLSCAN, we now see IXSCAN in the executionStages field, indicating that the database smartly used the index. Moreover, the totalDocsExamined drastically reduces to 199,916, showcasing the immense power of indexing. It's important to remember why such improvements in query performance are valuable: querying less data means using less time to do it. This efficiency translates into freeing up compute cycles, allowing other work to proceed more smoothly. Consequently, there's a direct impact on database hosting costs, as reduced processing time and resource usage can lead to significant cost savings.

Bridging Understanding for Efficiency

Understanding the feedback from MongoDB is akin to listening intently in a conversation. By focusing on what MongoDB is 'saying', we can adjust our questions (queries) to be more in tune with the database's strengths. Indexing, for instance, becomes a clear pathway to optimization once we've grasped how powerful it can be from the explain() outputs.

Mastering Indexes in MongoDB: Balancing Adaptability and Performance.

Imagine trying to find a particular book in a colossal library. Without any categorization or system, the task would be daunting and time-consuming. Similarly, in the grand library of MongoDB data, indexes serve as our guiding system, directing our queries with precision and speed. We've just witnessed how a well-placed index can turn an exhaustive COLLSCAN into a much more performant IXSCAN. But what's behind this magic? Let's journey into the heart of MongoDB indexing.

A Deep Dive into Indexing's Role

In essence, an index in MongoDB is a data structure that holds a subset of the collection's data in an easy-to-traverse form. This subset, ordered by the value of a specific field or set of fields, allows MongoDB to avoid scanning every document in a collection when searching for data. As seen in our earlier example with product categories, indexing can greatly enhance search efficiency.

The Colorful Palette of Indexes in MongoDB

MongoDB offers a properly rich set of index types and options for anyone seriously interested in query tuning. Each index type is crafted to serve specific use-cases, ensuring efficient querying and data retrieval. Here's a deep dive into these index types, explaining their purpose, how they work internally, and their primary use-cases:

Single Field:
- Purpose: To support efficient searches on individual fields.
- How it works: This index type creates a B-tree structure on a single field, enabling fast lookups.
- Use-case: Suited for situations where queries primarily focus on one field, like looking up users by their username.
Compound:
- Purpose: To handle queries that combine multiple fields.
- How it works: This index spans multiple fields within a document. A B-tree structure is maintained for the combination of these fields.
- Use-case: Useful when querying on conditions that involve multiple fields, like finding orders based on both customer ID and product ID.
Multikey:
- Purpose: To index array data effectively.
- How it works: Indexes each individual element of an array.
- Use-case: Suited for situations where queries target specific array values, like filtering products by tags.
Text:
- Purpose: To provide efficient full-text search capabilities.
- How it works: It tokenizes strings and stores them as an inverted index, making text searches faster.
- Use-case: Suitable for applications that require text search functionality, like searching for articles containing specific words or phrases.
Geospatial:
- Purpose: To facilitate queries based on geographical locations.
- How it works: Uses a special indexing mechanism to efficiently store and query data based on geographic coordinates.
- Use-case: Ideal for location-based services, like finding nearby restaurants or calculating distances between points.
Hashed:
- Purpose: Primarily to support sharding.
- How it works: It hashes the field's value and then indexes this hash, ensuring an even distribution of data.
- Use-case: Suitable for databases that use sharding to distribute data across multiple servers or clusters.
Wildcard:
- Purpose: To offer flexibility by indexing fields dynamically.
- How it works: Instead of indexing specified fields, it indexes all fields, ensuring no field is left unindexed.
- Use-case: Useful in scenarios where the structure of documents is dynamic or not well-defined in advance.
Sparse:
- Purpose: To conserve memory.
- How it works: Only includes documents where the indexed field is present, ensuring the index remains lightweight.
- Use-case: Ideal for collections where only some documents contain the indexed field, like optional attributes in user profiles.
TTL (Time-to-Live):
- Purpose: To automate data pruning.
- How it works: Monitors the indexed timestamp field and removes documents once they've existed past a specified duration.
- Use-case: Perfect for data with a natural expiration, such as session data or temporary logs.
Partial:
- Purpose: To ensure resource efficiency.
- How it works: Only indexes a subset of documents, based on a defined filter condition.
- Use-case: Designed for large collections where only a specific fraction of documents are relevant for indexing, like indexing only 'active' user accounts.

Each of these indexes plays a crucial role in ensuring MongoDB's versatility and performance. By understanding the strengths and ideal use-cases for each type, developers can craft optimized, responsive, and resource-efficient applications.

The Delicate Balance: Tradeoffs of Indexing

However, with great power comes great responsibility. Indexing isn't a silver bullet. While they speed up read operations, they introduce overheads:

Write Operation Delays: Every time data is written or updated in the collection, the corresponding index also needs an update. This can introduce latency, especially if multiple indexes exist for a collection.
Space Overheads: Indexes consume disk space. In some scenarios, especially with multiple compound indexes, the indexes' size can exceed the size of the data itself.

In the world of databases, there's always a balancing act between read and write efficiency and the resource footprint. The art lies in strategically creating indexes based on the specific needs of the application and the query patterns, ensuring that the benefits outweigh the costs.

MongoDB's Best Practices: Streamlining Data Operations with Native Techniques

Paging Through Your Data: From Clunky Steps to Graceful Strides

Pagination is one of those ubiquitous data operations that sounds simple but can be tricky to implement efficiently, especially when dealing with copious amounts of data.

Traditional Approach: Skip and Limit

A conventional method involves using skip() and limit():

collection.find({}).skip(50000).limit(100)

This method skips the first 50,000 documents and then fetches the next 100. It works perfectly for smaller datasets. But as the amount of data grows, the performance can degrade. Imagine skipping over a million records to get to the next set. Imagine trying to jump to the last chapter of a book but having to flip through every single page to get there.

The Efficient Turn: Range Queries

A more efficient strategy involves using range queries. Instead of skipping records, you position your query based on a certain field's range.

Let's say we're working with a timestamp. Instead of skipping over documents, we'd pull records that have a timestamp greater than the last one we've processed:

collection.find({"timestamp": {"$gt": lastKnownTimestamp}}).limit(100)

Here, we've bypassed the clunky skip and are now taking graceful strides through the data, enhancing the efficiency manifold.

Textual Battles: Regex vs. Text Search

Text searches are pivotal for any database, and MongoDB offers two primary tools: regular expressions (regex) and text search.

Original Approach: Text Search

MongoDB's text search is designed to be nuanced, taking into account language-specific stop words and stemming. However, MongoDB mandates that a text search step must be the primary operation in an aggregation pipeline:

collection.find({$text: {$search: "coffee"}})

The Efficient Route: Regex

If you can replace the text search with a regex, especially in scenarios where a match query reduces the number of documents to be scanned, the regex will be more efficient. This is because it can be applied post the match step:

collection.aggregate([
  {$match: {"productName": {"$exists": true}}},
  {$match: {"productName": /coffee/i}}
])

Here, the regex operates on a reduced set of documents due to the preceding match step, resulting in a more efficient query.

Choosing Wisely: The $in Operator vs. the $or Operator

When we're dealing with selecting documents based on multiple criteria, MongoDB presents us with two operators: $in and $or. Both serve a purpose, but there's a marked difference in how they impact performance. Let's embark on a journey to understand when and why you might favor one over the other.

Original Approach: Using $or

Consider this scenario: you want to fetch products from categories 'A' and 'B' but only those priced at $500 or less. Using the $or operator, your query would resemble:

db.getCollection("products").find({
    "$or": [
        {"category": "A", "price": {"$lte": 500}},
        {"category": "B", "price": {"$lte": 500}}
    ]
}).explain()

Here, MongoDB is essentially evaluating two distinct conditions. For every document, it questions, "Is this a product from category 'A' with a price of $500 or less? If not, is it from category 'B' and similarly priced?". Now, this method might be efficient for modest datasets, but with an increasing volume of data, the two conditions being checked independently can consume more time and resources. If there's no index to assist, the process can further lag.

The Streamlined Method: Using $in

Enter the more succinct $in operator:

db.getCollection("products").find({
    "category": {"$in": ["A", "B"]},
    "price": {"$lte": 500}
}).explain()

This refined query is essentially an encapsulation of the previous $or conditions. But what sets $in apart? When MongoDB encounters the $in operator and an index exists on the category field, it can quickly pinpoint the documents with categories "A" and "B". Instead of evaluating two distinct conditions like $or, it checks the indexed values for the presence of both "A" and "B" categories, ensuring a more streamlined retrieval.

The choice between $in and $or might appear nuanced at first glance, but these subtleties can profoundly influence database performance. In our endeavor to master the MongoDB vernacular, recognizing such distinctions enables us to converse more efficiently with our data repositories. Knowledge is power, and in this context, it's also speed.

4. Decoding Query Performance with Studio 3T's IntelliShell

Studio 3T is one of the foremost MongoDB GUIs and provides a myriad of tools to facilitate seamless database interaction, including the IntelliShell, which gives developers the ability to write, edit, and execute MongoDB commands in a shell-like environment.

When trying to decipher the efficiency of different MongoDB operations, the .explain() method becomes crucial. Using Studio 3T's IntelliShell tool, we can run .explain() on queries to garner insights into how MongoDB processes them. Let's dive deep into this by comparing the $in vs. $or operators using the provided examples.

Step-by-Step Guide:

1. Launch IntelliShell:

In Studio 3T, click on the IntelliShell button to open a new tab where you can input MongoDB commands.

![Studio 3T screenshot](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/my2m2gc5tlyp7qti1jxa.png)

2. Execute the $in Query:

   db.products.find({
       "category": {"$in": ["A", "B"]}, 
       "price": {"$lte": 500}
   }).explain()

Upon executing, you'll get a JSON response. The key sections to pay attention to:

winningPlan: This section reveals the optimized plan chosen by MongoDB to execute your query. As seen in the provided output, the chosen plan uses a FETCH stage to filter the price, and then an IXSCAN stage to use the index on the category field.
indexName: This confirms that the query is using the category_1 index, signifying the use of the indexed field which boosts query performance.

3. Execute the $or Query:

   db.products.find({
       "$or": [
           {"category": "A", "price": {"$lte": 500}},
           {"category": "B", "price": {"$lte": 500}}
       ]
   }).explain()

In the $or query response:

The winningPlan section shows the SUBPLAN stage, which is essentially MongoDB evaluating multiple query plans. It then uses an OR stage to combine the results of two independent FETCH and IXSCAN stages.
Notably, even though the $or query uses the same category_1 index, it evaluates each condition in the $or array separately. This distinction highlights why the $in operator is generally more efficient than the $or operator for this specific use-case.

4. Comparative Analysis:

Both queries utilize the category_1 index. However, the execution is different.
$in offers a streamlined approach with a singular FETCH and IXSCAN.
In contrast, $or separately evaluates each condition leading to two separate FETCH and IXSCAN operations.

Conclusion:

Using the IntelliShell tool in Studio 3T, we can vividly see the under-the-hood differences in how MongoDB executes the $in and $or operations. For the given example, the $in operation proves to be more efficient as it employs a singular indexed scan for multiple values. On the other hand, the $or operation separately assesses each condition, adding overhead to the query processing.

Understanding these nuances empowers developers to write optimized queries, ensuring swifter response times and efficient use of resources. Remember, the devil is in the details, and with Studio 3T's IntelliShell, you can unveil these crucial details.

Identifying MongoDB Performance Pitfalls

Roman Right — Thu, 26 Oct 2023 12:25:01 +0000

MongoDB's reputation in the NoSQL community is built on its query optimization capabilities. In the previous article, we explored how MongoDB queries work and how to use native optimization techniques to improve performance. Understanding these concepts is vital for crafting efficient queries.

However, even experienced developers sometimes face challenges that slow down performance. To achieve the best performance, we need to know not only how MongoDB queries work but also how to avoid common problems that can waste time and resources. Knowledge helps us find better efficiency and speed. Let's explore this together, highlighting common challenges and discussing how to tackle them.

The N+1 Problem: Unnecessary Resource Overhead

Imagine a scenario. Your application needs to fetch data about a set of users and their respective posts from the database. A common approach involves making one query to retrieve the users and then, for each user, making additional queries to fetch their posts. While it may seem straightforward, this N+1 query problem slyly drains the vitality from your application's performance.



// Example of the N+1 problem in MongoDB
users = db.users.find();
users.forEach(user => {
  user.posts = db.posts.find({ userId: user._id });
});

In the above code, for every user fetched from the users collection, an additional query is made to the posts collection to retrieve their posts, resulting in N+1 queries (where N is the number of users). It’s a classic trap, with each additional query incrementally throttling the speed and efficiency, making the application progressively sluggish, and devouring server resources as it goes.

The antidote? Batch the queries, letting MongoDB handle multiple requests in a single go, minimizing the back-and-forth and ensuring the resources are judiciously used.



// Batching queries to resolve the N+1 problem
userIds = db.users.find().map(user => user._id);
userPosts = db.posts.find({ userId: { $in: userIds } });

By utilizing the $in operator, the second query now fetches all posts for the user IDs in a single sweep, effectively countering the N+1 query problem and conserving your application's robustness and vitality. The art lies in striking a balance, ensuring that the strategies employed enhance performance, not stifle it, keeping the MongoDB queries in the fast lane, smoothly sailing towards optimal efficiency.

The Silent Delay: Unpacking the Effect of Missing Indexes

When working with MongoDB, even well-crafted queries can experience performance drops if the right indexes are missing. Without these indexes, MongoDB is forced to perform a collection scan, meaning it goes through every document in the collection to find the needed data. While this might be okay for small datasets, it can cause delays as the data grows.

Look at this typical query:



db.orders.find({"brand": "Apple"})

Without an index on brand, MongoDB must scan through the entire orders collection, which may not be effective for large datasets.

But, adding an index can greatly improve query speed:



// Indexing the brand field
db.orders.createIndex({ "brand": 1 })

With this index, MongoDB can quickly find relevant documents, reducing unneeded processing time.

Still, indexing isn't about just adding them everywhere. Too many or poorly set up indexes can actually slow things down. It's vital to carefully choose which fields to index based on how queries are made, and to keep an eye on index size and memory use to get the best results.

The Hidden Bumps on the Road: Other Overlooked Pitfalls

Unoptimized Aggregation Pipelines

Take the case where you're aggregating documents to find the total sales for each item. Without optimization, the aggregation pipeline processes every document in the collection.



// An unoptimized aggregation query
db.sales.aggregate([
  {
    $group: {
      _id: "$item",
      totalSales: { $sum: "$amount" }
    }
  }
])

The query above groups the sales collection by the item field and calculates the total amount. It might work efficiently for smaller datasets but can become a bottleneck as the data grows because it processes each document in the collection.

An enhancement would be to use the $match stage early in the pipeline to filter documents based on certain criteria. For instance, suppose you’re only interested in calculating total sales for items that have sold at least once in the past week.



// Optimized aggregation query
db.sales.aggregate([
  { $match: { lastSoldDate: { $gte: new Date("2023-09-23") } } },
  {
    $group: {
      _id: "$item",
      totalSales: { $sum: "$amount" }
    }
  }
])

By introducing $match, the aggregation pipeline first filters out the documents that do not meet the criteria (items sold in the past week), thereby reducing the number of documents that the $group stage has to process. This optimized pipeline ensures that only relevant documents are processed, boosting the efficiency and speed of the aggregation operation.

In this example, the $match stage serves to narrow down the data that is processed in the subsequent stages, demonstrating a typical scenario where the strategic arrangement of pipeline stages contributes to enhanced performance.

Ignoring Sharding for Large Datasets

Another unnoticed pitfall is the neglect of sharding for large datasets. Sharding distributes the data across multiple servers, ensuring that the database can scale beyond the limits of a single server. Ignoring it could lead to unbalanced and overloaded systems, tarnishing performance and speed.



// Implement sharding
sh.enableSharding("database")
sh.shardCollection("database.collection", { "shardKey": 1 })

With these simple commands, sharding is enabled for the specified database and collection, providing a vital boost for handling vast datasets.

Overlooking Connection Pooling

A further overlooked detail is not utilizing connection pooling. Opening a new database connection every time is not only expensive in terms of resources, but it also adds unnecessary latency.



// Use connection pooling
const client = new MongoClient(uri, { poolSize: 50 });
client.connect();

Here, a connection pool of size 50 is created, reusing existing connections and averting the overhead of establishing a new one each time.

Studio 3T Spotlight: Fine-Tuning Aggregation Pipeline Efficiency

I'd like to pick up one of the pitfalls we discussed earlier and show you how Studio 3T can help you improve the performance.

Imagine we have a service that collects anonymized user click data. Each document in the dataset might resemble the following:



{
  "_id": ObjectId("6531f27d8a57e4c3a95738fd"),
  "x": 450,
  "y": 320,
  "timestamp": "2023-10-17T14:08:59.000+0000",
}

Where x and y represent the click coordinates on the page. The task is to aggregate this data to find regions (or bins) of the page that receive the most clicks.

Here's the initial aggregation pipeline:



// Original Pipeline
[
    {
        "$project": {
            "binX": {"$floor": {"$divide": ["$x", 100]}},
            "binY": {"$floor": {"$divide": ["$y", 100]}}
        }
    },
    {
        "$group": {
            "_id": {"binX": "$binX", "binY": "$binY"},
            "count": {"$sum": 1}
        }
    },
    {"$sort": {"count": -1}},
    {"$limit": 3}
]

Preparation

Start by opening the Aggregation Editor for the "clicks" collection. This is where you will input the aggregation pipeline steps, setting the stage for further analysis and optimization. This initial setup is crucial for ensuring that the pipeline runs efficiently, and it aids in quickly identifying any potential issues that may arise.

Identifying the Problem and Implementing the Solution

With the pipeline set, it’s time to pinpoint the bottleneck. The Explain button is a powerful tool that provides a detailed breakdown of the pipeline’s execution, highlighting the time taken at each stage.

Run the Explain for the aggregation.
Detect the slowdown at the $group step, with a 9-second execution time.
Introduce the $sample step before the aggregation, drastically improving the performance by reducing the data to a manageable sample size.

Pipeline



// Optimized Pipeline
[
    {"$sample": {"size": 100000}},
    {
        "$project": {
            "binX": {"$floor": {"$divide": ["$x", 100]}},
            "binY": {"$floor": {"$divide": ["$y", 100]}}
        }
    },
    {
        "$group": {
            "_id": {"binX": "$binX", "binY": "$binY"},
            "sampledCount": {"$sum": 1}
        }
    },
    {"$sort": {"sampledCount": -1}},
    {"$limit": 3}
]

You can find the script that generates the experiment data here.

It's important to note that sampling can quickly process large datasets but adds some approximation. It works best on extensive data where a random sample gives a representative snapshot of the full dataset.

Conclusion

In the world of MongoDB, achieving optimal performance can sometimes be a challenge. But with the right information and methods, these challenges become opportunities for improving efficiency and speed. The strategies discussed here help protect your MongoDB operations from unnecessary delays and resource use. As you keep using and refining these methods, your MongoDB tasks will not only sustain but flourish, offering top performance and helping your projects succeed effortlessly. Remain alert and informed, and let each step on this journey be a move towards continuous excellence in MongoDB.

Detective de'Code and the Dueling Dishes

Roman Right — Wed, 11 Oct 2023 18:50:43 +0000

The Call

On a foggy evening, Louis de'Code was deeply focused in the green glow of his terminal when an unexpected notification jolted him. His old-fashioned, ringtone echoed through his dimly-lit office: "Gelátoz needs your help!" Gelátoz, the talk-of-the-town food delivery app, had become synonymous with the annual Feast of Fables - a playful holiday celebrating the most treasured culinary tales from around the world.

The message was clear and urgent: "Detective! Gelátoz hangs and hiccups just as our customers are finalizing their festive orders. And just our luck, due to the new privacy law that forbids retaining user food preferences, the logs—rich with food content—had to be deleted! Feast of Fables is days away, and this hiccup could turn our grand celebration into a grand fiasco."

A holiday where mythic dishes took center stage, Feast of Fables was Gelátoz's crowning achievement, and a malfunction now would not only spell a gastronomical disaster but also might just put the final nail in their digital coffin. The stakes were high, and the clock was ticking.

With his keen senses tingling, detective de'Code felt the weight of this crisis. Clearing the code clutter on his screen, he mentally geared up for a deep dive into the realm of Gelátoz, where culinary wonders met code. The game was on.

Tech Trio

In the heart of the city, the grand Gelátoz headquarters resembled an architectural masterpiece, where vintage aesthetics embraced the throbbing pulse of modern tech innovation. Inside this wonder, culinary fantasies transmuted into digital commands, awaiting their journey to patrons eagerly anticipating the Feast of Fables.

As de'Code made his entry, the symphony of coding keystrokes and server hums hinted at the technological dance unfolding. Immediately, he discerned the trio that wove the tech tapestry of Gelátoz:

Fiona, the Frontend Maestro: Donning an artsy beret and a demeanor of fierce focus, Fiona was fluent in the poetry of React. She delicately crafted user interfaces that felt less like screens and more like tantalizing culinary presentations. When not immersed in her digital creations, Fiona lost herself to the rhythm of dance, her body painting stories in the air with grace.
Benny, the Backend Virtuoso: His quirky "Life's better in Python" tee told tales of his expertise. Orchestrating the data dynamics using FastAPI, Benny ensured that the heartbeats of Gelátoz's operations always played in rhythm. Away from the digital waves, Benny felt a deep connection to the vastness of the seas, his heart echoing the call of seafaring adventures.
Alice, the Database Guardian: With her serene aura, and a mug that cleverly quipped, "Keep calm and query on," Alice guarded the sanctum of Postgres. Ensuring every data point, every transaction was as meticulous as a chef's recipe, she made sure the digital ingredients always sang in harmony. In her free time, the world of chess captivated her – where she meticulously planned moves and strategies, mirroring her precision at work.

Surveying the trio and the digital masterpiece they'd crafted, detective de'Code braced himself. With a malfunctioning app, mysteriously vanished logs, and the Feast of Fables ticking closer, his plate was full. The culinary coding caper had just begun.

Rhythmic Riddles

The grand room of Gelátoz's headquarters was punctuated by the determined pacing of detective de'Code. "The culprit must lie somewhere in the path of promo dish recommendation," he mused. First stop: the Frontend - the shop window of the app. Fiona, with a nervous glance, handed him access to the console, where the notorious JavaScript snippet lay in wait.

import { useState, useEffect } from 'react';

function PromoDishDetails({ dishId }) {
 const [dish, setDish] = useState(null);
 const [chefDetails, setChefDetails] = useState(null);

 useEffect(() => {
   async function fetchData() {
     let response = await fetch(`/api/promoDishes/${dishId}`);
     let data = await response.json();
     setDish(data);

     // Fetching chef details after getting the dish details
     let chefResponse = await fetch(`/api/chefs/${data.chefId}`);
     let chefData = await chefResponse.json();
     setChefDetails(chefData);
   }

   fetchData();
 }, [dishId]);

 return (
   <div>
     Dish Name: {dish?.name} <br/>
     Promo Price: {dish?.promoPrice} <br/>
     Chef: {chefDetails?.fullName}
   </div>
 );
}

Examining the lines, de'Code felt the rhythm of the sequential API calls, like notes in a well-composed melody. "Sequential requests," he thought aloud. "But executed gracefully."

He imagined it as a choreographed dance: a performer moves to the stage, dances, and only when the routine is done does the next performer take their turn. Each step is well-rehearsed; there's no tripping or overstepping.

Fiona sighed with relief. "So, our front-end dance is smooth?"

Detective de'Code nodded, appreciating the finesse in the script. "It's not here that our villain hides. The choreography is well-executed, no step out of place."

Tales from the Depths

Pivoting, Louis de'Code ventured into the darker, more enigmatic world of the backend. Like a sailor about to navigate uncharted waters, Benny, holding his code snippet, handed over the map to the next mystery.

async def fetch_promo_dishes():
  async with httpx.AsyncClient() as client:
     for attempt in range(MAX_RETRIES):
        try:
            response = await client.get(f"{SUPPLIER_URL}/promo")
            response.raise_for_status()
            return response.json()
        except httpx.RequestError:
            if attempt < (MAX_RETRIES - 1):
                await asyncio.sleep(RETRY_DELAY * (2 ** attempt))
            else:
                raise


@app.get("/promoDishes/")
async def get_promo_dishes():
   try:
       dishes = await fetch_promo_dishes()
       return dishes
   except httpx.RequestError:
       return {"error": "Failed to fetch promotional dishes."}

The detective's fingers trailed over the lines detailing retries, a clever system reminiscent of a sailor adjusting his course amidst changing winds and tides. "Resilient code," he admired. "Just like a captain not letting a few storms dissuade his journey, but knowing when to dock for shelter. Benny, this is masterful navigation. You've safeguarded against turbulent third-party waters."

Benny, beaming with a mix of pride and relief, responded, "We don't want to be those sailors who fight against a storm until their ship sinks. We adjust the sails, make a few attempts, then find safe harbor."

As the backend's innocence became evident, de'Code's gaze wandered, seeking another thread. "Considering the system stalls post-order but prior to promo dish recommendation, our focus should now be on the order placement mechanism." The game was still afoot.

The Stalemate Standoff

Amid the dim lighting of the server room, detective de'Code, with narrowed eyes, stared at the enigma on the screen. The cursor blinked in a rhythmic pattern, but what unfolded was nothing short of a fierce chess match between two pieces: process_normal_order and process_combo_order.

"Notice," the detective whispered, fingers caressing the keyboard, "how these two functions handle transactions in reverse order. The devil, as they say, is in the details."

async def process_normal_order(conn, order_dish_ids):
    async with conn.transaction():
        await conn.execute(
            """
            UPDATE 
                inventory
            SET 
                stock = stock - 1
            WHERE
                id = ANY($1::int[]);
            """,
            order_dish_ids,
        )
        await conn.execute(
            """
            INSERT INTO
                orders (status, dish_ids)
            VALUES
                ('created', $1);
            """,
            order_dish_ids,
        )


async def process_combo_order(conn, combo_dish_ids):
    combo_dish_ids = combo_dish_ids + [COLA_ID, FRIES_ID]
    async with conn.transaction():
        await conn.execute(
            """
            INSERT INTO
                orders (status, dish_ids)
            VALUES 
                ('created', $1);
            """,
            combo_dish_ids,
        )
        await conn.execute(
            """
            UPDATE 
                inventory
            SET
                stock = stock - 1
            WHERE
                id = ANY($1::int[]);
            """,
            combo_dish_ids,
        )


@app.post("/placeOrder/")
async def place_order(order: Order):
    conn = await asyncpg.connect(DATABASE_URL)
    if order.tag == "combo":
        await process_combo_order(conn, order.dish_ids)
    else:
        await process_normal_order(conn, order.dish_ids)
    await conn.close()
    return {"status": "Order placed successfully!"}

The room, filled with developers and stakeholders, buzzed with confusion. Alice, the database expert, stepped forward. "When two transactions compete for the same resources but in reverse order, it's a classic recipe for a deadlock. It's as if two chess players tried to checkmate each other, but their moves end up in a stalemate."

Detective de'Code nodded in agreement. "Exactly. In a high traffic scenario, when both functions are invoked almost simultaneously, one waits for the other indefinitely, creating a standstill - our mastermind antagonist: the deadlock."

Resolution Recipe

With the problem identified, Louis de'Code, ever the genius, set forth on rectifying it. He swiftly rearranged the code, ensuring consistency in transaction order, preventing the chess pieces from getting trapped.

async def process_order(conn, order_dish_ids):
    # Let's combine the logic so that 
    # transaction order is consistent
    async with conn.transaction():
        await conn.execute(
            """
            INSERT INTO
                orders (status, dish_ids)
            VALUES
                ('created', $1);
            """,
            order_dish_ids,
        )
        await conn.execute(
            """
            UPDATE
                inventory
            SET
                stock = stock - 1 
            WHERE 
                id = ANY($1::int[]);
            """,
            order_dish_ids,
        )


@app.post("/placeOrder/")
async def place_order(order: Order):
    conn = await asyncpg.connect(DATABASE_URL)
    dish_ids = order.dish_ids
    if order.tag == "combo":
        dish_ids += [COLA_ID, FRIES_ID]
    await process_order(conn, dish_ids)
    await conn.close()

    return {"status": "Order placed successfully!"}

The room erupted in applause. Alice, impressed, remarked, "By ensuring that both types of orders follow the same transaction pattern, you've eliminated the potential for a deadlock. Brilliant!"

But amidst the cheers, de'Code, ever modest, simply adjusted his hat. "It's all about ensuring the right recipe, a harmonized blend of ingredients. Tech problems, after all, are like puzzles waiting to be solved."

And with that, the detective packed his toolkit. "Another tech enigma awaits," he remarked, leaving the Gelátoz team in awe, their gratitude palpable, their system more robust.

Setting the Stage: Unraveling MongoDB Query Performance

Roman Right — Wed, 27 Sep 2023 11:51:14 +0000

The heart of digital interactions: Why MongoDB's speed and efficiency matter.

In our warp-speed digital age, isn't database performance more than a perk? Imagine a bustling online store during a massive sale. A lag isn't just some annoying spinner—it's missed profit.

Now, picture a financial trading platform on a high-stakes day. Delays? Those could lead to significant financial repercussions.

Moreover, we're now in an era of real-time data analytics and machine learning, where data isn't just retrieved but also processed at lightning speeds. Real-time recommendations, fraud detection, or even simple tasks like auto-fill in forms, rely heavily on the efficiency of these databases.

However, it's not just about speed. The concept of efficiency also translates to the system’s ability to handle vast amounts of data with minimal resources. As businesses grow and scale, so do their databases. Herein lies the beauty of MongoDB – its design allows it to balance the scales of speed and resource management.

Anatomy of a MongoDB query: Breaking it down.

So, what makes a MongoDB query tick? Let's dive in.

First, it's essential to understand that MongoDB is a NoSQL database. Unlike its relational counterparts, it uses JSON-like documents with optional schemas, meaning data can be stored in varied formats. This feature itself lends a degree of flexibility to MongoDB that is often absent in traditional databases.

Every query begins with a client sending a request to the MongoDB server. This could be anything from a simple find request to retrieve documents to more complex aggregate functions.

Namespace

The first step in this journey is the Namespace. Think of it as a combination of the database name and the collection or index name. It's the address of sorts, directing the query where to go.

For example, consider the following MongoDB shell command:

use usersDatabase
db.userCollection.find()

Here, usersDatabase.userCollection is the namespace.

Query Selector

Once the namespace is identified, the next step is the Query Selector. This is where the conditions for the query are set. For example, if you want to find all users from a particular city, the query selector will contain parameters to match against documents in the chosen collection.

In the MongoDB shell, it might look something like this:

db.userCollection.find({"address.city": "New York"})

Here, {"address.city": "New York"} is the query selector.

Indexes

MongoDB is smart. Instead of scanning every document, it uses indexes to speed up the search process. If the appropriate index exists, MongoDB can locate the documents in a fraction of the time it would have otherwise taken.

For instance, you might create an index on the city field like so:

db.userCollection.createIndex({"address.city": 1})

Projection

Finally, once the documents are located and retrieved, we have the Projection. This determines which fields from the documents are returned. Think of it as a spotlight on specific information, focusing on what's essential and discarding the rest.

In the MongoDB shell, a projection can be illustrated as:

db.userCollection.find(
    {"address.city": "New York"}, 
    {"name": 1, "address.city": 1, "_id": 0}
)

In this example, only the name and address.city fields will be returned for documents that match the query, while the _id field is excluded.

MongoDB's thought process: Delving into query execution

MongoDB isn't just any ordinary database that takes in a query and spits out a result. There's an intricate dance of logic, planning, and execution that takes place behind the scenes, ensuring the speedy and efficient retrieval of data.

Query Planner

Whenever a query is issued to MongoDB, it doesn't just jump right into fetching the data. First, it takes a moment to strategize. This is the domain of the Query Planner.

The Query Planner assesses possible strategies to retrieve the data. If there are indexes that can be used, it evaluates each of them. Essentially, it's looking for the most efficient path, much like how a GPS navigates through various routes to determine the quickest way to your destination.

During this phase, MongoDB might even execute multiple query plans concurrently, racing them against each other to figure out the fastest one.

Query Executor

Having mapped out its strategy, MongoDB then hands over the baton to the Query Executor. This component is all about action. It takes the chosen plan from the Query Planner and executes it, retrieving the required documents from the collection.

For performance optimization, the Query Executor might at times, under certain conditions, interrupt an ongoing query execution and re-evaluate a more optimal plan, especially if it notices that the initial plan isn't as efficient as anticipated.

Studio 3T Spotlight: Visual Explain - A first look at visualizing your queries

While understanding the inner workings of MongoDB is enlightening, there's a distinct difference between reading about it and seeing it in action. Enter Studio 3T's Visual Explain.

Why Visualization Matters

Let's face it, while many of us can comprehend intricate details from textual explanations, there's something about visual representation that simplifies complexity. Much like how a roadmap can illustrate a journey better than a list of directions, Visual Explain showcases the journey of your MongoDB query.

Getting Started with Visual Explain

Studio 3T's Visual Explain is as intuitive as it gets. When you create a query in Studio 3T, there's an option to visualize the query's execution plan. By selecting this, you're immediately presented with a visual breakdown of how MongoDB plans to, or has, executed your query.

The visualization covers everything:

The Namespace and the path taken to reach the desired collection.
The potential Indexes that MongoDB considers, along with which ones are eventually used.
The Query Selector and how the documents are filtered out based on the provided conditions.
And finally, the Projection which highlights the specific fields fetched from the documents.

Benefits of Visual Explain

With Visual Explain, you gain the ability to:

Optimize Your Queries: By observing the paths taken during query execution, you can identify bottlenecks and optimize your queries for better performance.
Understand Index Usage: Visualize which indexes are being utilized and how they affect query performance. This can be invaluable when deciding on creating new indexes.
Educational Value: For those new to MongoDB, seeing the visualization can accelerate understanding, making the learning curve a bit gentler.

Visual Explain doesn’t just clarify; it's a bridge to MongoDB's deeper workings.

A Practical Use Case

Imagine you operate an online computer store. Your database brims with product details stored in specific structured documents. These documents serve as the backbone for various operations, from inventory checks to customer queries. But lately, there's a hitch – performance has hit a snag. Let's dive into how Visual Explain can be our detective and solution in one.

Take a glance at a sample document from our collection:

{
    "name": "Premium Laptop",
    "price": 1299.99,
    "manufacturer": {
        "name": "TechMaster Inc.",
        "location": "San Francisco, USA"
    },
    "reviews": [
        {
            "user": "LaptopLover123",
            "rating": 5,
            "comment": "Love it! Super fast."
        },
        {
            "user": "TechGuru22",
            "rating": 4,
            "comment": "Great build, needs better battery."
        },
        {
            "user": "GamingEnthusiast",
            "rating": 5,
            "comment": "Gaming champ! No lag."
        }
    ],
    "tags": ["laptop", "premium", "tech", "high performance"]
}

So, what's our mission? To retrieve laptops priced below $600. Here's how our query looks:

{"tags":"laptop", "price":{"$lt":600}}

In Studio 3T, Visual Explain shows how MongoDB navigates this:

Pre-indexing landscape.

The Indexing Decision

Our visual feedback screams for efficiency. It takes almost half a second to find the laptops we need. The culprits? The database is performing a full scan over the "tags" and "price" fields, which is anything but quick. The way out? A compound index for these fields. However, a word of caution: while indexes supercharge read operations, they might add a tad delay to write processes. Striking the right balance is paramount.

Studio 3T’s Indexing Flow

Launch Index Manager: Expand your collection and double-click on Indexes.
Build the Compound Index: Click Add Index, choosing both "tags" and "price".
Review the Impact: After going back to Visual Explain, we could clearly see the improvement. The searches are now 4 times faster!

With Visual Explain, we found that full scans were the issue. By adjusting our indexes, we made our searches quicker.

Beyond This Introduction

While we've touched upon the basics of MongoDB's query performance and indexing here, the depths and details, including how indexes function and the diverse types MongoDB offers, await in the upcoming articles of this series.

Concluding Remarks: The Journey Forward

As we come to the end of this introductory voyage into the heart of MongoDB's query performance, a few key thoughts emerge. Databases, as the silent orchestrators behind the symphony of digital interactions, bear an incredible weight of expectation. It's not just about storing data anymore – it's about doing so with agility, efficiency, and sophistication. MongoDB, with its non-relational structure and intelligent mechanics, rises to meet these challenges with aplomb.

The tools and techniques we've discussed, especially Studio 3T's Visual Explain, shed light on a world that often remains hidden beneath layers of code and abstracted complexity. They allow us, whether we're database novices or seasoned professionals, to visualize, understand, and optimize the intricate dance of query planning and execution.

However, as deep as we've delved today, it's essential to remember that this is but the tip of the iceberg. Optimizing MongoDB's performance is a rich topic, layered with nuances and intricacies. Understanding and enhancing technology should always be seen as an ongoing journey, not a destination.

That said, this is merely the opening act. In the upcoming installments of our series, "Enhancing MongoDB Query Performance", we'll explore more strategies, tools, and best practices. We'll dive deeper, ask tougher questions, and seek even more refined solutions to ensure that MongoDB doesn't just perform – it excels.

Stay tuned and, as always, never stop questioning, learning, and optimizing. The digital age awaits, and we're just getting started.

Announcing AnnaDB - next-gen NoSQL database

Roman Right — Tue, 13 Sep 2022 10:20:53 +0000

I'm excited to introduce AnnaDB - the next-generation developer-first NoSQL data store.

I work with many small projects daily - proofs of concepts and experiments with new frameworks or patterns. For these purposes, I needed a database that works with flexible data structures, as I change it frequently during my experiments. And it must support relations out of the box, as this is a natural part of the structures' design - links to other objects. I tried a lot (if not all) databases, but nothing fit my requirements well. So, I decided to make my own then. This is how AnnaDB was born.

AnnaDB is an in-memory data store with disk persistence. It is written with Rust, a memory-safe compilable language. AnnaDB is fast and safe enough to be and the main data storage, and the cache layer.

Features

Flexible object structure - simple primitives and complicated nested containers could be stored in AnnaDB
Relations - you can link any object to another, and AnnaDB will resolve this relation on finds, updates, and other operations.
Transactions - out of the box

Basics

I want to start with the basic concepts and examples of the syntax here and continue with the usage example.

Collections

AnnaDB stores objects in collections. Collections are analogous to tables in SQL databases.

Every object and sub-object (item of a vector or map) that was stored in AnnaDB has a link (id). This link consists of the collection name and unique uuid4 value. One object can contain links to objects from any collections - AnnaDB will fetch and process them on all the operations automatically without additional commands (joins or lookups)

TySON

The AnnaDB query language uses the TySON format. The main difference from other data formats is that each item has a value and prefix. The prefix can mark the data or query type (as it is used in AnnaDB) or any other useful for the parser information. This adds more flexibility to the data structure design - it is allowed to use as many custom data types as the developer needs.

You can read more about the TySON format here

Data Types

There are primitive and container data types in AnnaDB.

Primitive data types are a set of basic types whose values can not be decoupled. In TySON, primitives are represented as prefix|value| or prefix only. Prefix in AnnaDB shows the data type. For example, the string test will be represented as s|test|, where s - is a prefix that marks data as a string, and test is the actual value.

Container data types keep primitive and container objects using specific rules. There are only two container types in AnnaDB for now. Maps and vectors.

Vectors are ordered sets of elements of any type. Example: v[n|1|,n|2|,n|3|,]
Maps are associative arrays. Example: m{ s|bar|: s|baz|,}

More information about AnnaDB data types can be found in the documentation

Query

Query in AnnaDB is a pipeline of steps that should be applied in the order it was declared. The steps are wrapped into a vector with the prefix q - query.

collection|test|:q[
   find[
   ],
   sort[
      asc(value|num|),
   ],
   limit(n|5|),
];

If the pipeline has only one step, the q vector is not needed.

collection|test|:find[
   gt{
      value|num|:n|4|,
   },
];

Server

To run AnnaDB locally please type the next command in the terminal:

docker run --init -p 10001:10001 -t romanright/annadb:0.1.0

Client

AnnaDB shell client is an interactive terminal application that connects to the DB instance, validates and handles queries. It fits well to play with query language or work with the data manually.

The client can be installed via pip

pip install annadb

Run

annadb --uri annadb://localhost:10001

Usage example

You are prepared for the fun part of the article now. Let's play with AnnaDB!

I'll create a database for the candy store to show the features.

Insert primitive

Let's start with categories. I'll represent categories as simple string objects. Let's insert the first one into the categories collection.

Request:

collection|categories|:insert[
    s|sweets|,
];

collection|categories| shows on which collection the query will be applied. In our case - categories.

insert[...] - is a query step. You can insert one or many objects using the insert operation.

s|sweets| - is the object to insert. In this case, it is a string primitive. Prefix s means that it is a string, | wrap the value of the primitive. Other primitive types could be found in the Data Types section.

Response:

result:ok[
    response{
        s|data|:ids[
            categories|d2e9fecd-8b3d-429d-9a9b-34810120a221|,
        ],
        s|meta|:insert_meta{
            s|count|:n|1|,
        },
    },
];

If everything is ok, the result will have an ok[...] vector with responses for all the transaction pipelines. Each response contains data and meta information. In our case, there is only one response with a vector of ids in data and a number of inserted objects in meta.

Insert container

Let's insert a more complicated object now - a chocolate bar. It will have fields:

name
price
category

For the category, I'll use the already created one.

Request:

collection|products|:insert[
    m{
        s|name|:s|Tony's|,
        s|price|:n|5.95|,
        s|category|:categories|d2e9fecd-8b3d-429d-9a9b-34810120a221|,
    },
];

The query is similar to the previous one, but the object is not a primitive but a map. The value of the category field is a link that was received after the previous insert.

Response:

result:ok[
    response{
        s|data|:ids[
            products|17b12780-349c-4091-9bd2-7e08ad509ad0|,
        ],
        s|meta|:insert_meta{
            s|count|:n|1|,
        },
    },
];

The response is nearly the same as before - link in data and number of inserted objects in meta.

Get object

Let's retrieve the information about this chocolate bar now. I'll use the get operation for this, to the object by id

Request:

collection|products|:get[
    products|17b12780-349c-4091-9bd2-7e08ad509ad0|,
];

This time I use the get[...] query step. Using this step you can retrieve one or many objects using object links.

Response:

result:ok[
    response{
        s|data|:objects{
            products|17b12780-349c-4091-9bd2-7e08ad509ad0|:m{
                s|category|:s|sweets|,
                s|price|:n|5.95|,
                s|name|:s|Tony's|,
            },
        },
        s|meta|:get_meta{
            s|count|:n|1|,
        },
    },
];

In the response here you can see the objects{...} map, where keys are links to objects and values are objects. objects{} map keeps the order - it will return objects in the same order as they were requested in the get step, or as they were sorted by the sort step.

The category was fetched automatically and the value was returned.

Let's insert another chocolate bar there to have more objects in the collection:

collection|products|:insert[
    m{
        s|name|:s|Mars|,
        s|price|:n|2|,
        s|category|:categories|d2e9fecd-8b3d-429d-9a9b-34810120a221|,
    },
];

I use the same category id for this bar.

Modify primitive

Let's modify the category to make it more accurate.

Request:

collection|categories|:q[
    get[
        categories|d2e9fecd-8b3d-429d-9a9b-34810120a221|,
    ],
    update[
        set{
            root:s|chocolate|,
        },
    ],
];

The query here consists of 2 steps. Get the object by link step and modify this object step. The update[...] operation is a vector of update operators. Read more about the update.

Response:

result:ok[
    response{
        s|data|:ids[
            categories|d2e9fecd-8b3d-429d-9a9b-34810120a221|,
        ],
        s|meta|:update_meta{
            s|count|:n|1|,
        },
    },
];

The response of the update operation contains the ids of the updated objects as data and the number of the updated objects as meta.

Let's take a look at how this affected the chocolate objects.

Request:

collection|products|:find[
];

To find objects, I use the find[...] operation. It is a vector of find operators. If it is empty, all the collection objects will be returned.

Response:

result:ok[
    response{
        s|data|:objects{
            products|0b6ddf36-b8ba-487f-acd8-4dfee05d5177|:m{
                s|price|:n|2|,
                s|name|:s|Mars|,
                s|category|:s|chocolate|,
            },
            products|17b12780-349c-4091-9bd2-7e08ad509ad0|:m{
                s|price|:n|5.95|,
                s|name|:s|Tony's|,
                s|category|:s|chocolate|,
            },
        },
        s|meta|:find_meta{
            s|count|:n|2|,
        },
    },
];

The category was changed for both products, as the category object was linked with these objects.

Modify container

Now I'll increase the price of the bars, where it is less than 2

Request:

collection|products|:q[
    find[
        lt{
            value|price|:n|3|,
        },
    ],
    update[
        inc{
            value|price|:n|2|,
        },
    ],
];

The find step can stay before the update step as well. All the found objects will be updated. Read more about find operation and operators here.

Response:

result:ok[
    response{
        s|data|:ids[
            products|0b6ddf36-b8ba-487f-acd8-4dfee05d5177|,
        ],
        s|meta|:update_meta{
            s|count|:n|1|,
        },
    },
];

The response is similar to the previous one.

Here is how all the products look like after the update:

result:ok[
    response{
        s|data|:objects{
            products|0b6ddf36-b8ba-487f-acd8-4dfee05d5177|:m{
                s|category|:s|chocolate|,
                s|name|:s|Mars|,
                s|price|:n|4|,
            },
            products|17b12780-349c-4091-9bd2-7e08ad509ad0|:m{
                s|name|:s|Tony's|,
                s|price|:n|5.95|,
                s|category|:s|chocolate|,
            },
        },
        s|meta|:find_meta{
            s|count|:n|2|,
        },
    },
];

Sort objects

To sort objects, I'll use the sort operation against the price field.

Request:

collection|products|:q[
    find[
    ],
    sort[
        asc(value|price|),
    ],
];

The sort[...] operation is a vector of sort operators - asc and desc. Sort operators are modifiers that contain paths to the sorting value. The sort operation is not an independent step, it can stay only after find-like operations that return objects. You can read more about sort here

Response:

result:ok[
    response{
        s|data|:objects{
            products|0b6ddf36-b8ba-487f-acd8-4dfee05d5177|:m{
                s|name|:s|Mars|,
                s|price|:n|4|,
                s|category|:s|chocolate|,
            },
            products|17b12780-349c-4091-9bd2-7e08ad509ad0|:m{
                s|category|:s|chocolate|,
                s|price|:n|5.95|,
                s|name|:s|Tony's|,
            },
        },
        s|meta|:find_meta{
            s|count|:n|2|,
        },
    },
];

Objects in the response are sorted by price now.

It is useful to use limit and offset operations together with sort. You can read about them in the documentation

Delete objects

After any find-like step, you can use the delete operation to delete all the found objects. Or it can be used independently to delete the whole collection.

Request:

collection|products|:q[
    find[
        gt{
            value|price|:n|5|,
        },
    ],
    delete,
];

The delete operation is a primitive without value.

Response:

result:ok[
    response{
        s|data|:ids[
            products|17b12780-349c-4091-9bd2-7e08ad509ad0|,
        ],
        s|meta|:update_meta{
            s|count|:n|1|,
        },
    },
];

The response contains affected ids in data and the number of deleted objects in meta.

Using from your app

AnnaDB has a Python driver. It has an internal query builder - you don't need to learn AnnaDB query syntax to work with it. But it supports raw querying too.

I'll add drivers for other languages soon. If you can help me with it, I'll be more than happy :)

Plans

This is the very early version of the database. It can already do things, and I use it in a few of my projects. But there are many features to work on yet.

Drivers

I plan to add drivers to support the most popular languages, like JS, Rust, Go, and others. If you can help with this - please get in touch with me.

Rights management

This is probably the most important feature to implement. Authentication, authorizations, roles, etc.

Performance increase

There are many performance-related things to improve now.

Query features

Projections
More find and update operators
Developer experience improves

Data Types

I plan to add more data types like geo points and graph vertices to make AnnaDB more comfortable working with different data fields.

Managed service

My big goal is to make a managed data store service. Hey, AWS, Google Cloud, MS Azure, I'm ready for collaborations! ;)

Extend 3rd party classes. Rust's Implementations pattern in Python.

Roman Right — Mon, 17 Jan 2022 12:50:31 +0000

Intro

There is a pattern called Implementations in the Rust language. Using it you create methods and interfaces for the structures.

struct Number {
    odd: bool,
    value: i32,
}

impl Number {
    fn is_strictly_positive(self) -> bool {
        self.value > 0
    }
}

I decided to make the same but for Python. Meet Impler.

The basic syntax

You can extend any class with your own methods or even interface (class) using the @impl decorator.

from impler import impl


class Point:
    def __init__(self, x: float, y: float):
        self.x = x
        self.y = y


point = Point(3, 4)


@impl(Point)
def distance_from_zero(self: Point):
    return (self.x ** 2 + self.y ** 2)**(1/2)


print(point.distance_from_zero()) # 5.0

The same way you can implement class or static methods

@impl(Point)
@staticmethod
def distance(left, right):
    return ((left.x + right.x) ** 2 + (left.y + right.y) ** 2)**(1/2)

point1 = Point(1, 3)
point2 = Point(2, 1)

print(Point.distance(point1, point2)) # 5.0

You can implement the whole interface also.

Here is an example of the base interface

from pathlib import Path


class BaseFileInterface:
    def dump(self, path: Path):
        ...

    @classmethod
    def parse(cls, path: Path):
        ...

This is how you can implement this interface for Pydantic BaseModel class:

from impler import impl
from pydantic import BaseModel
from pathlib import Path


@impl(BaseModel, as_parent=True)
class ModelFileInterface(BaseFileInterface):
    def dump(self, path: Path):
        path.write_text(self.json())

    @classmethod
    def parse(cls, path: Path):
        return cls.parse_file(path)

If the as_parent parameter is True the implementation will be injected into the list of the target class parents.

Then you can check if the class or object implements the interface:

print(issubclass(BaseModel, BaseFileInterfase))
# True

Real world use-case

There is a csv file with data of some products:

+----+-----------+--------------+----------------+
|    | name      | created at   |   expire after |
|----+-----------+--------------+----------------|
|  0 | milk 3.5% | 2022-01-15   |              7 |
|  1 | sausage   | 2021-12-10   |             60 |
|  2 | yogurt    | 2021-12-01   |             20 |
+----+-----------+--------------+----------------+

I need to display if the product was expired already using pandas

For this, I need to convert dates to the number of passed days. It would be great if DataFrame would have this method. Let's add it there.

from typing import Dict

import numpy as np
import pandas as pd
from impler import impl
from pandas import DataFrame


@impl(DataFrame)
def to_age(self: DataFrame, columns: Dict[str, str], unit: str = "Y"):
    for target, new in columns.items():
        self[new] = ((pd.to_datetime("now") - pd.to_datetime(
            self[target])) / np.timedelta64(1, unit)).astype(int)
    return self


df = pd.read_csv("products.csv").to_age({"created at": "age in days"}, "D")
df["expired"] = df["expire after"] < df["age in days"]
print(df)

result:

+----+-----------+--------------+----------------+---------------+-----------+
|    | name      | created at   |   expire after |   age in days | expired   |
|----+-----------+--------------+----------------+---------------+-----------|
|  0 | milk 3.5% | 2022-01-15   |              7 |             2 | False     |
|  1 | sausage   | 2021-12-10   |             60 |            38 | False     |
|  2 | yogurt    | 2021-12-01   |             20 |            47 | True      |
+----+-----------+--------------+----------------+---------------+-----------+

As you can see, the to_age method returns self. It means this can be used in pandas methods chain, like this:

age_gt_10 = (pd.read_csv("products.csv").
    to_age({"created at": "age"}, "D").
    query("age > 10").shape[0]
)
print(age_gt_10)  # 2

Conclusion

You probably noticed that the lib logo is a warning sign:

I want to warn you here too. Please, be careful - this lib patches classes.

There are many scenarios where this tool could be used. I'd say it is good for extending 3rd party classes in research or experiment projects. It can be used in web services too but should be well tested.

Links

GH Project: https://github.com/roman-right/impler
PyPi: https://pypi.org/project/impler/
API Doc: https://github.com/roman-right/impler/blob/main/docs/api.md

Announcing Beanie ODM 1.8 - Relations, Cache, Actions and more!🎉🚀

Roman Right — Mon, 29 Nov 2021 13:16:11 +0000

I'm happy to introduce to you the new version of Beanie and a lot of new features, that come with it.

Here is the feature list:

Relations
Event-based actions
Cache
Revision

Relations

This feature is perhaps the most anticipated of all. It took some time, but finally, it is here. Relations.

The document can contain links to other documents in their fields.

Only top-level fields are fully supported for now.

Direct link to the document:

from beanie import Document, Link

class Door(Document):
    height: int = 2
    width: int = 1


class House(Document):
    name: str
    door: Link[Door]  # This is the link

List of the links:

from typing import List

from beanie import Document, Link

class Window(Document):
    x: int = 10
    y: int = 10


class House(Document):
    name: str
    door: Link[Door]
    windows: List[Link[Window]]  # This is the list of the links

Other link patterns are not supported for now. If you need something more specific for your use-case, please leave an issue on the GitHub page - https://github.com/roman-right/beanie

Write

The next write methods support relations:

insert(...)
replace(...)
save(...)

To apply the writing method to the linked documents, you should set the respective link_rule parameter

house.windows = [Window(x=100, y=100)]
house.name = "NEW NAME"

# The next call will insert a new window object 
# and replace the house instance with updated data
await house.save(link_rule=WriteRules.WRITE)

# `insert` and `replace` methods will work the same way

Or Beanie can ignore internal links with the link_rule parameter WriteRules.DO_NOTHING

house.door.height = 3
house.name = "NEW NAME"

# The next call will just replace the house instance 
# with new data, but the linked door object will not be synced
await house.replace(link_rule=WriteRules.DO_NOTHING)

# `insert` and `save` methods will work the same way

Fetch

Prefetch

You can fetch linked documents on the find query step, using the parameter fetch_links

houses = await House.find(
    House.name == "test", 
    fetch_links=True
).to_list()

All the find methods supported:

find
find_one
get

Beanie uses a single aggregation query under the hood to fetch all the linked documents. This operation is very effective.

On-demand fetch

If you don't use prefetching, linked documents will be presented as objects of the Link class. You can fetch them manually then.
To fetch all the linked documents you can use the fetch_all_links method

await house.fetch_all_links()

It will fetch all the linked documents and replace Link objects with them.

Or you can fetch a single field:

await house.fetch_link(House.door)

This will fetch the Door object and put it in the door field of the house object.

Delete

Delete method works the same way as write operations, but it uses other rules:

To delete all the links on the document deletion you should use the DeleteRules.DELETE_LINKS value for the link_rule parameter

await house.delete(link_rule=DeleteRules.DELETE_LINKS)

To keep linked documents you can use the DO_NOTHING rule

await house.delete(link_rule=DeleteRules.DO_NOTHING)

Event-based actions

You can register methods as pre- or post- actions for document events like insert, replace and etc.

Currently supported events:

Insert
Replace
SaveChanges
ValidateOnSave

To register an action you can use @before_event and @after_event decorators respectively.

from beanie import Insert, Replace

class Sample(Document):
    num: int
    name: str

    @before_event(Insert)
    def capitalize_name(self):
        self.name = self.name.capitalize()

    @after_event(Replace)
    def num_change(self):
        self.num -= 1

It is possible to register action for a list of events:

from beanie import Insert, Replace

class Sample(Document):
    num: int
    name: str

    @before_event([Insert, Replace])
    def capitalize_name(self):
        self.name = self.name.capitalize()

This will capitalize the name field value before each document insert and replace

And sync and async methods could work as actions.

from beanie import Insert, Replace

class Sample(Document):
    num: int
    name: str

    @after_event([Insert, Replace])
    async def send_callback(self):
        await client.send(self.id)

Cache

All the query results could be locally cached.

This feature must be turned on in the Settings inner class explicitly.

class Sample(Document):
    num: int
    name: str

    class Settings:
        use_cache = True

Beanie uses LRU cache with expiration time. You can set capacity (the maximum number of the cached queries) and expiration time in the Settings inner class.

class Sample(Document):
    num: int
    name: str

    class Settings:
        use_cache = True
        cache_expiration_time = datetime.timedelta(seconds=10)
        cache_capacity = 5

Any query will be cached for this document class.

# on the first call it will go to the database
samples = await Sample.find(num>10).to_list()

# on the second - it will use cache instead
samples = await Sample.find(num>10).to_list()

await asyncio.sleep(15)

# if the expiration time was reached 
# it will go to the database again
samples = await Sample.find(num>10).to_list()

Revision

This feature helps with concurrent operations.

It stores revision_id together with the document and changes it on each document update. If the application with the old local copy of the document will try to change it, an exception will be raised. Only when the local copy will be synced with the database, the application will be allowed to change the data. It helps to avoid losses of data.

This feature must be turned on in the Settings inner class explicitly too.

class Sample(Document):
    num: int
    name: str

    class Settings:
        use_revision = True

Any changing operation will check if the local copy of the document has the actual revision_id value:

s = await Sample.find_one(Sample.name="TestName")
s.num = 10

# If a concurrent process already changed the doc, 
# the next operation will raise an error
await s.replace()

If you want to ignore revision and apply all the changes even if the local copy is outdated, you can use the parameter ignore_revision

await s.replace(ignore_revision=True)

Other

There is a bunch of smaller features, presented in this release. I would like to mention a couple of them here.

Save changes

Beanie can keep the document state, that synced with the database, to find local changes and save only them.

This feature must be turned on in the Settings inner class explicitly.

class Sample(Document):
    num: int
    name: str

    class Settings:
        use_state_management = True

To save only changed values the save_changes() method should be used.

s = await Sample.find_one(Sample.name == "Test")
s.num = 100
await s.save_changes()

The save_changes() method can be used only with already inserted documents.

On save validation

Pydantic has very useful config to validate values on assignment - validate_assignment = True. But unfortunately, this is a heavy operation and doesn't fit some use cases.

You can validate all the values before saving the document (insert, replace, save, save_changes) with beanie config validate_on_save instead.

This feature must be turned on in the Settings inner class explicitly.

class Sample(Document):
    num: int
    name: str

    class Settings:
        validate_on_save = True

If any field has a wrong value, it will raise an error on write operations (insert, replace, save, save_changes)

sample = await Sample.find_one(Sample.name == "Test")
sample.num = "wrong value type"

# Next call will raise an error
await sample.replace()

Conclusion

Thank you for reading. I hope you'll find these features useful.

If you would like to help with development - there are some issues at the GitHub page of the project - https://github.com/roman-right/beanie

Announcing Beanie 1.0 - MongoDB ODM with Query Builder🚀🎉

Roman Right — Tue, 11 May 2021 07:59:59 +0000

WARNING!!
This article is outdated.
Please, follow the current documentation of Beanie to use actual features and patterns.
Link to the doc - https://roman-right.github.io/beanie/

I'm happy to introduce to you Beanie 1.0.0 - Python ODM (Object Document Mapper) for MongoDB!

Two months ago I published the very first Beanie release. You can find the article about it by the link. I demonstrated there, how simple it is, to make a CRUD service with FastAPI and Beanie.

Many features were added since then. Today I want to show the most interesting things, which came with this major version update.

For this demo, I will use the Product document model.

from beanie import Document
from pydantic import BaseModel


class Category(BaseModel):
    name: str
    description: str


class Product(Document):
    name: str
    description: Optional[str] = None
    price: Indexed(float, pymongo.DESCENDING)
    category: Category
    num: int

    class Collection:
        name = "products"
        indexes = [
            [
                ("name", pymongo.TEXT),
                ("description", pymongo.TEXT),
            ],
        ]

Beanie Document is an abstraction over Pydantic BaseModel. It helps to make documents flexible and structured at the same time.

For this demo I set up indexes here in two ways:

Simple with the Indexed field
More complex with the Collection inner class

More information about document setup could be found by link

Create documents

Beanie provides and single document creation pattern and batch inserts:

chocolate = Category(name="Chocolate")

# One
caramel_and_nougat = Product(name="Caramel and Nougat ", 
                             num=20,
                             price=3.05,
                             category=chocolate)
await caramel_and_nougat.create()

# Many
peanut_bar = Product(name="Peanut Bar", 
                     num=4, 
                     price=4.44,
                     category=chocolate)
truffle = Product(name="Chocolate Truffle", 
                  num=40, 
                  price=2.50,
                  category=chocolate)
await Product.insert_many([peanut_bar, truffle])

Find queries

Now you can use Python native comparison operators with the document class fields:

Product.find(Product.category.name == "Chocolate")

The find() method will return the FindMany query, which uses an async generator pattern, and data can be available via async for loop.

async for item in Product.find(
        Product.category.name == "Chocolate"
):
    print(item)

To retrieve the list, I use to_list() method.

products = await Product.find(
    Product.category.name == "Chocolate"
).to_list()

FindMany queries provide also sort, skip, limit, and project methods:

class ProductShortView(BaseModel):
    name: str
    price: float


products = await Product.find(
    Product.category.name == "Chocolate",
    Product.price < 3.5
).sort(-Product.price).limit(10).project(ProductShortView)

Python comparison operators don't cover all the cases. Beanie provides a list of find operators, which could be used instead:

from beanie.operators import Text

products = await Product.find(Text("Chocolate")).to_list()

The whole list of the find operators can be found here

You can use native PyMongo syntax for fine-tuning here too:

products = await Product.find({"price": {"gte": 2}}).to_list()

If you need to find a single document, you can use the find_one method instead.

product = await Product.find_one(Product.name == "Peanut Bar")

The detailed tutorial about finding the documents can be found by link

Update

update() method allows updating documents using search criteria of the FindMany and FindOne queries.

You can do it, using update operators:

from beanie.operators import Inc, Set

# Many
await Product.find(
    Product.name == "Peanut Bar"
).update(Set({Product.price: 5}))

# One
await Product.find_one(
    Product.name == "Chocolate Truffle"
).update(Set({Product.price: 3}))

# or

product = await Product.find_one(Product.name == "Peanut Bar")
await product.update(Inc({Product.price: -1}))

The list of the update operators can be found by link

Native PyMongo syntax is also supported for this

await Product.find(
    Product.num <= 5
).update({"$set": {Product.price: 1}})

There is a list of preset update operations, which could be used as methods. Increment, for example:

await Product.find(
    Product.category.name == "Chocolate"
).inc({Product.price: 2})

To update all the documents without the find query you can skip the find step:

await Product.inc({Product.price: 2})

Aggregations

Aggregations, as updates, could be used over the whole collection or, usingFindMany searching criteria.

class TotalCountView(BaseModel):
    category: str = Field(None, alias="_id")
    total: int


# Over collection

total_count = await Product.aggregate(
    [{"$group": {"_id": "$category", "total": {"$sum": "$num"}}}],
    projection_model=TotalCountView
).to_list()

# Subset

total_count = await Product.find(Product.price < 10).aggregate(
    [{"$group": {"_id": "$category", "total": {"$sum": "$num"}}}],
    projection_model=TotalCountView
).to_list()

As for the update operations, there is a list of preset methods for the popular aggregations. For example, average:

avg_choco_price = await Product.find(
    Product.category.name == "Chocolate"
).avg(Product.price)

Here you can find the doc with all the preset methods.

Delete

Delete operations support the same patterns:

# Many
await Product.find(
    Product.category.name == "Chocolate").delete()

# One
product = await Product.find_one(Product.name == "Peanut Bar")
await product.delete()

# Without fetching

await Product.find_one(
    Product.name == "Chocolate Truffle"
).delete()

Conclusion

In my first article, I said, that Beanie is a micro ODM. I'm removing the prefix micro now. Beanie is a rich Python ODM for MongoDB with a lot of features, like query builder, projections, and migrations. It helps me to build services and applications a lot. I hope, it will help many other developers too.

There is a list of interesting things, that I plan to add there. You are always welcome to participate in the development :-) Thank you very much for your time!

Beanie Projections. Reducing network and database load.

Roman Right — Wed, 21 Apr 2021 10:03:49 +0000

WARNING!!
This article is outdated.
Please, follow the current documentation of Beanie to use actual features and patterns.
Link to the doc - https://roman-right.github.io/beanie/

Today I want to introduce to you a new Beanie feature. MongoDB projections are supported now. It helps to reduce database load and makes your services more efficient.

How does it work? You can set up two document models for the same collection:

A big one - for the regular operations
A short one - for the case, when you need only a few fields

Beanie is using MongoDB projections when it takes the data from the database. It means it is asking only the data, which was described in the model. Not filtering results but asking for limited data. This is reducing, and database load, and network load.

I will show on small FastAPI service example:

This is an ice cream store api. It has an Ice Cream entity. To create and get single ice cream I will use the long model:

class Nutrition(BaseModel):
    energy: float
    fat: float
    protein: float
    carbs: float


class IceCream(Document):
    name: str
    price: float
    summary: str
    description: str
    ingredients: List[str]
    per_100_gr: Nutrition

    class Collection:
        name = "ice-cream"

And to return the list of the ice creams I'll use the short version:

class IceCreamShort(Document):
    name: str
    price: float
    summary: str

    class Collection:
        name = "ice-cream"

And the endpoints are neat as usual with FastAPI:

# CREATE ICE CREAM

@ice_cream_router.post("/", response_model=IceCream)
async def new(ice_cream: IceCream):
    await ice_cream.create()
    return ice_cream

# GET SINGLE ICE CREAM

@ice_cream_router.get("/{ice_cream_id}", response_model=IceCream)
async def get(ice_cream_id: PydanticObjectId):
    ice_cream = await IceCream.get(ice_cream_id)
    if ice_cream is None:
        raise HTTPException(status_code=404, detail="Ice cream not found")
    return ice_cream

# GET LIST OF ALL THE AVAILABLE ICE CREAMS

@ice_cream_router.get("/", response_model=List[IceCreamShort])
async def get_list():
    print(await IceCreamShort.find_all().to_list())
    return await IceCreamShort.find_all().to_list()

Let's test. For the list endpoint it returns next:

[
    {
        "_id": "607fe424447d1f704c7b12ea",
        "name": "Ben & Jerry's Netflix & Chill'd",
        "price": 7.0,
        "summary": "Peanut Butter Dairy Ice Cream with Sweet & Salty Pretzel Swirls & Brownie Pieces"
    },
    {
        "_id": "607fe670447d1f704c7b12eb",
        "name": "Chip Happens",
        "price": 6.0,
        "summary": "A Cold Mess of Chocolate Ice Cream with Chocolatey Chips & Crunchy Potato Chip Swirls"
    }
]

While for the single ice cream information it gives this:

{
    "_id": "607fe670447d1f704c7b12eb",
    "name": "Chip Happens",
    "price": 6.0,
    "summary": "A Cold Mess of Chocolate Ice Cream with Chocolatey Chips & Crunchy Potato Chip Swirls",
    "description": "Sometimes “chip” happens and everything’s a mess, but we Nailed It! with this chip-filled limited batch. When smooth chocolate ice cream meets chocolatey chips & salty swirls, they pack a serious one-two crunch. The best part? There won’t be anything left to clean up.",
    "ingredients": [
        "Cream (MILK) (26%)",
        "water",
        "condensed skimmed MILK",
        "sugar",
        "vegetable oils (coconut, fully refined soybean, sunflower)",
        "glucose syrup",
        "potatoes",
        "cocoa powder (1.5%)",
        "starch",
        "fat reduced cocoa powder (1%)",
        "free range EGG yolk",
        "vanilla extract",
        "salt",
        "emulsifier (SOY lecithin)",
        "stabilisers (guar gum, carrageenan)",
        "MILK fat",
        "natural vanilla flavouring. Sugar, cocoa, vanilla: mass balance is used to match Fairtrade sourcing, total 20% F. F Visit info.fairtrade.net/sourcing"
    ],
    "per_100_gr": {
        "energy": 277.0,
        "fat": 17.0,
        "protein": 3.7,
        "carbs": 27.0
    }
}

In this case, it reduces database and network loads several times for the list view.

The whole working project can be found by link

Links:

MongoDB indexes with Beanie

Roman Right — Wed, 24 Mar 2021 15:20:45 +0000

WARNING!!
This article is outdated.
Please, follow the current documentation of Beanie to use actual features and patterns.
Link to the doc - https://roman-right.github.io/beanie/

Beanie - Python ODM (Object Document Mapper) for MongoDB.

A few days ago Beanie 0.3.0 was released. The most important feature of this version is Indexes support. In my previous article I told about the general use-cases of this library. This time I would like to show in examples, what indexes are needed for and how to use them with Beanie.

For this demo I will create a geo service to search interesting places around. Next functions will be provided:

Upload map files to create the places. Files in .KML format.
Search for places by names and descriptions.
Search for places around based on distance.

I will use FastAPI to handle the requests. It is a very popular API framework now and fits perfectly with this demonstration.

Data Structure

To store the places I will use Beanie Document class. It is an abstraction over Pydantic BaseModel that provides methods for working with MongoDB.

from enum import Enum
from typing import Tuple

from beanie import Document
from pydantic import BaseModel


class GeoType(str, Enum):
    point = "Point"


class GeoObject(BaseModel):
    type: GeoType = GeoType.point
    coordinates: Tuple[float, float]


class Place(Document):
    name: str
    description: str
    geo: GeoObject

    class Collection:
        name = "places"

Place data model contains fields:

name - the name of the place
description - a short description of the place
geo - geo information. As a datatype, I use GeoObject here. It follows the data structure GeoJSON Point of MongoDB. More information about this can be found here

I also use the inner class Collection. It is optional. It helps to set up the MongoDB collection, where the documents are stored. For now, I'm just setting up the collection name there.

Initialisation

I'm creating the FastAPI app and init Beanie with the Place document structure. No surprises here.

# ... some code skipped

@app.on_event("startup")
async def app_init():
    client = motor.motor_asyncio.AsyncIOMotorClient(Settings().mongo_dsn)
    await init_beanie(client.beanie_db, document_models=[Place])
    app.include_router(place_router, prefix="/v1", tags=["places"])

Upload the map file

I will create an endpoint for uploading .KML map files to the service. To parse the file I will use the PyKML library.

@place_router.post("/upload/", response_model=StatusResponse)
async def places_from_file(file: bytes = File(...)):
  root = parser.fromstring(file)
  places = []
  for folder in root.Document.Folder:
    for place_mark in folder.Placemark:
      try:
        description = str(place_mark.description).strip()
      except AttributeError:
        description = ""

      place = Place(
        name=str(place_mark.name).strip(),
        description=description,
        geo=GeoObject(coordinates=str(place_mark.Point.coordinates).strip().split(",")[:2])  # long-lat
      )
      places.append(place)
  await Place.insert_many(places)
  return StatusResponse(status=ResponseStatuses.OK)

Click to see request details

POST localhost:10001/v1/upload with file

Output:

{
    "status": "OK"
}

The endpoint receives the file in bytes format and provides it to the PyKML parser. It extracts all the data, and I use it to create a list of Place objects. Then I insert all the created objects together using the batch insert method await Place.insert_many(places). It is a very efficient way for inserting data when you have many objects.

In this example, I'm using a map I found through the Google My Maps service. It is a nice map file with a collection of places in Belin. The file can be found in the demo project on github.

Text search

The next goal after uploading the map file is to make it able to search for places based on their names and descriptions. To do this I need to upgrade class Place a little:

class Place(Document):
    name: str
    description: str
    geo: GeoObject

    class Collection:
        name = "places"
        indexes = [
            [("name", pymongo.TEXT), ("description", pymongo.TEXT)],
        ]

I added TEXT index for the name and description fields. Now I can use MongoDB text search to find places:

await Place.find_many({"$text": {"$search": "coffee"}}).to_list()

This returns a list of Place objects, that have the string coffee in the name or description field. So handy.

Now I'm moving this to the endpoint and will append sorting and pagination there:

class PlacesByPhraseInput(BaseModel):
    search_phrase: str
    skip: Optional[int] = None
    limit: Optional[int] = None


@place_router.post("/search/", response_model=List[Place])
async def places_by_name(input_data: PlacesByPhraseInput):
    return await Place.find_many(
        {"$text": {"$search": input_data.search_phrase}},
        skip=input_data.skip,
        limit=input_data.limit,
        sort="name"
    ).to_list()

Click to see request details

POST localhost:10001/v1/search

Input:

{
    "search_phrase": "coffee",
    "limit": 2
}

Output:

[
    {
        "_id": "605861b0a7bad9ea7250d130",
        "name": "19grams",
        "description": "Tiny hole in the wall serving Tres Cabezas locally roasted coffee & homemade cakes. Small space that fills up in winter then opens up onto the street in the summer. Perfect spot for a coffee refresh on the way to visit the Eastside gallery & Oberbaumbrucke.                                                                           Mon- Fri 08.oo- 18.oo Sat- Sun 10.oo- 18.oo",
        "geo": {
            "type": "Point",
            "coordinates": [
                13.4447623999999,
                52.4999605
            ]
        }
    },
    {
        "_id": "605861b0a7bad9ea7250d131",
        "name": "19grams",
        "description": "After 15 years of serving house roasted coffee paired with exceptional homemade cakes. Loads of seating, in'n'out & a perfect pitstop from the weekend fleamarkets at Boxhagener Platz & RAW.                                                         Mon- 08.oo- 20.oo Sat- Sun 09.oo- 20.oo",
        "geo": {
            "type": "Point",
            "coordinates": [
                13.4687180999999,
                52.507127
            ]
        }
    }
]

The endpoint returns the list according to the same search criteria, sorts it by name, and limits by skip and limit parameters.

Geo search

Now for the tastiest part - I'm going to implement a coordinate-based search. I will create an endpoint to lookup places around by the radius. To do this, I need to add one of the geospatial indexes to the Places:

class Place(Document):
    name: str
    description: str
    geo: GeoObject

    class Collection:
        name = "places"
        indexes = [
            [("name", pymongo.TEXT), ("description", pymongo.TEXT)],  # TEXT indexes
            [("geo", pymongo.GEOSPHERE)],  # GEO index
        ]

I appended the GEOSPHERE index to the geo field. Now, to find the surrounded places I will use the aggregate method. Suppose I am at Alexanderplatz with coordinates [13.413305998382263, 52.52203686798391] and I want to find the places in a distance of 1 kilometer:

point = GeoObject(coordinates=[13.413305998382263, 52.52203686798391])
radius = 1000  # 1km

places = await Place.aggregate(
    [
        {
            "$geoNear": {
                "near": point.dict(),
                "distanceField": "distance",
                "maxDistance": radius,
            }
        }
    ],
    item_model=Place
).to_list()

Done. it returns a list of the Place objects within 1 kilometer of me. Also, I used "distanceField": "distance", parameter in the query. This means I can get the distances to each point I found. I'll update the output model a bit to achieve this:

class PlaceWithDistance(Place):
    distance: float


places = await Place.aggregate(
    [
        {
            "$geoNear": {
                "near": point.dict(),
                "distanceField": "distance",
                "maxDistance": radius,
            }
        }
    ],
    item_model=PlaceWithDistance  # Here I use new output model
).to_list()

Now it also provides a distance together with other places information. I will wrap it to the endpoint:

class PlacesAroundInput(BaseModel):
    coordinates: Tuple[float, float]
    radius: float = 1000

@place_router.post("/around/", response_model=List[PlaceWithDistance])
async def places_by_radius(input_data: PlacesAroundInput):
    point = GeoObject(coordinates=input_data.coordinates)

    return await Place.aggregate(
        [
            {
                "$geoNear": {
                    "near": point.dict(),
                    "distanceField": "distance",
                    "maxDistance": input_data.radius,
                }
            }
        ],
        item_model=PlaceWithDistance
    ).to_list()

Click to see request details

POST localhost:10001/v1/around

Input:

{
    "coordinates": [13.413305998382263, 52.52203686798391],
    "radius": 2000
}

Output:

[
    {
        "_id": "605861b0a7bad9ea7250d0ea",
        "name": "Serious Communist buildings",
        "description": "",
        "geo": {
            "type": "Point",
            "coordinates": [
                13.418898,
                52.52335
            ]
        },
        "distance": 405.9842140775255
    },
    {
        "_id": "605861b0a7bad9ea7250d0f1",
        "name": "1.Berliner DDR Motorrad Museum",
        "description": "",
        "geo": {
            "type": "Point",
            "coordinates": [
                13.4080193,
                52.5238593
            ]
        },
        "distance": 411.55085454343913
    },
    ...
]

Done!

I'm really impressed with the things MongoDB can do with geo indexes.

Conclusion

I've demonstrated, what a useful and interesting thing indexes are and how comfortable it is to use them with Beanie. For sure there are many other use-cases and many other index types in MongoDB. And Beanie supports all of them and many other things. All the Beanie methods with examples can be found in the documentation.

I've created Beanie Discord server where you can ask your questions, tell me about bugs, share ideas or just chat. Everyone is welcome there.

Resources

Demo - https://github.com/roman-right/beanie-index-demo
Beanie project - https://github.com/roman-right/beanie
Beanie Documentation - https://roman-right.github.io/beanie/
Discord - https://discord.gg/ZTTnM7rMaz

Forem: Roman Right

Studio 3T Performance Suite: How to Identify and Resolve Performance Bottlenecks in MongoDB

Introduction

Understanding the Query Profiler

Validating Improvements using Visual Explain

Optimizing with the Index Manager

Checking Results in the Query Profiler

Conclusion

MongoDB Performance Tools

Mongostat

MongoDB Compass

Mongotop

Studio 3T

Query Profiler

Visual Explain

Aggregation Editor

Conclusion

Crafting Efficient MongoDB Queries

MongoDB's Response Mechanism: The Power of the explain() Method.

What's this explain() magic?

Deciphering the Talkback

Bridging Understanding for Efficiency

Mastering Indexes in MongoDB: Balancing Adaptability and Performance.

A Deep Dive into Indexing's Role

The Colorful Palette of Indexes in MongoDB

The Delicate Balance: Tradeoffs of Indexing

MongoDB's Best Practices: Streamlining Data Operations with Native Techniques

Paging Through Your Data: From Clunky Steps to Graceful Strides

Textual Battles: Regex vs. Text Search

Choosing Wisely: The $in Operator vs. the $or Operator

4. Decoding Query Performance with Studio 3T's IntelliShell

Step-by-Step Guide:

Conclusion:

Identifying MongoDB Performance Pitfalls

The N+1 Problem: Unnecessary Resource Overhead

The Silent Delay: Unpacking the Effect of Missing Indexes

The Hidden Bumps on the Road: Other Overlooked Pitfalls

Unoptimized Aggregation Pipelines

Ignoring Sharding for Large Datasets

Overlooking Connection Pooling

Studio 3T Spotlight: Fine-Tuning Aggregation Pipeline Efficiency

Preparation

Identifying the Problem and Implementing the Solution

Conclusion

Detective de'Code and the Dueling Dishes

The Call

Tech Trio

Rhythmic Riddles

Tales from the Depths

The Stalemate Standoff

Resolution Recipe

Setting the Stage: Unraveling MongoDB Query Performance

The heart of digital interactions: Why MongoDB's speed and efficiency matter.

Anatomy of a MongoDB query: Breaking it down.

Namespace

Query Selector

Indexes

Projection

MongoDB's thought process: Delving into query execution

Query Planner

Query Executor

Studio 3T Spotlight: Visual Explain - A first look at visualizing your queries

Why Visualization Matters

Getting Started with Visual Explain

Benefits of Visual Explain

A Practical Use Case

The Indexing Decision

Studio 3T’s Indexing Flow

Beyond This Introduction

Concluding Remarks: The Journey Forward

Announcing AnnaDB - next-gen NoSQL database

Basics

Collections

TySON

Data Types

Query

Server

Client

Usage example

Insert primitive

MongoDB's Response Mechanism: The Power of the `explain()` Method.

What's this `explain()` magic?