Enhancing MongoDB's Performance With Horizontal Scaling

Imagine you work for a small business that just boomed with that one TikTok or Instagram reel. The application is flooded with requests, while the company was already planning on adding some new products to the application.

As a team maintaining databases for your organization, you will likely encounter an inevitable challenge when your single database instance that once handled everything smoothly comes crashing down because of increased demand. As your product catalog and customer base expand exponentially, your single MongoDB instance starts to struggle. Query response times increase, impacting user experience, and scaling vertically (adding more resources to a single server) proves to be unsustainable and costly. With the growing infrastructure demand, you as the tech decision maker need a more efficient solution to address the performance bottlenecks and deliver a better user experience.

This is where scaling your application through MongoDB’s sharding transforms the architecture, helps increase in throughput, provides better performance, and much more.

In this article, we will gain a deeper understanding of MongoDB sharding capabilities, explore the difference between horizontal and vertical scaling, and discuss how you can scale your application better.

The difference between horizontal and vertical scaling

Before we get into the details of the horizontal scaling approach, it is equally important to understand the difference between both types of scaling strategies.

From the above table, we see that in certain scenarios, vertical scaling proves inefficient because, among other things:

• The CPU limit is reached after a point and no further vertical scaling is possible.
• The single point of failure would result in a crash and application downtime.
• The interrelation between performance and cost becomes exponentially worse.
• Setting up global users from a single server location introduces latency issues.

Based on parameters like cost, future growth, reliability, flexibility, and the complexity of the application, you have enough information to choose between the horizontal or vertical approach.

Understanding sharding in databases

Sharding in a database is the concept of distributing the data across different machines. To avoid excessive load on a single machine and improve the retrieval process, the database administrator (DBA) makes the decision to divide the data into small, manageable pieces from the existing database. These pieces are known as shards. This helps in:

• Managing only a subset of reads and writes on the complete application.
• Allowing queries in different shards to operate parallelly.
• Providing better scalability.

To address the system growth, MongoDB supports horizontal scaling using sharding.

In the next section, we will get into a deeper understanding of MongoDB’s sharded clusters.

A MongoDB sharded cluster

In MongoDB, when the data is distributed into different shards, there are a few terms which are important:

1. Shard: The small pieces of the larger dataset are known as chunks. These chunks are distributed across various shards and each shard is typically a replica sets in themselves.
2. Mongos: When a request is received, a mongos is responsible to route the query to the responsible shard.
3. Config server: These maintain the metadata and configuration settings for the cluster.
4. Shard keys: This is the key parameter to divide the dataset into smaller shards. Choosing the right shard key is the most essential and important step in creating the shards. Therefore, this decision should be made wisely.

The below diagram is a representation of sharded cluster architecture used in production:

Creating a sharded cluster

To create a sharded cluster on a local deployment, follow the steps in the official MongoDB documentation. Maintaining this is not as daunting as it seems, but if you are looking for a more streamlined process, MongoDB Atlas offers a fully managed sharded cluster setup.

With MongoDB Atlas, you can deploy a sharded cluster in just a few clicks, and the platform handles all aspects of maintenance, including monitoring, backups, and automatic scaling. This allows you to focus on developing your application without worrying about the underlying infrastructure.

Monitoring and balancing a sharded cluster

Managing a sharded cluster is necessary to get optimal performance from the application. After a collection is sharded, MongoDB will try to distribute it across the shards created based on the selection of the shard key. Therefore, it becomes important to analyse the shards and understand how data is being distributed across. To help monitor the shards better, MongoDB provides basic commands like:

sh.status()  

And:

db.printSghardingStatus()  

These commands provide basic information about the shards and existing chunks in the cluster.

It is also important to note that selecting the right shard key plays an important role in creating hot shards and jumbo chunks. To help with managing these chunks, the MongoDB Sharded Cluster Balancer plays an important role. This monitors the amount of data in each shard for each collection.

In essence, effective shard key selection and regular monitoring with the balancer are vital to maintain a well-distributed and high-performing MongoDB sharded cluster.

To shard or not to shard?

When to shard

MongoDB supports horizontal scaling using sharding. The question here is: What are the ideal scenarios when sharding is a better approach?

1. Increasing data size: When the data can no longer fit into single server storage space, sharding is what one should consider as it horizontally scales and distributes the data across different shards.
2. Increase in performance: When the application is required to have high read and write rates, a single replica set would not be sufficient to do this, hence spreading across reduces the contention and improves the performance of the database.
3. Scalability requirements: If the application is expected to grow in terms of users and traffic, and it is always good to shard to handle the load in a manageable way.

While knowing when to shard your database is important, it is equally important to understand when not to shard.

When not to shard

As a software engineer, knowing the best fit is the most crucial step before we reach a decision. To decide whether to shard or not to shard, it is important to understand:

1. If the database is relatively small and fits within the storage, there is probably no requirement to shard the collection.
2. Selection of the shard key is the most important step for sharding the collection. Messing up the selection of the shard key can result in jumbo chunks and might not help improve the application.
3. Selection of the wrong shard key could also result in operating on scatter gather queries and could therefore impact the performance of the application.
4. Managing the backups and restores in a sharded collection is more complex compared to a non-sharded collection. Therefore, maintaining the backups could be difficult to manage.
5. Finally, sharding in a multi-cloud environment can result in latency problems between the shards.

Sharding best practices

If you have a sharded cluster locally or on-prem, there are a few considerations:

1. Selecting the right sharding key helps in even distribution of data and read and write operations.
2. Avoid scatter gather queries.
3. Use hash based sharding to help in uniform distribution of reads and writes.
4. Add a new shard before the current shards gets overloaded, as a precautionary measure.

Conclusion

As modern day applications scale rapidly, ensuring the database can keep up is an important step. Horizontal sharding in MongoDB offers a powerful way to scale, distribute storage, and ease the compute across multiple machines. This article aims to give you a clear and concise understanding of sharding in MongoDB, what it is, how it works, and why it matters. As your application scales and your data grows, knowing when to move from a single-node setup to a sharded cluster can make all the difference in performance and reliability.

To know more about the technical concepts, visit MongoDB’s official documentation.