Forem: Gracija

Comparing Data Management Tools for Robotics

Gracija — Mon, 14 Apr 2025 00:00:00 +0000

Data management is a very important element in modern robotics systems. As robots become more advanced, they generate massive amounts of information from various sources, such as sensor readings, system logs, and video feeds, and being able to manage this data effectively can mean the difference between a robot that performs well and one that fails to meet expectations.

In this article, we will compare different data management tools for robotics by looking into their key features, strengths, and the types of use cases they are best suited for. Understanding these differences will help robotics engineers and developers choose the right tool for their specific needs.

Key Criteria for Comparison

When evaluating data management tools for robotics, several critical factors must be considered to ensure the selected solution meets the unique demands of robotic systems:

Data Type Support: Different robotics applications generate and require various types of data, each with its own storage and processing needs:
- Lightweight telemetry data: Includes GPS coordinates, IMU readings, joint positions, and system status. This data requires low bandwidth and offers near real-time access, making it valuable for analytics and monitoring robot performance over time.
- Downsampled sensor data represents a middle ground when it comes to the volume. It consists of reduced framerate camera feeds and lower-resolution lidar scans, making it particularly useful for monitoring robot operations and investigating incidents without consuming excessive storage.
- Full-resolution data comprises the complete, uncompressed outputs from sensors - raw camera frames at maximum framerate, high-frequency LiDAR point clouds, and detailed depth maps. Even though this data requires a larger storage capacity, it's essential for in-depth debugging and retraining machine learning models to improve robot performance and capturing specific events for incident triage.
Integration & Ecosystem: Data management tools must integrate well with other systems and tools, such as ROS (Robot Operating System), cloud platforms, and development environments. Good integration leads to smoother data flow, which leads to reduced time and effort needed to synchronize components of the system. Compatibility with cloud platforms is also important, as it offers scalability and the flexibility to store large amounts of data without needing on-premise infrastructure.
Performance & Scalability: Fast data processing in robotics is essential, especially for tasks like autonomous navigation and robotic manipulation where even small delays can cause errors or safety concerns. Additionally, as data volumes grow, systems must scale effectively without performance loss. Storing large files, such as ROS bags, on edge devices can overwhelm resources, leading to overheating and slower performance, and transferring these files to the cloud may also cause delays. Efficient data management helps avoid these issues, ensuring efficient performance and scalability as the system grows.
Ease of Use & API Support: The data management tool's ease of use and API support can make a big difference in how easily it fits into day-to-day development. A clean, intuitive interface helps engineers and data scientists to quickly familiarize themselves with the tool and therefore increase productivity, while having reliable, well-documented API support makes it easier to automate tasks, connect with other tools, and adapt the system to match specific needs.

In-Depth Comparison of Tools

ReductStore

ReductStore is a high-performance storage and streaming solution for robotics and industrial IoT applications optimized for managing and storing large volumes of unstructured data. Its design makes it especially suitable for high-frequency sensor data, telemetry, and full-resolution sensor data like raw camera images and lidar frames coming from autonomous vehicles, industrial robots, drones, and other robotic applications.

One key aspect of ReductStore is its ability to effectively manage data both locally on edge and in a central storage (either on-prem or in the cloud). It enables efficient storage of large files, such as rosbag files, by allowing users to split them by topic and save each topic as a separate record or entry, rather than storing the entire rosbag file. This approach provides greater flexibility in managing the data, enabling different storage, replication, and retention strategies for different data types.

As edge devices often have limited storage capacity, ReductStore offers volume-based retention policies based on the First In, First Out (FIFO) principle. This approach ensures that older data is deleted only when the storage reaches full capacity. These retention policies are customizable and can be adjusted separately for each data bucket, allowing users to make different data management strategies according to their specific needs and thus avoid storage overrun on the robot.

When it comes to syncing the robot's data to a cloud or central storage, ReductStore uses conditional replication, which basically means that users can set filters in order to replicate only specific entries or a subset of records that satisfy a given set of rules. This is achieved through a JSON-like query language in the ReductStore HTTP API, enabling efficient data transfer based on customized conditions. The API also supports batch operations, reducing the number of requests and improving overall performance when transferring large datasets.

ReductStore integrates with Google Cloud Storage, offering the fastest and most cost-effective cloud storage for time series objects. It cuts data storage costs by storing infrequently accessed data in lower-cost tiers, such as Cool or Cold. Additionally, it uses the pay-as-you-go model, allowing users to pay only for the storage they use.

Foxglove & MCAP

Foxglove is a web-based visualization and observability platform for robotics data, with tight support for ROS 1, ROS 2, and MCAP logs. It supports a wide range of data types that robotics teams commonly work with, like telemetry, camera feeds, lidar point clouds, and depth maps.

For large, high-resolution data, Foxglove relies on MCAP, its open-source log format built for robotics and optimized for both speed and storage. The tool provides an efficient way to explore and visualize these MCAP files, allowing users to stream only the specific data they need.

To facilitate real-time monitoring, Foxglove provides a simple agent that automatically detects new MCAP files on the robot and sends them to the Foxglove platform. To use the Foxglove agent effectively, robots must continuously record data with rosbag, creating small, rotating files (typically 1-5 minutes each). Once a file reaches its set limit, it's closed, and a new one is created. While this works, it is highly inefficient because the robots only store data in the file system in separate files. All in all, Foxglove is a good solution for managing bag files and data visualization, but it definitely has its limitations.

Foxglove integrates with key systems and platforms commonly used in robotics development. It works great with ROS, allowing direct ingestion of data from ROS topics, services, and actions. It also supports integration with external systems via APIs, such as WebSocket and REST APIs. For cloud integration, Foxglove can connect to platforms like AWS, Azure, or Google Cloud for scalable storage solutions, ensuring that your data is accessible without local storage limitations.

Foxglove offers a clean, intuitive interface with a strong emphasis on handling large sensor and time-series data commonly found in robotics. It focuses heavily on data streaming, with interactive visualizations such as graphs, plots, and camera feeds. It also supports a "drag-and-drop" interface for easy interaction with data.

Rerun

Rerun is an open-source visualization toolbox and data platform for multimodal and temporal data. It is designed to handle a wide variety of data types such as images, point clouds, tensors, LiDAR scans, depth maps, and sensor data. Its main strength is in its ability to combine easy log handling with a fast, built-in visualizer designed for robotics and XR applications.

When working with large-scale time-series data, Rerun provides a special, Column-Oriented API (apart from its row-based API), which optimizes both data ingestion speed and memory usage. Rerun also leverages efficient data structures, like the Chunk structure, to minimize memory allocations and boost data manipulation efficiency. Furthermore, the platform allows for selective logging, enabling the capture of only necessary data.

When it comes to integration, it's important to note that Rerun currently does not have native support for ROS. However, users can still use it with ROS by implementing custom logging within their ROS nodes.

Rerun supports embedding within Jupyter notebooks and web pages and it offers integrations with native loggers in languages like Python, Rust, and C++, allowing you to stream logs directly into the Rerun Viewer.

Rerun's UI is more focused on real-time 3D visualization, with an interactive approach that allows users to explore robot environments in depth. It features easy-to-use 3D overlays and it supports live streaming and tracking, letting users explore their data in real-time and view different data types in a unified 3D space.

Heex

The biggest difference between Heex and the other platforms is that, instead of recording everything all the time, Heex focuses on capturing only what's important — like when an error happens or a specific event is triggered. This tool is built to efficiently handle large amounts of different data, such as logs, images, and sensor data, without overwhelming the users' system.

A key feature of Heex is its RDA (Resource and Data Automation) for ROS 2, which automates data capture, processing, and management in real time. It works in the following way: the robot runs a small software agent that monitors its activity and knows when to start recording based on specific rules or scenarios set by the user. These rules can be updated remotely in real-time, without the need to plug anything in or restart the robot.

The platform manages recorded data using the MCAP format—similar to Foxglove—and even embeds Foxglove's visualization tools directly into its platform, allowing users to explore data in context. Once relevant data is captured, it's sent to the cloud, where teams can review it through the Heex dashboard. This setup makes it easy to manage and analyze data across an entire fleet of robots, even if they're in different locations. Since only important moments are logged, the system runs faster, uses less bandwidth, and keeps storage manageable.

Heex integrates easily with your systems using SDK Agents for real-time data collection, APIs to connect with other tools and workflows, and a Command Line Interface (CLI) for advanced users who prefer direct control. It also integrates with ROS 1 and ROS 2, offering native logging and data streaming.

Heex's UI offers interactive dashboards where users can view and analyze data from connected robots in real time. The platform allows users to customize their dashboards to highlight specific events or metrics that are important for their workflows, such as errors, system status, and sensor readings. It also provides detailed insights into the robot's state, with easy navigation through timelines and data streams.

Comparative Analysis Table

To help visualize the differences between the various data management tools discussed, here's a comparison table that summarizes their key differences:

Tool	ReductStore	Foxglove	Rerun	Heex
Primary Focus	High-frequency unstructured time-series robotics data, large-scale storage	Real-time data streaming and visualization	Real-time 3D multimodal data visualization and XR applications	Real-time, event-driven data capture and fleet management
Data Types	High-frequency sensor data, telemetry, complex data streams	Telemetry, camera feeds, lidar, depth maps, ROS data	Images, point clouds, tensors, text, multimodal data	Logs, images, sensor data
Performance Advantages	High-throughput ingestion, time-series queries, and optimized for edge-to-cloud workloads	Web-based visualization of MCAP files with timeline syncing and stream filtering	Data model enabling efficient in-memory storage, fast access, and real-time rendering of multimodal data	Event-driven data offloading system that captures only relevant moments from continuous streams into MCAP
ROS Integration	Supports ROS (via custom setups)	Native ROS1 and ROS2 support	No native ROS support (custom integration possible)	Native ROS1 and ROS2 support
UI	Web-based UI and CLI tools	Web-based (browser UI with timeline, 2D/3D viewers)	App-based (cross-platform native app; browser UI in dev)	Web-based dashboard with filters, events, alerts

Conclusion

In this article, we covered the key challenges of managing robotics data, such as data type support, performance and scalability, integration with other systems, and ease of use. We also examined several data management platforms like ReductStore, Foxglove, Rerun, and Heex, each with its own set of strengths and disadvantages. Understanding these differences can help you make informed decisions about which solution best fits your specific needs.

Thanks for reading, I hope this article will help you choose the right storage strategy for your vibration data. If you have any questions or comments, feel free to use the ReductStore Community Forum.

3 Ways to Store ROS Topics

Gracija — Tue, 21 Jan 2025 00:00:00 +0000

The Robot Operating System (ROS) is a powerful framework for developing and managing robotic systems. It simplifies integration, communication, and development through various tools and libraries. ROS is built around a communication system that uses a publish-subscribe model to connect components, where some, like sensors or cameras, act as publishers, and others, like motors or processors, are subscribers. The data shared between these components is organized into topics.

To make the most of this data, especially when it's needed later for analysis, debugging, or sharing, it's crucial to store it efficiently. In this article, we'll dive into three methods for storing ROS topics, comparing their benefits and limitations to help you choose the best one for your needs. In case you need to gain a broader understanding of how to handle robotics data effectively, make sure to check out our article on storing and managing robotics data first.

Why Storing ROS Topics is Important

Storing ROS topics enables robotic engineers to better manage their data for various purposes, such as:

Post-Processing : After collecting data, engineers can use the data to analyze the robot's behavior, performance, and decision-making in detail, helping identify patterns or issues. They can also use the analyzed information for algorithm optimizations or for training machine learning models.
Debugging : When something goes wrong in a robot's operation, having a stored history of ROS topics can be useful for reviewing past data and pinpointing exactly when and where the problem occurred.
Reproducibility : Storing topics makes it possible to recreate the same scenario over and over again, which is useful for troubleshooting or testing with different configurations.
Documentation : Having the data that the robot produced stored somewhere can be useful for reporting, training, or improving systems based on previous results.

Each method of storing ROS topics has its strengths, so let’s dive into the three main approaches.

Method 1: Store ROS Topics to File System Using Rosbag

Rosbag is the default and most commonly used tool in ROS for handling recorded data. It works by capturing the messages being published on ROS topics and saving them into a .bag file. Since rosbag supports recording multiple topics at once, it is a great tool for scenarios where you need to capture many different data streams, such as sensor readings, images, or even logs. However, in case you don’t need the data from all the topics that exist on the system, there is an option to record only a subset of them.

Advantages :

Easy to set up : Rosbag is a built-in ROS tool, so no additional software installation is required.
Offline analysis : There is no need to connect to the live robot in order to analyze the recorded data.

Drawbacks :

Limited scalability : Since Rosbag is made for debugging but not for storing data continuously, managing and storing large Rosbag files can become very difficult as the amount of data grows.
No advanced filtering : Rosbag doesn’t provide advanced tools for filtering or querying data efficiently.
No built-in User Interface : Rosbag is a command line tool and as such, it has no built-in UI, meaning engineers have to interact with Rosbag through terminal commands.

The Bag Database, an open-source tool, addresses the challenges of working with Rosbag files by offering a user-friendly, web-based user interface for managing bag files. This tool makes it easier to search for specific bags, download them, or run post-processing scripts. Additionally, the Bag Database provides a REST API, allowing engineers to automate tasks like indexing or organizing bag files.

Method 2: Store Rosbag Data in Time-Series Object Storage

Another option is to store compressed Rosbag files in a time-series object storage solution like ReductStore. Time-series storage works really well for ROS data because it sorts and retrieves information by time, which fits naturally with how robots process and use data. This approach also lets you add helpful labels to the files, like AI labels, sensor types, or recording locations, making it easier to manage and access your data.

Advantages :

Advanced filtering : Labels make it easier to search and filter large datasets based on specific criteria.
Scalable : ReductStore can easily handle large amounts of data, making it suitable for long-term storage of robotics data.

Drawbacks :

Learning curve : Engineers new to time-series storage may need some time to set up and learn how to use them effectively.
Costs : Using cloud-based object storage can cost more than simply storing the files on your system.

Method 3: Store Topics in Separate ReductStore Records for Granular Control

This method involves storing each ROS topic as a separate record in ReductStore, giving you great flexibility over how your data is managed. By grouping data by topic, you gain way better control over filtering, searching, and replicating the data. Each record can be configured with its own compression settings depending on the type of the data that’s being sent over that topic. Moreover, having specific metadata labels for each topic makes it easier to create useful and efficient queries.

This approach also makes it easier to replicate data to the cloud. For example, you can make a rule to replicate only the most important topics - such as camera data or specific events - while excluding less relevant information.

Advantages :

Better control over your data : You can customize how each topic is stored, compressed, and labeled.
Conditional filtering and replication : With the ability to filter records based on label, you can quickly find what you need. You can also have granular replication by replicating only the important topics to the cloud.

Drawbacks :

Requires careful management : Storing topics as separate records means you'll need to plan ahead to keep track of the data and its metadata, ensuring everything stays well-organized and manageable.

Examples :

Vibration Data : Vibration sensors generate a large volume of data, which can quickly become challenging to manage. To handle this efficiently, data is often chunked and stored as objects. Keeping these chunks in separate records makes filtering and analysis much easier, especially for tracking machinery or robots where vibration patterns reveal potential issues. For more tips on managing vibration sensor data, check out this blog article on storing vibration sensor data.
LiDAR Data : LiDAR data can be quite large, so applying compression is key to reducing storage needs. There are special compression formats designed specifically for LiDAR, which help shrink the size without losing the quality of the 3D point clouds. By storing LiDAR data as a separate record in ReductStore, you can apply these specialized compression techniques to that particular topic without affecting other types of data.
Logs and JSON Data : Storing logs and JSON data in separate records allows you to organize and query them easily based on metadata labels. Logs may include useful details like timestamps, event types, or sensor statuses, while JSON is great for storing structured information. Keeping them as individual topics allows you to quickly filter and retrieve only the data you need, making analysis or debugging simpler.
Images : Imagine that the robotic system has two separate topics, one for sharing image data, and the other for position data (like X, Y coordinates). Now, what you can do is sync these two topics by storing them together - in this scenario, the images would be the data itself, while the coordinates would be the labels of the record. This allows you to link the image to specific location data, enabling you to query images based on the robot’s position. For a step-by-step guide with code examples on how to store images in ROS using ReductStore, take a look at this blog article on storing images in ROS.

Best Practices for Storing ROS Topics

When choosing how to store ROS topics, consider the following:

Data Size and Scalability : For large datasets like LiDAR or images, ReductStore's time-series storage is ideal. Simpler solutions like Rosbag are enough for smaller data, but become difficult to manage as the project grows.
Query and Filtering Needs : Storing each topic separately in ReductStore allows you to query data based on labels, enabling you to retrieve only the specific data you need. So, in case you need more granular filtering of the data, the best option would be storing each topic in its own record.
Cost and Complexity : Storing topics separately offers great flexibility, but it takes more planning and management. If you're on a budget, simpler solutions like Rosbag or storing entire bag files in a time-series storage like ReductStore can work well for smaller projects.

Conclusion

In conclusion, there are various ways to store ROS data, each with its own set of advantages and disadvantages. Choosing the right storage method for your ROS topics depends on data size, query needs, and scalability. For larger datasets and advanced filtering, ReductStore's time-series storage is a great fit. Smaller projects may find simpler options like Rosbag more suitable. Therefore, it's important to consider the specific requirement of your project and plan accordingly.

If you have any questions or comments, feel free to use the ReductStore Community Forum.

How to Store and Manage Robotic Data

Gracija — Tue, 07 Jan 2025 00:00:00 +0000

Robots generate massive amounts of data that must be managed effectively. Challenges like limited on-device storage, the need for real-time processing, and high cloud storage costs make it essential to find efficient solutions. Balancing edge and cloud storage while keeping data synchronized is a key part of effective management.

This article begins by outlining these challenges and offering practical strategies, such as using time-series databases and implementing retention policies. We will then introduce ReductStore, a specialized database designed to meet the unique needs of robotic systems. With features like real-time ingestion, efficient querying with batching, smart retention policies, and edge-to-cloud replication, ReductStore offers a cost-effective and high-performance solution for storing and managing robotic data.

We’ll also explore a hand-on example where we’ll show how you can set up ReductStore and use it for storing and managing data. Finally, we will compare ReductStore with MongoDB, explaining why ReductStore is the better choice for robotics. This comprehensive guide is designed to help engineers and developers overcome the challenges of robotic data management and optimize their systems.

What we’ll cover:

Challenges in Robotic Data Management

Robots often operate in dynamic and unpredictable environments and are continuously generating large amounts of data. Therefore, finding an efficient way to store and manage this data can at times be challenging, mainly due to the following factors:

High Frequency and Real-Time Requirements : Robots often operate in real-time environments; for example, a drone navigating through a city must process camera and sensor data in milliseconds to avoid potential obstacles and stay on track. Data storage solutions must be able to manage these high-frequency streams and make them easily and quickly accessible in order to ensure fast analysis and good decision-making.
Limited On-Device Storage : Most robots cannot store all the data they generate because they have limited edge storage capacity due to size, weight, and power limitations. That’s why it’s important for engineers to make effective data management strategies and ensure that important data is maintained without exceeding storage limits.
High Volume of Data : An autonomous vehicle can produce up to 5 terabytes of data every hour, including camera feeds, LiDAR scans, radar data, GPS logs, and sensor readings. Processing, storing, and managing such large datasets requires storage systems designed for scalability, which is something traditional relational databases often lack.
Cloud Storage Costs : Sending all robotic data to the cloud is impractical and costly. Cloud services charge for storage and data transfer, and since robots generate many terabytes of data, the costs can grow quickly. Balancing what to store locally and what to offload to the cloud is an important part of cost-effective data management.
Reduction Challenges : Implementing good reduction strategies can be tricky, as it requires balancing storage efficiency with maintaining important information. Without clear strategies, valuable data might be lost, or storage could fill up with unnecessary details.

General Strategies for Managing Robotic Data

In order to have an effective data management system for handling robotic data, engineers must have a clear strategy and use various specialized tools. The key is to implement solutions that can handle high-frequency data streams while at the same time optimizing storage space and costs.

Time-Series Object Stores : Time-series object stores, such as ReductStore, are designed to handle high-frequency, time-stamped unstructured data streams, which makes them ideal for managing robotic data. They offer efficient storage, real-time querying, and high-performance data writing, ensuring that robots can operate effectively while keeping costs manageable.
Balance Edge and Cloud Storage : To optimize storage, keep critical data - such as the latest logs or details of high-priority events - on edge devices for immediate access, while moving less time-sensitive data to the cloud for long-term analysis and archiving. With such hybrid model, you can definitely reduce cloud storage, while also making sure that there will be a minimal delay when accessing decision-making data.
Implement Retention Policies : Retention policies are automated rules for data lifecycle management, and they help manage storage by automatically deleting or archiving outdated or non-essential data. Robot's Edge devices are quite space-constrained, and often we just want to keep as much history as possible. The best retention policy in this case is a FIFO quota based on storage volume. For example, an entry with a 1TB FIFO quota would not delete older records until the quota was full.
Compress and Optimize Data : Another way to reduce storage but still maintain usability is to compress large datasets, such as images or videos. Advanced compression formats like H.265 for video and JPEG for images allow robotics teams to save loads of storage space.
Prioritize Data Relevance : Not all robotic data holds the same value, so it’s important for engineers to know how to make the distinction between what’s considered critical data and should be kept, and what data is not important and can be discarded.

ReductStore: A Purpose-Built Solution for Robotic Data

In this section, we’ll get into more detail about what ReductStore actually is and why it is such a good solution when it comes to storing and managing high volumes of robotic data.

Key Features and Benefits

ReductStore is a high-performance, time-series database specifically designed to handle the complex data needs of robotics systems and the challenges that we’ve discussed about in the previous sections. It is designed to handle unstructured data and is optimized for real-time data ingestion, allowing it to process high-frequency sensor readings and large data streams generated by autonomous vehicles, drones, industrial robots, and more.

Cost-Efficiency

ReductStore can actually achieve 10-100x better performance for 1/10th of the cost compared to traditional time-series object stores when working with records of around 100KB. But how exactly does it do it? Well, firstly, it can integrate with blob storages like Azure Blob Storage or Google Cloud Storage with a Fuse driver. In the example below, we’ll be using Azure Blob Storage.

Azure Blob Storage : A scalable object storage service optimized for large amounts of unstructured data.
BlobFuse : An open-source virtual file system driver for Azure Blob Storage.

ReductStore leverages Azure's storage tiers and stores infrequently accessed data in lower-cost tiers like Cool or Cold, and it also utilizes a pay-as-you-go model. Moreover, traditional databases require high IOPS (Input/Output Operations per Second) and often rely on expensive storage options like SSDs to maintain performance. ReductStore can store data in blob storage using FUSE drivers, eliminating the need for redundant, costly storage setups. BlobFuse allows ReductStore to treat blob storage like a local file system, meaning that data can be quickly accessed without having duplicate data in other layers like caches or local databases. Plus, unlike traditional databases, Blob Storage can scale virtually infinitely. ReductStore takes advantage of this scalability without additional costs for maintaining database indexes or clusters.

When it comes to data replication, ReductStore allows replication at the bucket level, not the entire dataset, meaning we may only replicate high-priority sensor data or logs while leaving unimportant data untouched. This minimizes storage and network costs compared to traditional databases that replicate entire datasets. In addition, ReductStore replicates data incrementally, which ensures that only changes or new data will be replicated and in this way avoids extra costs for potential duplicate files. Additionally, ReductStore supports storing summary metrics as labels alongside each record. These labels act as metadata, making it easier to filter important records and design efficient replication strategies for selectively replicating only the most critical data to the cloud.

Query Language and Batching Capabilities

ReductStore uses a flexible and powerful query language that is based on JSON syntax and is optimized for robotics data. The query language allows for real-time and historical analysis of data, as well as powerful filtering, aggregation, and joining of datasets. For robotics applications, where performance and precision are critical, the ability to run complex queries in real-time is essential.

The query language also supports time-series operations, which allow engineers to analyze sensor data over specified time windows. For example, you can query for vibration data from a specific robot within a defined timeframe or compare sensor data from different robots in parallel.

Additionally, ReductStore provides APIs and SDKs for integration with popular programming languages, making it easy to interface with robotic systems and third-party software applications. Whether you're working in Python, C++, JavaScript, or Rust, ReductStore ensures that you can access and manipulate data with minimal overhead.

ReductStore optimizes data retrieval through batching, allowing multiple records to be grouped into a single query based on a time range. This reduces the number of requests, improving performance and reducing delays. Queries are processed iteratively, returning batches of records per iteration, which is beneficial for managing large datasets. For more details, visit the official guide on data querying.

Replication Strategies and Edge-To-Cloud Syncing

In this example, there are a couple of industrial robots that produce a large amount of data which then gets ingested by Kafka and then distributed to a local storage on the edge. This is where the replication comes into play - in order to optimize costs, only a part of the data that’s stored locally is actually going to be stored on the cloud. For example, if we ingest and store sensor data every second, only 1 in 10 seconds are actually going to be stored on the cloud, as well as some common metrics which may be useful for analysis later on. Due to the fact that we can create custom policies for the replication strategy, ReductStore offers a great solution to the edge-to-cloud syncing problem and shows that it is possible to have great performance while still significantly cutting on costs.

Retention Strategies

As mentioned in the previous section, implementing effective retention policies is crucial for ensuring that the local storage doesn’t fill up and lead to potential bottlenecks, but we also must make sure that all the data that we need quick access to is actually stored at the edge.

ReductStore implements volume-based retention policies that follow the FIFO (first-In, First-Out) principle. That means that old data will be deleted only when the storage is at full capacity in order for new data to be stored. These retention policies are customizable and can be adjusted separately for each data bucket.

This is different from how most of the other databases are handling retention – they’re implementing them based on time periods. This can be problematic because even if the system was not working for some time due to an outage or any other problem and the storage is not filled up, the database will delete data that it mustn’t have deleted, only because the predefined timeframe has passed.

Example Applications

Autonomous Vehicles : Autonomous vehicles generate massive amounts of sensor data that need to be processed and analyzed in real-time. ReductStore’s ability to handle time-series data and support high-throughput ingestion makes it an excellent solution for managing this data.
Industrial Robots : In manufacturing, robots rely on real-time sensor data to adjust operations and monitor performance. ReductStore is ideal for storing and analyzing diagnostic data, enabling predictive maintenance and continuous optimization.
Drones and UAVs : Drones often operate in remote areas with limited connectivity. ReductStore’s offline data synchronization and edge storage capabilities allow for reliable operation, even in environments with poor network coverage.

MongoDB vs. ReductStore for Robotics

MongoDB and ReductStore are both robust databases, but they are built for different purposes. ReductStore is great at handling high-speed time-series data, making it perfect for storing unstructured “blob” data. It uses buckets to organize data by time, which helps with fast real-time processing. MongoDB, on the other hand, is a flexible database for semi-structured data, but it struggles with big files unless you use GridFS, which can slow things down.

With features like batch querying, smart edge-to-cloud syncing, and simple retention policies to avoid storage issues, ReductStore is a perfect solution for robotics applications, where data is processed locally and only important pieces are sent to the cloud. MongoDB has good features like replication and time-based data deletion, but it’s not as well-suited for these specific needs. Moreover, in performance tests, ReductStore was up to 900% faster for writing data and 2300% faster for reading compared to MongoDB.

For a deeper analysis, explore the full comparison here: MongoDB vs ReductStore: Choosing the Right Database for Robotics Applications.

Hands-On Example: Storing and Handling Robotic Data in ReductStore

In this section, we’ll walk through a practical example of how to store and handle robotic data using ReductStore. ReductStore is very efficient for storing “episodes” - for example, 10-second raw data with all the logs generated by robots that would be around 100KB. For simplicity reasons, we’ll show how to store trajectory data, but in reality, engineers will most probably work with bigger and more diverse data.

For this example, we’ll use Python, so make sure you have Python 3.8+ installed in your environment.

Setting Up ReductStore

We’ll start by using Docker to set up ReductStore in a containerized environment. Docker ensures that our setup is isolated, reproducible, and easy to manage. To get started, create a folder on your system, and inside of it, create a docker-compose.yaml file with the necessary configuration to run ReductStore:

version: "3.8"

services:
  reductstore:
    image: reduct/store:latest
    ports:
      - "8383:8383"
    volumes:
      - data:/data
    environment:
      - RS_API_TOKEN=my-token

volumes:
  data:
    driver: local

Then, to start the ReductStore container, simply run:

docker compose up -d

This command will pull the latest ReductStore image and start the container. Once the container is up, ReductStore will be available on port http://127.0.01:8383 and can be accessed by your robotic system.

You can check the status of your container by running:

docker ps

One last thing you need to do before we begin is make sure you have the necessary libraries installed by running:

pip install reduct-py numpy

Store and Manage Data

Now that ReductStore is running and we have everything that we need installed, let's look at how we can store robotic data. In our example, we’ll work with trajectory data like coordinates, speed, and orientation.

Create (or Get) a Bucket

Let us first create a new bucket for storing the robot trajectory data named trajectory_data.

async def create_trajectory_bucket():
 async with Client("http://localhost:8383", api_token="my-token") as client:
  settings = BucketSettings(
     quota_type=QuotaType.FIFO,
     quota_size=1000_000_000,
    )
  await client.create_bucket("trajectory_data", settings, exist_ok=True)

The quota type of this bucket will be FIFO and its size will be 1 GB. In case we don’t define this as a setting, the bucket will be created with a default quota type of NONE.

Generate Trajectory Data

Now, let’s create a generate_trajectory_data function that simulates the generation of trajectory data points for a robot moving in a 2D space.

async def generate_trajectory_data(frequency: int = 10, duration: int = 1):
 interval = 1 / frequency
 start_time = datetime.now()

  for i in range(frequency * duration):
    time_step = i * interval
    x = np.sin(2 * np.pi * time_step) + 0.2 * np.random.randn()
    y = np.cos(2 * np.pi * time_step) + 0.2 * np.random.randn()
    yaw = np.degrees(np.arctan2(y, x)) + np.random.uniform(-5, 5)
    speed = abs(np.sin(2 * np.pi * time_step)) + 0.1 * np.random.randn()
    timestamp = start_time + timedelta(seconds=time_step)

    yield {
      "timestamp": timestamp.isoformat(),
      "position": {"x": round(x, 2), "y": round(y, 2)},
      "orientation": {"yaw": round(yaw, 2)},
      "speed": round(speed, 2),
    }
    await asyncio.sleep(interval)

The frequency is the number of data points generated per second (default is 10 Hz), while the duration is the total time, in seconds, for which the data generation will run (the default is 1 second). X and y are the coordinates and they dictate the position of the robot, yaw is its orientation, speed is its approximate speed based on position changes, while timestamp is the current timestamp for the current data point.

Calculate Metrics

Now, we would like to calculate some important metrics that can be useful for further analysis of our robots. In this example, we will calculate the average speed and total distance of the robot by analyzing the data that we generated in the previous section:

def calculate_trajectory_metrics(trajectory: list) -> tuple:
  positions = np.array([[point["position"]["x"], point["position"]["y"]] for   point in trajectory])
  speeds = np.array([point["speed"] for point in trajectory])

  deltas = np.diff(positions, axis=0)
  distances = np.sqrt(np.sum(deltas**2, axis=1))
  total_distance = np.sum(distances)

  average_speed = np.mean(speeds)

  return total_distance, average_speed

Storing data in ReductStore

async def store_trajectory_data():
  trajectory_data = []
  async for data_point in generate_trajectory_data(frequency=10, duration=1):
    trajectory_data.append(data_point)

  total_distance, average_speed = calculate_trajectory_metrics(trajectory_data)

  labels = {
    "total_distance": total_distance,
    "average_speed": average_speed
  }

  packed_data = pack_trajectory_data(trajectory_data)

  timestamp = datetime.now()

  async with Client("http://localhost:8383", api_token="my-token") as client:
    bucket = await client.get_bucket("trajectory_data")
    await bucket.write("trajectory_data", packed_data, timestamp, labels=labels)


def pack_trajectory_data(trajectory: list) -> bytes:
  """Pack trajectory data json format"""
  return json.dumps(trajectory).encode("utf-8")

In the store_trajectory_data function, we’re making use of the functions we previously created to generate the data and calculate the metrics. Then, we need to somehow pack the data, therefore we create the pack_trajectory_data function where, as an example, we convert the trajectory list output to a json dump. Next step is to store that data in the bucket we have already created. The fact that we’re calling the functions for generating the data and calculating the metrics inside of the store_trajectory_data function is due to simplicity reasons, but in general, these functions should be called from the outside and then simply fed to this function.

Now, if we take a look at the Buckets section in ReductStore, we’ll see that our bucket now has data inside.

Retrieving Data

The next step would be to retrieve the data we just stored queried by a specific label. Here’s the query:

async def query_by_label(bucket_name, entry_name, label_key, label_value):
  async with Client("http://localhost:8383", api_token="my-token") as client:
    bucket = await client.get_bucket(bucket_name)

    async for record in bucket.query(
      entry_name,
      when={
        label_key: {"$gt": label_value}
      },
    ):
      # Do something with the record

Note that in order to get all records with no filtering at all, you just have to remove the ‘when’ condition.

Running Everything

This is our main function in which we’re calling the functions for creating the bucket, storing the trajectory data (which inside includes functions for generating the data and its labels), and retrieving records filtered by a specific label.

async def main():
  await create_trajectory_bucket()
    await store_trajectory_data()
    label_query_result = await query_by_label("trajectory_data", "trajectory_data", "&total_distance", HIGH_DISTANCE)
    if label_query_result:
      print(f"Data queried by label: {label_query_result}")

asyncio.run(main())

Conclusion

Managing robotic data doesn't have to be complicated. With the right tools and strategies, you can handle large amounts of data efficiently while keeping costs low. ReductStore offers a practical solution that meets the unique needs of robotics systems, from real-time processing to smart storage management. By implementing these approaches, robotics teams can focus less on data management and more on building better robots.

Ready to improve your robotic data management? Try ReductStore today at reduct.store or check out our documentation to get started.

Thanks for reading. If you have any questions or comments, feel free to use the ReductStore Community Forum.

MongoDB vs ReductStore: Choosing the Right Database for Robotics Applications

Gracija — Tue, 26 Nov 2024 00:00:00 +0000

Robotics applications generate and process a wide variety of data, such as sensor readings, video streams, logs, and AI model outputs. Managing this data efficiently is crucial because it affects the performance, scalability, and reliability of the entire system.

In this article, we'll compare ReductStore and MongoDB , two databases designed to handle different aspects of data management. ReductStore is a time-series blob storage solution optimized for managing large amounts of data coming from continuous streams. MongoDB, on the other hand, is a popular NoSQL database known for its flexibility, scalability, and support for unstructured and semi-structured data.

By understanding the strengths and limitations of each, you can make an informed decision to meet your project's specific data needs.

TL;DR;

Feature	ReductStore	MongoDB
Architecture	Time-series string messages with metadata stored as key-value pairs	Collections of JSON-like documents
Data Querying	Query requests via time ranges with filters	MongoDB Query Language (MQL)
Communication Protocol	Uses HTTP/HTTPS	MongoDB Wire Protocol (binary, BSON-based)
Data Batching Capabilities	Supports data batching	Supports data batching, but it’s not optimized for time-series data
Replication	Granular data filtering support	Replicates the entire dataset; no native support for filtering during replication.
Retention	Retention policy based on data volume with ‘First-in, First-out’ (FIFO) deletions	Uses TTL (Time to Live) indexes for time-based data deletion
Performance	Optimized for unstructured data; up to 900% faster for writes and 2300% faster for reads.	General-purpose; slower with large blobs.

Understanding the General Architectural Differences

First, let's look at MongoDB and ReductStore's differences in terms of their general architectural principles.

ReductStore has a specialized approach to managing unstructured, “blob” data organized by time. Its architecture revolves around buckets, entries, and records—concepts comparable to databases, tables, and rows in SQL systems. This structure, combined with fixed-size blocks for chronological data storage, makes ReductStore highly efficient for real-time processing and retrieval of data coming from continuous streams. It's also important to be mentioned that ReductStore has no limits to the size or the type of the data you can store.

MongoDB , in contrast, uses a document-oriented model based on collections of JSON-like documents. This flexible structure supports complex hierarchies and adapts well to applications with dynamic or loosely structured data, such as e-commerce platforms or content management systems. While MongoDB has a 16MB document size limit, it can handle larger files using GridFS, a feature designed to store and retrieve large binary files by splitting them into smaller chunks. However, storing data with GridFS is slower compared to working with standard JSON documents, which can affect performance in applications requiring frequent or rapid access to large files.

Development Experience

In this section, we will compare the development experience of MongoDB and ReductStore, focusing on their query languages, communication protocols, and integration capabilities.

Query languages and SDK support

ReductStore supports querying data in several different ways – either by using its CLI, SDKs, or HTTP API. This database solution improves data retrieval by batching records within a time range, minimizing requests and delays. Apart from querying by a time range, there is a possibility to retrieve a record by its timestamp, but you have to keep in mind that such querying is less efficient due to the lack of batching. ReductStore provides client SDKs for Python, JavaScript, C++, and Rust.

MongoDB uses its own query language, MongoDB Query Language (MQL), which allows for flexible and powerful data querying. MQL supports complex filtering, aggregation, and indexing, making it a great choice for applications that need to work with semi-structured data. MongoDB also provides official SDKs for various languages, such as Java, Python, and C#, making it easy to integrate with various development environments.

Communication protocols

ReductStore uses HTTP/HTTPS protocols to handle data, focusing on reducing network load in environments with high latency. It processes large datasets by batching requests and delivering data in segments, making it efficient for both real-time and historical data access.

On the other hand, MongoDB uses its native MongoDB Wire Protocol, a lightweight, binary protocol optimized for fast communication between clients and servers. This protocol encodes data in BSON (Binary JSON), which supports efficient transmission and storage of JSON-like documents. These design choices make MongoDB a great choice for web-based applications, offering both flexibility and scalability.

Data Management Features

Data batching capabilities

Robots often generate high-frequency data from sensors and cameras, which requires efficient data transfer to avoid bottlenecks.

ReductStore enables efficient data batching by grouping multiple records into a single HTTP request, minimizing delays and reducing the number of queries needed. ReductStore optimizes data retrieval by batching multiple records into a single HTTP request, reducing network overhead and delays. The storage system ensures data records are batched optimally for the HTTP protocol, supporting use cases such as real-time monitoring and historical data analysis with minimal query overhead.

MongoDB , provides bulk write operations that enable inserting, updating, or deleting multiple documents in a single transaction. This is particularly useful for applications that need to modify large datasets without compromising performance. MongoDB's aggregation pipelines also support batch data processing, but this batching is more general and not a good option for time-series data.

Edge-to-Cloud Replication

In robotics, data is often processed locally (on edge devices) to reduce latency and dependence on cloud connectivity. However, important data still needs to be replicated to the cloud for centralized analysis, long-term storage, or to train machine learning models.

ReductStore offers granular control over data filtering during replication, making it easy to move specific subsets of data between edge devices and the cloud. This feature allows efficient syncing, ensuring only necessary data is transferred to reduce bandwidth usage. This makes ReductStore particularly well-suited for applications like robotics where data is processed in real-time at the edge, and only relevant or high-priority data needs to be replicated to the cloud for analysis or storage.

MongoDB's replication system mainly focuses on maintaining data consistency and fault tolerance across multiple replica sets. While it is very reliable and flexible for general-purpose applications, it is not specifically suited for edge-to-cloud workflows since it does not natively allow data filtering during the replication process.

Retention policies

Robotic systems generate large amounts of data, much of which quickly becomes irrelevant. Efficient retention policies are a must to maintain performance and avoid disk space issues on edge devices.

ReductStore's retention policies are based on volume thresholds, meaning that older entries are automatically removed as new ones are added. The platform follows a FIFO (first-in, first-out) rule, making it a perfect solution for applications where only recent data needs to be kept, such as sensor readings or telemetry logs. These retention policies can be customized for each data bucket, which ensures efficient data management.

MongoDB uses TTL (Time to Live) indexes to delete documents once they reach a certain age. This feature isn't automatic and requires setup, but it lets developers manage expiration policies for individual collections. TTL indexes are especially useful for applications like session storage, temporary logs, or data with defined lifespans.

Performance Analysis

MongoDB is often better suited for structured numerical data but requires a more complex setup with GridFS for handling blob storage. On the other hand, ReductStore simplifies the process by storing binary data directly into buckets, which makes it easier to manage time-series data.

In terms of performance, our analysis shows that ReductStore greatly outperforms MongoDB when handling large blobs of unstructured data. In write tests, ReductStore is up to 900% faster than MongoDB, while for read operations, it delivers up to 2300% faster results. In the table below, you can find the results of the tests conducted by measuring the speed of writing and reading of binary data of varying sizes on an NVMe drive.

Chunk Size	Operation	MongoDB, blob/s	ReductStore, blob/s	ReductStore, %
1 KB	Write	799	7994	+900%
	Read	2008	47979	+2300%
10 KB	Write	783	7431	+850%
	Read	1918	32888	+1600%
100 KB	Write	694	3612	+420%
	Read	1730	6250	+260%
1 MB	Write	246	663	+170%
	Read	776	540	-30%

Real-World Example

For a real-world robotics system, consider a scenario where numerous robots are operating on a manufacturing shop floor. These robots are continuously generating data, such as sensor readings, status updates, and environmental information. Kafka is used for ingesting this data in real-time and directing it into a local storage system for temporary retention and processing.

As the amount of data accumulates, it is very important to efficiently replicate this data to the cloud for further analysis and long-term storage. This is where ReductStore comes into play. The data replication strategy involves decreasing the volume of data by filtering out less important information automatically. By using ReductStore's edge-to-cloud replication and data filtering capabilities, only the most relevant time-series data is transmitted to the cloud, making sure that resources are used efficiently and bandwidth is optimized. This reduction strategy can be applied by using predefined rules or dynamic triggers, such as certain events or changes in sensor readings, helping improve performance and save on cloud costs.

Conclusion

ReductStore and MongoDB each have their own strengths, making them a good fit for different needs and use cases. MongoDB is great for flexible, general-purpose storage, especially with structured data, but when it comes to handling large-scale, time-sensitive data—like the kind found in robotics—ReductStore really shines. With optimized performance, fast read/write speeds, and features like edge-to-cloud replication, ReductStore is better suited for managing large, unstructured data sets. So, while MongoDB is a solid choice for many use cases, ReductStore stands out for robotics applications that need efficient, real-time data management.

I hope this tutorial has been helpful. If you have any questions or feedback, don't hesitate to use the ReductStore Community forum.

Forem: Gracija

Comparing Data Management Tools for Robotics

Key Criteria for Comparison​

In-Depth Comparison of Tools​

ReductStore​

Foxglove & MCAP​

Rerun​

Heex​

Comparative Analysis Table​

Conclusion​

3 Ways to Store ROS Topics

Why Storing ROS Topics is Important​

Method 1: Store ROS Topics to File System Using Rosbag​

Method 2: Store Rosbag Data in Time-Series Object Storage​

Method 3: Store Topics in Separate ReductStore Records for Granular Control​

Best Practices for Storing ROS Topics​

Conclusion​

How to Store and Manage Robotic Data

Challenges in Robotic Data Management​

General Strategies for Managing Robotic Data​

ReductStore: A Purpose-Built Solution for Robotic Data​

Key Features and Benefits​

Cost-Efficiency​

Query Language and Batching Capabilities​

Replication Strategies and Edge-To-Cloud Syncing​

Retention Strategies​

Example Applications​

MongoDB vs. ReductStore for Robotics​

Hands-On Example: Storing and Handling Robotic Data in ReductStore​

Setting Up ReductStore​

Store and Manage Data​

Create (or Get) a Bucket​

Generate Trajectory Data​

Calculate Metrics​

Storing data in ReductStore​

Retrieving Data​

Running Everything​

Conclusion​

MongoDB vs ReductStore: Choosing the Right Database for Robotics Applications

Understanding the General Architectural Differences​

Development Experience​

Query languages and SDK support​

Communication protocols​

Data Management Features​

Data batching capabilities​

Edge-to-Cloud Replication​

Retention policies​

Performance Analysis​

Real-World Example​

Conclusion​

Key Criteria for Comparison

In-Depth Comparison of Tools

ReductStore

Foxglove & MCAP

Rerun

Heex

Comparative Analysis Table

Conclusion

Why Storing ROS Topics is Important

Method 1: Store ROS Topics to File System Using Rosbag

Method 2: Store Rosbag Data in Time-Series Object Storage

Method 3: Store Topics in Separate ReductStore Records for Granular Control

Best Practices for Storing ROS Topics

Conclusion

Challenges in Robotic Data Management

General Strategies for Managing Robotic Data

ReductStore: A Purpose-Built Solution for Robotic Data

Key Features and Benefits

Cost-Efficiency

Query Language and Batching Capabilities

Replication Strategies and Edge-To-Cloud Syncing

Retention Strategies

Example Applications

MongoDB vs. ReductStore for Robotics

Hands-On Example: Storing and Handling Robotic Data in ReductStore

Setting Up ReductStore

Store and Manage Data

Create (or Get) a Bucket

Generate Trajectory Data

Calculate Metrics

Storing data in ReductStore

Retrieving Data

Running Everything

Conclusion

Understanding the General Architectural Differences

Development Experience

Query languages and SDK support

Communication protocols

Data Management Features

Data batching capabilities

Edge-to-Cloud Replication

Retention policies

Performance Analysis

Real-World Example

Conclusion