From Regex Rampage to Lazy Bliss: My rjq Performance Adventure

Mainak Bhattacharjee — Sat, 12 Oct 2024 16:19:08 +0000

Hey there, fellow Rustaceans 🦀!

I've been building a JSON filter tool called rjq, inspired by the awesome jq. But things took a turn for the worse when I hit a performance wall during lexing. The culprit? Compiling regular expressions in a hot loop . It turns out, regexes are like hungry hippos – they chomp up performance if you're not careful!
Here's the story of how I tamed the regex beast and saved my program from a slow, sluggish fate:

The Regex Rampage 🦖:

At first, I naively compiled the regex patterns within the lexing loop. This meant every iteration involved creating a brand new regex object. Think of it like baking a whole new pizza for every bite – inefficient, right? This constant creation caused a major performance bottleneck i.e. ~80% execution time was consumed by this.

The Lazylock Solution 🧙‍♂️:

Thankfully, the Rust gods (and some helpful folks on the r/Rust subreddit) pointed me towards lazy_static and a technique called lazy initialization. This magic combo allowed me to compile the regex only once and store it in a thread-safe location using a LazyLock. Now, it's like having a box of pizza ready with a fresh slices whenever you need it – much more efficient!

The Lazy Bliss ✨:

The impact was phenomenal! Performance soared, and my lexing code became as smooth as butter . No more regex rampage, just happy filtering .
Want to See the Code?
Curious about the details? Head over to my GitHub repo for rjq: https://github.com/mainak55512/rjq

Lessons Learned 📚:

Regex compilation can be expensive, avoid hot loops!
Embrace lazy initialization for performance gains.
There's always a better way to do things in Rust (and life!)

So, the next time you encounter a performance bottleneck, remember – there might be a lazy solution waiting to be discovered!

P.S. If you have any other tips or tricks for optimizing JSON filtering in Rust, leave a comment below!

But wait, there's more!

Let's dive deeper into the technical aspects of this adventure.
Understanding lazy_static and LazyLock

lazy_static: This macro provides a way to declare static variables that are initialized only once, even in a multi-threaded environment.
LazyLock: This is a type provided by the lazy_static crate that ensures thread-safety during initialization.

Here's a simplified example of how I used these to optimize the regex compilation in rjq:

Outside the hot loop:

static MATCH_NUMBER: LazyLock<Regex> = LazyLock::new(|| Regex::new(r"^\d+\.?\d+").unwrap());

...and so on

Inside the hot loop:

    if MATCH_NUMBER.is_match(&source_string[cursor..]) {
        match MATCH_NUMBER
            .find(&source_string[cursor..])
            .map(|x| x.as_str())
        {
            Some(val) => {
                cursor += val.len();
                token_array.push_back(token(TokenType::NUMBER, val.to_string()));
            }
            None => (),
        }
    } else if ... so on

As you can see, the MATCH_NUMBER variable is declared using LazyLock, and it's initialized only once when the code is first executed. The LazyLock within the code ensures that the initialization is thread-safe.

Additional Performance Tips

Profiling: Use tools like perf or cargo-flamegraph to identify other performance bottlenecks in your code.
Data Structures: Choose appropriate data structures for your use case. For example, consider using HashMap for efficient lookups.
Algorithms: Optimize algorithms to reduce computational complexity.
Memory Management: Be mindful of memory allocations and deallocations.

By following these tips and leveraging techniques like lazy initialization, you can significantly improve the performance of your Rust applications.

Happy coding 🎉!

Introducing rjq: A Fast and Lightweight CLI JSON Filtering Tool

Mainak Bhattacharjee — Fri, 11 Oct 2024 06:05:31 +0000

In the world of data manipulation, JSON has become a ubiquitous format, but filtering and querying JSON data can be cumbersome without the right tools. Enter rjq, a command-line JSON filtering tool developed in Rust🦀, designed to be a performant and lightweight alternative to the popular jq tool.

The Motivation Behind rjq

rjq began as a hobby project, driven by a desire to create a tool that prioritizes performance and simplicity. With a keen focus on providing a robust alternative to jq, rjq is compatible with both Linux and Windows, making it accessible to a wider audience.
Key Features

Performance:

rjq has been optimized for speed, running nearly 2x faster than jq when tested on a Linux machine with 4GB RAM and an Intel i3 6th Gen processor. This performance boost can significantly enhance workflows, especially for users dealing with large datasets.

Simplicity:

The query structure of rjq is designed to be intuitive. Writing queries feels akin to crafting simple conditional statements in any programming language, which lowers the barrier to entry for new users.

Lightweight:

With a minimalistic approach, rjq ensures that users can quickly load and filter JSON data without unnecessary overhead.

How to Use rjq

Using rjq is straightforward. You can load JSON data from a file using the --load flag, or you can pipe input directly into the tool. Here are some usage examples:


rjq --load="test.json" --query="<query string>" --params="<comma separated parameter list>"

Alternatively, you can pipe JSON output from other commands:


stto --json cpython | rjq --query="<query string>" --params="<comma separated parameter list>"

The Development Journey

The development of rjq has been a valuable learning experience, particularly in mastering the intricacies of Rust. The support from the Reddit community was instrumental in overcoming challenges. You can check out some of the discussions and insights from fellow developers in this Reddit post.

Future Plans

Looking ahead, the goal for rjq is to become the go-to choice for JSON filtering on both Linux and Windows. rjq is still in early development stages and there are plans to incorporate new features and enhance the tool's capabilities to meet the needs of users day-to-day usage. Any contribution, stars ⭐ and fork 🔗 to the rjq repo is greatly appreciated 👍.

Who Can Benefit from rjq?

rjq is tailored for:

Developers:

Those working with JSON data who need a reliable filtering tool.

Data Analysts:

Professionals seeking efficient data extraction methods.

DevOps Teams:

Teams automating data processing tasks in their workflows.

Installation

Getting started with rjq is easy. Binaries for both Linux and Windows are available in the releases section of the GitHub repository, allowing users to install the tool without hassle.

Conclusion

Whether you’re a developer, data analyst, or part of a DevOps team, rjq offers a fast, lightweight solution for filtering JSON data. With its performance, simplicity, and growing feature set, rjq is poised to become an essential tool in your data processing arsenal. Check out the GitHub repository to learn more and get started today!

Forem: Mainak Bhattacharjee