Forem: Thibault Cheneviere

A guide through rust declarative macros

Thibault Cheneviere — Wed, 20 Mar 2024 01:15:29 +0000

Macros are a way to extend the Rust syntax. As describe in the rust book, "macros are a way of writing code that writes other code, which is known as metaprogramming".

Rust have two types of macros, procedural macros which are invoked using the #[derive(..)] syntax and declarative macros. Examples of declarative macros are vec!, println! or format!.

I'll start with examples for beginners, then I'll try to cover more advanced examples.

Why do I want to use declarative macros ?

From now on, we will refer to declarative macros as "macros". Macros can resemble functions but are different in the way that they get expanded at compile time, whereas functions are invoked at runtime. This allows to perform actions that function cannot, such as implementing traits or creating structs. Moreover, the macro syntax allows macros to take an infinite number of parameters (or tokens, we'll see that later on). For examples, we can write:

// Println take one parameter here
println!("hello");

// Println take two parameters here
println!("hello {}", "world");

That being said, I think the best way to understand fully the potential and use of macros is to write some macros. Let's start by creating our first rust macro.

An introduction to macros

Firstly, here is the syntax to declare macros in rust. We use the macro_rules! keyword:

macro_rules! a_macro {
    ( ) => { };
//  ^^^    ^^^
//   |      └── body of the rule (transcriber)
//   └── the parameters of the rule (matcher)
}

A macro is made of rules that can match different patterns (we'll see that later on in more advanced examples). A rule is made of parameters and a body. The parameters part is a matcher, which means it'll try to match with the given pattern to execute the body (transcriber). This macro can be invoked with the following syntaxes a_macro!(), a_macro![] or a_macro!{}. There is no way to force a specific syntax, it just comes down to a convention that you should keep (for example vec![] using brackets).

The matcher

The matching patterns are quite like regex. For those who are familiar with it, it'll be easier to write macros. If you're not familiar with regex, don't worry, I'll try to cover the essentials here, and you can refer to the resources down below for more information.

Note: the matcher doesn't bother with spaces. I prefer to add unnecessary spaces for readability in my macros, but it's only for this purpose, and it has nothing to do with a specific syntax.

Variables arguments

Variable arguments in macros are prefixed with $ and they have a special fragment specifier.

The structure for a variable parameter is: $ name : fragment-specifier.

Both $ and : are fixed
The name follows the Rust naming convention.
The fragment-specifiers are not Rust types. You can think of them as categories. You can find the list of fragments here

Note: fragment-specifiers are called differently and there is no consensus on their naming. From source to source, you can find the names designator or even type. Here I'll stick with the definition given in The Little Book of Rust Macros.

Fixed arguments

Fixed arguments are just arguments without $ sign and fragment-specifier. They may be useful to identify a specific rule of a macro. For example:

macro_rules! pow {
    (squared $n:expr) => {
        $n.pow(2)
    }
    (cubed $n:expr) => {
        $n.pow(3)
    }
}

fn main() {
    assert_eq!(pow!(squared 2_i32), 4);
    assert_eq!(pow!(cubed 2_i32), 8);
}

Note: some fragment specifiers need specific follow-up. For example expr can be followed by ,, ; and =>. So if I wanted to write my fixed argument after the variable one I should have written $n:expr, squared for example. You can find a complete list of the follow-ups here.

Multiple matching

As shown in the previous example, macros can contain multiple matching patterns. The patterns are matched in order of appearance, which means usually you would like to write most-specific patterns first and least-specific ones at the end. You can find a more technical explanation here.

Repetition

We've covered macros that take variable parameters and fixed arguments, but what if I want to take an undefined number of parameters like vec![] ?

Well, we can use repetition operators to match patters. We can wrap up an expression inside $() and add a repetition operator at the end. Here are the repetition operators, you'll notice how similar they are to regex:

* - indicate any number of repetition from 0 to n
+ - indicate any number of repetition from 1 to n
? - indicate zero or one repetition

Let's say we want to add a list of numbers, we can use the following macro:

macro_rules! adder {
    () => { .. };
    ($($n:expr),*) => { .. };
}

Note: the , before the * if a follow-up of the expr fragment specifier. It states that the list is comma separated. Here I could have used any expr follow-ups or none if I just wanted a space separated list. We can also note that this comma separated list don't have trailing comma. Which means adder!(1,) is a bad syntax. To allow the optional trailing comma, we need to write the matcher with the following pattern $($n:expr),* $(,)?.

The transcriber

We now have covered a lot of the matcher part, but the generation of code is made in the transcriber. This is where we can use the matched parameters to write actual code. Let's see this by implementing the adder!() macro above.

macro_rules! adder {
    () => { 0 };
    ($($n:expr),*) => {
        {
            let mut sum = 0;
            $(
                sum += $n;
            )*
            sum
        }
    };
}

fn main() {
    assert_eq!(adder!(1, 2, 3, 4), 10);
    assert_eq!(adder!(1), 1);
    assert_eq!(adder!(), 0);
}

To handle the repetition we have in our matcher, we just have to wrap up the code we want to repeat inside a $()* code block (the repetition operator should match the one used in the matcher, that is why I used * here).

But what if I want to repeat different variables. For example, add some together and multiply others together ? Well, Rust handles it and you can write the following macro:

macro_rules! calculator {
    {add $($add:expr),*; mul $($mul:expr),*} => {
       {
            let mut sum = 0;
            let mut mul = 1;
            $(
                sum += $add;
            )*
            $(
                mul *= $mul;
            )*
            (sum, mul)
        }
    }
}

fn main() {
    assert_eq!(calculator!(add 1, 2, 3; mul 1, 2, 3), (6, 6));
}

Note: it's not possible to write the sum += $add; and mul *= $mul; lines in the same loop block because the Rust compiler won't know how many times the loop needs to be repeated if we give different numbers of arguments to both add and mul.
It is also important to note that the transcriber actually returns a block (note the curly braces around).

Export

We've seen that the macro should be declared before even calling it, so how can we export macros ?

There are two possibilities. The first one is the use of the #[macro_export] derive element.

#[macro_export]
macro_rules! calculator { .. }

You can now access this macro from anywhere in your crate with crate::calculator!().

Another possibility is the use keyword.

macro_rules! calculator { .. }

pub use calculator;

This allows you to access your macro in your code, but it's still scoped in your module. For example, if the previous macro is defined in the calculator module, you can access it from main with calculator::calculator!().

A more advanced macro

Now that we have covered a lot a theory, let's try to implement a more advanced macro. Let's try to implement a macro that can calculate an expression written in reverse polish notation. In order to speed up the process, I'll show you my implementation, and I'll try to explain it.

macro_rules! rpn {
    { @inner_op stack [$r:expr, $l:expr $(, $stack:expr)*]; $op:tt $($tt:tt)* } => {
        rpn! { @inner stack [$l $op $r $(, $stack)*]; $($tt)* }
    };
    { @inner stack [$res:expr]; } => { $res };
    { @inner stack $stack:tt; + $($tt:tt)* } => {
        rpn!{ @inner_op stack $stack; + $($tt)* }
    };
    { @inner stack $stack:tt; - $($tt:tt)* } => {
        rpn!{ @inner_op stack $stack; - $($tt)* }
    };
    { @inner stack $stack:tt; * $($tt:tt)* } => {
        rpn!{ @inner_op stack $stack; * $($tt)* }
    };
    { @inner stack $stack:tt; / $($tt:tt)* } => {
        rpn!{ @inner_op stack $stack; / $($tt)* }
    };
    { @inner stack [$($stack:expr),*]; $num:tt $($tt:tt)* } => {
        rpn!{ @inner stack [$num $(, $stack)*]; $($tt)* }
    };
    { $($tt:tt)* } => {
       rpn!{ @inner stack [ ]; $($tt)* }
    };
}

As you can see, there are multiple matching patters. The ones starting with the @inner fixed arguments are internal matching patters used to create the result. They are not to be used by the user.

This leaves the end user with access to the $($tt:tt)* matching rule. This is basically a rule that accept any list of tokens. The idea is then quite simple, I followed the implementation of the polish reverse notation, I create an empty stack that I'll use to store the expressions while parsing.

Note: the stack is a list of expr separated by coma.

Let's break down each rule:

{ @inner stack [$($stack:expr),*]; $num:tt $($tt:tt)* } => {
    rpn!{ @inner stack [$num $(, $stack)*]; $($tt)* }
};

This rule is made to match every number. Every time we encounter one, we pile it up on the stack.

Next on we have the 4 rules to match operators. For example, to match the + operator we have:

{ @inner stack $stack:tt; + $($tt:tt)* } => {
    rpn!{ @inner_op stack $stack; + $($tt)* }
};

Note: we can treat the stack as a single tt (means token-tree) because it's between brackets.

The rules matching operators calls the rule marked with the @inner_op keyword. This is only to avoid code repetition.

{ @inner_op stack [$r:expr, $l:expr $(, $stack:expr)*]; $op:tt $($tt:tt)* } => {
    rpn! { @inner stack [$l $op $r $(, $stack)*]; $($tt)* }
};

This rule is the one doing the logic of the reverse polish notation. It takes the last two expressions from the stack and put them back as a new expression with the right operator.

Finally, we have the final case, where all tokens have been processed:

{ @inner stack [$res:expr]; } => { $res };

Here, I just return the expression build during the recursive calls.

At this point, if you invoke this macro with a valid reverse polish notation, it will produce a valid result.

assert_eq! { rpn! {3 4 +}, 7};
assert_eq! { rpn! {3 4 + 5 +}, 12};
assert_eq! { rpn! {3 4 5 + *}, 27};
assert_eq! { rpn! {15 7 1 1 + - / 3 * 2 1 1 + + -}, 5};

Conclusion

That's all for today. There are still some macro patterns and technical details to cover, but I'll let you read the references if you want to dive deeper in the world of Rust declarative macros.

I hope you learned some useful stuff and enjoy the reading ! Feel free to give any feedback or suggestions, it'll be really appreciated.

Finally, you can find some follow me on GitHub and find some code snippets of this article here.

References

Cover image from Tsoding Daily

How I built a Formula1 API 🏎 - part 1

Thibault Cheneviere — Wed, 24 Jan 2024 18:32:03 +0000

Building a Formula1 REST API in Rust using diesel.rs and rocket.rs.

In this guide I'm going to walk through my process of building a REST API in Rust from scratch. I will talk about my decisions in terms of project architecture, choices of frameworks or crates. I'll try to reflect my reflection process as much as I can. You can follow the development of this project on GitHub.

This guide assumes that you have a basic-to-decent understanding of Rust concepts and the language syntax.

Let's start our journey through this project !

Why a Formula1 API ?

First of all, why did I choose to even build an F1 API ? Well don't you like cars, fast cars ? Well I do, and the only open-source API available, known as the Ergast API is quite cool and complete but is deprecated and will shutdown at the end of year 2024.

So, why not create a new open-source project providing an API to access the same data ? That's how I started this project !

Building the project architecture

Now that you know why I did start this project, let's start by creating our architecture. We'll be using the workspace feature of Rust and Cargo. This feature allows to have multiple packages inside a same project that share the same Cargo.lock and target/ folder.

cargo new rust_race_engine
cd rust_race_engine
rm -rf src/

We deleted the src folder as we don't need it in the top level crate. Next, we're going to create a new project for each layer of the application. We'll have the following layers :

api layer will handle API requests and contain our handlers
application layer will handle the logic behind the API requests. It'll contain the implementation of each routes
infrastructure layer will hold our migrations and the connection pool structure
shared layer will hold any other models needed in our project, for example the responses and parameters models.

cargo new --lib api
cargo new --lib application
cargo new --lib infrastructure
cargo new --lib shared

By the end of these steps, your folder architecture should look like this.

.  
├── Cargo.lock  
├── Cargo.toml  
├── api  
│ ├── Cargo.toml  
│ └── src  
│   └── lib.rs  
├── application  
│ ├── Cargo.toml  
│ └── src  
│   └── lib.rs  
├── infrastructure  
│ ├── Cargo.toml  
│ └── src  
│   └── lib.rs  
└── shared  
  ├── Cargo.toml  
  └── src  
    └── lib.rs

We are now going to link all these projects in the top-level Cargo.toml file. We'll delete everything inside and enter the following lines :

[workspace]
members = [
    "application",
    "api",
    "infrastructure",
    "shared"
]
resolver = "2"

That ends the configuration of our project. Now we can get to work on the actual code.

Database and migrations

Database setup

The first step will be to set up the database and the diesel.rs migrations to use the database in our Rust project. As I plan to recreate an API like the Ergast API, I'll use their open-source database as a starter.
The Ergast website provides MySql dumps of their database. I'll be using docker to host my database in a container. I'll not deep dive into how to use the mysql image but here is how I start the database server :

docker run \
    --name=rust_race_engine \
    -d \
    -p 3306:3306 \
    -v $(pwd)/db/f1db.sql:/docker-entrypoint-initdb.d/dump.sql \
    -e MYSQL_DATABASE="f1db" \
    -e MYSQL_USER="f1_user" \
    -e MYSQL_PASSWORD="formula1" \
    mysql/mysql-server:latest

To quickly explain, this hosts the database found in the db/f1db.sql file, which should be the file downloaded from the Ergast website. The database can then be accessed at the following address : mysql://f1_user:formula1@127.0.0.1:3306/f1db.

We now have an up and running MySql server.

Migrations

We can now focus on the database migrations. To do so, i'll follow the migrations guide from diesel.rs. I won't repeat everything here so you can refer to it as an additional content.

First, we'll need the diesel_cli tool. We can install it using this command :

cargo install diesel_cli --no-default-features --features mysql

Next, we'll create a diesel.toml file in our top-level folder and paste the following content in it :

# For documentation on how to configure this file,
# see https://diesel.rs/guides/configuring-diesel-cli

[print_schema]
file = "application/src/schema.rs"
custom_type_derives = ["diesel::query_builder::QueryId"]

[migrations_directory]
dir = "infrastructure/src/migrations"

We'll also create a folder migrations inside infrastructure/src and place a up.sql file in it. Inside this file we'll put the content of the f1db_tables.sql file that we can find on the Ergast website. This is the file used to create the tables.
We can now generate our diesel.rs tables with the two following commands :

echo mysql://f1_user:formula1@127.0.0.1:3306/f1db > .env
diesel migration run

If everything works fine you should see a new file inside your application package called schema.rs containing all diesel.rs tables. We'll be using these tables to create the data models to query the database.

Conclusion

So far we've covered the architecture setup of our project and the migrations of the Ergast API database using diesel.rs.

I hope your learnt a lot with this article and enjoyed it ! Make sure to read the next parts to continue to build this F1 API and don't forget to star the GitHub repository and feel free to give any feedback or suggestions you have.

Thanks for reading and see you in the next part !
Thibault