Forem: Jae Anne Bach Hardie

Learning Rust 2: A Tiny Database is Born

Jae Anne Bach Hardie — Sun, 30 Aug 2020 20:33:55 +0000

Hello again! Tonight we're drinking a Chardonnay from Murcia, Spain, so shoutout to my home country hope you're holding up I hear things have been rough lately. It's pleasantly dry if not particularly complex, tasting mostly like crunchy green apples.

Another thing that won't be particularly complex is my plan for the next steps in this project. Last time I said I was going to attempt to make a VCS — Version Control System, like Git or Mercurial — in Rust. We got as far as getting things running enough to read a file and print out it's contents, so not very far at all. Today I want to get away from just toying with the language and see if I can make something that's at least theoretically useful without being unachievable. When tackling a large project — especially when your only motivation is to learn things and maybe write silly blog posts — it's important to have small milestones that build on each other so you can evaluate your progress and learn as you go.

I've decided that my first milestone is to make a non-branching, client-only VCS. It's like you had a git repository where you only have your main branch and there's no remote to push to. It's still useful in that you can view all previous states of the repository and roll back to them but it has no collaboration features. This is pretty simple to start with but it's also an important component of a more fleshed-out VCS. No matter how big Ribbit gets, it will still have to track a branch's history.

The architecture for this kind of system can be really straightforward: a repository consists of a chronological list of Revisions, each of which consists of one or more Changes. Each Change is a transformation applied to a file. At least to start with we'll model every Change as completely replacing the previous state of the file since diffs and merging aren't necessary when we don't have collaboration features. We could store all this in many different ways but I figure SQLite is a reasonably scalable solution that seems to have bindings for Rust available. I'm not looking forward to manual management of database connections but this is part of what learning Rust is.

As a first step, I install the SQLite crate by adding it to my Cargo.toml:

[dependencies]
rusqlite = "0.24.0"

And I run cargo build. Cargo fetches and builds the crate no problem, so now I should be able to create a database that stores the state of our repository. Taking inspiration from git, I'll store it under .ribbit/repo.db. Let's see if it works:

use rusqlite::Connection;

fn create_database() -> rusqlite::Result<Connection> {
    let path = "./.ribbit/repo.db";
    let db = Connection::open(&path)?;
    return Ok(db);
}

fn main() -> rusqlite::Result<()> {
    let db = create_database()?;

    Ok(())
}

I put the database creation logic into its own function. I want to start gently dividing the logic I'm writing into functions. Eventually I'll be making multiple crates to organise the code into relevant modules and it will be easier if everything isn't in one large main() function.

When I try to run this, the compiler errors, complaining that it can't find "sqlite3.lib". I guess I don't have SQLite installed on my machine. I'm used to it being around but I must not have used it on any projects on this laptop yet. I install it via Chocolatey but that still doesn't work. I don't really want to spend my evening learning how to get C libraries to work on Windows so I follow the advice of the rusqlite README and enable the option to compile SQLite from source in my Cargo.toml:

[dependencies.rusqlite]
version = "0.24.0"
features = ["bundled"]

Ribbit now compiles again but returns an error: cannot open database file. I'm pretty sure Connection::open is meant to create the database file but it probably doesn't create the folder it should be in if that doesn't exist, so I need to figure out how to create a folder. I find the function I need in the std::fs crate but my naive implementation doesn't compile:

fn create_database() -> rusqlite::Result<Connection> {
    fs::create_dir("./.ribbit")?;
    let db = Connection::open("./.ribbit/repo.db")?;
    return Ok(db);
}

It turns out errors potentially returned from the fs crate are different from those returned from the rusqlite crate, so I can't just stick a ? after create_dir and count on it to just short-circuit if an error is returned. I could create a custom Result type that could contain either kind of error but I figure if I can't create the directory to store the repository data there's not much else the program can do anyway so I decide to use the .unwrap() method instead, which panics — aborts program execution — if there's an error. Panics don't need to be handled by the type system because the program terminates, but they do mean the error can't be caught later.

fn create_database() -> rusqlite::Result<Connection> {
    fs::create_dir("./.ribbit").unwrap();
    let db = Connection::open("./.ribbit/repo.db")?;
    return Ok(db);
}

This works! An empty "./.ribbit/repo.db" file has been created, and reminds me I need to add .ribbit to my .gitignore file. However, when I try to run the code again I see a problem. The program panics because we can't create a directory that already exists. That's a silly reason to abort execution because the database could still be created, so we'll have to check for the existence of the directory before trying to create it:

fn create_database() -> rusqlite::Result<Connection> {
    let ribbit_path = "./.ribbit";
    match fs::metadata(&ribbit_path) {
        Ok(_) => (),
        Err(error) => match error.kind() {
            io::ErrorKind::AlreadyExists => fs::create_dir(&ribbit_path).unwrap(),
            other_error => panic!(other_error),
        },
    };
    let db = Connection::open("./.ribbit/repo.db")?;
    return Ok(db);
}

Correction added 2020-08-31

This code is wrong. The error kind we need to match on is the file not existing rather than already existing. These are the perils of coding late at night, I got what I was doing all mixed up and then didn't test it properly. Part of my objective with this series is to show when and why mistakes happen during learning so I'm going to leave this here until I fix it in the next installment but I didn't want anyone to copy this code believing it should work as described.

End correction

This uses a language feature that I learned to love years ago in Elixir: pattern matching. The branches of a "match" block execute only if the input to the match matches the left hand side and if it does any variables declared within the match store the value in that place of the structure. This is hard to explain but intuitive to write and allow us to easily extract the error from the result and match on its kind. Now the only step that is missing is to create the database table we'll need:

fn create_database() -> rusqlite::Result<Connection> {
    let ribbit_path = "./.ribbit";
    match fs::metadata(&ribbit_path) {
        Ok(_) => (),
        Err(error) => match error.kind() {
            io::ErrorKind::AlreadyExists => fs::create_dir(&ribbit_path).unwrap(),
            other_error => panic!(other_error),
        },
    };
    let db = Connection::open("./.ribbit/repo.db")?;
    db.execute(
        "create table if not exists changes (
            id text primary key,
            date_created integer not null,
            revision_id text not null,
            file_path text not null
        )",
        rusqlite::NO_PARAMS,
    )?;
    return Ok(db);
}

In the future I'm pretty sure we'll need revisions at the very least to have their own table so they can have a message associated with them but for now both revisions and files existing simply as ids in the changes table will be enough. I confirm this code works by opening the database in SQLEctron and checking the table has been created.

This is enough for one night. I already feel myself becoming more confident with Rust's syntax, even though most of my code is still written by modifying copy-and-paste example code from various documentation sources. Next time I'll look into how to read command line arguments so I can add commands to ribbit and start not only initialising the repository but saving some revisions to it.

Learning Rust 1: Install things and read a file

Jae Anne Bach Hardie — Wed, 26 Aug 2020 23:33:24 +0000

I've tried to learn Rust multiple times. Unsuccessfully.

I want to be upfront about that because it's a bit hard to admit. I'm a pretty experienced software engineer, mostly in web development but I used to work in data science. I'm curious and stubborn and generally pride myself on picking things up quickly. However, I never studied Computer Science, I never did any serious coding in a low-level language and so while theoretically I read the Rust docs and I understand I've never managed to make it all click in a way which makes it intuitive to reason about.

I'm writing this in an attempt to break this cycle of picking it up, toying around with it, getting frustrated and going back to Javascript. I'm going to set myself up every few nights, pour myself a drink and write about my progress as I learn to solidify my thoughts and keep me honest. I think this might be useful to other Rust beginners (I certainly like reading other people's thought processed) but I'm not experienced enough to be properly didactic. Expect to come on a ride with me while I figure out why my code doesn't compile and hopefully learn to not make my mistakes, don't expect a coherent explanation of what the stack and the heap are. Those are two different words that both mean "a pile of things" and I keep forgetting which is which.

The drink tonight is a French Manhattan, which is where you replace the vermouth with Chambord. I feel it's a misnomer, if anything it tastes less french than its more traditional cousin, all the herbal sophistication replaced with the flavour of blackberries marinated in cough syrup. The cloying sweetness is at least somewhat celebratory.

Anyway, Rust.

The Project

I'm going to attempt to make a version control system. This sounds like a good idea because

It's the kind of thing that can be really big and complex but made out of simple building blocks.
We'll need to build both a CLI and a server, which is neat variety.
Manipulating lots of files performantly sounds like the kind of thing you'd use a language like Rust for.

I might regret this. It's a bit more ambitious than your standard TodoMVC but it's interesting, isn't it? Have you never stopped and wondered how someone starts coding something like git?

Installing Rust

It turns out I already had rustup installed. If you don't have rustup installed, the Rust website provides installers if it isn't already in your package manager. Rustup is itself like a little package manager, it allows you to download different versions of Rust and switch between them. This is the kind of thing I don't have a reason to care about yet but I know from experience is really useful when you're switching between projects which were made with different versions of a language.

I run rustup update to update to the latest stable version. It's been a while since I last updated and I don't want to be caught out by things that might be different from the docs available out there. Fortunately, it's that easy.

When I go to install a Rust plugin for my editor — VSCode, the editor that has taken over even my hardened, spacemacs-loving heart — I notice there's two: the official one and Rust Analyzer. It looks like Analyzer is an attempt to provide a more responsive IDE experience with things like go-to-reference and autocompletion. That's something I always thought was missing from the rust experience in my previous forays, so even though it's officially in alpha I take heart in the 5 star rating the plugin has and brave it. After all, if it doesn't work I can just switch to the regular language server.

Creating the project

The docs say to use cargo new to create a new Rust project. Cargo is, if I understand it right, the package manager for Rust packages, which are called crates (get it?). It's like Rust's npm. I've decided to name my project "ribbit" for no other reason than it starts with "R" and I think it will be funny to type in the terminal, so I type cargo new ribbit and that looks so funny it instantly validates my naming decision. Or maybe the Chambord is getting to me.

Opening the folder in VSCode reveals cargo.toml and cargo.lock files and I've used npm enough to take a good guess at what those are. I actually really like the TOML format, it's much more human-readable than JSON:

[package]
name = "ribbit"
version = "0.1.0"
authors = ["Jae Anne Bach Hardie <EMAIL REDACTED>"]
edition = "2018"

I don't know why "edition" is set to all the way back in 2018 but I decide not to worry about it. Other than that there's a .gitignore file (very useful) and an extremely simple main function in src/main.rs

fn main() {
    println!("Hello, world!");
}

I'm not immediately sure how to run a Rust project but Cargo has been pretty intuitive so far so I type in cargo run and am not disappointed to find it compiles the project in debug mode and outputs Hello, world! to my terminal. So far, so good. I commit everything to a new repository using VSCode's "publish" command that integrates with GitHub. Frustratingly, it creates the repository but doesn't add the origin to my local git and push, which I have to do manually. I guess they're still ironing out the kinks.

Reading a file

Time for the kind of thing that makes me really afraid. Let's see if we can read a file and print it to console. We're going to need to read a lot of files to make a VCS work!

I start by copying the "read a file" example from the stdlib:

use std::io;
use std::io::prelude::*;
use std::fs::File;

fn main() -> io::Result<()> {
    let mut f = File::open("Cargo.toml")?;
    let mut buffer = [0; 10];

    // read up to 10 bytes
    let n = f.read(&mut buffer)?;

    println!("The bytes: {:?}", &buffer[..n]);
    Ok(())
}

To go through this:

The use declarations import other crates, in this case the stdlib crates needed to open files.
let declares an immutable variable, let mut a mutable one. I love this in theory, in practice I still lack an intuition for when things need to be mutable. For example, I'd think you wouldn't need anything mutable to just read a file and not change it but I think the buffer here needs to be mutable so it can be changed to store the bytes of the file as they come in?
& passes something as a pointer, &mut as a mutable pointer. I think. I'm sure we'll get to pointers right now I just know it has something to do with where memory is allocated and that the compiler will shout at me if I use the wrong one.
The ? after an operation short-circuits the function to return failure if the op fails. This is how Rust does error handling and it's pretty clever. Functions explicitly return a Result type but you can write them without having to think about that. It's like async function() in Javascript, which will return a Promise despite being written like synchronous code with exceptions, except the exceptions are also typed which is great.

Anyway, this code only reads 10 bytes and then prints them as integers rather than as Unicode so we have some ways to go before we're printing out the whole file recognizably. The standard library gives us BufReader with an interface that makes it easier to read something in as a Unicode string:

use std::io;
use std::io::prelude::*;
use std::io::BufReader;
use std::fs::File;

fn main() -> io::Result<()> {
    let f = File::open("foo.txt")?;
    let mut reader = BufReader::new(f);
    let mut buffer = String::new();

    // read a line into buffer
    reader.read_line(&mut buffer)?;

    println!("{}", buffer);
    Ok(())
}

This prints out the first line of the file! We're a while loop away from printing the whole thing, so let's look up how to do that. The documentation is hard to navigate at first, because BufReader doesn't seem to have what I'm looking for. What is happening is that BufReader implements the BufRead trait. Traits are collections of methods that a type implements so BufReader implements BufRead, Debug, Read and Seek. It's BufRead that has the read_line method, as well as a lines method that returns an iterator. That's what we want, because iterators can be looped over with a for loop.

Before we do that, though, I'm starting to realize I'm not getting any IDE integration. I can't see the types of variables on hover and things like that. After some poking around it turns out I have to go to my VSCode settings and turn on individual Rust Analyzer features. Alpha software is alpha, I guess. I don't know which features I want so I check "Enable All Available Features" and hope nothing bad will happen. My code is immediately enriched with type information and I can see documentation on hover! This is amazing and reminds me of what I loved most about ReasonML, the fact that you don't have to manually declare types but the inferred types are added in by the IDE so you can see them.

I first try

for line in reader.lines() {
    println!("{}", line);
}

but my new IDE integration quickly informs me that line there is a Result type that cannot be printed. Makes sense, I guess reading a line could fail. I try adding one of those convenient question marks:

for line? in reader.lines() {
    println!("{}", line);
}

That's a syntax error. Now that I think about it, the ? operator goes on the operation not the name of the variable being assigned to. How about this?

for line in reader.lines()? {
    println!("{}", line);
}

That's a different error but fortunately Rust error messages are pretty helpful:

the `?` operator can only be applied to values that implement `std::ops::Try`
the trait `std::ops::Try` is not implemented for `std::io::Lines<std::io::BufReader<std::fs::File>>`
required by `std::ops::Try::into_result`

I see what is happening. lines() returns the iterator and we need to unwrap each individual value returned from the iterator, not the iterator itself. I'm starting to think there might not be a nice sugar to do this in a for loop declaration although I'd love if there was, just like Javascript has for await (foo of iter). Unwrapping it where it's used works fine, though, and makes me feel a bit silly for not having started there:

use std::io;
use std::io::prelude::*;
use std::io::BufReader;
use std::fs::File;

fn main() -> io::Result<()> {
    let f = File::open("Cargo.toml")?;
    let reader = BufReader::new(f);

    for line in reader.lines() {
        println!("{}", line?);
    }

    Ok(())
}

That compiles and prints out the file! Success!

Before I go for the night, though, I notice that I'm getting an error when trying to auto-format the file. The log says component rustfmt should be in the manifest so I'm guessing rustup doesn't install the formatter by default. Weird choice but OK. I run rustup component add rustfmt but that tells me

toolchain 'stable-x86_64-pc-windows-msvc' does not contain component 'rustfmt' for target 'x86_64-pc-windows-msvc'

Um. Ok. Time to do some Googling. There's an issue where others have had the same problem, the fix seems to be to reinstall the toolchain. I guess my version was too old and rustup got confused? The reinstall fixes it and the autoformatter runs, although my code was already formatted properly so it only reorders the imports. Still, very useful to have in place for later.

Good night everyone!