Forem: Ben Lovy

Oops, I Did It Again...I Made A Rust Web API And It Was Not That Difficult

Ben Lovy — Sun, 21 Nov 2021 23:02:27 +0000

Over two years ago (oof), I posted a walkthrough of my Rust implementation of todo-mvp by @gypsydave5 demonstrating how to build a simple Rust API without a framework. The core functionality was built using hyper, a lower-level HTTP library instead of a full-blown framework.

It turns out I wrote that post about six months too early. I published it in May 2019, and in November, Rust released 1.39.0 containing async/await syntax. Womp womp.

My intent here was to simply upgrade the existing application to use the new syntax where applicable and call it a day. But, you know that thing that happens when you go back and review code you wrote years ago? Yeah, that thing happened. We're starting from scratch.

What's New

Our result functions almost identically to the implementation in the previous post, so some of this code will look very similar. Here's a quick overview of the new stuff you'll find in this post not present in the previous version:

anyhow - Error handling for humans.
async/await - New syntax for expressing asynchronous computation - like a webserver!
catch_unwind - A panicking task shouldn't crash your whole server! Gracefully catch the panic and keep on servin'.
compression - Every response will be DEFLATE compressed, simply because we can.
Rust 2021 - the future of Rust.
state management - Instead of a global variable, we'll use protocol extensions to access the app state.
tracing - The log crate is old news. All the cool kids use tracing now.
unit testing - We didn't have any last time - tsk tsk! Learn how to write async unit tests for your handlers.

Setup

To follow along, you'll need a stable Rust toolchain. See the install page for instructions to install rustup for your platform. You should prefer this method to your distribution's package manager. If you're a NixOS freak, I recommend fenix.

Once your environment is set up, start a new project and make sure you can build it:

$ cargo new simple-todo
   Created binary (application) `simple-todo` package
$ cd simple-todo
$ cargo run
   Compiling simple-todo v0.1.0 (/home/deciduously/code/simple-todo)
    Finished dev [unoptimized + debuginfo] target(s) in 0.19s
     Running `target/debug/simple-todo
Hello, world!

$

Open up your Cargo.toml and make it look like this:

[package]
authors = ["Cool Person <cool.person@yourcoolsite.neato>"]
edition = "2021"
rust-version = "1.56"
name = "simple-todo"
version = "0.1.0"

[dependencies]
anyhow = "1"
backtrace = "0.3"
clap = {version = "3.0.0-beta.5", features = ["color"] }
flate2 = "1"
futures = "0.3"
hyper = { version = "0.14", features = ["full"] }
lazy_static = "1.4"
serde = { version = "1", features = ["derive"] }
serde_json = "1"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
tera = "1"
tokio = { version = "1", features = ["full"] }
uuid = { version = "0.8", features = ["serde", "v4"] }

[dev-dependencies]

pretty_assertions = "0.7"
select = "0.5"

There are some new elements here already, up in the package section. Rust 2021 was recently released, and along with it, the rust-version metadata key, allowing you to specify the Minimum Supported Rust Version (MSRV) directly in the package.

This post is only concerned with Rust and will use assets identical to the last post. Create a folder called templates at the project's top-level, and place this index.html inside. You will also need to create a directory at src/resource and fill it with these files. There is a stylesheet and a handful of SVG files. Your structure should look like this:

$ tree
.
├── Cargo.lock
├── Cargo.toml
├── src
│  ├── main.rs
│  └── resource
│     ├── check.svg
│     ├── plus.svg
│     ├── tick.png
│     ├── todo.css
│     ├── trashcan.svg
│     └── x.svg
└── templates
   └── index.html

Good to go!

Entrypoint

This whole app will live in src/main.rs. Start off by adding the imports:

use anyhow::Result;
use backtrace::Backtrace;
use clap::Parser;
use flate2::{write::ZlibEncoder, Compression};
use futures::{future::FutureExt, Future};
use hyper::http;
use lazy_static::lazy_static;
use serde::Serialize;
use std::{
    cell::RefCell,
    convert::Infallible,
    io::Write,
    panic::AssertUnwindSafe,
    path::PathBuf,
    sync::{Arc, RwLock},
};
use tera::Tera;
use uuid::Uuid;

It's nice to allow the user to specify where to run their app. The clap library provides a convenient way to specify command-line arguments in a struct. This feature is still in beta but will stabilize soon. We can create a struct and use the Parser feature to generate the options:

#[derive(Parser)]
#[clap(version = concat!(env!("CARGO_PKG_VERSION")), about = "Serve a TODO list application.")]
struct Args {
    #[clap(
        short,
        long,
        about = "Address to bind the server to.",
        env,
        default_value = "0.0.0.0"
    )]
    address: String,
    #[clap(short, long, about = "Port to listen on.", env, default_value = "3000")]
    port: u16,
}

When run without any arguments, the server will bind to 0.0.0.0:3000, a reasonable default. This allows users to either use the ADDRESS and PORT environment variables or -a/--address and -p/--port command-line arguments. It also provides a nice --help implementation:

simple-todo 0.1.0

Serve a TODO list application.

USAGE:
    todo-mvp-rust [OPTIONS]

OPTIONS:
    -a, --address <ADDRESS>    Address to bind the server to. [env: ADDRESS=] [default: 0.0.0.0]
    -h, --help                 Print help information
    -p, --port <PORT>          Port to listen on. [env: PORT=] [default: 3000]
    -V, --version              Print version information

Our main() function will just parse these arguments and pass them off to our app routine:

fn main() -> Result<()> {
    let args = Args::parse();
    app(args)?;
    Ok(())
}

I've brought anyhow::Result into scope, making error handling super easy to use. We don't need to specify all our Error types. It can automatically convert any errors that implement std::error::Error, which should be all of them. If an error propagates all the way up to main(), we'll get all the info it's captured printed to stdout.

The app() function is our real entrypoint:

#[tokio::main]
async fn app(args: Args) -> Result<()> {
    tracing_subscriber::fmt::init();

    let addr = std::net::SocketAddr::new(args.address.parse()?, args.port);

    let todos = Todos::new();
    let context = Arc::new(RwLock::new(todos));

    serve(addr, context, handle).await?;

    Ok(())
}

This function is tagged with #[tokio::main], meaning it will execute an async runtime. All the functions we're using now can be marked async, meaning we can await the result. Under the hood, Rust converts these functions into constructs called Futures, and in the previous iteration we built these manually. Futures can either be Ready or Waiting. You can call the poll() method on a future, which just asks: "are you ready, or are you waiting?" If the future is ready to resolve, it will pass up the response, and if it's waiting, it will just respond with Pending. It can return a wake() callback, letting the caller know there's a value. When wake is executed, the caller will know to poll() this future again, resolving it to a value.

This all involved a lot of ceremony. This async/.await syntax abstracts these concepts for us. We just write functions like normal, except we mark them async, and when control flow hits a point where we need to wait for a result to resolve, we can use await. If these are used within an executor like #[tokio::main], the runtime will cover all the details. Instead of blocking control flow, each of these await points will yield to other running tasks on this executor until the underlying Future returns from a poll() request as Ready<T>. This makes our types easier to reason about and our code much more straightforward to write.

This top-level function reads our argument struct to build the SocketAddr to bind to, starts up the logging system with tracing_subscriber, and builds our state management. Then, we call the serve() async function and await on the result. This executor will run forever until the user kills the process and can handle multiple concurrent connections for us seamlessly.

For a sneak preview, this is the signature of serve():

async fn serve<C, H, F>(
    addr: std::net::SocketAddr,
    context: Arc<C>,
    handler: H,
) -> hyper::Result<()>
where
    C: 'static + Send + Sync,
    H: 'static + Fn(Request) -> F + Send + Sync,
    F: Future<Output = Response> + Send,
{
    // ...
}

It requires we pass an address, then an Arc<C>. That C type is our app state, and for this to work, it must implement the Send and Sync traits. Send means the value can be sent to another thread, and Sync means it can be shared between threads simultaneously.

In our case, we don't need to think too hard about this. We're using an Arc<RwLock<T>> to allow mutation to provide a type that can be accessed from multiple tasks and safely mutated as needed. As long as each task grabs the proper lock, we don't need to worry about concurrent tass clobbering each other. Only one task will be able to write to this type at a time, so every new reader will always have the correct data.

Finally, we need to add a handler with type H. These types start to peel back a little of what async is doing for us. Stripping out the Send + Sync trait bounds, this function satisfies the trait bound Fn(Request) -> Future<Output = Response>. Because we're in an async environment, we can just write async fn handle(request: Request) -> Response - it's an asynchronous function from a request to a response. Sounds like a web server to me! Using Rust's async/.await, we get to simply write what we mean.

We'll come back to the handler shortly - first, we have a little bit of setup to take care of.

Templates

This application only consists of a single page that will be refreshed whenever the state changes. We placed markup at templates/index.html, an HTML file using Jinja-style templating. We'll use Tera to handle this in Rust.

The templates need to be compiled before use, but this only needs to happen once. We can use lazy_static to ensure this compilation happens the first time the templates are accessed, and then reuse the compiled result for all subsequent access:

lazy_static! {
    pub static ref TERA: Tera = match Tera::new("templates/**/*") {
        Ok(t) => t,
        Err(e) => {
            eprintln!("Unable to parse templates: {}", e);
            std::process::exit(1);
        }
    };
}

Now we can use the TERA global. If Tera could not compile the templates for any reason, the process will exit here and display the error. Our server only works if this step completes successfully.

State

Next, we need to define the app state. The application revolves around a Todo type with a name, an ID, and a boolean tracking whether it's been completed:

#[derive(Debug, Serialize)]
pub struct Todo {
    done: bool,
    name: String,
    id: Uuid,
}

impl Todo {
    fn new(name: &str) -> Self {
        Self {
            done: false,
            name: String::from(name),
            id: Uuid::new_v4(),
        }
    }
}

We just need to provide a string name, like Todos::new("Task"), and this type will generate a new unique ID and set it to incomplete.

Storage is pretty simple:

#[derive(Debug, Default)]
struct Todos(Vec<Todo>);

We need methods to add new todos, remove existing todos, and toggle the done boolean:

impl Todos {
    pub fn new() -> Self {
        Self::default()
    }

    pub fn push(&mut self, todo: Todo) {
        self.0.push(todo);
    }

    pub fn remove(&mut self, id: Uuid) -> Option<Todo> {
        let mut idx = self.0.len();
        for (i, todo) in self.0.iter().enumerate() {
            if todo.id == id {
                idx = i;
            }
        }
        if idx < self.0.len() {
            let ret = self.0.remove(idx);
            Some(ret)
        } else {
            None
        }
    }

    pub fn todos(&self) -> &[Todo] {
        &self.0
    }

    pub fn toggle(&mut self, id: Uuid) {
        for todo in &mut self.0 {
            if todo.id == id {
                todo.done = !todo.done;
            }
        }
    }
}

Notably, we don't have to write any unique code to make this thread safe. We can write it exactly as we would in a single-threaded, synchronous context and trust that Rust won't let us mutably access this unsafely. This is what got instantiated up in app() with these lines:

let todos = Todos::new();
let context = Arc::new(RwLock::new(todos));

Wrapping this all in an Arc means that any task accessing this value can get their own reference to it, and the RwLock will allow multiple concurrent readers or exactly one writer at a time. When the lock is released, the next waiting task will be able to take control.

Handler

We're finally ready to take a look at the handler. Per the signature up above, we know we need a function with the following signature: async fn handle(request: Request) -> Response. Each time our web server receives an HTTP request, it will call this function to produce a Response to send back. First, add type aliases for the request and response:

type Request = http::Request<hyper::Body>;
type Response = http::Response<hyper::Body>;

The hyper crate defines all the types we need. In both cases, the request and response will carry a hyper::Body. Every single handler will use this type signature, so defining these aliases saves us a lot of typing.

The request contains information about how it was sent. We can use Rust's match construct to read both the incoming URI and HTTP method to correctly dispatch a response. Here's the whole body:

async fn handle(request: Request) -> Response {
    // pattern match for both the method and the path of the request
    match (request.method(), request.uri().path()) {
        // GET handlers
        // Index page handler
        (&hyper::Method::GET, "/") | (&hyper::Method::GET, "/index.html") => index(request).await,
        // Style handler
        (&hyper::Method::GET, "/static/todo.css") => stylesheet().await,
        // Image handler
        (&hyper::Method::GET, path_str) => image(path_str).await,
        // POST handlers
        (&hyper::Method::POST, "/done") => toggle_todo_handler(request).await,
        (&hyper::Method::POST, "/not-done") => toggle_todo_handler(request).await,
        (&hyper::Method::POST, "/delete") => remove_todo_handler(request).await,
        (&hyper::Method::POST, "/") => add_todo_handler(request).await,
        // Anything else handler
        _ => four_oh_four().await,
    }
}

Each match arm will match with a specific combination of HTTP verb and path. For example, GET /static/todo.css will properly dispatch the stylesheet() handler, but POST /static/todo.css is not supported and will fall through to four_oh_four(). Each of these handlers is itself an async function, but we don't want to return up to the caller until they've been polled as ready, and return an actual Response. Remember, Rust is doing this for us - when we write async fn() -> Response, we actually get a Fn() -> impl Future<Output = Response>. We can't use that return type until the future resolves! That's what the .await syntax signifies. Once the future is ready, we'll use the resulting Response output but not before.

The most straightforward handler is four_oh_four():

async fn four_oh_four() -> Response {
    html_str_handler("<h1>NOT FOUND!</h1>", http::StatusCode::NOT_FOUND).await
}

This response doesn't depend on the request - the request didn't make any sense to us, after all! There's no input parameter, but like all handlers, it produces a Response. Because all of our routes need to build responses, I pulled this logic out into a series of building-block functions.

Response Builders

Most of our Response building shares a lot of logic. We usually send back some form of string, and we want to attach a content type and a status code. Because we care about our user's bandwidth usage, we also want to compress our responses, ensuring as little as possible is sent over the wire. The most common case is a successful response containing HTML:

async fn ok_html_handler(html: &str) -> Response {
    html_str_handler(html, http::StatusCode::OK).await
}

This, in turn, calls the HTML string handler:

async fn html_str_handler(html: &str, status_code: http::StatusCode) -> Response {
    string_handler(html, "text/html", status_code).await
}

Our four_oh_four() handler used this directly to include a different status code. Ultimately, though, it's all just strings:

async fn string_handler(body: &str, content_type: &str, status_code: http::StatusCode) -> Response {
    bytes_handler(body.as_bytes(), content_type, status_code).await
}

async fn ok_string_handler(body: &str, content_type: &str) -> Response {
    string_handler(body, content_type, hyper::StatusCode::OK).await
}

These helpers allow for other content types as long as the body is still passed as a string. This covers all the body types we need for this application. At the bottom, we get to bytes_handler:

async fn bytes_handler(body: &[u8], content_type: &str, status_code: http::StatusCode) -> Response {
    let mut encoder = ZlibEncoder::new(Vec::new(), Compression::default());
    encoder.write_all(body).unwrap();
    let compressed = encoder.finish().unwrap();
    hyper::Response::builder()
        .status(status_code)
        .header(hyper::header::CONTENT_TYPE, content_type)
        .header(hyper::header::CONTENT_ENCODING, "deflate")
        .body(hyper::Body::from(compressed))
        .unwrap()
}

This function takes a byte slice (&[u8]), and DEFLATE compresses it. It adds the proper Content-Encoding header so that any connected client can uncompress the payload before presenting it back to the user. This is a rapid operation, and the smaller your payloads, the better. Every response that contains a body will eventually pass through this function before bubbling back up to the top-level handle() function and back to the client.

There's one more response type for this application, and this doesn't use a body at all. Whenever our app state changes, we will use the 301 HTTP status code to trigger the client to redirect back to the index. Every time the index is rendered, it reads the current app state, meaning any mutation executed in the handler will be automatically reflected on the refresh. This function calls Response::builder() directly:

async fn redirect_home() -> Response {
    hyper::Response::builder()
        .status(hyper::StatusCode::SEE_OTHER)
        .header(hyper::header::LOCATION, "/")
        .body(hyper::Body::empty())
        .unwrap()
}

Main page

Now, we have everything we need to render the app. Our index function is the first handler that requires state:

async fn index(request: Request) -> Response {
    // Set up index page template rendering context
    let mut tera_ctx = tera::Context::new();
    let todos_ctx: Arc<RwLock<Todos>> = Arc::clone(request.extensions().get().unwrap());
    {
        let lock = todos_ctx.read().unwrap();
        let todos = lock.todos();
        let len = todos.len();
        tera_ctx.insert("todos", todos);
        tera_ctx.insert("todosLen", &len);
    }
    let html = TERA.render("index.html", &tera_ctx).unwrap().to_string();
    ok_html_handler(&html).await
}

In the previous iteration of this app, the Todos struct was instantiated alongside our TERA templates in a global static variable. A better solution is to thread it through to our handler using the Request itself. We'll look at the implementation lower down, but by the time we get here, there's a fresh Arc containing the context ready to be read. We can use request.extensions().get() and Arc::clone() to grab our very own reference to the app state to use for building this response. The request extensions use the type of what's stored for access, so we need to explicitly add the type of todos_ctx to indicate what we're looking for.

Next, we build the index page using the current state of the app. This handler won't perform any mutation, so we can use todos_ctx.read(). By introducing a child scope, we ensure our read lock gets dropped when we're done with it, allowing any writers waiting for access to grab their own locks. If we needed to wait, no problem! We're in an async function, the caller can poll us any time, and we'll just return Pending until we're ready to go. Nice and neat.

Once we've received our handle to the app state, we can pass it through to Tera. TERA.render() will return an HTML string with all template values resolved using our app state. Then, we can use our trusty ok_html_handler() response builder to tag it with a proper content type and status code and compress the result before returning to the caller.

The index.html template will request a style sheet when it loads from /static/todo.css. That's a pretty simple handler:

async fn stylesheet() -> Response {
    let body = include_str!("resource/todo.css");
    ok_string_handler(body, "text/css").await
}

The include_str!() macro actually bundles the string content directly in your compiled binary. These files need to be present at compile-time but do not need to be distributed in production. The compiled binary already includes everything it needs.

SVG

All the image assets in this application are SVG, which is represented as XML. This means we just need to read these strings and pass the proper content-type:

async fn image(path_str: &str) -> Response {
    let path_buf = PathBuf::from(path_str);
    let file_name = path_buf.file_name().unwrap().to_str().unwrap();
    let ext = match path_buf.extension() {
        Some(e) => e.to_str().unwrap(),
        None => return four_oh_four().await,
    };

    match ext {
        "svg" => {
            // build the response
            let body = match file_name {
                "check.svg" => include_str!("resource/check.svg"),
                "plus.svg" => include_str!("resource/plus.svg"),
                "trashcan.svg" => include_str!("resource/trashcan.svg"),
                "x.svg" => include_str!("resource/x.svg"),
                _ => "",
            };
            ok_string_handler(body, "image/svg+xml").await
        }
        _ => four_oh_four().await,
    }
}

I just used a catch-all - any GET request that's not for the index or the stylesheet is assumed to be for an image. At the top, there's a little extra logic to make sure we're looking for an image file. If there's no extension, i.e. /nonsense, this handler will dispatch a four_oh_four(). Otherwise, we press forward and try to find the actual SVG file. If we succeed, we just pass the string back, and if not, we also four_oh_four().

State Handlers

The remaining handlers are involved with mutating the state. All of them will pass a request body with an item. For a new todo, it will be item=Task, and for toggling or removing, it will hold the id: item=e2104f6a-624d-498f-a553-29e559e78d33. In either case, we just need to extract the value after the equals sign:

async fn extract_payload(request: Request) -> String {
    let body = request.into_body();
    let bytes_buf = hyper::body::to_bytes(body).await.unwrap();
    let str_body = String::from_utf8(bytes_buf.to_vec()).unwrap();
    let words: Vec<&str> = str_body.split('=').collect();
    words[1].to_owned()
}

The body may come in chunks, so we use hyper::body::to_bytes() to produce a single Bytes value with everything concatenated. We can then convert the bytes to a UTF-8 string and split on the = to grab the actual payload. All of our state mutation handlers call this function on the incoming request:

async fn add_todo_handler(request: Request) -> Response {
    let todos_ctx: Arc<RwLock<Todos>> = Arc::clone(request.extensions().get().unwrap());
    let payload = extract_payload(request).await;
    {
        let mut lock = todos_ctx.write().unwrap();
        (*lock).push(Todo::new(&payload));
    }
    redirect_home().await
}

async fn remove_todo_handler(request: Request) -> Response {
    let todos_ctx: Arc<RwLock<Todos>> = Arc::clone(request.extensions().get().unwrap());
    let payload = extract_payload(request).await;
    {
        let mut lock = todos_ctx.write().unwrap();
        (*lock).remove(Uuid::parse_str(&payload).unwrap());
    }
    redirect_home().await
}

async fn toggle_todo_handler(request: Request) -> Response {
    let todos_ctx: Arc<RwLock<Todos>> = Arc::clone(request.extensions().get().unwrap());
    let payload = extract_payload(request).await;
    {
        let mut lock = todos_ctx.write().unwrap();
        (*lock).toggle(Uuid::parse_str(&payload).unwrap());
    }
    redirect_home().await
}

Each handler grabs its own unique reference to the app state, then extracts the payload. Like we did in index(), we open a new scope to interact with the RwLock mutex, and in this case, we use todos_ctx.write() to request a mutable lock. This blocks all other tasks until the mutation is complete. Then, we just redirect_home(). This prompts the client to send a GET / request, which leads our to-level handler to call index(), which reads the newly-mutated app state to build the page.

Groovy! That's a full-fledged TODO app.

Serve

There's one missing piece. We defined our handle() function, but we haven't talked about serve() beyond the type signature. This one is pretty beefy:

async fn serve<C, H, F>(
    addr: std::net::SocketAddr,
    context: Arc<C>,
    handler: H,
) -> hyper::Result<()>
where
    C: 'static + Send + Sync,
    H: 'static + Fn(Request) -> F + Send + Sync,
    F: Future<Output = Response> + Send,
{
    // Create a task local that will store the panic message and backtrace if a panic occurs.
    tokio::task_local! {
        static PANIC_MESSAGE_AND_BACKTRACE: RefCell<Option<(String, Backtrace)>>;
    }
    async fn service<C, H, F>(
        handler: Arc<H>,
        context: Arc<C>,
        mut request: http::Request<hyper::Body>,
    ) -> Result<http::Response<hyper::Body>, Infallible>
    where
        C: Send + Sync + 'static,
        H: Fn(http::Request<hyper::Body>) -> F + Send + Sync + 'static,
        F: Future<Output = http::Response<hyper::Body>> + Send,
    {
        let method = request.method().clone();
        let path = request.uri().path_and_query().unwrap().path().to_owned();
        tracing::info!(path = %path, method = %method, "request");
        request.extensions_mut().insert(context);
        let result = AssertUnwindSafe(handler(request)).catch_unwind().await;
        let start = std::time::SystemTime::now();
        let response = result.unwrap_or_else(|_| {
            let body = PANIC_MESSAGE_AND_BACKTRACE.with(|panic_message_and_backtrace| {
                let panic_message_and_backtrace = panic_message_and_backtrace.borrow();
                let (message, backtrace) = panic_message_and_backtrace.as_ref().unwrap();
                tracing::error!(
                    method = %method,
                    path = %path,
                    backtrace = ?backtrace,
                    "500"
                );
                format!("{}\n{:?}", message, backtrace)
            });
            http::Response::builder()
                .status(http::StatusCode::INTERNAL_SERVER_ERROR)
                .body(hyper::Body::from(body))
                .unwrap()
        });
        tracing::info!(
            "Response generated in {}μs",
            start.elapsed().unwrap_or_default().as_micros()
        );
        Ok(response)
    }
    // Install a panic hook that will record the panic message and backtrace if a panic occurs.
    let hook = std::panic::take_hook();
    std::panic::set_hook(Box::new(|panic_info| {
        let value = (panic_info.to_string(), Backtrace::new());
        PANIC_MESSAGE_AND_BACKTRACE.with(|panic_message_and_backtrace| {
            panic_message_and_backtrace.borrow_mut().replace(value);
        })
    }));
    // Wrap the request handler and context with Arc to allow sharing a reference to it with each task.
    let handler = Arc::new(handler);
    let service = hyper::service::make_service_fn(|_| {
        let handler = handler.clone();
        let context = context.clone();
        async move {
            Ok::<_, Infallible>(hyper::service::service_fn(move |request| {
                let handler = handler.clone();
                let context = context.clone();
                PANIC_MESSAGE_AND_BACKTRACE.scope(RefCell::new(None), async move {
                    service(handler, context, request).await
                })
            }))
        }
    });
    let server = hyper::server::Server::try_bind(&addr)?;
    tracing::info!("🚀 serving at {}", addr);
    server.serve(service).await?;
    std::panic::set_hook(hook);
    Ok(())
}

I know, I know. There's a lot here. The meat of this function happens right at the end:

let server = hyper::server::Server::try_bind(&addr)?;
tracing::info!("🚀 serving at {}", addr);
server.serve(service).await?;

We build a hyper::Server, bind it to the address we constructed from the Args struct, and serve this service thing. The service is built right above that:

let handler = Arc::new(handler);
let service = hyper::service::make_service_fn(|_| {
    let handler = handler.clone();
    let context = context.clone();
    async move {
        Ok::<_, Infallible>(hyper::service::service_fn(move |request| {
            let handler = handler.clone();
            let context = context.clone();
            PANIC_MESSAGE_AND_BACKTRACE.scope(RefCell::new(None), async move {
                service(handler, context, request).await
            })
        }))
    }
});

We also wrap the handler function in an Arc. Our context is already wrapped, so we clone both to get a local reference within this closure. This allows the executor to spin up multiple concurrent versions of our handler service that all have access to the same state and logic.

The beefy stuff happens in this service() closure above. This is where we take an incoming request and match it up with our handler and context. A new instance is executed for each incoming request, and all this ceremony allows this to happen without interfering with other simultaneous requests.

For one, this is where we add our context to the request:

request.extensions_mut().insert(context);

When we call request.extensions().get() in our mutating handlers, we're pulling out the context added at this stage.

There's also some logging added. We trace the request's specifics and start a timer that reports how long the request took. To see this logging in action, set the environment variable RUST_LOG=info when executing the server process.

Catching Panics

The most exciting part (to me, at least) is the panic handler. We always want our requests to succeed. However, there are situations where we may encounter a panic. This will cause the entire Rust program to crash and print out a stack trace in normal usage. However, this is a web service. A panic situation in one request handling process shouldn't prevent other requests from executing. We don't want the whole server to crash; we still want to handle these situations gracefully. We can intercept the normal panic behavior and instead simply produce a different response containing the details.

At the top, we create a task-local storage location:

tokio::task_local! {
    static PANIC_MESSAGE_AND_BACKTRACE: RefCell<Option<(String, Backtrace)>>;
}

This is local to just the currently executing Tokio task, not the whole program. Then, we replace the default panic logic with our own:

let hook = std::panic::take_hook();
std::panic::set_hook(Box::new(|panic_info| {
    let value = (panic_info.to_string(), Backtrace::new());
    PANIC_MESSAGE_AND_BACKTRACE.with(|panic_message_and_backtrace| {
        panic_message_and_backtrace.borrow_mut().replace(value);
    })
}));

// ...

std::panic::set_hook(hook);

First, we grab the existing hook and store it to the hook variable. Then, we overwrite our own. At the end of the function, we make sure to reset the global panic hook back to what it was. If the task panics inside this function - for example, one of our unwrap() statements encounter a non-success, we'll store the panic message and backtrace to this task-local location. However, we will not abort the process.

Up above, in the service location, we can catch this happening:

let result = AssertUnwindSafe(handler(request)).catch_unwind().await;

We attempt to build the response, but this result will not be a success if anything happens. If it was a success, great, we pass it back up. However, if we find an error value here, we can dispatch different logic:

let response = result.unwrap_or_else(|_| {
    let body = PANIC_MESSAGE_AND_BACKTRACE.with(|panic_message_and_backtrace| {
        let panic_message_and_backtrace = panic_message_and_backtrace.borrow();
        let (message, backtrace) = panic_message_and_backtrace.as_ref().unwrap();
        tracing::error!(
            method = %method,
            path = %path,
            backtrace = ?backtrace,
            "500"
        );
        format!("{}\n{:?}", message, backtrace)
    });
    http::Response::builder()
        .status(http::StatusCode::INTERNAL_SERVER_ERROR)
        .body(hyper::Body::from(body))
        .unwrap()
});

For most requests, that result.unwrap() went fine, and we just stored our response to Response. However, if it was an error, we can read the result of the panic from this task-local area. We emit an error trace on the server-side and then build a new response with status code INTERNAL_SERVER_EROR. This response includes the full backtrace as the body. This means our server can keep handling other requests without interruption, but the specific client that caused the panic gets a complete log of the problem, and our server has logged the backtrace as well. We can diagnose the issue without losing uptime for any other client.

Now, no matter what happened while processing the request, we've stored a valid hyper::Response to this response value, and we can pass that back to the caller even if something catastrophic occurs. We can safely use Ok::<_, Infallible>, signifying that there is no possible way for control to fail to hit this point. Our server will always generate a response and continue, even if something terrible happens. Good stuff.

Tests

Finally, we want to ensure we can automate tests. I'll just demonstrate a test of our 404 handler, which includes all the pieces needed to build a robust test suite:

#[cfg(test)]
mod test {
    use super::*;
    use flate2::write::ZlibDecoder;
    use pretty_assertions::assert_eq;
    use select::{document::Document, predicate::Name};

    #[tokio::test]
    async fn test_four_oh_four() {
        let mut request = hyper::Request::builder()
            .method(http::Method::GET)
            .uri("/nonsense")
            .body(hyper::Body::empty())
            .unwrap();
        let context = Arc::new(RwLock::new(Todos::new()));
        request.extensions_mut().insert(Arc::clone(&context));
        let response = handle(request).await;

        assert_eq!(response.status(), http::status::StatusCode::NOT_FOUND);

        let body = hyper::body::to_bytes(response.into_body()).await.unwrap();
        let mut decoder = ZlibDecoder::new(Vec::new());
        decoder.write_all(&body).unwrap();
        let uncompressed = decoder.finish().unwrap();
        let result = String::from_utf8(uncompressed).unwrap();

        let document = Document::from(result.as_str());
        let message = document.find(Name("h1")).next().unwrap().text();
        assert_eq!(message, "NOT FOUND!".to_owned());
    }
}

Tokio provides a #[tokio::test] macro for building async tests. We can use hyper::Request to construct a request and build our context the same way we did in the server. Because our handler is just a function from a Request to a Response, we can test it very simply: let response = handle(request).await;.

We first assert the status code matches what we expect, then we have to decode the body. We use the ZlibDecoder to read the response body and decompress it back to a string.

Once we have our string response, we can use the select.rs library to ensure the structure matches our intent. In this case, we are asserting we've received an h1 element with a text body matching the string NOT FOUND!.

Fin

This implementation is an improvement over the previous in several fundamental ways. The async/.await syntax allows us to write code closely matching our intent without getting bogged down with Boxes and Futures. We avoid polluting the global scope and use the Request itself to handle concurrent access to the app state and even catastrophic request handling errors gracefully without affecting other clients. Our handler is straightforward to test. This application provides a solid, performant base for building more complex applications and keeps the dependencies and therefore compile times and bundle sizes to a minimum.

While there are many different frameworks for building web applications in Rust, it's worth asking yourself whether you actually need that level of abstraction. For many web service needs, putting the building blocks together yourself isn't much more complicated, and you retain control over how you build your application. If we want to deconstruct the request URI, we can do that already. If we return JSON, we just need to create a struct that implements serde::Serialize.

The conclusion here is the same as before: when the end goal is simple enough, why not use simple tools?

Cover photo by Danny Howe on Unsplash

Prepare your Rust API docs for Github Pages

Ben Lovy — Tue, 09 Nov 2021 01:30:45 +0000

Rust comes with a built-in documentation system, rustdoc. It's great, use it.

Github comes with a built-in web hosting service, Github Pages. It's great, use it.

To use Github Pages, you can either create a directory in your repository to host your deployed web content or a separate branch and then activate the feature in the "Pages" section of your repository settings:

Great!

You get a directory located at <project>/target/doc when you run cargo doc to generate your Rust documentation. It contains the complete web content ready to publish. If you execute cargo doc --open, your default browser will open up and show you what it generated.

Unfortunately, this folder can't be copied strictly as-is. If you try, you'll get a 404. This is because Github Pages is looking for an index.html file in the root directory of wherever it's pointed. However, cargo doc created a subdirectory for your crate specifically, where index.html lives. Here's my example for a Rust package called build_wheel:



$ tree target/doc/
target/doc
├── ayu.css
├── brush.svg
├── build_wheel
│  ├── all.html
│  ├── fn.build_wheel.html
│  ├── index.html          <-- RIGHT HERE
│  ├── sidebar-items.js
│  ├── struct.Config.html
│  └── struct.Tag.html
├── clipboard.svg
/* ... etc, etc... */

Uh-oh, it's nested. Github can't find it. You can't just make a symbolic link to an index.html in the root directory because all the relative links inside will break.

Luckily, the fix is easy. You simply need to create an index.html that reroutes where you want. This file only consists of one line:



<meta http-equiv="refresh" content="0; url=build_wheel">

This file uses the http-equiv meta attribute to point immediately to the build_wheel subdirectory and load the index.html file found there. Just replace the directory name with the name of your package.

You can easily automate this in a simple script:



cargo doc --no-deps
rm -rf ./docs
echo "<meta http-equiv=\"refresh\" content=\"0; url=build_wheel\">" > target/doc/index.html
cp -r target/doc ./docs

Feel free to get as fancy as you want for pulling the actual name. I've just hardcoded it ¯\(ツ)/¯.

When Github Pages tries to build your content, now it will find this root index page. Users will load this page and immediately refresh the page you intended to display, and all the internal links generated by rustdoc will work as expected.

Neat.

Cover photo by Kelly Sikkema on Unsplash

Workstation Management With Nix Flakes: Build a Cmake C++ Package

Ben Lovy — Sun, 31 Oct 2021 15:42:35 +0000

Last time, we looked at how to produce a development shell using Nix Flakes that contained the Python interpreter alongside a few dependencies the project required. In this post, we'll create a compiled binary, using source code hosted on GitHub, for other people to include in their environments.

The Target

For this demonstration, I'll be packaging the LightGBM CLI tool. Nixpkgs already provides derivations for the native libraries for Python and R, which will suffice for most users, but I didn't see one to build the CLI tool directly (and, of course, will submit mine upstream as well).

Per the documentation, building this application from source is relatively straightforward:

git clone --recursive https://github.com/microsoft/LightGBM
cd LightGBM
mkdir build
cd build
cmake ..
make -j4

We can learn a few things: this git repository has submodules, meaning we need to use --recursive, and the build can be parallelized. These instructions also have you create a build directory, but with Nix, we can skip that. The whole build will take place in a purpose-made build directory.

The Flake

As before, I'll show the complete flake first. We'll walk through it in pieces below.

{
  inputs = {
    nixpkgs = {
      url = "github:nixos/nixpkgs/nixos-unstable";
    };
    flake-utils = {
      url = "github:numtide/flake-utils";
    };
  };
  outputs = { nixpkgs, flake-utils, ... }: flake-utils.lib.eachDefaultSystem (system:
    let
      pkgs = import nixpkgs {
        inherit system;
      };
      lightgbm-cli = (with pkgs; stdenv.mkDerivation {
          pname = "lightgbm-cli";
          version = "3.3.1";
          src = fetchgit {
            url = "https://github.com/microsoft/LightGBM";
            rev = "v3.3.1";
            sha256 = "pBrsey0RpxxvlwSKrOJEBQp7Hd9Yzr5w5OdUuyFpgF8=";
            fetchSubmodules = true;
          };
          nativeBuildInputs = [
            clang
            cmake
          ];
          buildPhase = "make -j $NIX_BUILD_CORES";
          installPhase = ''
            mkdir -p $out/bin
            mv $TMP/LightGBM/lightgbm $out/bin
          '';
        }
      );
    in rec {
      defaultApp = flake-utils.lib.mkApp {
        drv = defaultPackage;
      };
      defaultPackage = lightgbm-cli;
      devShell = pkgs.mkShell {
        buildInputs = [
          lightgbm-cli
        ];
      };
    }
  );
}

If you read the last post, much of this looks the same. We have inputs and outputs, a let...in section defining variables, and a devShell in the output.

The new bits are the lightgbm-cli variable, defined using stdenv.mkDerivation, and the defaultApp and defaultPackage keys in the outputs section.

The Walkthrough

I won't repeat the explanations of the shared elements. Take a look at the previous post for the full descriptions. This post will focus on the mkDerivation section.

The pkgs.stdenv.mkDerivation function is a wrapper around the low-level Nix concept of a derivation. See the Nix Pills for a thorough walkthrough of how to work with them. Using mkDerivation sets some sensible defaults. It provides more tools on top of this base to greatly streamline the process, but it's important to know that this derivation concept is what ultimately gets evaluated under the hood.

We want to refer to this derivation in multiple places, so it gets a specific name assigned in the let...in block:

lightgbm-cli = (with pkgs; stdenv.mkDerivation {
    # ...
  }
);

By prefixing the function using with pkgs; we can avoid explicitly referring to pkgs every time we use something from this set. The first example is right here. the stdenv object lives in pkgs, so fully qualified, this opener would be pkgs.stdenv.mkDerivation.

Name

Inside, we first set the package name. You can directly set a name, but the preferred way to handle this is to set the package name using pname, and the version separately. Nix will then combine them into a whole name key for you with the format ${pname}-${version}.

pname = "lightgbm-cli";
version = "3.3.1";

Source

Then, we need to point Nix at our source code. If you were packaging a source that you were building yourself, and this flake lived in the project directory, you could point at the current folder:

src = ./.;

However, we're just creating a derivation for an existing codebase hosted online in a Git repository. We can use the pkgs.fetchgit function to direct Nix to download the source directly before building:

src = fetchgit {
  url = "https://github.com/microsoft/LightGBM";
  rev = "v3.3.1";
  sha256 = "pBrsey0RpxxvlwSKrOJEBQp7Hd9Yzr5w5OdUuyFpgF8=";
  fetchSubmodules = true;
};

We provide the URL and the specific tag, so we're not just pointing right at the main branch. You can also set a particular commit, tagged or not, by giving the complete git commit hash:

rev = "d4851c3381495d9a065d49e848fbf291a408477d";

If we didn't need to fetch any submodules, and the repo was on GitHub specifically, there's an even better function to use called pkgs.fetchFromGithub:

src = fetchFromGitHub {
  owner = "microsoft";
  repo = "LightGBM";
  rev = "v3.3.1";
  hash = "sha256-pBrsey0RpxxvlwSKrOJEBQp7Hd9Yzr5w5OdUuyFpgF8=";
}

You don't need to provide a full URL, Nix needs the owner and repo name, and you can use the more generic hash key, prefixing the actual hash so that Nix knows how to resolve it. However, I was unable to get this working with submodules, and this wouldn't work for code hosted elsewhere, like GitLab or sourcehut.

Before you run your derivation the first time, you probably won't know the sha256 hash of the source repository. While you could download it separately and use the nix hash-path path/to/clone command, I find it more convenient to provide a bad hash first and let Nix tell you what it should be. There's even a built-in feature for this:

sha256 = lib.fakeSha256;

On the first run, you'll get an error like this:

$ nix build
error: hash mismatch in fixed-output derivation '/nix/store/5ysvmxay83fnc14w5r2s450i39byd4ks-source.drv':
         specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
            got:    sha256-pBrsey0RpxxvlwSKrOJEBQp7Hd9Yzr5w5OdUuyFpgF8=
error: 1 dependencies of derivation '/nix/store/jaip55q9ix8hq4l7srd9kigxlq7nimyr-lightgbm-cli-3.3.1.drv' failed to build

It downloaded the repository, checked the hash, and (thankfully) noticed it's not the same as the fakeSha256 hash you provided. Conveniently, it tells us what the hash should be so that you can change your code:

- sha256 = lib.fakeSha256;
+ sha256 = "pBrsey0RpxxvlwSKrOJEBQp7Hd9Yzr5w5OdUuyFpgF8=";

When you rerun it, the hashes will match, and Nix will continue to the next phase.

Inputs

Now that we have the source, we need to tell Nix what tools are necessary to build the software. We can use the nativeBuildInputs key for this:

nativeBuildInputs = [
  clang
  cmake
];

Configuration is handled via cmake, and we'll have clang be our C++ compiler.

Package

To control the build, you can use the phases provided by mkDerviation: unpack, patch, configure, build, check, and install. However, if you need default behavior for any of these, you can omit them. You only need to define logic when it differs from the defaults. To learn more about these phases, see this article.

Setting cmake in our nativeBuildInputs means that Nix already configures this project by default. We don't have any unique unpacking to do or any patches to apply, and we're not going to check the compiled output. These defaults mean we only need to define a buildPhase and an installPhase.

The build part is easy:

buildPhase = "make -j $NIX_BUILD_CORES";

By this point, Nix has already implicitly run cmake ., so our Makefiles are in place and ready to go. We could omit this phase and have Nix execute make, but we want to use as many cores as possible to speed up the build. Nix provides a special environment variable $NIX_BUILD_CORES that we can pass to make's job flag to be sure this build runs as efficiently as it can.

Then, we need to copy the resulting binary to the output of the flake:

installPhase = ''
  mkdir -p $out/bin
  mv $TMP/LightGBM/lightgbm $out/bin
'';

The result of your derivation gets assigned a special $out variable. If your derivation ultimately produces one or more binaries, you can create a $out/bin directory to hold these. Any Nix expression that depends on your derivation will automatically add the contents of this directory to your $PATH.

In this case, we need the lightgbm executable file produced in the build phase. When we downloaded the GitHub source, Nix placed it in the LightGBM directory within the special $TMP build directory for the derivation. $TMP is an alias for $NIX_BUILD_TOP. You can also use $TMPDIR, $TEMPDIR, or TEMP. Nix creates all these while running your derivation to prevent processes from using /tmp outside this special Nix-y place.

If you had opted to pin to a specific commit instead of a tag, this folder would be called, e.g. $TMP/LightGBM-d4581c3, including the beginning of the complete commit hash.

If you get confused about the directory structure, I find the easiest trick is to dump the whole thing:

installPhase = ''
  mkdir $out
  cp -r $TMP $out
'';

When you run nix build, you'll see the entire contents of the build directory in the result symlink, which you can explore to understand where everything ended up.

Fixed-Output Derivations

I don't use this feature directly in this example, but it's worth mentioning that you don't have access to the internet in your phase descriptions. This restriction is to keep Nix derivations pure. However, what if you wanted to? You might need to curl something or use maven to download dependencies before building. There are myriad reasons you may wish to access the internet beyond simply grabbing the source code to build a given project.

To allow this to happen, you can provide an output hash of the result so that Nix can prove to itself that your derivation is reproducible. Like before, you won't know this result ahead of time, so you'll need to watch it fail once and tell you the intended result, but once you add that to your flake, Nix will happily grab these remote files and include them in your derivation result.

First, you need to add cacert to your derivation and set the SSL_CERT_FILE variable:

buildInputs = [ cacert ];
SSL_CERT_FILE = "${cacert}/etc/ssl/certs/ca-bundle.crt";

Now, Nix will pull the required SSL certificates into the build sandbox so that tools like curl can access them to make network requests.

Then, you add the hash:

outputHash = "sha256-I4UGDcrtmX/1TAQz89peXsqoetZmCM+1b3XYqexv/VA=";

This config should be sufficient if your derivation result is a file. However, if you're producing a folder structure, you'll also need to add this key:

outputHashMode = "recursive";

By default, outputHashMode is set to "flat", so it will fail to hash a recursive directory tree properly. Luckily it's easy enough to toggle. Your derivation is still totally reproducible and safe to use in a pure evaluation environment, even if it requires downloading additional files from the internet.

In fact, this is what the pkgs.fetchgit convenience helper is doing under the hood. Everything in Nix is a derivation, and the hash you provide to this function gets used to satisfy this exact requirement.

Outputs

Now, the lightgbm-cli variable points to the complete derivation. For usage, we want this flake to be usable via nix build using defaultPackage, nix develop using devShell, or nix run using defaultApp. All of these are easy to do now:

defaultApp = flake-utils.lib.mkApp {
  drv = defaultPackage;
};
defaultPackage = lightgbm-cli;
devShell = pkgs.mkShell {
  buildInputs = [
    lightgbm-cli
  ];
};

Now, users can access a ready-made lightgbm executable no matter how they intend to interact with this flake or integrate it into their environments. We lean on the mkApp feature provided by flake-utils to produce the particular structure required by defaultApp/nix run and pass it the defaultPackage key. Because this whole block was marked rec, it doesn't matter what order we define them. Nix will recursively evaluate this block until everything is thoroughly defined.

If you have Nix installed, you should be able to create a new folder, paste the contents above into flake.nix, and use these commands to grab, compile, and use lightgbm yourself no matter what platform you're using.

Neat!

Cover photo by Henry & Co. on Unsplash

Workstation Management With Nix Flakes: Jupyter Notebook Example

Ben Lovy — Sat, 30 Oct 2021 13:58:52 +0000

What?

If you're reading DEV, your computer is probably your primary productivity tool. Whether it's a job, a hobby, or just a point of interest, your ability to install and use different types of software with ease is integral to your ability to keep learning and working.

This means that many of us end up with complicated workstation needs, with an interconnected web of dependency interactions. If you need to wipe your hard drive and start over, it would likely take a non-trivial amount of work to set everything back up exactly the same way, and each piece needs to be managed separately.

Nix is a tool designed to take the pain out of workspace management. Nix is declarative. You're not telling the computer how to build your environment step-by-step; you're simply describing the end state you need. Nix learns how to make it and then stores everything you need in a reproducible way so that subsequent invocations are near-instant. It is also unified. Every tool you use can be managed this way, with lots of built-in functionality for quickly setting up complex environments with minimal effort on your part.

This post will show you how I'm using Nix to manage my Python environment and dependencies for using Jupyter Notebooks for school assignments. However, this same strategy can be used to build environments for whatever tool you need.

If you just want to see the config file, jump down to The Flake.

Who?

While Nix can control your entire operating system, you don't need to go all-in to reap the benefits. You can install the Nix package manager on Linux (i686, x86_64, or aarch64) or macOS (currently just x86_64, but ARM support will show up eventually). See the Quick Start guide to get up and running.

This post uses a new feature of Nix called Nix Flakes, which you need to enable after you install. See the Flakes wiki page and look for the "Non-NixOS" section. I promise it's not too hard.

However, if you end up liking what you see, I think NixOS is the best way to use Nix. Just recently, I ended up wiping my hard drive. I re-partitioned and then ran one single command. After letting it build, I could boot into an operating system identical to how I had left it, with all my tools installed and configuration settings set. That's pretty hard to beat. You can see my personal config on Github.

Why?

Python is a great programming language. It has both a low barrier of entry for people learning the craft and a bustling ecosystem for scientific/numerical computing and web development (and much, much more).

However, one sticking point for me has always been dependency management. For most Python uses, you will need to install some packages, and keeping this organized is not always straightforward.

I'm using this example specifically because I recently started taking a class that uses Jupyter Notebooks for assignments and labs. I love Jupyter. It's an excellent tool for this sort of thing and is pretty widely used. However, when I opened up my first assignment, the first interactive cell contained these lines:

import numpy as np
import pandas as pd

While this is super standard and to be expected, I still felt myself thinking, "Oh boy, here we go again."

There is a tool for installing dependencies. It's not like you're just manually downloading Python files, putting them in a sensible place yourself, and somehow telling your Python interpreter where to find them (looking at you, C and C++). We have pip! With a properly configured Python installation, you can just type pip install numpy, and the tool will automatically take care of the details.

However, this installs to a global location. Every Python program in any project on your computer will refer to the same place. No good - what if you want to run some code that depends on a different version of some package?

The numpy dependency isn't a global dependency of your python installation. The need is local to whatever you're currently doing. There's an answer for that as well, of course. It involves, well, a virtualenv, a requirements file, and a substantial local folder with a full copy of your environment. This environment is pretty specific to exactly how it was installed too. If anything happens to it, or you change computers, you have to start over.

It, you know, works, but it doesn't exactly "spark joy" for me. With each new Python project, the tangled mess that comprises "my workstation" grows, and I don't need that kind of background anxiety in my life.

How?

With Nix, we have a declarative solution. We can create one text file and describe what we need, and Nix will get it done. This file will work the same way anywhere we want to use it and doesn't leave brittle artifacts littered in our project codebase. As long as this single text file is present in your project, you can produce the environment you need the same exact way every time and pretty much just forget about the whole thing. It will even work if we drop it into a different computer running on a different platform and OS, as long as it's supported by Nix.

The Flake

I'll start by showing you the whole file, and we'll dig through it in pieces below. This gets saved as flake.nix in the top level of your project directory.

{
  inputs = {
    nixpkgs = {
      url = "github:nixos/nixpkgs/nixos-unstable";
    };
    flake-utils = {
      url = "github:numtide/flake-utils";
    };
  };
  outputs = { nixpkgs, flake-utils, ... }: flake-utils.lib.eachDefaultSystem (system:
    let
      pkgs = import nixpkgs {
        inherit system;
      };
    in rec {
      devShell = pkgs.mkShell {
        buildInputs = with pkgs; [
          (python3.withPackages(ps: with ps; [
            ipython
            jupyter
            numpy
            pandas
          ]))
        ];
        shellHook = "jupyter notebook";
      };
    }
  );
}

Not so bad, right? Let's take a closer look from the outside in.

The Walkthrough

A Nix Flake is just an object - check out those surrounding curly braces. This object has two keys, inputs and outputs. The inputs are where we define the flake's dependencies and where to find all the tools we use. This one has two, nixpkgs and flake-utils. Each of these just points to a GitHub URL, and if you follow those links, you'll see each repo provides its own flake.nix. The outputs of each remote flake get piped into the inputs of my flake, so we can use what they provide.

Next, we define our own output object:

outputs = { nixpkgs, flake-utils, ... }: flake-utils.lib.eachDefaultSystem (system:
  # ...
);

We specifically bring each name we need from the inputs with {{ nixpkgs, flake-utils, ... }} so we can refer to them inside, then use a tool from flakeUtils called eachDefaultSystem. This is a convenience helper. It provides a special variable called system (see, right at the end: (system:) which refers to the specific platform you're on. On my computer, this resolves to x86_64-linux. This means we can use the exact same flake on any supported system, and Nix will know how to find the correct versions of anything inside.

First, we create a special pkgs variable to refer to the Nix package repository:

let
  pkgs = import nixpkgs {
    inherit system;
  };
in rec {
  #...
};

If you're a functional programmer, you might recognize the let...in pattern. We're just defining variables to be used below. This flake just needs the one, and using inherit system - the system from just above - we can expressly point to the subset of nixpkgs that applies to our platform. We add rec to allow Nix to fully resolve any Nix variables in the following object recursively until thoroughly evaluated.

To discover what's available, you can use the NixOS Search page - yes, this applies to Nix users on other operating systems as well.

Finally, we're ready to tell Nix what we need. This flake doesn't build a package; it just defines a development shell. We can activate it by using the command nix develop from anywhere in this folder. This is called a devShell:

devShell = pkgs.mkShell {
  #...
};

To build it, we can use the built-in mkShell feature. This is contained in pkgs, which we've already appropriately configured above.

We want this shell to have stuff available for use, using the buildInputs key of mkShell:

buildInputs = with pkgs; [
          (python3.withPackages(ps: with ps; [
            ipython
            jupyter
            numpy
            pandas
          ]))
        ];

We use with pkgs; first, so that we don't have to fully qualify pkgs.python3 for each package inside, but that's what we mean for each item in this array.

This array only contains one input, python3. However, we don't want just a plain Python environment. We know some packages we'll need access to already. This is the part that replaces the virtualenv setup entirely. We can use the withPackages option to define the list: this environment will grab ipython, jupyter, numpy, and pandas for us. This will install Jupyter for us and the packages used inside, so the import statements from above will just work without any further action on your part. If you need a new package at any point, you can just add it to this list and re-run nix develop.

Additionally, you can pin to a minor version of Python by simply replacing python3 in the snippet with, for example, python39.

Finally, to make this as easy as possible to use for this purpose, we can have it immediately launch Jupyter:

shellHook = "jupyter notebook";

You can run any arbitrary code here. This will execute immediately once the environment is ready. You can use multiple lines:

shellHook = ''
  mkdir cool_dir
  echo "cool file contents" > cool_dir/cool_file.txt
  jupyter notebook
'';

I don't go anywhere without my cool_file.txt, and Nix definitely can deliver.

With my particular shellHook in place, running nix develop in this folder will automatically launch the Jupyter server and open the landing page in my browser. I just store all my .ipynb files in the same directory, and they're all ready to use. I am thrilled that I could use Nix to escape all the Python package management problems I've experienced in the past with under 30 lines of config. This solution is portable, reproducible, and once you understand the building blocks, easy to use and adapt.

The first time you run nix develop will be the slowest because Nix needs to download everything from scratch. However, once it's in your nix store, subsequent runs will be rapid. If nothing has changed, it will be instant, and if you tweak something, it will only recalculate the difference.

The inputs are hashed, so you always use the same revision upstream. You can run nix flake update to grab updated hashes if you want to pull in new changes. This will update the flake.lock file that Nix creates and manages.

Your Nix store will grow over time because it will keep fully specified and hashed versions of whatever you use. To clean up unneeded files, you can periodically run nix-collect-garbage.

Okay, And?

This example is just a taste. You can use Nix to completely manage your entire workstation from the ground up, as well as declaratively package any type of application you would like to distribute to others.

In future posts, we'll look at how to package your code for others to use, how to use Nix Flakes to define your entire operating system, and take a deeper look at what Nix is doing under the hood to turn these files into development environments.

Nix is a vast topic, but it can significantly streamline how you manage your computer and the tools you need every day. If there's something specific you'd like to see in this series, please let me know!

Cover photo by CHUTTERSNAP on Unsplash.

Pretend You're Using A Different Linux Distribution With One Docker Command

Ben Lovy — Mon, 25 Oct 2021 19:14:29 +0000

The Why

Lots of developers use Linux, but "Linux" is a vast category. There are a lot of similarities, but a lot of differences, too. Ubuntu and Fedora look and feel pretty similar until you try to install a package - whoops, apt is a little different than dnf. Specific system settings may be stored in different places, and particular commands may be included by default in one but not the other. Then there's even more niche stuff like Arch, which you install piece by piece from a very minimal package set, or Gentoo, which is similar to Arch with the additional caveat that the user compiles all the software locally for their specific hardware. Users of these distros may end up with pretty different-looking operating systems that all fall under the broad Linux umbrella.

All of the above adhere to a structure called the Filesystem Hierarchy Standard, FHS for short. This specifies the standard top-level hierarchy common to these different flavors, like /etc for configuration, /boot for bootloader files, /proc for process management, and /home for user-specific home directories. See the Wikipedia link for a more complete list. If you're a Linux user, this structure will feel familiar to you.

However, even the FHS is not universal. My personal development machine is running a super weird Linux flavor called NixOS. This fully declarative system stores every single component of functionality in a unique directory called /nix/store and maintains a web of symlinks. Software compiled for standard Linux distributions won't run on NixOS or vice versa without specifically patching the resulting executable binaries. To complicate things further, I'm pinning to the unstable channel instead of a tagged release, meaning the package set is liable to change at any time. While there are a lot of benefits, it means my local machine is fundamentally incompatible with the Linux computers I want the software I produce to run on.

The Point

I'm primarily writing code in Rust, which has powerful facilities for cross-compiling non-native targets built-in, and Nix can help me fill in the rest. This is great! From my local computer, I can produce working binaries for many different types of computers.

For example, we want to support Ubuntu 18.04, one Long-Term Support release behind the current LTS, 20.04. This is several years old by this point, and as a result, only has, for example, glibc version 2.27, instead of the current 2.34. This is crucial for compatibility, because almost every program depends on your OS providing this library and being able to use whichever version it finds.

However, how would I know that my result on my bleeding-edge NixOS box works as intended? Containers to the rescue! We can ask Docker to build an Ubuntu 18.04 container and drop us into a shell with the current filesystem available. It's kind of like the su command, except instead of switching the active user, you're changing your whole OS on the fly.

Here's the line:

$ docker run --rm -it -v $PWD:/working-dir docker.io/ubuntu:18.04
root@6bb49a338644:/# cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.6 LTS"
root@487693de818e:/# cd working-dir/
root@487693de818e:/working-dir# ls
Cargo.lock  Cargo.toml  README.md  custom-target.json  dist  flake.lock  flake.nix  hello  hello_build  main.rs  scripts  target  x86_64-unknown-linux-gnu2.24.json  x86_64-unknown-linux-musldynamic.json  zig
root@7ae91a94a888:/working-dir# exit
$

Perfect! The /working-dir directory inside your new Ubuntu 18.04 container now has the contents of whichever directory you were in when you ran this command. That's the -v $PWD:/working-dir part. The $PWD variable returns the current working directory, and after the colon, you provide a location in the new container to mount this directory.

As far as any software inside is concerned, it's running in a standard Ubuntu installation. This lets me quickly verify that my program's cross-compiled, binary-patched version runs as expected on this target environment. When you're done, just type exit to return to your native shell. The -it flag made the container interactive, and the --rm flag will clean up the Docker container when it quits.

You can check out the Docker Hub or quay.io for other available docker containers to spin up.

This tip works with any tool that supports the Docker API, in addition to Docker itself. I'm running it via Podman, and it works the same way.

Now you can use whatever crazy environment you want and still responsibly ensure whatever you're compiling will work as intended for your users.

Cover image photo by Braydon Anderson on Unsplash

Quickly Grab Stuff From Your Git History

Ben Lovy — Fri, 22 Oct 2021 23:30:11 +0000

While working through a problem, my colleague remembered a prior version of our application had a syntax example we could use. Thankfully, the codebase has been checked into git! We knew the code in question revolved around the mkDerivation functionality in nix.

Since then, this particular code has been moved out of our codebase and added to nixpkgs, so we can pull the derivation from the main tree instead of defining our own out-of-tree logic. My first instinct was to start digging through that (huge) repository to find the file and use that as a reference.

No need! We have it in our git history. We can query this using git log -S:

$ git log -S mkDerivation
commit 3a275488e740ae1b4314208a908c5300f9563ee0
Author: David Yamnitsky <david@yamnitsky.com>
Date:   Mon Jul 19 11:51:47 2021 -0400

    use mold and wasm-bindgen from nixpkgs

commit a3a042b5b90ad57ff11bc47a5db6e68dc1ca55e7
Author: David Yamnitsky <david@yamnitsky.com>
Date:   Wed Jun 16 10:26:35 2021 -0400

    use mold as the linker to speed up incremental compiles on x86_64-linux

Beautiful - that top commit looks like it represents when we switched to pull this derivation directly from nixpkgs. Removing the code is sufficient - each git commit represents a diff. This commit should show us the code:

$ git show 3a275488e740ae1b4314208a908c5300f9563ee0
commit 3a275488e740ae1b4314208a908c5300f9563ee0
Author: David Yamnitsky <david@yamnitsky.com>
Date:   Mon Jul 19 11:51:47 2021 -0400

    use mold and wasm-bindgen from nixpkgs
...
flake. nix

───┐
1: │
───┘
{
  inputs = {
    nixpkgs = {
      url = "github:nixos/nixpkgs/nixos-unstable";
      url = "github:nixos/nixpkgs/nixos-unstable-small";
    };
    flake-utils = {
      url = "github:numtide/flake-utils";

────┐
48: │
────┘
        CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER = toString ./. + "/scripts/clang";
        CARGO_TARGET_WASM32_UNKNOWN_UNKNOWN_LINKER = "lld";
        buildInputs = with pkgs; [
          (stdenv.mkDerivation {
            pname = "mold";
            version = "0.9.1";
            src = fetchgit {
              url = "https://github.com/rui314/mold";
              rev = "v0.9.1";
              sha256 = "sha256-yIkW6OCXhlHZ1jC8/yMAdJbSgY9K40POT2zWv6wYr5E=";
            };
            nativeBuildInputs = [ clang_12 cmake lld_12 tbb xxHash zlib openssl git ];
            dontUseCmakeConfigure = "true";
            buildPhase = "make -j $NIX_BUILD_CORES";
            installPhase = "mkdir -p $out $out/bin $out/share/man/man1 && PREFIX=$out make install";
          })
          cachix
          cargo-insta
          clang_12

There it is, in the text! In my terminal, additions are highlighted in green and removals are in red. This was a removal, but you still get the full removed text. I was able to copy that stdenv.mkDerivation code and work from there. Thanks, git. (Thit).

As an aside, I highly recommend the following git alias:

l = "log --all --graph --decorate --abbrev-commit --format=format:'%C(bold blue)%h%C(reset) - %C(bold white)%an%C(reset) %C(bold yellow)%d%C(reset)%n%C(bold cyan)%aD%C(reset) - %C(bold green)(%ar)%C(reset)%n%C(white)%s%C(reset)'";

It's a mess of text, but it produces super easy to read git histories:

* 9b24232 - Ben Lovy  (HEAD -> main, origin/main, origin/HEAD)
| Fri, 22 Oct 2021 11:49:42 -0400 - (8 hours ago)
| Remove maplit
* 65016b6 - David Yamnitsky 
| Fri, 22 Oct 2021 11:39:12 -0400 - (8 hours ago)
| update deps
| * 6d55d3e - Ben Lovy  (refs/stash)
|/| Fri, 22 Oct 2021 11:49:01 -0400 - (8 hours ago)
| | WIP on main: e07e691 clear 1.56 warnings
| * 6bf8ab1 - Ben Lovy 
|/  Fri, 22 Oct 2021 11:49:01 -0400 - (8 hours ago)
|   index on main: e07e691 clear 1.56 warnings
* e07e691 - David Yamnitsky 
| Fri, 22 Oct 2021 11:09:02 -0400 - (8 hours ago)
| clear 1.56 warnings

The coloration doesn't reflect here, in your terminal this will be even cooler. As a git novice, this sort of output is instrumental in keeping track of changes to the codebase.

Rust Function Pointers Across FFI Boundary

Ben Lovy — Fri, 10 Sep 2021 13:39:40 +0000

Hi DEV - it's been ages! I'm stumped on a problem, and remembered my old adage: "ask DEV stuff, they know things". So, here's hoping someone can unstick me!

EDIT: Figured it out. Left the solution in the comments!

I'm trying to store a pointer to a function defined in C in a Rust struct, and call it from the Rust side. The rest of my FFI seems to be hooked up okay, but I'm getting a segfault when I make the actual function call. Debugging sessions have not proved useful.

I'll try to keep the context streamlined, but let me know if you need more to see what's going on.

First, here's the C externs, the function I'm pointing to, and the passage across the FFI boundary:

typedef struct scheduler scheduler_t;
typedef struct job job_t;
typedef void (*unit_to_unit_t)(void);

extern void run(job_t *, scheduler_t *, unit_to_unit_t);
extern job_t *every(uint32_t);
extern job_t *seconds(job_t *);

// Define a job
void job(void)
{
    //printf("Hello!  It is now %s\n", now());
    printf("Hello!");
}

run(seconds(every(8)), scheduler, job);

Super exciting stuff. The commented out line calls a now()function that returns a char *, but I wanted to get anything working first. The every() function returns job_t *, and seconds() takes a job_t * and returns a job_t *, which is finally passed into run(). I have a typedef named unit_to_unit_t for the job function pointer - it's a function taking no arguments and returning nothing. The scheduler is also an FFI entity.

This is the definition of run() on the Rust side:

#[no_mangle]
pub unsafe extern "C" fn run(job: *mut Job, scheduler: *mut Scheduler, work: *const fn() -> ()) {
    let job = {
        assert!(!job.is_null());
        Box::from_raw(job)
    };
    let mut scheduler = {
        assert!(!scheduler.is_null());
        &mut *scheduler
    };

    let work: &fn() -> () = {
        assert!(!work.is_null());
        &*work
    };

    job.run(&mut scheduler, *work)
        .unwrap_or_else(|e| eprintln!("Error: {}", e));
}

I'm passing the C function as the last argument here, and trying to cast it to work. This step doesn't complain, seems to go okay.

On the Rust side, when that job.run() method gets called, the following trait/struct is used to store the function pointer and call it:

pub trait Callable {
    /// Execute this callable
    fn call(&self) -> Option<bool>;
    /// Get the name of this callable
    fn name(&self) -> &str;
}

/// A named callable function taking no parameters and returning nothing.
#[derive(Debug)]
pub struct UnitToUnit {
    name: String,
    work: fn() -> (),
}

impl UnitToUnit {
    pub fn new(name: &str, work: fn() -> ()) -> Self {
        Self {
            name: name.into(),
            work,
        }
    }
}

impl Callable for UnitToUnit {
    fn call(&self) -> Option<bool> {
        // gets HERE just fine...
        (self.work)();
        None
    }
    fn name(&self) -> &str {
        &self.name
    }
}

This all works fine with Rust function pointers. I determined that with the C version, we do get inside the call() implementation - the segfault happens when I use (self.work)();

I'm not sure if it's relevant, but the actual call is triggered here in C:

extern void run_pending(scheduler_t *);    

// Run some jobs
for (int i = 0; i < 100; i++)
{
    run_pending(scheduler);
    sleep(1);
}

Which corresponds to this Rust interface fn:

#[no_mangle]
pub unsafe extern "C" fn run_pending(ptr: *mut Scheduler) {
    let scheduler = {
        assert!(!ptr.is_null());
        &mut *ptr
    };

    scheduler
        .run_pending()
        .unwrap_or_else(|e| eprintln!("Error: {}", e));
}

Here's scheduler::run_pending():

    /// Run all jobs that are scheduled to run.  Does NOT run missed jobs!
    pub fn run_pending(&mut self) -> Result<()> {
        //let mut jobs_to_run: Vec<&Job> = self.jobs.iter().filter(|el| el.should_run()).collect();
        self.jobs.sort();
        let mut to_remove = Vec::new();
        for (idx, job) in self.jobs.iter_mut().enumerate() {
            if job.should_run() {
                let keep_going = job.execute()?;
                if !keep_going {
                    debug!("Cancelling job {}", job);
                    to_remove.push(idx);
                }
            }
        }
        // Remove any cancelled jobs
        to_remove.sort_unstable();
        to_remove.reverse();
        for &idx in &to_remove {
            self.jobs.remove(idx);
        }

        Ok(())
    }

And finally, job::execute():

    /// Run this job and immediately reschedule it, returning true.  If job should cancel, return false.
    ///
    /// If the job's deadline has arrived already, the job does not run and returns false.
    ///
    /// If this execution causes the deadline to reach, it will run once and then return false.
    pub fn execute(&mut self) -> Result<bool> {
        if self.is_overdue(self.now()) {
            debug!("Deadline already reached, cancelling job {}", self);
            return Ok(false);
        }

        debug!("Running job {}", self);
        if self.job.is_none() {
            debug!("No work scheduled, moving on...");
            return Ok(true);
        }
        let _ = self.job.as_ref().unwrap().call(); // CALLED RIGHT HERE
        self.last_run = Some(self.now());
        self.schedule_next_run()?;

        if self.is_overdue(self.now()) {
            debug!("Execution went over deadline, cancelling job {}", self);
            return Ok(false);
        }

        Ok(true)
    }

I added a comment where the actual call() happens.

This whole FFI shindig is just an experiment, I don't actually have a real need for this to work - but now that I got this far, I kinda want to know what I'm getting wrong! Thanks in advance.

Using Bitwise Operators: Why Waste Space Use Many Bits When Few Bits Do Trick?

Ben Lovy — Sun, 09 Aug 2020 20:06:27 +0000

These days, we live in the future. Memory is very cheap. When we need to store an integer, in most cases it makes sense to just use an entire multi-byte value to represent it even if your value is low. Of course, that does result in a lot of wasted space. The number 2 as a 32-bit integer looks like this:

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0

Just one of these bits is flipped towards the end - all the rest is just padding.

I'm working on a Chip8 emulator. To build such an emulator, you create a software representation of the entire computer. It has a space for the program memory, some registers for short-term storage, some counters, some boolean flags - everything it needs to execute a program.

The Chip8 is a much more limited environment than a modern computer. The entire space for memory only comprises 4096 bytes - and some of that is reserved for the system itself. With such a constraint it doesn't make sense to waste all this space. Whenever possible we want to concern ourselves with a single byte at a time, so that surrounding bytes can be meaningfully applied to other parts of the program.

In fact, as we'll see, we don't even need to limit ourselves to a byte. Using bitwise operators we can efficiently use individual bits within a byte to store and communicate information with minimal wasted space.

The Machine

I'm doing mine in Rust, but you don't need to know Rust to follow this post. This is what my struct representing the machine looks like:

/// The top-level software representation of the Chip8 machine
pub struct Machine {
    opcode: Opcode,
    /// Available memory space - 4K
    memory: [u8; 4096],
    /// CPU Registers
    registers: [u8; 16],
    /// Index register
    pub idx: u16,
    /// Program counter
    pub pc: u16,
    /// Graphics system - 2048 total pixels, arranged 64x32, each containing 0 or 1
    pub screen: [u8; 64 * 32],
    /// Delay timer - 60Hz, counts down if above 0
    pub delay_timer: u8,
    /// Sound timer - buzzes at 0.  60Hz, counts down if above 0
    pub sound_timer: u8,
    /// Call stack
    pub stack: [usize; 16],
    /// Stack pointer
    pub sp: usize,
    /// Keep track of the keypad - 0x0-0xF
    keys: [bool; 16],
    /// Flag to signal the screen state has changed and must be redrawn
    pub draw_flag: bool,
}

The biggest thing you need to know is that a u8 is an unsigned 8-bit value, or a single byte, and a u16 is an unsigned 16-bit value. The usize types are machine-dependent pointer-sized values, used to index into arrays. In my case they're analogous to u64.

There is a software representation for each part of the computer a program will interact with. Then, you can programmatically hook up actual inputs and outputs however you like, and the running program is none the wiser. Inside this struct everything works exactly as it would on a physical version of this machine.

Hexadecimal Representation

When working with individual bytes hexadecimal is much more convenient than decimal. A single hexadecimal digit maps neatly to a 4-bit value, or a nibble, and thus can express every possible configuration of 4 bits:

0x0 -> 0 0 0 0
0x1 -> 0 0 0 1
0x2 -> 0 0 1 0
0x3 -> 0 0 1 1
0x4 -> 0 1 0 0
0x5 -> 0 1 0 1
0x6 -> 0 1 1 0
0x7 -> 0 1 1 1
0x8 -> 1 0 0 0
0x9 -> 1 0 0 1
0xA -> 1 0 1 0
0xB -> 1 0 1 1
0xC -> 1 1 0 0
0xD -> 1 1 0 1
0xE -> 1 1 1 0
0xF -> 1 1 1 1

It's clear to see the pattern. This makes hexadecimal a natural way to talk about memory in such small quantities. A byte is 8 bits, which is two nibbles next to each other. Using two hex digits we can represent every possible byte from 0x00, which is all 8 zeros, to 0xFF, which is all 8 ones.

Opcodes

To interact with the machine state, a program consists of a series of opcodes. Each cycle, the machine reads the current opcode and then updates the state in memory and registers according to which specific instruction is found. When the program loads these opcodes are copied into the computer's memory sequentially, beginning at the starting address:

/// Locate a program file by filename and load into memory.  Returns total bytes loaded.
pub fn load_game(&mut self, name: &str) -> Result<usize> {
    // Clear the memory to make way
    self.reset();
    let rom = SHELF.rom(name)?; // I have all game data pre-loaded in memory and tagged by name
    // Load in memory starting at location 512 (0x200), which is where the pc pointer starts
    for (idx, &byte) in rom.iter().enumerate() {
        self.memory_set(idx as u16 + self.pc, byte);
    }
    Ok(rom.len())
}

It all just gets copied one byte at a time, and the self.pc pointer is used to keep track of where the machine is currently looking. However, each instruction is actually more than a single byte long. There is a full list on Wikipedia, 35 in total. Here's a few examples:

1NNN -> Jump to address NNN
8XY4 -> Add the value in register Y to the value in register X
3XNN -> Skip the next instruction if the value in register X equals 0xNN

Each of these digits is hexadecimal so each opcode is 2 bytes long, or 16 bits. In Rust, I use the u16 datatype to work with these numbers. Further, each one carries different amounts of information. There is a hardcoded prefix and/or suffix to determine which exact instruction, and then within that various nibbles are used to denote different things. Look at the differences between these codes:

0x1234 -> Jump to address 0x234
     1       2         3        4
0 0 0 1 | 0 0 1 0 | 0 0 1 1 | 0 1 0 0 

Extract value 0x234 - pad to 0x0234
0 0 0 0 0 0 1 0 0 0 1 1 0 1 0 0
___

0x8234 -> Add the value in register 3 to the current value in Register 2
    8       2         3        4
1 0 0 0 | 0 0 1 0 | 0 0 1 1 | 0 1 0 0 

Extract values 0x2 and 0x3
0 0 1 0
0 0 1 1

___

0x3234 -> Skip the next instruction if the value in register 2 equals 0x34
    3       2         3        4
0 0 1 1 | 0 0 1 0 | 0 0 1 1 | 0 1 0 0

Extract values 0x2 and 0x34
0 0 1 0
0 0 1 1 0 1 0 0

The only difference in bit-level representation between 0x1234 and 0x8234 is which bit is flipped in the first nibble. However, the information stored is not used in the same way. To successfully address all 4096 memory locations a single byte is insufficient, but 12 bits is enough, so we pull the final three nibbles and understand them as a single number. To specify registers, though, there's only 16 of those - we don't need all the extra space. A single nibble is enough to uniquely address any of the registers. Thus, the second and third nibbles are used to specify the targets and the final nibble is simply used as another hard-coded opcode designator. The 3XNN code uses a third pattern, extracting two values but one is a nibble and the other is a full byte. It skips an instruction of the value at register X is equal to the value in NN.

The point is that while specific patterns of bits may be the same or similar, how they're understood within the program can vary. There's a lot of different ways to efficiently utilize a pattern of 16 bits depending on how many total possibilities there are for each unit of information. This means that you can't approach an opcode as a single number or even two separate bytes. It's really four nibbles, and what you do with them will depend on the specific opcode. It makes most sense to work with these as a pattern of individual bits.

Bitwise Manipulation

Bitwise operators look at the individual bits in each operand to produce a result, instead of dealing at the higher abstraction level that "numbers" introduce. We'll look at &, |, <<, and >>, with some passing mention of ^ as well.

Extract Specific Nibbles

The first example was 1NNN. The relevant information the machine needs to execute this opcode consists of the last three nibbles of the opcode. After we have determined that 0x1234 is an opcode of this type, all we need to do is grab that 0x234 value. To do this you can use the bitwise &, or AND:

AND
A  B  Result

0  0  0
1  0  0
0  1  0
1  1  1

The AND operation checks if both values are set to 1, or true. If they are, A & B = 1. If they aren't both set, the result is 0, or false. We can use this to find the set bits of only a portion of a 16-bit value. If we take 0x1234 & 0x0FFF, the result will contain the exact bits that are set in the last three nibbles and completely ignore the first nibble:

0x1234 -> 0 0 0 1 0 0 1 0 0 0 1 1 0 1 0 0
0x0FFF -> 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1
&        ________________________________
0x0234 -> 0 0 0 0 0 0 1 0 0 0 1 1 0 1 0 0

By applying the AND rules to each pair of bits in sequence, the final total has only zeros for the not-relevant first nibble and successfully retains the information from the rest of the first byte. We now have a bit pattern that fits neatly into a u16 containing the extracted information.

We can use the same logic to grab, for instance, the last byte, like we do to pull 0x34 from 0x3234:

0x3234 -> 0 0 1 1 0 0 1 0 0 0 1 1 0 1 0 0
0x00FF -> 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1
&        ________________________________
0x0034 -> 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 0

In Rust:

pub fn last_three_digits(&self) -> u16 {
    self.0 & 0x0FFF
}

It's important to note once again that this neat mapping of place value to nibble only works with hexadecimal - 0x34 in base 16 is 52 in base 10! It's easy to get confused.

Extract Any Nibble

As an extension of the previous concept, many of these opcodes require extracting just the second and/or third nibble, which in memory are actually stored in consecutive bytes. To get the second you can use 0x0F00, or 0x00F0 for the third.

You could write a separate function for each position, but it's not hard to use bit-shifting to generalize it:

pub fn nibble_from_left(&self, from_most: u8) -> u8 {
    // Make sure the parameter is legal
    if from_most > 3 {
        panic!(
            "cannot get the {}th nibble from 2-byte number {:#0x}",
            from_most, self.0
        );
    };
    let bits = 4 * (3 - from_most);
    ((self.0 >> bits) & 0xF) as u8
}

Let's use this function to grab the 0x2 from 0x8234. This function expects the index from the most-significant digit, or the left, starting at 0, so to get at the second nibble the index passed as a parameter will be 1.

First, we can observe that anything less significant - or to the right - of the nibble in question is totally irrelevant. We can discard it completely. We are only interested in a single nibble, which means that running our bitwise & operator against just 0xF will be sufficient to pull what we need and discard anything before it. By using the >>, or right-shift operator, we can move the nibble in question right to the end of the number:

0x8234 -> 1 0 0 0 0 0 1 0 0 0 1 1 0 1 0 0
>> 8
0x0082 -> 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0

Shifting right by one discard the rightmost bit, pads a new 0 bit to the left, and moves everything over. If we do this eight times we've discarded the last two nibbles and moved the one we're interested in right to the end. Now we can run the &:

0x0082 -> 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0
0xF    -> 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1
&      ___________________________________
0x0002    0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0

Brilliant! Using >> and &, we were able to produce a bit pattern that corresponds to the exact digit we need. The missing piece was how much to shift right - 4 * (3 - from_most); means that we will always shift bits by a multiple of 4 to always deal with a whole nibble at a time. We just need to know how many nibbles to shift. To get the 3rd least significant digit, like this case, we take 4 * (3 - 1) = 4 * 2 = 8. This way we can reuse the same logic to pull any of the four nibbles out. The only thing that changes is how much data at the end we discard - if we want the last nibble after all, this would be 4 * (3 - 3) = 0, and applying >> 0 to a number will leave it as-is.

This is used very commonly to parse the opcodes used in the Chip8 machine. Many of them specifically carry information in the middle two nibbles so I just made a helper function to pull those:

pub fn middle_nibbles(&self) -> (u8, u8) {
    (self.nibble_from_left(1), self.nibble_from_left(2))
}

This will very neatly return (0x2, 0x3) when called on 0x8234.

Combine Bytes

When we loaded the program - or series of opcodes - into memory, we copied them one single byte at a time. This means that each opcode is occupying two adjacent memory locations. We need to wrap them up together and understand that sequence of 16 consecutive bits as a single number. In other words, we need to find the u16 you get when you place two u8 values next to each other. If you have the bytes 0xAB and 0xCD, how do you produce the 16-bit value 0xABCD?

On paper, we can visually see the solution:

0xAB   -> 1 0 1 0 1 0 1 1
0xCD   ->                 1 1 0 0 1 1 0 1
0xABCD -> 1 0 1 0 1 0 1 1 1 1 0 0 1 1 0 1

You just need to grab a space big enough and put the more significant byte to the left of the less significant byte. Of course, we have to get there with bitwise operators, though, lacking an in-software pencil and paper.

To isolate a nibble from a longer bit sequence, we used the right-shift >> above to shift the nibble we need to the end. The reverse left-shift operation << also exists, which will discard more significant digits to the left and pad with zeros on the right. If we shift left by 8 bits we will essentially insert a blank byte:

0xAB   -> 1 0 1 0 1 0 1 1
<< 8
0xAB00 -> 1 0 1 0 1 0 1 1 0 0 0 0 0 0 0 0

Great! To pop our actual intended second byte in, we can use bitwise OR, or |. This has a similar function to AND but with the following truth table:

OR
A  B  Result

0  0  0
1  0  1
0  1  1
1  1  1

The difference is that this operator returns true/1 if either of the operands is set to 1. It's only false if both bits are false. We can use this to combine our newly-expanded first byte with the intended second byte:

0xAB00 -> 1 0 1 0 1 0 1 1 0 0 0 0 0 0 0 0
0xCD   -> 0 0 0 0 0 0 0 0 1 1 0 0 1 1 0 1
|       _________________________________
0xABCD -> 1 0 1 0 1 0 1 1 1 1 0 0 1 1 0 1

This operation just pulls down whatever is relevant from either byte, effectively combining them. So, first, shift the more significant byte, then OR it against the less significant byte to pull out your single 16-bit value:

pub fn combine_bytes(byte_one: u8, byte_two: u8) -> u16 {
    (byte_one as u16) << 8 | byte_two as u16
}

When the emulator emulates a cycle it reads two bytes starting at the current program counter and combines them using this method to understand the current opcode:

fn fetch_opcode(&self) -> Result<Opcode> {
    let first_byte = self.current_byte();
    let second_byte = self.memory_get(self.pc + 1);
    Ok(Opcode::new(first_byte, second_byte)?)
}

In my implementation the Opcode::new() logic contains the call to this internal combine_bytes method.

Misc

These are the bitwise operations I've used in handling Chip8 opcodes, but understanding these operators can lead to some very efficient implementations of some other commonly needed operations.

Even or Odd

One handy one to keep under your belt is to check whether a given number is even or odd. A common idiom many of us reach for is the modulo operator: byte % 2 == 0. If the remainder after dividing by two is 0 then it's an even number.

There's actually two ways (that I know) to do this. We went over bitwise AND in this article, so I'll cover that - it's similar to the trick we used to extract a nibble. In a binary representation of a number, all we need to determine whether a number is even is the very last bit. If it's even, this will be a zero, and if it's odd, this will be a one. If that observation isn't intuitive, go back up and look at the mappings from hex digits to nibbles and see if you can see why this is true.

This means that it's sufficient to AND your value against 1. You will get back a 1 if the value to check ends in a 1 bit, or a 0 if it's zero.

5 -> 0 1 0 1
1 -> 0 0 0 1
&  _________
1 -> 0 0 0 1

___

6 -> 0 1 1 0
1 -> 0 0 0 1
&  _________
0 -> 0 0 0 0

Then, just check what you got. If the bit was set, it's an odd number, otherwise, it's even. In Rust:

fn is_even(val: u32) -> bool {
    val & 1 == 0
}

This will work on values of any size, because it only ever looks at the least significant bit.

While in most applications the performance benefit is likely negligible, this solution will be more efficient than using %. In some situations it may be enough to prefer this.

I'll leave the other solution I know as an exercise - can you do it using bitwise XOR (exclusive-or)? This operator looks like ^ in many languages. Here's the truth table:

XOR
A  B  Result

0  0  0
1  0  1
0  1  1
1  1  0

The ^/XOR operation is similar to |/OR, but only returns true if one or the other bit is set and not both. Show me what you get in the comments!

Double/Halve Numbers

The left and right shift operations we used above are super handy. When you push the bits in a binary number to the left, you end up doubling the value:

6  -> 0 1 1 0
<< 1
12 -> 1 1 0 0

Looking at the above it's clear to see that this will work the other way, too. Shifting right instead will undo the above, effectively halving the value. You can extend this to any power of two:

6  -> 0 0 1 1 0

<< 2
24 -> 1 1 0 0 0

<< 3
48 -> 1 1 0 0 0 0

Shifting right by n is the same thing as multiplying by 2^n. Neat! As with the even/odd trick, this will probably be more efficient than multiplying - however, you do lose some readability. As with any optimization, it always depends on context.

There are numerous other bitwise "hacks" like this. Share some of your favorites below!

Acknowledgements

If you're going to tackle your own Chip8 emulator, while the Wikipedia page has a lot of what you need, I also recommend How to write an emulator by Laurence Muller. So far, this blog post and Wikipedia have been the only two resources I've needed.

Cover image: Photo by Federica Galli on Unsplash

Prime Sieve in (Hopefully) Idiomatic Ruby (And Some Books You Should Read)

Ben Lovy — Thu, 06 Aug 2020 04:14:49 +0000

Hey y'all. Been awhile. Books are at the bottom.

Earlier this week, I got my first "star" on an exercism.io for a solution very early on in the Ruby track I wrote months ago, so I went back and looked at what I wrote. I'm far enough removed from writing this solution that reading the code was interesting, and I can tell I was largely eager to flex all the cool toys I just learned. So, I want to tell you what I did, and I want you to tell me how you'd do it better.

The problem is the Sieve of Eratosthenes. I did say it was an early one, but I'd argue problems that are easy to wrap your head around are the best way to flex in a new environment. That's "era-TOSS-the-knees", if you're like me and read it way before you heard it or said it. Basically, you can find prime numbers up to n by going up a number list one by one, calling each element x and crossing out all numbers from x up to n that are a multiple of x. If it's a multiple of something, it's not prime. Eventually, the list remaining comprises all the primes from 2 to n. Easy enough.

What's neat, though, is that in Ruby you can kinda just say that:

# Sieve of Eratosthenes
class Sieve
  def initialize(n)
    @all = n > 1 ? 2.upto(n).to_a : []
  end

  def primes
    return @all if @all.empty?

    @all.each do |x|
      @all.select { |y| y > x }.each do |z|
        @all.delete(z) if (z % x).zero?
      end
    end
    @all
  end
end

The first thing you learn when learning Ruby is how deeply Object-Oriented Programming permeates its design. Instead of thinking in terms of calling methods and executing functions, which a fundamentally imperative way of approaching a task, it's important to try to shift towards "passing messages" and receiving responses. The biggest thing I've learned is that design is not so much about the classes you have, it's about how they interface with each other and the messages they can utilize.

Ruby is an idiosyncratic language in that many of its idioms arise from the wealth of messages that are available to pass. Instead of checking if 2 == 0, you pass the object 2 - which is an instance of class Integer - the message zero? and see what it says. Quite often, I fire up pry and enter questions like this:

$ pry
[1] pry(main)> 2.methods.grep /z/
=> [:rationalize, :size, :zero?, :nonzero?, :frozen?, :freeze]

Cool. That's all the methods that the number 2 knows how to respond to that contain the letter z. Why?

[2] pry(main)> 2.class
=> Integer

Right. Any Integer can. But it goes deeper:

[2] pry(main)> 2.class
=> Integer
[3] pry(main)> 2.class.superclass
=> Numeric
[4] pry(main)> 2.class.superclass.superclass
=> Object
[5] pry(main)> 2.class.superclass.superclass.superclass
=> BasicObject
[6] pry(main)> 2.class.superclass.superclass.superclass.superclass
=> nil

It's BasicObjects all the way down. If you pass a message to an object, it will first check the object itself for an implementation. If it isn't found, it will then check the class for the implementation (or "go right"). If that class doesn't have it, it will check superclasses (or "go up") until it finds what it needs, or doesn't. Right one, then up until a match or nil. Easy enough. Even Class is a class (it subclasses Module), you can call methods on it like attr_reader. You often see it without parens (attr_reader :name, :address, :email), but Ruby doesn't require parentheses for method calls. It's all "just Ruby". Whoa. I highly recommend Metaprogramming Ruby 2 for grokking the significance of all this.

This high degree of runtime reflection is part of what makes Ruby, well, Ruby. All this to say that the Integer class implements the zero? method. The visual leap from 2 == 0 to 2.zero? isn't huge, but the conceptual leap is important. I am working with a bunch of objects living in memory, and I can ask any of them fairly complex questions about themselves. Coming from C++ and Rust, where by the time your code executes it's already just "some machine code at some address", it's a big shift in how you understand and work with a running application. I'm used to writing my own abstractions this way, but not necessarily interacting with the language itself like this.

Which brings me back to the code. To start off with the sieve, I populate an instance variable @all in the class constructor:

def initialize(n)
  @all = n > 1 ? 2.upto(n).to_a : []
end

When executing the sieve, you have an upper limit for the primes you want to calculate, and this is passed to the constructor as n. I used the tertiary conditional operator to ask if this upper limit is greater than 1. If it isn't, there's no primes to report, so the fallback at the end is an empty list []. However, most invocations will pass a larger integer, in which case @all gets instantiated with 2.upto(n).to_a.

Another super idiosyncratic thing about Ruby is how pervasive Enumerator is. I find this tendency seeping into the all the code I write, which is why I love learning different types of languages. The first message we pass 2 is upto(n). This returns an Enumerator:

[13] pry(main)> n = 15
=> 5
[14] pry(main)> 2.upto(n)
=> #<Enumerator: ...>

For this application, though, I decided to immediately collect the results in an array with to_a:

[6] pry(main)> 2.upto(n).to_a
=> [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]

To calculate the primes, the English version was "going up a number list one by one, calling each element x and crossing out all numbers from x up to n that are a multiple of x." Here's the Ruby:

@all.each do |x|
      @all.select { |y| y > x }.each do |z|
        @all.delete(z) if (z % x).zero?
      end
    end

We'll take that list we just made, [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], and successively call each one x. Then, we look at every other element higher in the list, calling it z, and delete any in that same list that satisfy (z % x).zero?. If the remainder when dividing z by x is zero, it's a multiple and therefore not a prime. Deleting it from the list ensures we won't waste any more work on an eliminated member of the set.

Before we even got to all this, though, there's a one-liner guard clause:

return @all if @all.empty?

If @all.empty? returns true, we know @all == [], because arrays can respond to that message. When that's the case, the empty list does indeed comprise all the primes up to n, so pass it on up before moving further. This is one particular case where I think this code should be improved - I know in the constructor that the result will be the empty list. However, I don't know how the requirements of this class might change (hypothetically), so I decided to not actually return that result until the primes message is explicitly passed. What's your instinct?

However...that's it. It's more or less what we said in English, but in Ruby. When we eliminate all the numbers that are a multiple of one smaller, we get the primes:

[2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
[2, 3, 5, 7, 9, 11, 13, 15] # after 2
[2, 3, 5, 7, 11, 13] # after 3
[2, 3, 5, 7, 11, 13] # None of the rest have any multiples in the set - these are the primes under 15

All done. You can almost read the code like the English: "take each from @all as x, but specifically select the ones that are higher than x. Delete it from @all if it's a perfect multiple of x". And all along the way, each object already knew how to do or answer everything I needed. All you have to do is ask.

How would you write this simple program?

Aside

On a more personal note, I've been pretty quiet this pandemic - haven't really known what to write while I shift gears and figure out the path forward, and it's been a bunch of rebuilding fundamentals in a more practical context. There's been ample book-readin' (for tool-learnin' and ecosystem-familiarizin') and some problem solving on LeetCode-esque-sites (which have thoroughly humbled and redirected me), and some, you know, messing around and such (incoming chip8 post, I got way bit-wiser). And, like, a bunch of completely non-coding concerns - right? Who's with me? 2020 can go suck it?

Anyway, I've found it all challenging and necessary, but not necessarily post-inspiring. However, I've definitely had my mind blown more than once, and these books are to blame:

The Ruby-specific ones:

It's tough to recommend any over the other. They're all quite different, and I'm finding each indispensable.

The Random Other Crap:

The Art Of Computer Programming - hardly needs an introduction. This is slow going, but not one page is a waste. Put your Assembly cap on.
RHCSA 8 Cert Guide - off topic, but also great if this is in your interests. I also am watching this author's video course, and recommend his Linux instruction material. Straight and to the point, does not waste your time. I found I was already closer than I thought, this is perfect for filling in the gaps. I'm taking the exam in a few weeks.
Calculus and its Applications - Also off-topic. I'm taking Calc 1 in school for the first time, as a 28-year-old would-be engineer, and honestly, it's the most challenging material on here while also blowing my mind. While there is additional course material, I have largely succeeded in the course by reading this textbook and attempting the homework, as someone who is not adept at math by nature. Why didn't any of you tell me calculus was cool?

Cover image: Photo by Matt Seymour on Unsplash

Share Your Conway's Game Of Life

Ben Lovy — Sun, 12 Apr 2020 17:02:37 +0000

Sadly, John Horton Conway has passed away. While Conway's contributions to computational mathematics over the years are numerous and eclectic, many aspiring coders are introduced to his work via Conway's Game of Life, a cellular automaton that's relatively simple to implement.

In the Game of Life, you provide an initial state, or seed, to a square grid of cells. Each cell is either "on" or "off", and the next state of the grid is calculated via these rules, as stated on Wikipedia:

Any live cell with two or three live neighbors survives.
Any dead cell with three live neighbors becomes a live cell.
All other live cells die in the next generation. Similarly, all other dead cells stay dead.

As this is relatively simple to write logic for, many programmers take a stab at implementing such a program early in their careers. Here's mine on GitHub, from July 2017, as one of my first Rust programs - but not terribly interesting to look it as it just renders world states in text to the terminal. However, it does demonstrate how trivial the core logic is to write:


//tick_cell takes a Coord and a World and returns the next state of the cell
fn tick_cell(c: &Coord, w: &World) -> bool {
    let s = moore_sum(c, w);
    if get_cell(c, w) {
        match s {
            2 | 3 => true, //ALIVE
            _ => false, //overcrowded or starved
        }
    } else {
        match s {
            3 => true, //3 gives birth
            _ => false, //barren
        }
    }
}

Do you have a Game of Life to show us? Bonus points for an implementation we can play with - my entry must be downloaded and compiled, but I'm sure some of you web-dev-y types have something more interactive!

What are some of your favorite seeds? Are there other cellular automata that interest you?

Alternatively, what other John Conway work has inspired you? According to this redditor, Conway was frustrated that the Game of Life was his most well-known contribution. What else do you remember him for?

Image: wikimedia commons

Quick And Dirty Java Makefile

Ben Lovy — Wed, 08 Apr 2020 15:07:24 +0000

I'm not a Java user and know next to nothing about it. Today was my first time ever running a line of Java directly. If you've ever used Java before, this is not going to blow your mind.

As part of my Ruby learning journey I'm working on translating some Java (specifically from the excellent Crafting Interpeters book), and while my Ruby implementation is the focus I want to be able to follow along locally with both languages. However, I didn't want to spend a ton of time learning about a whole ecosystem that I'm not targeting. Previously, I'd always assumed Java required bulky build tools and IDEs and lots of peripheral knowledge to hook everything up. I thought I needed use a full-fledged tool like Eclipse or IDEA to set up all the details for you and had no sense of how the build process worked.

As it turns out, while you can lean on tooling, you can of course also just pop some Java code in a text file and build and run it by hand! There is a javac CLI tool. Either use it directly or use GNU make to invoke it automatically.

Here's what I'm starting with:

JAVAC=javac
sources = $(wildcard *.java)
classes = $(sources:.java=.class)

all: program

program: $(classes)

clean:
    rm -f *.class

%.class: %.java
    $(JAVAC) $<

jar: $(classes)
    jar cvf program.jar $(classes)

.PHONY: all program clean jar

edit: added phony targets! Thanks Michael.

With this, you can invoke make to compile a Thing.class bytecode file for each Thing.java source file present. Then you get to run java Thing to execute it! You can also use make jar to roll everything together into a jar file.

All in all, not so different from anything else I've ever used!

This is, as promised, "quick and dirty". You can go forth from here and further customize your makefile, but if you want to do more than just quick experimentation, I do still think the IDE/Java-specific build tool route (Maven, Gradle, etc) is gonna be the way to go. For non-Java-users who just want to plunk about without wasting too much time on peripherals, though, this'll do you just fine.

Photo by Jakub Dziubak on Unsplash

Setting Up A Fresh Ruby Project

Ben Lovy — Thu, 19 Mar 2020 13:05:20 +0000

To start up a Ruby project, you don't need to do anything other than put some Ruby code in a file and invoke it with ruby program.rb.

As projects grow, though, you'll likely add dependencies and tests. You can mitigate this with documentation - tell users what gems they need to pull, and which files contain the test suites - but that becomes unwieldy quick. It's much better to lean on ecosystem tools to specify and automate your build steps.

To follow along, you'll need Ruby installed. The latest at time of writing is version 2.7.0.

EDIT: If you don't want to do this from scratch, try bundler gem:

Comment Not Found

Make a directory

Don't just spread your ruby source all across your home directory - keep your projects self contained. Create a directory:



$ mkdir ruby_project

Open this directory in your favorite editor. Everything else in this post goes in here.

Add Bundler

To manage dependencies, use Bundler. This tool allows you to specify gems required to run your project and lets users issue a single bundle install command to pull down any required.

First, install the gem in you haven't already:



$ gem install bundler

Then, create a file called Gemfile in your project root. For now, we'll just specify the package repository URL:



source 'https://rubygems.org'

As we need dependencies, we'll add them here below this line.

Add a Test file

The first dependency we need is for testing. This example uses minitest. Add that to your Gemfile:



  source 'https://rubygems.org'
+ gem 'minitest'

Now you can invoke bundle install to pull this down automatically. When you do, a new file called Gemfile.lock will be generated containing the specific package version in use.

This is really up to you, but I like keeping my tests in a separate subdirectory:



$ mkdir test

Create a file in this folder called test/cool_program_test.rb with the following contents:



# frozen_string_literal: true

require 'minitest/autorun'
require_relative '../lib/cool_program'

# Test program coolness
class CoolProgramTest < Minitest::Test
  def test_coolness_off_the_charts
    # skip
    assert_equal CoolProgram.new.coolness, 11
  end
end

The first line, # frozen_string_literal: true, is a magic comment that is kind of like calling Object.freeze on every string literal. This is done for performance reasons, but you can opt out for a given string by prefixing it with a + character: +'my newly mutable string literal'. Now you can do my_str << some_other_str. In Ruby, though, I'm not finding myself manipulating strings like this.

We know we'll be able to use the minitest dependency in the first line because we specified it with Bundler. The actual source code doesn't exist yet, though!

Add an Implementation File

This is also personal preference, but it's a sensible idea to keep your implementations in a separate subdirectory as well:



$ mkdir lib

Create the file we pulled into the test file at lib/cool_program.rb with these contents:



# frozen_string_literal: true

# The coolest program
class CoolProgram
  attr_reader :coolness
  def initialize
    @coolness = 11
  end
end

puts "Coolness: #{CoolProgram.new.coolness}/10"

Groovy! Everything's in place.

Add a Rakefile

Now, we could just direct users to invoke ruby themselves on the proper files - ruby lib/cool_program.rb to run the program, or ruby test/cool_program_test.rb for the tests. It would be better if we could specify those paths and abstract into test or run operations. That's what rake is for! This tool is a lot like GNU Make, but for Ruby.

First, add it to your Gemfile:



  source 'https://rubygems.org'
  gem 'minitest'
+ gem 'rake'

Then, add a new file in your project root called Rakefile:



task default: %w[test]

task :run do
  ruby 'lib/cool_program.rb'
end

task :test do
  ruby 'test/cool_program_test.rb'
end

If you know how make works, this is similar (if you don't, I've got ya covered). The first line defines the default task, or what happens when you invoke rake without specifying a specific task. In this case, we just have it run the test task as a dependency with no block itself. This is a list, you can specify multiple tasks here.

Each task below can be invoked at the command line directly - you'd use rake run to actually execute your program and just rake on its own to run the test suite.

Add a Linter

I've become fairly reliant on the Rubocop linter to keep me in line with the Ruby style guide. It now ships with a ready-to-go Rake task.

First, add it to your Gemfile:



  source 'https://rubygems.org'
  gem 'minitest'
  gem 'rake'
+ gem 'rubocop'

Then, modify your Rakefile:



+ require 'rubocop/rake_task'

- task default: %w[test]
+ task default: %w[lint test]

+ RuboCop::RakeTask.new(:lint) do |task|
+   task.patterns = ['lib/**/*.rb', 'test/**/*.rb']
+   task.fail_on_error = false
+ end

  task :run do
    ruby 'lib/cool_program.rb'
  end

  task :test do
    ruby 'test/cool_program_test.rb'
  end

It's my personal preference to run the linter and the tester by default, but not fail on lint errors. Season to taste.

Optional - Add a GitHub Action

I host almost all my code on GitHub and have become a fan of GitHub Actions. Using YAML, you can have GitHub automatically run your tests whenever you commit or open a PR. Create a new file at <project root>/.github/workflows/ruby.yml:



name: Ruby

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2
    - name: Set up Ruby 2.7
      uses: actions/setup-ruby@v1
      with:
        ruby-version: 2.7.x
    - name: Build and test with Rake
      run: |
        gem install bundler
        bundle install --jobs 4 --retry 3
        bundle exec rake

You can tweak each step very easily here - the commands specified will get run and if everything completes without errors, you get a green check mark. Sweet! Using rake allows this file to stay concise, and then you'd be able to manage changes to your structure in your Rakefile without having to worry about this file at all.

Add a README and stuff

We're just about done, but you should always include a README:



# ruby_project

The coolest dang program that you ever did see.

## Dependencies

* [Ruby](https://www.ruby-lang.org/en/).  Written with version [2.7.0](https://www.ruby-lang.org/en/news/2019/12/25/ruby-2-7-0-released/) - *[docs](https://docs.ruby-lang.org/en/2.7.0/)*.

## Usage

Install deps: `gem install bundler && bundle install`.  Run `bundle exec rake` to run the tests, or `bundle exec rake run` to run the program.

Add a .gitignore:



*.gem
*.rbc
/.config
/coverage/
/InstalledFiles
/pkg/
/spec/reports/
/spec/examples.txt
/test/tmp/
/test/version_tmp/
/tmp/

## Environment normalization
/.bundle/
/vendor/bundle
/lib/bundler/man/

# Used by RuboCop. Remote config files pulled in from inherit_from directive.
.rubocop-https?--*

You should probably add a LICENSE file, too - here's the BSD-3-Clause:



Copyright 2020 Coolness McAwesome

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

At the end, your directory should look like this:



$ tree
.
|____Gemfile.lock
|____.gitignore
|____.github
| |____workflows
| | |____ruby.yml
|____Gemfile
|____LICENSE
|____Rakefile
|____test
| |____cool_program_test.rb
|____README.md
|____lib
| |____cool_program.rb

Finally, git init && git add . && git commit -m "Initial commit". Happy hacking!

If you don't feel a pressing need to do this from scratch, you can just use this GitHub template with the code from this post.

Photo by Jukan Tateisi on Unsplash