Forem: Michael Gattozzi

Oxidizing the technical interview

Michael Gattozzi — Tue, 19 May 2020 17:29:23 +0000

Inspired by Reversing the technical interview and Hexing the technical interview

Years ago, as a young'in from the Rust Belt, you dreamed of becoming a systems programmer, an Abyss Gazer if you will. You wanted to live, breathe, and die by being as close to the metal as you could. You grew up in a harsh environment of reading assembly output to shave off a few instructions in an attempt to make the fastest code, segfaults, and poorly documented hardware, all to become a systems programmer. You look back upon those years fondly as it's turned you into the programmer you are today. An unparalleled Abyss Gazer, an Eldritch One, one who makes mortal men fear and worship the code you've wrought upon this world that holds the very foundations of it together. Today you've come for an interview at the request of a friend for a position using Rust. They hope to be able to hire you so that you may channel the will of Ferris the Rustacean into all the code that flies from your fingers for them. Of course they can't just give you the job. Your potential future comrades must see for themselves whether the rumors about your skill are true. You look to your interviewer who asks if you're ready to get started. You nod, ready to take on any task given to you.

"You're given an array of size n + 1 and the integers 0 to n. Find the duplicate in the list using an algorithm that optimizes for size and one that optimizes for execution in terms of Big O. Make sure to only use the stable release of Rust as we do not use nightly here.", your interviewer explains. "Does it matter if I just do 1 algorithm that can do both in O(1)?", you ask. The interviewer looks over their notes in confusion as this was not one of the answers they were told was possible. They're a younger engineer and you are being tested for a Principal Engineering position. It's nice that you're already able to prove you can do your job to teach those beneath you in terms of seniority. "Uh sure. If you can uh prove it.", they say.

You whip out your MacBook to act as a conduit to Wozniak himself, so that he may guide you as he has many times before. You empty yourself to become the vessel for those who gazed into the abyss before you. "Let's get started by creating a project to house the summoning." you say, making sure to explain your process. The interviewer gives a slight nod, but they seem more on guard than they were earlier. Maybe summoning wasn't the right word. Incantation maybe? Either way, you type the magic words to your console like all Rustaceans before, now, and after will to start the world anew.

cargo new interview

"Alright let's setup our manifest to ship high quality artisanal hand crafted code to our end users." You open up the Cargo.toml file and fill it with the incantation of optimization you've ingrained into your soul

[package]
name = "interview"
version = "0.1.0"
authors = ["Z̴̧̡̢̨̛A̷̡̡̢̢̡L̶̢̧̢̪̬Ģ̵̡̢̢̧̛Ơ̵̡̡̬͇̬ ̵̡̧̢̛̳Ċ̷̢̨̢̼̰Ǫ̷̨̧̢̛͎M̵̡̨̖̹̣E̴̢̧̨̨̛S̸̱̠̹̮̣ <Z̴̧̡̢̨̛A̷̡̡̢̢̡L̶̢̧̢̪̬Ģ̵̡̢̢̧̛Ơ̵̡̡̬͇̬ ̵̡̧̢̛̳Ċ̷̢̨̢̼̰Ǫ̷̨̧̢̛͎M̵̡̨̖̹̣E̴̢̧̨̨̛S̸̱̠̹̮̣@gmail.com>"]
edition = "2018"

[dependencies]

[profile.release]
panic = "abort"
lto = true
codegen-units=1
opt-level="s"

"Wait what's up with your name?", the interviewer asks. You wave them aside and reply, "Worry not sweet summer child, lest yee be cursed as I have been. The important part is that we've optimized for code gen size and to strip out as many unnecessary assembly instructions as possible from the code." "Wouldn't that mean you should use z for size?", they ask. "What? And add one whole useless instruction to the output? I think not."

You quickly open up your editor to src/main.rs. You crack your knuckles and start off your incantation by releasing the limiters.

#![feature(const_fn)]
#![feature(const_fn_union)]
#![feature(const_panic)]
#![feature(const_if_match)]
#![feature(const_raw_ptr_deref)]
#![feature(lang_items)]
#![feature(no_core)]
#![feature(optin_builtin_traits)]
#![feature(allow_internal_unstable)]
#![feature(intrinsics)]
#![feature(untagged_unions)]

"Wait wait wait. I said use a stable Rust compiler and these are all nightly features. Also what in the world do you need all this for?" asks your interviewer in anger. "I am using a stable Rust compiler. Just like you asked" you say showing them the version number, though this is more a technicality, but they don't know it yet. "Uhhh okay whatever keep going".

"No libstd, no core, only me" you whisper as you place the line that will enable true freedom from the shackles of having any Rust dependencies whatsoever.

#![no_core]

It's then that you see the look of abject horror on your interviewer's face. Good good let them stare into the depths of what system's programming truly is. "Alright so IO is a thing as much as functional programmers try to avoid it by using 'pure' functions. If we want to see the output of something we'll need to make some kind of syscall since operating systems exist. We'll link to libc so that we can use printf at least. We'll link directly to the OSX libc here with this line, but depending on the OS we would want to link a different libc implementation instead."

It pains you to have any dependency whatsoever, but you're not trying to also write a whole Operating System today, that's for next week's interview. There's no librs you can use yet unfortunately and so, like a blacksmith carefully linking metal rings in a set of chainmail, you link to the venerable libc.

#[allow(non_camel_case_types)]
pub type c_char = i8;
#[allow(non_camel_case_types)]
pub type c_int = i32;
#[link(name = "System")] // OSX libc
extern "C" {
    pub fn printf(format: *const c_char, ...) -> c_int;
}

"Okay okay, but are you going to even get to the actual problem?", the interviewer asks, clearly miffed that you're bothering to use C code instead of println. Oh the sweet summer child. Everything is built on top the work of Kerninghan & Ritchie. We can't escape their legacy that easily.

"Okay let's add some macros to cut down on some of the repetitive stuff later. We have to put them here otherwise they won't exist at all if they come after the places we invoke them at." you utter, in an attempt to convince the clearly bereaved interviewer that you do actually understand the language you're using and that you're not yanking them around for no reason.

macro_rules! matches {
    ($expression:expr, $( $pattern:pat )|+ $( if $guard: expr )?) => {
        match $expression {
            $( $pattern )|+ $( if $guard )? => true,
            _ => false
        }
    }
}
macro_rules! copy_clone_eq_impls {
   ($($t:ty)*) => {
     $(impl Copy for $t {}
       impl Clone for $t {
         fn clone(&self) -> Self {
            *self
         }
       }
       impl PartialEq for $t {
           fn eq(&self, other: &$t) -> bool {
               (*self) == (*other)
           }
       }
     )*
   }
}

"Why not derive for those?", they ask. You look at them dead in the eyes and explain to them, "I can't implement these for primitives via a derive macro. I have to *become* the compiler itself in order for these to work. Now let's continue. We have an apple pie to make and first we must invent the universe. Let's add some traits and import some compiler intrinsics we're going to need."

#[lang = "sized"]
pub trait Sized {}
#[lang = "freeze"]
auto trait Freeze {}
extern "rust-intrinsic" {
    fn offset<T>(dst: *const T, offset: isize) -> *const T;
}
#[lang = "receiver"]
trait Receiver {}
#[lang = "index"]
trait Index<Idx: ?Sized> {
    type Output: ?Sized;
    fn index(&self, index: Idx) -> &Self::Output;
}
impl<T, I> Index<I> for [T]
where
    I: SliceIndex<[T]>,
{
    type Output = I::Output;
    fn index(&self, index: I) -> &I::Output {
        index.index(self)
    }
}
trait SliceIndex<T: ?Sized> {
    type Output: ?Sized;
    fn get(self, slice: &T) -> Option<&Self::Output>;
    fn get_mut(self, slice: &mut T) -> Option<&mut Self::Output>;
    unsafe fn get_unchecked(self, slice: &T) -> &Self::Output;
    unsafe fn get_unchecked_mut(self, slice: &mut T) -> &mut Self::Output;
    fn index(self, slice: &T) -> &Self::Output;
    fn index_mut(self, slice: &mut T) -> &mut Self::Output;
}
#[lang = "copy"]
trait Copy: Clone {}
trait Clone: Sized {
    fn clone(&self) -> Self;
    fn clone_from(&mut self, source: &Self) {
        *self = source.clone()
    }
}
#[lang = "eq"]
trait PartialEq<Rhs: ?Sized = Self> {
    fn eq(&self, other: &Rhs) -> bool;
    fn ne(&self, other: &Rhs) -> bool {
        !self.eq(other)
    }
}
#[lang = "partial_ord"]
trait PartialOrd<Rhs: ?Sized = Self>: PartialEq<Rhs> {
    fn partial_cmp(&self, other: &Rhs) -> Option<Ordering>;
    fn lt(&self, other: &Rhs) -> bool {
        matches!(self.partial_cmp(other), Some(Less))
    }
    fn le(&self, other: &Rhs) -> bool {
        matches!(self.partial_cmp(other), Some(Less) | Some(Equal))
    }
    fn gt(&self, other: &Rhs) -> bool {
        matches!(self.partial_cmp(other), Some(Greater))
    }
    fn ge(&self, other: &Rhs) -> bool {
        matches!(self.partial_cmp(other), Some(Greater) | Some(Equal))
    }
}
#[lang = "not"]
trait Not {
    type Output;

    fn not(self) -> Self::Output;
}
#[lang = "neg"]
trait Neg {
    type Output;
    fn neg(self) -> Self::Output;
}
#[lang = "sub"]
trait Sub<Rhs = Self> {
    type Output;
    fn sub(self, rhs: Rhs) -> Self::Output;
}
trait Termination {
    fn report(self) -> i32;
}

Like any good Rustacean worth their sea salt, you've drilled every bit of libstd and libcore into your mind so that you only pull in the items that you need and nothing more. You believe in zero cost abstractions, no, you are zero cost abstractions. Even the possibility of paying for just a CPU cycle or bit more disgusts you. Only run what must be run and nothing more.

"Wait aren't these traits and items from libcore and libstd? What. What's going on here?" your interviewer asks, clearly distressed that it's taking this much code. "Well yes since we're using #![no_core] they simply do not exist anymore. We're only defining exactly what we need from there and nothing more. Besides I'm being as explicit as possible by letting you stare at the true cost it takes to program on any machine, by taking the syntax sugar and prebuilt code away. Speaking of sugar got any coffee? I like it black like the soul I sold to 'them' to get better at this kind of stuff." Oof maybe the self deprecating joke was a bit much, even if it was true. You were only trying to lighten the mood, but the interviewer seems to have pretended to hear nothing.

"Okay now lets implement all these traits then!" you say, fingers flying over the keyboard, never leaving insert mode. Esc, h, j, k, l? Ha those keys are for those who haven't dedicated themselves to the abyss.

impl<T, I> Index<I> for [T]
where
    I: SliceIndex<[T]>,
{
    type Output = I::Output;
    fn index(&self, index: I) -> &I::Output {
        index.index(self)
    }
}
trait SliceIndex<T: ?Sized> {
    type Output: ?Sized;
    fn get(self, slice: &T) -> Option<&Self::Output>;
    fn get_mut(self, slice: &mut T) -> Option<&mut Self::Output>;
    unsafe fn get_unchecked(self, slice: &T) -> &Self::Output;
    unsafe fn get_unchecked_mut(self, slice: &mut T) -> &mut Self::Output;
    fn index(self, slice: &T) -> &Self::Output;
    fn index_mut(self, slice: &mut T) -> &mut Self::Output;
}
#[lang = "copy"]
trait Copy: Clone {}
trait Clone: Sized {
    fn clone(&self) -> Self;
    fn clone_from(&mut self, source: &Self) {
        *self = source.clone()
    }
}
#[lang = "eq"]
trait PartialEq<Rhs: ?Sized = Self> {
    fn eq(&self, other: &Rhs) -> bool;
    fn ne(&self, other: &Rhs) -> bool {
        !self.eq(other)
    }
}
#[lang = "partial_ord"]
trait PartialOrd<Rhs: ?Sized = Self>: PartialEq<Rhs> {
    fn partial_cmp(&self, other: &Rhs) -> Option<Ordering>;
    fn lt(&self, other: &Rhs) -> bool {
        matches!(self.partial_cmp(other), Some(Less))
    }
    fn le(&self, other: &Rhs) -> bool {
        matches!(self.partial_cmp(other), Some(Less) | Some(Equal))
    }
    fn gt(&self, other: &Rhs) -> bool {
        matches!(self.partial_cmp(other), Some(Greater))
    }
    fn ge(&self, other: &Rhs) -> bool {
        matches!(self.partial_cmp(other), Some(Greater) | Some(Equal))
    }
}
#[lang = "not"]
trait Not {
    type Output;

    fn not(self) -> Self::Output;
}
#[lang = "neg"]
trait Neg {
    type Output;
    fn neg(self) -> Self::Output;
}
#[lang = "sub"]
trait Sub<Rhs = Self> {
    type Output;
    fn sub(self, rhs: Rhs) -> Self::Output;
}
trait Termination {
    fn report(self) -> i32;
}
#[lang = "slice"]
impl<T> [T] {
    #[allow(unused_attributes)]
    #[allow_internal_unstable(const_fn_union)]
    const fn len(&self) -> usize {
        unsafe { Repr { rust: self }.raw.len }
    }

    fn as_ptr(&self) -> *const T {
        self as *const [T] as *const T
    }

    fn as_mut_ptr(&mut self) -> *mut T {
        self as *mut [T] as *mut T
    }
}

#[lang = "const_ptr"]
impl<T: ?Sized> *const T {
    unsafe fn add(self, count: usize) -> Self
    where
        T: Sized,
    {
        self.offset(count as isize)
    }

    unsafe fn offset(self, count: isize) -> *const T
    where
        T: Sized,
    {
        offset(self, count)
    }
}
#[lang = "mut_ptr"]
impl<T: ?Sized> *mut T {
    unsafe fn add(self, count: usize) -> *mut T
    where
        T: Sized,
    {
        self.offset(count as isize)
    }
    unsafe fn offset(self, count: isize) -> *mut T
    where
        T: Sized,
    {
        offset(self, count) as *mut T
    }
}
impl<T> SliceIndex<[T]> for usize {
    type Output = T;
    fn get(self, slice: &[T]) -> Option<&T> {
        if self < slice.len() {
            unsafe { Some(self.get_unchecked(slice)) }
        } else {
            None
        }
    }
    fn get_mut(self, slice: &mut [T]) -> Option<&mut T> {
        if self < slice.len() {
            unsafe { Some(self.get_unchecked_mut(slice)) }
        } else {
            None
        }
    }
    unsafe fn get_unchecked(self, slice: &[T]) -> &T {
        &*slice.as_ptr().add(self)
    }
    unsafe fn get_unchecked_mut(self, slice: &mut [T]) -> &mut T {
        &mut *slice.as_mut_ptr().add(self)
    }
    fn index(self, slice: &[T]) -> &T {
        &(*slice)[self]
    }
    fn index_mut(self, slice: &mut [T]) -> &mut T {
        &mut (*slice)[self]
    }
}
#[allow(unconditional_recursion)]
impl PartialOrd for usize {
    fn partial_cmp(&self, other: &usize) -> Option<Ordering> {
        self.partial_cmp(other)
    }
}
impl Not for bool {
    type Output = bool;
    fn not(self) -> bool {
        !self
    }
}
impl Neg for isize {
    type Output = isize;

    fn neg(self) -> isize {
        -self
    }
}
impl Sub for usize {
    type Output = usize;
    fn sub(self, rhs: usize) -> Self::Output {
        self - rhs
    }
}
impl Termination for () {
    fn report(self) -> i32 {
        0
    }
}
impl<T: ?Sized> Receiver for &T {}
impl<T: ?Sized> Receiver for &mut T {}
copy_clone_eq_impls!(usize u64 bool);

"Wait hold on, the op Sub uses - as part of it's definition? The thing it's supposed to define and wait Not does this too with ! and wait wait wait." Your interviewer is sputtering in disbelief. Maybe they were too young to be corrupted by your influence. It's too late now. You've already shown them what the grimoire contains. You can only go deeper now. You're almost done.

#[allow(dead_code)] // It's actually needed! Zombie Ordering
enum Ordering {
    Less = -1,
    Equal = 0,
    Greater = 1,
}
use Ordering::*;
enum Option<T> {
    Some(T),
    None,
}
use crate::Option::*;
#[repr(C)]
pub(crate) union Repr<T> {
    pub(crate) rust: *const [T],
    rust_mut: *mut [T],
    pub(crate) raw: FatPtr<T>,
}
#[repr(C)]
pub(crate) struct FatPtr<T> {
    data: *const T,
    pub(crate) len: usize,
}

You create the only concrete types that you'll need beyond the primitives, but they're transitory like the feeble lives we live. Your interviewer just looks at you as you show them the depths of what Rust truly is and the secrets it hides behind the veil. Your interviewer begins to sputter unintelligible words. "nmbrs. enms in em?!?" You better wrap this up quickly and bring them back to the living before they cross the river Styx. You're trying to get a job here.

#[lang = "start"]
fn start<T: Termination + 'static>(main: fn() -> T, _: isize, _: *const *const u8) -> isize {
    main().report() as isize
}

fn main() {
    unsafe {
        printf(
            "The value is: %d\n\0" as *const str as *const u8 as *const c_char,
            DUPLICATE,
        );
    }
}

This is it. The final stretch. The anticipation is killing you. "What's with this type signature? What's start?! Rust has no runtime!", your interviewer sputters. You turn to them, "The greatest trick the Devil ever played was convincing C and Rust programmers their language has no runtime." You turn back to your computer. You sigh and say, "Look we're almost done. This is the last bit I swear." You begin typing in the last words that will let you have an O(1) runtime and O(1) space usage when you finally execute the code. Truly the most magical of the incantations you've wrought thus far.

pub const DUPLICATE: u64 = {
    pub const LIST: &'static [u64] = {
        unsafe {
            &*Repr {
                raw: FatPtr {
                    data: &[
                        1u64, 2u64, 3u64, 7u64, 4u64, 5u64,
                        6u64, 7u64, 8u64, 9u64, 10u64,
                    ] as *const u64,
                    len: 11,
                },
            }
            .rust
        }
    };

    const fn recurse(item: usize, compare: usize) -> u64 {
        if compare == 0usize && item != 0usize {
            recurse(item - 1, 10)
        }
        // else if compare == 0 && item == 0
        // Pigeon Hole principle means we're guaranteed to handle this case
        else if compare == item {
            recurse(item, compare - 1)
        } else {
            if LIST[compare] == LIST[item] {
                return LIST[compare];
            } else {
                recurse(item, compare - 1)
            }
        }
    }
    recurse(10, 10)
};

"Wait wait wait that's the naive solution! This is O(n^2) runtime!", your interviewer says exasperated at the rigamarole you've run them through. "No no no no that's only theoretically true!", you say, "In practice the use of const causes the compiler to make this constant time and space! Watch!" You crack your hands and open up the gates of hell yourself typing the heretical incantation only the privileged rustc coders are allowed to use to compile the compiler lest the spawn of the underworld destroy the illusion of stability.

interview is 📦 v0.1.0 via 🦀 v1.43.0
❯ RUSTC_BOOTSTRAP=1 cargo run --release
   Compiling oxidizing-the-interview v0.1.0 (/Users/Z̴̧̡̢̨̛A̷̡̡̢̢̡L̶̢̧̢̪̬Ģ̵̡̢̢̧̛Ơ̵̡̡̬͇̬ ̵̡̧̢̛̳Ċ̷̢̨̢̼̰Ǫ̷̨̧̢̛͎M̵̡̨̖̹̣E̴̢̧̨̨̛S̸̱̠̹̮̣/interview)
    Finished release [optimized] target(s) in 0.17s
     Running `target/release/interview`
The value is: 7

interview is 📦 v0.1.0 via 🦀 v1.43.0
❯ RUSTC_BOOTSTRAP=1 cargo rustc --release -- --emit asm
    Compiling interview v0.1.0 (/Users/Z̴̧̡̢̨̛A̷̡̡̢̢̡L̶̢̧̢̪̬Ģ̵̡̢̢̧̛Ơ̵̡̡̬͇̬ ̵̡̧̢̛̳Ċ̷̢̨̢̼̰Ǫ̷̨̧̢̛͎M̵̡̨̖̹̣E̴̢̧̨̨̛S̸̱̠̹̮̣/interview)
     Finished release [optimized] target(s) in 0.19s

interview is 📦 v0.1.0 via 🦀 v1.43.0
❯ cat target/release/deps/oxidizing_the_interview-6201aba656776dbc.s
    .section    __TEXT,__ text,regular,pure_instructions
    .macosx_version_min 10, 7
    .globl _main
_main:
    pushq %rbp
    movq %rsp, %rbp
    leaq l___unnamed_1(%rip), %rdi
    movl $7, %esi
    xorl %eax, %eax
    callq _printf
    xorl %eax, %eax
    popq %rbp
    retq

    .section __TEXT,__ const
l___unnamed_1:
    .asciz "The value is: %d\n"

.subsections_via_symbols

interview is 📦 v0.1.0 via 🦀 v1.43.0
❯ cat target/release/deps/interview-6201aba656776dbc.s | wc -l
    20

"See I told you I would only use a stable compiler. You just had to trust me! Now look at this code's asm output. Only 20 lines! 17 if you cut out the empty newlines! No handwritten assembly needed! It's all high level Rust code and look the 7 is in there and it just gets pushed into a register and printed out, so I've proved it's constant time and space in practice!"

Your interviewer, eyes white and rolled back into their head says, "We'll contact you later. Thanks for coming in." Oh no, you've heard this line before. You might not get the job. "Look I know I wrote the code with only one array, but I could make this work for any array by using a build.rs script to insert them into the file at compile time based off an external file used as input to define what numbers it should contain and thus allowing the code to find any duplicates like the problem asked for!", you say hoping to impress the interviewer in a last ditch effort. "No really. We'll contact you. You've more than answered the question." your interviewer sputters, rushing to get you out of the room. You can't help but feel like you've failed the interview, but your dark patron is pleased for showing another the truth and the way of the computer they use. Oh well, onto the next interview. Maybe they'll be ready for your enlightenment.

You can find the code from this blog post here.

Redistributing Distributed Work

Michael Gattozzi — Mon, 02 Dec 2019 19:13:37 +0000

I’ve been working on a project recently that aims to make code, tooling, and planning live side by side in git. There are two big problems as I see them today. Code is stored on some centralized service like GitHub, Atlassian, or GitLab tying you in to them due to their tooling around things like tickets and integrations like CI. The other issue is that these things live outside the history of your code, despite being as critical if not more so to understanding why changes were made.

While things like GitHub have made it more accessible than ever for people to join in and code, it has concentrated power around a few companies, abstracting over a tool designed for distributed work and centralizing it. What I’ve been working on is as much a political statement as it is an attempt to simplify today’s workflows to do more work in repo on disk and have history that stores not only code but the plans, tooling, and other critical context that code needs. ICE is bad for a whole host of reasons which includes locking kids in cages and GitHub and it’s CEO Nat Friedman as well as their now parent company Microsoft are absolutely okay with this. Anything we can do to move people off these platforms is an inherently good thing. I also think that the current trend towards centralization is a big mistake that has really killed the vibrancy of the internet. I want to help bring that back.

There’s a trick to doing this and being successful though. It has to be software that’s easy to use and intuitive and needs to have the same value add as GitHub. This means abstracting over the not particularly fun or easy parts git to create a user experience that’s as nice as the services that people use today. I’d like to talk a little bit about the first part of this suite of tools I’ve been working on, what future tools and plans are, as well as some of my reasoning for staying a somewhat private project for now.

Ticket

The first thing I’ve built and done the most work on is Ticket. It’s an in repo ticket manager. Your issues get stored as part of the repo so that not only do you know when a particular issue came up, but when it closed in relation to the code in your repo. I’ve also been working on a TUI to browse the issues as well. It’s fairly bare bones as is all projects when started, but it works and it’s how I’m keeping track of

issues and things that need to get done inside of dev-suite (tentative name for now). My most immediate work has been finishing out the basic TUI browser and switching to using uuids that are generated with timestamps and can be ordered by them, over than a singular id number increasing over time.

The TUI displaying title, ticket assignee, what the status is and a description of the currently selected ticket

Once those are done I’ll be able to move on to doing some of the following:

Adding comments to issues
Assigning one or more users to issues
Commenting from the TUI
Assigning users via the TUI
Adding custom configurable states such as "Won't Fix"
Issue tags
Search/Filter

and more as it gains better functionality. The designing philosophy around this tool is "Don't be Jira". The less time you need to use this and other tools because you get what you need quickly, the better. It should not take 20 hoops to get what you want.

Hooked

Hooked is the in repo githook manager I've been working on. It sets up all the hooks you might want for your code and symlinks them to .git/hooks for you automatically. No need to remember which way the args for ln -s go! This also means that your hooks live inside the repo and version controlled while there are lots of services you can use to handle linting etc. it's really nice to just enforce it automatically with hooks that are able to travel with your code. Now there's a lot of

teams that might not want that and I get it. For me though it's been great to know that every commit builds and that I can get the same experience across all my machines. The goal of hooked is to make it easy to initialize scripts, in the scripting lang you choose, for the hooks. This can make it easier for teams to enforce styling, but skip with --no-verify if just making one off junk commits and not wanting to run the whole test suite. Really what I hope people use this for is to embrace trunk based development, commiting straight to master, all while following the "not rocket science" rule and making sure master is green before you push. githooks are a tool and how you use them is up to you. This is about creating a suite of tools that work well together, but you should only really use the ones you want to use.

Future Tools

So far that's all I've worked on, so it's fairly early days, but other tools I want to work on or have thought about and not settled on are:

Easy to do distributed pull requests that stores comments as commits, saves the PR as commits that can be viewed later in time, and only actually merging the changes that were made, not the comments easily
Project planning tools since tickets the way JIRA and other software does is a bit of an anti-pattern
git fork, clone a repo locally that sets the upstream automatically and setting the origin as the repo you plan to push your code to.
Better tooling around emailing and code. Email will be around till I die probably, so finding out how it can be effectively used for communication and maybe even storing it as part of the history. This is one of my more vague ideas
Changelog generator

There's probably even way more that can be done here, but the main thing for now is recreating that GitHub/GitLab ease of use and experience in repo and further enhancing it. We want to redistribute where code lives and how work is done, but that doesn't mean we need to regress and give up the quality experiences we have today.

Why private?

For now it's a single person project because frankly I'm just not good at maintaining code or working on things due to mental health issues. When it is my job and I'm paid for it it's not as hard cause I like to eat and you know live. For side projects though it's like video games to me. I pick them up and play for awhile put it down and forget about it for some time and then get hyper focused months later. This isn't exactly sustainable, but getting consistent contributors is hard to do as well. For now I just want to work on this alone because I won't feel guilty if I need to put it down for a month or three, and because it's a lot of real small tools working together in one project and not one large monolithic tool I'll feel isn't making progress it really leans into my ability to hyper focus sometimes on interesting things, but there are 50 things to choose from which increases the chances of hyper focusing.

The tl;dr mental health is tough, but as I've begun to understand who I am it's made it easier to build my environment around how I work rather than trying to adapt to the environment. I do think I'll release some initial builds of some tools at some point, but no guarantees right now.

git commit -m "Finish writing the blog post" && git push

All in all it's been fun working on this so far and the reactions to what I've shown on Twitter have been very positive, so I feel like I'm on to something. I'm hoping to post updates on a more frequent basis. I feel like this is a project I might be able to stick with for awhile.

Turning GitHub Into Your Own Registry

Michael Gattozzi — Tue, 11 Jun 2019 23:49:18 +0000

Occasionally I have ideas. They're not necessarily good ideas and occasionally I make the mistake of acting on them. What you're about to read is one of those times.

Look I'm not saying I turned @github into an alternate registry for @rustlang packages, but actually I did. Now if they add actual support to the package repos beta that would be neat. pic.twitter.com/GadqKxXSbJ

— Michael Gattozzi (@mgattozzi ) June 11, 2019

So how do you, dear reader, do this weird (sort of useless) thing? Follow me and you'll be in a world of pure configuration.

Index me like one of your git repos

Well the first step is you've got to actually set up an index. The way cargo alternative registries work is that it assumes you have an index to tell it what crates you have available as well as where to get the things. First up you'll want to set up a git repo on GitHub. Now I haven't tested this with private repos but I have a suspicion it won't work for authentication reasons. The token aspect of cargo login for a registry might work. It probably won't. For our purposes assume we're using a public repo for everything.

Create a file in the top level of your repo called config.json it should look something like this (replace mgattozzi with your github username):

{
    "dl": "https://github.com/mgattozzi/{crate}/releases/download/v{version}/{crate}-{version}.crate"
}

There is another field for the web api if you are running a crates.io instance but we aren't here so we can just drop it. What we're telling cargo is that all the crate files it needs will be in some repo we own, with the crate name being the repo name, and as part of tagged releases. This allows us to set a deterministic url for downloading, version our crates by release, and have it work for all of our own personal crates. Now the v in /v of the url doesn't need to be there. It's just the prefix I use to tag releases. As long as you're consistent and but the version as part of it when you tag your release you should be good to go.

Cool so we have a config file. What about a way to actually list crates? Well there's a file format and a way to store the file. Here are the rules for the file hierarchy in the docs:

The rest of the index repository contains one file for each package, where the filename is the name of the package in lowercase. Each version of the package has a separate line in the file. The files are organized in a tier of directories:

Packages with 1 character names are placed in a directory named 1.

Packages with 2 character names are placed in a directory named 2. -Packages with 3 character names are placed in the directory 3/{first-character} where {first-character} is the first character of the package name.

All other packages are stored in directories named {first-two}/{second-two} where the top directory is the first two characters of the package name, and the next subdirectory is the third and fourth characters of the package name. For example, cargo would be stored in a file named ca/rg/cargo.

So in this example my crate's name is abrade, so my file would be under ab/ra/abrade.

Here's what it looks like:

{"name": "abrade","vers": "0.1.0","deps": [],"cksum": "65b07c5782e771125d9feda1ec08d908087997051339c492220346a14d6dd936","features": {},"yanked": false,"links": null}

Note each version for a crate has an entry and has to be on it's own line otherwise it just won't work. This means pretty multiline JSON is a no go. The cksum field is a SHA256 of the .crate file we'll need to generate. After doing that step (which we'll cover next) commit the file and the config and push it to GitHub. The docs for the registryare a bit light but go more indepth as to what an entry should look like. There's also the actual indexfor crates.io that you can look at for more examples.

Box

Okay so how do you generate a .crate file anyways? Luckily for us cargo has a subcommand for us! After you know what commit you'll use for the release is and you've pushed it to GitHub, just run cargo package. If you look under target/package you should see a file {crate}-{version}.crate. In my case it was abrade-0.1.0.crate. That's it. Next thing you have to do is tag a release on GitHub and make this .crate file as part of the release. Finally to use it in our own code.

Me, My Registry, and I

Okay so in whatever project you have you will want to specify it should use your alternative registry. In your Cargo.toml put something like this (depending on your crate):

[dependencies]
abrade = { version = "0.1.0", registry = "my-registry" }

Now cargo doesn't know what my-registry is, so we need to tell it. Open up this file as we need to modify it ~/.cargo/config (or the equivalent on Windows). Now this file might not exist if you haven't done this before. That's fine! You'll need to add something like this to it:

[registries]
my-registry = { index = "https://github.com/mgattozzi/mgattozzis-crates-on-github-index.git" }

This is the http url for your index. Replace my username/repo with yours for your index. Then that's it. cargo build and you should see that it pulls it down and compiles it!

You can find my repos for reference below:

You can find more about the config file for cargo at this link.

Cargo Culting

Should you actually do this? I mean no probably not, but you can and that's the cool thing. There's some limitations (like only your crates) and it's a manual process since you can't just cargo publish but that's an automatable thing. Still if you want the option is there if you want to be a bit of a renegade, but you'd be better of just using crates.io or setting up a registry on your own server. I hope you enjoyed following along a bad idea. It's just always fun to see systems used in unconventional ways.

GitHub Actions: An introductory look and first impressions

Michael Gattozzi — Tue, 04 Dec 2018 23:21:39 +0000

I recently had the benefit of gaining access to GitHub’s Actions beta. What is GitHub Actions? According to the site it says, “With GitHub Actions you can automate your workflow from idea to production”. Essentially we get to run code that responds to GitHub webhooks by spinning up a Docker container and running it. This means we can automate pretty much anything you can think of:

When you change the version number in a certain file tag the release in GitHub, build static binaries, and push them to the tag
When someone has their PR merged, and the branch is on your repo, delete the branch
If you push a new commit to master, create a Docker container, and tag it as latest, then push it to a container registry.

These are just a few things you could do. In fact with a little know how with the GitHub API, their webhooks, and your own server you could already do this. However, not everyone has infrastructure to do this or can afford it (which it should be noted you can only use Actions on private repos, which costs money, as of December 4th, 2018 so who knows if that will change). I’m personally hoping GitHub will make this free for open source code to lower barriers of entry and automation with DevOps.

What does this mean though for GitHub Actions currently? In your code you’ll have access to the webhooks, details about the repo, an API token scoped to the repo to interact with the API, and the ability to link multiple actions together with access to the state change between linked actions. Meaning you don’t need to setup a whole server just to run actions. GitHub does it for you and provides you the information you need to be able to actually automate decisions for your repo. That way you can spend less time manually doing things and more time actually coding.

I’m gonna go over a small thing I did and then go over my experience with it, which keep in mind this is Beta software, and then some final conclusions. Hopefully this will give you a starting point of something beyond the Hello World Example. Let’s get started!

Rusty Containers

Anyone who follows what I do knows I use Rust extensively and so I’m also writing my repo workflows with it, but fret not a lot of what I’m covering is still relevant to whatever language you decide to use with GitHub Actions! Really, I want you to pick up some of the following from this section:

Understanding the GitHub API
Understanding how containers are setup with Actions (i.e. ENV Vars)
Understanding limitations of actions

Let’s take a look at what I got working. I’ve been wanting to make an automated management bot like Rust’s bors for forever. I called mine Thearesia and in fact this was originally why I started to write github-rs. Admittedly I’ve let the project languish because I don’t enjoy maintenance and like making new stuff that does weird things. I created a repo called Operator21O named after the character of the same name from NieR: Automata (easily one of my top 3 games this year and top 10 all time).

What it does so far is that if an Owner, Member of an Org, or a Collaborator says, “r+” on a PR it will automatically merge it. Some caveats to this; it has not been implemented to handle if the merge isn’t green. If Travis is still running or a collaborator blocked it with their PR review it’ll be red. That’s fine though! This was a great first step and this alone gives us a lot to work with and learn from!

Let’s first take a look at the workflow file that goes in the .github folder at the root of the repo:

workflow "PR Issue Comment" {
  resolves = "IssueComment"
  on = "issue_comment"
}

action "IssueComment" {
  uses = "./operator_issue_comment"
  secrets = ["GITHUB_TOKEN"]
}

What is this saying? We’ve created this file called main.workflow and in it is the text above. We have some kind of workflow called PR Issue Comment that I named as such. We say that it resolves IssueComment. This can be an array of actions we want to trigger actually, but we only use the one here. We then specify what GitHub webhook to trigger this workflow on. In this case we say, “Run this workflow if it’s an issue_comment event.” Neat! Workflows are our triggers and actions are what comes after. We only have one action here but you can chain multiple together so building a container could be one action and the next action then pushes it to a container registry if it had built successfully.

Let’s take a look at the action I wrote here. I’ve named it IssueComment. The uses section can be all sorts of things; containers on a registry or, in my case since I wanted to Dogfood my workflows, a path to a Dockerfile in the repo. The secrets field can contain, well, secrets. There’s docs on how to handle this but currently not using it for production is the current state it is in. I figured this out way later in the docs (which I’ll put in my review overall later), but if you want access to the GITHUB_TOKEN for API requests as an ENV var you have to specify it in the action first.

Great so we have our workflow file so let’s actually look at the code I wrote. Here’s the whole file:

#[macro_use]
extern crate serde_derive;
extern crate serde;
extern crate serde_json;
extern crate reqwest;
extern crate http;

use std::{fs, env, process::exit, error::Error};
use reqwest::header;
use http::StatusCode;
fn main() {
    if let Err(e) = run() {
        eprintln!("{}", e);
        exit(1);
    }
}

fn run() -> Result<(), Box<Error>> {
    let event = serde_json::from_str::<IssueCommentEvent>(
        &fs::read_to_string(env::var("GITHUB_EVENT_PATH")?)?
    )?;

    let mut headers = header::HeaderMap::new();
    headers.insert(header::ACCEPT, header::HeaderValue::from_static("application/vnd.github.v3+json"));
    let client = reqwest::Client::builder()
        .build()?;

    let aa = &*event.issue.author_association;
    if aa == "COLLABORATOR" || aa == "OWNER" || aa == "MEMBER" {
        let cb = &*event.comment.body;
        if cb.contains("r+") {
            let hurl = &*event.issue.html_url;
            if hurl.contains("pull") {
                let rurl = &*event.issue.repository_url;
                let num = event.issue.number;
                let result = client
                    .put(&format!("{}/pulls/{}/merge", rurl,num))
                    .bearer_auth(&env::var("GITHUB_TOKEN")?)
                    .send()?
                    .status();

                if result == StatusCode::OK {
                    println!("PR Merged!");
                } else {
                    eprintln!("Unable to merge PR!");
                    exit(1);
                }
            } else {
                eprintln!("Can't r+ on an issue only a PR");
            }
        } else if cb.contains("r-") {
        }
    }
    Ok(())
}

#[derive(Serialize, Deserialize, Debug)]
struct Comment {
  author_association: String,
  body: String,
  created_at: String,
  html_url: String,
  id: i64,
  issue_url: String,
  node_id: String,
  updated_at: String,
  url: String,
  user: User,
}

#[derive(Serialize, Deserialize, Debug)]
struct Issue {
  assignee: Option<String>,
  assignees: Vec<String>,
  author_association: String,
  body: String,
  closed_at: Option<String>,
  comments: i64,
  comments_url: String,
  created_at: String,
  events_url: String,
  html_url: String,
  id: i64,
  labels: Vec<String>,
  labels_url: String,
  locked: bool,
  milestone: Option<String>,
  node_id: String,
  number: i64,
  repository_url: String,
  state: String,
  title: String,
  updated_at: String,
  url: String,
  user: User,
}

#[derive(Serialize, Deserialize, Debug)]
struct Repository {
  archive_url: String,
  archived: bool,
  assignees_url: String,
  blobs_url: String,
  branches_url: String,
  clone_url: String,
  collaborators_url: String,
  comments_url: String,
  commits_url: String,
  compare_url: String,
  contents_url: String,
  contributors_url: String,
  created_at: String,
  default_branch: String,
  deployments_url: String,
  description: String,
  downloads_url: String,
  events_url: String,
  fork: bool,
  forks: i64,
  forks_count: i64,
  forks_url: String,
  full_name: String,
  git_commits_url: String,
  git_refs_url: String,
  git_tags_url: String,
  git_url: String,
  has_downloads: bool,
  has_issues: bool,
  has_pages: bool,
  has_projects: bool,
  has_wiki: bool,
  homepage: Option<String>,
  hooks_url: String,
  html_url: String,
  id: i64,
  issue_comment_url: String,
  issue_events_url: String,
  issues_url: String,
  keys_url: String,
  labels_url: String,
  language: String,
  languages_url: String,
  license: Option<String>,
  merges_url: String,
  milestones_url: String,
  mirror_url: Option<String>,
  name: String,
  node_id: String,
  notifications_url: String,
  open_issues: i64,
  open_issues_count: i64,
  owner: User,
  private: bool,
  pulls_url: String,
  pushed_at: String,
  releases_url: String,
  size: i64,
  ssh_url: String,
  stargazers_count: i64,
  stargazers_url: String,
  statuses_url: String,
  subscribers_url: String,
  subscription_url: String,
  svn_url: String,
  tags_url: String,
  teams_url: String,
  trees_url: String,
  updated_at: String,
  url: String,
  watchers: i64,
  watchers_count: i64,
}

#[derive(Serialize, Deserialize, Debug)]
struct IssueCommentEvent {
  action: String,
  comment: Comment,
  issue: Issue,
  repository: Repository,
  sender: User,
}

#[derive(Serialize, Deserialize, Debug)]
struct User {
  avatar_url: String,
  events_url: String,
  followers_url: String,
  following_url: String,
  gists_url: String,
  gravatar_id: String,
  html_url: String,
  id: i64,
  login: String,
  node_id: String,
  organizations_url: String,
  received_events_url: String,
  repos_url: String,
  site_admin: bool,
  starred_url: String,
  subscriptions_url: String,
  #[serde(rename = "type")]
  _type: String,
  url: String,
}

Most of this is just struct definitions for deserialization code. What we care about with these structs is that IssueCommentEvent represents the webhook for issue comments. As a side note if you’re unfamiliar with GitHub’s API under the hood every PR and Issue is an Issue. That’s why if, say, in a new repo you open an issue, the issue is numbered 1. If you then open a PR its number is 2. This will be important in a little bit!

Let’s look at the opening bit of code:

#[macro_use]
extern crate serde_derive;
extern crate serde;
extern crate serde_json;
extern crate reqwest;
extern crate http;

use std::{fs, env, process::exit, error::Error};
use reqwest::header;
use http::StatusCode;
fn main() {
    if let Err(e) = run() {
        eprintln!("{}", e);
        exit(1);
    }
}

This is probably the last bit of non Rust 2018 edition code I’ll write since I wanted to make sure this worked on stable and the newest version comes out this week which makes it the default. Here we’re importing some external crates. serde for our deserialization of JSON, reqwest to make simple http requests and http for all the types used amongst various http libs in Rust.

We make a few imports of types and modules with our use statement and then have our main function. We say that when we get a Result type from our run function, that if it was an error, print it out on stderr then exit with a non zero status code! Most of this code is standard boilerplate we would want to write to make sure we can handle errors and print out the result for our Rust programs, but I’m making no assumptions about who’s reading the article here.

Let’s take a look at the actual logic that does stuff now that we have gone through the setup and type code.

fn run() -> Result<(), Box<Error>> {
    let event = serde_json::from_str::<IssueCommentEvent>(
        &fs::read_to_string(env::var("GITHUB_EVENT_PATH")?)?
    )?;

    let mut headers = header::HeaderMap::new();
    headers.insert(header::ACCEPT, header::HeaderValue::from_static("application/vnd.github.v3+json"));
    let client = reqwest::Client::builder()
        .build()?;

    let aa = &*event.issue.author_association;
    if aa == "COLLABORATOR" || aa == "OWNER" || aa == "MEMBER" {
        let cb = &*event.comment.body;
        if cb.contains("r+") {
            let hurl = &*event.issue.html_url;
            if hurl.contains("pull") {
                let rurl = &*event.issue.repository_url;
                let num = event.issue.number;
                let result = client
                    .put(&format!("{}/pulls/{}/merge", rurl,num))
                    .bearer_auth(&env::var("GITHUB_TOKEN")?)
                    .send()?
                    .status();

                if result == StatusCode::OK {
                    println!("PR Merged!");
                } else {
                    eprintln!("Unable to merge PR!");
                    exit(1);
                }
            } else {
                eprintln!("Can't r+ on an issue only a PR");
            }
        } else if cb.contains("r-") {
        }
    }
    Ok(())
}

run has a function signature that says “I’ll do things, but in order to signal that it all went well return a unit type (()) but it if goes wrong send me back any type that implements the standard library’s Error trait”. This gives us flexibility with handling errors from various crates. Tf they did it right they’ll have implemented the trait!

Okay so next is actually reading the webhook file. GitHub stores this in an ENV var called GITHUB_EVENT_PATH. So we’re querying for it here. The ? is basically saying, “If the Result of this was an error return the error for this function, otherwise get me the value you got out of the Result.” We then use that to read the file from the disk and again use ? saying that, “If you can’t read throw an error, otherwise give me the contents of the file.” We then use from_str to deserialize it into the IssueCommentEvent type we made earlier. If that works then ? unwraps the value and stores it in event!

Next up we set up some default headers for any request we’ll make to the API in this program to create our Client which will make the request. We’re setting a header that tells the GitHub API that we only want to use v3 REST API and not the GraphQL version at all.

Next we get a reference to a field in the type called author_association and store it in aa this is mostly for a shorter variable name for the comparison here, but also so I don’t need to also stick & in front of each of the strings with " around them, thus the &*, but that’s a whole other blog post on it’s own. Suffice to say we’ll be doing this to grab values we need.

We then want to check if they’re an OWNER, COLLABORATOR, or MEMBER. We don’t just want someone who wouldn’t have commit access to merge things, though with MEMBER the granularity could have it that they do not have write access. We then check if the comment contained r+ in it. We then check the html_url (yes I know I shortened it to hurl that’s how I feel about my code sometimes) to see if it has pull in it. Remember when I was talking about how they’re all issues under the hood? Well we need to use the html_url to determine if it was a comment on a pull request or a comment on an issue in the issue tracker. issue_url does not make that distinction. If it is a common on a PR then we setup a request to do a merge on the PR with the token we added as a secret. If it works we print out it did, if it did not we print out it that it did not and exit with a non zero exit code. Future work will make an error type for this and send that up to the top level instead.

So what does this look like when it actually runs?

And from the log files:

It works! Hopefully, this gave you a bit of a primer on how to get it working!

The Good, The Bad, and The Ugly

Let’s start with the bad/ugly cause I like to keep negativity to a minimum and I overall really like actions. This is a beta so I only expect it to get better over time:

Slow startup times: It can take half a minute to startup a container. I think GitHub would benefit immensely from using something like firecracker
The build, push, run, debug cycle is long and painful. If there was some way to mock this locally to reduce that I would absolutely love it
No streaming logs: I can only see the logs after it runs
The Actions tab seems a bit wonky and feels like it could have 3 separate sub tabs. This one is kind of hard to describe unless you use it.
Docs are hit and miss. Overall they’re good, other packages/apis I’ve used have not had docs this good. However, I’d prefer schema definitions of JSON payloads with the examples. I had three things have null fields that did not show up in the example for the event type. That and I feel like Actions docs still have some ways to go. It’s not so bad that you can’t get anything done, but you’ll be skipping around trying to figure stuff out. For example, the GITHUB_TOKEN was made to seem like it’s part of the environment if you look at the storing secrets docs, but it’s not until you go to the docs on what environment variables are available that you find out that you have to add it to the action definition to get access to it. This is fine if you read the docs in order, but I did not so I ran into this issue. Having it be reaffirmed is good, albeit redundant.
Being able to test out workflows on feature branches so that I’m not pushing to master all the time would be dope. I’ll need to experiment if this is the case already, but if it is the docs do not mention it at all.
I’d love to see this be available to open source for free, but that costs money so I don’t really know if we’ll see that at all. Again it’s still limited to private repos for now.

That’s all I really had issue with and these can get better with time I believe. Let’s cover the good because Actions is overall impressive:

Arbitrary code execution in a serverless environment. Just typing those words gives me nightmares, but it’s impressive what GitHub has managed to pull off bringing this to their product
I don’t have to run my own infrastructure which is so nice
Visually being able to see how actions connect is neat
Being able to store secrets into a repo itself and access them from Actions is great, it’s got a GitOps vibe that I’m very here for
Whatever lang you want you can use, just bring a Dockerfile or container
Uses cached versions of your containers to reduce the need to rebuild, only rebuilds if it needs too
Sharing workflows is as easy as unzipping files into your directory. Would love to see this be as easy as pulling releases from like an Actions App Store to use in your repo
Arbitrary Code Execution. Seriously it’s so damn cool and you can now manage everything in one spot.

Color me impressed. I’m excited to see what more can be done with this.

Conclusion

We covered how to get started with GitHub Actions in a more advanced way, as well as my thoughts on the matter. I’d like to see this get rolled out to more people because it’s really cool. To those who already have it, happy hacking!

Plumbing the Deps of the Crate: Caching Rust Docker Builds

Michael Gattozzi — Tue, 23 Oct 2018 03:32:10 +0000

I’ve been using Docker for Rust code recently, but one of the problems is that using a Dockerfile like the following:

COPY . /opt/my_build_dir
RUN cargo build

Ends up causing the container to download the dependency crates. Every. Single. Time. It’s terrible. If you’ve ever had a 15 minute clean build for a Rust project you can see how much it sucks to just rebuild from scratch every time just to test out some changes.

How do we deal with this? What’s the solution? Let’s look at how Docker works with the COPY instruction and file system caching and then move to the solution I used to hack around the limitation of not being able to only build the dependencies of a crate.

COPY, Filesystems, and You

One of the things Docker does is that it caches the container at different layers. The commands FROM, COPY, RUN and CMD each create a different layer. This is good! That means if everything up to line 20 in my Dockerfile is the same, Docker can skip those build steps. How does COPY know it’s “invalid” and that the command needs to change again? It checks to see if the files referenced in the COPY command have changed. This is good! If I change a file then I would want the new file in the container so it can build properly.

Now see if you can spot the problem with the first snippet of a Dockerfile I shared above. Think you got it? If you’re not sure then think about this: if I use COPY . I’m saying that, “If anything at all in this directory changes, copy the data in”. You change the source files? A whole new copy of the directory is made as the command has to be rerun. This is also a new OS so even if you copy in a target directory Rust will still rebuild it, as the cached files in your target dir will no longer be valid for compilation.

This right here is why we’re downloading everything over and over and over again on changes. We haven’t created a layer that only builds the deps and caches it. What we really want is to only rebuild those deps when Cargo.toml/Cargo.lock change! How do we do it?

She sed, I know what it’s like to build deps

To be clear what we’re going to do is definitely a hack. It’s not pretty. It’s absolutely not elegant, but it works. We’re gonna use a small feature of Cargo.toml files, a bit of sed magic, and some clever uses of COPY to accomplish it. Okay let’s get started. First up you’ll need to open up the Dockerfile in your directory. I’m assuming you have cargo already in the container and are doing something that looks like this:

# The beginning stuff up to this point
COPY . /your_work_dir
RUN cargo build
# The rest of your file

The first thing we want to do is change our file to this instead:

# We'll get to what this file is below!
COPY dummy.rs /your_work_dir
# If this changed likely the Cargo.toml changed so lets trigger the
# recopying of it anyways
COPY Cargo.lock /your_work_dirCOPY Cargo.toml /your_work_dir
# We'll get to what this substitution is for but replace main.rs with
# lib.rs if this is a library
RUN sed -i 's/src/main.rs/dummy.rs/' Cargo.toml
# Drop release if you want debug builds. This step cache's our deps!
RUN cargo build --release
# Now return the file back to normal
RUN sed -i 's/dummy.rs/src/main.rs/' Cargo.toml
# Copy the rest of the files into the container
COPY . /your_work_dir
# Now this only builds our changes to things like src
RUN cargo build --release

Now in the top level of the directory create a file dummy.rs it should like this:

fn main() {}

That’s it! It’s a stub to get cargo to build the deps, but not build our actual code! If you’re building a library then use an empty file instead. Alright so the last bit of magic is to modify your Cargo.toml to contain this in it under the [package] section:

[[bin]]
name = "your_project_name"
path = "src/main.rs"

If you’re making a library then put:

[lib]
name = "your_project_name"
path = "src/lib.rs"

Remember those sed commands? They’ll actually change the line with path on it. It’ll change it to point at the dummy file, compile it (which it’s basically empty so not too much time), and all of the deps listed in the Cargo.toml file. Then it changes back to the actual entry point to build the correct project!

Like I said it’s an absolute hack but hey it works very well! Try it out and see how much you can speed your builds up by.

Closing thoughts

Ideally cargo would be smart enough to just do cargo build --dependencies-only (there is an issue open for this) and then having Docker setup to only rebuild changes to the src directory. For now though, the above is, while fragile, a good enough alternative to the issue. It would be great to see Rust’s excellent tooling to work in a way that it will meet production needs. I hope you’ll spend more time working on code and futzing with deployment configs now, rather than worrying about your build times!