Forem: Kushal Joshi

Cap'n Proto - RPC at the speed of Rust - Part 2 of 2

Kushal Joshi — Sun, 12 Jun 2022 15:20:54 +0000

This is Part 2 of a two part article on Cap'n Proto (capnp). I'd strongly suggest reading Part 1 so Part 2 makes sense.

The code is available here:

https://github.com/kushalj/capnproto-demo

Title photo by Anders Jildén on Unsplash

As a reminder, this is our example capnp schema:

@0xb068ff5fb1c4f77e;

using Rust = import "rust.capnp";
$Rust.parentModule("server");

struct Point {
    x @0 :Float32;
    y @1 :Float32;
}

interface PointTracker {
    addPoint @0 (p :Point) -> (totalPoints :UInt64);
}

Setting up the client and server

The objective here it so use the Cap'n Proto RPC features and not explore setting up a server so the Tokio code below is copied from the capnp-rpc examples and modified slightly towards simplification so we can explore the capnp-rpc features and see them in the context of the Tokio code.

Starting with the code at the end of Part 1, we have this as our current server.rs:

#![allow(dead_code)]

#[path = "./schema/point_capnp.rs"]
mod point_capnp;

pub mod point_demo {
    use crate::server::point_capnp::point;
    use capnp::serialize;
    use std::fs::File;

    pub fn write_to_stream() -> std::io::Result<()> {
        let mut message = ::capnp::message::Builder::new_default();

        let mut demo_point = message.init_root::<point::Builder>();

        demo_point.set_x(5_f32);
        demo_point.set_y(10_f32);

        // This Result should be consumed properly in an actual app
        let _ = serialize::write_message(&mut ::std::io::stdout(), &message);

        // Save the point
        {
            let file = File::create("point.txt")?;
            let _ = serialize::write_message(file, &message);
        }

        // Read the point from file
        {
            let point_file = File::open("point.txt")?;

            // We want this to panic in our demo incase there is an issue
            let point_reader =
                serialize::read_message(point_file, ::capnp::message::ReaderOptions::new())
                    .unwrap();

            let demo_point: point::Reader = point_reader.get_root().unwrap();
            println!("\n(x = {}, y = {})", demo_point.get_x(), demo_point.get_y());
        }

        Ok(())
    }
}

The code is suppressing Results with unwraps and is dirtier than a cloth seat on UK public transport but that's ok - we are just using this to gain intuition. When the time comes to use capnp in production, you will not code like this. Please don't code like this.

Moving on, Cap'n Proto (capnp) facilitates RPC via a tiny built-in concurrency event-loop, very similar to JavaScripts's called KJ. An odd discovery for me, as these are my initials - but the docs say they don't stand for anything and were chosen as they were easy to type. 😕 Why not give it some kind of useful name? Names are strongly related to learning and knowledge transition. This mildly irritating - but not really that important as, fortunately, the Rust version of capnp has the sensible name RPCSystem which makes the code a little more readable.

Within the capnproto-rust repo, there is a capnp-rpc folder with some example of how to use the RPC system. The Docs refer to something called EzyRPCServer and this no longer exists in the code but there are some examples of Tokio based async servers, which is a good thing as that is how we should build this anyway (with Tokio), rather than abstracting away some magical server code.

We can start by changing main.rs to be a simplified version of the capnp-rpc hello-world example:

#[tokio::main(flavor = "current_thread")]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let args: Vec<String> = ::std::env::args().collect();
    if args.len() >= 2 {
        match &args[1][..] {
            "client" => return client::main().await,
            "server" => return server::main().await,
            _ => ()
        }
    }

    println!("usage: {} [client | server] ADDRESS", args[0]);
    Ok(())
}

So now we need to adjust Cargo.toml:

...

[dependencies]
capnp = "0.14"
capnp-rpc = "0.14"
tokio = { version = "1.0.0", features = ["net", "rt", "macros"]}
tokio-util = { version = "0.6.0", features = ["compat"] }
futures = "0.3.0"

...

...and add a skeleton main() to server.rs:

use std::net::ToSocketAddrs;

...

pub async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let args: Vec<String> = ::std::env::args().collect();
    if args.len() != 3 {
        println!("usage: {} server ADDRESS:PORT", args[0]);
        return Ok(());
    }

    let addr = args[2]
        .to_socket_addrs()
        .unwrap()
        .next()
        .expect("could not parse address");

    tokio::task::LocalSet::new()
        .run_until(async move {
            let _listener = tokio::net::TcpListener::bind(&addr).await?;

            println!("Server running");
            loop {}
        })
        .await
}

(_listener has an underscore to remove warnings. We will remove this underscore shortly)

...and finally we need to add skeleton a client.rs:

#![allow(dead_code)]

use futures::AsyncReadExt;
use std::net::ToSocketAddrs;

#[path = "./schema/point_capnp.rs"]
pub mod point_capnp;

pub async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let args: Vec<String> = ::std::env::args().collect();
    if args.len() != 3 {
        println!("usage: {} client HOST:PORT", args[0]);
        return Ok(());
    }

    let addr = args[2]
        .to_socket_addrs()
        .unwrap()
        .next()
        .expect("could not parse address");

    tokio::task::LocalSet::new()
        .run_until(async move {
            let stream = tokio::net::TcpStream::connect(&addr).await?;

            println!("Connected to TCP Stream");

            stream.set_nodelay(true)?;
            let (_reader, _writer) =
                tokio_util::compat::TokioAsyncReadCompatExt::compat(stream).split();

            // RPC code

            Ok(())
        })
        .await
}

(_reader and _writer have underscores to remove warnings. We will remove these underscores shortly)

These are the basics (from the capnp example code) to create a network connection and pass a message (I've excluded capnp code so we can build that up gradually and you can get an idea of what is boiler-plate and what is specific to our capnp client/server example).

Let's test the client:

❯ cargo run client 0.0.0.0:8080
   Compiling capnproto-demo v0.1.0 (/Users/kushaljoshi/code/rust/capnproto/capnproto-demo)
    Finished dev [unoptimized + debuginfo] target(s) in 1.46s
     Running `target/debug/capnproto-demo client '0.0.0.0:8080'`
Error: Os { code: 61, kind: ConnectionRefused, message: "Connection refused" }

That Connection refused error makes sense as the Server is not running - and functions as an adequate failing test to let us know the client tried to connect. We can open another terminal and run the server from the same folder:

❯ cargo run server 0.0.0.0:8080
    Finished dev [unoptimized + debuginfo] target(s) in 0.09s
     Running `target/debug/capnproto-demo server '0.0.0.0:8080'`
Server running

...and now the client again:

❯ cargo run client 0.0.0.0:8080
    Finished dev [unoptimized + debuginfo] target(s) in 0.09s
     Running `target/debug/capnproto-demo client '0.0.0.0:8080'`
Connected to TCP Stream

That's better. This seems to indicate that the client is connecting to the server (there is no error on connection), so we can assume we have a basic server and a client that can connect to it.

Magical flying clients

At this point the capnp docs describe how to use the RPC framework in C++ only at this time, so I had to dive into the code sample and breakdown the intention of the code and rummage around to see what is happening just under the surface. I tried to refer back to the C++ docs but there are numerous differences in usage and implementation.

The capnp-rpc README has some information that helps and contains the following magical description: ""

Then you can convert your object into a capability client like this:

(capnp example)

let client: foo_capnp::bar::Client = capnp_rpc::new_client(MyBar {});

This new client can now be sent across the network. You can use it as the bootstrap capability when you construct an RpcSystem, and you can pass it in RPC method arguments and results.

Say what now?

So my server makes a client and sends the client to the... client?

Ok sure, I think I follow. Let's make the server-client thing then:

The generated code also includes a Server trait for each of your interfaces. To create an RPC-enabled object, you must implement that trait.

So for this schema (capnp example):

@0xa7ed6c5c8a98ca40;

interface Bar {
    baz @0 (x :Int32) -> (y :Int32);
}

interface Qux {
    quux @0 (bar :Bar) -> (y :Int32);
}

I need to build this (capnp example):

struct MyBar {}

impl ::foo_capnp::bar::Server for MyBar {
     fn baz(&mut self,
            params: ::foo_capnp::bar::BazParams,
            mut results: ::foo_capnp::bar::BazResults)
        -> Promise<(), ::capnp::Error>
     {
         // `pry!` is defined in capnp_rpc. It's analogous to `try!`.
         results.get().set_y(pry!(params.get()).get_x() + 1);

         Promise::ok(())
     }
}

That looks ok - the generated code seems to have types for the params and results of the baz interface method already. Let's start with that for our PointTracker interface. We will call it PointTrackerImpl rather than MyPoint (which is also the convention in the rest of the examples):

pub struct Point {
    x: f32,
    y: f32,
}

struct PointTrackerImpl {
    points: Vec<Point>,
}

impl point_tracker::Server for PointTrackerImpl {
    fn add_point(
        &mut self,
        params: point_tracker::AddPointParams,
        mut results: point_tracker::AddPointResults,
    ) -> Promise<(), ::capnp::Error> {
        let point_client = pry!(params.get()).get_p();

        if let Ok(received_point) = point_client {
            self.points.push(Point {
                x: received_point.get_x(),
                y: received_point.get_y(),
            });
        }
        results.get().set_total_points(self.points.len() as u64);

        Promise::ok(())
    }
}

(This code is in server.rs)

and now we can point at this point-tracker from our server code, using the boiler-plate from the hello-world example:

use capnp::capability::Promise;
use capnp_rpc::{pry, rpc_twoparty_capnp, twoparty, RpcSystem};

use futures::{AsyncReadExt, FutureExt};
use std::net::ToSocketAddrs;


// existing code...

...


pub async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let args: Vec<String> = ::std::env::args().collect();
    if args.len() != 3 {
        println!("usage: {} server ADDRESS:PORT", args[0]);
        return Ok(());
    }

    let addr = args[2]
        .to_socket_addrs()
        .unwrap()
        .next()
        .expect("could not parse address");

    tokio::task::LocalSet::new()
        .run_until(async move {
            let listener = tokio::net::TcpListener::bind(&addr).await?;

            // Cap'n Proto point_tracker client initialised here
            let point_tracker_client: point_tracker::Client =
                capnp_rpc::new_client(PointTrackerImpl { points: Vec::new() });

            println!("Server running");
            loop {
                let (stream, _) = listener.accept().await?;
                stream.set_nodelay(true)?;
                let (reader, writer) =
                    tokio_util::compat::TokioAsyncReadCompatExt::compat(stream).split();

                let network = twoparty::VatNetwork::new(
                    reader,
                    writer,
                    rpc_twoparty_capnp::Side::Server,
                    Default::default(),
                );

                let rpc_system =
                    RpcSystem::new(Box::new(network), Some(point_tracker_client.clone().client));

                tokio::task::spawn_local(Box::pin(rpc_system.map(|_| ())));
            }
        })
        .await
}

(This code is in server.rs)

And our server is nearly done!

Vat?

If that was a little bit like drawing an owl, I understand - It's ok, we're going to break that last bunch of code down a bit and explain what the Cap'n is doing.

The Promise we are using above is part of the RPC system and facilitates the async functionality within the RPC runtime. The actual response, as you will be able to figure out eventually after some head-scratching, is the results.get() function and the other code it contains.

So in the add_point() point_tracker::Server trait implementation of PointerTrackerImpl we are doing the following:

Get the point-client from the request which should give us a Result
If the Result is ok, grab the x & y coordinates for the point directly from the message
Save the total number of points as a new message
Send the message as the response

This part is a little fiddly and I had to fart around with the available helper code and examples to shape this. I'm not 100% happy with it but it will do as an example for our purposes here.

So this is our "Server" but it needs to be Served somehow. In the main() function of server.rs we set up something a little odd now. A two-party-vat network.

To explain this I will first share some information from the capnp docs site. Half way through a list of features on the RPC Protocol page there is a vague reference:

Three-way interactions. A network of Cap’n Proto vats (nodes) can pass object references to each other and automatically form direct connections as needed.

In case you missed it, this is the (only?) reference to the 'Vat' word we can see in the code. Apparently a Vat is a node. Cool. Not sure why they didn't call a node a.. "Node". 🤷 But ok. Now we know.

So this magical code...

 let network = twoparty::VatNetwork::new(
                    reader,
                    writer,
                    rpc_twoparty_capnp::Side::Server,
                    Default::default(),
                );

...takes an input stream & output stream (the TCPListener stream that we wrapped with Tokio (out of scope of this article - please go explore Tokio articles) a Side (Server or Client) and some internal options. It's easier to see this with the Types annotated by Rust Analyzer:

It returns a capnp RPC vat (node) network.

Then we set up the RPC system:

let rpc_system =
   RpcSystem::new(Box::new(network), Some(point_tracker_client.clone().client));

A lot going on there, Let's get the Rust Analyzer view again:

Ok we have a new RPC System that contains a pointer (Box) to the VatNetwork we made above. It takes a "bootstrap" param which is our server-client thing we made earlier. And we may need to cycle around the loop again if the spawn'd task ends, so we clone it.

Finally we spawn the task and can call map() on the RPC System to start it. Another mild oddity but I can live with it. I guess it make sense if I want to process the return value of the RPC System for some reason; then this nests it neatly.

Make the RPC Client

The client code is similarly built except there is no loop. I left a comment in the code above that says //RPC code and left a space for the capnp-rpc code. This code looks as follows:

            let rpc_network = Box::new(twoparty::VatNetwork::new(
                reader,
                writer,
                rpc_twoparty_capnp::Side::Client,
                Default::default(),
            ));
            let mut rpc_system = RpcSystem::new(rpc_network, None);
            let point_tracker: point_tracker::Client =
                rpc_system.bootstrap(rpc_twoparty_capnp::Side::Server);

            tokio::task::spawn_local(Box::pin(rpc_system.map(|_| ())));

            let mut request = point_tracker.add_point_request();

            // let's make a Point:
            let mut message = ::capnp::message::Builder::new_default();
            let mut new_point = message.init_root::<point::Builder>();
            new_point.set_x(5_f32);
            new_point.set_y(10_f32);

            request.get().set_p(new_point.into_reader())?;

            let reply = request.send().promise.await.unwrap();

            println!(
                "Total points in Point Tracker: {}",
                reply.get().unwrap().get_total_points()
            );

So again we are making a two-party-vat-network (node network) but this time we are using the capnp config Side enum of Client. We then boot the RPC System with the network and a None because we don't have server code on this side to attach.

In fact this is a mental hook for understanding what's happening under the surface: The None is replaced by a capability on the server side (our point-tracker server-client thingy, which is a capability::Client) - but because we are currently building the client side of the RPC, there is no capability to bootstrap the RPC system with. We just want to connect and call a remote procedure - but before that we need the interface from the other side.

The next line confusingly also says "bootstrap" (same name as the RpcSystem::new() param). Yes, there are two things called Client and two things called bootstrap. Serious name-grokking challenge in this project but it's still evolving so I will appreciate what it does and not how it looks...

Anyway, back to rpc_system.bootstrap(). This one is interesting - it's the opposite of the other bootstrap (which is a weak argument for the function having the same name as a param...). This function sends a vat (node) ID and responds with the "bootstrap interface" of the other side! This is why the result is attached to the point_tracker variable.

You might notice that we have to explicitly state the type as point_tracker::Client. This does actually make sense. If you remove the explicit type, this happens, two lines down:

The compiler has no idea if point_tracker has an add_point_request() function and what it might return - because this is just code. Nothing has been sent yet!

Let's send some points!

Once the RPC System is up and running (with the .map() as before), we can send our message.

To make the request we use the add_point_request() function which is provided in the generated code.

Unfortunately a lot of the examples use on-the-fly creation of the capnp message they are sending. This is not practical if the message needs to be created by some other code in our application. So in the example, I'm manually creating a message and initialising it with point::Builder, as we did in Part 1, and then setting this as the message in the request.

Finally, we send the message and use the RPC promise system as our async approach to await the response. And that's it!

If we run the server in one terminal window, we get the confirmation message that it is running:

And if we run the client from another terminal window, we can start filling up the Point Tracker with points!

You can keep running the client and we can see the total points increasing as the server stores the points we are sending.

Final Words

This write up took a little longer than I planned because, to be frank, the Cap'n Proto documentation (for Rust at least) is a bit pants at the moment (a UK colloquialism for "it needs some work"). However, I really like this project and really, I've just touched the surface of what it's capable of in this small tour. With some better documentation and some evolution around its RPC features, I think this is going to be one to watch. It feels oddly cool to have the ability to extract data directly from network messages. I'm going to integrate capnp into a project I'm working on and I may do a second write-up at that point.

There should be a "so what" at the end I guess. There is: From an abstract perspective, network data requires conversion to a serialised stream of bytes that can be sent via some protocol that can reconstruct (deserialise) that data back into something a network node can consume for its data processing purposes. As we progress down the road of gigantic user data (my phone has half a TB of memory, don't tell me we're not going there - and Genshin Impact does a multi-gigabyte update every time you load it) combined with an increasing need to provide information immediately, there is one step in the process we will be able to remove completely with Cap'n Proto - the deconstruction/reconstruction steps. Similarly to Rust's energy saving side effect, this probably makes Cap'n Proto the most energy efficient protocol out there. I can imagine the Cap'n is going to be a big deal just a little further down that road.

Cap'n Proto - RPC at the speed of Rust - Part 1 of 2

Kushal Joshi — Sun, 15 May 2022 13:29:48 +0000

We're entering into a kind of niche subject here but I want to make it more accessible if I can. At the time of writing this post, at my work we've been using GRPC for a few years, because at the time when we made the decision, it was simple, fast, had some Rust support, and corporate sponsorship (it used to be that G stood for Google...). It's not that we've reached the end of using GRPC but I keep wondering what else is possible, as our work context requires us to go faster and faster.

Title photo by Emre Karataş on Unsplash

GRPC is a binary RPC protocol that serialises pretty damn quick and transfers data more than fast enough across our global system. We could use something faster but, to date, we've had other fish to fry in the area of optimisation.

However, today I have some personal projects that need a data protocol and my leaning is to go with RPC. I need speed and I want it to be well supported and simple to integrate.

While it would be easy to use GRPC again, I want to try out another RPC protocol - Cap'n Proto.

Why? because the author of Cap'n Proto was one of the original authors of ProtoBuf 2 (Protocol Buffers 2) which is the open source serialisation format used in GRPC. Also because Cap'n Proto claims to have no serialisation/deserialisation at all once a message is created, which means it should be very fast for transferring data around a distributed system. This includes consideration for saving that same data structure that doesn't need serialisation. It's apparently all handled by the protocol definition, all the way down to the endian consideration for saved data.

The Cap'n Proto site is both funny and full of information as to why it was created - but I can't immediately find a way to get an example up and running or to understand how to convert my app's types to Cap'n Proto types. I think everything I need should be there as I can see there is a section on Encoding which should explain this.

The only hurdle I have is that while the documentation is extensive it is a little confusing in places and mainly focuses on C++ and the C++ RPC system which is a little different to the Rust code. There are Rust examples in the official repo which I will try and leverage here.

Installing Cap'n Proto

There is a note on their site that Homebrew can be used to install on a Mac. But at the time of writing I couldn't figure out what to install.

After some hunting I found that we need the relevant tool to process the Cap'n Proto (capnp) Schema files: https://capnproto.org/capnp-tool.html

I found this can be installed on a Mac with:

brew install capnp

If you don't have Homebrew for you Mac, go here: https://brew.sh/

If you don't have a Mac, there are installation instructions here: https://capnproto.org/install.html

once installed we can make sure it runs and look at the help:

capnp --help

Output

Usage: capnp [<option>...] <command> [<arg>...]

Command-line tool for Cap'n Proto development and debugging.

Commands:
  compile  Generate source code from schema files.
  convert  Convert messages between binary, text, JSON, etc.
  decode   DEPRECATED (use `convert`)
  encode   DEPRECATED (use `convert`)
  eval     Evaluate a const from a schema file.
  id       Generate a new unique ID.

See 'capnp help <command>' for more information on a specific command.

Options:
    -I<dir>, --import-path=<dir>
        Add <dir> to the list of directories searched for non-relative imports
        (ones that start with a '/').
    --no-standard-import
        Do not add any default import paths; use only those specified by -I.
        Otherwise, typically /usr/include and /usr/local/include are added by
        default.
    --verbose
        Log informational messages to stderr; useful for debugging.
    --version
        Print version information and exit.
    --help
        Display this help text and exit.

Ok it's working. What do I do now?

I guess we can start with a message example.

The docs say:

Cap’n Proto messages are strongly-typed and not self-describing. You must define your message structure in a special language, then invoke the Cap’n Proto compiler.

Ok let's have a look at the compiler tool docs.

It says I can do this:

capnp compile -oc++ myschema.capnp

This is fine but I want Rust, not C++ code which this command seems to generate. Looking around, there is a bunch Rust crates that I think will help, plus an examples folder, all in this repo:

https://github.com/capnproto/capnproto-rust

But the example contains an ID in the schema file, so I'm not sure if I need to generate this or it is generated by the tool and... inserted into the schema?

Some more hunting around and text searching for "generate" brought me to the language page where I found this:

So it looks like I need to generate at least 1 id and put it in my schema.

❯ capnp id
@0xb068ff5fb1c4f77e;

And let's use the example from the capnproto-rust repo, but with our ID:

I will call this file src/schema/point.capnp

@0xb068ff5fb1c4f77e;

struct Point {
    x @0 :Float32;
    y @1 :Float32;
}

interface PointTracker {
    addPoint @0 (p :Point) -> (totalPoints :UInt64);
}

What does this describe? It looks like an RPC call to add a Point (with x & y coords defined as f32's) to something like a list of points, and it returns the totalPoints, which is a u64. As this type is not a collection I will assume it means the total-number-of-points.

Quick review of the schema basics:

Capnp comments use a "#"
The capnp types are:
- Void: Void
- Boolean: Bool
- Integers: Int8, Int16, Int32, Int64
- Unsigned integers: UInt8, UInt16, UInt32, UInt64
- Floating-point: Float32, Float64
- Blobs: Text (UTF8 NUL terminated), Data
- Lists: List(T) - the T is a Capnp built-in or defined capnp Schema Struct
Struct fields are consecutively numbered (like protobuf) - but with an "@"
There are Enums but also Unions.
Interfaces wrap methods (the PointTracker interface above contains addPoint method)
".capnp" files can import other ".capnp" files
Types for a field are declared with a :colon

The plan

As a rough plan, I want to be able to serve this interface and use or save the file in some way as a demo of the capnp capabilities. The challenge will be to make it as simple as possible so it facilitates what is an exploratory reference (for me at least) and hopefully some info/learning for anyone else looking at this protocol or learning/exploring Rust.

I've now made a cargo new project folder and added a src/schema folder for the file above.

In case generating a capnp ID sounds like a pain - the vscode-capnp extension for vs-code can generate a capnp ID anytime you need it.

(In fact I accidentally found out later that if you forget, the compiler throws an error and generates the ID for you so you can just copy and paste it in)

Generating a Cap'n Proto Schema

Let's see what the cli tool says about compiling now:

❯ capnp help compile
Usage: capnp compile [<option>...] <source>...

Compiles Cap'n Proto schema files and generates corresponding source code in one
or more languages.

Options:
    -I<dir>, --import-path=<dir>
        Add <dir> to the list of directories searched for non-relative imports
        (ones that start with a '/').
    --no-standard-import
        Do not add any default import paths; use only those specified by -I.
        Otherwise, typically /usr/include and /usr/local/include are added by
        default.
    -o<lang>[:<dir>], --output=<lang>[:<dir>]
        Generate source code for language <lang> in directory <dir> (default:
        current directory).  <lang> actually specifies a plugin to use.  If
        <lang> is a simple word, the compiler searches for a plugin called
        'capnpc-<lang>' in $PATH.  If <lang> is a file path containing slashes,
        it is interpreted as the exact plugin executable file name, and $PATH is
        not searched.  If <lang> is '-', the compiler dumps the request to
        standard output.
    --src-prefix=<prefix>
        If a file specified for compilation starts with <prefix>, remove the
        prefix for the purpose of deciding the names of output files.  For
        example, the following command:
            capnp compile --src-prefix=foo/bar -oc++:corge foo/bar/baz/qux.capnp
        would generate the files corge/baz/qux.capnp.{h,c++}.
    --verbose
        Log informational messages to stderr; useful for debugging.
    --version
        Print version information and exit.
    --help
        Display this help text and exit.

Aha:

the compiler searches for a plugin called 'capnpc-' in $PATH...

Not sure if I have that. Let's see what the autocomplete finds:

❯ capnpc
capnpc        capnpc-c++    capnpc-capnp

Nope. Ok let's install capnpc-rust:

I couldn't find anything about needing to install this. Maybe it's magical and I can just select Rust as the language:

❯ capnp compile -orust src/schema/point-schema.capnp
rust: no such plugin (executable should be 'capnpc-rust')
rust: plugin failed: exit code 1

Yup, it's not magical.

Hmm... maybe it's a Cargo crate?

❯ cargo install capnpnc-rust
    Updating crates.io index
error: could not find `capnpnc-rust` in registry `crates-io` with version `*`

Nope.

Ok maybe I'm going about this the wrong way. I guess I could compile the capnpc-rust to a binary by cloning the repo but that may be unnecessary as what I really want is to compile it from within my own code. Isn't it? 🤷 - This is just a guess from reading the capnproto-rust repo:

It's also strongly hinted at in the capnproto-rust docs:

We can try...

crate::Cargo.toml:

[package]
name = "capnproto-demo"
version = "0.1.0"
edition = "2021"
build = "build.rs"

[dependencies]

[build-dependencies]
capnpc = "0.14"

crate::build.rs:

fn main() {
    capnpc::CompilerCommand::new()
        .src_prefix("src/schema")
        .file("src/schema/point.capnp")
        .run()
        .expect("schema compiler command failed");
}

And it compiles and runs the build cargo build! But it doesn't do anything. 😞 Or maybe it did and there's a schema somewhere on my drive?

It's probably this missing Env-var from the examples:

...but I think I want to specify the output folder myself:

fn main() {
    capnpc::CompilerCommand::new()
        .src_prefix("src/schema")
        .file("src/schema/point.capnp")
        .output_path("src/schema")
        .run()
        .expect("schema compiler command failed");
}

Ok! Now we have a generated schema file that is around 500 lines of code:

I'm going to cargo build again to see what happens when the schema already exists:

❯ ll src/schema 
total 56
-rw-r--r--  1 kushaljoshi  staff   159B 30 Apr 15:58 point.capnp
-rw-r--r--  1 kushaljoshi  staff    20K 30 Apr 17:54 point_capnp.rs

❯ cargo build
    Finished dev [unoptimized + debuginfo] target(s) in 0.04s

❯ ll src/schema
total 56
-rw-r--r--  1 kushaljoshi  staff   159B 30 Apr 15:58 point.capnp
-rw-r--r--  1 kushaljoshi  staff    20K 30 Apr 17:54 point_capnp.rs

Nothing (I ran the second cargo build at 18:00)! this looks good so far. I don't want to be pointlessly regenerating the schema on every build.

Right, now we have a schema and automatically generated code in our build. That's quite nice. Now how do we use it?

Using the generated code

In the generated code there's pub mod point module wrapper so this seems like a good places to start. Let's use that module in our project:

We'll keep it nice and simple. First we can make a server module that will be the capnp server.

Cargo.toml:

...

[dependencies]
capnp = "0.14"

...

main.rs:

mod server;

fn main() {
    println!("Hello, world!");
}

I've left the default new project code for now as a sign-post for very new people to see what is happening and how we are building up the project.

server.rs:

#[path = "./schema/point_capnp.rs"]
mod point_capnp;

use point_capnp::{point, point_tracker};

I'm guessing we need to tell the compiler where the code is.

There's a small issue when we try to build this. The generated code expects the point_capnp mod to be at the top level and doesn't like it being declared inside server:: :

That's a little annoying. The generated code is hard coded to crate::point_np.

I had a read of the issues for a few hours and found this has been addressed, albeit in what feels like a hacky way, and was raised/found as an issue in an old blog article from Hoverbear, which helped immensely here (thanks Ana!).

The simple answer for us right now (if there is a better/simpler solution, please comment) is to add this file - rust.capnp to the schema folder and include it in each schema like this:

point.capnp:

@0xb068ff5fb1c4f77e;

using Rust = import "rust.capnp";
$Rust.parentModule("server");

struct Point {
    x @0 :Float32;
    y @1 :Float32;
}

interface PointTracker {
    addPoint @0 (p :Point) -> (totalPoints :UInt64);
}

This is an irritant as it's a manual change to every schema file but it works great and compiled fine with tons of "associated function is not used" warnings for the generated code. Adding #![allow(dead_code)] at the top of the server.rs file fixed this for now. This is a pattern that works for now but probably won't scale - I'll let my server module "own" the capnp generate code for each schema that server is a host for.

I'm making a first commit to the repo at this point as I have a compiling capnp schema 🎉.

Getting to the Point

At this stage we are almost at the end of most of the available documentation regarding Rust but the capnproto-rust repo contains both serialisation and RPC examples. Deconstructing those, I'm hoping to make the simplest implementation I can here.

Let's make a point from our Point. The docs say:

In Rust, the generated code for the example above includes a point::Reader<'a> struct with get_x() and get_y() methods, and a point::Builder<'a> struct with set_x() and set_y() methods.

To understand how to use these, we have to jump back to the beginning of the documentation to understand how capnp works:

Cap’n Proto generates classes with accessor methods that you use to traverse the message.

Ok so we need to make a message that will contain our Point. I think.

In the address book example capnp::serialized_packed is used to read and write this message to a stream. Docs for this are here.

We can copy this address book code structure to make our Point.

server.rs:

#![allow(dead_code)]

#[path = "./schema/point_capnp.rs"]
mod point_capnp;

pub mod point_demo {
    use crate::server::point_capnp::point;
    use capnp::serialize_packed;

    pub fn write_to_stream() -> ::capnp::Result<()> {
        let mut message = ::capnp::message::Builder::new_default();

        let mut demo_point = message.init_root::<point::Builder>();

        demo_point.set_x(5_f32);
        demo_point.set_y(10_f32);

        serialize_packed::write_message(&mut ::std::io::stdout(), &message)
    }
}

main.rs:

mod server;

fn main() {
    let _ = server::point_demo::write_to_stream();
}

Output:

❯ cargo run
   Compiling capnproto-demo v0.1.0 (/Users/kushaljoshi/code/rust/capnproto/capnproto-demo)
    Finished dev [unoptimized + debuginfo] target(s) in 1.13s
     Running `target/debug/capnproto-demo`
 ̠@ A%

Fantastic! We "serialized" our Point and packed it into a capnp message. The message is not readable (it's the underscore, at-symbol, space, capital-A, percent-symbol) because it is capnp's binary type that does not need further serialization/deserialization over a stream to be used with an application. Can we check it?

Yes! The capnp tool provides a decode feature that needs the schema and the data structure:

❯ cargo run | capnp decode ./src/schema/point.capnp Point
    Finished dev [unoptimized + debuginfo] target(s) in 0.04s
     Running `target/debug/capnproto-demo`
capnp decode: The input is not in "binary" format. It looks like it is in "packed" format. Try that instead.
Try 'capnp decode --help' for more information.

Ok so this didn't work because we need to either tell capnp that it's a packed (compressed) message, or we need to print the raw message to STDOUT. Let's do both to increase our intuition of what is happening here. First we just need to add --packed to the CLI command:

❯ cargo run | capnp decode ./src/schema/point.capnp Point --packed
    Finished dev [unoptimized + debuginfo] target(s) in 0.04s
     Running `target/debug/capnproto-demo`
(x = 5, y = 10)

Now we can see that capnp can unpack (decompress) the message and print out the Point coords that we set. But we may not always have packed data so let's send the Point in its raw message format and make sure we can decode it as we would expect. We need to make a change to server for that:

server.rs

...

pub mod point_demo {
    use crate::server::point_capnp::point;
    use capnp::serialize;

    pub fn write_to_stream() -> ::capnp::Result<()> {
        let mut message = ::capnp::message::Builder::new_default();

        let mut demo_point = message.init_root::<point::Builder>();

        demo_point.set_x(5_f32);
        demo_point.set_y(10_f32);

        serialize::write_message(&mut ::std::io::stdout(), &message)
    }
}

Output:

❯ cargo run | capnp decode ./src/schema/point.capnp Point --packed
   Compiling capnproto-demo v0.1.0 (/Users/kushaljoshi/code/rust/capnproto/capnproto-demo)
    Finished dev [unoptimized + debuginfo] target(s) in 0.65s
     Running `target/debug/capnproto-demo`
capnp decode: The input is not in "packed" format. It looks like it is in "binary" format. Try that instead.
Try 'capnp decode --help' for more information.

A very helpful message that confirms what we know we did. We can remove the --packed flag now.

Output:

❯ cargo run | capnp decode ./src/schema/point.capnp Point
    Finished dev [unoptimized + debuginfo] target(s) in 0.05s
     Running `target/debug/capnproto-demo`
(x = 5, y = 10)

Fabulous.

If you have followed along and got this working, you may want to see the benefit more clearly so for that we can save the data and load it back in without any further serialization/deserialization.

server.rs:

pub mod point_demo {
    use crate::server::point_capnp::point;
    use capnp::serialize;
    use std::fs::File;

    pub fn write_to_stream() -> std::io::Result<()> {
        let mut message = ::capnp::message::Builder::new_default();

        let mut demo_point = message.init_root::<point::Builder>();

        demo_point.set_x(5_f32);
        demo_point.set_y(10_f32);

        // This Result should be consumed properly in an actual app
        let _ = serialize::write_message(&mut ::std::io::stdout(), &message);

        // Save the point
        {
            let file = File::create("point.txt")?;
            let _ = serialize::write_message(file, &message);
        }

        // Read the point from file
        {
            let point_file = File::open("point.txt")?;

            // We want this to panic in our demo incase there is an issue
            let point_reader =
                serialize::read_message(point_file, ::capnp::message::ReaderOptions::new())
                    .unwrap();

            let demo_point: point::Reader = point_reader.get_root().unwrap();
            println!("\n(x = {}, y = {})", demo_point.get_x(), demo_point.get_y());
        }

        Ok(())
    }
}

Output:

❯ cargo run
   Compiling capnproto-demo v0.1.0 (/Users/kushaljoshi/code/rust/capnproto/capnproto-demo)
    Finished dev [unoptimized + debuginfo] target(s) in 0.63s
     Running `target/debug/capnproto-demo`
@ A
(x = 5, y = 10)

So... it doesn't look like much happened there and the output, by design, looks the same.

However, you may have missed what just happened and how awesome this 😄 !!

Let's go through it:

We created a serialized Point from our Point Schema
We set data inside the serialized Point (no need to deserialize Point or serialize x & y float 32 values)
We saved the serialized data to disk using the standard file tools
We read in the serialized data using the standard file tooling (endianness is considered in the filetype)
We used accessor methods on the serialized data and printed the value without deserializing the data.

It's ok - if you are thinking "so what?", then your project use cases may not have been performance critical so far. If they have though, then this should be a wondrous thing to behold!

For the uber skeptical: The Reader above is not a deserializer. It is literally a Reader. It needs a schema and some data and it know how to set the pointers in the data (which is made up of ordered segments) to make the accessor methods point at the correct parts of the data. For more information have a read of the capnp encoding page.

You can decode the file data the same way as the STDOUT output above:

cat point.txt | capnp decode ./src/schema/point.capnp Point
(x = 5, y = 10)

Now this really quite interesting; if it can be saved, it can be thrown over a network and used by any client that has the appropriate map (schema) to read the received data, without any interim deserialization steps.

That's what we will try next in Part 2.

18 factors powering the Rust revolution, Part 3 of 3

Kushal Joshi — Mon, 11 Apr 2022 21:59:06 +0000

Cover Photo by Markus Spiske on Unsplash

Disclaimer: I work at Keyrock but views expressed here are my own. However, if you like the views, do go checkout the company as it is full of awesome people and I have learned much whilst working there, including Rust.

If you haven't read Parts 1 & 2, I highly recommend you start with those:

Part 1

Syntax
Separation of Data and Code
Type safety
Declarative Programming and Functional Composition features

Part 2

Evolved package management
Concurrency ready std lib and great async support
Clear, helpful compiler errors
Zero cost abstractions (Optimisation)
Energy efficiency
WASM
Tauri - High performance, secure, cross-platform desktop computing

Part 3

Macros!
Corporate Engagement
Cryptocurrency and Blockchains
Everybody else's tooling
Speed
No garbage collection
Memory safety features

Macros!

Some people may be scarred and jaded from out of control macro usage in certain languages (especially low-level languages). In the world of Ruby people would decry "Rails" developers and later in the Elixir universe, people were encouraged to start without Phoenix, that giant macro-driven-code-generating beauty, for fear that they would never really explore the language properly and become "Phoenix" developers; I understand why - but in these examples - like with Elixir metaprogramming (using macros to generate and change code) - Macros were generally used for good, and supercharged the language the right way. Usually.

Macros in Rust are generally very sensible, and easy to identify - either with their exclamation mark identifiers for declarative style macros (like println!() in the std lib), or used with derive or as custom function attributes, which usually generate code from your code.

Generally the macro system is a platform for extended functionality and simplifying the interface to libraries. This hints at the potential growth still to come.

So how is this a major factor? I believe Macros are very similar to a custom type system. If the type is heavily documented and openly shared, it becomes part of the language and general knowledge of the system, known by all that claim to be its adherents. Where macros are not maintained, not tested well, not documented well, and are part of a closed system, they can become a knowledge-barrier to those attempting to change/fix/test that system's code. - hence the resistance many people have.

The Rust community has, through common practice, set some pretty heavy expectations for people producing macros that become part of the general knowledge of the language: As an example, the Serde library, known to almost anyone that will use Rust for web service development, is a heavily documented set of macros that allow for very granular transformation when serialising and derserialising data. There is even a Serde "book". Approached in this disciplined way, Rust Macros are going to provide an explosion of metaprogramming in the years to come, that will amplify the existing Rust growth.

More info:

https://serde.rs/

https://happens.lol/posts/a-love-letter-to-rust-macros/

https://jstrong.dev/posts/2020/productive-rust-implementing-traits-with-macros/

Corporate Engagement

We're way past the initial gradual incline slope of Rust growth. At this point, we're approaching hypergrowth. Try and find a company interested in performance systems programming that's not looking at Rust or starting to teach it themselves. With Microsoft and AWS sponsoring major rust packages, it feels very much like all the ducks are already neatly lined up and ready for take-off.

Let's take a tiny peek at how this is happening and what this corporate engagement looks like:

“We’re not just hiring a couple of folks for Tokio and the Rust compiler,” Shane Miller says. “Rust is a critical component of our long-term strategy, and we’re investing to deliver Rust engineering at Amazon scale. That includes developer tools, infrastructure components, interoperability, and verification.”

— AWS Open Source Blog, November 2020

News headline:

"AWS's Shane Miller to head the newly created Rust Foundation"

— ZDNet, April 2021

The Rust Foundation Community Grants Program is made possible by the generous support of corporate partners, with Amazon Web Services (AWS), Google, and Huawei amongst the first contributors...

“A large part of the current success of the Rust Project is due to the vibrant community and volunteers who contribute not only to engineering tasks, but to all aspects of making the Project successful,” said Lars Bergstrom, Google Director of Engineering. “We believe that one way that the Rust Foundation can best support the Rust Project is through providing Community Grants to those critical volunteers.”

— Businesswire, December 2021

News headline:

Google's Android Team is backing an effort to introduce Rust as a second programming language in the Linux kernel.

— ZDNet, April 2021

News headline:

In Rust we trust: Shoring up Apache, ISRG ditches C, turns to wunderkind lang for new TLS crypto module
Subheading: Elder server httpd to be revitalized with Google-funded memory-safe add-on

— The Register, Feb 2021

News headline:

Fuchsia OS uses Rust to write code extensively
"According to the overall code analysis of Fuchsia, Rust has the most lines of code."

— Infotech News, meterpreter, May 2021

News headline:

"Announcing three new Rust Foundation Board Members"

Article detail:

The Rust Foundation is delighted to announce that three new individuals have joined the Board of Directors.

The members:

– Ryan Levick, Developer Advocate, Microsoft
– Andrew Wafaa, Senior Director of Software Communities, Arm
– Eric Garcia, Software Engineering Manager, Meta

— Rust Foundation news blog, March 2022

This is really a tiny proportion of the Rust news out there. Additionally all of the tech giants are launching their own Rust onboarding and training material. As an idea, introducing Rust at a giant enterprise might sound experimental but when everyone's doing it, well, then it's at worst, a fashionable trend; at best, corporates jumping onboard to position themselves around a technology that constitutes part of the core competitive advantage of tomorrows solutions.

More info:

https://docs.microsoft.com/en-us/learn/paths/rust-first-steps/

https://developer.ibm.com/articles/os-using-rust/

Cryptocurrency and Blockchains

Assuming you were not trapped in a CIA black-site for the last 5 years, you've probably seen another unstoppable tidal wave, besides Rust; call it what you want: Web 3.0, The blockchain revolution, The rise of DEFI, NewFI, GameFI, The New Fiat... Cryptocurrency has arrived and it's here to stay (note: I might be slightly biased - see my history in Part 1). And what else has come along with this new industry? A demand for performance: Performance transaction/block processing, performance historical data processing, and performance contract execution - all on various distributed platforms, each claiming superiority over the others, all burning millions in energy (see 'Energy efficiency' in Part 2), and each not wanting live code to fail (see 'Safety features', below) - all so some other excited people can buy $2m (two million dollar) cartoon apes, or a piece of virtual land etc etc. If you were starting one of these projects today, what technology might you decide to bet on?

Here's hint - checkout Substrate, a blockchain framework, written in Rust.

Some Blockchain projects using Rust:

Article title:

"Solana Uses Rust to Pull in Developers and Avoid Copypasta"

The New Stack, Jan 2022

News Headline:

"Concordium Launches DevX Initiative to Support Rust Devs"

CoinQuora, April 2021

And when the many-named, many-faced hydra of social data wanted to make a cryptocurrency (Libra/Diem), they also went full Rust, back in 2019.

Everybody else's tooling

There is a naive argument that can be made at some point: "So why doesn't everyone just use Rust?" - well, there are lots of reasons: Legacy code (Cobol, C++ that no-one wants to touch but they want to extend - see all Banks), availability of developers, capability of engineering managers (😇), context and requirements (do we need quick code, or need to code quick?), suitability of application to the context (Some front-end frameworks are very mature - why fix what is not borked?), availability of specialist libraries (e.g. Python/Julia), architecture priority (e.g. we want 100% functional, with Actors), addiction to pain (Haskell, Java, C++), etc, etc.

However, The tooling around other languages does not need to be limited by the limits of those languages. An outcome of energy efficiency requirements, speed, and concurrency features, for example, might lead to someone writing the compiler/interpreter for another language in Rust 🙂 . More interestingly, once it is possible to transpile a language into Rust, it can be recompiled into WASM and run in the browser!

This means the Rust will likely form the core of other languages' tooling as we enter the next decade.

Examples:
A Node.js interpreter written in Rust: https://deno.land/

A web tool pipeline written in Rust: https://swc.rs/

The fastest dataframe library with Python links, written in Rust: https://pola-rs.github.io/polars-book/user-guide/index.html

A Clojure interpreter written in Rust: https://github.com/clojure-rs/ClojureRS

A Python interpreter written in Rust: https://rustpython.github.io/

The Erlang BEAM (VM) written in Rust: https://getlumen.org/

Ok, now we get to the "big three". Often when I am out and about and professionally participating in the local requirement to support the testing of grain, potato, and fruit fermentation operations, I'm often asked "Why Rust?". As the previous 15 factors in these articles would involve a rather extended evening of technical discussions, which can be challenging considering my previously mentioned professional activities, and not to mention immensely boring for non-developers, I usually default to the following three factors.

Speed

It may be obvious that low level compiled languages are faster; but it may not be obvious that they are faster - I mean hundreds and sometimes thousands of times faster than interpreted languages (I didn't realise that this was not common knowledge until speaking to a JS dev recently who said "Thousands? Are you sure?"). Rust speed is generally on par with C++ and in some cases faster - usually when C++ is not "cheating" with what Rust would call unsafe code. There are rules in the world of Rust this is going to add a few nanoseconds of overhead. For most application contexts, the difference will be a moot point.

More interesting is the meeting point of high-level abstractions (see Part 1) and speed. This allows for very high speed code that is easy to maintain, fix, test, and learn (knowledge transition amongst a team).

More info:

https://www.techempower.com/benchmarks/

https://www.logdna.com/blog/coding-for-performance-why-we-chose-rust

https://programming-language-benchmarks.vercel.app/c-vs-rust

https://programming-language-benchmarks.vercel.app/cpp-vs-rust

https://dev.to/deepu105/concurrency-in-modern-programming-languages-rust-vs-go-vs-java-vs-nodejs-vs-deno-36gg

No garbage collection

Interpreted or compiled, high level languages are usually very memory safe, due to the common language feature of a garbage collector. Memory is tracked and when variables are no longer in scope, the memory is tidied up and we never see a null pointer or experience segmentation fault/memory access violation. In fact most people with a high level language background that I speak with about this subject say the same thing every time: I've never really had to think about memory, memory safety or how I use memory. You can thank your friendly neighbourhood garbage collector.

So what's the issue? Well, the JavaScript garbage collector (GC) can take around 150ms in some cases to finish its "sweep" of memory. That is a long time in a realtime system. Go-lang claims to have the fastest garbage collection and they have managed to eliminate large memory scanning pauses; they claim to have been able to get a GC pause down to 100 microseconds. This is impressive and likely makes GC pauses insignificant for most contexts.

However, what if we are creating an autopilot for an airplane, or a self-driving car? Do we ever want GC pauses? Unless we like to make extremely dangerous (and borderline psychotic) gambles, probably not [There are exceptions, in the case of a well designed system where GC is distributed and controlled, like Pony - but they start with the words "If you know what you are doing..."]. Similarly most High Frequency Trading systems in the traditional financial trading world are written in C++ for the same reason. Even a 100 microsecond pause could lead to a significant loss from that period of "blindness".

This sounds great and you may be thinking "Sure, but I'm making a web-server so I don't think I need to worry about this GC stuff." You are probably right - except maybe from one other perspective; I'm going to hit someone's buzzword bingo here - machine empathy. The lack of a garbage collector makes the code more efficient, as seen by Rust's energy profile, because memory is not being intermittently scanned/deallocated/reallocated etc. This can be understood intuitively: a GC pause is a performance hit that takes place because (usually) all threads must be suspended (stopping of the world) to ensure no memory access takes place during the sweep [Except in the Erlang/Elixir and Pony universes of course]. Multi-core systems can even experience a detrimental impact on throughput with an increase in number of threads (as time spent collecting garbage increases).

Some popular options to avoid GC are then C/C++/Rust/Objective-C/Swift/some other ancient languages like Cobol/Pascal/Fortran. Objective-C & Swift are generally thought of as an Apple device language with associated/relevant tools (although Server-side Swift is apparently a thing now) but not as cross platform general use languages (yet) - and Swift reference counting is thought of as a type of GC (the compiler inserts release/retain code at compile time) so we can remove those. So then, eliminating the ancient languages leaves C/C++... and Rust.

Taking these other factors into consideration, which would you choose?

More info:

https://groups.google.com/g/golang-dev/c/Ab1sFeoZg_8/m/_DaL0E8fAwAJ?pli=1

https://www.oracle.com/java/technologies/javase/gc-tuning-6.html

https://wiki.c2.com/?LanguagesWithoutGarbageCollection

https://docs.swift.org/swift-book/LanguageGuide/AutomaticReferenceCounting.html

Safety features

Rust provides an incredible amount of code safety for a low-level language (more than just the glossed-over object-lifetime tracking that is often mentioned as the main feature). The first line of defence comes from the strong static typing that is use to check all type usage at compile time. Similarly, the Trait system creates a structured way to check for characteristics of generic types in "dynamic" code, meaning a function can accept a generic as long as the generic type expresses (or implements in the Rust terminology) specific Traits. A simple example is that the println!() function (which is actually a std-lib macro) can accept any type but to print it to stdout, the type must implement either the Display trait or the Debug trait (which has different usage but with this same macro). This means the code will never crash because an undisplayable type was sent to this macro - because that code would not compile in the first place!

Memory safety comes from a feature called Ownership. The theory behind ownership is two-fold; no two pieces of code should be able to write to the same piece of memory at once time, and no code should be able to access a memory location that is no longer in scope in its parent/calling function - or rather is "alive" long enough for the usage (the concept of reference Lifetimes). This is primarily expressed by "passing" ownership of variables from one part of the code to another, and by tracking the lifetime of a reference at compile time. References can be borrowed as well, so they can be read but they cannot be written to unless borrowed as mutable, which is an alternative to ownership. As you can imagine, this brand new paradigm gets complicated really fast when it is first encountered and is often the source of frustration for those new to rust.

I think there is something interesting in encountering a new paradigm like this. We assume we know how code works. We come across a new language and look for "hooks" that are familiar paradigms and philosophies, or architectures, that we can use as the foundation of our learning. Along comes a new, unique, and unfamiliar paradigm. Now, unless we take step back and accept we do not know how this feature works and have almost no existing basis for understanding it, we will start to stumble around and make mistakes, or worse, avoid that area of the language entirely because it is difficult to grok.

So is it worth learning all this stuff? Is Rust actually safer? A couple of examples and a proof:

i.

Nick Mathewson, co-founder of the Tor project started learning Rust by rebuilding Tor with Rust. He soon found that:

"Roughly half of Tor’s security issues since 2016 would have been impossible in Rust, and many of the other issues would have been much less likely, based on our informal audit."

There now exists a Rust based Tor project (ARTI) that's in development as we speak.

ii.

Microsoft has produce some research that has become almost iconic in the rust community. They examined bugs and vulnerabilities in their software going back to 2006 and stated that:

"roughly 70% of the security issues that the MSRC assigns a CVE to are memory safety issues. This means that if that software had been written in Rust, 70% of these security issues would most likely have been eliminated."

Note: CVE = "Common Vulnerabilities and Exposures"

The proof:

Beyond simple reasoning around the description of these features, how do we know they contribute towards "safety" - specifically memory safety in the language. Well, you could write lots of tests and set up many different scenarios to demonstrate this but you can further (if you really want to): In 2018, Ralf Jung, Jacques-Henri Jourdan, Robbert Krebbers, and Derek Dreyer went a step further and documented a formal safety proof for the language, which included verification conditions for libraries which, even if they contain unsafe code features, can be used to demonstrate their safety and inclusion as an extension to the language.

This is important because without these verification conditions, libraries that contain C++ could be seen as inherently unsafe. If there there is an "external" set of conditions that the library needs to meet, it's inclusion into the ecosystem becomes a less contentious process - and could provide for much more crossover with other low level systems programming languages that have decades of history solving problems that may not have been solved by Rust as yet.

More info:

https://doc.rust-lang.org/nomicon/ownership.html

https://devopedia.org/rust-language

https://msrc-blog.microsoft.com/2019/07/22/why-rust-for-safe-systems-programming/

https://msrc-blog.microsoft.com/2019/07/18/we-need-a-safer-systems-programming-language/

https://cacm.acm.org/magazines/2021/4/251364-safe-systems-programming-in-rust/fulltext

https://people.mpi-sws.org/~dreyer/papers/rustbelt/paper.pdf
https://plv.mpi-sws.org/rustbelt/

https://etaps.org/user-profile/archive/47-etaps-2018/etaps-2018/373-speaker-6

https://pvs-studio.com/en/blog/posts/0733/

In summary

Is Rust my favourite language? No. There are plenty of areas for improvement: Excessively implicit patterns (From/Into, implicit return values that are easy to miss, implicit traits like Display), some more complex internal architecture that requires "putting in the work" (e.g. Futures & "Pinning" memory when using async self-referencing futures), considerable boiler plate around custom Errors (some libraries help, but also have their own odd quirks) - I can continue - but none of them are dealbreakers. These things are being improved at an acceptable rate by a dedicated core team and a giant, motivated global community of devs (rustaceans 🦀).

The Rust revolution is not only about Rust being a very enjoyable language learn and code with - actually it's sometimes the opposite when getting started - It's about inevitability. Rust is a solution that meets several (around 18) needs, bridges programming paradigms, and provides enough added value benefits and tooling that it's difficult not to consider it for everything from WASM, Data Science, performance distributed computing, and embedded systems.

So where are going? I had a fellow JavaScript dev challenge my passion and conviction for Rust about three years ago, when I was fanboying the technology with a less explicit understanding of it. He said "So you think everyone will be using Rust in the next 5 years?" I said "More or less, yes, I think that's possible." He said "Wanna make a bet?" Today I really wish I took that bet. The last language that had so much breadth, and that was also similarly championed by its community... was JavaScript. And that is where I think we're going with Rust: Ubiquity; Rust will be soon growing everywhere.

18 factors powering the Rust revolution, Part 2 of 3

Kushal Joshi — Sun, 10 Apr 2022 21:35:18 +0000

Cover Photo by Trent Pickering on Unsplash

If you have haven't read Part 1, I strongly encourage you to do that first. It covers:

Syntax
Separation of Data and Code
Type safety
Declarative Programming and Functional Composition features

In this part we are going to cover:

Evolved package management
Concurrency ready std lib and great async support
Clear, helpful compiler errors
Zero cost abstractions (Optimisation)
Energy efficiency
WASM
Tauri - High performance, secure, cross-platform desktop computing

Evolved package management

Here's something I don't see many articles noting about the "why Rust" conversation: Package management. I remember when I was first learning C++ over 20 years ago and you had to create folders, add header files, reference the folders in compiler params/settings, go through manual linking and a bunch of initially confusing activities like non-standard build tools that are dependent on the library creators choices. Later it felt like cumbersome yak shaving that was just waiting for some automation.

Today C++ developers can use some nascent package management tools - however from the looks of it (I haven't used them - happy to hear more from today's C++ devs) it's still a painful area of the language. There's an interesting conversation here.

What about other languages? Well I used Pip (Python) for many years and battled circular/changing references and the (still) extra step of managing virtual environments. ~~It was ok but left a lot to be desired~~ [edit: I changed my mind - I recently completed the Udacity ML for Trading course and I spent more time fixing dependency issues than learning. Working with Pip is still a load of arse]. JavaScript was a small step forward but the solution there seemed to be "pile all the dependent code into a folder and compile everything". It evolved over time (tree-shaking!) but today the "10k files in node_modules" is a well known ~~joke~~ pain point.

For me, Go-lang was the first language with well thought out package management and compilation tooling, with some foresight around name-spacing the dependencies and linking to existing tools like github (easily). It was flexible, simple, and featured the ability to very easily compile to some target hardware from some other hardware (a feature spreading across other tool-chains now). I have a friend that swear by Go because of the benefits of it's simple effective tooling (and now there are package management options as the language evolves). New languages like Elixir now set the the standard with package managers like Hex that have batteries included and support building and publishing of packages with ease.

The Rust language developers have made some excellent choices around tooling and fortunately learned from the history of package management pain in other languages. For downloading, compiling, pushing open-source, and building CI tool chains, the tooling is mostly a help rather than a hindrance.

More info:

https://doc.rust-lang.org/book/ch01-03-hello-cargo.html

https://medium.com/blackode/how-to-write-elixir-packages-and-publish-to-hex-pm-8723038ebe76

https://news.ycombinator.com/item?id=19989188

Concurrency ready std lib and great async support

The rust-lang blog has a great article that lays out the rust approach to concurrency - called Fearless Concurrency. It lays out architectural and language principles and it's a good place to start towards understand the features of Rust that support concurrent programming.

To try and summarise the fabulous concurrency toolkit is quite a challenge but here goes:

The std lib contains "concurrency primitives" made up of both tooling and types that are built from the ground up to be supportive of the principles of fearless concurrency and to be thread safe for those wanting to experiment and try alternative methods/patterns of concurrent (or even parallel) programming.

Two major projects (non std lib but extremely commonly used) stand out in the area of async programming: Async std and Tokio - no doubt familiar to anyone that has turned an eye towards Rust for a second too long. Async architecture in general is likely very familiar to JavaScript programmers but in Rust there are some extra considerations (like ownership of the data that is thrown into an async function). Tokio is fast becoming a heavily supported and road tested async framework, with a thread scheduling runtime "baked in" that has learned from the history of Go, Erlang, and Java thread schedulers.

Tokio is a "take what you need" framework, whilst Async-std started as an "everything the box" solution. Today both have a lot of crossover with micro async runtimes like smol becoming the foundation one of framework and optionally usable in the other. The ability to rip out a small dependent sub-crate (dependent package) like smol and use it independently with ease never get's boring, by the way. It's great way to include a test runtime in an async library without forcing the inclusion of a giant async framework.

Both have rich documentation and both should be explored as a learning point for those new to Rust.

However, this isn't even 50% of what's out there: Need raw parallel power (and maybe don't need an async runtime)? Checkout Rayon. Need simple Actors for concurrent processing? Checkout Actix. Need a larger Actor system for fault tolerance/CQRS messaging? Checkout Riker. Damn, I sound like a youtube advert 🤦 - For real though, this is the tip of the concurrency iceberg. There is so much more - and it's growing.

More info:

https://blog.rust-lang.org/2015/04/10/Fearless-Concurrency.html

https://tokio.rs/blog/2019-10-scheduler

Clear, helpful compiler errors

from: https://www.reddit.com/r/ProgrammerHumor/comments/8nwyma/c_is_a_beautiful_language/

I remember wondering what "Syntax Error" meant when I was about 11, as it was the response to pretty much anything I typed into my Commodore 64. I didn't know what a "syntax" was but it was fun to see that this beige box responding to anything I typed, as if I was meeting a digital creature speaking an alien language and was at least having some kind of dialogue, with the hope it will one day understand me.

Later, as I learned various programming languages I learned the art of reading errors and stack traces and how to decipher the junk that compilers spewed out when everything went wrong. It was a strange "normal" and a part of learning to code in many languages, especially low level languages.

When I first came across Evan Czaplicki's "Compiler Errors for Humans" article and approach (expressed his landmark FP front-end language, Elm) I was gob-smacked. I had never realised how, or why, I had grown to accept the ridiculous, complicated, and often unintelligible, junk that passed for compiler "messaging", as being the norm.

It feels like the common sense of user-experience-centric design principles applied to compiler (or interpreter) error messaging became a tidal wave in new languages. I remember I was learning Elixir and the community forums were discussing the desire for Elm-like errors (today Elixir's error messages are wonderful too, by the way).

Rust has very much adopted this approach from the ground up and error information is extremely helpful and constantly improving. The complexities of the language, that generally stem from the vast menu of features, need this type of support, and while the initial experience can be frustrating, like a nagging, fusty, computer science pedant, lecturing you on every save, eventually you will become friends with it and it will be... emotional (I promise).

This is however, an inadequate summary of the errors, notes, critique, and alternative solutions the compiler suggests. One of the best reactions was from a developer to whom I gave an initial exercise towards learn rust. He had a background in Go-lang, PHP, JavaScript and more. I had a catchup early on and he said "It's only been 3 days and I'm love with the compiler." That's a more adequate summary.

What's creating this reaction is that, in Rust, a lack of compiler errors carries the implicit guarantee that (as long as there is no badly managed unsafe code) the compiler says your code is type-safe and memory-safe (also meaning thread-safe when data is being passed around). This increases confidence in the dev and makes refactoring far less dangerous. I even have straight-out-of-uni interns work on critical production code and I know that the compiler has got our backs.

More info:

https://ferrous-systems.com/blog/the-compiler-is-your-friend/
https://www.reddit.com/r/rust/comments/8lse7e/rust_compiler_errors_are_appreciated_apparently_d/

Example advanced response:
https://twitter.com/b0rk/status/954366146505052160?s=20&t=Ot4RtYCeRDbotl-TpoaPog

Zero cost abstractions

I was very sad to hear of the death of Joe Armstrong in 2019. He was without doubt a pioneer in our wider industry of software engineering. Joe had some classic quotes he is known for and I love them all. However, I have a bone to pick with this one:

"Make it work, then make it beautiful, then if you really, really have to, make it fast. 90 percent of the time, if you make it beautiful, it will already be fast. So really, just make it beautiful!"

— Joe Armstrong, quoted by Martin Logan in Erlang and OTP in Action, 2010, Manning

I'm going to say this quote works in Erlang or Elixir and maybe some other contexts but it's not a generalisation we can apply to all languages, especially imperative languages (see 'Declarative Programming and Functional Composition features' in Part 1). Try making "beautiful" optimised code in C++ or Go-lang that still maintains the implicit knowledge in the code required for long term maintenance by a team of devs. Usually there will be some "cheats" or alternative methods of looping, buffering data, sometimes referencing data in an unsafe way, or forced, dangerous tail-recursion, where the language doesn't support this feature in an efficient way. Even watching experts writing manually optimised code in some languages, the outcome is not intuitive and its ugliness is often hidden behind a veil of "that's the pattern to use there". The worst is often branchless-code fanatics trying to wedge a transformation into a 0 or 1 outcome to avoid a "costly" if branch ( 🤦 ).

Added to this general lack of aesthetic focus in low-level code is the overwhelming cognitive load added to the codebases where this optimisation has taken place. I can hear the responses already - in fact I heard this in an interview once - "but there are tools to alleviate this and the way I write code means I don't have that problem." I'm not going to respond much there except to say that as soon as a person is thinking about the code they write rather than the code than can be written and understood by a team of people with different levels of experience, we're already on different pages regarding teamwork and scalable productivity.

Rust helps to alleviate the dangers of this minefield of optimisation and yak-shaving boiler-plate coding by providing an ever evolving set of high-level "primitive" functions (especially around functional declarative features) that are said to be "zero cost abstractions" (a concept that will be familiar to C++ devs, in who's world the term originated).

What does this mean? Well, in general, it means, in around 99% of cases, you will not be able to code something more efficient if you tried to code it yourself. There are some exceptions and they will be addressed over time but in general it works this way: The compiler will look at the code that uses some looping or high level function and will "unfold" that code to be as fast as it can be. This means (for example) a loop that runs twice may be unfolded into the exact code running twice instead of a loop, or a generic function will be statically generated more than once in the compile (for each type) rather than dynamically derived at runtime (if that is deemed as more efficient - this can also be a dev's choice in the code) or, as the language evolves, the methods to collate, parse, or (default) sort/search data will improve over time, and we get that optimisation for free (zero cost).

What's the outcome? We get to maintain lovely high-level declarative code without having to muddy the waters with imperative optimisation cheats (that will need maintaining and reviewing in the future) - and which often impedes the knowledge transition of the intent of the author to the reader of the code - it increases the ability to code like a systems thinker (yes, that's a self-plug link but I think it's fairly relevant and interesting here, as I wrote that article before I learned Rust).

I see imperative developers often diving into the abstractions, finding issues with efficiency or making changes and pasting the code into code of their projects, as if they have achieved something great with this approach. That thinking is a little obtuse at best and a behavioural Not Invented Here mindset problem at worst. There are exceptions to every rule but in general, code is written for humans; to communicate an intent towards solving a data processing problem. This implies that code is written for knowledge transmission and therefore we should embrace high-level abstractions and cleaner code wherever possible. Rust enthusiastically embraces this philosophy.

More info:

https://boats.gitlab.io/blog/post/zero-cost-abstractions/

https://blog.rust-lang.org/2015/05/11/traits.html

Energy efficiency

With eye-catching headlines figures like "98% more efficient than Python" and using "23% less energy than C++" Rust is paving the way towards a greener future with the ability to reduce energy usage in server farms on a gigantic scale. A 30% reduction in electricity bills for your raspberry pi might seem borderline pointless - but imagine a 30% saving for a web services business running on tight margins. That's considerable, especially with the oncoming energy crises hitting some Western European countries.

But is it all about money? For business maybe, but consider this: It was recently claimed (and much debated) that (based on today's technology requirements, code, and architecture, and sources of energy etc, etc) watching a 30 minute show on Netflix could consume a small but significant amount of energy - I'm avoiding stating the amount because of the intense debates it creates; go read about it yourself.

Regardless of the amount of energy, the implication is thought provoking: When I use a digital service, I am increasing my carbon footprint by amounts that differ exponentially, depending on the language and architectural choices of the digital service provider!

Note: I wrote the section above as the Russian invasion of Ukraine was just getting started. Today we see spiking energy prices and a looming energy crisis in the UK, where I live most of the time. The ability of a new technology to drastically lower energy consumption in today's world is not merely an added extra; it's a competitive factor that will drive a switch to Rust in many industries, and for tooling across the board (see 'Everybody else's tooling' in Part 3).

More info:

https://greenlab.di.uminho.pt/wp-content/uploads/2017/10/sleFinal.pdf

https://haslab.github.io/SAFER/scp21.pdf

https://www.theregister.com/2021/11/30/aws_reinvent_rust/

https://energyinnovation.org/2020/03/17/how-much-energy-do-data-centers-really-use/

WASM

In JavaScript, there is a beautiful, elegant, highly expressive language that is buried under a steaming pile of good intentions and blunders.

— Douglas Crockford, JavaScript: The Good Parts (ed. 2008)

JavaScript, or ECMAScript (which is its true name that around 0% of people use), is my guilty pleasure. It really is an expressive tool that allows for free flowing thought to bleed into the editor, with functional composition features, easy to use async, simple baked in handling of JSON, what's not to like? Well... A lot, actually. There are many ways to discuss this but the simplest is the one that is the most frightening: I once read somewhere that anything that is possible to be structured and written in a programming language will eventually appear in the code (can't remember the reference - I'll add the reference when I find it one day). This means all the crazy stuff that's possible with JavaScript (e.g. confusing usage of syntax, accidental mathematical errors, parsing quirks, implicit type conversation to the wrong type, overly flexible architecture, silent errors, missing semi-colon issues etc) will eventually be a bugs in JS code everywhere.

Anyone that has spent time working with JS knows there are ways to work around these issues and that comes with experience, but in general this sounds like 😱 and I feel goes a step towards explaining why front-end development is 10x more complicated than any back-end-only dev already thinks it might be!

However, baked into all the frontend JavaScript engines in nearly all major browsers is an intermediate language known as Web Assembly (WASM), that JavaScript is transformed into internally. Today, other languages can generate Web Assembly and ship it to the browser and it already works great everywhere. I mean really great:

For example, this is a Python interpreter, written completely in Rust, then cross-compiled to WASM, and loaded into the browser as a component: https://rustpython.github.io/demo/.

So the question is this: If all the benefits of Rust can also be applied to frontend code, as the frameworks in this area develop, why would would people continue to choose JavaScript frameworks for web apps?

What am I saying? That Rust will end the era of JS in the browser? I don't know - probably not. JS can be really fun too. But there already is an Angular Rust project, WASM integration into some desktop application frameworks, and several rapidly growing WASM projects, so things are moving in an interesting direction. At my current work we were able to compile an entire backend rust service into a WASM module that gets loaded into a Vue based UI. We then stream the context that the backend service is receiving to the WASM module and we can "see" the output the service will be producing, in (throttled) realtime - but it's available in the browser with zero load on the backend service itself needing to deliver that data. It's a very interesting application of Rust and provokes thoughts of new alternative architectures.

More info:

https://www.decipherzone.com/blog-detail/front-end-developer-roadmap-2021

https://webassembly.org/roadmap/

https://rustwasm.github.io/

https://yew.rs/

https://github.com/MoonZoon/MoonZoon

Tauri - High performance, secure, cross-platform desktop computing

Enabled by the emergence of the webview library, Tauri opens a door that, until now, was half closed unless we were willing to sacrifice some security, performance, choice of tooling, performance, memory management and often performance... Cross platform desktop applications have been a veritable holy grail, just out of reach for those not prepared to sacrifice language-choice, speed, concurrency features, security, and memory usage. Yes, I'm talking about that incredible laptop-fan-stimulating heap of JS junk: Electron.js (Hey, it's just an opinion. Leave me alone). Electron was a great idea and still facilitates a good option for Desktop applications where security or performance is not a major concern, like the WhatsApp desktop client (:halo: ). In fact it's used for a considerable number of apps that you are probably using right now, including one that is used by many Rust developers, including me, because it's pretty awesome: Visual Studio Code 😞.

Today however, we have Tauri. I'm not going to fly the Tauri flag too much here, except to say that it's an extremely interesting project. Grab you beverage of choice and join Denjell from the Tauri Core Team at their fireside video where he presents what they are working on. Seriously - it's a another Rust revolution driver - it's currently the fourth highest ranked Rust project, even above projects you may know well such as rust-analyzer, Actix, and Tokio.

I'll say it explicitly for those skim-reading and missing the point. Rapid localised, desktop, rust applications, with a high-speed UI that leveraged the skill of almost any front-end framework developer, are about to become a reality. It won't be surprisingly to see every front-end dev adding Rust to their already ridiculously over-weight tool-belts.

More info:

https://tauri.studio/

https://evanli.github.io/Github-Ranking/Top100/Rust.html

Creating Tiny Desktop Apps With Tauri and Vue.js, Smashing Magazine, Kelvin Omereshone, July 2020

Thanks for reading up to this point. If this article is interesting, or even infuriating, I can promise more of the same in Part 3 where I will cover:

Macros!
Cryptocurrency and Blockchains
Everybody else's tooling
Speed
No garbage collection
Memory safety features

18 factors powering the Rust revolution, Part 1 of 3

Kushal Joshi — Sun, 10 Apr 2022 20:52:33 +0000

Cover Photo by Marie P on Unsplash

Where we were

In 2019, whilst working at Keyrock, the dev team (around 4-6 of us during that period) had reached the limits of what our JavaScript prototype trading services could do when it came to handling the complex world of high frequency trading (HFT). We had to process multiple feeds of information concurrently and have data available, within certain time constraints, inside a single service, with reliability. Of course, I am aware JS was probably not the best choice but we had a team-wide capability and have you ever tried to prototype in JS? It's so fast... Anyway, we had rapidly prototyped our first generation of services in JavaScript and whilst JS is suited to many use cases, (first strong opinion of the article) HFT is not one of them. Not that this stopped us trying 😇.

In choosing a new technology for the second generation of services we went through a research phase, trying out Go-lang, Kotlin, Elixir, and also another language I was tinkering with... Rust. We knew we didn't want to go the C++ route for reasons I will expand on in a different article. I'd done some Rust tests with basic sorting algorithms and couldn't quite believe the speed and kept going on about it at our office in Brussels. Another senior dev challenged the technology (vs what we had in production) and said he didn't want to keep hearing me go on about it unless it was significantly faster than what we had; he did some deeper experiments himself and soon went on to become our most ardent Rust emissary. Rust had the speed we needed, approachable concurrency primitives, acceptable syntax, and great tooling. We dove into this new world and today most of our production code is Rust.

However, there was something else about this language, even at the early stages, that I couldn't quite put my finger on. While sometimes complex to deal with, at the time I knew our decision was right even though I couldn't expand much on why, beyond the speed, lack of garbage collector, and concurrency support. Was that risky? I didn't think so - we did (just) enough research and I personally had Apple's philosophy ringing in my head - that we should use the opportunity to both eliminate a technology we had maxed out, and replace it with a new technology entering it's dawn.

Three years later I can express explicitly what I think I sensed intuitively - And I feel compelled to write about it! If you are new to Rust, I know where you are going... it is a world of initial confusion and frustration, followed eventually by a feeling that this just makes sense.

Today we see rust everywhere. Blockchains, web services, libraries supporting Python and JavaScript modules, even inside the holy of holies - the Linux kernel. This three-part article is about why that is the case.

Why is this happening?

Is it a cult? Is it mass hysteria? Or it just the latest fashion in hipster coding languages that's running out of control? Well, maybe there's an element of this - but there are also many concrete reasons this revolution is happening. And make no mistake - it is a revolution.

From the 2021 Stack Overflow Developer Survey

We can begin with the factors that are easy to understand and accept, and then I'll get into the less obvious but immensely powerful features of the language.

Let's start with the entire list and then break them down:

Part 1

Syntax
Separation of Data and Code
Type safety
Declarative Programming and Functional Composition features

Part 2

Evolved package management
Concurrency ready std lib and great async support
Clear, helpful compiler errors
Zero cost abstractions (Optimisation)
Energy efficiency
WASM
Tauri - High performance, secure, cross-platform desktop computing

Part 3

Macros!
Cryptocurrency and Blockchains
Everybody else's tooling
Speed
No garbage collection
Memory safety features

Syntax

At university I had some experience with Pascal and Smalltalk (it was a long time ago), before that I was coding Motorola 68000 series assembler on a Commodore Amiga and before that (my first exposure to low-level coding) I was playing with machine code on the MOS Technology 6510 in the Commodore 64.

Why am I sharing this? Because the coding syntax in that world was not "C" style. During the following years I also used C, C++, Java, JavaScript, Perl, C#, and later R, which are known as C-type or C-family of languages. (Python is a little different but it's not far off). The other languages above are not and this often leads to frustration in learning the language and difficulty in finding people that can "convert" to those languages.

The C-type languages above are actually a very small selection of the number of languages using C-family syntax (around 73). Why so many? Well, after the success of C & C++, it was theorised that it would be much easier to dive into a familiar syntax if learning a new language. For example Sun Microsystems worked on a project to clean up and and make C++ better by being able to run it everywhere - called Oak. Later they needed to change the name and they called it Java.

The fact that Rust retains this C-type "DNA" is by design. Systems programming is primarily the domain of C & C++ developers, with a smaller group of Objective-C (a superset of C) and Swift developers in the Apple universe. Rust honours this legacy - and targets those developers - by fusing a very familiar syntax together with symbols and features that facilitate new paradigms in a way that is only mildly distracting at the beginning of the Rust journey, and makes a lot of sense when you get used to it.

I've had a tech-challenge submissions from C++ devs with only 1-2 months of Rust experience and they "get it", often much more quickly than devs with other backgrounds. These submissions are not idiomatic Rust at all but they show the power of a familiar syntax.

Separation of Code and Data

I'm a big fan of Functional Programming (FP) - and Elixir and Erlang in particular. The original sub-heading here was: A reasonable step away from the era of misunderstood Object Oriented philosophy and architecture, towards separation of code and data, using Traits and Functions.

That's a long section title. I rewrote it several times. It was really difficult to find a title that represents this factor that wouldn't trigger the masses of converts of the confusing, and usually misunderstood, interpretation of Object Oriented Programming (OO, or OOP), who have spent years teaching and developing methods and patterns for structuring code, based on an approach that has virtually zero evidence but instead was the result of millions spent in marketing, over around 3-4 decades, plus the directed efforts of non-technical managers to split coding into "units of work".

There are various histories that document this (my favourite is by the eternally affable Richard Feldman, author of the Elm programming language). A very short summary:

A long time ago programs were made of data and procedures and were often a highly coupled mess of data sharing. Then a guy call Alan (Kay, 1966) said programs should pass messages instead of share data (an idea from Simula, 1962). He and others made a language of objects with associated functions that passed messages between objects (Smalltalk '72). But it seems (to keep the story short) he was somewhat mis-heard, and later languages implemented these ideas with their own odd assumptions.

“C++ is a significant barrier to learning and correctly applying object-oriented methodologies and technologies.”

— Bruce Webster, Pitfalls of object-oriented development
(New York: M&T Books, 1995), 139 (quoted in Hsu, Mahoney - see below)

Alan also tried to explain some of the confusion many years later:

“I made up the term ‘object-oriented’, and I can tell you I didn’t have C++ in mind.”

— Alan Kay, OOPSLA ‘97

“OOP to me means only messaging, local retention and protection and hiding of state-process, and extreme late-binding of all things.”

— Alan Kay, 1998, https://wiki.c2.com/?AlanKayOnMessaging

"I'm sorry that I long ago coined the term "objects" for this topic because it gets many people to focus on the lesser idea."

— Alan Kay, 2014, http://lists.squeakfoundation.org/pipermail/squeak-dev/1998-October/017019.html

Unfortunately though, by that time the world had mostly put Functional Programming, and landmark computer science theories such as Carl Hewitt's Actor model theory (which were partly inspired by Smalltalk '72) aside, and progressed the "OO super-soldier" ideas of Objective-C and C++ and mutated them into the uncontrollable "Hulk" of Java. (a language which once had so much kudos that another mostly unrelated language was named after it, and mutated to look like it).

"Object-oriented programming, for instance, was seen by one adherent as a way to impose the traditional workplace relationships of manufacturing. Taylorism was implicit in all the proposed solutions to the crisis."

— Hsu, Cornell University, 2020

By the time the corporate sponsors and their agents had finished with computer magazine articles and adverts, university course sponsorships, and mass marketing of their consultancy services behind the languages, the idea that OO was the de facto way to code had become ubiquitous. That word will back later in this article.

There are exceptions: Managing a handler for a side-effect should (for me) pretty much always be architecture similar to an "object + methods" because it just works wonderfully (e.g. DB driver code, wrapped in a Class or Struct + Methods in Rust) because there is an exclusive association between the code and datatype, in that case. But for flexible, sane, maintainable architecture for concurrent, distributed systems, (meaning mass interconnectivity of functionality) there are less troublesome, more flexible, elegant, and testable alternatives (for examples, please go read Carl Hewitt and argue with him and others like the creators of Erlang, and not me, please. Or better, first watch the excellent Brian Will and Richard Feldman videos linked below).

"I think the lack of reusability comes in object-oriented languages, not functional languages. Because the problem with object-oriented languages is they’ve got all this implicit environment that they carry around with them. You wanted a banana but what you got was a gorilla holding the banana and the entire jungle.

If you have referentially transparent code, if you have pure functions — all the data comes in its input arguments and everything goes out and leave no state behind — it’s incredibly reusable."

— Joe Armstrong on software reusability.

Thankfully, the pain of the "taught" way of OO coding is finally leading the world to rediscovery of FP benefits (even Java has an FP/OO hybrid successor - Kotlin) and alternative practices and architectural approaches. Mozilla and the Rust core-team thankfully chose an alternative architectural approach when making Rust: Traits. This was a stroke of genius in my opinion as it has resulted in (very amusingly) people migrating towards FP architectural practices without even knowing they are doing this, often whilst complaining about FP.

As a simple intro, Rust devs can create data structures called Structs and associated functions and/or methods (with an explicit self reference) with those Structs. This however has limitations when creating abstractions and eventually leads to the discovery of Rust Traits. Traits are akin to interfaces in some OO languages. A trait usually consists of function signatures that need to be expressed by a struct claiming to express that trait. In this way they are like a set of interrelated behaviours for some unknown type. Not quite a pure function but a constrained function, with expectations of the type that will be transformed by the function.

So what? Well... it means we can:

Create a set of pure functions that process data, or we can...
define a set of methods as behaviours for some generic type with some expected traits, or...
we can create a known Struct with a set of traits and expressed functions for those traits, specifically designed for that Struct - a little like an Object + methods - but with the trait we retain the principle of the struct - or "object" - being open for extension - one of the first five principles of object oriented design, all of which can still be expressed in Rust without permanently coupling data to code.

This lack of opinion in the features of the language means this is enough flexibility (for example) to build applications with the hailed "functional core, imperative (OO) shell" architecture, and yet provide the option to lean on the safety of Rust (see Part 3) when coding this way. In this way Rust makes for an incredible language that allows for a wide scope of architectural styles. It says "don't be afraid! If you love the pain of early architectural assumptions that OO enforces via the myth of encapsulation, you can still feel that hurt with Rust!". ;)

More Info:

Sun preps 500m Java brand push

Object-Oriented Programming is Bad, Brian Will, Youtube, 2016

Why isn't functional programming the norm? Feldman, Youtube 2019

Connections between the Software Crisis and Object-Oriented Programming, Hansen Hsu

No Silver Bullet, Brooks, 1987

Functional Core, Imperative Shell, destroyallsoftware.com, 2012 by @garybernhardt

Type safety

Static Types

In some of the few studies carried there are observed benefits to statically typed languages. If you haven't heard of Quorum, check them out - they are bringing evidence to software development - an industry where our practices were prescribed by managers and shaped from a desire to deskill and control developers (see also, Hsu, above) rather than via gathering evidence of best practice.

For developer convenience in IDEs static typed languages win all day long, as they do when it comes to automated error checking (e.g. functional signatures) in code. Typing is so important to some developers that JavaScript has an entire superset language created around it that enforces types (cheers Microsoft - 👍 you did a good. I don't get to say that often).

Dynamic Types

Look I'm not selling the point - I love me some JavaScript when I'm playing around or I'm guiding friends/relatives to learn code. But if I want to create a financial system/trading bot etc, in that context, explicitness rules. I don't want to second guess type inference anywhere. But I really like dynamic types; enter Rust Generics and Traits...

Rust Generics

The generics system in Rust allows for the ability to specify the Traits (see section above) that the the "type" expresses. In simplistic terms this means we can use dynamic types (a generic type) but restrict those types to those that express a specific set of Traits. It often feels like the best of both worlds. It can create some complexity in areas but in general it works extremely well for modelling complex abstractions.

More Info:

Human factors evidence in language design, Quorum

Software as a Labour Process, Ensmenger & Aspray, 2002

TypeDevil: Dynamic Type Inconsistency Analysis for JavaScript, Pradel, Shuh & Sen

The Plot to Deskill Software Engineering, Glass, Communications of the ACM, 2015

Declarative Programming and Functional Composition features

There are common problems with attempting to encapsulate the world as objects and behaviours all the time (see Separation of Code and Data, above). Once those behaviours start mixing, the variations require more and more complex objects with multiple parents and specific variations of methods, often leading to a lot of duplication of code and code that is difficult to maintain when changes need to be applied (like hybrid objects that facilitate/abstract communication between completely unrelated objects) - The way a method expresses behaviour (the knowledge of it) is spread throughout the 'children' of the objects (it's not very DRY code and it's not very flexible). There's an entire industry of rules and patterns to help deal with this mess.

Functional Programming uses a paradigm that helps to solve these types of scenarios in a rather elegant way: Functional Composition. Generic, "pure" functions replace methods-attached-to-data and instead "process" data that conforms to defined characteristics. This way every inherited variation of the data structure can be processed by a set of functions that describe behaviours, and those behaviours can then be composed to create more complex behaviours. But each behaviour is only expressed once (#極度乾燥). You have probably used these approaches already if you have played with Iterators or Streams.

How does Rust facilitate this? With Generics, Traits, the Fn() and related types, closures, and iterators. Most importantly, these features are not pushed into your face with enforced interior immutability like other FP languages. Rust instead provides a buffet of functional features that one may or may not choose to use - but it's a heavily stacked buffet, leading to two outcomes: Experienced FP devs generally won't go hungry, and people new to FP can learn, experiment, and introduce architectural changes gradually.

A small example that will make sense to FP devs but may irritate or confuse people new to the FP paradigm: By default all variables are immutable and functions all implicitly return their "final value", or the result of their final expression, as long as there is no semi-colon ending the expression. If there is no value (if there is a semi-colon that ends the last expression), the function will still return an empty tuple. Taken together this means Rust functions (by default, without usage of the mut keyword) have no internally mutable variables. They process some input, and output a new value/s based on that input. If we then remove the usage of mut on the params input to the function, we have an FP style data-processing function that cannot mutate it's input or it's internal variables, and thereby results in an idempotent function (always produces the same effect as the first time it's run).

How can these be used with traditional loops and if/else branching? Well that would be clumsy - so Rust also contains high level abstractions like map, filter, fold, and other reducers that allow the building of iterators that apply functions; hello declarative programming! Structured correctly, this raises the possibility of creating highly optimised branchless code - but branchless code that is still readable!

If this sounds alien to your "normal" coding world, it's likely you are coming from an imperative/object-oriented programming paradigm - and that's fine. Even without knowing the definition of these mysterious words: We can add mut to function params, and mut in front of variables, and all the usual if/else/for/while tools are present.

In this way Rust uniquely exists as a bridging or transition tool from one paradigm to the other.

More info:

https://www.freecodecamp.org/news/imperative-vs-declarative-programming-difference/

https://www.fpcomplete.com/blog/2018/10/is-rust-functional/

https://github.com/hemanth/functional-programming-jargon

https://hbfs.wordpress.com/2008/08/05/branchless-equivalents-of-simple-functions/

That's it for Part 1. In Part 2 I'm going to focus on:

Evolved package management
Concurrency ready std-lib and great async support
Clear, helpful compiler errors
Zero cost abstractions (Optimisation)
Energy efficiency
WASM
Tauri - High performance, secure, cross-platform desktop computing

Connect to a single MongoDB instance from rust

Kushal Joshi — Mon, 03 Feb 2020 10:20:28 +0000

The rust driver for Mongo is still in its infancy. It is well documented in terms of the signatures of each function and feature but very limited when it comes to examples. I spent way to long trying to make a simple connection to local and staging Mongo instances and the examples just don't work.

Solution:
They are missing the direct_connection option flag.

fn main() {
    // Set URI
    let uri = "mongodb://localhost:27017/";

    // Setup options
    let mut options = ClientOptions::parse(uri).unwrap();
    options.direct_connection = Some(true);

    // Create the Client
    let client = Client::with_options(options).unwrap();

    // Use the Client
    let _dbs = dbg!(client.list_database_names(None));
}

Note for noobs, you don't need the dbg!() wrapper - that is just to log the output easily. You can just:

let _dbs = client.list_database_names(None);