Forem: Dan

A full-stack serverless application with AssemblyLift and Next.js

Dan — Tue, 11 Oct 2022 19:21:45 +0000

Today we published a new demo illustrating several features of the latest AssemblyLift release!

This demo deploys a simple "hello world" Next.js application to an AWS environment. Visits to the app are counted by invoking a logging function from the server function.

The demo repository is on GitHub. You can see it live here!

Topics illustrated:

Serving static web assets from an AssemblyLift function
Serving a private API from an AssemblyLift function
Authoring AssemblyLift functions in both Rust and Ruby
Sandboxed WebAssembly network access using an IO Module
- Making HTTP calls to another function or to a 3rd party service
- Making calls to an AWS service (Secrets Manager)
Deploying a service to a custom domain name
Automated TLS certificate configuration & provisioning

AssemblyLift is an open platform for cloud-native application development. AssemblyLift provides a portable, function-oriented framework and WebAssembly-based runtime which can be deployed to AWS Lambda or Kubernetes. The AssemblyLift CLI generates HashiCorp Terraform infrastructure code from simple TOML definitions, and takes care of compiling and packaging functions and services for deployment. To make a clichéd comparison, think of it as Infrastructure on Rails 😛

Next.js by Vercel is a popular JavaScript/TypeScript frontend framework based on React. Next allows us to export our React application as static HTML which we can serve with AssemblyLift.

Deep dive

Services & Functions

AssemblyLift applications are composed of services, each of which are made up of functions. Services are stored in the services directory. Each service is made up of a service manifest named service.toml alongside one or more functions. This demo application deploys one service named www which contains two functions, server and counter, each of which do exactly what it says on the tin :).

The server function is our HTTP server which serves web content. The counter function is a private function (protected by IAM) which is called from server; this function updates a simple count of visits by IP in a Xata database.

For performance, the server function is written in Rust. Rust compiles natively to WebAssembly, and so has faster cold-start time as well as faster execution speed compared to Ruby -- ideal for our server! We leverage a Rust crate called rust_embed which allows us to embed assets inside a compiled binary. We use this to embed the assets generated by our Next.js build inside the WebAssembly module which AssemblyLift will deploy as our Lambda function!

The counter function is written in Ruby. Since Ruby is an interpreted language, AssemblyLift deploys a customized Ruby 3.1 interpreter compiled to WebAssembly, which executes the function handler. Since the interpreter is somewhat large, the cold-start time of a Ruby function tends to be larger than that of a Rust function. Our counter is being run in the backround, so we're fine with it being a little bit laggy at times 😉.

HTTP API

Regardless of infrastructure provider, AssemblyLift services are placed behind some kind of API Gateway service. When deployed to AWS, each AssemblyLift service is placed behind an an Amazon API Gateway endpoint. Each function may define an HTTP route which will invoke it. The server function is invoked by GET /{path+}; the {path+} token is a path parameter where the + indicates that it is a greedy parameter. This means that everything after the first / is mapped to a parameter called path. This allows us to map the requested path to an embedded path in our function binary. The counter function is invoked by POST /api/counter/{ip}, where {ip} is a regular path parameter named ip.

A function's HTTP route is defined in service.toml:

[[api.functions]]
name = "counter"
language = "ruby"
size_mb = 3584
http = { verb = "POST", path = "/api/counter/{ip}" } # This defines the HTTP route for counter
authorizer_id = "iam"

The counter function is protected by API Gateway's built-in IAM authorizer. AssemblyLift doesn't yet support defining access permissions between functions in TOML, so for this demo a small amount of Terraform is included to attach IAM policies to each function. AssemblyLift instantiates the user_tf directory (if found) as a module alongside its generated module(s).

Domain name mapping

AssemblyLift projects can specify one or more domain names to which services can be mapped using a DNS provider. At the moment, the only available DNS provider is Amazon Route53.

Domain names are defined as an array in assemblylift.toml:

[[domains]]
dns_name = "demos.asml.akkoro.io"
[domains.provider]
name = "route53"
[domains.provider.options]
aws_region = "us-east-1"

With Route53, the dns_name must correspond to an existing Route53 Hosted Zone.

By default, services are mapped to subdomains according to the pattern service-name.project-name.domain-name.tld. For example, the www service would be www.nextjs.demos.asml.akkoro.io.

A service can indicate that it is the root service, omitting the service subdomain from the path. For example in service.toml:

[api]
domain_name = "demos.asml.akkoro.io"
is_root = true

yields the domain nextjs.demos.asml.akkoro.io for the www service.

When deployed to AWS, AssemblyLift will provision TLS certificates for your services using Amazon Certificate Manager (ACM).

IO Modules

The counter function uses a Xata database for storage, which provides a JSON-oriented HTTP API which we can access using the HTTP IO Module. WebAssembly's sandboxing means that by default, functions cannot communicate over a socket or access the filesystem. AssemblyLift provides an RPC interface allowing communication with services called IO Modules or IOmods. IOmods are deployed per-service, i.e. every function in a service share the same set of IOmods. An IOmod is essentially a library of remote functions, each at an assigned coordinate organization.namespace.module.call.

For example, importing the standard HTTP module:

[[iomod.dependencies]]
coordinates = "akkoro.std.http"
version = "0.3.0"
type = "registry"

exports a single function named request at akkoro.std.http.request.

Next Frontend

There is no specific convention for adding a frontend to AssemblyLift; in this project we have created the frontend directory to mimic the services directory. Here we have used create-next-app to create a Next.js project called www after our service of the same name.

The frontend must be exported to a static site, using next build && next export. The contents of the generated out directory are copied into the server binary when it is compiled (on asml cast).

Deploy your own!

You can deploy this project yourself! You will need to update it to use your own domain of course ;)

You will need...

Required:

An AWS account
Rust & Cargo CLI
Node & NPM CLI
Latest AssemblyLift CLI

Nice to have:

A domain name in Route53

An AssemblyLift project is built with the asml cast command; casting is the process of compiling WASM, transpiling TOML, building images, etc which comprises the build of an application. Artifacts and plans are serialized to the net/ directory.

To deploy a project you use the command asml bind, which will deploy artifacts and apply the underlying Terraform plan.

Reach out!

Get in touch with us on Discord and follow @akkorocorp on Twitter.

AssemblyLift alpha latest: easy API Gateway for Kubernetes functions, Ruby language support 💎

Dan — Wed, 22 Jun 2022 16:41:59 +0000

Hello again y'all! 👋🏻

Since our last post introducing the AssemblyLift v0.4-alpha series, we've had two new alpha releases which introduced some major additions & improvements!

In this post we'll talk about each change and what it means, and then we'll walk through a short example demonstrating the new features. 🙂

Revised TOML manifest spec

Manifests (assemblylift.toml and service.toml) now use arrays to describe most entries which used to be written using tables.

For example, the [services] table in assemblylift.toml is now the [[services]] array.

Why?

The primary data structures in TOML are arrays and tables.

Previous releases leaned heavily on tables. A typical assemblylift.toml for example might be:

[project]
name = "my-project"

[services]
my_service = { name = "my-service" }
another = { name = "another-service" }

Both [project] and [services] are tables. The my_service key also points to a table. Expressing services this way (as well as functions, authorizers, etc.) is a bit verbose considering that these things are also named.

By comparison, using arrays to add multiple services to assemblylift.toml for example looks like:

[project]
name = "project"

[[services]]
name = "service1"

[[services]]
name = "service2"

Which avoids redundant information (the table key) and becomes easier to read.

Ruby language support

AssemblyLift is now bilingual! Functions can now be written in Ruby in addition to Rust.

To declare a function a Ruby function, set the language parameter.

For example:

[[api.functions]]
name     = "my-ruby-func"
language = "ruby"
http     = { verb = "GET", path = "/my-service/my-ruby-func" }

A Ruby function requires handler.rb to be the entrypoint. AssemblyLift ships a custom build of the Ruby runtime which includes a module for interop with the AssemblyLift platform.

We'll explore this in more detail in the demo section below!

Why?

We ❤️ Ruby! Most commonly associated with the Rails framework, Ruby is a powerful & mature scripting language suitable for many purposes.

Ruby recently gained support for WebAssembly, and we almost immediately went about integrating it into AssemblyLift! 💨

While you can't (yet?) lift your existing Rails app to a Function, the Ruby language is familiar to many devs. Ruby could especially be preferable in cases where performance is less of a concern, such as automation tasks for example.

Support for Gloo Edge API Gateway

Another feature addition we're very excited about is HTTP API support for the new Kubernetes provider. The previous alpha release could deploy functions to a K8s cluster, but there was no practical way yet to invoke them!

This release will automatically install Gloo Edge on the target cluster. AssemblyLift will deploy a Gateway, and set up routing for any functions running in a K8s service which define an HTTP API.

Why?

An API Gateway is central to the AssemblyLift architecture. AWS Lambda has a natural pairing with Amazon API Gateway, but Kubernetes of course is open-ended.

For this first iteration, Gloo Edge was chosen as it is open-source, easy to automate, and very full-featured. Being itself based on Envoy Proxy we can eventually explore its support for WASM traffic filters in Gloo Edge Enterprise.

A potential drawback is that Gloo may be heavier than needed for your application (our test deployment required 2vCPU per node to run Gloo + an AssemblyLift service). We'll soon look at support for other (lighter) reverse proxies. Eventually we want to support multiple API Gateway providers per backend, and also allow for cross deployment (Amazon APIGW in front of K8s, Gloo in front of Lambda).

A short demonstration

Let's look at an example! If you want to try AssemblyLift yourself and follow along, you will need the prerequisites below. Otherwise if you just want the gist of things, you can skip that and keep reading 🙂

In this example we'll deploy a basic Ruby function to a Kubernetes cluster that writes to the function log, and curl it via an API Gateway.

Prereqs

You will need a running Docker daemon on your development machine. You will also need a configured k8s config at the default ~/.kube/config. So far AssemblyLift has been tested against Docker Desktop Kubernetes as well as Digital Ocean Kubernetes. It isn't strictly necessary, but is recommended for testing and debugging, to have kubectl installed.

You will also need a Docker image registry. Right now AssemblyLift supports Docker Hub and AWS ECR, however ECR is recommended as your registries can be provisioned/destroyed automatically. Currently, AssemblyLift will attempt to use AWS credentials from your default profile in ~/.aws/credentials.

You can grab AssemblyLift for macOS with

curl -O public.assemblylift.akkoro.io/cli/0.4.0-alpha.3/x86_64-apple-darwin/asml

Or for Linux with

curl -O public.assemblylift.akkoro.io/cli/0.4.0-alpha.3/x86_64-linux-gnu/asml

Make the binary executable by running chmod +x asml and ensure it is available on your system PATH.

If your Linux system's glibc isn't compatible with the asml binary you may need to build from source instead. The easiest way to do this is to install with cargo by running cargo install assemblylift --version 0.4.0-alpha.3.

Create a Ruby function

We'll create a new project with a Ruby template:

asml init -n ruby-example -l ruby
cd ruby-example

This will create a basic Ruby function called my-function inside a service my-service. These can be renamed as you like, however the directory names also currently need to be updated manually to match (we'll have this sync in a future release :)).

Add an HTTP route definition to the function in service.toml:

[[api.functions]]
name = "my-function"
language = "ruby"
registry = "ecr" # or "dockerhub" if using Docker Hub
http = { verb = "ALL", path = "/my-service/my-function" }

Set provider and configure container registry

Again in service.toml, add the following to use the Kubernetes backend provider:

[service.provider]
name = "k8s"

Then in assemblylift.toml, configure the container registry you wish to use:

[[registries]]
host = "ecr"
options = { aws_account_id = "12345890", aws_region = "us-east-1" }

[[registries]]
host = "dockerhub"
options = { registry_name = "<name>", username = "<username>", password = "<password>" }

Write some Ruby!

Next, let's open up the function's handler.rb:

require 'asml'
require 'base64'
require 'json'

def main(input)
    # TODO implement your function code here!
    Asml.success(input.to_s)
end

main(JSON.parse(Asml.get_function_input()))

The asml module includes the Asml class, which exposes basic methods for interacting with the AssemblyLift runtime environment.

The input object has shape:

{
  method: String,
  headers: Object,
  body_encoding: String,
  body: String
}

We can decode the body and log it:

def main(input)
    body = input['body_encoding'] == 'base64' ? Base64.decode64(input['body']) : nil
    if body != nil then
        Asml.log(body)
    end

    Asml.success("OK")
end

Build, deploy, test!

Generate your deployment by running asml cast. If there are no errors, you should see a Terraform plan in the output.

Deploy your service with asml bind! Once deployed successfully, your cluster should have a Gloo Edge installation in the gloo-system namespace and a function deployed as ClusterIPs in an asml-* namespace for your service (you can check this by running kubectl get namespaces).

The Gloo Edge Gateway will have created a LoadBalancer resource, which should be deployed by your Kubernetes host (e.g. Digital Ocean). You should be able to curl the LoadBalancer public IP with your function's route!

curl -X POST -d "Hello World" http://<ip>/my-service/my-function

Reach out!

Whether you have questions about using AssemblyLift or just want to chat, you can find us on Discord!

AssemblyLift v0.4.0-alpha released with Kubernetes support, WASI (WebAssembly on K8s)

Dan — Wed, 27 Apr 2022 18:00:37 +0000

Hello devs! 👋🏻 We're back!

Today we're making available the first alpha release of AssemblyLift v0.4. This initial release brings a few exiting improvements to AssemblyLift including

WASI modules; Asml functions now target wasm32-wasi and are compiled as executables instead of libraries
Simplified Rust function boilerplate (thanks to WASI)
Experimental Kubernetes backend provider
Hyper-based HTTP function runtime

Let's take a look!

Installing

We're going to use the new k8s provider, so as a prerequisite you'll need docker installed on your system as well as a configured Kubernetes cluster. Functions are written in Rust and will need cargo to be available as well. Finally, as of writing the AssemblyLift runtime images are hosted on public ECR, which may require authentication using the aws CLI.

You can grab AssemblyLift for macOS with

curl -O public.assemblylift.akkoro.io/cli/0.4.0-alpha.1/x86_64-apple-darwin/asml

Or for Linux with

curl -O public.assemblylift.akkoro.io/cli/0.4.0-alpha.1/x86_64-linux-gnu/asml

You can make the binary executable by running chmod +x asml.

If necessary, authenticate docker with ECR using

aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws

Deploy a function to Kubernetes

Let's spin up a new project to experiment with

asml init -n asmlnetes && cd asmlnetes

Next, open up service.toml and update our service & function definition to use the k8s backend.

[service]
name = "my-service"

[service.provider]
name = "k8s-hyper-alpine"
[service.provider.options]
registry_name = "my-dockerhub-registry"
# ECR is also supported! Instead of registry_name use:
# registry_type = "ecr"
# aws_account_id = "1234567890"
# aws_region = "ca-central-1"

[api.functions.my-function]
provider = { name = "k8s-hyper-alpine" }
name = "my-function"

We've also deployed the HTTP IOmod as a container which can be specified via the iomod section as follows

[iomod.dependencies.http]
type = "container"
version = "0.2.0"
coordinates = "akkoro.std.http"

If you want to play with the HTTP module, you'll need to add the corresponding dependency to the functions' Cargo.toml.

http = { package = "assemblylift-iomod-http-guest", version = "0.2" }

Our function handler used to be defined in lib.rs, however with WASI support we now use a regular main function in main.rs.

use asml_core::*;

#[handler]
async fn main() {
    FunctionContext::log("Hello, world!".into());
}

On both Lambda and Kubernetes platforms, function input is now stored in the variable ctx.input, which is injected by the handler macro.

The input shape used by the Hyper runtime isn't finalized, but as of writing you deserialize function input as follows

use std::collections::BTreeMap;

use asml_core::*;
use serde::Deserialize;

#[handler]
async fn main() {
    let event: HttpFunctionRequest = serde_json::from_str(&ctx.input).unwrap();

    // The event body is encoded as Z85; in this case we assume it's a string, but it could be binary!
    let body = std::str::from_utf8(z85::decode(event.body.unwrap()).unwrap()).unwrap();
    FunctionContext::success(format!("Body = {:?}\n", body));
}

#[derive(Deserialize)]
struct HttpFunctionEvent {
    method: String,
    headers: BTreeMap<String, String>,
    body: Option<String>,
}

IOmods are called as they have been in previous versions, but the new version of the HTTP module guest includes a handy request builder to make things a little easier 🙂

use std::collections::BTreeMap;

use asml_core::*;
use serde::Deserialize;

use http::HttpRequestBuilder;

#[handler]
async fn main() {
    let http_req = HttpRequestBuilder::new()
        .method("GET")
        .host("akkoro.io")
        .path("/")
        .build();
    let http_res = http::request(http_req).await.unwrap();
    FunctionContext::log(format!("HTTP status: {}", http_res.code));

    FunctionContext::success(format!("Body = {:?}\n", http_res.body));
}

#[derive(Deserialize)]
struct HttpFunctionEvent {
    method: String,
    headers: BTreeMap<String, String>,
    body: Option<String>,
}

Running asml cast should compile the function and generate a Terraform plan, which if everything is defined correctly should create some Kubernetes resources in a namespace (asml-{project_name}-{service_name}).

This should work against whatever cluster is defined in ~/.kube/config, however at the moment only a NodePort service is created for each function -- LoadBalancer support for cloud deployments on AKS and such will come in a later update.

After running asml bind to deploy your service(s), you'll be able to find the function pods for example with

kubectl get pods -n asml-asmlnetes-my-service

If you're running on a local K8s cluster (I use the distro provided by Docker Desktop personally), you can cURL requests to localhost on the exposed NodePorts for each function.

Find the port by listing the function services

kubectl get services -n asml-asmlnetes-my-service

What's next?

The 0.4.0-alpha.x series will iterate on the above compute platform enhancements, concentrating mainly on k8s support.

The data model we wrote about a while ago is still going through more of a design phase. The plan is to introduce that functionality in a 0.4.0-beta series, and iterate on that up to the final 0.4 release.

What's next for AssemblyLift? Data-oriented cloud dev through WebAssembly and capability-based security

Dan — Tue, 19 Oct 2021 16:07:45 +0000

After nearly two years in part-time development, AssemblyLift has been through three major revisions, sitting now at v0.3.2.

The focus up to now has been on creating a low-friction environment for building applications with functions and APIs, and on the core functionality of the WebAssembly runtime. Using a simple TOML definition, AssemblyLift makes it easy to deploy Rust functions to AWS Lambda fronted by API Gateway. TL;DR it's a nifty little API-oriented compute framework!

Conspicuously missing from this framework however is any notion of data. Where do we store it? Who is allowed to access it? I've personally never written a dynamic web service that didn't need to interact with a data store in some way or another!

It's common to pair serverless compute like Lambda with a serverless database like DynamoDB. This pairing requires the Lambda function to have an attached IAM policy granting it the necessary access to DynamoDB resources. With AssemblyLift currently, attaching IAM policies is done either via the AWS Console or by including your own Terraform code with the AssemblyLift project (this process is covered in another blog post).

This service-to-service permissions model is common among cloud providers, and is usually implemented as Role-Based Access Control or RBAC. In our example above, a Lambda function is assigned an execution role which has one or more policies attached. If the function is allowed to assume the role, and if the attached policies allow it the function can access DynamoDB. Or more accurately, it can access whichever DDB resources are specified in the policy resources list. AWS policies also allow you to specify a list of conditions on the data which can be accessed, for example by only allowing GetItem for items with a particular prefix on the primary key. The official AWS docs contain more details on DynamoDB access management.

In a serverless context, this model of access control can create some problems. To maintain the Principle of Least Privilege it is recommended to keep a one-to-one relationship between functions and roles, i.e. functions should not share roles. At scale you may have hundreds of functions, for each of which a unique access policy is maintained. This means that changes to access policy for any data potentially has to be replicated many times. This complexity increases the risk of having misconfigured and/or inconsistent access policies, even if they are represented as code. Using policy conditions to restrict data access is problematic, as your database "schema" (the format of your keys) is then hard-coded separately from the function code.

We didn't want to expose this complexity in AssemblyLift. It's the kind of "plumbing" code nobody enjoys writing, and it's also the kind that tends to get glossed over simply because it's hard to get right. Fortunately, it is now 2021 and we have some new tools at our disposal!

Entity-Capability Data Access Control

Entity-Capability Data Access Control (ECDAC) is the tentative name for the data access model being developed for AssemblyLift (as a kind of nod to the Entity-Component-System architecture which inspired it). The model is intended to alleviate the issues with policy definition described above. It is also meant to address issues with maintaining & communicating entity/object schema definitions between many functions and services, which will be discussed below.

In this scheme there are (as you might have guessed) two essential components:

Entities
Capabilities

When we talk about an Entity, we mean an object of some kind with a unique ID (e.g. Person, Account, Post, Comment, etc). This is the same sort of entity concept which is often used in database design, including DynamoDB "single-table" design.

Capabilities refers to capabilities in the capability-based computing/security sense. Connor Hicks at Suborbital wrote an excellent blog post which discusses the details and benefits of the capability-based model better than I probably can here. In short, capabilities define what an application component can do. A Capability is typically some kind of unforgeable token or object, which embeds a set of access rights.

An important distinction between access rights encoded as a Capability, and access rights encoded in an RBAC policy, is that Capabilities can be transmitted and shared and possessing a valid Capability implies valid access. By possessing a valid Capability, application code can verify that it is allowed to perform an action. In an RBAC system on the other hand the entire function must assume a role (effectively pretending to be a "user"), and in turn take on the union of all permissions provided by the attached policies. Access is validated independently of the function code (after the code has already executed) by an authorization service and rejected if the policy doesn't allow the action.

In our ECDAC model, an entity is composed of a unique ID, a shape describing any necessary schema, and a set of capabilities defining how data described by this entity can be accessed. This inversion of control -- keeping access control with the data rather than the service -- is made possible by using WebAssembly modules to implement our Entities!

The implementation details are still being worked out, but the central idea is that our data Capability is compiled as an executable WASM module, which is an Entity insofar as it implements the Entity ABI. An Entity is a Capability as a WASM module.

A service defines an Entity as a kind of dependency. This means the service possesses an Entity, which in turn means possessing a Capability. An AssemblyLift guest will interact with Entities through a high-level API, which delegates to the Entity module to verify the Capability, and operate on data in a particular location according to the Entity shape/schema. The Entity module is self-verifying and needs no outside processes to validate the requested action. If we want Entities to perform database operations on behalf of the guest, they will need either an IOmod or WASI environment. Otherwise it will return back to the AssemblyLift host to complete the action (I'm personally torn on what the best choice here is 🙃).

An example Entity definition for an AssemblyLift project might look like the following:

[entity]
name = "user"

[[fields]]
name = "name"
type = "string"
attributes = ["searchable"] # Optional

[[fields]]
name = "email"
type = "string"
attributes = ["unique", "primary"] # Optional

[[actions]]
action = "load"
[actions.cap]
location = "asml/service/service-name/*"
effect = "allow"

[[actions]]
action = "store"
[actions.cap]
location = "dynamodb/region/table-name"
effect = "allow"

In this example we have an Entity representing a user which has defined a name and email field. The entity specifies two actions load and store, each of which have an associated Capability. The load action specifies a location, which is a unique identifier specifying a resource which can perform the action on the Entity (i.e. the Entity says "who can retrieve me?"). Similarly the store action specifies a location indicating where it can be stored (i.e. the Entity says "who can receive me?").

From within our function code, we can interact with an API which abstracts the low-level Entity system implementation.

let user_prototype = Entity::new("user");
match user_prototype.query_all().await {
    ....
}

We see a number of benefits over traditional RBAC/IAM:

Reduced policy complexity: by defining access in terms of database entities, we eliminate the need to maintain unique policies for each function in an application.
Simplified access definition: data-oriented access control decouples resource access from data access, and lets us reduce the surface area of the policy language.
Improved data access transparency: "who can access this data?" is answered by the data's Entity definition.
Consistent entity schema: the shape of an Entity is stored in the module, distributing schema rules to every function which has the Entity.

When is this coming?

Development is in progress now, and the plan as of writing is to get a version of this introduced in v0.4.

Lambda function HTTP authorization with Auth0 and AssemblyLift (WebAssembly + Lambda + API Gateway + Rust)

Dan — Sat, 09 Oct 2021 13:34:06 +0000

Oh, hello! Sorry, I didn't see you there.

Since you're here, let's talk about Lambda function authorization!

By itself, the only way a Lambda function has to reject an invocation is with an IAM policy. This will let you prevent another service from programmatically invoking the function via the AWS SDK. In most cases however it's considered bad practice to directly invoke a function in this way, due to the coupling it can introduce between two services.

A better approach is to invoke the function via HTTP API Gateway, and make the underlying function an implementation detail. In addition to decoupling, with an API Gateway we can now use HTTP authorization schemes such as OAuth to protect our functions!

Auth0 offers a highly-regarded (and easy to use, personally speaking) authentication platform, which we can use to provide an OAuth authorizer for our API. Auth0 offers a free plan supporting up to 7000 users!

In this guide we'll use AssemblyLift to deploy our Lambda function, and define our API and an authorizer. AssemblyLift is an open platform which allows you to quickly develop high-performance serverless applications. Service functions are written in the Rust programming language and compiled to WebAssembly. The AssemblyLift CLI takes care of compiling and deploying your code, as well as building and deploying all the needed infrastructure!

Preparation & assumed knowledge

To follow this guide you will need an AWS account if you do not have one already.

You will also need the Rust toolchain and NPM installed. The Rust toolchain is installed using the rustup interactive installer. The default options during installation should be fine. After installation you will need to install the wasm32-unknown-unknown build target for the Rust toolchain (rustup toolchain install wasm32-unknown-unknown).

Once you have these prerequisites you can install AssemblyLift with cargo install assemblylift-cli. Run asml help to verify the installation.

⚠️NOTE⚠️: if you have previously installed AssemblyLift, please make sure you are on the latest version before you begin! Version 0.3.2 contains bugfixes affecting this guide. 🙂

Assumed knowledge

We are going to write a simple function in Rust which writes an item to a DynamoDB table, so you should be familiar with Rust (our usage of DynamoDB will be very basic). In particular, you should be comfortable with async/await in Rust.

While this guide should be fairly straightforward, we will be using some slightly more advanced features of AssemblyLift. If this is your first time using AssemblyLift, consider taking a look this post on deploying an Eleventy static site with AssemblyLift which is a little simpler.

You should also be familiar with AWS IAM, as we will need to add permissions to our function enabling access to DynamoDB. Optionally if you know some Terraform, we will also demonstrate how you can set these permissions from within the project code. 😀

Project setup

In your favourite projects directory, create a new AssemblyLift application and change to that directory.

$ asml init -n auth-tutorial
$ cd auth-tutorial

Let's start by adding a data service, in which we will have a put function that allows us to add a new item to a DynamoDB table.

Generate the new service & function:

$ asml make service data
$ asml make function data.put

In order for AssemblyLift to recognize the new service, it must be added to the project manifest assemblylift.toml:

[project]
name = "auth0-tutorial"

[services]
data = { name = "data" }

If you like, you can delete the default service from the manifest as well as remove its directory under services/.

Create an Auth0 API

With the base of our AssemblyLift project set up, now would be a good time to create a new API in Auth0. This will provide JWT authorization to our data.put function.

In your Auth0 dashboard, navigate to Applications > APIs and hit Create API.

Leave the signing algorithm as RS256, and give your API a name. The identifier is a unique string which is used within OAuth 2.0 to identify the authorization server which issued a token. It is recommended to use the URL of the endpoint which will be associated with this authorizer, however you are free to use whichever naming scheme you like!

Configure the `data` service

Next let's open up the data service manifest services/data/service.toml, and replace the default configuration with the following:

# data/service.toml
[service]
name = "data"

[api.functions.put]
name = "put"
http = { verb = "PUT", path = "/put" }
authorizer_id = "auth0"

[api.authorizers.auth0]
auth_type = "JWT"

Configure the authorizer

A JWT authorizer at minimum requires audience and issuer parameters. The audience is the identifer you chose for your API in the Auth0 console. The issuer is going to be a URL of the form https://<tenant-id>.<region>.auth0.com. You can find your tenant ID and region on the settings tab of the Auth0 console.

Add the parameters to the auth0 definition:

# data/service.toml
[api.authorizers.auth0]
auth_type = "JWT"
audience = ["your-identifier-here"]
issuer = "https://<tenant-id>.<region>.auth0.com"

Writing data with the `put` function

This will be a relatively simple function which writes some data to a DynamoDB table, as an example of an operation that really ought to be protected by an authorizer 😜

We will need to add a dependency to the DynamoDB IOmod. IOmod dependencies are defined in service.toml:

# data/service.toml
[iomod.dependencies.dynamodb]
coordinates = "akkoro.aws.dynamodb"
version = "0.1.5"

We will also need to add a Cargo dependency to our put function. IOmod dependencies are paired with "guest crates" which provide an API to the IOmod. If you are familiar with the C or C++ programming language, you can think of this as analogous to a header file.

# Cargo.toml
[dependencies]
assemblylift-iomod-dynamodb-guest = "0.1"

Next let's look at the function code. Find put/src/lib.rs and update it with the following:

// data/put/src/lib.rs
extern crate asml_awslambda;

use asml_core::GuestCore;
use asml_awslambda::*;

use assemblylift_iomod_dynamodb_guest::{put_item, structs::*};

handler!(context: LambdaContext<ApiGatewayEvent>, async {
    let user_id = context.event.request_context
                .unwrap()
                .authorizer
                .unwrap()
                .claims
                .unwrap()
                .get("sub")
                .unwrap()
                .to_string();

    let mut input = PutItemInput::default();
    input.table_name = "auth0-tutorial".to_string(); // Change this to the name of your own DynamoDB table!
    input.item = PutItemInputAttributeMap::default();

    let mut pk_value = AttributeValue::default();
    pk_value.s = Some(user_id);
    input.item.insert(AttributeName::from("pk"), pk_value);

    match put_item(input).await {
        Ok(response) => http_ok!(response),
        Err(err) => http_error!(err.to_string()),
    }
});

This function calls the DynamoDB PutItem action, to write the authenticated user's ID to a table called auth0-tutorial with a primary key named pk. The user ID is written as the primary key, as you might if this were a real table tracking user data.

All IOmod calls in Rust return a type implementing Future, giving us support for await syntax! The action is called with the input struct we built, and an HTTP response is returned when the action is resolved.

Function permissions

In order for our function to access DynamoDB, we will have to attach an IAM policy to the function's execution role which allows access.

Unfortunately this is not handled in the AssemblyLift TOML directly at the moment (we are taking our time sorting out how permissions will be modelled in AssemblyLift 🙂). Your options are to find the role in the AWS IAM console and add the policy there, or you can use a feature of AssemblyLift which lets you include your own Terraform code with the code generated by the CLI!

AssemblyLift generates IAM roles for functions named in a particular format; asml-{project name}-{service name}-{function name}.

Your own Terraform can be included with the AssemblyLift project by adding a Terraform module inside a directory named user_tf at the project root (next to assemblylift.toml).

An example of such a module is below:

provider aws {
  region = "us-east-1"
}

data aws_caller_identity current {}
data aws_region current {}

locals {
    account_id = data.aws_caller_identity.current.account_id
    region     = data.aws_region.current.name

    prefix     = "asml-auth0-tutorial"
    table      = "auth0-tutorial"
}

data aws_iam_role data-put {
    name = "${local.prefix}-data-put"
}

data aws_iam_policy_document allow-dynamodb-put {
    statement {
      actions   = ["dynamodb:PutItem"]
      effect    = "Allow"
      resources = ["arn:aws:dynamodb:${local.region}:${local.account_id}:table/${local.table}"]
    }
}

resource aws_iam_policy data-put-policy {
    name   = "${data.aws_iam_role.data-put.name}-dynamodb"
    policy = data.aws_iam_policy_document.allow-dynamodb-put.json
}

resource aws_iam_role_policy_attachment data-put-policy {
    role       = data.aws_iam_role.data-put.name
    policy_arn = aws_iam_policy.data-put-policy.arn
}

Terraform's data lookup feature is used to import the role AssemblyLift created for you. A caveat here is that you must have run asml bind at least once prior to adding this lookup, in order for it to be found! See the next section below on building & deploying!

Build & deploy the `data` service!

This part is quite easy with AssemblyLift! ☺️

Use the cast command to compile your function code, and generate a Terraform plan. Then use the bind command to deploy the serialized code & infra to AWS!

Before running bind, check the output of all the processes executed during cast to verify that the function compiled OK and that the infrastructure changes reported by Terraform aren't unexpected.

$ asml cast
$ asml bind

Testing our function authorizer

The easiest way to test the authorizer is to use the curl commands generated for you in the Auth0 dashboard! Navigate to the test tab of your API in the dashboard. The command you are looking for is of the form:

curl --request GET \
  --url http://path_to_your_api/ \
  --header 'authorization: Bearer <TOKEN>'

Your API endpoint URL can be found in the AWS API Gateway console. This endpoint corresponds to the data service. Our put function is at the path (and verb) we defined in the service definition TOML:

curl --request PUT \
  --url https://<api-id>.execute-api.<region>.amazonaws.com/put \
  --header 'authorization: Bearer <TOKEN>'

Calling the endpoint without the authorization header should return an HTTP Forbidden error. Otherwise it should return HTTP OK with an empty JSON object {}!

Open the DynamoDB web console and verify that the PutItem call worked; there should be an item with your Auth0 user ID!

That's all, folks!

If you have any questions, please don't hesitate to reach out in the comments below or on GitHub!

Deploy an ultra-fast blog in minutes with Eleventy and AssemblyLift (WebAssembly + Lambda + API Gateway + Rust)

Dan — Tue, 28 Sep 2021 20:49:55 +0000

Hello fellow developers! 😊

Let's talk about static site generators for a minute. Static Site Generators (SSGs) are popular these days for lots of applications, not least of all personal blogs. Static sites offer quick response times owing to easy caching by CDNs. They also have the advantage of not needing access to a database for content. Perhaps best of all, they can often be hosted for free using a simple S3 bucket or a SaaS such as Netlify.

An alternative approach is a serverless backend with AWS Lambda and API Gateway. This would allow our "static" file server to do some processing before serving the final HTML if we like. With Lambda, we can control our "server" performance by adjusting the function's memory/CPU allocation. Using API Gateway allows us to monitor and secure our routes (among other things). Lots of benefits while still being cheap, if not free, and high-performance!

Normally what we've described above would be a pretty complicated set up (and a much longer article 😜). Luckily we can use AssemblyLift to do all the heavy work for us!

AssemblyLift is an open platform which allows you to quickly develop high-performance serverless applications. Service functions are written in the Rust programming language and compiled to WebAssembly. The AssemblyLift CLI takes care of compiling and deploying your code, as well as building and deploying all the needed infrastructure!

In this walkthrough we're going to build a blog using Eleventy (11ty), a JavaScript SSG. While I was preparing to write this I started putting together my own blog; the source of which you can take a look at on GitHub if you'd like to see approximately what we'll be building (your's will undoubtedly be prettier than mine).

Preparation & assumed knowledge

To follow this guide you will need an AWS account if you do not have one already.

Once you have these prerequisites you can install AssemblyLift with cargo install assemblylift-cli. Run asml help to verify the installation.

Assumed knowledge

AssemblyLift functions are written in Rust, so it will be helpful to have a working knowledge of Rust. That said if you are only interested in getting your site deployed, you can safely smile and nod at the Rust code and skip ahead 🙃. Eleventy is a JavaScript framework, however the only JS you need is its configuration in .eleventy.js.

Project setup

In your favourite projects directory, create a new AssemblyLift application for your blog and change to that directory.

$ asml init -n my-blog
$ cd my-blog

Before we can configure our Eleventy front-end, we'll need to set up our AssemblyLift backend and create our web service.

Configure the default project

With asml init a new AssemblyLift project is generated with the bare minimum needed to build and deploy an application; a single service containing a single function.

First let's take a look at the application manifest, assemblylift.toml:

# Generated with assemblylift-cli

[project]
name = "my-blog"

[services]
default = { name = "my-service" }

We need a service which will serve our site assets, so we can start by renaming the default service. Update the services table and name the service something relevant:

# assemblylift.toml
[services]
web = { name = "web" }

We will also need to rename the service's directory:

$ mv services/my-service services/web

Next let's open up the web service manifest, services/web/service.toml:

# Generated with assemblylift-cli

[service]
name = "my-service"

[api.functions.my-function]
name = "my-function"

Rename the service to web. Then as with our service in the project manifest, we should rename the default function in the service manifest to something relevant:

# web/service.toml
[service]
name = "web"

[api.functions.server]
name = "server"

And rename the function directory:

$ mv services/web/my-function services/web/server

Finally, we'll need to rename the function inside web/server/Cargo.toml:

# server/Cargo.toml
[package]
name = "server"
version = "0.0.0"
.
.

The server function

Let's take another look inside the service manifest, at the api.functions table.

# web/service.toml
[api.functions.server]
name = "server"

Pretty sparse right? As is, this will deploy a lambda function without any sort of API (though it can still be invoked like any other function via the AWS SDK!). Let's add an HTTP route to our function so we can invoke it from our browser:

# web/service.toml
[api.functions.server]
name = "server"
http = { verb = "GET", path = "/{path+}" }

The http block has a verb and a path. The verb is the HTTP method the function is invoked with (e.g. GET, POST, PUT, etc.). The path (or "route" in some frameworks) is the HTTP path which is mapped to our function. In this case we are using a feature of API Gateway called a proxy path. This is the {path+} variable, which globs the entire path as a single variable called path inside our function code.

The function code

So far we've got our infrastructure defined, but the default function code doesn't do much of anything.

Open up web/server/src/lib.rs, and overwrite it with the following:

// lib.rs
// www/server

extern crate asml_awslambda;

use std::io::Write;

use base64::encode;
use flate2::write::GzEncoder;
use flate2::Compression;
use mime_guess;
use rust_embed::RustEmbed;

use asml_awslambda::*;
use asml_core::GuestCore;

handler!(context: LambdaContext<ApiGatewayEvent>, async {
    let path = context.event.path;
    let path = match path.ends_with("/") {
        true => format!("{}index.html", &path[1..path.len()]),
        false => String::from(&path[1..path.len()]),
    };

    AwsLambdaClient::console_log(format!("Serving {:?}", path.clone()));

    match PublicAssets::get(&path.clone()) {
        Some(asset) => {
            let mut gzip = GzEncoder::new(Vec::new(), Compression::default());
            let mime = Some(
                mime_guess::from_path(path.clone())
                    .first_or_octet_stream()
                    .as_ref()
                    .to_string(),
            );
            let data = asset.data.as_ref();
            gzip.write_all(data).unwrap();
            let body = encode(gzip.finish().unwrap());
            http_ok!(body, mime, true, true) // true, true: we always gzip & encode base64
        }
        None => http_not_found!(path.clone()),
    }
});

#[derive(RustEmbed)]
#[folder = "../../../www/_site"]
struct PublicAssets;

This function uses the rust-embed crate, which embeds the contents of a directory inside the compiled binary. We use this to store the site generated by Eleventy inside the function!

We can then use the path variable passed to the function via the function input (remember our path parameter, {path+}?) to select which resource to return as our function output. A match block is used to detect if we have a path ending in a forward slash, indicating that we should default to serving index.html from that path.

Other crates are imported as well, to handle Gzip & Base 64 encoding and as well as guessing MIME types for our response headers. In all, you should have the following dependencies in your Cargo.toml:

[dependencies]
base64 = "0.13"
direct-executor = "0.3.0"
flate2 = "1"
mime_guess = "2"
rust-embed = "6"
serde = "1"
serde_json = "1"
asml_core = { version = "0.2", package = "assemblylift-core-guest" }
assemblylift_core_io_guest = { version = "0.3", package = "assemblylift-core-io-guest" }
asml_awslambda = { version = "0.3", package = "assemblylift-awslambda-guest" }

Installing Eleventy

As of now our base infrastructure is ready, but compiling our function will fail because our www directory doesn't exist yet! Let's fix that; first by creating the directory in our project root:

$ mkdir www
$ cd www

Create a new package in this directory with npm and then install Eleventy:

$ npm init -y
$ npm install --save-dev @11ty/eleventy

At this point, it will be helpful to take a look at the Eleventy base blog repo on GitHub. I recommend copying .eleventy.js into www and installing these additional packages:

npm install --save-dev @11ty/eleventy-navigation @11ty/eleventy-plugin-rss @11ty/eleventy-plugin-syntaxhighlight luxon markdown-it markdown-it-anchor

From there the structure of your site is really up to you! Look at examples and determine what you like best.

Running npx @11ty/eleventy will generate the static site from any template files it finds in the current directory, and by default write the output to _site/. You will at least need index.* in order to generate an index.html to serve.

A powerful feature of Eleventy is collections. Pages with the same tag are grouped into a collection by the same name, and made available inside our templates. In the base blog repo for example, all posts are placed in a directory and a single JSON file specifies the tags assigned to everything in the directory. Now adding a post to your blog is as easy as adding a new file to your posts directory!

Build & deploy our new site

Once you have something which builds without error in www/_site, you should have everything necessary to build our application and deploy!

AssemblyLift applications are built with the cast command, inside the project root (the directory where assemblylift.toml is found):

$ asml cast

This will compile our web service & server function, and generate a Terraform infrastructure plan. All build artifacts are serialized in the net/ directory. When our Rust function is compiled, our static site assets are bundled with the resulting WebAssembly binary!

To deploy our new service, simply run:

$ asml bind

Testing the web service

If everything in the bind command completed without error, you should now be able to find a new API Gateway endpoint inside the AWS console. The API will be named asml-{projectName}-{serviceName}. Navigate to your API details, where you should find an endpoint listed for the $default stage. Opening this URL in a browser should render your new statically generated blog!

Further enhancements

For production use, you will probably want to create a CloudFront distribution using your web server service as the distribution source. The API Gateway service also supports custom domain names, as the generated APIGW/CloudFront URLs are quite ugly 😁.

If in the future you want to expand your blog with dynamic content, you can create additional services & functions. Note however that if you intend on using CloudFront or other CDN, the web service should only contain the server function and all other functionality should reside in other services. This is because the server function uses a proxy path; no other functions in the service would receive invocations!

In AssemblyLift a new service is created by running:

$ asml make service my-service-name

You can then add a new function to the service with:

$ asml make function my-service-name.my-function-name

Any new functions you make which you don't want to be publicly accessible should attach an authorizer. Take a look at this post on using Auth0 with AssemblyLift for an in-depth guide!

That's all, folks!

If you have any questions, don't hesitate to reach out in the comments below or on GitHub!

For more details on AssemblyLift, please see the official documentation.

Deploy a Jamstack site on AWS Lambda with API Gateway in 10 minutes or less 💨

Dan — Thu, 12 Aug 2021 16:48:53 +0000

Jamstack architecture is all the rage these days, owing to its focus on high performance and scalability. With an emphasis on serving pre-generated content, for some it may feel a little bit like a return to the old days. The underlying infrastructure however is anything but!

This article will walk through building and deploying a simple static site, served using AWS Lambda and API Gateway. To accomplish this we will use AssemblyLift, an open-source platform designed to quickly & easily accomplish such a task.

Getting started

You will need an AWS account if you do not have one already. The small application you will build here fits comfortably in the free tier. 🙂

You will also need the Rust toolchain and NPM installed. In addition you will need to install the wasm32-unknown-unknown build target for the Rust toolchain (rustup toolchain install wasm32-unknown-unknown).

Once you have these prerequisites you can install AssemblyLift with cargo install assemblylift-cli. Run asml help to verify the installation.

Creating a Jamstack project

We will base our application on the AssemblyLift project template available on GitHub. You can clone this, or click the "use this template" button to create a new repo in your account from the template.

The template project includes a simple static site that we will build with Webpack. It also includes an AssemblyLift service called www which implements our server function.

Project structure

Let's take a closer look at the project.

├── README.md
├── assemblylift.toml
├── babel.config.json
├── package-lock.json
├── package.json
├── services
│   └── www
│       ├── server
│       │   ├── Cargo.lock
│       │   ├── Cargo.toml
│       │   └── src
│       │       └── lib.rs
│       └── service.toml
├── web
│   ├── images
│   │   └── AssemblyLift_logo_with_text.png
│   ├── main.js
│   ├── style
│   │   └── main.css
│   └── views
│       └── index.ejs
└── webpack.config.js

The project root contains configuration files for AssemblyLift, Babel, NPM, and Webpack. There is also the AssemblyLift services directory, and a directory called web which we've chosen for our frontend code. Feel free to rename the web directory if you like, just don't forget to update the Webpack config accordingly!

This project has one service www containing one function server (sometimes written www/server or www.server). The server function is a Rust crate containing the function handler.

A closer look

Next let's look at the handler function in www/server. You won't need to change it for this walkthrough, but it will be helpful to understand how it works if you want to expand on it yourself 🙂.

The code

extern crate asml_awslambda;

use std::io::Write;

use base64::encode;
use flate2::write::GzEncoder;
use flate2::Compression;
use mime_guess;
use rust_embed::RustEmbed;

use asml_awslambda::*;
use asml_core::GuestCore;

handler!(context: LambdaContext<ApiGatewayEvent>, async {
    let path = context.event.path;
    let path = match path == "/" {
        true => String::from("index.html"),
        false => String::from(&path[1..path.len()]),
    };

    AwsLambdaClient::console_log(format!("Serving {:?}", path.clone()));

    match PublicAssets::get(&path.clone()) {
        Some(asset) => {
            let mut gzip = GzEncoder::new(Vec::new(), Compression::default());
            let mime = Some(
                mime_guess::from_path(path.clone())
                    .first_or_octet_stream()
                    .as_ref()
                    .to_string(),
            );
            let data = asset.data.as_ref();
            gzip.write_all(data).unwrap();
            let body = encode(gzip.finish().unwrap());
            http_ok!(body, mime, true, true) // true, true: we always gzip & encode base64
        }
        None => http_not_found!(path.clone()),
    }
});

#[derive(RustEmbed)]
#[folder = "../../../dist/"]
struct PublicAssets

How does it work?

Most of the heavy lifting in this function is done by rust-embed, which embeds the contents of dist (our Webpack output directory) in the compiled binary. This allows our static assets to be deployed to Lambda bundled inside the function code, as long as the resulting binary is less than 50MB.

The rest of the function code deals mainly with API Gateway. The path received by the function is matched against the embedded assets; if one is found, it is gzipped and then encoded base64 as required by API Gateway to serve binary content. The content-type header is set using the mime_guess crate, defaulting to application/octet-stream if type cannot be guessed.

Configure, build, deploy

Technically you can deploy the template as-is without any configuration, but you will likely want to at least rename the project. To do this look at assemblylift.toml:

[project]
name = "assemblylift-template-jamstack"

# In production you should configure an S3 bucket & DynamoDB table for terraform state
#[terraform]
#state_bucket_name = "my-tf-state"
#lock_table_name = "my-tf-lock"

[services]
default = { name = "www" }

You can learn more about the AssemblyLift TOML documents on the official docs, however this is where you would set the name of the project (included in the names of the created AWS resources) as well as add more services if you want. There is also a block for configuring Terraform remote state storage, which is highly recommended for use in production!

To build an AssemblyLift application you use the asml cast command; this will compile everything which needs to be deployed and store it immutably in the net directory.

If you haven't built the Webpack project however, you will get an error! Build your web assets first with npm run build. Once they are available in dist, your function code should build no problem.

Finally to deploy our service! AssemblyLift applications are deployed with asml bind. This process delegates to Terraform and will require appropriate AWS credentials to be available.

Testing the server function

Unfortunately you'll have to leave the command line for the next part, as we'll need to access the AWS console to find our new service URL. Ideally in a future release the Asml command line will do more of this for you :).

First navigate to the API Gateway section of the AWS Console. You should see an API by the name you gave above listed here.

Click on the listed API to view more details.

This is where each API stage is listed; AssemblyLift always uses the default stage. The URL is the endpoint used to invoke your API.

Our API defines only one route, /{path+} which is called a proxy path. This instructs API Gateway to take the entire path as a variable called path. Our server function will map the path / to /index.html.

If you click on the URL, you should get a very simple static site with the AssemblyLift logo returned back to you!

Next steps

The first thing you'll want to do of course is replace the template site with your own. You can start by just replacing the content, but you're not obligated to keep using Webpack! This function should be compatible with any framework or site generator that ends with a directory of static assets.

If you paid attention to the function code above, it might have occurred to you that the assets we serve don't have to be completely static. For example we could instead embed a handlebars template or similar & render it with the handlebars crate, allowing us "semi-dynamic" content for lack of a better name.

You may also want to create a CloudFront distribution for your API. I have found performance adequate without a CDN, however it will likely improve with one. Putting a CDN in front of the function should also help hide or smooth out latency due to cold start time, if the function is not invoked regularly. CloudFront is not expensive, however whether it is worth it will be up to you 🙂.

Learn More

The best way to learn about AssemblyLift right now is by visiting the official documentation.

You can also reach out to us on Twitter, via Element at #assemblylift:matrix.org, or in the comments below!

AssemblyLift v0.3 Released

Dan — Wed, 04 Aug 2021 16:42:45 +0000

After several months in development, AssemblyLift v0.3 has been released and made available via Cargo! 🥳

If you've never heard of it or just need a reminder, AssemblyLift is an open platform for developing cloud-native applications. At its core are the AssemblyLift Runtime and the AssemblyLift CLI. The runtime provides an environment for executing compiled WebAssembly (WASM) on infrastructure such as AWS Lambda, and is one of the more unique aspects of the platform which we'll dig into more below.

If you'd like to install the CLI and explore the things being discussed in this post, get started by running

cargo install assemblylift-cli

Note the CLI is currently supported on Linux and macOS only.

State of the World

There's an adage I can't remember the origin of, which states that something must be built three times in order to get it right. On the first pass you are learning what you wish to build. On the second, you are learning how to build it. Then on the third, you build it for real.

It's never quite that simple of course (the road to 1.0 is long), but much of the work that has gone into the v0.3 release is "beneath the surface", making this a rather foundational release so to speak. Lots of key areas were reworked or completely rewritten based on the lessons of v0.1 & v0.2, which make implementing new features easier now and in the future.

That said, there are also many user-facing improvements which make this the most "complete" offering of AssemblyLift to date!

Below we'll highlight the major changes between v0.2 and v0.3, and walk through each in more detail in the following sections. Towards the end, I'll discuss a little more about where the project is headed next. 🙂

Change List

Support multiple infrastructure providers for services & functions
- Experimental Lambda runtime based on Alpine Linux
New TOML->HCL transpiler pipeline
Support for arbitrarily large data buffers which can be paged into WASM memory (function input and IOmod responses)
Deployment of & support for the IO Module Registry
Bump critical dependencies to latest versions! In particular, Tokio 1.x is now used throughout the Asml crate ecosystem.

Key Changes

Infrastructure Providers

I'll be honest; this is the change which took the most work and is easily the least visible 😂

But! That doesn't mean it wasn't important.

While AWS Lambda is still the de facto provider, a goal of the project has always been to support deployment to a variety of kinds of infrastructure.

To accomplish this, most of the code which translated your AssemblyLift TOML into Terraform HCL needed to be thrown away and rewritten. This resulted in the new transpiler mentioned above!

The new provider implementation was tested with an experimental provider for Lambda based on Alpine Linux. The original purpose for this provider was to support faster, LLVM-compiled WASM binaries which are incompatible with the default Lambda OS. Unfortunately, despite the tiny image size I discovered that cold-start time a for Docker-based Lambda is much too high to be worth it.

Buffers & Memory Management

A well-published feature of WebAssembly is its memory model, which is a sandboxed linear memory. This essentially means that WASM code has a large array of bytes to work with inside the sandbox, but there are no pointers or other means of escaping elsewhere. A side-effect of this is that to get new data into WebAssembly from the host, the host must write into a known region of the linear memory.

In previous versions, AssemblyLift did not buffer data on the host before passing it to the WASM guest. If the data could not all fit in the defined region within the guest, tough luck!

Clearly that's no good. So there are now host-side buffers which can reach arbitrary size, and then be paged into a location in WASM via ABI calls as needed. This change shows up both on function input, as well as in IOmod memory management as responses are buffered and handled by the guest.

IO Module Registry

Another biggin' that has been in-progress for an age.

IO Modules or IOmods are a unique feature of AssemblyLift which are used to provide the WASM guest with access to outside functionality.

As of v0.3 the CLI can fetch IOmod dependencies from the new public IOmod Registry, while IOmods themselves are now shipped in Zip-archived packages. As of writing, the modules available on the registry are limited, but more are planned (including modules for all remaining AWS services).

Upcoming Improvements

The next point release will focus on improving quality & error handling in a few areas of the code, as well as improving the DX* around the API Gateway/HTTP macros. At the moment for example, the http_ok! macro expects your response to be JSON. This suits API applications fine, but we would like to be more flexible and to support returning HTML from our functions. This opens us up to full-stack development with AssemblyLift in the future! 😎

In the longer term, we'll be expanding what can be managed by AssemblyLift directly. For example function permissions and/or IAM policies can not be set in the service TOML at the moment, which is a major usability issue that should be addressed.

We'll also be evaluating additional providers to support in addition to AWS Lambda.

* Developer eXperience

Learn More

The best way to learn about AssemblyLift right now is by visiting the official documentation.

You can also reach out to us on Twitter, via Element at #assemblylift:matrix.org, or in the comments below!

AssemblyLift v0.2.9: BIG performance boost + new features! 🚀

Dan — Wed, 10 Feb 2021 04:35:07 +0000

So many updates, so little time!

As usual I've been neglecting to write stuff down, but I've still been busy. :)

AssemblyLift has been quietly updated several times recently (mostly in December), iteratively adding features that I needed while building the IOmod registry infrastructure.

The most recent update, 0.2.9, was just published the other day and includes an upgrade to Wasmer 1.0 inside the AssemblyLift Lambda runtime. The latest build of Wasmer gives us significant gains, so I thought now would be a good time to document all the recent changes! 🎉

Changes for Rust guest code

Guest-facing changes were relatively minor, but offer a lot more flexibility on function input.

The only breaking change, introduced via 0.2.8, is a change which adds a type parameter to to LambdaContext. This is the type of the incoming lambda event.

For a function invoked via HTTP for example:

handler!(context: LambdaContext<ApiGatewayEvent>, async {...})

The ApiGatewayEvent struct itself also now deserializes the request_context field, which gives access to authorization information (such as user ID), among other things.

For example:

// "sub" is the UUID of the authorized user
let user_id = event.request_context
                .unwrap()
                .authorizer
                .unwrap()
                .claims
                .unwrap()
                .get("sub")
                .unwrap()
                .to_string();

Changes to service & function definitions

Service TOML definitions now support specifying authorizers by ID for each function. These map to API Gateway authorizers; currently only IAM and JWT are supported. See the pull request for examples.

The fields timeout and size are now recognized optional parameters for function definition. Timeout refers to the function timeout in seconds; the duration after which the function will terminate. Size specifies the function size in megabytes, defaulting to 1024Mb.

Finally, it is now possible to include custom Terraform code in your AssemblyLift app. This code is placed in a directory which must be called user_tf. This directory will be interpreted as a Terraform module and injected in the HCL code which is generated by asml during cast.

Changes to core & runtime

As mentioned, the big news here is the upgrade to Wasmer 1.0. Wasmer is the WebAssembly runtime AssemblyLift uses internally to execute compiled WASM code.

Internally there were many enhancements & fixes to the AssemblyLift runtime, which includes the removal of nearly all Rust code marked unsafe! In particular the ABI code was simplified a bunch, with 447 additions and 848 deletions! 🙂

Pre-compiled WASM

Previously, WASM modules were loaded & compiled to native binary by Cranelift during Lambda initialization. For simple modules this wasn't so bad, but for real-world use the cold-start times (i.e. the time to start the Lambda runtime) were high -- around 1 second in my usage.

Wasmer 1.0 supports cross-compiling WASM to native binary and serializing the result, to be loaded later by the runtime. We use this in AssemblyLift, by adding this compilation step to asml cast. After compiling Rust to WASM, asml will invoke Cranelift and compile the WASM bytes to Amazon Linux-compatible binary. This brings our cold-start times down to around 200ms! I should disclaim though that this is an anecdotal benchmark; I'll need time to do something more formal.

Execution speed appears to be faster overall, though I will need to test this further as well. FWIW I can say that a function I have, which performs a batch-write to DynamoDB and then invokes another service function to perform a PUT to S3, has a warm execution time which is usually sub-300ms.

It is also possible, but not yet supported, to use LLVM in place of Cranelift. This should result even faster execution speed once we get it implemented!

Getting Started

If you're coming across AssemblyLift for the first time, you can install it via cargo with:

cargo install assemblylift-cli

Check out the landing page for links to resources! Unfortunately, said resources are a tad scarce at the moment, but I assure you it's being worked on. 🙃

Thanks for looking!

Build a "todo list" backend with AssemblyLift 🚀🔒

Dan — Sat, 24 Oct 2020 14:43:38 +0000

I don't think it's possible to publish a new framework without demonstrating the classic todo app with it. Definitely frowned upon. Maybe even illegal (check your local laws).

Fermenting AssemblyLift to the point that it could run the "hello world" of web & cloud frameworks was the first usability milestone I wanted to hit, and here we are! 🎉

This post will introduce some AssemblyLift concepts, and walk you through writing and deploying a backend for a simple todo list app.

Getting Started

The AssemblyLift CLI is available via Cargo with cargo install assemblylift-cli. At least Rust 1.39 is required for async support, however I haven't confirmed if there are any other features used that would require a higher version. For reference, I've been working with 1.45.

As of writing, the CLI is compatible with macOS and Linux.

You will also need an Amazon AWS account, as AssemblyLift deploys to AWS Lambda & API Gateway (with support for other clouds planned!). Additionally, this guide uses AWS DynamoDB as a database. If you don't have an account already, note that each of these services offer a free tier which should be plenty to experiment with 🙂.

⚠️
Please reach out if these services are not offered in your region! I can’t do much short-term, but I’d like to be aware of blind spots :)

⚠️
For this article I am assuming some familiarity with both AWS and Rust.

Creating a New Project

The AssemblyLift CLI installs itself as asml. Run asml help to see a list of commands, as well as the version. This guide is current for version 0.2.5 (but I'll try to keep it updated :)).

Create your project by running asml init -n todo. This will create a project named todo in ./todo. Then cd into the directory, as the remaining commands should be run from the project root (the location of assemblylift.toml).

The CLI should have generated a basic project skeleton including a default service, itself including a default function.

🔎
A project in the AssemblyLift sense is essentially a collection of services. What that collection represents is up to you. For example a project may be a collection of services consumed by an application (like our todo app), or a group of related services that are consumed by other services and/or apps.

Let's take a look first at assemblylift.toml. The one generated for you should like something like this:

[project]
name = "examples"
version = "0.1.0"

[services]
default = { name = "my-service" }

At the moment this configuration file defines only some brief info about the project, and the services it contains. It may be expanded with options for project-wide configuration in future releases.

Let's update the generated values with something specific to our todo project.

[project]
name = "todo-app"
version = "0.1.0"

[services]
default = { name = "todo-service" }

You will also need to rename the service's directory via mv ./services/my-service ./services/todo-service.

Your First Service

Next, let's open up the service.toml file for todo-service.

[service]
name = "my-service"

[api]
name = "my-service-api"

[api.functions.my-function]
name = "my-function"
handler_name = "handler"

Start by editing the names to match the ones we gave in assemblylift.toml above. We'll also update the default function name to the first function we'll demonstrate here, the "create todo" function. Like with the service, you should rename the function's directory with mv ./services/todo-service/my-function/ ./services/todo-service/create/.

[service]
name = "todo-service"

[api]
name = "todo-service-api"

[api.functions.create]
name = "create"
handler_name = "handler"

The handler_name field should generally be left alone (and will probably be elided in a future release); you should only change it if you know what you're doing! 🤓

🔎
A service at its core is a collection of functions. A service may optionally declare an HTTP API, mapping (verb,path) pairs to functions. Services are required to have at least one function (otherwise there's not much point, right? 😆).

Since we're here, let's also define the HTTP API for our create function.

[api.functions.create]
name = "create"
handler_name = "handler"
http = { verb = "POST", path = "/todos" }

With http defined, AssemblyLift will create an HTTP API for the service with routes defined by the functions' verb (method) and path. Take a look at the API Gateway docs for information on using path parameters.

Adding Dependencies

Our services are going to depend on IO modules (IOmods). IOmods are similar to packages in other environments such as Node.js. However in AssemblyLift, IOmods are implemented as binary "plugins" and they are how our functions communicate with the outside world.

Add the following blocks to your service.toml:

[iomod.dependencies.aws-dynamodb]
version = "0.1.0"
type = "file"
from = "/your/path/to/iomod/akkoro-aws-dynamodb"

[iomod.dependencies.std-crypto]
version = "0.1.0"
type = "file"
from = "/your/path/to/iomod/akkoro-std-crypto"

Currently AssemblyLift only supports importing local files. Hopefully in the near future we'll have some registry infrastructure for IOmods. In the meantime, we have a manual step to fetch the latest build of the IOmod standard library. You can download a zip from GitHub here, and extract the binaries to a directory somewhere on your local system. At the moment the stdlib contains a whopping 2 (!!) modules, for DynamoDB and basic crypto respectively.

Your First Function

Finally, some Rust code! 🦀 Let's look at the lib.rs that was generated for us.

extern crate asml_awslambda;

use asml_core::GuestCore;
use asml_awslambda::{*, AwsLambdaClient, LambdaContext};

handler!(context: LambdaContext, async {
    let event = context.event;
    AwsLambdaClient::console_log(format!("Read event: {:?}", event));

    AwsLambdaClient::success("OK".to_string());
});

This is the bare minimum you need to have a complete AssemlyLift function written in Rust -- essentially just enough to prove that events are read in and status is written out correctly. The handler! macro provides the entry point to our function, and hides the boilerplate code necessary to bootstrap the function.

Let's make it more interesting. We'll start by rewriting this for HTTP invocation:

extern crate asml_awslambda;

use asml_core::GuestCore;
use asml_awslambda::*;

handler!(context: LambdaContext, async {
    let event: ApiGatewayEvent = context.event;
    match event.body {
        Some(content) => {
            // TODO
        }
        None => {
            http_error!(String::from("missing request payload"));
        }
    }
});

The http_error! macro is a helper which wraps AwsLambdaClient::success and returns an HTTP 520 with a JSON response indicating a function error.

Our create function is going to use the DynamoDB PutItem call to store a new todo item. Each item will use a UUID for its primary key, for which we'll use the UUID v4 call from the crypto IOmod.

First we'll need to import a few crates; add the following to the Cargo dependencies:

serde = "1.0.53"
asml_iomod_dynamodb = { version = "0.1.2", package = "assemblylift-iomod-dynamodb-guest" }
asml_iomod_crypto = { version = "0.1.1", package = "assemblylift-iomod-crypto-guest" }

We'll need serde to serialize/deserialize our request & response JSON. The other two are the "guest" crates for each IOmod we depend on.

Next, let's add simple request & response structs to lib.rs.

use serde::{Serialize, Deserialize};
.
.
.
#[derive(Serialize, Deserialize)]
struct CreateTodoRequest {
    pub body: String,
}

#[derive(Serialize, Deserialize)]
struct CreateTodoResponse{
    pub uuid: String,
}

The request contains only the body field, which will store the text of the todo item. The item's ID and timestamp will be generated on the function.

Let's move on to the body of our function. We'll start by deserializing the request body to our CreateTodoRequest struct.

.
.
Some(content) => {
    let content: CreateTodoRequest = serde_json::from_str(&content).unwrap();
}
.
.

Easy enough. Next, lets use one of our IOmods! Import the uuid4 call from crypto, and use it to generate a UUID for the item.

use asml_iomod_crypto::uuid4;
.
.
Some(content) => {
    let content: CreateTodoRequest = serde_json::from_str(&content).unwrap();

    let uuid = uuid4(()).await;
}
.
.

IOmod calls always take a single argument for the call's input, and always return a Future. The uuid4 call doesn't actually require any input, so we pass () for the input argument.

We'll use this value as the primary key for the todo item. We'll construct this item as part of the PutItemInput struct that our DynamoDB call will take as input.

use asml_core_io::get_time;
use asml_iomod_crypto::uuid4;
.
.
Some(content) => {
    let content: CreateTodoRequest = serde_json::from_str(&content).unwrap();

    let uuid = uuid4(()).await;

    let mut input: structs::PutItemInput = Default::default();
    input.table_name = String::from("todo-example");
    input.item = Default::default();
    input.item.insert(String::from("uuid"), val!(S => uuid));
    input.item.insert(String::from("timestamp"), val!(N => get_time()));
    input.item.insert(String::from("body"), val!(S => content.body));
}
.
.

The timestamp is generated by get_time(), which returns the system time as the duration in seconds since the "Unix epoch". The get_time function is an AssemblyLift ABI call, and is always available.

The val! macro provides a shorthand for writing the DynamoDB value JSON, and is borrowed from the rusoto_dynamodb crate (as are the structs 🙃).

The last thing we need to add is a call to DynamoDB's PutItem, and a response from our a function.

use asml_iomod_crypto::uuid4;
use asml_iomod_dynamodb::{structs, structs::AttributeValue, *};
.
.
Some(content) => {
    let content: CreateTodoRequest = serde_json::from_str(&content).unwrap();

    let uuid = uuid4(()).await;

    let mut input: structs::PutItemInput = Default::default();
    input.table_name = String::from("todo-example");
    input.item = Default::default();
    input.item.insert(String::from("uuid"), val!(S => uuid));
    input.item.insert(String::from("timestamp"), val!(N => get_time()));
    input.item.insert(String::from("body"), val!(S => content.body));

    match put_item(input).await {
        Ok(_) => 
            let response = CreateTodoResponse { uuid };
            http_ok!(response);
        }
        Err(why) => http_error!(why.to_string())
    }
}
.
.

Here we've introduced the http_ok! macro, which returns our response as an HTTP 200. If the call to put_item fails for some reason, we call our error macro again.

Putting it all together, the code for our create function should like like this!

extern crate asml_awslambda;

use serde::{Serialize, Deserialize};

use asml_core::GuestCore;
use asml_awslambda::*;

use asml_iomod_crypto::uuid4;
use asml_iomod_dynamodb::{structs, structs::AttributeValue, *};

handler!(context: LambdaContext, async {
    let event: ApiGatewayEvent = context.event;
    match event.body {
        Some(content) => {
            let content: CreateTodoRequest = serde_json::from_str(&content).unwrap();

            let uuid = uuid4(()).await;

            let mut input: structs::PutItemInput = Default::default();
            input.table_name = String::from("todo-example");
            input.item = Default::default();
            input.item.insert(String::from("uuid"), val!(S => uuid));
            input.item.insert(String::from("timestamp"), val!(N => get_time()));
            input.item.insert(String::from("body"), val!(S => content.body));

            match put_item(input).await {
                Ok(_) => {
                    let response = CreateTodoResponse { uuid };
                    http_ok!(response);
                }
                Err(why) => http_error!(why.to_string())
            }
        }

        None => {
            http_error!(String::from("missing request payload"));
        }
    }
});

#[derive(Serialize, Deserialize)]
struct CreateTodoRequest {
    pub body: String,
}

#[derive(Serialize, Deserialize)]
struct CreateTodoResponse{
    pub uuid: String,
}

Building the Project

Before getting too far, now would be a good time to try building our project to make sure everything is working.

The CLI command you'll use for this is asml cast. The cast command compiles all function source & generates an infrastructure plan with Terraform. All build artifacts are written to the net directory, which is the frozen (and deployable) representation of your project.

You may need to scroll the output a little to verify there were no errors -- the Terraform plan will still run even if function compilation fails (we'll fix this soon :)).

Adding Additional Functions

I'm going to leave the implementation of the remaning functions as an exercise to the reader. This is already long, plus there's a complete example on Github if you'd like the full source.

You should be aware of the asml make command while you do this, which will let you generate & add a new service or function to your existing project.

Running asml make service <service-name> will, as you probably guessed, create a new service. This uses the same template as the init command to generate the stub.

What about adding a function to this service, that we already have? You do this by running asml make function <service-name>.<function-name>.

For example, if you want to add a delete function to our todo service you might run asml make function todo.delete.

For now you will still need to update your .toml files by hand, but I would like make to take care of that for you in the future as well in a future update 🙂.

Deployment

The last command you're going to need is asml bind. This command will take an existing net structure and bind it to the backend (i.e. AWS Lambda).

If you aren't getting errors during cast, there shouldn't be any issues running bind successfully. If everything goes smoothly (or even if not), you should receive output from the underlying Terraform process.

If it worked, head over to your AWS console and take a look for your API in API Gateway. You should be able to grab the endpoint URL it generated and start testing!

🚧
AssemblyLift doesn't yet provide a built-in means of adding IAM policies to the Roles generated for each Lambda. In the meantime you will have to attach policies manually (such as for DynamoDB access). A unique role has been generated by asml for each function in your service, and should be easily identified by name in the console.

fin

That's all! Please please please reach out here or on Github if you have any issues! Things can't be fixed if I don't know that they're broken :).

I'll do my best to keep this guide updated as changes & fixes are introduced. We'll keep a changelog here when the time comes.

Happy coding!

AssemblyLift v0.2 preview: RPC-based IO modules

Dan — Tue, 15 Sep 2020 15:15:26 +0000

I feel like it's been a looooong minute since I was last able to give an update on AssemblyLift. Real life is busy, summer heat makes me sluggish, and I finally took a vacation!

Still, in my spare time I've managed to finish core work around the next major revision of AssemblyLift. It took almost exactly twice as long as I had expected & planned, but I think it was worth the extra effort.

I spent a month thinking I was writing a plugin system, but it turned out to be fairly brittle and gave me trouble with respect to memory safety. So I spent another month re-working it into a solution based on Cap'n Proto RPC, and so far it seems to have paid off!

What are we talking about, anyway?

AssemblyLift is a framework I've been developing, which provides code & tooling for building serverless cloud applications running on services like AWS Lambda. AssemblyLift apps are written in Rust (more client languages to come!) and are compiled to WebAssembly (WASM).

Running serverless functions as WebAssembly comes with several benefits:

WebAssembly modules are inherently isolated; they run inside their own memory space, and are by default unable to open sockets or access the underlying filesystem.
They are often faster than their counterparts written in JavaScript or Python, both common language choices for Lambda.
They can be much lighter in weight, in terms of memory footprint and size of deployment package.
Several languages compile to WebAssembly, offering developer choice on a common runtime environment.

On its own, that first point might look like a problem -- we're probably going to need to communicate with the outside world at some point. Luckily, WebAssembly allows the implementer to provide its own system ABI to the module code. AssemblyLift provides its own ABI, which facilitates running our modules in places like AWS Lambda.

Rather than trying to provide low-level APIs to the module (such as POSIX sockets for example), AssemblyLift provides a standard interface for registering plugin-like modules which the WASM module may access. Tasks which require network or storage access are implemented in these plugin modules, which the WASM module communicates with via the runtime host.

In the AssemblyLift framework, these plugin modules are referred to as IO modules (which I often abbreviate to IOmod). The systems' design & implementation were inspired by Haskell's approach to IO, hence the name 🙂.

In the v0.1 series of AssemblyLift, the IOmod portion of things is roughly present however the (currently lone) DynamoDB module is compiled statically into the rest of the host binary (essentially as a proof-of-concept). My goal was to allow IOmods to be distributed as packages, which naturally means that they must be able to be loaded (or not) at run-time on a per-function basis.

Neat-O! So how does it all work? What does it look like?

At a high level, it looks a little bit like the following diagram; I did my best to understand & stick to the C4 model.

Obviously we're glossing over some implementation details, but the basic idea really is as simple as it looks. To dive a little deeper, let's look at an AssemblyLift example and walk through how a simple network call is made 🙂.

extern crate asml_awslambda;

use direct_executor;
use asml_core::GuestCore;
use asml_awslambda::{*, AwsLambdaClient, LambdaContext};

use asml_iomod_dynamodb::{structs, list_tables};

handler!(context: LambdaContext, async {
    let input: structs::ListTablesInput = Default::default();
    let response = list_tables(input).await; // this is our IOmod call

    AwsLambdaClient::success(format!("Got response {:?}", response));
});

The above example uses a DynamoDB IOmod to call ListTables. This method is implemented by list_tables, which takes a single input argument.

Every IOmod call has this structure (ie output_struct = call(input_struct)), and calls are always asynchronous. The list_tables function comes from an IOmod "guest crate" -- a Rust crate providing the WASM-compatible guest interface to the IO module.

When calling list_tables, the guest calls the AssemblyLift ABI to invoke the method on its behalf. The AssemblyLift runtime in turn locates and invokes the method via RPC in the IOmod host process.

Packaging & Deployment

In the current implementation, IOmods are defined per-service in an AssemblyLift app. When deployed to AWS Lambda, each service is backed by a Lambda Layer which contains all of the IOmod binaries required by the service's functions. When the AssemblyLift runtime starts, it spawns all binaries that have been layered into the environment.

When will it be ready?

I'm aiming for early October, by the time I clean everything up and do more testing. This IOmod stuff also doesn't include everything I had planned for v0.2.x, but I may push everything else out to the next major release. We'll see -- stay tuned! 😊

Update Nov 8 2020: IOmods are now available in the v0.2.x line of AssemblyLift

Why the Rust module system might be hard to understand

Dan — Wed, 22 Jul 2020 19:37:15 +0000

The module system provided by Rust is, by most accounts, kind of weird.

But so is Rust in many ways, so it kind of makes sense. However it seems like a side-effect is that some concepts, like the module system, are particularly tricky to figure out.

There was recently a blog post linked on /r/rust, which I think does a really great job of explaining how it works (if you landed here looking for a tutorial, let me send you there instead!).

The conversation on Reddit got me thinking about why it's difficult to understand. People seem to have different opinions on it, and I think a lot of that might have to do with how much everyone's previous experience can vary when coming into the Rust ecosystem.

The Effect of Past Experiences

First of all, I'm going to make an assumption: that the vast majority of people coming into Rust right now, are not doing so with it as their first programming language. They most likely already work in another language, given that Rust is still relatively young. Rust is also, I think, rarely recommended as something one should try learning first.

This assumption is important, because that's what my first point is kind of based on.

No matter what you're learning, you are using your past experiences and accumulated knowledge to help you understand a concept, whether it's something completely novel or a "next step" of something else.

In the psychological sense, priming is "a phenomenon whereby exposure to one stimulus influences a response to a subsequent stimulus, without conscious guidance or intention".

For Rust, I see see at least two potential priming vectors.

The first, is the word "module" itself. In software development, "module" is a very overloaded term. That often means that one of the first things you have to do (mentally) when reading the docs, is disambiguate the word we use to describe a seriously central concept of Rust. Asking "what is a module?" is the kind of question that gets closed on Stack Overflow for being impossible to answer definitively.

This is a kind of example of priming (if you are a native English speaker; more on that in a second), except that you have been primed with what is essentially mush in my opinion. You know we're talking about a collection of code that is separate from the other modules (hopefully), and from there it's just a vague memory of every other kind of module you've ever encountered. Plus any other IRL-kinds of modules you can think of, even if only subconsciously.

I have no idea what it is like coming across terms like module, with English as a second language. I expect the effect is the same, since it's unlikely your last framework or language used anything other than the English word to describe the idea. And I'm also assuming priming works the same way for a "symbol" regardless of what language it's written in. Let me know if you can shed more light on this part!

As for the second vector, the Reddit discussion I linked also indicated that people didn't necessarily expect Rust's module system to be independent of the filesystem. In part, this was based on their previous experience in other languages. Sound familiar? 🙂

The Effect of Novel Jargon

In addition to using overly ambiguous language adopted from the aether of software engineering, Rust also makes use of its own novel or invented jargon to express ideas. This isn't a criticism of Rust in isolation -- a lot of (most?) software engineering terms are completely made up.

The novel jargon I'm talking about here, is the use of the word crate to describe a Rust package. Or is a crate a module, and the crate in a package? Who knows, unless you can make it through the extra cognitive effort I described earlier to parse the official Rust book. That, or you hope someone writes a well-worded tutorial.

I'm assuming they're called "crates" as a cutesy name derived from Cargo's name, or perhaps it's the reverse.

In either case, if you speak English you think of a crate as something that holds stuff in order to ship it (my internal monologue: "ohhhhh, now I get it!"). And they are managed with what is de facto known as a package manager, called Cargo. Ok, so far so good, as long as crate is a word that already means something to you. And identifying Cargo as a package manager was probably easy.

On packages and crates, the Rust book states:

A crate is a binary or library. The crate root is a source file that the Rust compiler starts from and makes up the root module of your crate (...). A package is one or more crates that provide a set of functionality.

Okay...

Let's look at just the first bit:

A crate is a binary or library. The crate root is a source file

On my first read (or more, maybe), I don't see the word root after the second use of crate. I'm still processing what I was just told a crate is, and then I take in another "a crate is" sentence. My brain immediately aborts. I've got two things a crate is, and none of them are what I expected -- a crate is a binary or library? I thought we installed crates with a package manager...making them packages?

A little further:

that the Rust compiler starts from and makes up the root module of your crate

This is the first place that the word module is used in this section, and it's got the modifier root. But I still don't know what an ordinary module is in this context yet, or a what a crate really is for that matter.

And finally:

A package is one or more crates that provide a set of functionality

Now we learn that a package is really multiple crates. That still doesn't fit what we were primed with, and the explanation hasn't made anything clearer.

I also have to point out how much work the words "a set of functionality" are doing. Speaking of ambiguity!

Altogether, there is quite of bit of jargon thrown at you "just" to learn how things are structured. And a lot of that jargon is pretty ambiguous.

The Solution Space

This isn't meant to be a roast of the Rust book authors. Technical writing is hard (and woefully undervalued as a profession, but that's another post). Overall the Rust book is extremely well written, and it's been useful nearly every day I've worked with Rust.

My overall point, is that I think a lot more care could be taken about the words and phrases that are selected to describe abstract concepts. Maybe not just inside the Rust community, but industry-wide. It may make communicating with each other easier!

Rust is difficult, so I think describing it is going to be more difficult in many cases. Unfortunately, I believe this may ultimately be a case of something requiring a bit more effort (than is maybe even possible right now). Either taking more time with documentation before it's released or dedicating more resources to it. Which is obviously tough, being a (mostly?) volunteer-run project.

The End

If you're still here, thank you! If you have any corrections, please reach out!

Credits

Cover Photo by Brunno Tozzo on Unsplash

Forem: Dan

A full-stack serverless application with AssemblyLift and Next.js

Deep dive

Services & Functions

HTTP API

Domain name mapping

IO Modules

Next Frontend

Deploy your own!

You will need...

Reach out!

AssemblyLift alpha latest: easy API Gateway for Kubernetes functions, Ruby language support 💎

Revised TOML manifest spec

Why?

Ruby language support

Why?

Support for Gloo Edge API Gateway

Why?

A short demonstration

Prereqs

Create a Ruby function

Set provider and configure container registry

Write some Ruby!

Build, deploy, test!

Reach out!

AssemblyLift v0.4.0-alpha released with Kubernetes support, WASI (WebAssembly on K8s)

Installing

Deploy a function to Kubernetes

What's next?

What's next for AssemblyLift? Data-oriented cloud dev through WebAssembly and capability-based security

Entity-Capability Data Access Control

When is this coming?

Further reading

Lambda function HTTP authorization with Auth0 and AssemblyLift (WebAssembly + Lambda + API Gateway + Rust)

Preparation & assumed knowledge

Assumed knowledge

Project setup

Create an Auth0 API

Configure the data service

Configure the authorizer

Writing data with the put function

Function permissions

Build & deploy the data service!

Testing our function authorizer

That's all, folks!

Deploy an ultra-fast blog in minutes with Eleventy and AssemblyLift (WebAssembly + Lambda + API Gateway + Rust)

Preparation & assumed knowledge

Assumed knowledge

Project setup

Configure the default project

The server function

The function code

Installing Eleventy

Build & deploy our new site

Testing the web service

Further enhancements

That's all, folks!

Deploy a Jamstack site on AWS Lambda with API Gateway in 10 minutes or less 💨

Getting started

Creating a Jamstack project

Project structure

A closer look

The code

How does it work?

Configure, build, deploy

Testing the server function

Next steps

Learn More

AssemblyLift v0.3 Released

State of the World

Change List

Key Changes

Infrastructure Providers

Buffers & Memory Management

IO Module Registry

Upcoming Improvements

Learn More

AssemblyLift v0.2.9: BIG performance boost + new features! 🚀

Changes for Rust guest code

Changes to service & function definitions

Configure the `data` service

Writing data with the `put` function

Build & deploy the `data` service!