Forem: Mike Blydenburgh

Rust + Lambda using CDK & Github Actions (Part 1)

Mike Blydenburgh — Mon, 07 Mar 2022 14:40:19 +0000

I have recently been on a Rust kick trying to see why it is consistently ranked as a most loved language and seeing what makes it tick.

After going through the book, it was time to try to apply what I've learned to something I do at work all the time - deploy code to a Lambda. Standing up the project myself made me realize that I had to reference a number of different sources to get things working correctly and there was not a good pattern to follow to automatically deploy via CDK so I wanted to share what I learned along the way!

I am by no means a Rust expert and am not claiming that this is the most proficient way to set things up, but hopefully this can help someone else on their Rust journey in the cloud!

Overview

What we're going to build is a simple lambda function that receives a request via API Gateway and performs an operation on a DynamoDB table. GitHub Actions will be used as a CI/CD pipeline to deploy a CDK stack to AWS.

Since AWS does not currently provide an official Lambda runtime for Rust, we will configure a custom runtime for Amazon Linux 2.

Part 1 will go over the Rust code, and in Part 2 we'll deploy using GitHub Actions and CDK!.

Generating Project & Setting Dependencies

To get generate a new project we'll call rust_lambda, run cargo new rust_lambda.

The following dependencies (crates) will be needed in Cargo.toml:

[package]
name = "rust_lambda"
version = "0.1.0"
authors = ["Your name"]
edition = "2021"

[dependencies]
aws-config = "0.6.0"
aws-sdk-dynamodb = "0.6.0"
tokio = { version = "1", features = ["full"] }
lambda_runtime = "0.4.1"
lambda_http = "0.4.1"
serde = "1.0.136"
serde_derive = "1.0.136"
serde_json = "1.0.78"
log = "0.4.0"
uuid = { version = "0.8.2", features = ["v4"]}

Let's quickly go over how these crates will be used:

aws-config: used to create an AWS environment config
aws-sdk-dynamodb: used to define a Dynamo client and perform operations on a table
tokio: Rust async network runtime that provides building blocks to build web apps
lambda_runtime: provides methods to execute a function as a lambda handler
lambda_http: provides additional handler and request definitions used in create an http invoked lambda
serde: Rust serialization/deserialization library
serde_derive: provides the derive macro that is used to implement various traits on specific struct definitions
serde_json: provides methods to handle json
log: Rust logging library that provides various additional logging macros
uuid: generates a random uuid

Data Model

Overview

Being getting into the code it would be useful to get an idea of how we're going to be receiving requests and saving items into our table.

For our request, let's assume we are going to get a POST request with the following JSON payload:

{
  "firstName": "firstName",
  "lastName": "lastName"
}

The DynamoDB table configuration will be pretty simple:

PartitionKey(userId): String
SortKey(modelTypeAndId): String
FirstName: String
LastName: String

Setting up the partition and sort keys like this enable a relational data model of many different item types on a single dynamo table (Posts that are owned by Users for example).

Creating a Model Module

In order to keep the struct definitions separate, a models folder can be created, and inside a mod.rs and user.rs.
user.rs:

use serde::{Deserialize, Serialize};

#[derive(Deserialize, Serialize, Debug)]
#[serde(rename_all = "camelCase")]
pub struct AddUserEvent {
    pub first_name: String,
    pub last_name: String,
}

#[derive(Deserialize, Serialize, Debug)]
#[serde(rename_all = "camelCase")]
pub struct User {
    pub uuid: String,
    pub first_name: String,
    pub last_name: String,
}

mod.rs:

pub mod user;

In user.rs we define a struct for the incoming add user event, and a struct that will be used to map a dynamo GetItem result to. Using serde's rename_all we can maintain Rust's convention for snake_case in code but has camelCase in json requests/responses.

mod.rs is defining a public a module that can be imported elsewhere.

Defining the Lambda Handler

Boilerplate

Starting off with writing the lambda handler itself, the usual entry-point is the main function:

use lambda_http::{handler, Request};
use lambda_runtime::{Context, Error as LambdaError};

#[tokio::main]
async fn main() -> Result<(), LambdaError> {
    lambda_runtime::run(handler(handler_func)).await?;
    Ok(())
}

A few things are happening here...

First, the function needs to be annotated for tokio in order to execute the function in the tokio runtime (read more here). Normally, the main function cannot be async, but adding this annotation enables this.

Next, we have to set the return type to be a Result<(), LambdaError> so we can gracefully handle any errors.

Finally, we are using the lambda_runtime crate to easily bootstrap (as the name implies) a working lambda runtime. Since our lambda is going to be invoked via an API Gateway request, we can then use the lambda_http crate to set ourselves up to accept an Http Request. We .await? the call to lambda_runtime::run to allow for execution and bubble up any errors.

This is all boilerplate code that will only have to been done until Rust gets an official Lambda runtime from AWS.

Basic Routing Handler

At its most basic, the handler function has a signature that should be familiar to anyone who has written a lambda handler before- it takes in an event object and a context object (unused here so by convention labeled with _), and returns a value or error.

async fn handler_func(event: Request, _c: Context) -> Result<Value, LambdaError> {

Here, Request represents an HttpRequest type that has the properties you would expect to find:

method
uri
version
headers
extensions
body

Value represents a json value for a response.

Since this lambda will handle multiple HTTP methods, some basic routing will be needed in order to execute different functions depending on the incoming request method.

use lambda_http::{handler, Request};
use lambda_runtime::{Context, Error as LambdaError};
use serde_json::{json, Value};

--snip--

async fn handler_func(event: Request, _c: Context) -> Result<Value, LambdaError> {
let result = match event.method() {
        &lambda_http::http::method::Method::GET => {
            json!("get")
        }
        &lambda_http::http::method::Method::POST => {
            json!("post")
        }
        _ => {
            println!("Handling other request");
            json!("other")
        }
    };

    Ok(json!(result))
}

Here, a match statement can be used to run different code based upon the incoming Request Method. Separate handlers for both creating a user and getting a user will be defined.

Add User Handler

All the created handlers can be placed in a new folder called handlers. For now create a mod.rs and create_user_handler.rs

mod.rs:

pub mod create_user_handler;

The handler itself needs the Dynamo client defined in the parent handler, as well as request payload so it can serialize it into a struct (this is done be using serde_json). Finally in order to successfully submit a PutItem request, a uuid needs to be generated.

create_user_handler.rs:

use aws_sdk_dynamodb::model::AttributeValue;
use aws_sdk_dynamodb::Client;
use serde_json::{json, Value};
use lambda_runtime::{Error as LambdaError};
use uuid::Uuid;

use crate::{
    models::{
        user::{AddUserEvent, User}
    }
};

pub async fn create_user(client: &Client, payload: &str) -> Result<Value, LambdaError> {
    let uuid = Uuid::new_v4().to_string();
    let add_user_event: AddUserEvent = serde_json::from_str(payload)?;

    let request = client
        .put_item()
        .table_name("rust-lambda-table")
        .item("userId", AttributeValue::S(String::from(&uuid)))
        .item(
            "modelTypeAndId",
            AttributeValue::S(format!("User#{}", String::from(&uuid))),
        )
        .item("first_name", AttributeValue::S(add_user_event.first_name.clone()))
        .item("last_name", AttributeValue::S(add_user_event.last_name.clone()));

    request.send().await?;
    let created_user = User {
        uuid: uuid,
        first_name: add_user_event.first_name,
        last_name: add_user_event.last_name
    };
    let user_json = serde_json::to_value(&created_user).unwrap();
    Ok(json!(user_json))
}

Now we have a basic handler function that takes in a json payload and create a new Dynamo item. In order to send back the correct Value type, we can useserde_json::to_value(). Now we're returning a dynamic response based on the payload!

Get User Handler

With a handler created for creating a User, let's move on to defining the handler required for retrieving that User from the table. Create a new get_user_handler.rs. This handler will also need a Dynamo client as an argument as well as an id of the User that is to be requested.

get_user_handler.rs:

use serde_json::Value;
use aws_sdk_dynamodb::model::AttributeValue;
use aws_sdk_dynamodb::Client;
use lambda_runtime::{Error as LambdaError};

use crate::{
    models::{
        user::User
    }
};

pub async fn get_user(client: &Client, id: &str) -> Result<Value, LambdaError> {
    let request = client
        .query()
        .table_name("rust-lambda-table")
        .key_condition_expression("#id = :uuid")
        .expression_attribute_names("#id", "userId")
        .expression_attribute_values(":uuid", AttributeValue::S(id.to_string()));

        let result = request.send().await?;
        let user = match result.items.unwrap().first() {
            Some(res) => User {
                uuid: res.get("userId").unwrap().as_s().unwrap().clone(),
                first_name: res.get("first_name").unwrap().as_s().unwrap().clone(),
                last_name: res.get("last_name").unwrap().as_s().unwrap().clone()
            },
            _ => User {
                uuid: String::from(""),
                first_name: String::from(""),
                last_name: String::from("")
            }
        };

        let user_json = serde_json::to_value(&user).unwrap();
        Ok(user_json)
}

For now, if no User is found a blank User will be returned. Since we created a second handler we want to export as part of the handlers module, mod.rs needs to be updated.

mod.rs:

pub mod create_user_handler;
pub mod get_user_handler;

Update Routing Handler

The only thing left to do is introduce the DynamoDB configuration and implement the calls to the newly created handlers.

main.rs:

use aws_config::meta::region::RegionProviderChain;
use aws_sdk_dynamodb::model::AttributeValue;
use aws_sdk_dynamodb::Client;
use self::{
    models::{
        user::AddUserEvent
    },
    handlers::{
        create_user_handler::create_user,
        get_user_handler::get_user
    }
};

--snip--
async fn handler_func(event: Request, _c: Context) -> Result<Value, LambdaError> {
    let region_provider = RegionProviderChain::default_provider().or_else("us-east-1");
    let config = aws_config::from_env().region(region_provider).load().await;
    let client = Client::new(&config);

    let body_string: &str = match event.body() {
        lambda_http::Body::Text(text) => text.as_str(),
        _ => "",
    };


    let result = match event.method() {
        &lambda_http::http::method::Method::GET => {
            println!("Handling GET request");

            json!(get_user(&client, event.query_string_parameters().get("id").unwrap()).await?)
        }
        &lambda_http::http::method::Method::POST => {
            println!("Handling POST request");
            json!(create_user(&client, body_string).await?)
        }
        _ => {
            println!("Handling other request");
            json!("other")
        }
    };

    Ok(json!(result))
}

Here us-east-1 region is used as the default, defining a default configuration object and passing that into a client constructor method to instantiate an instance of a DynamoDB client for use.

Using the AWS SDK makes interacting with Dynamo fairly straightforward. With a client instantiated, we can use it in the different user handlers to access methods to perform actions on a given table.

With that done, we now have all the handler code written and are ready to set up the cloud resources that will be required to successfully deploy and invoke this lambda! That will all be covered in part 2.

Github Link: https://github.com/mblydenburgh/rust-lambda

Rust + Lambda using CDK & Github Actions (Part 2)

Mike Blydenburgh — Fri, 04 Mar 2022 14:41:09 +0000

In continuing where things left off in part 1, here we will be using GitHub Actions to automatically deploy a CDK stack to provision all the cloud resources.

There are a few prerequisites that going to be assumed here:

You have an AWS account and know you way around the console
You have a Github Repository
The CDK CLI is installed (npm i -g cdk)

CDK Stack

To have things work as intended, an IAM User will be required that has an Access Key Id and Secret Access Key that can be passed into the deployment pipeline in Github. The permissions that this role has will depend on the resources that is required to provision, but for our purposes setting the following AWS-managed policies should be sufficient:

IAMFullAccess
AmazonS3FullAccess
AmazonDynamoDBFullAccess
AmazonAPIGatewayAdministrator
AmazonSSMFullAccess
AWSCloudFormationFullAccess
AWSLambda_FullAccess

Note: Caution should be used when creating IAM Users with too much access, but as a general rule User's should be creating with the minimum needed permissions. The User defined here is overly simplified for demo purposes.

With a User created, you will need to create an Access Key and download the credentials excel file. These will be the secrets that need to be injected into the Github repo. To add them, go to your repository settings on Github, then from the Security section create two secrets named AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with their respective values set:

Next we can create everything required for the CDK stack. Create a new directory in the project root folder called cdk. In this folder a new cdk project should be initialized via cdk init app --language typescript, which will generate all the required files needed to deploy a stack. Some important ones to note:

/cdk/package.json: Where stack dependencies are defined
/cdk/lib/cdk-stack.ts: Where the stack definition will go
/cdk/bin/cdk.ts: File that instantiates an instance of the stack defined in cdk-stack.ts and sets tags, environment config, etc
/cdk/cdk.json: Invokes the code in cdk.ts when cli commands are ran to initiate deployment

Some other things will generate such as a tsconfig.json file and some Jest tests, but those can be ignored for now.

First we can define what dependencies will be needed by updating the package.json file:

{
  "name": "cdk",
  "version": "0.1.0",
  "bin": {
    "cdk": "bin/cdk.js"
  },
  "scripts": {
    "build": "tsc",
    "watch": "tsc -w",
    "test": "jest",
    "cdk": "cdk"
  },
  "devDependencies": {
    "@aws-cdk/core": "1.143.0",
    "@aws-cdk/aws-apigatewayv2": "1.143.0",
    "@aws-cdk/aws-apigatewayv2-integrations": "1.143.0",
    "@aws-cdk/aws-dynamodb": "1.143.0",
    "@aws-cdk/aws-iam": "1.143.0",
    "@aws-cdk/aws-lambda": "1.143.0",
    "@types/jest": "^26.0.10",
    "@types/node": "17.0.15",
    "jest": "^26.4.2",
    "ts-jest": "^26.2.0",
    "aws-cdk": "2.10.0",
    "ts-node": "10.4.0",
    "typescript": "4.5.5"
  },
  "dependencies": {
    "aws-cdk-lib": "2.15.0",
    "constructs": "^10.0.0",
    "source-map-support": "^0.5.16"
  }
}

@aws-cdk/core provides basic Stack definitions, while all the other @aws-cdk/ prefixed dependencies provide constructs to provision resources related to that service. Everything else is boilerplate dependencies required to compile the project, though with updated versions.

There is a section in cdk.json that can uncommented to set the account and region that will be used for deployment. The name of stack class will be generated based upon the project name but can renamed if desired (renamed to CdkStack here). Additionally, we are going to use cdk from @aws-cdk/core instead of aws-cdk-lib:

import cdk = require('@aws-cdk/core');

new CdkStack(app, 'CdkStack', {
  env: { account: accountNumber, region: 'us-east-1' },
});

This will create a new CDK app using a CdkStack class defined in cdk-stack.ts:

import cdk = require('@aws-cdk/core');

export class CdkStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const appName = "rust-lambda"

As you can see, we are just extending the provided base stack that AWS provides. The appName variable will be referenced while naming all of our resources. to begin with, we know we need a DynamoDB table to save records to so let's start with that.

import { AttributeType, BillingMode, Table } from "@aws-cdk/aws-dynamodb"

export class CdkStack extends cdk.Stack {
--snip--
const dynamoTable = new Table(this, `DynamoTable`, {
      tableName: `${appName}-table`,
      partitionKey: { name: "userId", type: AttributeType.STRING },
      sortKey: { name: "modelTypeAndId", type: AttributeType.STRING },
      billingMode: BillingMode.PAY_PER_REQUEST
    })
--snip--
}

Lambda's will have an attached IAM Role that is used to control what services the Lambda is able to interact with. Creating a new role is just as simple:

--snip--
const lambdaRole = new Role(this, `LambdaRole`, {
            roleName: `${appName}-role`,
            assumedBy: new ServicePrincipal("lambda.amazonaws.com"),
            managedPolicies: [
ManagedPolicy.fromAwsManagedPolicyName("service-role/AWSLambdaVPCAccessExecutionRole")
            ]
})

dynamoTable.grantReadWriteData(lambdaRole)
        lambdaRole.addToPolicy(new PolicyStatement({
            effect: Effect.ALLOW,
            actions: ["dynamodb:Query", "dynamodb:Scan", "dynamodb:GetItem", "dynamodb:PutItem", "dynamodb:DeleteItem"],
            resources: [
                `arn:aws:dynamodb:${Aws.REGION}:${Aws.ACCOUNT_ID}:table/${dynamoTable.tableName}/index/*`
            ]
        }))
--snip--

Here, a Role is being provisioned and we are granting it permission to use a specific set of DynamoDB actions. If we omitted dynamodb:PutItem for instance, the lambda would not be able to perform that operation. This gives very fine grain control on what a particular Lambda should be able to do.

Next, we can provision the actual Lambda, assign it the Role that we just created, and create what is called a Lambda Integration:

--snip
const lambdaFunction = new Function(this, `LambdaFunction`, {
      functionName: `${appName}-lambda`,
      runtime: Runtime.PROVIDED_AL2,
      role: lambdaRole,
      code: Code.fromAsset("../target/x86_64-unknown-linux-musl/release/lambda.zip"),
      handler: "main",
      environment: {
        RUST_BACKTRACE: '1'
      }
    })
    lambdaFunction.grantInvoke(new ServicePrincipal("apigateway.amazonaws.com"))

    const lambdaIntegration = new HttpLambdaIntegration("HttpLambdaIntegration", lambdaFunction)
--snip--

Since Rust does not currently have an official Rust runtime, we need to deploy it as a custom runtime using Amazon Linux 2. The code asset that is required needs to point to zip file that has the executable in a file called Bootstrap (read more on custom runtimes here). We will define the actual build steps to support this requirement last.

Since we want to invoke the Lambda via an http request, we need to state that the API Gateway service can invoke it. We also create a HttpLambdaIntegration that will be passed to the individual api routes. This pattern enables each different route to invoke different Lambdas.

There are a couple different ways to set up an APIGateway, but here we can use an HttpApi. Refer to this page to see the differences between ApiGateway's HttpApi and RestApi.

const api = new HttpApi(this, `RestAPIGateway`, {
    apiName: "rust-lambda-api",
    corsPreflight: {
        allowHeaders: ['Authorization', 'Access-Control-Allow-Origin','Access-Control-Allow-Headers','Content-Type'],
        allowMethods: [
            CorsHttpMethod.ANY
        ],
        allowOrigins: ['*'],
    },
})

api.addRoutes({
    path: "/users",
    methods: [HttpMethod.GET, HttpMethod.POST],
    integration: lambdaIntegration
})

api.addRoutes({
    path: "/users/{proxy+}",
    methods: [HttpMethod.GET, HttpMethod.POST],
    integration: lambdaIntegration
})

For this example we will just need a /users endpoint that will accept only POST and GET requests. If desired, you can allow all methods will HttpMethod.ANY. Having the additional /users/{proxy+} endpoint will enable any endpoint that extends from /users/. Now we have a complete stack that we're ready to deploy!

Note: If your IDE is complaining about about argument of type 'this' is not assignable to parameter of type 'Construct', you can add an //@ts-ignore to the line right above the resource provisioning.

GitHub Actions

In order to automatically deploy the stack to your AWS account, let's set up an Actions Workflow. Github will automatically scan your project for a .github/workflows directory, and execute files it finds.

A number of things need to be accounted for in order for the build and deployment to be successful:

AWS credentials are injected
Rust is installed, set for a Linux target like x86_64-unknown-linux-musl
Executable is named bootstrap and zipped into a file

name: Deploy CDK Stack

on: 
  push:
    branchs: [main]


jobs:
  aws_cdk:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        target:
          - x86_64-unknown-linux-musl
    steps:
      - name: Checkout Code
        uses: actions/checkout@v2

      - name: Set AWS Credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{secrets.AWS_ACCESS_KEY_ID}}
          aws-secret-access-key: ${{secrets.AWS_SECRET_ACCESS_KEY}}
          aws-region: us-east-1

      - name: Install Node
        uses: actions/setup-node@v1
        with:
          node-version: 14

      - name: Install Rust
        uses: actions-rs/toolchain@v1
        with:
          toolchain: stable
          target: ${{ matrix.target }}
          override: true

      - name: Install Stack Dependencies
        run: 'sudo npm ci'
        working-directory: ./cdk

      - name: Install NPM
        run: 'sudo apt update -y && sudo apt install nodejs npm -y'

      - name: Install CDK CLI
        run: 'sudo npm install -g aws-cdk'

      - name: Run Build
        uses: actions-rs/cargo@v1
        with:
          use-cross: true
          command: build
          args: --release --all-features --target=${{ matrix.target }}

      - name: Rename binary to bootstrap
        run: 'mv ./rust_lambda ./bootstrap'
        working-directory: ./target/x86_64-unknown-linux-musl/release

      - name: Zip Code for Deployment
        run: 'sudo zip -j lambda.zip bootstrap'
        working-directory: ./target/x86_64-unknown-linux-musl/release

      - name: CDK Synth
        run: 'cdk synth'
        working-directory: ./cdk

      - name: Run CDK Bootstrap
        run: 'cdk bootstrap aws://415023725722/us-east-1'
        working-directory: ./cdk

      - name: CDK Deploy
        run: 'cdk deploy --require-approval never'
        working-directory: ./cdk

As configured, this Action will execute anytime something to the main branch gets pushed. Let's give that a go now! Rust takes a little while to compile, especially on the Github VMs, so you might want to grab a cup of coffee 🙂

Invoking

With everything deployed, we are finally ready to try to invoke our lambda! To get the endpoint that is generated, go to your AWS console and navigate to your Lambda's page, then go to the Configuration tab, then Triggers. You should see an API Gateway listed with a given URL:

You should be able to make a POST request in Postman, Thunder Client, etc and get a successful 200 response!

Github Link: https://github.com/mblydenburgh/rust-lambda