Forem: Sebastian Bille

Radical Simplicity in Cloud Architecture

Sebastian Bille — Tue, 04 Mar 2025 09:04:10 +0000

For a long time, both professionally with clients in my consulting work and in the AWS community, I've been evangelizing the "serverless" paradigm shift. To me, it's crystal clear that you should focus on building what differentiates your business, and not on managing infrastructure or solving problems that are already solved by others. The term "serverless" may have been watered down by marketing in recent years, but the core idea of utilizing managed services and focusing on business logic remains.

People argue about cold starts, vendor lock-in, or learning curves, and while they are valid concerns, I believe that the benefits of serverless far outweigh the drawbacks - especially when you consider the total cost of ownership over the lifetime of a project. But I'm not here to argue about these concerns today, I'm here to talk about something else.

Luc van Donkersgoed covers some of these concerns and others in my favorite blog post of all time here. Go ahead and read it but pinky promise you'll come back!

Today I want to talk about an aspect of serverless that I think is often overlooked: the developer experience and the unexpected use cases of "boring" technology that emerge when it's exposed in the form of a radically simple to consume and configure managed service.

Your personal production environment

Do you know how many lines of code or clicks in the console you need to set up a DynamoDB table? An S3 Bucket? A Lambda function?

The answer is about 2.

This sets up a service that will scale to virtually any load, without further configuration. Perhaps more importantly though, it also scales to 0, and you literally pay nothing when it's not being used.

People tend to focus on the scaling up part, not realizing what the scaling down part actually entails. It's not just about saving money, it's about the freedom to experiment and build things in a way that you wouldn't have otherwise.

Want to have a personal carbon-copy production environment? Sure, hit deploy and have it ready in 30 seconds and it will always be free.

When you merge code to main, you can be absolutely certain that it will work exactly the same in production as it did during development, because you've been testing in "production" all along.

Want to spin up an ephemeral environment on every pull request, run integration tests against it, and tear it down when the pull request is merged? Sure, it's literally a few lines of code and it will cost you nothing.

And once these resources are deployed, you will never have to worry about them again. They will scale up and down to meet demand, there is nothing to patch or maintain, and the availability and security of the services will be the responsibility of one of the most experienced and capable companies in running cloud infrastructure in the world - so that you can spend your time where it actually sets you apart.

Gone are the days of sharing a database in the development environments to keep cost down. Gone are the days tinkering with scaling groups and load balancers. Gone are the days of waking up in the middle of the night to patch a vulnerability.

When the log4net vulnerability was announced, I was working in the platform services team at a major automotive company running hundreds of workloads in AWS. Do you know what we did to patch it? Not a damn thing.

The same company at one point ran a superbowl ad and traffic was naturally expected to explode. Do you know what we did to prepare for it? Not a damn thing - we sat back to enjoy the show.

The idea that serverless compute is more expensive than traditional alternatives is less and less true the more you look at the full picture and lifecycle of your applications. Is there a break-even point where it becomes more cost-efficient to move away from it? Sure. But I'd argue that that point is way (way) higher than you'd think if you consider the total cost of ownership. That is, not only the compute layer itself, but also:

development environments and feedback loops
labor cost of maintenance, configuration management
overprovisioning to meet demand, or underprovisioning and risking business objectives, and the labor of finding the right balance
the opportunity cost of not spending time on what actually matters to the business

This isn't meant to be a blanket statement claiming that serverless is always cheaper. But it is cheaper in more cases than you might think at first glance. It's also not a binary choice. Employing a serverless-first strategy is wise, where you default to “more serverless” and move further back in the abstraction spectrum as needed, with parts of your application (or application landscape) being serverful where it's more fitting.

Unexpected Use Cases from Radically Simple Services

Traditionally, setting up a globally distributed, highly available database required intricate configurations: manual sharding, complex replication setups, and constant vigilance to maintain consistency across regions. This complexity often deterred teams from pursuing global distribution, limiting application performance and resilience.

Enter services like DSQL and DynamoDB Global Tables. With the latter as an example, you can select the regions where you need your data replicated, and DynamoDB handles the rest, eliminating the complexity and operational burden of deploying and managing multi-region replication. This means that updates performed on a replica table in one region are automatically replicated to the replica tables in other regions, ensuring low-latency access for users worldwide. By abstracting away traditional pain points, fully managed services make many problems that were previously very hard and costly to solve completely disappear.

And this pattern repeats itself across many other services and problem domains. Whether it's APIs with API Gateway, Cron jobs and scheduled tasks with EventBridge or workflow orchestration with StepFunctions - what once required dedicated infrastructure, maintenance, and tuning is now an API call away.

When you have managed services that are so simple to consume and configure, you start to see unexpected use cases for technology that you wouldn't have considered before.

My friend and colleague Elias Brange has two perfect examples of this.

The first one is covered in his blog post "Test Event-Driven Architectures with EventBridge and AppSync Events" where he explores the idea of using AppSync Events (a managed Websockets API) to test event-driven architectures end-to-end. Spinning up a websockets API as part of your test suite and streaming events through it to test your system's behavior is a brilliant idea, and it's a fantastic solution to a problem that's hard to solve with traditional tools.

The other one is about spinning up a Wiremock service as a Lambda Layer to mock external services during early development or as part of integration tests without the need to change any application code.

While it's obviously not a new idea to mock external services, the simplicity of setting up a Wiremock service in a few lines of code, deploying it as a "sidecar" to your dev stack, and not having to pay a dime for it is, honestly, a game changer.

If you talk to developers who've been working with serverless for a while, you'll find that it's not uncommon for many of them to have tricks like this up their sleeve.

Conclusion

When you really start to lean into the serverless paradigm shift, you realize that perhaps the biggest benefit isn't just avoiding infrastructure management or infinite scalability. It's the ability to focus entirely on solving business problems instead of wrangling things that "should just work. It may even unlock new ways of thinking about how you can use technology to solve problems that you wouldn't have considered before, and in, dare I say, the vast majority of cases, it will be cheaper in the long run.

Hi there, I'm Sebastian Bille! If you enjoyed this post or just want a constant feed of memes, AWS & serverless talk, and the occasional new blog post, make sure to follow me on 𝕏 at @TastefulElk or on LinkedIn 👋

Elva is a serverless-first consulting company that can help you transform or begin your AWS journey for the future

⚡️ Testing Individual StepFunction States

Sebastian Bille — Thu, 07 Dec 2023 07:46:59 +0000

AWS StepFunctions is a fantastic service for orchestrating complex workflows. But testing StepFunctions has always been... tricky.

You could execute the entire state machine, observing side effects and final outcomes, but this approach often feels like using a sledgehammer for a task that needs a scalpel. It's effective for smaller workflows but quickly becomes unwieldy with more complex ones.

Unit testing Lambda functions or other compute components can cover the core logic of a task, but this doesn't cover data transformations or control flows between states.

Local emulation of StepFunctions is another approach but it often leads to IAM access inconsistencies and other headaches common with emulating cloud services.

While these methods are useful, they each come with drawbacks that risk that the beautiful state machine you wrote to handle that one complicated payment workflow becomes just another mess of logic that is hard to maintain or evolve. This is particularly ironic considering that many state machines are built explicitly to untangle complex logic.

💡 Did you know? While StepFunctions is the name of the AWS Service, a workflow in StepFunctions is called a state machine. Each step in a workflow is called a state, and a task state represents a state where another AWS service performs the work.
Right before re:Invent 2023, however, AWS released a new capability to test individual states in a state machine.

Right before re:Invent 2023, however, AWS released a new capability to test individual states in a state machine.

This feature is available in the StepFunction console, but, more importantly, it also allows us to write isolated integration tests in code. These can test output, data transformations, and control flow for each individual state in our state machines - all without the hassle of trying to force the entire state machine to go down a particular route to validate the same thing.

Let's take a look at how we can use it.

Integration Tests

First, assume we have a small state machine that upgrades a user account, but only if they have enough credits to cover the cost.

Using the Node AWS SDK, there's a new TestStateCommand. This command takes:

A task definition
A task input payload
An IAM role that the StepFunction service will assume to execute the state

It executes the state, and returns:

The status of the execution - whether or not it errored, succeeded, or caught an error
The output of the task
The state that would be next in line
Metadata about the execution

An example response could look like this:

{
  '$metadata': {
    httpStatusCode: 200,
    requestId: '2916bb74-418d-402b-a903-18c0a9c3e9c9',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  nextState: 'Choice',
  output: '{"ExecutedVersion":"$LATEST","Payload":{"cost":75,"credits":100},"SdkHttpMetadata":{"AllHttpHeaders":{"X-Amz-Executed-Version":["$LATEST"],"x-amzn-Remapped-Content-Length":["0"],"Connection":["keep-alive"],"x-amzn-RequestId":["256f8768-f84e-46b0-ba79-b48dbfd250fa"],"Content-Length":["25"],"Date":["Wed, 06 Dec 2023 19:48:24 GMT"],"X-Amzn-Trace-Id":["root=1-6570d008-34fc67a96baf085e418d0745;sampled=1;lineage=055c6c5a:0"],"Content-Type":["application/json"]},"HttpHeaders":{"Connection":"keep-alive","Content-Length":"25","Content-Type":"application/json","Date":"Wed, 06 Dec 2023 19:48:24 GMT","X-Amz-Executed-Version":"$LATEST","x-amzn-Remapped-Content-Length":"0","x-amzn-RequestId":"256f8768-f84e-46b0-ba79-b48dbfd250fa","X-Amzn-Trace-Id":"root=1-6570d008-34fc67a96baf085e418d0745;sampled=1;lineage=055c6c5a:0"},"HttpStatusCode":200},"SdkResponseMetadata":{"RequestId":"256f8768-f84e-46b0-ba79-b48dbfd250fa"},"StatusCode":200}',
  status: 'SUCCEEDED'
}

Let's put together a couple of helper functions that can help us fetch a deployed state machine's definition, and one that executes a given state in the state machine definition.

// util.ts

import {
  DescribeStateMachineCommand,
  SFNClient,
  TestStateCommand,
} from '@aws-sdk/client-sfn';

const client = new SFNClient({});

interface TestStateInput {
  stateMachineDefinition: string;
  roleArn: string;
  taskName: string;
  input?: string;
}

export const fetchStateMachine = async (stateMachineArn: string) => {
  const stateMachine = await client.send(
    new DescribeStateMachineCommand({
      stateMachineArn: stateMachineArn,
    })
  );

  if (!stateMachine.definition) {
    throw new Error('State machine definition not found');
  }

  if (!stateMachine.roleArn) {
    throw new Error('State machine roleArn not found');
  }

  return {
    definition: stateMachine.definition,
    roleArn: stateMachine.roleArn,
  };
};

const getTask = (taskName: string, definition: string) => {
  // parse the definition and return the given state
  const task = JSON.parse(definition).States[taskName];
  if (!task) {
    throw new Error(`Task ${taskName} not found`);
  }

  return JSON.stringify(task);
};

export const testState = async ({
  roleArn,
  stateMachineDefinition,
  taskName,
  input,
}: TestStateInput) => {
  const task = getTask(taskName, stateMachineDefinition);

  return await client.send(
    new TestStateCommand({
      definition: task,
      roleArn: roleArn,
      input: input,
    })
  );
};

fetchStateMachine takes a state machine ARN and returns the deployed state machine's definition and the IAM role it's configured to use.

testState takes that definition, role, and the name of the state that we want to test. It executes the state using the StepFunction service, and it returns the execution information.

We can now use these to start writing our tests:

// account-upgrade.test.ts
import { describe, it, expect } from 'vitest';
import { fetchStateMachine, testState } from './util';

describe('[accountUpgrade]', async () => {
  const stateMachine = await fetchStateMachine(
    'arn:aws:states:us-east-1:123456789012:stateMachine:accountUpgrade'
  );

  describe('[CheckCreditBalance]', async () => {
    it('returns credit balance', async () => {
      const res = await testState({
        stateMachineDefinition: stateMachine.definition,
        roleArn: stateMachine.roleArn,
        taskName: 'CheckCreditBalance',
        input: JSON.stringify({
          cost: 75,
        }),
      });

      expect(res.status).toBe('SUCCEEDED');
      expect(res.nextState).toBe('Choice');

      const output = JSON.parse(res.output || '{}');
      expect(output.Payload).toEqual({
        balance: expect.any(Number),
        cost: 75,
      });
    });
  });

  describe('[Choice]', async () => {
    it('flows to account upgrade if credit balance more than cost', async () => {
      const res = await testState({
        stateMachineDefinition: stateMachine.definition,
        roleArn: stateMachine.roleArn,
        taskName: 'Choice',
        input: JSON.stringify({
          cost: 75,
          balance: 100,
        }),
      });

      expect(res.status).toBe('SUCCEEDED');
      expect(res.nextState).toBe('UpgradeAccount');
    });
  });
});

With this, we have everything we need to start to fully cover the full configuration and logic of our state machines.

For a more complete demo, which gives us type safety for the resource ARN and the state names, I've published a demo SST project here.

Additionally, Lars Jacobsson has added support to his samp-cli project for interactively running the individual state tests - it even lets you re-use recent live execution payloads. Incredible!

If you want to dig further into more strategies for testing StepFunctions, Yan Cui has some excellent writing on it here.

In conclusion, testing StepFunctions is hard, but it did just get a whole lot easier. As we've seen, the ability to test individual StepFunction states marks a leap forward in how we can build more robust and well-tested state machines. This feature isn't just a neat trick; it addresses some of the core frustrations we've faced in complex workflow orchestration. It doesn't replace end-to-end testing and unit testing of state machines, but it's a well-appreciated complement!

Hi there, I'm Sebastian Bille! If you enjoyed this post or just want a constant feed of memes, AWS & serverless talk, and the occasional new blog post, make sure to follow me on 𝕏 at @TastefulElk or on LinkedIn 👋

Elva is a serverless-first consulting company that can help you transform or begin your AWS journey for the future

⚡️Serverless Frameworks for 2023

Sebastian Bille — Mon, 23 Jan 2023 07:36:32 +0000

In this post, we'll look at some of the most popular frameworks for building serverless applications on AWS, including their features, strengths, and weaknesses. We'll also look at examples of how each framework can be used to build an example application. Whether you're a seasoned serverless developer or are just getting started with it, this post will give you a good overview of the different frameworks available and help you decide which one is best for your needs.

Example Application: A To-Don’t app

To showcase how each framework is used, we'll build a To-Don't application which, obviously, is an app to list things you don't want to do, rather than a To-Do app.

The architecture will look like this:

An API Gateway with a POST /item method
A “Persist” Lambda function that writes the item to a DynamoDB table
A DynamoDB table with Streams enabled
A “Fan-out” function that subscribes to the DynamoDB stream and then fans out the item to an SNS Topic

Implementing an endpoint to fetch items is left as an exercise for the reader; we're on a tight deadline here.

ℹ️ tl;dr: If you want to skip ahead and just compare the structure and configuration for each project, including the code for the actual Lambda functions, you can find them all here.

Serverless Framework

Serverless Framework is an open-source tool built specifically to simplify building serverless apps. While it does support other clouds, such as Azure and Google Cloud, AWS is definitely where the focus is. It's found a carefully balanced way of abstracting away “just enough” of the parts you may not care much about while allowing you the flexibility of CloudFormation when you need it. For a long time, Serverless Framework was the clear choice for many development teams.

It's got a plugin system with wide community support, and it's unlikely there isn't a plugin that solves your problem if you're lacking something from the base framework.

Lately, however, the innovation speed seems to have dwindled, and its competitors have far outrun it in feature additions.

All in all, Serverless Framework is a solid choice - a safe choice, really - but this one won't wow you.

🛠 Working with Serverless Framework

To install the Serverless Framework CLI tool, run npm install -g serverless. You can then initialize a new project by running serverless, and you'll get to choose from a number of starting points and examples.

Serverless Framework apps use a serverless.yml file that, in its simplest form, looks a little something like this:

# name of our app
service: to-dont-app

# let the provider know that we want to deploy to AWS
provider:
  name: aws

# define out Lambda functions
functions:
  hello:
    handler: hello.handler
    runtime: nodejs16.x

ℹ️ Since Serverless Framework supports clouds other than AWS, we need to instruct the provider that we want to deploy to AWS.

Given the above serverless.yml and AWS credentials configured, running serverless deploy will deploy a ‘hello’ Lambda function. The CLI also includes a bunch of utility functions that help you skip a couple of trips to the AWS console. You can, for example, invoke your newly deployed function by running serverless invoke -f hello, tail the CloudWatch logs by running serverless logs -f hello --tail, or generate test events, such as an API Gateway event, by running serverless generate-event -t aws:apiGateway.

👷‍♂️ To-Don't example

Our To-Don't application's serverless.yml looks like this, with added comments explaining what's going on:

service: to-dont

plugins:
  # Add a plugin that allows us to specify more fine grained IAM policies
  - serverless-iam-roles-per-function

# require serverless v3
frameworkVersion: '3'

provider:
  name: aws
  # default config for all functions
  runtime: nodejs16.x
  region: eu-north-1

functions:
  persist:
    handler: src/persist.handler
    # Set an environment variable with the DynamoDB table name,
    # based on the reference to the table in the CloudFormation 
    # Resources block below
    environment:
      TABLE_NAME: !Ref DynamoTable
    # Expose the function for POST requests on /item in an ApiGateway
    events:
      - http:
          path: /item
          method: post
    # Allow the function to write to our DynamoDB table
    iamRoleStatements:
      - Effect: Allow
        Action:
          - dynamodb:PutItem
        Resource:
          - !GetAtt DynamoTable.Arn
  fanout:
    handler: src/fanout.handler
    # Set an environment variable with the SNS topic ARN
    environment:
      TOPIC_ARN: !Ref SnsTopic
    # Trigger the function when a new item is added to the DynamoDB
    # table by listening to the DynamoDB stream
    events:
      - stream:
          type: dynamodb
          arn:
            !GetAtt DynamoTable.StreamArn
    iamRoleStatements:
      - Effect: Allow
        Action:
          - sns:Publish
        Resource:
          - !Ref SnsTopic

# DynamoDB tables and SNS Topics has to be set up in CloudFormation,
# and the Resources block below allows us to do just that
resources:
  Resources:
    DynamoTable:
      Type: AWS::DynamoDB::Table
      Properties:
        BillingMode: PAY_PER_REQUEST
        AttributeDefinitions:
          - AttributeName: pk
            AttributeType: S
        KeySchema:
          - AttributeName: pk
            KeyType: HASH
        StreamSpecification:
          StreamViewType: NEW_AND_OLD_IMAGES
    SnsTopic:
      Type: AWS::SNS::Topic

You can find the full Serverless Framework version of our To-Dont app here.

⚖️ Pros and Cons

Pros:

Strikes a great balance between abstracting away messy CloudFormation code that you likely don't care about and giving you the flexibility to use it when needed.
The plugin ecosystem is fantastic, and you'll most likely find whatever you need.
Battle-tested and proven for years.

Cons:

The configuration can be confusing on what is Serverless Framework config and, what is actual CloudFormation, and how they interact.
Innovation speed and support have dwindled lately.
Not possible to define re-usable components across stacks

📚 Resources

Get started with Serverless Framework

Lightning fast & simple Typescript Serverless builds

6 Serverless CLI commands you didn't know existed

Serverless Framework Serverless Patterns Collection

AWS SAM

AWS's open-source take on a declarative framework specifically for deploying serverless applications is called AWS SAM, or Serverless Application Model. It had a slow start but has had an incredible cadence of pumping out features and quality-of-life additions in the last year or two and is now a very serious contender.

It uses a superset of CloudFormation, which should make anyone familiar with that should feel right at home, and, much like Serverless Framework, it aims to abstract away the ugly parts and streamline the development process.

SAM is an excellent choice for building your serverless apps; it's got great support from AWS, they're active in the community to take in feedback and ideas, and they churn out features like there's no tomorrow. The configuration is however a bit verbose, which might tilt the learning curve slightly for beginners.

🛠 Working with AWS SAM

To install the SAM CLI, follow the installation instructions for your platform here, or if you use Homebrew, run:

brew tap aws/tap
brew install aws-sam-cli

Bootstrapping a new project with SAM is done by running sam init\ and, again, you'll get to choose from a couple of different starting points.

SAM, since it's a superset of CloudFormation, uses a template.yaml file for its configuration, and a simple example might look like this:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Resources:
  HelloFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: src/app.lambdaHandler
      Runtime: nodejs16.x

SAM also optionally uses a samconfig.toml file that specifies default parameters for its commands, such as what region you want to deploy to or the stack name. The great thing about separating the definition of the app and how the app is built in two different files is the template.yaml file can be written very generically so that it can be shared and re-used.

Given the above template.yaml and AWS credentials configured, running sam deploy --guided will give you a wizard that will automatically create the samconfig.toml file for you and then deploy the application and our ‘hello’ Lambda function.

The SAM CLI also includes helpful utility functions to make your life easier. Unfortunately, you can't invoke the deployed function directly from the CLI, but you can tail the CloudWatch logs by running sam logs -n hello --tail and generate test events, such as an API Gateway event, by running sam local generate-event apigateway aws-proxy - but perhaps the most useful one is sam sync which first deploys your application and then listens to code changes in your Lambda functions and deploys them in a matter of seconds.

👷‍♂️ To-Don't example

Our To-Don't application's template.yaml looks like this, with added comments explaining what's going on:

AWSTemplateFormatVersion: "2010-09-09"
Description: >-
  to-dont

Transform: AWS::Serverless-2016-10-31

# Default Lambda function config
Globals:
  Function:
    Timeout: 10
    MemorySize: 512
    Runtime: nodejs16.x

Resources:
  persistFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: src/persist.handler
      # Set an environment variable with the DynamoDB table name, 
      # based on the reference to the table defined below
      Environment:
        Variables:
          TABLE_NAME: !Ref DynamoTable
      Policies:
        # Give Create/Read/Update/Delete Permissions to the DynamoDB table
        - DynamoDBCrudPolicy:
            TableName: !Ref DynamoTable
      # Expose the function for POST requests on /item in an ApiGateway
      Events:
        Api:
          Type: Api
          Properties:
            Path: /item
            Method: POST
  fanoutFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: src/fanout.handler
      # Set an environment variable with the SNS topic ARN
      Environment:
        Variables:
          TOPIC_ARN: !Ref SnsTopic
      # Give the function permission to publish to the SNS topic
      Policies:
        - SNSPublishMessagePolicy:
            TopicName: !GetAtt SnsTopic.TopicName
      # Trigger the function when a new item is added to the DynamoDB
      # table by listening to the DynamoDB stream
      Events:
        Stream:
          Type: DynamoDB
          Properties:
            Stream: !GetAtt DynamoTable.StreamArn
            StartingPosition: LATEST
            BatchSize: 10

  DynamoTable:
    Type: AWS::DynamoDB::Table
    Properties:
      BillingMode: PAY_PER_REQUEST
      AttributeDefinitions:
        - AttributeName: pk
          AttributeType: S
      KeySchema:
        - AttributeName: pk
          KeyType: HASH
      StreamSpecification:
        StreamViewType: NEW_AND_OLD_IMAGES
  SnsTopic:
    Type: AWS::SNS::Topic

Outputs:
  WebEndpoint:
    Description: "API Gateway endpoint URL for Prod stage"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/"

You can find the full AWS SAM version of our To-Dont app here.

⚖️ Pros and Cons

Pros:

SAM Sync greatly speeds up the development flow.
It can be combined with AWS CDK.
Rapidly evolving

Cons:

Verbose config
No plugin system or reusable components

📚 Resources

Getting started with AWS SAM

Serverless Application Repository

SAM Serverless Patterns Collection

AWS CDK

AWS CDK, Cloud Development Kit, is very different from our previous two candidates. It takes an imperative approach and lets you write your infrastructure code in the same language that you already use to write the rest of the application, and then synthesizes the code to CloudFormation to deploy it. A CDK app consists of building blocks called constructs, and these constructs can be shared and re-used. A perhaps unexpected aspect is that the nature of how you build your CDK apps makes it very easy to write actual unit tests for your infrastructure.

It is, however, not strictly meant for building serverless applications, which means that the CLI does not include the kind of utility functionality around your development that other, more focused frameworks provide. But on the other hand, you can combine CDK with SAM to fill the gaps!

CDK is a fantastic tool for managing your AWS infrastructure, but it does lack some quality-of-life features and development utilities that are handy for serverless development.

🛠 Working with AWS CDK

You can install CDK globally by running npm install -g aws-cdk.

When you initialize a CDK project by running cdk init you'll be asked what language you want to use to define your infrastructure. A minimal project in TypeScript may look like this:

A /bin/my-app.ts file that serves as the entry point of your application and creates the stack(s) that defines your application:

#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
import { MyAppStack } from '../lib/my-app-stack';

const app = new cdk.App();
new MyAppStack(app, 'MyAppStack', {});

and then a lib/my-app-stack.ts that defines what resources your app is built from:

import * as cdk from 'aws-cdk-lib';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import { Construct } from 'constructs';

export class MyAppStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const hello = new lambda.Function(this, 'HelloHandler', {
      runtime: lambda.Runtime.NODEJS_16_X,
      code: lambda.Code.fromAsset('src'),
      handler: 'hello.handler'
    });
  }
}

Given the above CDK app and configured AWS credentials, running cdk deploy will deploy a CloudFormation stack with a ‘hello’ Lambda function.

When you change your infrastructure code, you can run cdk diffto preview what changes would be made if it were deployed, and running cdk deploy --watch will watch your files for changes and deploy them automatically.

A neat thing about the --watch feature is that if the changes don't need a CloudFormation update, such as modifying Lambda function code, it'll skip the CloudFormation update and update the resource directly - greatly speeding up the workflow.

👷‍♂️ To-Don't example

The bin/to-dont.ts file in our To-Don't project looks like this:

#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
import { ToDontStack } from '../lib/to-dont-stack';

const app = new cdk.App();
new ToDontStack(app, 'CdkStack', {});

and the lib/to-dont-stack.ts, a bit meatier, looks like this:

import { Construct } from 'constructs';
import * as path from 'path';
import * as cdk from 'aws-cdk-lib';
import * as dynamodb from 'aws-cdk-lib/aws-dynamodb';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as apigateway from 'aws-cdk-lib/aws-apigateway';
import * as sns from 'aws-cdk-lib/aws-sns';
import { NodejsFunction } from 'aws-cdk-lib/aws-lambda-nodejs';

export class ToDontStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Create our DynamoDB table
    const ddb = new dynamodb.Table(this, 'table', {
      billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
      partitionKey: {
        name: 'pk',
        type: dynamodb.AttributeType.STRING,
      },
      stream: dynamodb.StreamViewType.NEW_AND_OLD_IMAGES,
    });

    // Create our SNS topic
    const topic = new sns.Topic(this, 'topic');

    // Create our API Gateway
    const api = new apigateway.RestApi(this, 'api');

    // Create the 'persist' Lambda function
    const persistFunction = new NodejsFunction(this, 'persistFunc', {
      // set the entry point to the function
      entry: path.join(__dirname, '../src/functions/persist.ts'),
      // set the DynamoDB table name as an environment variable
      environment: {
        TABLE_NAME: ddb.tableName,
      },
    });

    // grant the persist function read/write permissions to the DynamoDB table
    ddb.grantReadWriteData(persistFunction);

    // Create the 'fanout' Lambda function
    const fanoutFunction = new NodejsFunction(this, 'fanoutFunction', {
      entry: path.join(__dirname, '../src/functions/fanout.ts'),
      environment: {
        TOPIC_ARN: topic.topicArn,
      },
    });
    // grant the fanout function publish permissions to the SNS topic
    topic.grantPublish(fanoutFunction);

    // grant the fanout function Stream read permissions to the DynamoDB table
    ddb.grantStreamRead(fanoutFunction);

    // Add a DynamoDB stream event source to the fanout function
    fanoutFunction.addEventSourceMapping('mapping', {
      eventSourceArn: ddb.tableStreamArn,
      startingPosition: lambda.StartingPosition.TRIM_HORIZON,
      batchSize: 10,
    });

    // Map the POST /item endpoint to the persist function
    const itemResource = api.root.addResource('item');
    itemResource.addMethod(
      'POST',
      new apigateway.LambdaIntegration(persistFunction, { proxy: true }),
    );
  }
}

You can find the full AWS CDK version of our To-Dont app here.

⚖️ Pros and Cons

Pros:

Use the same language to define your infrastructure as your application code (TypeScript, JavaScript, Python, Java, C# or Go)
Lets you create reusable constructs to share publicly or within your organization.
Easily write unit tests for your infrastructure.

Cons:

Every CDK app is going to look at least slightly different from any other CDK app, which makes it harder to understand a new project quickly.
Using high-level constructs that abstract away the nitty-gritty can accelerate your workflow, and it's easy to fall into the trap of abstracting away too much. Be careful; you're the one that owns the deployed infrastructure, and you need to understand what's actually getting deployed to do that responsibly.

📚 Resources

Getting started with the AWS CDK

Getting started with AWS SAM and the AWS CDK

CDK Serverless Patterns Collection

SST

SST is a batteries-included framework built on top of AWS CDK with a strong focus on developer experience. It includes high-level constructs for setting up common serverless resources, lets you build and deploy your frontend app in the same stack, and has a feature called Live Lambda that - hold on to your chai latte - lets you attach a debugger to a deployed Lambda function running in AWS.

SST does a great job at improving the developer experience and helps with the transition to deploying early and often as part of your development flow. And since SST uses CDK, you can use any CDK constructs in your applications, which means you benefit from the developments and community of that project too.

🛠 Working with SST

You can initialize a new SST application by running npm create sst@latest which will let you choose from a few different starter examples.

A simple SST application could look like this:

A stacks/index.ts that defines the entry point for our application, some default options, and which stack(s) are in it:

import { MyStack } from "./MyStack";
import { App } from "@serverless-stack/resources";

export default function (app: App) {
  app.setDefaultFunctionProps({
    runtime: "nodejs16.x",
    srcPath: "services",
    bundle: {
      format: "esm",
    },
  });
  app.stack(MyStack);
}

and then a Stack,stacks/MyAppStack.ts, which includes the constructs that make up the application:

import { StackContext, Function } from "@serverless-stack/resources";

export function MyStack({ stack }: StackContext) {
  new Function(stack, "helloFunction", {
    handler: "functions/hello.handler",
  });
}

Given the above configuration, npx sst deploy will deploy a ‘hello’ Lambda function. Runningnpx sst start similarly first deploys the application and starts to watch your files for changes and re-deploys the application when needed. If the changes don't require a CloudFormation update, like a Lambda function code update, the changes are instantly (no really, instantly) updated. While the debug session is active, your CloudWatch logs will be streamed directly to your CLI.

But perhaps the killer feature of this framework is that debug session allows you to attach an actual debugger to the process, which means you can use breakpoints and step through your code line-by-line - on a Lambda function running in AWS. Since the function, again, is actually running in AWS, you avoid a whole slew of problems that usually comes with trying to emulate Lambda, and other AWS services, locally.

👷‍♂️ To-Don't example

Thestacks/index.ts in our To-Don't application looks like this:

import { ToDontStack } from "./ToDontStack";
import { App } from "@serverless-stack/resources";

export default function (app: App) {
  // set default props for all functions
  app.setDefaultFunctionProps({
    runtime: "nodejs16.x",
    srcPath: "services",
    bundle: {
      format: "esm",
    },
  });

  // add our ToDontStack to the application
  app.stack(ToDontStack);
}

and the stacks/ToDontStack.ts looks like this:

import {
  Api,
  StackContext,
  Table,
  Topic
} from "@serverless-stack/resources";

export function ToDontStack({ stack }: StackContext) {
  // create our SNS topic
  const topic = new Topic(stack, "topic");

  // create our DynamoDB table
  const ddb = new Table(stack, "table", {
    fields: {
      pk: "string",
    },
    primaryIndex: { partitionKey: "pk" },
    stream: "new_and_old_images",
    consumers: {
      fanout: {
        function: {
          handler: "functions/fanout.handler",
          // bind the SNS Topic to the function so we can access it
          // typesafely from the Lambda function instead of env vars
          bind: [topic],
        },
      }
    }
  });

  // create our ApiGateway
  const api = new Api(stack, "api", {
    routes: {
      // map POST requests to /item to the persist function
      "POST /item": {
        // create the persist function
        function: {
          handler: "functions/persist.handler",
          bind: [ddb],
        },
      },
    },
  });

  // add a CloudFormation output for the API endpoint
  stack.addOutputs({
    ApiEndpoint: api.url,
  });
}

You can find the full SST version of our To-Dont app here.

⚖️ Pros and Cons

Pros:

Live lambda debug really is a gamechanger
Useful abstractions for common serverless resources
Great development experience with fast deploys

Cons:

Every SST app is going to look, at least slightly, different from any other SST app, which makes it harder to understand a new project quickly
Using high-level constructs that abstract away the nitty-gritty can accelerate your workflow, and it's easy to fall into the trap of abstracting away too much. Be careful; you're the one that owns the deployed infrastructure, and you need to understand what's actually getting deployed to do that responsibly.

📚 Resources

SST Quick Start

Live Lambda Development

CDK Serverless Patterns Collection

🥈Honorable mentions - niche & generic options

There are a couple of fairly common options for building serverless AWS apps that I've chosen not to include in this list, either because they're more generic and aren't focused on AWS or because they are more niche and focus on a subset of services or developers. These tools are all excellent at what they do though, so here are a few honorable mentions that didn't make the list, but may still be the best choice for you, depending on your type of application, your organization's policies and standards, or your preferences:

Terraform

Terraform is a widely used Infrastructure-as-Code tool with open-source modules available for virtually any API, including AWS. In contrast to all previously discussed tools, Terraform does not produce and deploy CloudFormation. Instead, it talks directly to the AWS API and keeps track of its state on its own.

Pulumi

Pulumi is another universal Infrastructure-as-Code tool, but instead of using a DSL, you can use Python, JavaScript, YAML, or other familiar programming- and markup languages. Like Terraform, Pulumi manages its own state instead of using CloudFormation.

Architect

Architect is a heavily opinionated framework for building FWA's, Functional Web Apps. It uses AWS SAM under the hood but provides a layer on top with simplified abstractions that lets developers define and use AWS infrastructure without necessarily knowing what service is backing their "events" construct.

AWS Amplify

Amplify is a tool meant to let frontend- and mobile developers build full-stack apps on AWS. It, among other things, features a CLI and a web console that helps you build out your backend and takes care of hosting and deployment. It's an excellent tool for rapid prototyping and building mobile apps.

☑️ Summary

In conclusion, there are a bunch of different serverless frameworks available to developers today, each with its own set of features and capabilities. In this article, we looked at some of the most popular frameworks and demonstrated how they could be used to build serverless applications. We also compared their features and discussed their pros and cons.

It's worth noting that the serverless space is rapidly evolving and changing. New features and capabilities are being added all the time, and new frameworks are constantly emerging.

CDK has been disrupting the space in the last years, but Infrastructure from Code is another interesting new concept with tools such as Ampt (previously Serverless Cloud) reimagining how we think about infrastructure (if you're interested in this concept, Allen Helton has a great post on it here).

I'm excited to see what the future holds for serverless and the tooling around it. Next year's version of this blog post is sure to be even more exciting as the space continues to grow and evolve. Keep an eye out for new developments, and don't be afraid to experiment with different frameworks to see which one works best for your particular use case!

Hi there, I'm Sebastian Bille! If you enjoyed this post or just want a constant feed of memes, AWS/serverless talk, and the occasional new blog post, make sure to follow me on Twitter at @TastefulElk or on LinkedIn 👋

Elva is a serverless-first consulting company that can help you transform or begin your AWS journey for the future

Separate stateful infrastructure with Serverless Compose

Sebastian Bille — Tue, 05 Jul 2022 08:10:12 +0000

One of the most significant benefits of serverless is that the development teams can fully own and manage all of their infrastructure themself. Keeping all of that infrastructure in the same CloudFormation stack can sometimes be troublesome however, and there are many reasons why you might consider having parts of it in a separate stack. A common approach is to keep stateful infrastructure, such as your DynamoDB tables or S3 buckets, in one stack, and stateless infrastructure, such as your Lambda functions or API gateways, in another. This way, you can more freely update or even tear down the stack that's naturally often changing while not risking affecting the long-living infrastructure that your application depends on. But on the other hand, that can lead to complex deployment orchestration or overcomplicated mono repo setups just to be able to deploy your stacks in the correct order with the right dependencies. This post will show you how you can use Serverless Compose to easily separate your stateful infrastructure into a separate stack and deploy it without the hassle!

♫ Serverless Compose

Serverless Compose is a feature of the Serverless Framework that simplifies orchestration and configuration when you have multiple related CloudFormation stacks. Apart from providing a simple configuration format for orchestrating and defining the services and their respective dependencies, it also provides CLI tooling to interact with those services once deployed, for example to stream the logs of those services directly to your stdout at the same time.

🎹 Composing an Application

The first thing we're going to need is a serverless-compose.yml file in the root of our project. This file will contain the configuration that instructs Serverless Compose what services (CloudFormation stacks) our application will consist of and how their deployments should be orchestrated. In an empty directory, create a file called serverless-compose.yml and list two services in it, one called infrastructure and one called application.

services:
  infrastructure:
    path: infrastructure

  application:
    path: application

The infrastructure service will contain our stateful infrastructure, which in this case will be a DynamoDB table, while the application service will contain stateless resources, in this case, a Lambda function.

Let's start by implementing the infrastructure service by creating a subdirectory called infrastructure. Inside that directory, add a serverless.yml file and add the configuration to set up a DynamoDB table:

service: infrastructure

frameworkVersion: '3'

provider:
  name: aws

resources:
  Resources:
    DynamoTable:
      Type: AWS::DynamoDB::Table
      Properties:
        AttributeDefinitions:
          - AttributeName: id
            AttributeType: S
        KeySchema:
          - AttributeName: id
            KeyType: HASH
        BillingMode: PAY_PER_REQUEST
        TableName: dunder-mifflin-clients
  Outputs:
    TableName:
      Value: !Ref DynamoTable

Note that we've added an Outputs field that contains the name of the DynamoDB table. We can use this output to easily pass parameters between our services without having to specify the same properties in multiple places or storing them in a parameter store. To inject this property as a parameter in the application service, add a parameters field to the service definition in the root serverless-compose.yml file, referring to the output of the infrastructure stack

services:
  infrastructure:
    path: infrastructure

  application:
    path: application
    params:
      tableName: ${infrastructure.TableName}

Next, it's time to implement the application service. Again, in the root of the project, create a subdirectory called application and add a serverless.yml file:

service: application

frameworkVersion: '3'

provider:
  name: aws
  runtime: nodejs14.x

functions:
  hello:
    handler: handler.hello
        environment:
        TABLE_NAME: ${param:tableName}

This configuration defines a Lambda function called "hello" that takes a TABLE_NAME as an environment variable. That variable is filled by the parameter we injected with the value from our infrastructure service.

To complete the application service, add the Lambda function handler.js

module.exports.hello = async (event) => {
  return {
    statusCode: 200,
    body: JSON.stringify(
      {
        message: `I see you ${process.env.TABLE_NAME} 👀`,
        input: event,
      },
      null,
      2
    ),
  };
};

Finally, we don't want to risk our application service accidentally being deployed before our infrastructure service in case we do any changes to the DynamoDB table that in turn requires code changes to our application service. We can make sure we always deploy the infrastructure service first by specifying that the application service depends on the infrastructure service:

services:
  infrastructure:
    path: infrastructure

  application:
    path: application
    params:
      tableName: ${infrastructure.TableName}
    dependsOn: infrastructure

We're now ready to deploy our application! In the root of the project, run npx serverless deploy 🚀

If you enjoyed this post and want to see more, follow me on Twitter at @TastefulElk
where I frequently write about serverless tech, AWS, and developer productivity!

This post was originally published on the Elva blog. Elva is a serverless-first consulting company that can help you transform or begin your AWS journey for the future.

⚡️Combining Serverless Framework & AWS CDK

Sebastian Bille — Mon, 25 Oct 2021 21:51:11 +0000

Serverless Framework is a fantastic tool for defining and managing your Lambda functions. But let's face it; it's not very good at handling other types of resources and infrastructure. CDK is though. This post will take a look at how we can combine the two!

Serverless has excellent utility for describing and managing Lambda functions, but as soon as you need basically any other type of infrastructure, you'll have to resort to raw CloudFormation. Using the CDK with its declarative and straightforward constructs to define infrastructure is more expressive and easy to understand once the infrastructure stack grows. What if we could use them together and utilize each of them for its respective strengths, and what it does best? Unfortunately, there's no native way to use the two tools together, like there is for SAM.

Let's look at how we can work around that and whether or not it makes sense!

TL;DR - give me the code!

🏗 Setting up the project

Let's start by initializing a Typescript Serverless app. To keep things simple, we'll use a Yeoman generator to get us started with a project. Install it with:

npm install -g yo generator-sls-node

and then, in an empty folder, run the generator with:

yo sls-node

Since we'll be using CDK to manage our infrastructure, we'll want to manage CDK, its dependencies, and transpiling the infrastructure code separately from our actual application. Therefore, let's create a folder called infrastructure in the root of our project.

mkdir infrastructure

Next, navigate to the created folder and initialize a CDK project:

cd infrastructure
npx cdk init --language typescript

You should now have a project with the following structure:

- infrastructure # Our CDK infrastructure
    ↳ bin     # CDK executable
    ↳ lib     # infrastructure stack definition
    ↳ test    # infrastructure tests
- src                    # application code
- tests                  # application tests
- serverless.yml # Serverless config

🔨 Building our app

The app we're building will contain a Lambda function, exposed in a HTTP API, which will put an item on an SQS queue for another Lambda to finally process the item.

Let's start by adding an SQS queue to our CDK stack. To do that, we need to first add the @aws-cdk/aws-sqs package:

yarn add @aws-cdk/aws-sqs

I prefer Yarn over NPM, which is the default of the CDK init boilerplate. If you prefer to use Yarn too, remove the package-lock.json so you don't get conflicting lock files.

Then add a queue to infrastructure/lib/infrastructure-stack.ts

// infrastructure/lib/infrastructure-stack.ts
import * as cdk from '@aws-cdk/core';
import { Queue } from '@aws-cdk/aws-sqs';

export class InfrastructureStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const sqs = new Queue(this, 'SQS', {
      queueName: 'sls-cdk-demo-queue'
    });
  }
}

As discussed earlier, we want to use Serverless and CDK for what they do best respectively. That means our API and Lambda functions will be defined in our serverless.yml, seeing as Serverless handles those very capably and brings lots of functionality to interact with them once deployed. They're also the resources most likely to change often, meaning they don't really belong in our more long-living infrastructure stack.

For this simple demo use case, all we need to do to set up our API Gateway, Lambda function and to map the function to the /todo path, is to add the following to our serverless.yml, using the events mapping:

# serverless.yml
...
functions:
  createTodo:
    handler: src/createTodo.handler
    events:
      - http:
          path: /todos
          method: post

For the createTodo function to send anything to our SQS buffer, it's going to need the URL of the queue. It's also going to need the queue's ARN to set up the necessary sqs:SendMessage IAM permission. However, since the SQS queue is not defined in the same CloudFormation stack as our Lambda function, we'll need to export those two properties from our CDK stack so that we can reference them in our Serverless stack.

Cross Stack References

CloudFormation stacks can export a set of properties, as "Outputs" which can then be referenced in other CloudFormation stacks. Conveniently, Serverless has built-in support to reference Outputs of other stacks, which means we can use this to easily transfer information from our CDK stack to our Serverless stack.

In the CDK stack, we start by creating our output variables:

// infrastructure/lib/infrastructure-stack.ts
...
new cdk.CfnOutput(this, 'queueUrl', {
  value: sqs.queueUrl
});

new cdk.CfnOutput(this, 'queueArn', {
  value: sqs.queueArn
});

Now, back in our serverless.yml, we can reference those two variables with the ${cf:stackName.variableName} syntax:

# serverless.yml
...
provider:
  name: aws
  region: eu-north-1
  runtime: nodejs14.x
  iam: # Set up IAM permission for sending messages to the queue
    role:
      statements:
        - Effect: Allow
          Action: sqs:SendMessage
          Resource: ${cf:InfrastructureStack.queueArn}

functions:
  createTodo:
    handler: src/createTodo.handler
    events:
      - http:
          path: /todos
          method: post
    environment: # Set environment variable with the queue URL
      QUEUE_URL: ${cf:InfrastructureStack.queueUrl}

and while we're at it, lets also add the definition for our second Lambda, the queue processor, again by using the cross stack reference to the queue ARN:

# serverless.yml
...
functions:
    createTodo:
    ...
    queueProcessor:
        handler: src/queueProcessor.handler
        events:
      - sqs:
          arn: ${cf:InfrastructureStack.queueArn}

That should be all we need for the infrastructure, all that's left is writing the actual Lambda code.

Lambda Code

Our Lambda code is going to be very simple. The createTodo handler, the one exposed in the API, will take the body of the request and put it directly on the SQS queue. The queueProcessor will then automatically consume the queue. Starting with the createTodo function, add the following:

// src/createTodo.js
import { SQS } from 'aws-sdk';

export const handler = async (event) => {
  const sqs = new SQS();

  const payload = event.body || '';

  await sqs
    .sendMessage({
      QueueUrl: process.env.QUEUE_URL,
      MessageBody: payload,
    })
    .promise();
};

and then let's just pretend to do some processing of the item in the queueProcessor.js:

// src/queueProcessor.js
export const handler = (event) => {
  console.log('Doing some heavy processing right now. I promise! Don\'t come in!', event);
};

That's it, now all we need is to be able to deploy our app along with the infrastructure!

🚀 Deploying

Since our application will live in two separate CloudFormation stacks, we need to do two separate deployments. The Serverless stack will depend on our infrastructure stack, but never the other way around - meaning we should always deploy the infrastructure stack first.

To be able to deploy our app in one step, rather than manually first having to navigate to the infrastructure folder, deploy it, navigate back to the project root and then deploy our Serverless app, we can add a script to our root package.json that does us for us in one swift motion:

// package.json
...
"scripts": {
  "deploy": "cd infrastructure && yarn cdk deploy && cd .. && yarn sls deploy",
...

We can also add a CDK wrapper script that lets us easily run any CDK command without having to navigate to the infrastructure folder.

// package.json
...
"scripts": {
    "cdk": "cd infrastructure && yarn cdk",
...

The cdk script utilizes the fact that Yarn forwards arguments, and thus we can run, for example, yarn cdk diff from the root of our app, even though CDK isn't actually in the root. Neat! 👵

What's even neater, though, is that we can now just run yarn deploy. Since it's the first time we're deploying the app, it'll take a little longer than on subsequent deploys but after about half the time it takes refill your coffee, the app will be up and running! We can verify that it works by sending a POST request to the API Gateway endpoint printed in the output, and by then looking at the logs of the final queueProcessor lambda:

curl -X POST -H 'Content-type: application/json' --data '{"text":"Hello, World!"}' https://abcdefghij.execute-api.eu-north-1.amazonaws.com/dev/todos

yarn serverless logs --function queueProcessor

Conclusions

CDK is excellent at defining and making sense of complex infrastructure and resources. However it does come with quite a bit of overhead. The CDK project itself has dependencies that you'll need to manage, and, most importantly; the infrastructure code is, well, code, which you'll need to maintain, lint and test. On the other hand, being able to test it, is definitely also one of the strengths. It's also generally a good idea to separate your more long-living infrastructure into a separate CloudFormation stack. It minimizes the risk of accidentally modifying, or even deleting, important stateful resources such as a DynamoDB table.

There are ongoing projects that tries to bring CDK constructs natively right into Serverless via plugins, one example being Lift, and other frameworks that are built on top of the CDK, such as Serverless Stack. Lift looks promising, but it doesn't yet support using custom constructs, and many organizations and projects have invested a lot into Serverless Framework and aren't ready to jump ship to an alternative like Serverless Stack.

For smaller projects, or projects where the infrastructure and/or the interaction between the infrastructure isn't complex, using CDK and Serverless together is unlikely to be worth the overhead. For larger, more complex apps, though, where you might want to unit test your resources or just be able to describe them better than with CloudFormation, which quickly gets very messy - it might be worth considering!

If you enjoyed this post and want to see more, follow me on Twitter at @TastefulElk where I frequently write about serverless tech, AWS, and developer productivity!

⚡️ 6 Serverless CLI Commands You Didn't Know Existed

Sebastian Bille — Tue, 24 Aug 2021 20:57:31 +0000

The Serverless Framework CLI can do a lot more than just deploy your app. Here are a few more ways you can use it in your everyday work that might not be obvious at first sight!

1. Bootstrap a Project

The CLI can help you initialize and bootstrap your new project, letting you choose between quite a few different project types and languages.

Running just serverless lets you configure your project through a setup wizard with the most common project types and leads you all the way up to deploying the project for the first time.

serverless create is similar but instead expects you to provide it with either the name of a built-in project starter or an external one from Github or Bitbucket. Example usage: serverless create --template aws-nodejs-typescript.

CLI Reference

If you want a slightly more fully-fledged starter project that includes testing, linting & optional TS support with esbuild, feel free to check out my Yeoman generator or my post on how to set up lightning fast builds with esbuild instead!

Lightning fast & simple Typescript Serverless builds

Sebastian Bille for AWS Community Builders ・ Jul 1 '21

#serverless #typescript #tutorial #aws

2. Generate Mock Events

Have you ever deployed a project, published a message to the SNS topic, gone to CloudWatch, scoured the logs for the right output from your Lambda - just so that you could copy the SNS payload to use for local debugging or as a test event? I sure have.

As it turns out, there's a much easier way! The Serverless CLI can help generate mock events from SNS, SQS, DynamoDB, and many other service integrations, just by running serverless generate-event

CLI Reference

3. Install tab-completion

Install tab-completion to get the CLI to help you suggest available parameters and subcommands!

Currently supported shells are fish, bash, and zsh.

CLI Reference

4. Invoke Deployed Functions

How to invoke Lambda functions locally, with the popular serverless-offline plugin, or by running serverless invoke local is something that I think new serverless devs usually pick up on pretty fast.

Something that's surprisingly easy to overlook, however, is that the "invoke" command can also be used to invoke already deployed functions. This can be incredibly helpful in shortening the feedback loop when debugging those pesky issues that never seem to happen on your local machine!

Simply run serverless invoke --function myFunction, optionally including a payload, to run the deployed function!

CLI Reference

5. Print or Tail Logs

Another nifty feature to further shorten the feedback loop is the ability to print or tail the logs of a deployed Lambda function to avoid having to open up the console for the CloudWatch logs. With this, you can even tail multiple function logs at the same time, side by side, by running the command in more than one window.

serverless logs --function myFunction [--tail]

CLI Reference

6. Print Metrics

Print the metrics for the service or a given function to get a glance at how it's performing, right from the comfort of your CLI. This could for example be used to quickly figure out whether or not there have been any errors in any of the functions in the service or if the average execution time is creeping up.

serverless metrics [--function]

CLI Reference

If you enjoyed this post and want to see more, follow me on Twitter at @TastefulElk where I frequently write about serverless tech, AWS, and developer productivity! This post even started out as a thread over there!

Step Functions Workflow Studio with Serverless Framework

Sebastian Bille — Wed, 11 Aug 2021 07:02:06 +0000

The serverless-step-functions plugin is a nifty Serverless Framework extension that allows you to define step functions directly within the serverless.yml. However, the definitions for step functions tend to get messy fast and it's hard to visualize how a workflow actually looks and flows just from looking at the YAML. The Step Functions Workflow Studio helps you with exactly this by allowing you to interactively sketch out your state machine. In this post, I'll give you a brief walkthrough of how we can use the Workflow Studio together with the Step Functions plugin.

Creating a State Machine

To get started, navigate to the Step Functions page in the AWS Console and press "Create state machine".

Next, choose "Design your workflow visually", then go to Next.

From here, you can drag and drop Actions and Flows, tinker with the properties, flow, and error handling until you're happy with the design.

Set any resources that you want to eventually define in your serverless app to refer to a made-up value in the meantime - setting the Lambda Function name to just a string name for example. Once you're done, click "Next".

At last, we have our generated JSON schema for the state machine! But wait. Our Serverless template is in YAML, not JSON - how do we translate this into something that we can actually use?

As it turns out, YAML is just a superset of JSON, meaning all JSON is valid YAML. The JSON that was generated is in the form of Amazon State Languages spec, and that same spec is used by our serverless-step-functions plugin. This means that we can translate our JSON snippet to YAML using an online tool, such as Code Beatify, or a CLI tool such as yq.

Now that we've got the state machine defined in YAML, all we need to do is actually use it in the serverless.yml.

Putting it together

If you don't already have a serverless project set up, check out my guide to a simple esbuild setup or just run serverless in your terminal to choose from one of the boilerplates.

The serverless-step-functions plugin extends the serverless.yml format with a root stepFunctions property. We'll use that to define our step function by pasting in our YAML snippet as the definition of the state machine.

// serverless.yml
[...]

stepFunctions:
  stateMachines:
    myStateMachine:
        name: MyStateMachine
        [your generated state machine goes here]

Keep in mind that any references to Lambda functions or other services that we used placeholder references for needs to be updated to refer to the actual resources we want them to point to. As an example, a Lambda definition might look like this when generated:

FirstFunction:
  Type: Task
  Resource: arn:aws:states:::lambda:invoke
  OutputPath: $.Payload
  Parameters:
    Payload.$: $
    FunctionName: firstFunction

and we can change that to instead refer to a firstFunction Lambda defined in our stack by changing it to:

FirstFunction:
  Type: Task
  Resource: !GetAtt firstFunction.Arn
  OutputPath: "$.Payload"
  Parameters:
        Payload.$: "$"

For the workflow from the examples above, this is what the full serverless.yml might look like.

service: step-functions-demo

plugins:
  - serverless-step-functions

provider:
  name: aws
  runtime: nodejs14.x

stepFunctions:
  stateMachines:
    myStateMachine:
      name: MyStateMachine
      definition:
        Comment: This is your state machine
        StartAt: First function
        States:
          First function:
            Type: Task
            Resource: !GetAtt firstFunction.Arn
            OutputPath: "$.Payload"
            Parameters:
              Payload.$: "$"
            Retry:
            - ErrorEquals:
              - Lambda.ServiceException
              - Lambda.AWSLambdaException
              - Lambda.SdkClientException
              IntervalSeconds: 2
              MaxAttempts: 6
              BackoffRate: 2
            Next: Wait
          Wait:
            Type: Wait
            Seconds: 5
            Next: Second function
          Second function:
            Type: Task
            Resource: !GetAtt secondFunction.Arn
            OutputPath: "$.Payload"
            Parameters:
              Payload.$: "$"
            Retry:
            - ErrorEquals:
              - Lambda.ServiceException
              - Lambda.AWSLambdaException
              - Lambda.SdkClientException
              IntervalSeconds: 2
              MaxAttempts: 6
              BackoffRate: 2
            End: true

functions:
  firstFunction:
    handler: src/functions/first.handler
  secondFunction:
    handler: src/functions/second.handler

If you enjoyed this post and want to see more, follow me on Twitter at @TastefulElk where I frequently write about serverless tech, AWS, and developer productivity! 👋

Lightning fast & simple Typescript Serverless builds

Sebastian Bille — Thu, 01 Jul 2021 21:42:03 +0000

Slow builds tend to cost more than just the time they waste. More than a few seconds and you're off checking Slack and all of a sudden it's been a few minutes. Any longer and all of a sudden you've been on Reddit for 20 minutes. Having a fast build means you've got a faster feedback loop and less risk of having to context switch due to getting distracted.

esbuild is "An extremely fast JavaScript bundler" and in this tutorial, I'm going to show you how we can accomplish blazingly fast builds for Typescript Serverless Framework projects, all while drastically reducing the configuration complexity compared with the alternatives.

Demo Project

Let's start in an empty folder by running npm init -y to initialize a new project. Next, we create a function that will act as our Lambda handler, in src/function.ts:

// src/function.ts
export const handler = async (event) => {
  console.log(event);
  return {
    status: 200
  };
};

Now since we'll be using Serverless and esbuild to package and deploy our app, we'll need to install Serverless and the Serverless esbuild plugin: npm install serverless serverless-esbuild --save-dev

Finally, to deploy our app all we need to do is create a serverless.yml

# serverless.yml
service: esbuild-demo

plugins:
  - serverless-esbuild

provider:
  name: aws
  runtime: nodejs14.x

functions:
  function:
    handler: src/function.handler

and now we can run npx serverless deploy to get our app up and running in AWS - transpiled, three shaken & ready to rock. No additional configuration is necessary but you can of course choose to configure esbuilds behavior if needed. The transpile target is chosen automatically from the Lambda runtime from the Serverless provider setting, but it will also automatically discover and respect tsconfig.json if you have it.

Setting up unit testing with Jest is almost just as simple. First, we need to add Jest, Jest types, and the Jest esbuild transformer:

npm install jest esbuild-jest @types/jest --save-dev

and then configure Jest to use the esbuild transformer:

// jest.config.json
{
  "transform": {
    "^.+\\.(j|t)sx?$": "esbuild-jest"
  }
}

We can now also write Typescript unit tests:

// tests/function.test.ts
import { handler } from '../src/function';

describe('[function]', () => {
  it('should return status 200', async () => {
    const res = await handler(null)
    expect(res).toEqual({
      status: 200
    });
  });
});

and run them with npx jest!

Summary

Having migrated a few more or less complicated projects from Webpack setups, esbuild tend to "just work" every time as a drop-in replacement. It provides significantly faster builds all while requiring a fraction of the config! 🚀

You can find the complete demo project here!

If you enjoyed this post and want to see more, follow me on Twitter at @TastefulElk where I frequently write about serverless tech, AWS, and developer productivity!

Serverless Framework vs SAM vs AWS CDK

Sebastian Bille — Fri, 07 May 2021 13:24:30 +0000

ℹ️ Edit: An updated version, with more options included, exists here

⚡️Serverless Frameworks for 2023

Sebastian Bille for AWS Community Builders ・ Jan 23 '23

#serverless #aws #cloud #productivity

When building serverless apps on AWS today there's a couple of different toolkits available that helps you develop, test and deploy your project. Serverless Framework was the king for a long time but both AWS SAM and the CDK have been increasing in popularity lately. But which one is the best to use in a new project and what's the difference between them anyway. After all, they're all just tools to produce Cloudformation templates anyway, right?

To get an understanding of the strengths and disadvantages of each option, I decided to build an identical example application across all three and compare the approaches.

By the end of this post, I hope you'll have a basic understanding of the Serverless Framework, AWS SAM, and the CDK and that you'll be able to make an educated choice on what'll suit your next project best based on your needs and preferences.

Our Example Application

To keep it interesting, the app we're using to showcase each framework certainly isn't your typical ToDo-app - it's a ToDont-app. A user can send a POST request to an API Gateway describing something they really shouldn't do, a Lambda function takes the ToDont-item and puts it on an SQS queue that acts as a buffer before finally another Lambda function consumes the buffer queue, pretends to do some heavy processing on the item and persists it in a DynamoDB table.

The application architecture is simple enough to easily comprehend but "complex" enough to resemble an actual app. To keep the code compact and readable, best practices and common sense have sometimes had to be omitted. All configs are complete and fully functional however and if you want to play around with the examples and deploy the apps yourself, you can find the code and the full examples here.

Our POST Lambda function looks like this

// src/post.js
const { SQS } = require('@aws-sdk/client-sqs');

const sqs = new SQS();

const handler = async (event) => {
  console.log('event', event);
  const { id, title } = JSON.parse(event.body);

  await sqs.sendMessage({
    QueueUrl: process.env.QUEUE_URL,
    MessageBody: JSON.stringify({
      id,
      title,
    })
  });

  return {
    statusCode: '200',
  };
};

module.exports = { handler };

the Process Lambda looks like this

// src/process.js
const { DynamoDB } = require('@aws-sdk/client-dynamodb');
const { marshall } = require("@aws-sdk/util-dynamodb");

const ddb = new DynamoDB();

const handler = async (event) => {
  console.log('event', event);

  const tasks = event.Records.map((record) => {
    const { id, title } = JSON.parse(record.body);
    return ddb.putItem({
      TableName: process.env.TABLE_NAME,
      Item: marshall({
        title,
        id,
      }),
    });
  });

  return Promise.all(tasks);
};

module.exports = { handler };

Prerequisites

If you want to follow along and deploy the apps, please note the following:

Each of the comparisons below assumes that you've installed the following packages as dependencies in your project
- @aws-sdk/client-dynamodb
- @aws-sdk/util-dynamodb
- @aws-sdk/client-sqs
While Yarn is used as the package manager & script runner below you could of course use NPM instead with the corresponding commands.
All of the examples assume that you've got an AWS credentials default profile configured

Serverless Framework

Serverless Framework ("Serverless" below) has been around for a long time now and has long been the preferred framework for a large part of the community. It's a simple tool that abstracts away and simplifies many of the nastier parts of CloudFormation and comes packed with features to simplify testing and deployment of your app.

The preferred way to run the Serverless CLI is to install it as a (dev)dependency in your project by running yarn add serverless -D and then all that's missing is a serverless.yml file which is used to define your application and its infrastructure. You can find the full configuration reference here but in short, the serverless.yml consists of two parts:

Your Serverless Framework configuration is used to describe your application stack, AWS environment, and lambda functions
Any additional infrastructure defined as CloudFormation resources, such as our DynamoDB table and SQS queue.

Here's how the serverless.yml for our application looks.

// serverless.yml
service: sls-todont

provider:
  name: aws
  region: eu-north-1
  runtime: nodejs14.x
  environment: # Inject environment variables
    TABLE_NAME: ${self:custom.tableName}
    QUEUE_URL: !Ref todontsQueue
  iamRoleStatements: # Configure IAM role statements
    - Effect: Allow
      Action: sqs:sendMessage
      Resource: ${self:custom.queueArn}
    - Effect: Allow
      Action: dynamodb:putItem
      Resource: ${self:custom.tableArn}

custom: # Custom variables that we can reference elsewhere
  tableName: ${self:service}-table
  queueName: ${self:service}-queue
  tableArn: # Get ARN of table with CloudFormation helper
    Fn::GetAtt: [todontsTable, Arn]
  queueArn: # Get ARN of queue with CloudFormation helper
    Fn::GetAtt: [todontsQueue, Arn]

functions: # Define our two Lambda functions
  post:
    handler: src/post.handler
    events: # Invoke on post requests to /todonts
      - http:
          method: post
          path: todonts
  process:
    handler: src/process.handler
    events: # Consume SQS queue
      - sqs:
          arn: ${self:custom.queueArn}

# CloudFormation below to define our infrastructure resources
resources: 
  Resources:
    todontsTable:
      Type: AWS::DynamoDB::Table
      Properties:
        TableName: ${self:custom.tableName}
        AttributeDefinitions:
          - AttributeName: id
            AttributeType: S
        KeySchema:
          - AttributeName: id
            KeyType: HASH
        BillingMode: 'PAY_PER_REQUEST'
    todontsQueue:
      Type: AWS::SQS::Queue
      Properties:
        QueueName: ${self:custom.queueName}

Now to deploy our application, all we need to do is run yarn serverless deploy.

The Serverless CLI includes some utility features that can be used to print or tail the logs of a deployed function by running yarn serverless logs --function process [--tail] or even invoke the function with yarn serverless invoke --function process. Most of the time during development, however, you're not going to be invoking the function. Instead, you'll let Serverless emulate and run the functions locally and you can do that by running yarn serverless invoke local --function post.

➕ Pros

Large & helpful community
The plugin ecosystem
Simple configuration with neat variable support
Great debugging and testing utilities

➖ Cons

Most apps will need to resort to CloudFormation definitions for some parts of the infrastructure
Hard to share configuration and components
Only YAML configurations. It's technically supported to write the configuration in JS but the documentation for it is close to non-existent
I've seen a lot of devs struggle with understanding where the line between Serverless configuration and CloudFormation configuration actually or why they have to change the syntax in the middle of the file

Resources:

Get started with Serverless Framework

Serverless Stack tutorial for deplying a production Serverless app

AWS SAM

Much like Serverless Framework, SAM (or the Serverless Application Model) is a combination of an abstraction layer to simplify CloudFormation and a CLI with utilities to test and deploy your app.

Here's the official install instructions for the SAM CLI which is installed globally on your system. SAM uses a samconfig.toml file to describe information about your app, such as the name and where and how it should be deployed, and a template.yml file to describe the actual resources your app will use.

The template.yml format follows the CloudFormation template anatomy templates but with a few added fields. Let's have a look:

// template.yml
# Boilerplate to identify template as SAM template
AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31
Description: sam-todonts

Globals:
  Function:
    Runtime: nodejs14.x
    Environment: # Inject environment variables
      Variables:
        QUEUE_URL: 
          Ref: TodontsQueue
        TABLE_NAME:
          Ref: TodontsTable

Parameters: # Parameters which can be filled by the CLI on deploy
  TableName: 
    Description: Name of DynamoDB table
    Type: String
    Default: sam-todonts-table
  QueueName:
    Description: Name of SQS queue
    Type: String
    Default: sam-todonts-queue

Resources: 
  PostFunction:
    Type: AWS::Serverless::Function
    FunctionName: sam-todonts-post
    Properties:
      Handler: src/post.handler
      Events:
        Post: #  Invoke on post requests to /todonts
          Type: HttpApi
          Properties:
            Path: /todonts
            Method: post
      Policies:
        - SQSSendMessagePolicy: # Use predefined IAM policy
            QueueName:
              Fn::GetAtt: [TodontsQueue, QueueName]

  ProcessFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: src/process.handler
      Events:  # Consume SQS queue
        SQSQueueEvent:
          Type: SQS
          Properties:
            Queue:
              Fn::GetAtt: [TodontsQueue, Arn]
      Policies: # Use predefined IAM policy
        - DynamoDBWritePolicy:
            TableName:
              Ref: TodontsTable

  TodontsTable:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: sam-todonts-table
      AttributeDefinitions:
        - AttributeName: id
          AttributeType: S
      KeySchema:
        - AttributeName: id
          KeyType: HASH
      BillingMode: PAY_PER_REQUEST

  TodontsQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: sam-todonts-queue

The configuration is a bit verbose but luckily the CLI can help you with a place to start by running sam init and answering a few questions about what your planning to build.

We can generate the samconfig.tomlfile and deploy at the same time by running sam deploy --guided.

Again, much like the Serverless Framework CLI, SAM comes loaded with utility features to test and debug your app. sam local invoke [functionName] can be used to run a Lambda function, or you can start a local HTTP server that hosts your function by running sam local start-api. You can also easily fetch the logs from a deployed function by running sam logs --name [functionName].

The great thing about separating the definition of the app and how the app is built in two different files is that the template.yml file can be written very generically so that it can be shared and re-used, you'll just have a different samconfig.toml in each project. SAM also integrates very well with CodeBuild to enable blue-green deployments.

➕ Pros

Enables sharing & re-use of templates
Well integrated with AWS build pipelines
Great debugging and testing utilities
Can be combined with CDK

➖ Cons

Verbose configuration
CLI is missing some features you'd expect, such as tearing down a deployed app.

Resources:

Getting started with AWS SAM

Serverless Application Repository

Serverless Patterns Collection

AWS CDK

The AWS Cloud Development Kit (CDK) isn't purely a tool for creating serverless apps, rather it's a full-blown infrastructure-as-code framework that allows you to use code, not config, to define your application.

You can install the CDK CLI by running yarn global add aws-cdk and then generate a starter project by running cdk init app --language --language typescript. There's a bunch of project configuration files and boilerplate that's generated when you run the init command but let's have a look at how the lib/cdk-stack.ts file looks like after we've described our ToDont-app in it.

// lib/cdk-stack.ts
import * as cdk from '@aws-cdk/core';
import lambda = require('@aws-cdk/aws-lambda-nodejs');
import sqs = require('@aws-cdk/aws-sqs');
import dynamodb = require('@aws-cdk/aws-dynamodb');
import { ApiEventSource, SqsEventSource } from '@aws-cdk/aws-lambda-event-sources';
import { Runtime } from '@aws-cdk/aws-lambda';

export class CdkStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // define our DynamoDB table
    const dynamoTable = new dynamodb.Table(this, 'cdk-todonts-table', {
      tableName: 'cdk-todonts-table',
      partitionKey: { name: 'id', type: dynamodb.AttributeType.STRING },
    });

    // define our SQS buffer queue
    const sqsBuffer = new sqs.Queue(this, 'cdk-todonts-queue', {
      queueName: 'cdk-todonts-queue',
    });

    // define our processing lambda
    const processLambda = new lambda.NodejsFunction(this, 'cdk-todonts-process', {
      runtime: Runtime.NODEJS_14_X,
      handler: 'handler',
      entry: 'src/process.js',
      events: [new SqsEventSource(sqsBuffer)],
      environment: {
        TABLE_NAME: dynamoTable.tableName
      }
    });

    // grant write access for the processing lambda to our dynamo table
    dynamoTable.grantWriteData(processLambda);

    // define the lambda backing our API
    const postLambda = new lambda.NodejsFunction(this, 'cdk-todonts-post', {
      runtime: Runtime.NODEJS_14_X,
      entry: 'src/post.js',
      handler: 'handler',
      events: [new ApiEventSource('POST', '/todonts')],
      environment: {
        QUEUE_URL: sqsBuffer.queueUrl,
      }
    });

    // grant write access to the SQS buffer queue for our API lambda
    sqsBuffer.grantSendMessages(postLambda);
  }
}

The basic building blocks of a CDK application are called constructs which represent a "cloud component", whether that's a single service instance such as an SQS Queue or a set of services encapsulated in a component. Constructs can then be shared and reused between projects and there's a fantastic community that has built a massive collection of high-quality components for you to use. Having the app and its infrastructure described fully in code also means that we can write actual tests against our setup - pretty darn cool, huh?

Before we can deploy the app for the first app, we need to bootstrap the AWS environment (account & region combination) to provision some resources that the CDK uses to deploy the app. After that, we can run cdk deploy to deploy our application.

The CDK CLI doesn't bring the same utility around testing and debugging as SAM and Serverless does but it is possible to use the SAM CLI together with the CDK to help bridge the gap. There's also a newcomer on the block, Serverless-Stack, an extension of the CDK, that brings a lot of testing utility and serverless specific constructs.

➕ Pros

Enables sharing & re-use of components
Large & helpful community
Makes the infrastructure testable
You can (likely) use the same programming language to define the infrastructure of your application as your actual application

➖ Cons

Need to use another tool, such as SAM or the AWS CLI, if you want to invoke or print the logs of a deployed function.

Resources:

Getting started with the AWS CDK

CDK Patterns

CDK Day

Wrapping Up

There's a lot that's happening in this space at the moment and while I think these are the three most prominent players right now, alternatives popping up left and right. Each framework has its own strengths and benefits and there's rarely a wrong or right when choosing which one will work best in your project.

Please let me know in the comments which one is your favorite and why!

If you enjoyed this post and want to see more, follow me on Twitter at @TastefulElk where I frequently write about serverless tech, AWS, and developer productivity!

Production ready Typescript Serverless project configuration

Sebastian Bille — Tue, 16 Mar 2021 12:54:17 +0000

Before we get started - if you're just here because you want a plug 'n' play starter, here's a project generator and here's an example repository! 🚀

Getting Typescript to run is usually pretty simple. You just follow a few lines of instructions from the first google link. Getting the project from that point to something that you can deploy with confidence is often surprisingly hard though. To get there, you'll at the very least want to have testing and linting set up. Tree shaking is another thing you'll probably want to be able to drastically reduce your bundle sizes.

This is going to be a walkthrough of my opinionated approach to achieving a simple but extensible setup that accomplishes these simple goals for a Serverless AWS project.

Project Initialization

Let's start off by creating a new project. Make sure you're in an empty directory and run npm init -y. We're going to need a few tools to set this up:

Serverless & Typescript
ESLint for linting
Jest for testing
Webpack for bundling and optimizing
ts-loader as a Typescript loader for webpack
ts-jest as a Typescript preprocessor for jest

Let's add those as devDependencies right of the bat as well

yarn add eslint jest @types/jest serverless serverless-webpack ts-jest ts-loader typescript webpack -D

Application Code & Tests

We'll need some example Typescript code to run in our app. I prefer placing the application code in an src folder and the test code in a tests folder to keep them separate.

// src/myFunction.ts
export const handler = async (event: any, context: any) => {
  return 'Hello from handler!';
};

// tests/myFunction.test.ts
import { handler } from '../src/myFunction';

describe('[myFunction]', () => {
  it('should not crash', async () => {
    await handler({}, {});
  });
});

In order to transpile and run the Typescript code, we'll need a tsconfig.json. We can generate one to start from by running yarn tsc --init. Let's change both the target and lib to "ES2020" as we want to be able to use modern ECMAScript features.

// tsconfig.json
{
  "compilerOptions": {
    "target": "ES2020", /* Specify ECMAScript target version: 'ES3' (default), 'ES5', 'ES2015', 'ES2016', 'ES2017', 'ES2018', 'ES2019', 'ES2020', or 'ESNEXT'. */
    "module": "commonjs", /* Specify module code generation: 'none', 'commonjs', 'amd', 'system', 'umd', 'es2015', 'es2020', or 'ESNext'. */
    "lib": ["ES2020"], /* Specify library files to be included in the compilation. */
    "strict": true, /* Enable all strict type-checking options. */
    "esModuleInterop": true, /* Enables emit interoperability between CommonJS and ES Modules via creation of namespace objects for all imports. Implies 'allowSyntheticDefaultImports'. */
    "skipLibCheck": true, /* Skip type checking of declaration files. */
    "forceConsistentCasingInFileNames": true /* Disallow inconsistently-cased references to the same file. */
  }
}

All we need now to be able to start developing and testing our app is to add a Jest configuration file. The default settings for Jest are pretty sensible to we don't have to care about most of those right now except for setting preset to "ts-jest" and the testEnvironment to "node".

// jest.config.js
module.exports = {
  preset: 'ts-jest',
  testEnvironment: 'node',
};

We can now finally run our tests with yarn jest to confirm that everything seems to work so far.

Excellent, our tests are passing! 🎉

Linting

We'll be using ESLint for static code analysis and code style enforcement. ESLint configurations can quickly become a mess and it's by nature very subjective so let's keep it simple here. When we run yarn eslint --init we'll get to answer a few questions about our preferences. In the image below I chose to create a small custom set of rules rather than using one of the "popular style guides" but it's completely up to you.

This is how mine ended up looking:

// .eslintrc.js
module.exports = {
  env: {
    es2021: true,
    node: true,
  },
  extends: ['eslint:recommended', 'plugin:@typescript-eslint/recommended'],
  parser: '@typescript-eslint/parser',
  parserOptions: {
    ecmaVersion: 12,
    sourceType: 'module',
  },
  plugins: ['@typescript-eslint'],
  rules: {
    indent: ['error', 2],
    'linebreak-style': ['error', 'unix'],
    quotes: ['error', 'single'],
    semi: ['error', 'always'],
  },
  ignore: ['webpack.config.js'],
};

The AirBnb style guide has a few conflicts with our Typescript setup out of the box so if you choose to extend AirBnb style guide you'll have to do these steps as well:

add "jest" to the env configuration
add 'plugin:import/typescript' to extends & import to plugins to get rid of module resolution errors
disable explicit file extension rule by adding 'import/extensions': 'off' to rules

Packaging & Deploying

Lastly, we need to configure Webpack and Serverless in order to build and deploy our app. For the Webpack configuration, we can use a pretty simple setup where we specify ts-loader to be used as the loader for Typescript files.

// webpack.config.js
const path = require('path');
const slsw = require('serverless-webpack');

module.exports = {
  mode: slsw.lib.webpack.isLocal ? 'development' : 'production',
  entry: slsw.lib.entries,
  resolve: {
    extensions: ['.ts', 'tsx']
  },
  target: 'node',
  module: {
    rules: [
      {
        test: /\.(tsx?)$/,
        loader: 'ts-loader',
        exclude: [
          [
            path.resolve(__dirname, 'node_modules'),
            path.resolve(__dirname, '.serverless'),
            path.resolve(__dirname, '.webpack'),
          ],
        ],
      },
    ],
  },
};

and finally, for the Serverless configuration we'll, in addition to some basic information about our app, add the following:

serverless-webpack to the list of plugins
Set package.individually to true in order to bundle each function separately so that we can reduce the bundle size for each function by only including exactly what's needed to run it
As we've been using Yarn we should set the Webpack packager to yarn so that it'll respect our lock file.

// serverless.yml
service: sls-ts

plugins:
  - serverless-webpack

package:
  individually: true

custom:
  webpack:
    packager: yarn

provider:
  name: aws
  region: eu-north-1
  runtime: nodejs14.x

functions:
  myFunction:
    handler: src/myFunction.handler

That's it! Hit yarn serverless deploy to deploy the project! We can verify that it works by running yarn serverless invoke -f myFunction

Wrapping up

Project configuration is highly subjective and the complexity often grows as the projects grows but I hope this can serve as a starting point to build upon!

If you enjoyed this guide and want to see more, follow me on Twitter at @TastefulElk where I frequently write about serverless tech, AWS and developer productivity!

Happy hacking! 🚀

Resources

Conditional git profile configuration

Sebastian Bille — Wed, 03 Mar 2021 19:30:15 +0000

I always found it inconvenient that I had to either keep changing the email in my .gitconfig every time I wanted to switch between working on a personal project and a work project, or I had to remember to configure the email in each repository as I cloned it. I often forgot and git discipline being something of a pet peeve of mine, I spent way too much time rewriting history just to change my darned email address so that it wouldn't be inconsistent in the git log. I have a problem, I know but let's keep on topic here mmhkay?

Turns out, there's a much easier way to do this. It's possible to conditionally extend the git configuration based on where your repository is located in your file system. That means we can have a global git configuration with all the default configuration and then in a few overrides for all repositories within, for example, the ~/code/work/ folder by adding a little something like this to the ~/.gitconfig file.

// ~/.gitconfig
[user]
    name = Sebastian Bille
    email = personal@example.com

[includeIf "gitdir:~/code/work/"]
    path = ~/code/work/.gitconfig

// ~/code/work/.gitconfig
[user]
    email = work@example.com

Now all repositories inside of the ~/code/work/ folder will use your overridden email but all others will keep using the default configuration.

This could of course be used to override or add any other configuration conditionally as well.

If you enjoyed this guide and want to see more, follow me on Twitter at @TastefulElk where I frequently write about serverless tech, AWS and developer productivity!

Happy hacking! 🚀

AWS EventBridge Replays

Sebastian Bille — Mon, 09 Nov 2020 23:57:23 +0000

AWS recently announced native replay functionality for EventBridge! In my opinion, replayability and event archiving in event-driven architectures are often overlooked and a resilient, cost-effective solution is often needlessly complicated - so this is fantastic news! 🚀

I was eager to try it out so I decided to build a small demo - come join me!

Project setup

npm init -y, install Serverless framework, serverless-pseudo-parameters and the AWS SDK

npm install serverless serverless-pseudo-parameters --dev

Let's start by creating a fake order producer that we later can pretend is some important service that lives somewhere else. The "createOrder" service will put an event on our message bus containing an order based on the orderId we pass.

// createOrder.js
const AWS = require('aws-sdk');

const eventBridge = new AWS.EventBridge({ region: 'eu-north-1' });

const handler = async (event) => {
  const order = {
    id: event.orderId,
  };

  return eventBridge.putEvents({
    Entries: [{
      EventBusName: 'orders',
      Source: 'elk.orders',
      DetailType: 'order',
      Detail: JSON.stringify(order)
    }]
  }).promise();
};

module.exports.handler = handler;

next, let's create our "orderCreated" function which will listen for any order events on the bus and work its magic on them.

// orderCreated.js
const handler = async (event) => {
  console.log(`handled order ${event.detail.id}`);
};

module.exports.handler = handler;

and finally, let's create our serverless.yml where we'll define all the infrastructure resources our service will need.

# serverless.yml
service: order-service

plugins:
  - serverless-pseudo-parameters

provider:
  name: aws
  region: eu-north-1
  runtime: nodejs12.x
  iamRoleStatements:
    - Effect: Allow
      Action: events:PutEvents
      Resource: ${self:custom.eventBusArn}

custom:
  eventBusArn: arn:aws:events:${self:provider.region}:#{AWS::AccountId}:event-bus/orders

functions:
  createOrder:
    handler: createOrder.handler
  orderCreated:
    handler: orderCreated.handler
    events:
      - eventBridge:
          eventBus: ${self:custom.eventBusArn}
          pattern:
            source:
              - elk.orders

resources:
  Resources:
    EventBus:
      Type: AWS::Events::EventBus
      Properties:
        Name: orders
    EventArchive:
      Type: AWS::Events::Archive
      Properties:
        ArchiveName: orders-archive
        SourceArn: ${self:custom.eventBusArn}

There's quite a lot going on here, so let's break it down a bit, shall we? We shall.

We're specifying that the orderCreated function should be triggered by all events on the "orders" EventBus. The EventBus itself is defined as a CloudFormation resource.
However, at the time of writing this, there's a bug in Serverless framework that prevents us from referencing the ARN of the CF resource in the event definition so we reference the "calculated" ARN in a custom property, eventBusArn, instead.

In addition to the EventBus resource, there's an Archive resource as well. This is the resource that defines our archive, in which all events put on the bus will be archived (and thus available to be replayed). We also set up the Lambda EventBridge trigger and specified that we're interested in all events coming from the "elk.orders" source.

We've also added an IAM role statement that allows our service to put events on the bus since we're hosting our event producer in the same service.

Alright, let's try it out.

From your nearest terminal, deploy the service by running npx sls deploy. We can create our first event by invoking the producer with an order

npx sls invoke --function createOrder -d '{ "orderId": "1" }'

if we now look at the logs of our orderCreated function, we should see that that it's received and "processed" the order.

npx sls logs --function orderCreated
[...]
START RequestId: d8069d8e-ed1a-4837-8e3f-b44f12e23773 Version: $LATEST
2020-11-09 00:18:41.268 (+01:00)    d8069d8e-ed1a-4837-8e3f-b44f12e23773    INFO    handled order 1
END RequestId: d8069d8e-ed1a-4837-8e3f-b44f12e23773

Great, it works - let's destroy something! If our orderCreated Lambda were to fail, the events would automatically be retried with an automatic exponential backoff - but what happens if we don't actually "fail", we just fail to process the event correctly? Unfortunately, it would be lost.

Let's simulate something like that happening by changing the name of our order's id field from id to orderId in the event.

// createOrder.js
[...]
const handler = async (event) => {
  const order = {
    orderId: event.orderId,
  };
[...]

deploy our service again, and then create another event, order 2

npx sls invoke --function createOrder -d '{ "orderId": "2" }'

The logs from the orderCreated function will now say

INFO    handled order undefined

Alright, I think we can figure out what the problem is, let's fix the bug

// orderCreated.js
const handler = async (event) => {
  console.log(`handled order ${event.detail.orderId}`);
};

module.exports.handler = handler;

and deploy again.

Replays to the rescue

We botched the handling of that second order but luckily we created an archive for our events when we created the event bus.

@mhlabs/evb-cli is a neat little CLI tool to manage EventBridge, so let's use that to trigger our replay.

npm i @mhlabs/evb-cli -g

Run the tool and specify the "orders" bus, evb replay -b orders, then adjust the time span as needed.

It'll take a minute or so for EventBridge to fetch and re-send your events from the archive, but in a moment we should be able to see that the second order has now been processed by our orderCreated handler! 🎉

INFO    handled order 2

Isn't that amazing? With just a handful of lines of code, we've set up an EventBus that durably stores all events for as long as we need them from which we can replay events sent within a given time span to any consumers that need to (re)process the events as well as a service that reads and writes to the bus.

Do keep in mind that when designing systems capable of replaying events, it's crucial to make sure your services are idempotent. If (when) the same order event is sent for the second time, your customer likely didn't buy two of the same Die Hard 2 DVD off your site at the same time.

The code for this demo can be found here.

PS.

Did you know EventBridge can automatically generate schemas and code bindings for Typescript, Python or Java? Using those would've prevented that whole id->orderId fiasco from before... Oh well.

If you enjoyed this guide and want to see more, follow me on Twitter at @TastefulElk where I frequently write about serverless tech, AWS and developer productivity!