Forem: Valentin BEGGI

🧙 Lambdalith Mastery: Elevating NestJS Deployment on AWS Lambda with CDK & Webpack

Valentin BEGGI — Thu, 16 Nov 2023 14:45:29 +0000

TL;DR 📚

Why is a Lambdalith approach a high ROI choice for many? 💸
Learn how to deploy a monolithic NestJS app on AWS Lambda using Webpack and AWS CDK. 🚀

The Lambdalith Edge for NestJS on AWS Lambda 🌟

🧠 A Lambdalith is a monolithic architecture approach for serverless applications where a single AWS Lambda function serves the entire API, rather than deploying separate functions for each endpoint.

Opt for a Lambdalith and reap multiple benefits for your NestJS API:

Faster Rollouts: Quicker deployments and streamlined management, irrespective of your number of routes.
Minimized Cold Starts: Enhanced performance through more frequent reuse of a single Lambda function.
Easier Logging: A single point for logs simplifies monitoring and alert setup.
Full NestJS Benefits: Fully exploit NestJS's rich features and community support.

While a Lambdalith might mean lengthier cold starts and broader control scopes, its efficiency, simplicity, and high return on investment are unmatched.

Monorepo structure 🚧📦: I strongly advice you embrace a monorepo structure, with a package for your API and a package for your infrastructure (CDK).

Setting Up the Infrastructure with AWS CDK 🏗️

AWS CDK transforms infrastructure into code. Kick things off by installing AWS CDK and initiating a TypeScript project with cdk init app --language typescript.

In the lib/my-stack.ts file, begin with the core of your setup: the Lambda function.

// LambdaNestStack in stack.ts
const apiNestHandlerFunction = new Function(this, "ApiNestHandler", {
  code: Code.fromAsset("api/dist"), // 👈 This is crucial
  runtime: Runtime.NODEJS_18_X,
  handler: "main.handler",
  environment: {}, // 👈 You might need env variables
});

Next up, create a Rest API with a Lambda proxy at its root. This API Gateway acts as the traffic controller, directing all requests to your Lambda-powered NestJS app. All route paths will be directed to your single Lambda. 🗾

const api = new RestApi(this, "Api", {
    deploy: true,
    defaultMethodOptions: {
    apiKeyRequired: true,
    },
});

api.root.addProxy({
    defaultIntegration: new LambdaIntegration(apiNestHandlerFunction, { proxy: true }),
});

const apiKey = api.addApiKey("ApiKey"); // 👈 to ease your testing

const usagePlan = api.addUsagePlan("UsagePlan", {
    name: "UsagePlan",
    apiStages: [
    {
        api,
        stage: api.deploymentStage,
    },
    ],
});

usagePlan.addApiKey(apiKey);

In this snippet, Code.fromAsset("api/dist") is crucial. It points to the location of our bundled NestJS app, ensuring efficient Lambda execution.

Prepping the NestJS App for Lambda 🦁

Start by creating a new NestJS app with nest new api. Then, install the @nestjs/platform-express and @vendia/serverless-express packages.
You now have a classic NestJS app, ready to be adapted for AWS Lambda.

Next to the main.ts file, create a new lambda.ts file. This file will be the entry point of our Lambda function.

// lambda.ts
import { NestFactory } from '@nestjs/core';
import { ExpressAdapter } from '@nestjs/platform-express';
import serverlessExpress from '@vendia/serverless-express';
import { Context, Handler } from 'aws-lambda';
import express from 'express';

import { AppModule } from './app.module';

let cachedServer: Handler;

async function bootstrap() {
  if (!cachedServer) {
    const expressApp = express();
    const nestApp = await NestFactory.create(
      AppModule,
      new ExpressAdapter(expressApp),
    );

    nestApp.enableCors();

    await nestApp.init();

    cachedServer = serverlessExpress({ app: expressApp });
  }

  return cachedServer;
}

const handler = async (event: any, context: Context, callback: any) => {
  const server = await bootstrap();
  return server(event, context, callback);
};

module.exports.handler = handler;

This code will be executed by AWS Lambda. It creates a NestJS app and adapts it to the AWS Lambda environment. It also ensures that the NestJS app is only created once, improving performance. ⚡

🗒️ Side Note: You could easily setup a single main.ts entry point by leveraging env variables to deduce the execution context: Lambda or local

Now, we need to bundle this TypeScript code into a single file...🤓

Packing it Up with Webpack 📦🧙‍♂️

There are several ways to bundle a NestJS app for AWS Lambda. You could use Lambda Layers, but this is not the most efficient approach. Instead, we'll use Webpack to bundle our NestJS app into a single file, which we'll then deploy with AWS CDK.

Let's start by creating a new webpack.config.js file in our API package. This file will define our Webpack configuration.

module.exports = function (options, webpack) {
  return {
    ...options,
    entry: ['./src/lambda.ts'],
    externals: [],
    output: {
      ...options.output,
      libraryTarget: 'commonjs2',
    },
    plugins: [
      ...options.plugins,
      new webpack.IgnorePlugin({
        checkResource(resource) {
          // Ignoring non-essential modules for Lambda deployment
          return lazyImports.includes(resource);
        },
      }),
    ],
  };
};

This configuration bundles our Lambda entry file (lambda.ts) and its dependencies, creating a lean and efficient package for AWS Lambda!

Make sure to create a build-lambda script in your package.json file!

{
  "scripts": {
    "build-lambda": "nest build --webpack --webpackPath webpack.config.js"
  }
}

Deploying the NestJS App: To the Cloud! ☁️

Your NestJS app is now a compact bundle, thanks to Webpack. Deploying? It's as simple as:

Build: Run npm run build-lambda in your API package.
Deploy: In your infrastructure package, execute cdk deploy.

And like that, your NestJS app ascends to AWS Lambda, primed for action. 💫

Your High-Performance NestJS App now lives on AWS 🚀

Congratulations! You've unlocked the strategy for a potent, scalable, and efficient NestJS app on AWS Lambda, all packaged neatly with Webpack and AWS CDK. 👏👏

Please feel free to comment if anything was unclear or if you found a better way to achieve this result! 💬

🥇 The Lambdalith Advantage: A Complete Guide to NestJS Deployment on AWS Lambda Using CDK

Valentin BEGGI — Tue, 14 Nov 2023 12:34:57 +0000

TL;DR 📚

Why is a Lambdalith approach a high ROI choice for many? 💸
Learn how to deploy a monolithic NestJS app on AWS Lambda using Webpack and AWS CDK. 🚀

The Lambdalith Edge for NestJS on AWS Lambda 🌟

🧠 A Lambdalith is a monolithic architecture approach for serverless applications where a single AWS Lambda function serves the entire API, rather than deploying separate functions for each endpoint.

Opt for a Lambdalith and reap multiple benefits for your NestJS API:

Faster Rollouts: Quicker deployments and streamlined management, irrespective of your number of routes.
Minimized Cold Starts: Enhanced performance through more frequent reuse of a single Lambda function.
Easier Logging: A single point for logs simplifies monitoring and alert setup.
Full NestJS Benefits: Fully exploit NestJS's rich features and community support.

While a Lambdalith might mean lengthier cold starts and broader control scopes, its efficiency, simplicity, and high return on investment are unmatched.

Monorepo structure 🚧📦: I strongly advice you embrace a monorepo structure, with a package for your API and a package for your infrastructure (CDK).

Setting Up the Infrastructure with AWS CDK 🏗️

AWS CDK transforms infrastructure into code. Kick things off by installing AWS CDK and initiating a TypeScript project with cdk init app --language typescript.

In the lib/my-stack.ts file, begin with the core of your setup: the Lambda function.

// LambdaNestStack in stack.ts
const apiNestHandlerFunction = new Function(this, "ApiNestHandler", {
  code: Code.fromAsset("api/dist"), // 👈 This is crucial
  runtime: Runtime.NODEJS_18_X,
  handler: "main.handler",
  environment: {}, // 👈 You might need env variables
});

const api = new RestApi(this, "Api", {
    deploy: true,
    defaultMethodOptions: {
    apiKeyRequired: true,
    },
});

api.root.addProxy({
    defaultIntegration: new LambdaIntegration(apiNestHandlerFunction, { proxy: true }),
});

const apiKey = api.addApiKey("ApiKey"); // 👈 to ease your testing

const usagePlan = api.addUsagePlan("UsagePlan", {
    name: "UsagePlan",
    apiStages: [
    {
        api,
        stage: api.deploymentStage,
    },
    ],
});

usagePlan.addApiKey(apiKey);

In this snippet, Code.fromAsset("api/dist") is crucial. It points to the location of our bundled NestJS app, ensuring efficient Lambda execution.

Prepping the NestJS App for Lambda 🦁

Next to the main.ts file, create a new lambda.ts file. This file will be the entry point of our Lambda function.

// lambda.ts
import { NestFactory } from '@nestjs/core';
import { ExpressAdapter } from '@nestjs/platform-express';
import serverlessExpress from '@vendia/serverless-express';
import { Context, Handler } from 'aws-lambda';
import express from 'express';

import { AppModule } from './app.module';

let cachedServer: Handler;

async function bootstrap() {
  if (!cachedServer) {
    const expressApp = express();
    const nestApp = await NestFactory.create(
      AppModule,
      new ExpressAdapter(expressApp),
    );

    nestApp.enableCors();

    await nestApp.init();

    cachedServer = serverlessExpress({ app: expressApp });
  }

  return cachedServer;
}

const handler = async (event: any, context: Context, callback: any) => {
  const server = await bootstrap();
  return server(event, context, callback);
};

module.exports.handler = handler;

This code will be executed by AWS Lambda. It creates a NestJS app and adapts it to the AWS Lambda environment. It also ensures that the NestJS app is only created once, improving performance. ⚡

🗒️ Side Note: You could easily setup a single main.ts entry point by leveraging env variables to deduce the execution context: Lambda or local

Now, we need to bundle this TypeScript code into a single file...🤓

Packing it Up with Webpack 📦🧙‍♂️

Let's start by creating a new webpack.config.js file in our API package. This file will define our Webpack configuration.

module.exports = function (options, webpack) {
  return {
    ...options,
    entry: ['./src/lambda.ts'],
    externals: [],
    output: {
      ...options.output,
      libraryTarget: 'commonjs2',
    },
    plugins: [
      ...options.plugins,
      new webpack.IgnorePlugin({
        checkResource(resource) {
          // Ignoring non-essential modules for Lambda deployment
          return lazyImports.includes(resource);
        },
      }),
    ],
  };
};

This configuration bundles our Lambda entry file (lambda.ts) and its dependencies, creating a lean and efficient package for AWS Lambda!

Make sure to create a build-lambda script in your package.json file!

{
  "scripts": {
    "build-lambda": "nest build --webpack --webpackPath webpack.config.js"
  }
}

Deploying the NestJS App: To the Cloud! ☁️

Your NestJS app is now a compact bundle, thanks to Webpack. Deploying? It's as simple as:

Build: Run npm run build-lambda in your API package.
Deploy: In your infrastructure package, execute cdk deploy.

And like that, your NestJS app ascends to AWS Lambda, primed for action. 💫

Your High-Performance NestJS App now lives on AWS 🚀

Congratulations! You've unlocked the strategy for a potent, scalable, and efficient NestJS app on AWS Lambda, all packaged neatly with Webpack and AWS CDK. 👏👏

Please feel free to comment if anything was unclear or if you found a better way to achieve this result! 💬

AWS Lambda Meets Bun through SST 🐰: Prime Time or Just Prime Hype? 🥵

Valentin BEGGI — Fri, 22 Sep 2023 15:33:22 +0000

TL;DR

⚡ Deploy Bun in AWS Lambda swiftly and efficiently with SST's streamlined process.
🐰 Dive into the integration of Bun with Lambda through SST and evaluate its readiness for your use-case.

1️⃣ Load the Bun Layer to your AWS Account

💡 Before you start, ensure you have an AWS account and that you're logged in via the AWS CLI.

To verify your login status, execute the following command:
aws sts get-caller-identity
If you are not logged in, run the command below:
aws configure

We'll be utilizing the official Bun layer for Lambda, provided by the Bun team 🐰.

Follow the instructions listed on the Bun Layer page to add the layer to your AWS account.

When executing bun publish-layer, ensure you choose the aarch64 architecture ⚡!

bun run publish-layer -- \
  --arch aarch64 \
  --region eu-west-1

Afterward, navigate to your AWS console, access the Lambda service, and verify that the layer is available.
Make a note of its ARN; we'll be using it later. 🧠

2️⃣ Deploy a new SST app 🔥

SST is a framework designed to streamline the deployment of Serverless applications on AWS. If you're curious about why we favor SST over the Serverless Framework, check out our previous article: Best Serverless Framework in 2023: A Data-Driven Showdown for AWS Projects

Proceed with the steps in SST's standalone get started section.

In essence, you'll execute:

npx create-sst@latest my-sst-app

Install your dependencies and then deploy your app (we already made sure you are logged in via the AWS CLI 😉). It will run cdk bootstrap under the hood so you don't have to worry about it.

📝 For a quick test of Bun in Lambda, consider removing all resources in MyStack.ts except the lambda resource. We'll adjust it later.

Upon completion, your app should be deployed with a Node.js Lambda function. Next, we'll modify your stack resources to make the lambda run Bun!

3️⃣ Add the Bun Layer to your lambda, and execute bun 🐰

Let's incorporate some Bun code into our Lambda.

First, add this to your tsconfig.json

{
  "compilerOptions": {
    "types": ["bun-types"] // add Bun global
  }
}

Then, draft a basic lambda handler leveraging the Bun.password API (this ensures Bun is operational!):

let hashedPassword = null;

export default {
  async fetch(request: Request): Promise<Response> {
    const { password, action } = (await request.json()).event as {
      password: string;
      action: "get" | "set";
    };

    switch (action) {
      case "set":
        hashedPassword = await Bun.password.hash(password);

        return new Response(
          JSON.stringify({
            statusCode: 200,
            body: { result: "Password has been set." },
          })
        );

      case "get":
        const isVerified = await Bun.password.verify(password, hashedPassword);

        return new Response(
          JSON.stringify({
            statusCode: 200,
            body: { result: isVerified },
          })
        );
    }
  },
};

💡 In the above code, the password is set in Lambda's memory. The variable will be lost whenever a new instance of this lambda is created.

Now, it's time to deploy our modifications! 🚀

In your MyStack.ts file, add the following code:

new Function(stack, "my-bun-function", {
  // 👇 This is required to use custom runtimes
  runtime: Runtime.PROVIDED_AL2,
  handler: "lambda.fetch",
  // 👇 That's why we published bun layer with aarch64 architecture
  architecture: Architecture.ARM_64,
  code: Code.fromAsset("out"),
  memorySize: 1024,
  timeout: Duration.seconds(10),
  layers: [
    LayerVersion.fromLayerVersionArn(
      stack,
      "bunLayer",
      // 👇 Replace with your layer ARN
      YOUR_BUN_LAYER_ARN
    ),
  ],
});

Lastly, in your package.json file, refresh the deploy command to:

    "deploy": "bun build --outdir=out packages/functions/src/PATH_TO_YOUR_HANDLER --target bun && sst deploy"

💡 If you need to deploy several lambdas, a dedicated build step to minify bun code would be necessary!

Now, run your deploy command, and go check your lambda in the AWS console. You should be able to test it with an input event that matches the one we used in our code.

{
  "password": "my-password",
  "action": "set"
}

To take it a step further, connect your Lambda to an API Gateway, or generate a function URL via the AWS console for testing!

4️⃣ Conclusion & Caveats 🥲

Using the custom Bun Layer comes at a cost: ~500ms cold start time. 🥶

After conducting tests on computationally intensive tasks ( calculating Fibonacci sequences), I observed performance to be twice as slow compared to the native Node.js runtime. 🐢

Personally, I believe Bun in Lambda isn't production-ready. We can only hope AWS introduces a native Bun runtime in the near future! 🤞

If you discovered a more efficient method to deploy Bun in Lambda or if your performance results differ from mine, please share your experiences in the comments! 🙏

🥇 The Best Serverless Framework in 2023: A Data-Driven Showdown for AWS Projects

Valentin BEGGI — Fri, 22 Sep 2023 15:12:26 +0000

1️⃣ Load the Bun Layer to your AWS Account

💡 Before you start, ensure you have an AWS account and that you're logged in via the AWS CLI.

To verify your login status, execute the following command:
aws sts get-caller-identity
If you are not logged in, run the command below:
aws configure

We'll be utilizing the official Bun layer for Lambda, provided by the Bun team 🐰.

Follow the instructions listed on the Bun Layer page to add the layer to your AWS account.

When executing bun publish-layer, ensure you choose the aarch64 architecture ⚡!

bun run publish-layer -- \
  --arch aarch64 \
  --region eu-west-1

Afterward, navigate to your AWS console, access the Lambda service, and verify that the layer is available.
Make a note of its ARN; we'll be using it later. 🧠

2️⃣ Deploy a new SST app 🔥

Proceed with the steps in SST's standalone get started section.

In essence, you'll execute:

npx create-sst@latest my-sst-app

Install your dependencies and then deploy your app (we already made sure you are logged in via the AWS CLI 😉). It will run cdk bootstrap under the hood so you don't have to worry about it.

📝 For a quick test of Bun in Lambda, consider removing all resources in MyStack.ts except the lambda resource. We'll adjust it later.

Upon completion, your app should be deployed with a Node.js Lambda function. Next, we'll modify your stack resources to make the lambda run Bun!

3️⃣ Add the Bun Layer to your lambda, and execute bun 🐰

Let's incorporate some Bun code into our Lambda.

First, add this to your tsconfig.json

{
  "compilerOptions": {
    "types": ["bun-types"] // add Bun global
  }
}

Then, draft a basic lambda handler leveraging the Bun.password API (this ensures Bun is operational!):

let hashedPassword = null;

export default {
  async fetch(request: Request): Promise<Response> {
    const { password, action } = (await request.json()).event as {
      password: string;
      action: "get" | "set";
    };

    switch (action) {
      case "set":
        hashedPassword = await Bun.password.hash(password);

        return new Response(
          JSON.stringify({
            statusCode: 200,
            body: { result: "Password has been set." },
          })
        );

      case "get":
        const isVerified = await Bun.password.verify(password, hashedPassword);

        return new Response(
          JSON.stringify({
            statusCode: 200,
            body: { result: isVerified },
          })
        );
    }
  },
};

💡 In the above code, the password is set in Lambda's memory. The variable will be lost whenever a new instance of this lambda is created.

Now, it's time to deploy our modifications! 🚀

In your MyStack.ts file, add the following code:

new Function(stack, "my-bun-function", {
  // 👇 This is required to use custom runtimes
  runtime: Runtime.PROVIDED_AL2,
  handler: "lambda.fetch",
  // 👇 That's why we published bun layer with aarch64 architecture
  architecture: Architecture.ARM_64,
  code: Code.fromAsset("out"),
  memorySize: 1024,
  timeout: Duration.seconds(10),
  layers: [
    LayerVersion.fromLayerVersionArn(
      stack,
      "bunLayer",
      // 👇 Replace with your layer ARN
      YOUR_BUN_LAYER_ARN
    ),
  ],
});

Lastly, in your package.json file, refresh the deploy command to:

    "deploy": "bun build --outdir=out packages/functions/src/PATH_TO_YOUR_HANDLER --target bun && sst deploy"

💡 If you need to deploy several lambdas, a dedicated build step to minify bun code would be necessary!

Now, run your deploy command, and go check your lambda in the AWS console. You should be able to test it with an input event that matches the one we used in our code.

{
  "password": "my-password",
  "action": "set"
}

To take it a step further, connect your Lambda to an API Gateway, or generate a function URL via the AWS console for testing!

Conclusion & Caveats 🥲

Using the custom Bun Layer comes at a cost: ~500ms cold start time. 🥶

Moreover, after conducting load tests on computationally intensive tasks (like calculating Fibonacci sequences), I observed performance to be nearly twice as slow compared to the native Node.js runtime.

Personally, I believe Bun in Lambda isn't production-ready. We can only hope AWS introduces a native Bun runtime in the near future! 🤞

I hope you found this article insightful! If you discover a more efficient method to deploy Bun in Lambda or if your performance results differ from mine, please share your experiences in the comments! 🙏

🥇 The Best Serverless Framework in 2023: A Data-Driven Showdown for AWS Projects

Valentin BEGGI — Wed, 16 Aug 2023 11:54:33 +0000

TL;DR

⚡ Opt for SST for new, simpler projects, rapid delivery, owning NextJS infra, or when having limited AWS expertise.
☁️ Go with CDK when integrating with or migrating existing infrastructures, for large enterprise projects, or when dealing with intricate infrastructures.

Why ❓

If you're wondering which serverless framework to adopt for your next AWS project in 2023, you're in the right place.

Whether you're starting from scratch or working on an intricate enterprise infrastructure, this comparison dives deep into the pros, cons, and best use cases for the top contenders.

Stick around and let's ensure your next choice is an informed one 🧠.

The contenders 🥊

Here are our three contenders

1 - Serverless + AWS CDK + Lift: An integration that amps up the traditional Serverless Framework with Lift's static frontend construct and CDK's robust infra definition.

💡 Have a look here to understand how the three are bridged together

2 - AWS CDK: AWS's native toolkit, offering a high-level, typesafe approach to AWS resources.

3 - SST: A TypeScript first framework streamlining serverless apps on AWS, from debugging to deployment.

Now let's dive right into our data-driven face-off between the frameworks. See how they stack up against each other across pivotal criteria. 🥊

Comparison Table 📈

Criteria	Description	1️⃣ Serverless Framework + CDK	2️⃣ AWS CDK	3️⃣ SST
Stars / Downloads		45k ⭐ - 1.2M downloads/week	✅ 10k ⭐ - 1.6M downloads/week	17k ⭐ - 35k downloads/week
Release frequency		1 / month	7 / month	✅ 36 / month
Ease of contribution	Codebase complexity and accessibility, communication with maintainers	"JS codebase Complex codebase"	❌ Slow processes hard to dev locally	🟠 Easy to chat with the team (Discord) Codebase Typescript + accessible PRs are very slow to merge
IaC Type-safety	Typescript definitions for all resources	"Base is YAML, but typescript support with @serverless/typescript. Serverless Framework resources are not natively typed."	✅ Fully typed	✅ Fully Typed
Lambda Definition	Ease to define a lambda resource and its triggers	✅ Config file per lambda Triggers for many services + cron	🟡 Lambda configuration is verbose. No abstraction for lambda triggers, need to define them manually.	✅ Both inline and verbose lambda configuration are available. Trigger definitions for API Gateway, Eventbus, SQS
Other Resources Definition	Ease of defining resources such as S3, DynamoDB, EventBridge, SQ	🔴 referencing serverless provisioned resources is hard CDK / serverless stacks are dissociated	✅	✅
Framework flexibility	Extending the capabilities of the framework to fit your need	✅ possible with plugins	✅ low level framework, you can do what you like	🟠 still a work in progress fallback to full cdk (losing other features)
Static Frontend Deployment	Existing solution to deploy a static website	✅ Lift	🟠 requires rebuild or use of less popular construct	✅ StaticSite in SST
Next.js App Deployment	Existing solution to deploy NextJs server	🟠 amplify plugin serverless-nextjs not maintained	🟠 amplify 🔴 cdk-nextjs (not well maintained)	✅ NexjsSite in SST
DevX / Debugging lambdas	Ease of debugging lambda executions	🟠	❌	✅ live debugging
Debugging deployment	Ease of debugging fails during deployment	✅	✅	🟠 SST adds a layer of liability
AWS Features completeness	amount of supported features	🟠 few releases, few contributors to implement new features	✅	🟡 small delay (~1week)
Multiple Runtime Deployment	Support for deploying multiple AWS Lambda runtimes	✅	✅	✅
Multi-stack Definition	Ability to define multiple stacks in a single file	❌ 1 serverless.ts per stack	✅	✅
Nested Stacks Support	Support for AWS CloudFormation nested stacks	❌	✅	✅
Documentation	quality of the frameworks documentation	🟠 yaml, no ready-to-use snippets but still quite complete	🟡	🟡

Our recommendation 🥇

Opt for SST if one of:

Starting a fresh project of low to moderate complexity
Prioritizing rapid delivery of value
Implementing NextJS without Vercel
Limited AWS expertise and need a more guided approach

Go with CDK if one of:

Integrating with or migrating an existing infrastructure
Engaging in large corporate projects deeply integrated with the company's Information System
Handling intricate and detailed infrastructures.

Wrapping Up 🌟

Deciding between SST and AWS CDK comes down to the unique needs of your project. As the serverless landscape evolves, so does our understanding.
Join the discussion, bring your experiences to the table, and please challenge our criterions! Let's journey together towards a more informed decision-making process in the serverless world. 💬🚀

Deploying a Lambda with a static IP has never been so simple 🍰

Valentin BEGGI — Fri, 24 Feb 2023 14:45:27 +0000

TL;DR

🧠 Learn how to deploy a Lambda with a static IP (for whitelisting concerns)
⚡ Perform NodeJS SFTP operations using this Lambda.

Your Serverless application might need to connect to a partner server that requires IP whitelisting.
We will use the Serverless Framework to deploy our Lambda function and the Typescript AWS CDK to create the infrastructure needed to give our Lambda a static IP 🔒.

If you are not familiar with these libraries, I recommend you check out this Ultimate Serverless DevX: Serverless Framework x CDK blog post to learn how to conciliate the two. 🤝

Moreover, we will use the NodeJS ssh2-promise package to perform SFTP operations in our Lambda function. 💽

This is what we will build 👷:

Let's get to work 💪🔨!

1️⃣ Create a VPC hosted Lambda function with a static IP

☁️ VPC, NAT Gateway and Elastic IP

First, we need to create a VPC with a public subnet and a private subnet. The private subnet will host our Lambda function and the public subnet will host our NAT Gateway.

const vpc = new Vpc(stack, 'Vpc', {
    vpcName: 'vpc',
    natGateways: 1, // 👈 Automatically creates an Elastic IP
    maxAzs: 1, // 👈 Use more if you need high availability
    subnetConfiguration: [
        {
        name: 'private-subnet-1',
        subnetType: SubnetType.PRIVATE_WITH_NAT,
        cidrMask: 24,
        },
        {
        name: 'public-subnet-1',
        subnetType: SubnetType.PUBLIC,
        cidrMask: 24,
        },
    ],
});

💸 Pricing Disclaimer
This architecture will cost some money. The NAT Gateway runs on a EC2 instance which will cost you around 30$/month. This price is multiplied by the number of AZs you use.

I also recommend that you use CloudFormation Outputs to store the IDs of the resources you will need later on. We want to slap that ✨clean IaC code✨ in our PR 🖐️.


export const vpcSecurityGroupOutputId = 'SgOutputId';
export const vpcPrivateSubnetOutputId = 'VpcPvSubOutputId';
...

const privateSubnets = vpc.selectSubnets(
    { subnetType: SubnetType.PRIVATE_WITH_NAT }
);
export const [privateSubnetId1] = privateSubnets.subnetIds;

// 👇 Outputing the IDs of the resources in case another stack needs them
new CfnOutput(stack, vpcSecurityGroupOutputId, {
    value: vpc.vpcDefaultSecurityGroup,
});
new CfnOutput(stack, vpcPrivateSubnet1OutputId, {
    value: privateSubnetId1,
});

⚡ Deploy the Lambda function

Now that we have our VPC, we can deploy our Lambda function, using the Serverless Framework.

export const staticIpLambda = {
  timeout: 15,
  handler: getHandlerPath(__dirname),
  vpc: {
    securityGroupIds: [ stack.resolve(vpc.vpcDefaultSecurityGroup) ],
    subnetIds: [ stack.resolve(privateSubnetId1) ],
    },
};

You can also check out this Serverless Framework x CDK article to learn how to leverage the stack.resolve() method 🧑‍🚀.

And bam! 🎉 Our Lambda function is deployed in our private subnet.

All the outbound traffic from our Lambda function will now go through the NAT Gateway, which will use the Elastic IP we created earlier. 🚀

You can retrieve the Elastic IP of your NAT Gateway using the AWS Console.
Go to the VPC service and click Elastic IPs. You should see the Elastic IP created by the CDK.

Now just message your partner and ask them to whitelist this IP. 📩

2️⃣ Perform SFTP operations in your Lambda function

❓ This part is actually optional.
We will cover a very specific SFTP use case and a problem I encountered while using the ssh2-promise and the serverless-esbuild plugin.

Now let's talk about a very specific use case: performing SFTP operations in your Lambda function.

I wanted to share with you this use case because I came across weird issues with the ssh2-promise package not being correctly bundled into my Lambda .zip code. 🤯

🔒 SSH Key
If your SFTP partner requires a SSH Key, I recommend you to store the key in AWS Secrets Manager. You can then retrieve it in your Lambda function using Middy Secrets Manager middleware. It will provide the key as a string in your Lambda function context. 🤓

💽 Store the file you want to send in your Lambda's `/tmp` folder

The ssh2-promise package will need file system access to send your file. One way to achieve that in a Lambda function context is to leverage the /tmp folder.
This folder is writable and will be deleted when your Lambda function is terminated.

🚧 Warning
The /tmp folder is not persistent. If you want to keep track of the files you've sent you should also use S3. 📦
Also be aware that the /tmp folder is shared between successive Lambda invocations in the same execution environment. Use a unique file name to avoid any bugs 🐛.

This is what the final code looks like with the ssh2-promise package:

fs.writeFileSync('/tmp/myFile.txt', 'Hello World!');

const sshPrivateKey = context["SSH_PRIVATE_KEY"]

const ssh = new SSH2Promise({
    host: 'sftpIpAddress',
    username: 'sftpUsername',
    privateKey: sshPrivateKey,
    port: "SFTP_PORT",
});

await ssh.connect();
const sftp = ssh.sftp();


await sftp.fastPut('/tmp/myFile.txt', "distant_name.txt");

await ssh.close();

The catch 🎣

If like me you are using the serverless-esbuild plugin to bundle your Lambda function, you might encounter some weird issues.
The ssh2-promise package is not correctly bundled into your Lambda function. 🤯

To solve this issue, I first patched the serverless-esbuild plugin to allow yarn3 module bundling exclusion. I want the ssh2-promise package NOT to be bundled by esbuild.

The patch is available here as a github gist

Then, I had to add the ssh2-promise package to the externals section of my esbuild config.

// serverless.ts

const serverlessConfiguration = {
  ..., // 👈 Your other serverless config
  custom: {
    esbuild: { ...esbuildConfig, external: ['ssh2-promise'] },
  },
};

Your Lambda function should now be able to perform SFTP operations. 🚀

Thanks for reading! 🙏

If you have any questions or feedback, feel free to reach out to me on Twitter or ask a question in the comments below. 🥸

Also check out my Dev.to for more articles about AWS, Serverless and Cloud Development. 📝

Deploying a Lambda with a static IP has never been so simple 🍰

Valentin BEGGI — Thu, 16 Feb 2023 13:17:50 +0000

TL;DR

🧠 Learn how to deploy a Lambda with a static IP (for whitelisting concerns)
⚡ Perform NodeJS SFTP operations using this Lambda.

Moreover, we will use the NodeJS ssh2-promise package to perform SFTP operations in our Lambda function. 💽

This is what we will build 👷:

Let's get to work 💪🔨!

1️⃣ Create a VPC hosted Lambda function with a static IP

☁️ VPC, NAT Gateway and Elastic IP

First, we need to create a VPC with a public subnet and a private subnet. The private subnet will host our Lambda function and the public subnet will host our NAT Gateway.

const vpc = new Vpc(stack, 'Vpc', {
    vpcName: 'vpc',
    natGateways: 1, // 👈 Automatically creates an Elastic IP
    maxAzs: 1, // 👈 Use more if you need high availability
    subnetConfiguration: [
        {
        name: 'private-subnet-1',
        subnetType: SubnetType.PRIVATE_WITH_NAT,
        cidrMask: 24,
        },
        {
        name: 'public-subnet-1',
        subnetType: SubnetType.PUBLIC,
        cidrMask: 24,
        },
    ],
});

💸 Pricing Disclaimer
This architecture will cost some money. The NAT Gateway runs on a EC2 instance which will cost you around 30$/month. This price is multiplied by the number of AZs you use.

I also recommend that you use CloudFormation Outputs to store the IDs of the resources you will need later on. We want to slap that ✨clean IaC code✨ in our PR 🖐️.


export const vpcSecurityGroupOutputId = 'SgOutputId';
export const vpcPrivateSubnetOutputId = 'VpcPvSubOutputId';
...

const privateSubnets = vpc.selectSubnets(
    { subnetType: SubnetType.PRIVATE_WITH_NAT }
);
export const [privateSubnetId1] = privateSubnets.subnetIds;

// 👇 Outputing the IDs of the resources in case another stack needs them
new CfnOutput(stack, vpcSecurityGroupOutputId, {
    value: vpc.vpcDefaultSecurityGroup,
});
new CfnOutput(stack, vpcPrivateSubnet1OutputId, {
    value: privateSubnetId1,
});

⚡ Deploy the Lambda function

Now that we have our VPC, we can deploy our Lambda function, using the Serverless Framework.

export const staticIpLambda = {
  timeout: 15,
  handler: getHandlerPath(__dirname),
  vpc: {
    securityGroupIds: [ stack.resolve(vpc.vpcDefaultSecurityGroup) ],
    subnetIds: [ stack.resolve(privateSubnetId1) ],
    },
};

You can also check out this Serverless Framework x CDK article to learn how to leverage the stack.resolve() method 🧑‍🚀.

You can retrieve the Elastic IP of your NAT Gateway using the AWS Console.
Go to the VPC service and click Elastic IPs. You should see the Elastic IP created by the CDK.

Now just message your partner and ask them to whitelist this IP. 📩

2️⃣ Perform SFTP operations in your Lambda function

❓ This part is actually optional.
We will cover a very specific SFTP use case and a problem I encountered while using the ssh2-promise and the serverless-esbuild plugin.

🔒 SSH Key
If your SFTP partner requires a SSH Key, I recommend you to store the key in AWS Secrets Manager. You can then retrieve it in your Lambda function using Middy Secrets Manager middleware. It will provide the key as a string in your Lambda function context. 🤓

💽 Store the file you want to send in your Lambda's `/tmp` folder

🚧 Warning
The /tmp folder is not persistent. If you want to keep track of the files you've sent you should also use S3. 📦
Also be aware that the /tmp folder is shared between successive Lambda invocations in the same execution environment. Use a unique file name to avoid any bugs 🐛.

This is what the final code looks like with the ssh2-promise package:

fs.writeFileSync('/tmp/myFile.txt', 'Hello World!');

const sshPrivateKey = context["SSH_PRIVATE_KEY"]

const ssh = new SSH2Promise({
    host: 'sftpIpAddress',
    username: 'sftpUsername',
    privateKey: sshPrivateKey,
    port: "SFTP_PORT",
});

await ssh.connect();
const sftp = ssh.sftp();


await sftp.fastPut('/tmp/myFile.txt', "distant_name.txt");

await ssh.close();

The catch 🎣

To solve this issue, I first patched the serverless-esbuild plugin to allow yarn3 module bundling exclusion. I want the ssh2-promise package NOT to be bundled by esbuild.

The patch is available here as a github gist

Then, I had to add the ssh2-promise package to the externals section of my esbuild config.

// serverless.ts

const serverlessConfiguration = {
  ..., // 👈 Your other serverless config
  custom: {
    esbuild: { ...esbuildConfig, external: ['ssh2-promise'] },
  },
};

Your Lambda function should now be able to perform SFTP operations. 🚀

Thanks for reading! 🙏

If you have any questions or feedback, feel free to reach out to me on Twitter or ask a question in the comments below. 🥸

Also check out my Dev.to for more articles about AWS, Serverless and Cloud Development. 📝

Don’t be a CRUD boomer 👨‍🦳, check out this new Event Sourcing library!

Valentin BEGGI — Wed, 02 Nov 2022 13:08:17 +0000

Event Sourcing is hard 🧗‍♂️

Implementing Event Sourcing in Typescript is painful. The power of this pattern is often counterbalanced by the complexity of its implementation and the difficulty of its maintainability 🤯. Key features like event replays can be tough to implement 🥵.

At Kumo, many of our projects used Event Sourcing, and we often stumbled upon the same issues again and again and ended up coding the same thing over and over. 🤡 It soon became obvious to us that developers needed new and better tools to do Event Sourcing on their Serverless projects.

❗ Event Sourcing is a pattern for storing data as a list of ordered events. If you are not familiar with this pattern, we invite you to read this Event Sourcing presentation.

We can do better

We thought that the best way to achieve our goal was to create an open-source library that would provide the following advantages:

💭 Abstractions first: Castore has been designed with flexibility in mind. It gives you abstractions that are meant to be used anywhere: React apps, containers, Lambdas... you name it!
⛑️ Full type safety: We love type inference and we know you will too! The lack of decently typed Event Sourcing library was one of the main reasons we created Castore.
📖 Best practices enforced: The Event Sourcing journey has many hidden pitfalls. We ran into them for you! Castore is opiniated. It comes with a collection of best practices and documented anti-patterns that we hope will help you out!
🙅‍♂️ We do NOT deploy resources: While some packages like DynamoDBEventStorageAdapter require compatible infrastructure, Castore is not responsible for deploying it.

Introducing Castore 🦫

As a result, after months of work ✨, we are happy to introduce Castore!

Castore is an open-source Typescript library! It provides a set of classes, methods, and adapters to offer the best typescript developer experience for implementing Event Sourcing and interacting with any storage solution you want to use. V1 of Castore is already live! 🌟

Castore API ⚙️

1. EventStore 🏪

Castore provides you with an EventStore class. This is the entry point of Castore. This class exposes several methods to interact with your events and squeeze data out of them. 🖐️🍋🖐️

To instantiate this class, you will need to define different event types. An event type is nothing more than a type-safe object describing the metadata and payload of your event.

You can use the EventType constructor 👷 to create basic events or use more sophisticated JsonSchemaEventType or ZodEventType to create events with runtime validation! 👨‍🏫

You will also need to provide a reducer when creating the EventStore. The reducer is the function that will be applied to your ordered events to build their summarized view which we call aggregate.

import { EventType } from "@castore/core"

export const userCreatedEvent = new EventType<
  // Typescript EventType
  'USER_CREATED',
  // Typescript EventDetails
  {
    aggregateId: string;
    version: number;
    type: 'USER_CREATED';
    timestamp: string;
    payload: { name: string; age: number };
  }
>({
  // EventType
  type: 'USER_CREATED',
});

const userEventStore = new EventStore ({
  eventStoreEvents: [userCreatedEvent],
  reducer:  (
    userAggregate: UserAggregate,
    event: UserEventsDetails,
  ): UserAggregate => {
    const { version, aggregateId } = event;

    switch (event.type) {
      case 'USER_CREATED': {
        const { name, age } = event.payload;

        return {
          aggregateId,
          version: event.version,
          name,
          age,
          status: 'CREATED',
        };
      }
      ...
    }
  }
  ...
});

2. Storing data 💽

Now that you have defined your EventStore and your types of Events, you want to store Events somewhere. We have created adapters to make your EventStore compatible with different data storage solutions. For now, DynamoDBAdapter and InMemoryAdapter are the two StorageAdapters available with Castore. But you can of course create your own if you want! 🔨

const userEventStore = new EventStore({
  eventStoreEvents: [...], 
  storageAdapter: new DynamoDbEventStorageAdapter({
    tableName: "MyTable", 
    dynamoDbClient: new DynamoDBClient()
  })
})

3. Interacting with the EventStore 🧠

To push and retrieve events from the event store, the EventStore class exposes methods like pushEvent or getAggregate. These methods use the adapter to interact with the data storage entity and add or retrieve events.

Here is a quick example showing how an application would use these two methods:

const removeUser = async (userId: string) => {
  // get the aggregate for that userId,
  // which is a representation of our user's state
  const { aggregate } = await userEventStore.getAggregate(userId);

  // use the aggregate to check the user status
  if (aggregate.status === 'REMOVED') {
    throw new Error('User already removed');
  }

  // put the USER_REMOVED event in the event store 🦫
  await userEventStore.pushEvent({
    aggregateId: userId,
    version: aggregate.version + 1,
    type: 'USER_REMOVED',
    timestamp: new Date().toIsoString(),
  });
};

Go check out the repo to find out about other cool features of Castore including Commands, Snapshots, custom mock helpers... 💫

Conclusion

Castore makes the Typescript developer experience so easy it should be illegal. You will be interacting with your event store like never before, optimization and best practices being abstracted away to let you focus on delivering business value as fast and seamlessly as possible. 🍰

We can’t wait to get your feedback ⭐ and add more features to Castore. The incoming features include events migration 🕊️, Projection classes 📽️... Feel free to contribute !

As a bonus, if you are a Serverless fanboy, here is a cool article to get a better grasp on serverless Event Sourcing architectures and their problematics written by one of our lovely colleagues at Theodo.

Written with ❤️ by @julietteff @charlesgery @thomasaribart @valentinbeggi

The Serverless API Versioning System you wish you had ⚡

Valentin BEGGI — Tue, 12 Jul 2022 11:33:36 +0000

Read if:

You want to know how to version your AWS Serverless API the way pure players do 🔥
You don’t like cold starts ❄️
You enjoy eating lambdas for breakfast 🍳

Introduction

What is API Versioning and why should I care?

Let’s say you are the owner of a very sophisticated API allowing your clients to post a username on your /users route and receive all the information you have about this username.

The request might look like this:

POST /users
body: {"username": "awesomeUser123", "age": 25, "job": "software engineer"}

and the response:

{
    "statusCode": 200,
    "body": {"id": "000024671"}
}

You now need the user's company and refactor the input this way:

{"age": 25, "job": { "title": "software engineer", "company": "Stroupe" }}

It would be a bad practice to update your code and deploy it right away because this would immediately break all the services relying on your API. A better and nicer solution would be to upgrade your API and communicate about the change the right way 📚.

🦸 API Versioning allows you to make breaking changes and keep your clients happy.

How do clients access versioned APIs?

💡 We will use date-based version names like “2022-01-01” (implementation date) instead of traditional v1, v2...
It is easier to track, to use, and to document.

There are 2 main ways your API consumers can query a specific version of your API, either with a path parameter, by calling POST /2022-01-01/users, or by using a dedicated HTTP header.

State of the art players, like Stripe, chose to provide header based API versioning. Indeed, the paths of your API will be kept as simple as possible. It's sexier to call a GET /users than a GET /users/2022-01-01 right?

We will detail two implementations of a header based API Versioning system using AWS API Gateway and Lambda.

Solutions

TL;DR : here is a quick summary of the pros and cons of each solution.
Our team chose solution 1

	Method 1: HTTP Proxy	Method 2: Custom Lambda Integration
Cost	❌ 1 request = 2 API Gateway calls	✅ 1 request = 1 API Gateway call
Performance	❌ Slight performance loss (~20ms) due to proxy.	✅ No loss
Code Simplicity	✅ Less CF Resources	❌ can’t use Lambda Proxy Integration. Pain for Serverless Framework users.
VTL Usage	✅ + (moderate usage)	❌ +++ (intensive usage)

🧠 Skip the next two parts if you’re already familiar with API Gateway REST API internals.

API Gateway Request and Response flow in REST APIs

This is how API Gateway handles request to a REST API.

There are 6 steps.

1️⃣ API Gateway receives the request.

2️⃣ It is passed to the Method Request, where we can perform input validation, headers validation, provide the route with authorizers.

3️⃣ Data is passed to Integration Request. We specify the hanler (a Lambda Function, another AWS Service, another HTTP endpoint …) and manipulate the body / headers that will be passed down using VTL.

4️⃣ The request is processed by the business logic handler 🧠.

5️⃣ In the Integration Response, we manipulate the handler output (status code, body) and override it with our own logic. For example, a 403 from DynamoDB Service could be changed to a 400 for the client with a custom error message.

6️⃣ Similarly to Method Request, we perform data output validation.

API Gateway Integration types

In this part, we will focus on the Integration Request part (3️⃣ in previous paragraph).

API Gateway provides its users several integration types, with a lot of services. We will cover two of them.

1. Lambda Proxy

Because the API Gateway <> Lambda duo is so powerful, API Gateway provides us with a simple, powerful, and nimble mechanism to build an API with the setup of a single API method.

It is the simplest way to integrate lambdas with API Gateway and, for Serverless Framework users, it is what is being built under the hood when you choose to deploy a lambda triggered by a REST API HTTP event.

In your code, your Lambda only returns a status code and a body.

export const greetUser = async ({ name }) => {
  if (typeof name !== "string") {
    return { statusCode: 400, body: "Your name must be a string" };
  }

  return { statusCode: 200, body: `Hello, ${name}!` };

This information is directly passed to the client, skipping Integration Response part.

2. Lambda Service Custom Integration

Directly integrates with other AWS Services APIs, including Lambda.

💡 When integrating with Lambda Function (instead of the more general Lambda Service), you can return a custom status code based on a regex rule applied on the error message of your lambda.
When integrating with AWS Service Lambda, you can map the status code you received from your call to the Lambda Service to a custom status code of your choice.

Now let’s build a header-based versioning system!

Method 1: HTTP Proxy Integration

We will use an intermediate proxy API route redirecting incoming requests to a layer of API routes. Routes in this layer are using Lambda proxy integration and have versions in their paths (e.g. POST users/2022-01-01).

I will define the steps to configure the first API route.

Integration Request

Use HTTP Integration type.
Optionally, use a mapping template to pass down the apiKey to the second API route and set a default version if no version header is provided.

#set($version = $method.request.header.Version)
#set($apiKey = $method.request.header.x-api-key)

#if(!$version ||$version == "")
  #set($version = "2022-01-01")
#end

#set($context.requestOverride.path.version = $version)
#set($context.requestOverride.header.x-api-key = $apiKey)

$input.json("$")

Integration Response

The Integration Response input is the status code API Gateway received from its HTTP call to our sub API Gateway route (potentially a 403 if it was not found).

We will use mapping template to add logic (using Velocity Template Language, VTL) to the response.
You will need a very useful method: $context.responseOverride.status

It allows you to override the final status code.

Now define each possible method response status (i.e. the status codes you want to expose to your client) and map them to the HTTP proxy call status code.

By default, map the HTTP response to a 500. Use a mapping template to return a {message: “Internal Server Error”} object. This way, any untackled behaviour ends up in a 500.
Next, we will take care of the 403 special case. In the case where an unknown version is passed by the client, the HTTP Proxy will try to invoke an undefined route, which will result in an API Gateway 403 response. I chose to return a custom error message telling the client that the requested version was not found.

#set($APIExposedVersions = ['20220101', '20220503'])

#set($version = $method.request.header.Version)

#if($version != "" and !$APIExposedVersions.contains($version))
  #set($context.responseOverride.status = 400)
  {message: "your custom version not found message"}
#else
  {message: "forbidden"}
#end

Finally map all relevant status codes of your API to themselves. They will be passed as is.

It should look like that:

Summing it up!

✅ Configuration remains quite simple

✅ Your lambdas are still integrated with Lambda Proxy

✅ Version can be accessed both with headers and with path

❌ It implies additional costs: each call to the proxy route will generate a second call to versioned routes

❌ Proxying takes around 20 to 40ms.

Method 2: Lambda Service Custom Integration

You have to lambdas, postUser20220101 and postUser20220305.

The goal is to have API Gateway directly call one lambda or the other based on the Version header.

It looks like this:

Let’s detail the Integration Request and Integration Response parts.

Integration Request

Chose AWS Service > Lambda integration
Use path override to tell API Gateway to invoke postUser{Version}
Use URL Path parameters and define version = method.request.header.Version It will allow API Gateway to invoke the right lambda based on the Version header

Integration Response

The integration response input is the Lambda Service status code (which is different from the Lambda status itself, a lambda could be in error but the Lambda Service will return a 200 HTTP status code). The output is the Lambda Service response payload.

Now define each possible method response status and map them to the Lambda Service status code.

First, define a mapping from Lambda Service 500 to a method response 500. Indeed if Lambda Service returns a 500 we just want to pass it through.
Then define a mapping from Lambda Service 404 to a 400.
Using a mapping template, check the payload you received from Lambda Service. If it mentions Lambda not found then override the response message with your custom Version not found message. To do that, just return in VTL an object with the information you want to provide your client with.
For example:
```
{"message": "The API version you requested: 
'$method.request.header.Version' is not managed" }
```
Else, use $context.responseOverride.status to override the status code to a 500
Finally, define a default mapping to 200. You can easily access, using VTL, the payload of your lambda response. Make sure your lambda returns an object containing the expected statusCode (the same as you would do with Lambda Proxy). Parse the payload and override the statusCode, using $context.responseOverride.status, with your Lambda’s.
That’s it, you kind of reimplemented the Lambda Proxy integration for the sake of versioning and client satisfaction 🎊.

We’re done, now fine-tune your VTL to make your version error message more explicit, by adding a list of supported versions for example!

Summing it up!

✅ Implies no additional cost

❌ Requires lots of API Gateway twisting and you can’t use Lambda Proxy integration.

❌ If you use the Serverless Framework, you will need to write a lot of cloudformation to deploy your versioned lambdas

Conclusion

We’ve learned two ways to define a header based API versioning system using AWS API Gateway and AWS Lambda. No unnecessary work is done by additional Lambdas, thus keeping cold starts as low as possible. All the work is done by blazing fast API Gateway internals 🔥.

I purposefully kept the implementation details light and simplified them the more that I could.
If you chose to implement one of these solutions but find it hard to do so, feel free ton contact me on Twitter or by leaving a comment below, I will be more than ✨API to help✨!

If you achieved the same goal a different way, please tell me how in the comments 💪

Cheers ☀️

Integration testing tool for DynamoDB

Valentin BEGGI — Tue, 11 Jan 2022 13:49:11 +0000

Integration testing in serverless architectures can be challenging. Testing specific outcomes within managed services is cumbersome. sls-test-tools provides a range of utilities, setup, teardown and assertions to make it easier to write effective and high quality integration tests for Serverless Architectures on AWS.

I’m happy to announce that sls-test-tools now ships with new DynamoDB assertions!

Let's jump in a quick example 🏃‍♂️ !

1️⃣ Function to test

Let's consider a simple Lambda function that uploads client transaction data to DynamoDB.

import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { DynamoDBDocumentClient, PutCommand } from "@aws-sdk/lib-dynamodb";

type TransactionEvent = { clientName: string; transactionData: string };

const ddbDocClient = DynamoDBDocumentClient.from(new DynamoDBClient({}));

export const handler = async (event: TransactionEvent): Promise<string> => {
  const { clientName, transactionData } = event;

  await ddbDocClient.send(
    new PutCommand({
      TableName: "Sells",
      Item: {
        PK: clientName,
        SK: new Date().toISOString(),
        transactionData,
      },
    })
  );

  return "Transaction saved !";
};

As you can see, the function just makes a single "put" call to DynamoDB using the AWS SDK. In our integration test we want to make sure that the data has indeed been written to Dynamo.

2️⃣ Writing the test!

We will be using sls-test-tools new assertion toExistInDynamoTable.
The integration testing can be done in 3 phases:

Triggering the initial event 🔥. In our scenario we call our lambda handler but we could also imagine sending an event in an Event Bus, uploading a file to S3...
Asserting the expected behavior ✅.
Cleaning what has been created 🧹. This step is very important in order to keep tests idempotent.

Following these three steps, the integration test implementation would be:

import { DocumentClient } from "aws-sdk/clients/dynamodb";
import MockDate from "mockdate";
import { AWSClient } from "sls-test-tools";

import { handler } from "./uploadTransactionDataToDynamo";

describe("Sells data upload integration testing", () => {
  const documentClient: DocumentClient = new AWSClient.DocumentClient();

  const mockDate = "2022-01-01T00:00:00.000Z";
  MockDate.set(mockDate);

  const event = { clientName: "John", transactionData: "someData" };

  afterAll(async () => {
    // 🧹 clean what you've created
    await documentClient.delete({
      TableName: "Sells",
      Key: { PK: "John", SK: mockDate },
    });
  });

  it("should upload transaction data to Dynamo", async () => {
    // 🔥 trigger the initial event
    await handler(event);

    // ✅ assert the functional behavior you are testing
    await expect({
      PK: "John",
      SK: mockDate,
    }).toExistInDynamoTable("Sells");
  });

All sls-test-tools assertions can of course be inverted using expect(..).not.

You can use those assertions on all type of DynamoDB table, on a composite primary index table containing a partition key PK and a sort key SK like in the previous example, as well as in simpler table where primary index is only a partition key.

it("should upload transaction data to Dynamo", async () => {
  await handler(event);

  await expect({ PK: "John" }).toExistInDynamoTable("Sells");

  await documentClient.delete({
    TableName: "Sells",
    Key: { PK: "John", SK: mockDate },
  });
});

3️⃣ What's next !

Our team is working on typing sls-test-tools and on adding more custom jest assertions. We are also writing a more exhaustive article on Serverless integration testing, please feel free to subscribe so you can be notified when it will come out!

⭐️⭐️⭐️ Happy 2️⃣0️⃣2️⃣2️⃣ ⭐️⭐️⭐️

For more background about sls-test-tools, you can check out these two articles: Integration Testing Strategy for EventBridge and Bringing Simplicity to Serverless Integration Testing.

Forem: Valentin BEGGI

🧙 Lambdalith Mastery: Elevating NestJS Deployment on AWS Lambda with CDK & Webpack

TL;DR 📚

The Lambdalith Edge for NestJS on AWS Lambda 🌟

Setting Up the Infrastructure with AWS CDK 🏗️

Prepping the NestJS App for Lambda 🦁

Packing it Up with Webpack 📦🧙‍♂️

Deploying the NestJS App: To the Cloud! ☁️

Your High-Performance NestJS App now lives on AWS 🚀

🥇 The Lambdalith Advantage: A Complete Guide to NestJS Deployment on AWS Lambda Using CDK

TL;DR 📚

The Lambdalith Edge for NestJS on AWS Lambda 🌟

Setting Up the Infrastructure with AWS CDK 🏗️

Prepping the NestJS App for Lambda 🦁

Packing it Up with Webpack 📦🧙‍♂️

Deploying the NestJS App: To the Cloud! ☁️

Your High-Performance NestJS App now lives on AWS 🚀

AWS Lambda Meets Bun through SST 🐰: Prime Time or Just Prime Hype? 🥵

TL;DR

1️⃣ Load the Bun Layer to your AWS Account

2️⃣ Deploy a new SST app 🔥

3️⃣ Add the Bun Layer to your lambda, and execute bun 🐰

4️⃣ Conclusion & Caveats 🥲

🥇 The Best Serverless Framework in 2023: A Data-Driven Showdown for AWS Projects

1️⃣ Load the Bun Layer to your AWS Account

2️⃣ Deploy a new SST app 🔥

3️⃣ Add the Bun Layer to your lambda, and execute bun 🐰

Conclusion & Caveats 🥲

🥇 The Best Serverless Framework in 2023: A Data-Driven Showdown for AWS Projects

TL;DR

Why ❓

The contenders 🥊

Comparison Table 📈

Our recommendation 🥇

Wrapping Up 🌟

Deploying a Lambda with a static IP has never been so simple 🍰

TL;DR

1️⃣ Create a VPC hosted Lambda function with a static IP

☁️ VPC, NAT Gateway and Elastic IP

⚡ Deploy the Lambda function

2️⃣ Perform SFTP operations in your Lambda function

💽 Store the file you want to send in your Lambda's /tmp folder

The catch 🎣

Deploying a Lambda with a static IP has never been so simple 🍰

TL;DR

1️⃣ Create a VPC hosted Lambda function with a static IP

☁️ VPC, NAT Gateway and Elastic IP

⚡ Deploy the Lambda function

2️⃣ Perform SFTP operations in your Lambda function

💽 Store the file you want to send in your Lambda's /tmp folder

The catch 🎣

Don’t be a CRUD boomer 👨‍🦳, check out this new Event Sourcing library!

Event Sourcing is hard 🧗‍♂️

We can do better

Introducing Castore 🦫

Castore API ⚙️

1. EventStore 🏪

2. Storing data 💽

3. Interacting with the EventStore 🧠

Conclusion

The Serverless API Versioning System you wish you had ⚡

Introduction

What is API Versioning and why should I care?

How do clients access versioned APIs?

Solutions

API Gateway Request and Response flow in REST APIs

API Gateway Integration types

1. Lambda Proxy

2. Lambda Service Custom Integration

Method 1: HTTP Proxy Integration

Integration Request

Integration Response

Method 2: Lambda Service Custom Integration

Integration Request

Integration Response

Conclusion

Integration testing tool for DynamoDB

1️⃣ Function to test

2️⃣ Writing the test!

3️⃣ What's next !

💽 Store the file you want to send in your Lambda's `/tmp` folder

💽 Store the file you want to send in your Lambda's `/tmp` folder