Forem: Subbusainath

Building Production Ready Lambda Extensions: Best Practices from 6 years of Serverless Experience

Subbusainath — Wed, 05 Mar 2025 17:28:19 +0000

Introduction:

After six years of diving deep into serverless architectures, I've seen the evolution of AWS Lambda first-hand. One of the most powerful additions in recent times is Lambda Extensions. They offer a way to augment Lambda functions with custom logic, but like any powerful tool, they require careful handling to avoid pitfalls. In this post, I'll share my insights on building production-ready Lambda Extensions with a focus on best practices, gleaned from real-world experiences.

Section 1: Understanding the Foundation – The Lambda Extension Lifecycle

Deep Dive into the Lifecycle:

It's crucial to understand the INIT, INVOKE, and SHUTDOWN phases.

Before we deep dive into practises, lets grasp the core of Lambda Extensions lifecycle which dictates how your extension interacts with the lambda function
- INIT Phase: Phase 1 where your extension get initializes. Consider this as a setup stage where you will often see configuration loading, connections to external services and necessary pre-processing happening here.
  - For Example, Think like you want to use a database in your code and consider this INIT Phase as connection pool establishing which help us to use the database for any subsequent invocations.
- INVOKE Phase: Phase 2 where most of our work happens, our extension get notified when the lambda function invoked. In here, you can perform actions before, after or even during the function execution.
  - For Example, A tracer extension which might capture the start and end times of the function invocation, for performance analysis.
- SHUTDOWN Phase: Final Phase of Our Extensions lifecycle, This phase will get the signal from the lambda environment when it is going to shutdown. So, that extension and wrap up its final process with the lambda function.
  - For Example, A database extension which help us to create a connection with the database from lambda, in this phase it will close its connection pool and prevent the resource leaks
Key takeaway : Understand the timing of each stage. For example, avoid heavy I/O operations during the INVOKE stage's extension portion.
Internal vs. External Extensions:

Clarify the differences and when to use each type. External extensions are great for long-running processes, while internal extensions are ideal for quick, in-process enhancements.
- Internal Extension
  - Run as separate processes within same lambda execution environment as the existing lambda function code. It shares same resources as the function.
  - Benefits - Easy Integration , Low Latency , Simplified Observability
- External Extensions
  - Run as separate processes outside the lambda function’s runtime environment
  - Benefits - Long Running Tasks, Language Flexibility, Persistent Processes, Asynchronous Processing and Resource Isolation

Section 2: Building Robust and Performant Extensions – Core Best Practices

Language Choice and Performance:

Go and Rust are excellent choices for extensions due to their performance characteristics. Python or Node.js can be used, but be mindful of potential overhead.
- Key takeaway : Consider the performance impact of your language choice.
Asynchronous Operations:

Leverage asynchronous operations to avoid blocking the main Lambda function. This is especially important for I/O-bound tasks.
- Key takeaway : Avoid blocking the Lambda functions execution time.
Error Handling and Resilience:

Implement robust error handling to prevent extension failures from impacting the Lambda function. Use retry mechanisms and circuit breakers where appropriate.
- Key takeaway : Build extensions that are fault-tolerant.
Logging and Monitoring:

Comprehensive logging and monitoring are essential for troubleshooting and performance analysis. Use structured logging and integrate with CloudWatch Logs.
- Key takeaway : Observability is paramount.
Configuration Management:

Avoid hardcoding configurations. Use environment variables or a configuration service to manage settings.
- Key takeaway : Externalize configuration.

Section 3: Security Considerations – Protecting Your Extensions

Principle of Least Privilege:

Grant extensions only the necessary permissions. Avoid overly permissive IAM roles.
- Key takeaway : Minimize the attack surface.
Secrets Management:

Never embed secrets directly in your extension code. Use AWS Secrets Manager or Parameter Store to securely retrieve secrets.
- Key takeaway : Protect sensitive data.
Code Reviews and Security Audits:

Regular code reviews and security audits are crucial for identifying and mitigating potential vulnerabilities.
- Key takeaway : Proactive security is essential.

Section 4: Deployment and Versioning – Streamlining the Process

Packaging and Deployment:

Use AWS SAM, AWS CDK or CloudFormation to automate the deployment of your extensions. This ensures consistency and reproducibility.
- Key takeaway : Automate deployment for consistency.
Versioning and Rollbacks:

Implement a versioning strategy for your extensions to facilitate rollbacks in case of issues.
- Key takeaway : Plan for rollbacks.
Testing:

Create unit and integration tests for your extensions to ensure they function correctly.
- Key takeaway : Test thoroughly.

Conclusion:

Lambda Extensions are a powerful tool for enhancing serverless applications. By following these best practices, you can build robust, performant, and secure extensions that improve the overall reliability and efficiency of your Lambda functions. Remember to prioritize performance, security, and observability in your extension development process. I encourage you to experiment with extensions in your own projects and share your experiences.

How to create L1,L2 and L3 CDK Constructs ? and When to use it ?

Subbusainath — Sun, 11 Feb 2024 18:57:15 +0000

Introduction

After a gap, here I am! with one of the key concepts to understand while using the CDK constructs in your CDK codes. In this blog, I am going to talk about the details of the CDK constructs and how to use them better to ease our resource creation.

Pre-Requisites

Y’all need these pre-requisites to follow along with this blog.

Code Editor of your choice ( I use VSCode )
AWS Account ( Required )
AWS CDK ( Required )
- Install CDK using following command. I am using WSL. So, I chose NPM
```
npm install -g aws-cdk
```
- Initialize your CDK project with a sample app by executing this command in your desired folder. We are going to use this sample app in this blog going further. I am choosing typescript as the coding language for this blog. But you can choose whatever language CDK supports and that you like!
```
cdk init sample-app --language=typescript
```
- Once, you executed the above-mentioned command. You will see files like these in the folder where you initiate the project. For Example,

Basic Knowledge of AWS Resources

What is CDK ?

CDK stands for Cloud Development Kit, which is an open source framework created by AWS to use it as an IaC (Infrastructure as a Code) that uses common programming languages to create AWS resources. Under the hood by provisioning it through AWS Cloudformation. In simple terms, you will be using a common programming language to create an AWS resource without worrying about YAML files or JSON files.

What is Constructs in CDK ? What are the different level of CDK Constructs ?

It contains code components that represent the AWS resources and their configurations, which are reusable. If anyone wants to create any resources from AWS, we can use these constructs to create those resources at ease whenever we want to. It’s like a building block that we use to build certain types of buildings while we are playing.

There are 3 different levels of CDK Constructs.

L1, which are very low-level constructs that directly map to AWS Cloudformation.
L2, which are higher-level constructs that are built on top of L1, It simplifies the configuration and management of AWS resources by providing convenient APIs.
And finally, L3 is a pre-configured construct that encapsulates some best practices and common use cases. It uses “patterns” (known as)

CDK Constructs enable you to define and deploy AWS resources in a programmatic and scalable manner, making it easier to manage infrastructure as code and automate the provisioning of resources in your AWS environment.

L1 , L2 and L3 Constructs - Examples

In this, we are going to create an API that returns hello world by creating apigateway and lambda in L1 constructs.

import { Stack, StackProps } from 'aws-cdk-lib';
import * as lambda from 'aws-cdk-lib/aws-lambda'
import * as apigatewayv2 from 'aws-cdk-lib/aws-apigatewayv2'
import { Construct } from 'constructs';

export class CdkConstructsStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    // this is l1 construct  code for apigateway
    const helloApi = new apigatewayv2.CfnApi(this, "helloAPi", {
      apiKeySelectionExpression: "string",
      basePath: "/",
      body: JSON,
      bodyS3Location: {
        bucket: "string",
        etag: "string",
        key: "string",
        version: "string",
      },
      corsConfiguration: {
        allowCredentials: false,
        allowHeaders: ["Content-Type/applicaton-json"],
        allowMethods: ["GET"],
        allowOrigins: ["CORS"],
        exposeHeaders: ["Content-Type/applicatio-json"],
        maxAge: 1,
      },
      credentialsArn: "string",
      description: "basic template to for l1 construct for an api",
      disableExecuteApiEndpoint: false,
      disableSchemaValidation: false,
      failOnWarnings: true,
      name: "helloApiL1Construct",
      protocolType: "string",
      routeKey: "string",
      routeSelectionExpression: "string",
      tags: {
        "name": "sai"
      },
      target: "string",
      version: "1.0",
    })
    // this is l1 construct code for lambda
    const helloLambda = new lambda.CfnFunction(this, "hello-world-l1", {
      architectures: ["arm_64"],
      code: {
        imageUri: "string",
        s3Bucket: "string",
        s3Key: "string",
        s3ObjectVersion: "string",
        zipFile: "string",
      },
      description: "this is just an hello world lambda using l1 constructs",
      environment: {
      },
      ephemeralStorage: {
        size: 512,
      },
      fileSystemConfigs: [],
      functionName: "helloFunction",
      handler: "string",
      imageConfig: {
        command: [],
        entryPoint: [],
        workingDirectory: "string",
      },
      kmsKeyArn: "string",
      layers: [],
      loggingConfig: {
        applicationLogLevel: "string",
        logFormat: "string",
        logGroup: "string",
        systemLogLevel: "string",
      },
      memorySize: 512,
      packageType: "string",
      reservedConcurrentExecutions: 4,
      role: "string", // Required
      runtime: "NODEJS20.X",
      runtimeManagementConfig: {
        runtimeVersionArn: "string",
        updateRuntimeOn: "string",
      },
      snapStart: {
        applyOn: "string",
      },
      tags: [],
      timeout: 60,
      vpcConfig: {
        ipv6AllowedForDualStack: false,
        securityGroupIds: [],
        subnetIds: [],
      },
    })

    // this is for l1 construct for lamnda integration with api
    const helloApiIntegration = new apigatewayv2.CfnIntegration(this, "apiIntegr", {
      apiId: helloApi.attrApiId, // Required
      connectionId: "string",
      connectionType: "string",
      contentHandlingStrategy: "string",
      credentialsArn: "string",
      description: "string",
      integrationMethod: "GET",
      integrationSubtype: "string",
      integrationType: "AWS_PROXY", // Required
      integrationUri: helloLambda.attrArn,
      passthroughBehavior: "string",
      payloadFormatVersion: "string",
      requestParameters: JSON,
      requestTemplates: JSON,
      responseParameters: JSON,
      templateSelectionExpression: "string",
      timeoutInMillis: 29,
      tlsConfig: {
        serverNameToVerify: "string",
      },
    })

    // this is l1 construct of iam for lambda with apigateway
    const lambdaPolicy = new lambda.CfnPermission(this, "id", {
      action: "ALLOW", // Required
      eventSourceToken: "string",
      functionName: helloLambda.attrArn, // Required
      functionUrlAuthType: "string",
      principal: "AWS_IAM", // Required
      sourceAccount: "string",
      sourceArn: helloApi.attrApiEndpoint,
    })
  }
}

In the above-mentioned snippet, you can see the exact properties that are similarly available in cloudfromation template for each resource. Which gives you the advantage of creating the resources with customization.

Same as above, hello world endpoint, but this time we used L2 Construct to create that resource.

// this is l2 construct  code for apigateway
    const helloApi = new RestApi(this, 'helloApiId', {
      restApiName: `helloRestApiName`,
      description: "This is an api using L2 Construct",
      endpointTypes: [EndpointType.EDGE]
    })

    // this is l2 construct for Lambda
    const helloLambda = new Function(this, 'helloFunction', {
      handler: "hello-handler.handler",
      runtime: Runtime.NODEJS_20_X,
      code: Code.fromAsset("./lib/functions/hello-handler.ts"),
      description: "this is a helloLambda using L2 Construct",
      architecture: Architecture.ARM_64,
      functionName: "L2HelloLambdaFunction"
    })

    // this is lambda integration using l2 construct

    const root = helloApi.root.addResource('hello')
    root.addMethod('GET', new LambdaIntegration(helloLambda))


    // this is lambda permission using l2 construct
    helloLambda.addPermission("lambdaPermissionForApiGateway", {
      principal: new ServicePrincipal("apigateway.amazonaws.com"),
      action: "lambda.InvokeFunction",
      sourceArn: helloApi.arnForExecuteApi('GET', "/hello", "DEV")
    })

But there are fewer properties with the abstracted layer and fewer lines of code with some of the default options.

So far, we have repeated code to create a simple hello-world endpoint. We are wasting so much time creating a simple endpoint along with lambda integration. Let's reduce this code repetition using the L3 construct.

First, we need to create our custom-patterned apigateway lambda construct like this:

import * as cdk from 'constructs';
import * as apigateway from 'aws-cdk-lib/aws-apigateway';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import { ServicePrincipal } from 'aws-cdk-lib/aws-iam';

export interface ApiGatewayLambdaConstructProps {
  /** Name of the API Gateway */
  readonly apiName: string;
  /** Lambda function to integrate with */
  readonly lambdaFunction: lambda.Function;
  /** Resources and methods to define (optional) */
  readonly resources?: Map<string, apigateway.MethodOptions>;
  /** Lambda Permission to define */

  lambdaPermission: Record<string, lambda.Permission>;
}

export class ApiGatewayLambdaConstruct extends cdk.Construct {
  constructor(scope: cdk.Construct, id: string, props: ApiGatewayLambdaConstructProps) {
    super(scope, id);

    const api = new apigateway.RestApi(this, props.apiName, {
        endpointTypes: [apigateway.EndpointType.EDGE]
    });

    // Add resources and methods based on props
    if (props.resources) {
      for (const [resourcePath, methodOptions] of props.resources.entries()) {
        const resource = api.root.addResource(resourcePath);
        resource.addMethod('GET', new apigateway.LambdaIntegration(props.lambdaFunction));
      }
    }

    props.lambdaPermission["permission"] = {
        principal: new ServicePrincipal("apigateway.amazonaws.com"),
        action: "lambda.InvokeFunction",
        sourceArn: api.arnForExecuteApi('GET', "/hello", 'DEV')
    }
  }
}

Once we build our custom construct, we can import it into our cdk stack like normal stacks.

// this is l2 construct for Lambda
    const helloLambda = new Function(this, 'helloFunction', {
      handler: "hello-handler.handler",
      runtime: Runtime.NODEJS_20_X,
      code: Code.fromAsset("./lib/functions/hello-handler.ts"),
      description: "this is a helloLambda using L2 Construct",
      architecture: Architecture.ARM_64,
      functionName: "L2HelloLambdaFunction"
    })

    // this to create /hello endpoint with lambda integration using L3 Construct 

    new ApiGatewayLambdaConstruct(scope, 'HelloApiGatewayLambdaConstruct', {
      apiName: "helloApi",
      lambdaFunction: helloLambda,
      lambdaPermission: {
        "resource": {
          principal: new ServicePrincipal("apigateway.amazonaws.com"),
          action: "lambda.InvokeFunction",
        }
      }
  })
  }

Like these, a few lines! We have now created an endpoint along with the lambda integration. It is very customizable and reusable. This is how we will be able to create the L3 construct. In the next section, we will be talking about when and where to use these constructs.

Where to use L1 Constructs & where not ?

If you need to have maximum granularity on the AWS resources for every single property, As I have mentioned before, L1 Constructs has a direct map with AWS Cloudformation.
If you need to try out any new services that have been released by AWS recently, You can try it out in L1 constructs before it is available in high-level constructs like L2 and L3.
It helps you troubleshoot issues with high-level constructs. Since both the L2 and L3 constructs are built on top of the L1 constructs,
If you want to try out unique cases that required to access specific Cloudformation properties.
If you don’t have in-depth knowledge about the cloudformation properties of AWS resources, then L1 is not suitable for you to use. Instead, use L2 or L3.
Maintaining resource creation is very hard in L1 constructs.
The portability issue is there. Since it is tightly coupled with cloudformation-related.

Where to use L2 Constructs & where not ?

If you don’t want to create any AWS resources from scratch by providing all information into the constructs similar to L1 Constructs and are willing to use some default configurations in the desired resources, you can use L2 Constructs.
Use it if you want to have consistency and a set of standards in your resource creation without any complexity.
Use it when you want to imply security practices for your AWS resources.
Use it when you don’t want to waste your time without a need.
It has limited configuration flexibility. So, if you want to use it for some edge-case scenarios, It is not suitable.
Troubleshooting the issue might be tricky since it adds more abstraction to resource creation. Just remember this while you are using L2 constructs.

Where to use L3 Constructs & where not ?

Use it when you want to reduce the complexity of creating resources with common patterns and redundant boilerplate code. It helps you maintain your resource stacks hustle-free.
Use it when you want to enforce a pattern for creating common infrastructure patterns, which saves you time. For example, as shown in the above L3 Construct Code Snippets Example, creating an ApiGateway with an endpoint backed by a lambda function.
Use it when you want to reuse it across the stacks. This will improve code consistency.
Use it when you want to create your own custom constructs that are not yet covered by the existing constructs. It gives you full control and flexibility over your infrastructure.
L3 constructs are not suitable if you want to have fine-grained access controls on your AWS resources. Because L3 constructs might be too restrictive.
It is not suitable if it does not have any common use cases where it is not effective as well.

💡 You can even publish your custom construct on https://constructs.dev/contribute

Conclusion

Always remember to understand the use of constructs when you use them for your purpose. CDK is a powerful tool; use it responsibly. Like in the movie Spiderman, where Uncle Ben advises Spiderman

With great power comes great responsibility

See you in next blog !

💡 official docs link for CDK : https://docs.aws.amazon.com/cdk/v2/guide/home.html

You can follow me on social medias:

Instagram : @_valandhavaney_

X : @SubbuSainath

Personal Site: subbusainath's biolink

LinkedIn: subbusainath-rengasamy

Github: subbusainath

How to create server side pagination REST api endpoint using nodejs, prisma with Postgresql?

Subbusainath — Mon, 19 Jul 2021 17:46:44 +0000

We know how prisma works well with GraphQL.But, Most of the programmers still use RESTful API over GraphQL. So here i am , Going to create the REST Api endpoint for pagination from server side with the help of well renown NodeJS with quality ORM for Postgresql (RDBMS).

Why nodejs ?

NodeJS is primarily used in backend developement since it is an single-threaded nature.And it is designed with real-time,push-based architecture. Most important thing of nodejs is IT IS AN OPENNNNNNNNNNNNNNNNN SOURCEEEEEEEEEEEEEEEEEEEEE🔥!

Tools used to start!

NodeJS V_12.X to latest
Prisma -version 2.16.1 @prisma/client, prisma-studio
PostgerSQL -version 9.3.1
Nodemon -2.0.7 (Looking for changes in file and automatically restart the server)
Visual Studio Code-Insider (probably for Mac M1 chip)
Dotenv - 8.2.0 for environment variable
Insomnia - 2021.2.2 for testing your API endpoints (You can use Postman)

Lets Start to Develope

Create a folder with the name you desire . In my case , I named my folder as "Server-side-Pagination"
Open your vscode with the folder
Run npm init to create package.json file



npm init

Your package.json file will be created with default structure and add dependencies like the snapshot which is mentioned below

Project Structure

Prisma schema setup for database models to create tables to store data



generator client {
  provider      = "prisma-client-js"
  binaryTargets = ["native", "rhel-openssl-1.0.x"]
}

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

model forensicDetails {
  id          Int     @id @default(autoincrement())
  first_name  String?
  last_name   String?
  Criminal     String?
  Blood_group String?
  height    Int?
  weight    Int?
  Evidence  String?
}

Run the following command



❯ dotenv -e .env npm run prisma:generate
❯ dotenv -e .env npm run prisma:migrate

first command is for generating prisma client

second command is for migrate it to local databse

Here comes the major part of this blog

In this blog, i am gonna use offset limit pagination which is good but not good as cursor pagination where it takes Primary Keys as a identifier for processing search.

Actually, How does offset works in prisma?



take: limit(data to retrieve from the table)
skip: offset(data to be skipped from the table)

offset based pagination is a popular technique where the client makes a request with parameters for limit(number of results) and offset(number of records to skip).

pagination.js

Folowing code snippet is for write a data into the table called forensicDetails.



const express = require("express");
const { PrismaClient } = require("@prisma/client");
const prisma = new PrismaClient();
const app = express();
const port = process.env.PORT || 5000;
app.use(express.json());
app.listen(port, () => {
  console.log(`Server is Running on the ${port}`);
});
app.post("/forensic", async (req, res) => {
  const {
    first_name,
    last_name,
    Criminal,
    Blood_group,
    height,
    weight,
    Evidence,
  } = req.body;
  return await prisma.forensicDetails
    .create({
      data: {
        first_name,
        last_name,
        Criminal,
        Blood_group,
        height,
        weight,
        Evidence,
      },
    })
    .then((data) => {
      return res.status(201).json(data);
    })
    .catch((err) => {
      return res.status(500).json(err);
    });
});

As we use LIMIT and OFFSET condition for selecting limited data from table. As i said earlier, we are going to implement this method use PRISMA ORM.



app.get("/forensic/page", async (req, res) => {
  const query = req.query;
  const page = parseInt(query.page) || 1;
  const limit = parseInt(query.limit) || 2;
  const last_page = req.query.last_page;
  const startIndex = (page - 1) * limit;
  const endIndex = page * limit;
  const result = {};
  const totalCount = await prisma.forensicDetails.count();
  const totalPage = Math.ceil(totalCount / limit);
  const currentPage = page || 0;
  try {
    if (page < 0) {
      return res.status(400).json("Page value should not be negative");
    } else if (page === 1 && !last_page) {
      result.totalCount = totalCount;
      result.totalPage = totalPage;
      result.currentPage = currentPage;
      result.next = {
        page: page + 1,
        limit: limit,
      };
      result.paginateData = await prisma.forensicDetails.findMany({
        take: limit,
        skip: startIndex,
        orderBy: {
          id: "desc",
        },
      });
      res.paginatedResult = result;
      result.currentCountPerPage = Object.keys(result.paginateData).length;
      result.range = currentPage * limit;
      return res.status(200).json(result);
    } else if (endIndex < totalCount && !last_page) {
      result.totalCount = totalCount;
      result.totalPage = totalPage;
      result.currentPage = currentPage;
      result.next = {
        page: page + 1,
        limit: limit,
      };
      result.paginateData = await prisma.forensicDetails.findMany({
        take: limit,
        skip: startIndex,
        orderBy: {
          id: "desc",
        },
      });
      res.paginatedResult = result;
      result.currentCountPerPage = Object.keys(result.paginateData).length;
      result.range = currentPage * limit;
      return res.status(200).json(result);
    } else if (startIndex > 0 && !last_page) {
      result.totalCount = totalCount;
      result.totalPage = totalPage;
      result.currentPage = currentPage;
      result.previous = {
        page: page - 1,
        limit: limit,
      };
      result.paginateData = await prisma.forensicDetails.findMany({
        take: limit,
        skip: startIndex,
        orderBy: {
          id: "desc",
        },
      });
      res.paginatedResult = result;
      result.currentCountPerPage = Object.keys(result.paginateData).length;
      result.range = currentPage * limit;
      return res.status(200).json(result);
    } else if (last_page === "true" && page === totalPage) {
      result.totalCount = totalCount;
      result.totalPage = totalPage;
      result.currentPage = totalPage;
      result.last = {
        page: totalPage,
        limit: limit,
      };
      result.paginateData = await prisma.forensicDetails.findMany({
        take: limit,
        skip: startIndex,
        orderBy: {
          id: "desc",
        },
      });
      res.paginatedResult = result;
      result.currentCountPerPage = Object.keys(result.paginateData).length;
      result.range = totalCount;
      return res.status(200).json(result);
    } else {
      return res.status(404).json({ error: "Resource not found" });
    }
  } catch (err) {
    console.error("error", err);
    return res.status(500).json(err);
  }
});

here the logic i used is User is able to adjust there LIMITs what ever they like except negative value.

Every page will be displayed with default LIMIT of 2. I stored the values in Objects.

Format for the api end point is consists of totalCount,totalPage,CurrentPage,next or previous or last ,paginatedData as array of objects, currentCountPerPage and range. This endpoint consists all the data preferable by front end developers.

Below mentioned screenshot , this is how it would look like in api endpoint testing tool

here i used insomnia,

Conclusion

This is how we able to create rest api endpoint for server side pagination using node js , Postgresql with Prisma. We can tweak this code based on the requirement by your friend developers.

Follow me for more technical blogs on Twitter.
Twitter
you can find complete code here Github link:
Blog_code