Forem: Rajan Panchal

What is Lambda Layers and how to use it with Java runtime?

Rajan Panchal — Thu, 19 Nov 2020 05:04:59 +0000

When developing serverless functions, you often need additional dependencies or libraries for the function to work. For Lambda functions in java, you need amazon dependencies of the services used in the function. Then you zip the function and its dependencies into a fat jar and upload it to the Lambda function. If you are not familiar with creating lambda function in java then refer to this post for a quick start.

The fat jar approach works fine but if you have multiple functions that use the same dependencies then creating a fat jar of each function is not a good approach and maintenance of such code is also difficult. A change in the version of a jar requires a change in each of the individual lambda functions.

A more elegant solution is to use Lambda layers.

What is Lambda Layers?

A layer is a ZIP archive that contains libraries, a custom runtime, or other dependencies. With layers, you can use libraries in your function without needing to include them in your deployment package. Layers let you keep your deployment package small, which makes development easier.

Layers are extracted to the /opt directory in the function execution environment. Each runtime looks for libraries in a different location under /opt, depending on the language.

You can specify up to 5 layers in your function's configuration, during or after function creation. You choose a specific version of a layer to use. If you want to use a different version later, update your function's configuration.

Let's see it in action

We will write a basic Lambda function that would just print lambda environment variables and event details. We will create 2 maven projects. In the first project, we include amazon and other essential dependencies. This project will serve as our Lambda Layer. In the second project, we include our first project as a dependency and will write the Lambda handler.

Create the first maven project called Lambda-Layer-Base and update pom as below:
This project contains all common dependencies required by the functions. In our case, we added lamdba-core and gson dependency. We also added maven-shade-plugin to create fat jar.

<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>net.rajanpanchal</groupId>
    <artifactId>lambda-layer-base</artifactId>
    <version>1</version>
    <name>Lambda-Layer-Base</name>
    <description>Lambda base</description>
    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <aws.java.sdk.version>2.14.11</aws.java.sdk.version>
    </properties>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>bom</artifactId>
                <version>${aws.java.sdk.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-lambda-java-core</artifactId>
            <version>1.2.0</version>
        </dependency>

        <dependency>
            <groupId>com.google.code.gson</groupId>
            <artifactId>gson</artifactId>
            <version>2.8.6</version>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>${maven.compiler.source}</source>
                    <target>${maven.compiler.target}</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.2.4</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

This project will only contain dependencies. Now execute maven clean install and it should generate a jar (lambda-layer-base-1.jar) in the target folder.

Create another maven project and update pom with the below contents. We included our first project as a dependency (lambda-layer-base). Also note, we no longer use maven shade plugin since we don't want dependencies to be included in the jar.

<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>net.rajanpanchal</groupId>
    <artifactId>lambda-java-demo</artifactId>
    <version>1</version>
    <packaging>jar</packaging>
    <name>Lambda-Layers-Demo</name>
    <description>HelloWorld Lambda Layers Demo</description>
    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <aws.java.sdk.version>2.14.11</aws.java.sdk.version>
    </properties>
    <dependencies>
    <dependency>
        <groupId>net.rajanpanchal</groupId>
        <artifactId>lambda-layer-base</artifactId>
        <version>1</version>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>${maven.compiler.source}</source>
                    <target>${maven.compiler.target}</target>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

Add a class file in src/main/java that will contain Lambda logic. Here we are just logging some stuff.

package net.rajanpanchal.handlers;

import java.util.Map;

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.LambdaLogger;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;

public class HelloWorldHandler implements RequestHandler<Map<String,String>, String>{

      public String handleRequest(Map<String,String> event, Context context)
      {
        LambdaLogger logger = context.getLogger();
        String response = new String("200 OK");
        logger.log("ENVIRONMENT:"+System.getenv());
        // log execution details
        Gson gson = new GsonBuilder().setPrettyPrinting().create();
        logger.log("ENVIRONMENT VARIABLES: " + gson.toJson(System.getenv()));
        logger.log("CONTEXT: " + gson.toJson(context));
        // process event
        logger.log("EVENT: " + gson.toJson(event));
        logger.log("EVENT TYPE: " + event.getClass().toString());
        return response;
      }
    }

Execute maven package command to create the jar for the project. A jar (lambda-java-demo-1.jar) should get created in target folder.

Creating Lamdba Layer

Before we head to the console to create the layer, let's complete one important step. Remember that layers gets extracted to a folder in /opt directory. You need to package the layer in folder structure as per your runtime. For java, you need to package jar in java/lib folder. The final folder structure, in the lambda runtime, would be /opt/java/lib/<your jar>. So now create folders java/lib and copy jar from the first project and zip the folder. Now we are ready to upload the jar as lambda layer.

Creating Lambda function

Now create the lambda function and specify the layer that we just created. Also, upload the jar from the second project as our function code.

Add Layers to the function

Upload function code jar and update runtime settings

Create a test event and execute the test.

Source Code:

Conclusion

Lambda layer makes your development easy and error-free. It reduces the size of the function deployment package. You can use the same layer in multiple functions.

If you like the post, feel free to share and follow me on Twitter for updates!

How to create dynamic content using AWS Lambda?

Rajan Panchal — Wed, 21 Oct 2020 03:44:19 +0000

Hello Everyone! Today we are going to see how you can generate dynamic content in Lamdba. We are going to use Java 8, Apache Maven, AWS SDK for Java, Apache Velocity to achieve this.

Uses Cases

There are several scenarios where you want to create dynamic content. For example, to create an HTML page response for an API method instead of returning just data in JSON/ XML. The other scenario is to generate a file or message with dynamic content. For instance, create welcome letters for your customers.

Let's Dive In

We are going to create welcome letters for our credit card customers. The letter's template is :


Hello ${name},

Say hi to your new credit card!
Your credit card number is ${creditCardNumber}.

Regards,
Your Credit Company

Lambda would populate name and credit card number. The generated file is uploaded to S3 for further processing like printing and mailing.

STEP 1:

Create an empty maven project and update pom to the following contents:


<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>net.rajanpanchal</groupId>
    <artifactId>lambda-java</artifactId>
    <version>1</version>
    <name>Dynamic-content-Lambda</name>
    <description>Generate Dynamic content in Lambda</description>
    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <aws.java.sdk.version>2.14.11</aws.java.sdk.version>
    </properties>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>bom</artifactId>
                <version>${aws.java.sdk.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
    <dependencies>
        <!-- https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-s3 -->
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-s3</artifactId>
            <version>1.11.865</version>
        </dependency>

        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-lambda-java-core</artifactId>
            <version>1.2.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.velocity</groupId>
            <artifactId>velocity-engine-parent</artifactId>
            <version>2.2</version>
            <type>pom</type>
        </dependency>
        <dependency>
            <groupId>org.apache.velocity</groupId>
            <artifactId>velocity-tools</artifactId>
            <version>2.0</version>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>${maven.compiler.source}</source>
                    <target>${maven.compiler.target}</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.2.4</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

We are adding AWS dependencies and apache velocity dependencies. The maven-shade-plugin to create a fat jar.

STEP 2:

We will have our template, discussed earlier, in the resources folder (src/main/resources) with the name welcomeLetter.vm.

STEP 3:

Create a class file with the following content:

package net.rajanpanchal.handlers;

import java.io.File;
import java.io.FileWriter;
import java.util.Map;
import java.util.Properties;

import org.apache.velocity.Template;
import org.apache.velocity.VelocityContext;
import org.apache.velocity.app.VelocityEngine;
import org.apache.velocity.exception.ParseErrorException;
import org.apache.velocity.exception.ResourceNotFoundException;
import org.apache.velocity.runtime.RuntimeConstants;

import com.amazonaws.regions.Regions;
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.PutObjectRequest;

public class WelcomeLetterGeneratorHandler implements RequestHandler<Map<String, String>, String> {
    Regions clientRegion = Regions.US_EAST_1;
    String fileObjKeyName = "welcome_letter.txt";
    String bucketName = "welcomelettersbucket";
    static VelocityContext vContext;
    static Template t;
    AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withRegion(clientRegion).build();
    static {
        try {
            // Create a new Velocity Engine
            VelocityEngine velocityEngine = new VelocityEngine();
            // Set properties that allow reading vm file from classpath.
            Properties p = new Properties();
            velocityEngine.setProperty(RuntimeConstants.RESOURCE_LOADER, "file,class");
            velocityEngine.setProperty("class.resource.loader.class",
                    "org.apache.velocity.runtime.resource.loader.ClasspathResourceLoader");
            velocityEngine.init(p);
            t = velocityEngine.getTemplate("welcomeLetter.vm");
            vContext = new VelocityContext();

        } catch (Exception e) {
            throw new RuntimeException(e);

        }
    }

    public String handleRequest(Map<String, String> event, Context context) {
        String response = null;
        LambdaLogger logger = context.getLogger();
        try {
            // Add data to velocity context
            vContext.put("name", "Rajan");
            vContext.put("creditCardNumber", "1234 5678 9012 3456");
            File f = new File("/tmp/"+fileObjKeyName);

            FileWriter writer = new FileWriter(f);
            // merge template and Data
            t.merge(vContext, writer);
            writer.flush();
            writer.close();

            // Upload a file as a new object with ContentType and title specified.
            PutObjectRequest request = new PutObjectRequest(bucketName, fileObjKeyName, f);
            ObjectMetadata metadata = new ObjectMetadata();
            metadata.setContentType("plain/text");
            metadata.addUserMetadata("title", "Welcome Letter");
            request.setMetadata(metadata);
            s3Client.putObject(request);
            response  = new String("Success");
        } catch (Exception ex) {
            response =  new String("Error:"+ex.getMessage());
        }

        return response;
    }
}

In the static block we are creating a velocity engine, setting resource loaders for the engine ('class' and 'file' resource loaders). Then we get the template and create a context to hold data to populate on the template.

In Lambda handler handleRequest, we add data like name and credit card number to the context, and using a FileWriter object we merge the template and the data. Note that we create the file in /tmp/ directory in the Lambda execution environment. Lambda provides some disk space to the function to use. Then we upload this file to the S3 bucket (welcomelettersbucket) with the filename welcome_letter.txt. My bucket exists in US-EAST-1 as indicated by the variable clientRegion.

STEP 4

Now execute maven package command to generate the fat jar with all dependencies.
A jar would be generated in the target folder.

STEP 5:

Go to AWS Console, create a Lambda function with runtime Java 8, let it create a default execution role. Upload the jar under Function Code section. Edit Basic Settings and add our handler package and method name

net.rajanpanchal.handlers.WelcomeLetterGeneratorHandler::handleRequest

STEP 6:

Add S3 access policy to the Lambda execution role. Click on Permissions tab - > Execution Role hyperlink. It will take you to the IAM Role. For simplicity, I added AmazonS3FullAccess permission, but you can add fine-grain S3 access.

STEP 7:

Go back to lambda and create Test event. Lambda executed successfully.

Check S3 bucket, the file is saved.

The file is ready for further processing!

Source Code:

Conclusion

It's super easy to create dynamic content in Lambda using Apache Velocity and Java SDK. In addition, using lambda you only pay for what you use. Let me know your thoughts on the article in the comments and how you are going to use this?

If you like the post, feel free to share and follow me on Twitter for updates!

What Is IKEA Effect And How To Benefit From It?

Rajan Panchal — Tue, 06 Oct 2020 20:27:04 +0000

For those who don't know, IKEA is a Swedish multinational group designs and sells ready-to-assemble furniture, kitchen appliances, home accessories, etc.

What is IKEA Effect

The IKEA effect is a cognitive bias in which consumers place a disproportionately high value on products they partially created. A 2011 study found that subjects were willing to pay more for furniture they had assembled themselves, than for equivalent pre-assembled items.

Now you might be wondering why we are discussing it on a tech blog? Read on and you would understand how it impacts you as an IT Professional.

When Labor Leads to Love

Normally, when you buy a product from IKEA it requires some level of assembly before the product can be used. The study found that when someone puts labor into something, they start liking it more, even if it is poorly constructed. Hence, the phenomenon,“IKEA effect”, is named in honor of the Swedish manufacturer whose products typically arrive with some
assembly required. The study also found that the "IKEA Effect" dissipates when the creations are destroyed. The creation has to stay intact for IKEA effect to be effective.

How you can use IKEA Effect to your advantage?

You might have experienced the IKEA effect even if you didn't know the term before. To take advantage of this, you need to build things. No matter how small it is. If you are learning a new language, a library or a technology, think of a small business problem and try to solve it using the knowledge that you have. For example, I am upskilling myself in AWS. So first I learn the concepts and then I think of a problem that can be solved using different AWS services. Once you solve it you will value your solution more than anything else even if it's not of production grade or perfect. This will keep you motivate to learn more and create more. You can also contribute to open source projects. Open source projects are more popular because IKEA effect is in place. Your name gets connected to projects you make contribution to. This gives a sense of accomplishment and motivates you to contribute more.

Also, don't throw away your solution, instead preserve it using your favorite version control like GitHub, GitLab, etc. Or you can also write blog about it.

Let me know your thoughts in comments !

Now, this post will be incomplete without these memes!

AWS Serverless Application Model : Guide to writing your first AWS SAM Application

Rajan Panchal — Tue, 29 Sep 2020 22:30:12 +0000

Today we are going to see how one can write a serverless application using Serverless Application Model (SAM).
First, let's understand what Serverless Application Model is.

What is Serverless Application Model?

What do you do when you want to create a resource on AWS, for example, Lambda function? You log in to the AWS console and manually go to the Lambda service to create a function. Now say you want to create 10 such functions you have to do the same process 10 times. This can be error-prone and time-consuming. With SAM you can just write once and deploy as many times you want flawlessly. It is similar to Cloud Formation. You can say SAM is the superset of Cloudformation.

The AWS Serverless Application Model (SAM) is an open-source framework for building serverless applications. It provides shorthand syntax to express functions, APIs, databases, and event source mappings. With just a few lines of code, you can define the application you want and model it using YAML. During deployment, SAM transforms and expands the SAM syntax into AWS CloudFormation syntax, enabling you to build serverless applications faster.

With the use of Environment variables and parameters, you can dynamically name all your resources and create different environments like DEV, TEST, PROD with the same set of resources.

Getting started with SAM

To use SAM you will need SAM-CLI installed. SAM CLI provides a Lambda-like execution environment that lets you locally build, test, and debug applications defined by SAM templates. You can also use the SAM CLI to deploy your applications to AWS.
You can follow the guide here to install SAM CLI.
Before we proceed make sure your user has all the permissions to create resources and configured on the machine. To configure AWS account, install AWS CLI and run aws configure command to set up credentials. I am using the admin user that has all the permissions.

Now we are ready to download sample application and examine its contents. Run the following command on terminal

sam init

It will ask you to make several choices. Note: you need to have your runtime already installed.

ubuntu@ubuntu-VirtualBox:~/MyWorkspace/BlogPosts/SAM$ sam init
Which template source would you like to use?
    1 - AWS Quick Start Templates
    2 - Custom Template Location
Choice: 1

Which runtime would you like to use?
    1 - nodejs12.x
    2 - python3.8
    3 - ruby2.7
    4 - go1.x
    5 - java11
    6 - dotnetcore3.1
    7 - nodejs10.x
    8 - python3.7
    9 - python3.6
    10 - python2.7
    11 - ruby2.5
    12 - java8.al2
    13 - java8
    14 - dotnetcore2.1
Runtime: 2

Project name [sam-app]: SAM application

Cloning app templates from https://github.com/awslabs/aws-sam-cli-app-templates.git

AWS quick start application templates:
    1 - Hello World Example
    2 - EventBridge Hello World
    3 - EventBridge App from scratch (100+ Event Schemas)
    4 - Step Functions Sample App (Stock Trader)
    5 - Elastic File System Sample App
Template selection: 1

-----------------------
Generating application:
-----------------------
Name: SAM application
Runtime: python3.8
Dependency Manager: pip
Application Template: hello-world
Output Directory: .

Next steps can be found in the README file at ./SAM application/README.md

Go to the file system and it would have created following file structure:

SAM application/
   ├── README.md
   ├── events/
   │   └── event.json
   ├── hello_world/
   │   ├── __init__.py
   │   ├── app.py            #Contains your AWS Lambda handler logic.
   │   └── requirements.txt  #Contains any Python dependencies the application requires, used for sam build
   ├── template.yaml         #Contains the AWS SAM template defining your application's AWS resources.
   └── tests/
       └── unit/
           ├── __init__.py
           └── test_handler.py

Open template.yml and let's breakdown the contents:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  SAM application

  Sample SAM Template for SAM application

# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 3

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
      Runtime: python3.8
      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /hello
            Method: get

Outputs:
  # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
  # Find out more about other implicit resources you can reference within SAM
  # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
  HelloWorldApi:
    Description: "API Gateway endpoint URL for Prod stage for Hello World function"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"
  HelloWorldFunction:
    Description: "Hello World Lambda Function ARN"
    Value: !GetAtt HelloWorldFunction.Arn
  HelloWorldFunctionIamRole:
    Description: "Implicit IAM Role created for Hello World function"
    Value: !GetAtt HelloWorldFunctionRole.Arn

AWSTemplateFormatVersion - The first line always stays the same for every template.yml.

Transform - It specifies one or more macros that AWS CloudFormation uses to process your template. This is a required section.

Description is self-explanatory.

Globals - It is unique to the SAM and defines properties shared by the whole template. In this case, the timeout will apply to all functions. Globals don't exist in cloud formation.

Resources - You define all AWS resources under this. This is also a mandatory section.
Here we have defined a function with the logical name HelloWorldFunction, it is of type AWS::Serverless::Function with a runtime of python 3.8, CodeUri points to the folder where our handler resides on the filesystem. Handler is the filename and the method. Events would create an API in the API Gateway with method type GET and path /hello.

Outputs- Describes the values that are returned and available for use after the stack is created. For example, in our template, HelloWorldApi output gives the API Url generated by the function. We are also setting Lambda and IAM Role ARN as output.
Also note !Sub & GetAtt functions in output, !Sub is used to substitute a variable with a value. Here it will substitute ${AWS::Region} and ${ServerlessRestApi}. GetAtt function returns the value of an attribute from a resource in the template. Here we are asking for ARN of the HelloWorldfunction.

Now examine the app.py file. It is straightforward and returns "hello world" as a response.

import json

# import requests


def lambda_handler(event, context):
    """Sample pure Lambda function

    Parameters
    ----------
    event: dict, required
        API Gateway Lambda Proxy Input Format

        Event doc: https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html#api-gateway-simple-proxy-for-lambda-input-format

    context: object, required
        Lambda Context runtime methods and attributes

        Context doc: https://docs.aws.amazon.com/lambda/latest/dg/python-context-object.html

    Returns
    ------
    API Gateway Lambda Proxy Output Format: dict

        Return doc: https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html
    """

    # try:
    #     ip = requests.get("http://checkip.amazonaws.com/")
    # except requests.RequestException as e:
    #     # Send some context about this error to Lambda Logs
    #     print(e)

    #     raise e

    return {
        "statusCode": 200,
        "body": json.dumps({
            "message": "hello world",
            # "location": ip.text.replace("\n", "")
        }),
    }

Get back to the terminal and cd into the directory where the template.yml resides, run the command: sam build

ubuntu@ubuntu-VirtualBox:~/MyWorkspace/BlogPosts/SAM/SAM application$ sam build
Building function 'HelloWorldFunction'
Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Deploy: sam deploy --guided

The sam build command builds any dependencies that your application has, and copies your application source code to folders under .aws-sam/build to be zipped and uploaded to Lambda. Check the contents of .aws-sam folder in project directory. It would be something similar to shown below. In HelloWorldFunction folder, there would be many other files along with app.py

.aws_sam/
   └── build/
       ├── HelloWorldFunction/
       └── template.yaml

Now its time to run sam deploy -g. It will ask you couple of configuration information like application name, region name where you want to create the stack, etc, and will wait for your confirmation if you asked for confirmation before deploy.

buntu@ubuntu-VirtualBox:~/MyWorkspace/BlogPosts/SAM/SAM application$ sam deploy -g

Configuring SAM deploy
======================

    Looking for samconfig.toml :  Not found

    Setting default arguments for 'sam deploy'
    =========================================
    Stack Name [sam-app]: my-first-sam-app
    AWS Region [us-east-1]: us-east-1
    #Shows you resources changes to be deployed and require a 'Y' to initiate deploy
    Confirm changes before deploy [y/N]: y
    #SAM needs permission to be able to create roles to connect to the resources in your template
    Allow SAM CLI IAM role creation [Y/n]: y
    HelloWorldFunction may not have authorization defined, Is this okay? [y/N]: y
    Save arguments to samconfig.toml [Y/n]: y

    Looking for resources needed for deployment: Found!

        Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-1xyg1t2j2ws5k
        A different default S3 bucket can be set in samconfig.toml

    Saved arguments to config file
    Running 'sam deploy' for future deployments will use the parameters saved above.
    The above parameters can be changed by modifying samconfig.toml
    Learn more about samconfig.toml syntax at 
    https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html
Uploading to my-first-sam-app/9601808ead19b558184dcec8285866a3  538937 / 538937.0  (100.00%)

    Deploying with following values
    ===============================
    Stack name                 : my-first-sam-app
    Region                     : us-east-1
    Confirm changeset          : True
    Deployment s3 bucket       : aws-sam-cli-managed-default-samclisourcebucket-1xyg1t2j2ws5k
    Capabilities               : ["CAPABILITY_IAM"]
    Parameter overrides        : {}

Initiating deployment
=====================
HelloWorldFunction may not have authorization defined.
Uploading to my-first-sam-app/362accae02d25f5921348967d73b9d29.template  1115 / 1115.0  (100.00%)

Waiting for changeset to be created..
CloudFormation stack changeset
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Operation                                                          LogicalResourceId                                                  ResourceType                                                     
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Add                                                              HelloWorldFunctionHelloWorldPermissionProd                         AWS::Lambda::Permission                                          
+ Add                                                              HelloWorldFunctionRole                                             AWS::IAM::Role                                                   
+ Add                                                              HelloWorldFunction                                                 AWS::Lambda::Function                                            
+ Add                                                              ServerlessRestApiDeployment47fc2d5f9d                              AWS::ApiGateway::Deployment                                      
+ Add                                                              ServerlessRestApiProdStage                                         AWS::ApiGateway::Stage                                           
+ Add                                                              ServerlessRestApi                                                  AWS::ApiGateway::RestApi                                         
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Changeset created successfully. arn:aws:cloudformation:us-east-1:<account #>:changeSet/samcli-deploy1599948737/03d65ab9-a943-494d-8db6-abf6aad17537


Previewing CloudFormation changeset before deployment
======================================================
Deploy this changeset? [y/N]:

Once you confirm, it will start creating stack. Below is the output after the stack creation completes:

2020-09-12 17:13:45 - Waiting for stack create/update to complete

CloudFormation events from changeset
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ResourceStatus                                    ResourceType                                      LogicalResourceId                                 ResourceStatusReason                            
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CREATE_IN_PROGRESS                                AWS::IAM::Role                                    HelloWorldFunctionRole                            -                                               
CREATE_IN_PROGRESS                                AWS::IAM::Role                                    HelloWorldFunctionRole                            Resource creation Initiated                     
CREATE_COMPLETE                                   AWS::IAM::Role                                    HelloWorldFunctionRole                            -                                               
CREATE_IN_PROGRESS                                AWS::Lambda::Function                             HelloWorldFunction                                -                                               
CREATE_IN_PROGRESS                                AWS::Lambda::Function                             HelloWorldFunction                                Resource creation Initiated                     
CREATE_COMPLETE                                   AWS::Lambda::Function                             HelloWorldFunction                                -                                               
CREATE_IN_PROGRESS                                AWS::ApiGateway::RestApi                          ServerlessRestApi                                 -                                               
CREATE_IN_PROGRESS                                AWS::ApiGateway::RestApi                          ServerlessRestApi                                 Resource creation Initiated                     
CREATE_COMPLETE                                   AWS::ApiGateway::RestApi                          ServerlessRestApi                                 -                                               
CREATE_IN_PROGRESS                                AWS::Lambda::Permission                           HelloWorldFunctionHelloWorldPermissionProd        Resource creation Initiated                     
CREATE_IN_PROGRESS                                AWS::ApiGateway::Deployment                       ServerlessRestApiDeployment47fc2d5f9d             -                                               
CREATE_IN_PROGRESS                                AWS::Lambda::Permission                           HelloWorldFunctionHelloWorldPermissionProd        -                                               
CREATE_COMPLETE                                   AWS::ApiGateway::Deployment                       ServerlessRestApiDeployment47fc2d5f9d             -                                               
CREATE_IN_PROGRESS                                AWS::ApiGateway::Deployment                       ServerlessRestApiDeployment47fc2d5f9d             Resource creation Initiated                     
CREATE_IN_PROGRESS                                AWS::ApiGateway::Stage                            ServerlessRestApiProdStage                        -                                               
CREATE_IN_PROGRESS                                AWS::ApiGateway::Stage                            ServerlessRestApiProdStage                        Resource creation Initiated                     
CREATE_COMPLETE                                   AWS::ApiGateway::Stage                            ServerlessRestApiProdStage                        -                                               
CREATE_COMPLETE                                   AWS::Lambda::Permission                           HelloWorldFunctionHelloWorldPermissionProd        -                                               
CREATE_COMPLETE                                   AWS::CloudFormation::Stack                        my-first-sam-app                                  -                                               
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

CloudFormation outputs from deployed stack
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Outputs                                                                                                                                                                                                
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Key                 HelloWorldFunctionIamRole                                                                                                                                                          
Description         Implicit IAM Role created for Hello World function                                                                                                                                 
Value               arn:aws:iam::<account #>:role/my-first-sam-app-HelloWorldFunctionRole-27OIM6WD99F0                                                                                                

Key                 HelloWorldApi                                                                                                                                                                      
Description         API Gateway endpoint URL for Prod stage for Hello World function                                                                                                                   
Value               https://3kuuurt63m.execute-api.us-east-1.amazonaws.com/Prod/hello/                                                                                                                 

Key                 HelloWorldFunction                                                                                                                                                                 
Description         Hello World Lambda Function ARN                                                                                                                                                    
Value               arn:aws:lambda:us-east-1:<account #>:function:my-first-sam-app-HelloWorldFunction-1U5YD9NICU5LP                                                                                   
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Successfully created/updated stack - my-first-sam-app in us-east-1

As you can see in the outputs, the HelloWorldApi URL was generated which you can hit in the browser and verify your API is working fine. You can also see the same information in Cloud Formation's output section on AWS console.

Great Job! You have successfully completed your First SAM App!

What else you can do in SAM?

This is not the end, you can also test the function locally before deploy. The AWS SAM CLI provides the sam local command to run your application using Docker containers that simulate the execution environment of Lambda.
I am on linux machine, hence I am going to run command : sudo /home/linuxbrew/.linuxbrew/bin/sam local start-api

SAM CLI now collects telemetry to better understand customer needs.

    You can OPT OUT and disable telemetry collection by setting the
    environment variable SAM_CLI_TELEMETRY=0 in your shell.
    Thanks for your help!

    Learn More: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-telemetry.html

Mounting HelloWorldFunction at http://127.0.0.1:3000/hello [GET]
You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2020-09-12 22:31:06  * Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)

When you hit the URL(http://127.0.0.1:3000/hello) in the browser, you should see the same response as before "hello world" and on the terminal, it should show something similar to this

Invoking app.lambda_handler (python3.8)
Failed to download a new amazon/aws-sam-cli-emulation-image-python3.8:rapid-1.1.0 image. Invoking with the already downloaded image.
Mounting /home/ubuntu/MyWorkspace/BlogPosts/SAM/SAM application/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated inside runtime container
START RequestId: f4945824-0a1a-1742-2f46-4d0bc2bbe515 Version: $LATEST
END RequestId: f4945824-0a1a-1742-2f46-4d0bc2bbe515
REPORT RequestId: f4945824-0a1a-1742-2f46-4d0bc2bbe515  Init Duration: 216.83 ms    Duration: 2.88 ms   Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 24 MB  
No Content-Type given. Defaulting to 'application/json'.
2020-09-12 22:38:28 127.0.0.1 - - [12/Sep/2020 22:38:28] "GET /hello HTTP/1.1" 200 -

Where to go from here?

This was a very basic SAM application but I hope you got the gist of it. As an exercise, you can add more resources like CloudFront, Route53, DynamoDB, ACM etc, and see how to model it in YAML. Take a look here and here for more complex SAM Templates.

If you like my articles feel free to follow me on Twitter for updates!

REST API using AWS Java SDK

Rajan Panchal — Tue, 22 Sep 2020 18:29:48 +0000

In the previous post, we looked at a simple Lambda handler using the AWS Java SDK. In this post, we're going to implement Rest API with Lambda (using Lambda Proxy Integration).

What is Lambda Proxy Integration

When a client submits an API request, API Gateway passes the request to the integrated Lambda function as-is, except that the order of the request parameters is not preserved. This request data includes the request headers, query string parameters, URL path variables, payload, and API configuration data.

The configuration data can include current deployment stage name, stage variables, user identity, or authorization context (if any). The backend Lambda function parses the incoming request data. The API gateway simply acts as a postman to pass requests and responses. One advantage is that you can change the implementation of the Lambda without impacting the API.

Let's implement using AWS Java SDK

Create an empty maven project and update the pom.xml with the following contents.

<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>net.rajanpanchal</groupId>
    <artifactId>lambda-api-demo</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>LambdaApiDemo</name>
    <description>Demonstrates API Request &amp; Response Using Lambda and Java SDK for AWS</description>
    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <aws.java.sdk.version>2.14.11</aws.java.sdk.version>
    </properties>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>bom</artifactId>
                <version>${aws.java.sdk.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
    <dependencies>
        <!-- https://mvnrepository.com/artifact/com.amazonaws/aws-lambda-java-events -->
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-lambda-java-events</artifactId>
            <version>3.2.0</version>
        </dependency>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-lambda-java-core</artifactId>
            <version>1.2.0</version>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>${maven.compiler.source}</source>
                    <target>${maven.compiler.target}</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.2.4</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

We are adding 2 main dependencies here:

aws-lambda-java-core - This provides RequestHandler
aws-lambda-java-events - This provides APIGatewayProxyRequestEvent and APIGatewayProxyResponseEvent

Added 2 plugins:

Compiler plugin - determines java version for the compiler.
maven-shade-plugin - to generate fat jar aka Uber jar (with all dependencies)

Create a class LambdaHandler, with the following contents.

package net.rajanpanchal.handlers;

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.LambdaLogger;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.lambda.runtime.events.APIGatewayProxyRequestEvent;
import com.amazonaws.services.lambda.runtime.events.APIGatewayProxyResponseEvent;

public class LambdaHandler implements RequestHandler<APIGatewayProxyRequestEvent, APIGatewayProxyResponseEvent> {
    @Override
    public APIGatewayProxyResponseEvent handleRequest(APIGatewayProxyRequestEvent requestEvent, Context context) {
        APIGatewayProxyResponseEvent responseEvent = new APIGatewayProxyResponseEvent();
        try {
            LambdaLogger logger = context.getLogger();
            logger.log("requestEventString:" + requestEvent.toString());
            logger.log("query string param:" + requestEvent.getQueryStringParameters());
            logger.log("headers:" + requestEvent.getHeaders());
            // fetching the value send in the request body
            String message = requestEvent.getBody();
            // Request request = gson.fromJson(message, Request.class);
            logger.log("message body:" + message);

            // setting up the response message
            responseEvent.setBody("Hello from Lambda!");
            responseEvent.setStatusCode(200);

            return responseEvent;

        } catch (Exception ex) {
            responseEvent.setBody("Invalid Response");
            responseEvent.setStatusCode(500);
            return responseEvent;
        }
    }
}

The above code is simple. requestEvent object is of type APIGatewayProxyRequestEvent. You will get headers, body, query-string, etc. from this object.
responseEvent object is of type APIGatewayProxyResponseEvent. You can set response headers, body, status, etc in this object.

Now execute maven command mvn package to generate a jar file in the target folder.
Login to AWS console, create Lambda function with Java 8 runtime, let it create execution role, and upload the jar file under Function Code section.
In Basic Settings, set the handler name in format :

*net.rajanpanchal.handlers.LambdaHandler::handleRequest*

Now go to API Gateway, create a REST API

Go to actions and create a new method of type GET. Select the following configuration:

To enable proxy integration, make sure to check the checkbox. Save to create a method.

Go to actions and click deploy API and give the following information and click deploy.

Use the URL generated to access the API

Testing

Hit the URL in the browser and you should see the below response:

Go to Cloud Watch, access Log Groups from the left-hand navigation, and access the logs corresponding to your lambda function. You should see the log statements from the lambda handler.

Source Code:

Here we complete our REST API with Lambda!

If you like the post, feel free to share and follow me on Twitter for updates!

Amazon S3 Quick Reference

Rajan Panchal — Wed, 16 Sep 2020 04:37:04 +0000

Amazon S3 Quick Reference for Certification, interview and study!

If you like the post, feel free to share and follow me on Twitter for updates!

AWS Lambda Performance Optimizations

Rajan Panchal — Thu, 10 Sep 2020 22:24:47 +0000

In this post we are going to see how you can optimize performance of AWS Lambda function. Although Lambda is a managed service by AWS and it automatically scales, based on shared responsibility model, the underlying service details will be managed by AWS but its client's responsibility to correctly use the service. By using these optimizations you can lower your Lambda bill.

Lets look at some of the Lambda Performance Optimizations

Execution context Reuse

When a Lambda function is invoked, AWS Lambda launches an execution context based on the configuration settings you provide like Memory, runtime, etc. The execution context is a temporary runtime environment that initializes any external dependencies of your Lambda function code, such as database connections or HTTP endpoints, etc. After the Lambda function completes the execution it maintains the execution context in hopes that another invocation of same function will happen. If it does happen, then it will reuse the context.
So the objects declared outside of the lambda handler (Lambda global variables) can be reused. You can declare any objects there. Following code snippets defines S3 as client that will be reused in subsequent invocation.


import json
import boto3
#DynamoDB defined outside of handler
dynamodb = boto3.resource('dynamodb')
def lambda_handler(event, context):
    table = dynamodb.Table("MyTable")
    response = table.get_item(Key={'userid': username})

    return {
        'statusCode': 200,
        'body': json.dumps(response['Item) 
      }

A word of caution: Its is not guaranteed that subsequent function invocation would go to the same execution context. Lambda may decide to create a new invocation context runtime so don't maintain any state information. Always have logic in Lambda handler to instantiate or reconnect the connections objects if cannot be reused.

Minimize Deployment Package

A deployment package is a ZIP archive that contains your compiled function code and dependencies. In case your Lambda handler is in Java, you need to have a fat-jar aka Uber Jar to upload to Lambda. Uber jar will contain all the dependencies required to execute the handler. Its is important that you only import those packages that are required. For example, if you only need basic Lambda implementation, then import only lambda-java-core dependency - It defines handler method interfaces and the context object that the runtime passes to the handler. If you define your own input types, this is the only library you need. You don't need whole aws-sdk dependency to write simple Lambda handlers. Use Bill of Materials POM (https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-bom) in the maven project and provides individual dependencies from BOM POM to import. This will reduce the fat jar size. Keep in mind that in lambda, you are working off a limited disk size and memory as defined in the configuration.
In Python and other languages you can only import modules that you need. For example, in TypeScript

// import entire SDK
import AWS from 'aws-sdk';
// import AWS object without services
import AWS from 'aws-sdk/global';
// import individual service
import S3 from 'aws-sdk/clients/s3';

Use Simple Frameworks

Instead of using full blown spring framework try to use lightweight frameworks that accomplish the same tasks. For example: If you need Java dependency injection in your app then use lightweight frameworks like Guice (pronounced 'juice' is a lightweight dependency injection framework for Java 6 and above, brought to you by Google)

Are there any other Lambda optimization techniques? Let us know in comments!

If you like my articles feel free to follow me on Twitter for updates!

Write your first AWS Lambda in Java

Rajan Panchal — Thu, 10 Sep 2020 22:23:06 +0000

So far I have been writing Lambda function in Python for all my AWS projects . In Python its easy, just import boto3 module and starting coding. Things are bit different when you write Lambda handlers in Java. Lets explore and see how you can write a simple HelloWorld Lambda handler in java. I have planned couple of how-to projects that I am going to write in Java. Hence I thought to do HelloWorld post before we dive into complex handlers. Feel free to follow me on twitter for upcoming updates.

To implement handler, we are going to use Maven, Eclipse and AWS SDK for Java. Below are the version:

Java: 1.8
Eclipse: Version: 2020-06 (4.16.0)
AWS SDK: 2.14.11
Maven: 3.6.3 (Included with eclipse)

To get started, create a new empty maven project in eclipse. Go to File-> New-> Maven Project. In new project dialog box. select "create simple project" and hit next. On next screen. add appropriate information about the project and hit finish. For example:

Now your project is created. Open pom.xml and add following code.

<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>net.rajanpanchal</groupId>
    <artifactId>lambda-java-demo</artifactId>
    <version>1</version>
    <name>Lambda-Java-Demo</name>
    <description>HelloWorld Lambda handler in java</description>
    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <aws.java.sdk.version>2.14.11</aws.java.sdk.version>
    </properties>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>bom</artifactId>
                <version>${aws.java.sdk.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-lambda-java-core</artifactId>
            <version>1.2.0</version>
        </dependency>

        <dependency>
            <groupId>com.google.code.gson</groupId>
            <artifactId>gson</artifactId>
            <version>2.8.6</version>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>${maven.compiler.source}</source>
                    <target>${maven.compiler.target}</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.2.4</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

We are setting some properties in <properties> tag like AWS SDK version and Java Version. We are adding AWS SDK dependency and Lambda core dependency with desired version. Google JSON dependency is added to convert between JSON and Java object and vice versa.
In plugins, we define maven compiler plugin to compile the code and another important plugin called maven-shade-plugin. This plugin helps to create fat jar a.k.a Uber jar. This jar will contain all the dependencies that are required to successfully run the lambda function.

Now lets create the handler.

package net.rajanpanchal.handlers;

import java.util.Map;

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.LambdaLogger;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;

public class HelloWorldHandler implements RequestHandler<Map<String,String>, String>{
      Gson gson = new GsonBuilder().setPrettyPrinting().create();
      @Override
      public String handleRequest(Map<String,String> event, Context context)
      {
        LambdaLogger logger = context.getLogger();
        String response = new String("200 OK");
        // log execution details
        logger.log("ENVIRONMENT VARIABLES: " + gson.toJson(System.getenv()));
        logger.log("CONTEXT: " + gson.toJson(context));
        // process event
        logger.log("EVENT: " + gson.toJson(event));
        logger.log("EVENT TYPE: " + event.getClass().toString());
        return response;
      }
    }

Here were are implementing RequestHandler that will accept a Map of String and outputs a String response. The handler is not doing much. Just logging some stuff from context and event and in response it sends 200 OK string.
After this right click on project, go to 'Run as' and click 'Maven build'. A configuration window will open. In 'Goals' type package and hit 'Run' . Build should be successful and will create a fat jar in target folder.

Now go to AWS console and create a new Lambda function with Java 8 as runtime and upload this jar from Function Code section. In Basic Settings, you have to specify package.class::methodName in the handler textbox. To test this Lambda, create a test event and execute the test.

The ouput might look similar to this:

START RequestId: c40b4095-27f5-4153-ac37-fd2c103f4476 Version: $LATEST
ENVIRONMENT VARIABLES: {
  "PATH": "/usr/local/bin:/usr/bin/:/bin:/opt/bin",
  "_AWS_XRAY_DAEMON_ADDRESS": "169.254.79.2",
  "LAMBDA_TASK_ROOT": "/var/task",
  "AWS_LAMBDA_FUNCTION_MEMORY_SIZE": "128",
  "TZ": ":UTC",
  "AWS_SECRET_ACCESS_KEY": "<keyid>",
  "AWS_EXECUTION_ENV": "AWS_Lambda_java8",
  "AWS_DEFAULT_REGION": "us-east-1",
  "AWS_LAMBDA_LOG_GROUP_NAME": "/aws/lambda/helloworldjava",
  "_HANDLER": "net.rajanpanchal.handlers.HelloWorldHandler::handleRequest",
  "LANG": "en_US.UTF-8",
  "LAMBDA_RUNTIME_DIR": "/var/runtime",
  "AWS_SESSION_TOKEN": "<session token>",
  "LD_LIBRARY_PATH": "/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib",
  "_X_AMZN_TRACE_ID": "Root\u003d1-5f545c5d-22c8a803badd636859a0f387;Parent\u003d23df427f78bc46e4;Sampled\u003d0",
  "AWS_SECRET_KEY": "<secret key>",
  "AWS_REGION": "us-east-1",
  "AWS_LAMBDA_LOG_STREAM_NAME": "2020/09/06/[$LATEST]57c598b33b164acf8e8151660249e50e",
  "AWS_XRAY_DAEMON_ADDRESS": "169.254.79.2:2000",
  "_AWS_XRAY_DAEMON_PORT": "2000",
  "AWS_XRAY_CONTEXT_MISSING": "LOG_ERROR",
  "AWS_LAMBDA_FUNCTION_VERSION": "$LATEST",
  "AWS_ACCESS_KEY": "<access key>",
  "AWS_LAMBDA_FUNCTION_NAME": "helloworldjava"
}CONTEXT: {
  "memoryLimit": 128,
  "awsRequestId": "c40b4095-27f5-4153-ac37-fd2c103f4476",
  "logGroupName": "/aws/lambda/helloworldjava",
  "logStreamName": "2020/09/06/[$LATEST]57c598b33b164acf8e8151660249e50e",
  "functionName": "helloworldjava",
  "functionVersion": "$LATEST",
  "invokedFunctionArn": "arn:aws:lambda:us-east-1:<AccountId>:function:helloworldjava",
  "cognitoIdentity": {
    "identityId": "",
    "poolId": ""
  },
  "logger": {}
}EVENT: {
  "key1": "value1",
  "key2": "value2",
  "key3": "value3"
}EVENT TYPE: class java.util.LinkedHashMapEND RequestId: c40b4095-27f5-4153-ac37-fd2c103f4476
REPORT RequestId: c40b4095-27f5-4153-ac37-fd2c103f4476  Duration: 516.04 ms Billed Duration: 600 ms Memory Size: 128 MB Max Memory Used: 79 MB  Init Duration: 393.54 ms

Here we complete our first Lambda handler in Java. In next post we will see how we can implement API using AWS SDK.

Automate Application deployment using GitHub Actions

Rajan Panchal — Wed, 02 Sep 2020 23:26:03 +0000

GitHub Actions makes it easy to automate all your software workflows. You can Build, test, and deploy your code right from GitHub. In this post we will explore how you can use GitHub Actions to automate serverless application deployment on AWS. You can also use AWS's own CI/CD services to achieve the same. But here we are going to keep our discussion limited to GitHub Actions.

How to use GitHub Actions?

Creating a GitHub action is simple. Go to your GitHub repository that you want to automate and click on "Actions"

You will be taken to Actions page where you can create a new Blank workflow or select existing actions from the marketplace. The actions from marketplace are reusable actions that you can use in your workflow. We are going to create a blank action and we will also use some actions from marketplace.

Lets rename the YAML file to workflow.yml. You can name anything you like. We are going to create a Lambda function with API gateway in Serverless Application Model (SAM) template and deploy it using GitHub Actions.
Below is our SAM template.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  GitHub Actions demonstration App
Resources:
  ApiGatewayApi:
    Type: AWS::Serverless::Api
    Properties:
      StageName: Prod
    Auth:
     UsagePlan:
      CreateUsagePlan: PER_API
      Description: Usage plan for this API
      Quota:
       Limit: 500
       Period: MONTH
      Throttle:
       BurstLimit: 100
       RateLimit: 50
  LamdbaFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: ./    
      Handler: lambda.handler
      Runtime: python3.8
      Events:
        getCounter:
          Type: Api
          Properties:
            Path: /hello
            Method: get
            RestApiId: !Ref ApiGatewayApi

lambda.py

def handler(event, context):
            return {
        'statusCode': 200,
        'headers': {
            'Content-Type': 'application/json',
            'Access-Control-Allow-Origin': '*'
        },
        'body':'Hello from Lambda!'
        ,
        "isBase64Encoded": False
    }

This contains one Lambda function and API with path hello. Lets first deploy manually using SAM CLI and then we will automate it. Create samconfig.toml with below details.
create s3_bucket that will be used for SAM deploy and update in samconfig.toml.

version = 0.1
[default]
[default.deploy]
[default.deploy.parameters]
stack_name = "sam-github-actions-app"
s3_bucket = "aws-sam-cli-managed-default-samclisourcebucket-1xyg1t2j2ws5k"
s3_prefix = "sam-app"
region = "us-east-1"
confirm_changeset = false
capabilities = "CAPABILITY_IAM"

Also create empty requirements.txt along with template.yml. Run SAM build and SAM deploy -g on CLI.

Go to API gateway and hit the url in browser. You should get "hello from Lambda!" response.

Go back to our workflow file on GitHub. We will deploy as soon as we push updates to the repo.
Below is our workflow.yml

# This is a basic workflow to help you get started with Actions
name: AWS Lambda & API gateway deployment demonstration
# Controls when the action will run. Triggers the workflow on push or pull request
# events but only for the master branch
on:
  push:
    branches: [ master ]
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "build"
  build:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest
    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
    # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
    - uses: actions/checkout@v2
    # Installs Python
    - name: Set up Python 3.8
      uses: actions/setup-python@v2
      with:
        python-version: 3.8
    # Installs PIP
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
    # Configures AWS credentials from github secrets
    - name: Configure AWS Credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: us-east-1
    # Build using SAM 
    - name: SAM Build
      uses: youyo/aws-sam-action/python3.8@master
      with:
        sam_command: build
      env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          AWS_DEFAULT_REGION:  us-east-1
    # Deploy on AWS
    - name: sam deploy
      uses: youyo/aws-sam-action/python3.8@master
      with:
          sam_command: 'deploy --stack-name myApp --no-fail-on-empty-changeset'
      env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          AWS_DEFAULT_REGION: us-east-1

We first indicate that we want to run this action on push to the master. Then we select runner (Ubuntu) on which our steps will execute. In Steps, we first checkout code, then install python and its dependencies. Then we use other actions from marketplace to configure AWS credentials and then we use another action to do SAM build and SAM deploy. Please note that we need ti supply AWS ACCESS KEY and SECRET ACCESS KEY to actions for commands to work. We setup here GitHub Secrets.

As soon as you check-in workflow.yml, the action would trigger.

Deploying stack

Now you can login to the AWS console and confirm the stack is created.

Go to Resources tab and access the API Gateway. Go to stages and access the Prod stage API.
Open the URL in browser with the path /hello and you should see below output!

Congratulations! You have successfully automated AWS deployment using GitHub Actions!
You can download the code from here:

AWS Certified Developer - Associate Exam Tips (2020)

Rajan Panchal — Wed, 02 Sep 2020 23:23:48 +0000

In June 2020, I passed the AWS Certified Solution Architect Exam after almost preparing for 5 months ( an hour or so everyday) and I was so pumped that I decided to take another AWS exam. I started preparing for the Developer Associate exam at the end of June and cleared by July end.

Here is my badage: AWS Badge

Developer associate is Developer focused and its bit different than the solution architect exam. I found Developer associate exam harder than the Solution Architect, but YMMV.

So here are my tips and preparation topics:

You success in exam depends upon recognizing keywords in the questions. All answers may look similar but keywords will lead to right answer. Pay attention to words. For example: 'low cost', 'encryption at rest vs encryption in transit'.
Know how to do Read Capacity Units (RCU) and Write Capacity Units (WCU) calculations. You might get one or two questions on determining RCU/WCU. Don't forget to pay attention to strongly consistent/ Eventual consistent keyword in question.
Sometimes you may find answer to questions in other questions. While reviewing see if any other question has answer to it.
Use all the time available for the exam. If finished early, spend time in reviewing all questions.
Attempt as many mock exams you can. This will prepare you on time management and questions patterns. Questions will be verbose in real exam.
There could be command level questions like what this command do, when to use etc.
You need to understand the concepts very well and should have some hands on. It is easy to eliminate two answers.
Know CI/CD services like CodeDeploy, CodeBuild, CodeCommit. Understand BuildSpec and AppSpec difference. Code Deploy lifecycle hooks. CodeDeploy Deployment Types for Lambda, ECS and EC2.
Cloudformation, KMS, SQS, SNS, ECS, API gateway, IAM (Inline policies/ Managed Policies) and Lambda (Concurrent execution Limit,
DynamoDB. Local Secondary Index vs Global. DynamoDB Accelerator. DynamoDB Streams. SCAN vs Query.
Amazon S3, EC2, Elastic Beanstalk and its deployment types. Knows CORS.
CloudWatch, CloudTrail, App X-Ray
You might also get architecture level questions but not much.

Good Luck!

How to serve premium/private content on your site?

Rajan Panchal — Sun, 23 Aug 2020 03:46:08 +0000

A lot of online companies (for example: Netflix, Udemy etc ) distribute contents over internet that is accessible only to the members of the site or who have subscribed or paid a premium. Today in this post we see how you can achieve this for your website using Amazon Web Services. We will see how you can securely serve private content to your users from AWS S3 bucket using S3 Presigned URLs

What are Presigned URLs?

A Presigned URL is a URL that provides limited permission and time to make a request. Anyone who receives the presigned URL can then access the object. For example, if you have a file in your bucket and both the bucket and the object are private, you can share the file with others by generating a presigned URL.

Some important points on Presigned URLs

The creator of Presigned URL should have access to object for URL to work, otherwise URL wont work.
You can create a presigned URL that's are not usable or doesn’t work.
It doesn’t cost anything to create Presigned URLs.
You can set expiration time on the URLs
If you created a presigned URL using a temporary token, then the URL expires when the token expires, even if the URL was created with a later expiration time
You can revoke the URL by removing the permissions to access the object from the user created the URL.

How to create Presigned URL?

When you create a presigned URL for your object, you must provide your security credentials, specify a bucket name, an object key, specify method and expiration time.

In Python, using Boto3, you can use generate_presigned_url method to generate the URL

response = s3_client.generate_presigned_url('get_object',
            Params={'Bucket': bucket_name,
            'Key': object_name},
            ExpiresIn=expiration)

The response object will contain the URL which will look similar to this

https://somebucketname-rep.s3.amazonaws.com/someFileName.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQS3IOUWUZF7YSXGA%2F20200821%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20200821T051228Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Security-Token=FwoGZXIvYXdzEH8aDCfJDxO0y6xQxYmdGCK2AXe71W%2FgZEg%2FSnSWC%2Fw%2FaJHeZ20M7OI7AqMEum5c98Chl6pSNPwE5Awsc3ySwokDF6L8a9wP0ceXWAmxT3WXLSoFeNHDbbEHfUKWnvGL8yFzAxdmf%2Fmi%2B5Tnl62td8Nad%2F0Ct1Sx11Mip1h2qdYxw80OX5bCTq7cAHHjpmupvaDt%2BZ3qVyIA9WZmeS63dCPOlieE9IiBZf%2FjxF4Mcs5w4ZIHtZL%2F3LvqMXAy3XfzCgnlYVZeCNczKLuv%2FfkFMi0mStwkzyO%2BfMIxWJ82GJmyNi7LZuY5r0Hx0mE%2BxLnre8jp9%2FACoV%2FM92GnsR0%3D&X-Amz-Signature=17046b630ad4dede85af1cd57204bba8adc462a1825a35d93e81b656c683ad75

You can use this URL in the browser and access the object! Simple.. isn't it?

Lets see it in Action

We are going to implement this on top of previous two posts, in the first post, we implemented custom identity broker(signup/login), using AWS KMS to encrypt decryt password. In second post, we used AWS STS to AssumeRole to read bucket files names and now we will extend that to use presigned url. Download code for second post from github.com/rajanpanchal/aws-kms-sts and modify it.
Here is how overall process looks like

Open file showFiles.py from the Lamdba folder and lets add function to generate presigned URL. Call this function from getFilesList function.

def getSignedUrl(key,s3_client):
    KeyUrl = {}

    response = s3_client.generate_presigned_url('get_object',
                        Params={'Bucket': os.environ['filesBucket'],
                        'Key': key},
                        ExpiresIn=3600)
    KeyUrl[key] = response
    return KeyUrl


# Returns list of files from bucket using STS    
def getFilesList():
    sts_client = boto3.client('sts')

    # Call the assume_role method of the STSConnection object and pass the role
    # ARN and a role session name.
    assumed_role_object=sts_client.assume_role(
        RoleArn=os.environ['s3role'],
        RoleSessionName="AssumeRoleSession1"
    )

    # From the response that contains the assumed role, get the temporary 
    # credentials that can be used to make subsequent API calls
    credentials=assumed_role_object['Credentials']

    # Use the temporary credentials that AssumeRole returns to make a 
    # connection to Amazon S3  
    s3_resource=boto3.resource(
        's3',
        aws_access_key_id=credentials['AccessKeyId'],
        aws_secret_access_key=credentials['SecretAccessKey'],
        aws_session_token=credentials['SessionToken'],
    )
    s3_client = boto3.client('s3',aws_access_key_id=credentials['AccessKeyId'],
aws_secret_access_key=credentials['SecretAccessKey'],
aws_session_token=credentials['SessionToken'])
    bucket = s3_resource.Bucket(os.environ['filesBucket'])
    files=[]
    for obj in bucket.objects.all():
        files.append(getSignedUrl(obj.key,s3_client))
    return files

We are using the temporary credentials obtained from AssumeRole to generate Presigned URL in getSignedUrl function. The getFilesList functions returns list of file names and presigned URL.

Now, modify showFiles.html, to iterate over response and create links for files

<html>
<head>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/semantic-ui/2.4.1/semantic.min.css" integrity="sha512-8bHTC73gkZ7rZ7vpqUQThUDhqcNFyYi2xgDgPDHc+GXVGHXq+xPjynxIopALmOPqzo9JZj0k6OqqewdGO3EsrQ==" crossorigin="anonymous" />
<script
  src="https://code.jquery.com/jquery-3.1.1.min.js"
  integrity="sha256-hVVnYaiADRTO2PzUGmuLJr8BLUSjGIZsDYGmIJLv2b8="
  crossorigin="anonymous"></script>

<script src="https://cdnjs.cloudflare.com/ajax/libs/semantic-ui/2.4.1/semantic.min.js"></script>
</head>
<body>

<div class="ui raised very text container">
<h1 class="ui header">File Access System</h1>
<i class="folder open icon"></i></i><div class="ui label">Files</div>
<div id="files" >Loading..</div>
</div>
</body>
<script>

fetch(" https://g4m3zpzp95.execute-api.us-east-2.amazonaws.com/Prod/showFiles/", {
  credentials: 'include'
})
  .then(response => response.text())
  .then((body) => {
    var files="";

    var obj = JSON.parse(body)
    for (i = 0; i < obj.length; i++) {
            var o = obj[i]

            for(x in o){
                files =  files+ "<i class='file alternate outline icon'><a href='"+o[x]+"' target='_blank'>&nbsp;&nbsp;"+x+"</a>"
            }
    }
    document.getElementById("files").innerHTML= files
  })
  .catch(function(error) {
    console.log(error); 
  });

</script>
</html>

Now do SAM build and Deploy.

Modify login.html, signup.html and showFiles.html to update the api urls from cloudformation outputs.
Upload these files to the bucket stsexamplebucket or the bucket you created. Keep these files Public

You can find the code here:

https://github.com/rajanpanchal/aws-kms-sts-presigned-url

Testing

Let me know if you have any questions or comments!

How to give access to AWS resources without creating 100s of IAM Users?

Rajan Panchal — Thu, 13 Aug 2020 03:35:15 +0000

This post was originally published at https://www.rajanpanchal.net/aws-sts-example/

Scenario:

Imagine you are a solution architect in a company with 100s of Sales employees and you are migrating from on premise to AWS Cloud. You want to use existing employee authentication system and you want to store files on S3 that employee uses in their day to day work. You don't want to keep S3 bucket public, that will expose all files to everybody.
You have 2 options:

Create single role with S3 bucket access and login credentials for all of the employees. With this user have to use 2 different logins. One to access their existing system and other to access S3 files.
Use AWS Security Token Service (STS) to assume role with S3 access and use that to give access to the files. User will still authenticate with their existing system.

In this post, we will explore and implement option # 2. Please note that we are building this example on top of previous post.

About Security Token Service (STS)

AWS Security Token Service (AWS STS) is a web service that enables you to request temporary, limited-privilege credentials for AWS Identity and Access Management (IAM) users or for users that you authenticate. You can use AssumeRole action on STS that returns a set of temporary security credentials that you can use to access AWS resources that you might not normally have access to. These temporary credentials consist of an access key ID, a secret access key, and a security token. Typically, you use AssumeRole within your account or for cross-account access. In our case, we are using AssumeRole within same account.

How to setup users and roles?

In our case, we are going to create a role called S3ReadOnlyAccessAssumeRole. As name suggests it has only S3 read access policy. We will also create a Trust Policy and attached to this S3 role. Trust policy will allow this role to be assumed by our lambda execution role. Here is how our SAM will look like for this role.

IamS3Role:
    Type: AWS::IAM::Role
    Properties: 
      AssumeRolePolicyDocument:
       Version: 2012-10-17
       Statement:
          - Effect: Allow
            Principal:
              AWS: !GetAtt ShowFilesFunctionRole.Arn
            Action:
              - 'sts:AssumeRole'
      Description: 'Readonly S3 role for lamdba to Assume at runtime'
      ManagedPolicyArns: 
      - arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess
      RoleName: S3ReadOnlyAccessAssumeRole

In above snippet, AssumeRolePolicyDocument attribute specifies trust policy that allow Lamdba execution role identified by principle AWS: !GetAtt ShowFilesFunctionRole.Arn to AssumeRole to which this policy is attached to i.e S3ReadOnlyAccessAssumeRole . The ManagedPolicyArns attribute species the policy for S3ReadOnlyAccessAssumeRole that allows read only access to S3 buckets.

Lambda Handler

Now, lets write our Lamdba handler that will use this role. Here is the SAM confgiuration

Origin:
    Type: String
    Default: https://stsexamplebucket.s3.us-east-2.amazonaws.com
  FilesBucket:
    Type: String
    Default: s3-sales-rep

ApiGatewayShowFilesApi:
    Type: AWS::Serverless::Api
    Properties:
      StageName: Prod
    Auth:
     UsagePlan:
      CreateUsagePlan: PER_API
      Description: Usage plan for this API
      Quota:
       Limit: 500
       Period: MONTH
      Throttle:
       BurstLimit: 100
       RateLimit: 50
  ShowFilesFunction:
    Type: AWS::Serverless::Function
    Properties:
      Environment:
        Variables:
          userTable: !Ref myDynamoDBTable
          s3role: !GetAtt IamS3Role.Arn
          origin: !Sub ${Origin}
          filesBucket: !Sub ${FilesBucket}
      CodeUri: Lambda/
      Handler: showfiles.lambda_handler
      Runtime: python3.8
      Policies:
        - DynamoDBCrudPolicy:
            TableName: !Ref myDynamoDBTable      
      Events:
        getCounter:
          Type: Api
          Properties:
            Path: /showFiles
            Method: GET
            RestApiId: !Ref ApiGatewayShowFilesApi

We are defining here couple of parameters that will be set as environment variables for Lambda. With Origin we specifying the origin domain for CORS. It is our S3 bucket's Virtual hosted style URL. FilesBucket is the bucket where files are stored. In Serverless Function definition, it uses showfiles.py as lambda handler. Function has permissions to use DB. We also create API for this lamdba with path /showFiles.

Lets see what we do in the Lambda handlers. We modified login.py lambda handler from previous post. We are setting a cookie once the user is authenticated. This is completely optional and really not required for STS to work but you might need some kind of system to identify that user is already authenticated.

 if decryptedPass == pwd : 
      token = secrets.token_hex(16)  
      response = table.update_item(
        Key={
            'userid': uname
        },
        AttributeUpdates={
            'token': {
                'Value': token,
            }
            }
        )

      return {
        'statusCode': 200,
        'headers':{
             'Set-Cookie':'tkn='+uname+'&'+token+';Secure;SameSite=None;HttpOnly;Domain=.amazonaws.com;Path=/',
             'Content-Type': 'text/html'
         },
        'body': '<html><head><script>window.location.href = \''+ os.environ['showFilesUrl']+'\' </script></head><body>Hello</body></html>'
      }
    else:
     response['status'] = 'Login Failed'
     return {
        'statusCode': 200,
        'body': json.dumps(response) 
      }

When user submits the username and password, Login lambda handler will authenticate the user, store the unique id in DB, set the cookie in response with location to showFiles url html page from S3 bucket. On page load, browser will change the location to showfiles url that will trigger the showFiles Lambda handler defined in showFiles.py

showFiles.html

<html>
<head>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/semantic-ui/2.4.1/semantic.min.css" integrity="sha512-8bHTC73gkZ7rZ7vpqUQThUDhqcNFyYi2xgDgPDHc+GXVGHXq+xPjynxIopALmOPqzo9JZj0k6OqqewdGO3EsrQ==" crossorigin="anonymous" />
<script
  src="https://code.jquery.com/jquery-3.1.1.min.js"
  integrity="sha256-hVVnYaiADRTO2PzUGmuLJr8BLUSjGIZsDYGmIJLv2b8="
  crossorigin="anonymous"></script>

<script src="https://cdnjs.cloudflare.com/ajax/libs/semantic-ui/2.4.1/semantic.min.js"></script>
</head>
<body>

<div class="ui raised very text container">
<h1 class="ui header">File Access System</h1>
<i class="folder open icon"></i></i><div class="ui label">Files</div>
<div id="files" >Loading..</div>
</div>
</body>
<script>

fetch("https://9nimlkmz74.execute-api.us-east-2.amazonaws.com/Prod/showFiles/", {
  credentials: 'include'
})
  .then(response => response.text())
  .then((body) => {
    var files="";
    var obj =  JSON.parse(body)
    for (i = 0; i < obj.length; i++) {
            files =  files+ "<i class='file alternate outline icon'><a href='#'>&nbsp;&nbsp;"+obj[i]+"</a>"
    }
    document.getElementById("files").innerHTML= files
  })
  .catch(function(error) {
    console.log(error); 
  });

</script>
</html>

We call the showFiles API, that gets the list of Files from S3 bucket and display on the page.

showFiles.py

import json
import logging
import boto3
import os

log = logging.getLogger()
log.setLevel(logging.INFO)

#retuns login cookie information userid and unique token
def getLoginCookie(cookies):
    data ={}
    for x in cookies:
      keyValue = x.split('=')

      if keyValue[0].strip() =='tkn':
        cookieValue = keyValue[1]
        tknvalues = cookieValue.split('&')
        data['uid']=tknvalues[0]
        data['tkn']=tknvalues[1]
      else:
        cookieValue =''
      return data

#verifies unique token that is saved in database vs in request      
def verifyLogin(data):
    dynamodb = boto3.resource('dynamodb')
    table = dynamodb.Table(os.environ['userTable'])
    response = table.get_item(Key={'userid': data['uid']})
    json_str =  json.dumps( response['Item'])

    resp_dict = json.loads(json_str)
    token = resp_dict.get("token")
    return bool(token == data['tkn'])

# Returns list of files from bucket using STS    
def getFilesList():
    sts_client = boto3.client('sts')

    # Call the assume_role method of the STSConnection object and pass the role
    # ARN and a role session name.
    assumed_role_object=sts_client.assume_role(
        RoleArn=os.environ['s3role'],
        RoleSessionName="AssumeRoleSession1"
    )

    # From the response that contains the assumed role, get the temporary 
    # credentials that can be used to make subsequent API calls
    credentials=assumed_role_object['Credentials']

    # Use the temporary credentials that AssumeRole returns to make a 
    # connection to Amazon S3  
    s3_resource=boto3.resource(
        's3',
        aws_access_key_id=credentials['AccessKeyId'],
        aws_secret_access_key=credentials['SecretAccessKey'],
        aws_session_token=credentials['SessionToken'],
    )

    bucket = s3_resource.Bucket(os.environ['filesBucket'])
    files=[]
    for obj in bucket.objects.all():
        files.append(obj.key)
    return files


def lambda_handler(event, context):
    headers = event.get("headers")
    cookies = headers['Cookie'].split(";")
    data = getLoginCookie(cookies)
    isVerified = verifyLogin(data)

    if(isVerified):
        response = getFilesList()

    return {
        'statusCode': 200,
        'headers': {
            'Content-Type': 'application/json',
            'Access-Control-Allow-Origin':os.environ['origin'],
            'Access-Control-Allow-Credentials': 'true'
        },
        'body': json.dumps(response)
    }

Focus on the lamdba_handler function here. We first get the cookie and verify the login. If verified we call the getFilesList function where the magic of STS happens. We get the arn of role to be assumed from Lambda environment variables. The assume_role function returns the credentials that contains the access key id, secret access key and session token. You can use this credentials to get S3 resources and access the bucket. We create list of files as array and send it as response.

You can find full SAM template.yml and other code for this here
https://github.com/rajanpanchal/aws-kms-sts

Before you run SAM deploy, create a S3 bucket that will store the html files. In our case, its stsexamplebucket. We use urls from this bucket in our SAM Template. On SAM Deploy, It will generate output with 3 URLS for the APIs. Modify the html files to point to those url and upload to S3 bucket. Make sure you make those files public.

Testing

You need to create a bucket with name specified in FilesBucket parameter in SAM template. This bucket will store the files for display.

Summary

In summary, we used a custom identity broker that authenticates the user and then AWS STS that allows access to AWS resources to those users. You might wonder we could have given access to S3 to Lambda Execution role instead of using STS. Of course you can and it will work. But the idea here is to use STS and you can use STS irrespective of Lambda i.e in your other applications like spring boot, java, python application, etc.
In next post, we will further extend this to use S3 signed url to give access to files stored in S3.

Feel free to point out any suggestions, errors and give your feedback in comments!