Forem: Kishan

AWS AppSync: The Game-Changing Access Method For Environment Variables

Kishan — Thu, 08 Feb 2024 03:26:18 +0000

Excitingggggg news! AWS has just rolled out support for environment variables in AppSync. Let's delve into incorporating this feature in our SAM & Serverless Framework.

Solution

Step1: When defining AppSync, we can now add environment variables similarly to how it's done during Lambda creation. In the properties section, include EnvironmentVariables and add key-value pairs as needed.

Resources:
  UserAppsync:
    Type: "AWS::AppSync::GraphQLApi"
    Properties:
      Name: "user-int-appsync"
      AuthenticationType: "AWS_IAM"
      EnvironmentVariables:
        USERTABLENAME: "user-int-table"

Step2: Having defined the environment in the YAML file, let's access it in our resolver file. Once environment variables are specified, they are passed for both request and response within the ctx (context) of the function parameter

//accessing the environment variable(NEW METHOD)
const tableName = ctx.env.USERTABLENAME

Accessing it is demonstrated below Inside the Resolver:

/**
 * @param {*} ctx - Context
 * @retuns {Object} - Request object
 */

export function request(ctx) {
  //accessing the environment variable(NEW METHOD)
  const tableName = ctx.env.USERTABLENAME
  return {
    operation: 'BatchGetItem',
    tables: {
      [tableName]: { keys },
    },
  };
}

/**
 * @param {*} ctx - Context
 * @retuns {Object} - Response object
 */
export function response(ctx) {
  return {};
}

As discussed, the environment is now passed within the ctx of both request and response. This enhancement simplifies the process of passing and accessing environment variables with the latest updates.

ServeLess Pattern: Lambda Error Handling Simplified With SQS and Dead Letter Queue (DLQ)

Kishan — Sun, 07 Jan 2024 06:17:13 +0000

Scenario

You just started with Lambda and your function receive lot's of request coming, and some of the request get failed and you not able to trace it?, let's use Simple Queue Service (SQS) and Dead Letter Queue(DLQ) and build a pattern which will catch all the failed request.

Architecture pattern

To catch all the request that got failed, we need to build a pattern where the Lambda is triggered from the SQS queue and then DLQ attached to it that it will catch the Lambda failure. Below is the Architecture design for the solution

let's implement the pattern, by understanding the resources to be created:

1. normalQueue: This Queue will be the starting point which will trigger the Lambda.

2. FailedLambda: The Lambda will receive the data from SQS and for now it is designed to throw error.

3. DeadLetterQueue: When the Lambda get failed the Message is stored in this DLQ.

How to Implement?

Step1: Create an SQS DLQ named "deadLetterQueueTest" with the type of Standard and a retention period of 14 days.

To make this queue available for other queue to use as dead letter queue(DLQ), we need to configure re-drive policy for this queue, which will allow all the queue in the region to access it.

Step2: Next Create a queue named "normalQueue" which will be used as a source to trigger the Lambda function. And Set the type to be Standard and retention period to 14 days.

In the Dead-letter queue section, select the DLQ created in Step 1 and click on create queue button.

Step3: Now Let's Create the Lambda function named "failedLambda" with permissions to receive SQS messages. Later Navigate to SQS and attach this lambda as trigger to the "normalQueue" as below.

In the Lambda function code, it will throw an error, such as an incorrect URL in an Axios call, to simulate a failed request

const axios = require('axios');

exports.handler = async (event) => {
    try {
      console.log('[INFO] Event', event);
      const result = await axios.get('https://jsonplaceholdr.typicode.com/todos/1');
      return result.data;
    } catch (err) {
      console.log('[ERROR] Error', err)
      throw err
    }
  };

Test

To Test Navigate to the "normalQueue", click on Send and receive message in the message tab, and send data that the Lambda can process. The data should be looking something similar to below

This will trigger the Lambda function, which will get failed. As a result, the corresponding message will be added to the DLQ. To verify you can then navigate to the DLQ, click on Send and receive message, and observe the failed message.

Summary

By Following this pattern to catch the failure in the Lambda made developer job easier to understand which request and what caused the request to get failed, by using SQS queue & Dead Letter Queue (DLQ) it made easier to catch the error or failure scenario more effective.

AWS SQS: Decode the Secret Power Of Short & Long Polling

Kishan — Tue, 19 Dec 2023 06:25:53 +0000

Discover the dynamics of Short & Long Polling in AWS SQS. When using the ReceiveMessage API call, SQS can be configured to perform short or long Polling, which will retrieve data from distributed components. Short Polling fetches a subset of servers, while long Polling queries all servers for message retrieval. let's understand each Polling mechanism for seamless integration into your application.

What is Short & Long Polling in SQS?

While invoking the ReceiveMessage API call in SQS, the option to configure Short & Long Polling arises.

Background

SQS, a distributed queue, distributes messages across various components in different regions. Short & long polling capitalises on this architecture, enhancing accessibility to fetch data from these distributed components.

Short Polling: This polling method optimise the user experience by sending data seamlessly fast by extracting the message in the subset of the server which stores the message.

Long Polling: On the opposite to short polling, this method search all the servers to fetch the messages in the server, which will take time to fetch all the server even there is no messages in those server.

Consuming Message through Short Polling

We Will Invoke SQS in two scenarios: an empty queue and a queue with data. For short polling, set WaitTimeSeconds to 0.

Code
On the code level, to make the api call to short polling you need to specify the parameter WaitTimeSeconds to 0, and MaxNumberOfMessages to 3 (you can set as much you want).

const aws = require('aws-sdk')
const sqs = new aws.SQS({ region: 'us-east-1' });

const shortPolling = async () => {
    try {
        const params = {
            QueueUrl: 'Queue-URL',
            WaitTimeSeconds: 0,
            MaxNumberOfMessages: 3
        }
        const data = await sqs.receiveMessage(params).promise();
        console.log('[INFO] Received messages from SQS by short polling', JSON.stringify(data));
    } catch (error) {
        console.log('[ERROR] Error while short polling the sqs', error);
        throw error;
    }
};

Scenario1:
When the queue is empty, SQS quickly looks over the server and returns an empty message.

Scenario2:
let assume you have 5 message in your queue and make a query to get those message. SQS fetches a subset of the server and returns messages such as MSG0,3(message with number postfix of 0 and 3).

For subsequent calls, it returns Message 2 as it finds the third message in the server.

Consuming Message through Long Polling

Let's Invoke SQS in two scenarios: an empty queue and a queue with data. For long polling, set WaitTimeSeconds to 20 seconds.

code
below code will be used for both the scenario, here i have specified the WaitTimeSeconds to be 20 seconds and maxMessage to be 3, this will make long polling request.

const aws = require('aws-sdk')
const sqs = new aws.SQS({ region: 'us-east-1' });

const longPolling = async () => {
    try {
        const startTime = new Date().getTime();
        const params = {
            QueueUrl: 'https://sqs.us-east-1.amazonaws.com/693651664797/PollingQueue',
            WaitTimeSeconds: 20,
            MaxNumberOfMessages: 3
        }
        const data = await sqs.receiveMessage(params).promise();
        const endTime = new Date().getTime();
        console.log('[INFO] Time taken for long polling', `${endTime - startTime} ms`);
        console.log('[INFO] Received messages from SQS by long polling', JSON.stringify(data));
    } catch (error) {
        console.log('[ERROR] Error while long polling the sqs', error);
        throw error;  
    }
};

Scenario1:
When attempting to invoke the queue devoid of data on the server, SQS patiently awaits for a duration of 20 seconds, which is scanning through all distributed servers to fulfil the request, ultimately responding an empty array.

Scenario2:
For this polling it will take all 20 seconds to fetch all the messages in the server, and after 20 seconds it return the response.

Summary

Harness the potential of Short & Long Polling in AWS SQS. Grasp the intricacies of each polling mechanism, tailor them to your scenarios, and optimise message retrieval for a responsive and efficient application. Tailor your SQS interactions with precision, leveraging the strengths of short and long polling based on your use case.

AWS DynamoDB: Comparing Lambda & Step Functions To Scan 1MB Data

Kishan — Sun, 10 Dec 2023 10:46:50 +0000

while working In a project i was exploring the Lambda, DynamoDB, and step function, while my aim was to fetch data from DDB in both the Lambda and in Step Function, which led me to an idea of comparing both the method with time, memory usage, and cost for each method. to this idea i added another layer of scanning 1MB of data from DynamoDB. now without wasting time let's explore the setup, execution and cost which should be considered.

Setup

To test both the scenario, let's setup required Lambda & step function, for both method will be sharing code to scan DB records, all the creation of Lambda & step function definition will be done in the aws console.

stepFunction Definition:

{
  "Comment": "Scan all the DB records",
  "StartAt": "Scan User Data",
  "States": {
    "Scan User Data": {
      "Type": "Task",
      "Parameters": {
        "TableName": "userInfo",
        "Limit": 50
      },
      "Resource": "arn:aws:states:::aws-sdk:dynamodb:scan",
      "End": true
    }
  }
}

Lambda Logic: Below code aims to scan the data from DDB, in the starting of the process i noted the time and and the completion of the task i again noted the time by the end i can find the difference from start to end, to find the total time taken to complete the task.

const { DynamoDBClient, ScanCommand } = require("@aws-sdk/client-dynamodb");

const dynamoDB = new DynamoDBClient({ region: 'us-east-1' });

const scanWholeTable = async (startKey) => {
    let items = [];
    const startTime = new Date().getTime();
    try {
      const params = {
        TableName: 'userInfo',
        ExclusiveStartKey: startKey,
        Limit: 25,
      };

      // prepare the params to scan the table
      const command = new ScanCommand(params);
      const data = await dynamoDB.send(command);

      items = items.concat(data.Items);

      // If there are more items to scan, recursively call the scanDynamoDBTable function with the last evaluated key
      if (data.LastEvaluatedKey) {
        return scanWholeTable(data.LastEvaluatedKey);
      }

      // check the time taken to complete the scan
      const endTime = new Date().getTime();
      const totalTime = endTime - startTime
      console.log('[INFO] time taken to complete the db scan', totalTime)
      return items;
    } catch (error) {
      console.log('[ERROR] Error while scanning the table', error);
      throw error;
    }
}

module.exports.handler = async (event, context) => {
  try {
      const finalUserInfo = await scanWholeTable(null);
      console.log('[INFO] UserInfo', finalUserInfo);
      return {
        statusCode: 200,
        headers: {
          'Content-Type': 'application/json',
        },
        body: JSON.stringify({userData:finalUserInfo, length: finalUserInfo.length}),
      }
    } catch (error) {
      console.info('Error in the handler:', error);
      return error;
    }
};

Time to complete the task.

Once all the setup is done, I triggered both the scenario multiple time to determine amount of time both method takes to scan 1MB of data from DDB.

Lambda:

I aimed trigger the lambda multiple times.
As I triggered multiple time, Lambda execution time ranged between 500-1100 milliseconds for scanning 1MB of data.
To complete the task Lambda execution took 1-2 seconds
And for The first trigger Lambda took 1253 milliseconds.

As triggering multiple time lambda showed reduced time of 544 milliseconds.

Step Function:

Continuously triggering the step function surprisingly took only 266 milliseconds to complete the task.
Task initialisation and completion took 400-500 milliseconds.
Apart from some lag, both Lambda and Step Function times are worth considering.

Cost Consideration.

Let's use the AWS Price Calculator to estimate the cost of scanning 1MB on DynamoDB.

Lambda Cost:
Let's configure the region (us-east-1), architecture (x84), memory (512), timeout (30), and trigger per day (let's keep it at 50).once all the configuration is set let's break down the cost.

For 50 requests per day, which total will be 1,520 per month, and memory usage of 608.33 GB for computing.
AWS provides 400,000 GB as a free tier for compute, resulting in a compute cost of 0 USD.
Currently AWS Lambda makes 1 million triggers as part of the free tier, which result in our process to cost 0 USD.
Running the Lambda will cost 0 USD until it reaches 400k compute and 1 million requests.

Step Function Cost:
Now configure the region (us-east-1), select standard workflow, and set the workflow requests to 50 per day, 5 state transition of the step function, let's consider there will be 5 states after it to transform data and send back the result.

With 1,520 requests triggered to the Step Function per month.
AWS provides 4,000 transitions as a free tier for the step function service, and for our step function it will be total of 7,604 transitions per month.
As a result above point the billable transitions amount will be 3,604 transitions after 7,604 - 4,000.
The cost for one transition state is 0.000025 USD.
leading to an estimated cost of 0.09 USD.

Summary

Scenario	Step Function	Lambda
Data Size(MB)	1MB	1MB
Time in Milliseconds	266	500 - 1100
Compute(GB)	686	not-specified
Cost(USD)	0.09	0
Disadvantage	As the Step Function send input to next state as marshalled data from DDB, and un-marshalling correctly mapping the "Null" type is problematic from the Payload of the step function in next states.	N/A

As i delve into comparing the Lambda & Step Function to scan 1MB of data from DynamoDb, yield insightful observations, on the time & cost for both methods, where Lambda free-tier benefits overshadow the Step Function consistent performance.

AWS IAM Auth: Calling AppSync Mutations from Lambda (Step-by-Step-Guide)

Kishan — Sun, 03 Dec 2023 15:23:25 +0000

Have you recently started working in AppSync and wondering how to trigger mutations or queries from Lambda functions? While you can use Axios to complete this, to make the call more secure, we will approach using IAM to call mutations from Lambda which will add an extra layer of protection. This blog aims to make dev life easier with the approach on how to implement the below scenario.

Scenario

In This blog i'll introduce the mutation schema which we will be using to call inside the Lambda, and then will be using IAM auth to get token so that lambda is allowed to call the mutation. now lets explore the approach

pre-requirities:

Create the required AppSync mutation.
Set up a Lambda function that will make the mutation call.
Ensure necessary permissions are in place to trigger the mutation from Lambda.

Soultion

Step1: Let's create an AppSync mutation which will be use to create user in DDB. Below is the schema for the createUser mutation:

      mutation MyMutation(
        $age: Int!, 
        $id: String!, 
        $isDeleted: String!, 
        $name: String!
      ) {
        createUser(
            age: $age, 
            id: $id, 
            isDeleted: $isDeleted, 
            name: $name
        ) {
          body {
            age
            id
            isDeleted
            name
          }
          message
          status
        }
      }

Step2: As the mutation is been created and we require IAM permission for your lambda, with the policy to trigger the appsync mutation, for that let's specify the region, accountId, and apiId in the resource.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Action": [
                "appsync:GraphQL"
            ],
            "Resource": [
                "arn:aws:appsync:region:accountId:apis/apiId/types/Mutation/fields/createUser"
            ]
        }
    ]
}

Step3: once all the necessary setup are done, you need to go to your lambda and paste the below code which will call the mutation, and install the required libraries which will be used to call the mutation:

'axios', '@aws-crypto/sha256-js', '@aws-sdk/credential-provider-node',  '@aws-sdk/signature-v4', '@aws-sdk/protocol-http'

Now, copy the below Lambda code:

const axios = require('axios');
const crypto = require('@aws-crypto/sha256-js');
const { defaultProvider } = require('@aws-sdk/credential-provider-node');
const { SignatureV4 } = require('@aws-sdk/signature-v4');
const { HttpRequest } = require('@aws-sdk/protocol-http');
const { Sha256 } = crypto;

const callUserMutation = async (mutationInput) => {
    try {
        const { id, name, age, isDeleted } = mutationInput;
      const query = `
      mutation MyMutation(
        $age: Int!, 
        $id: String!, 
        $isDeleted: String!, 
        $name: String!
      ) {
        createUser(
            age: $age, 
            id: $id, 
            isDeleted: $isDeleted, 
            name: $name
        ) {
          body {
            age
            id
            isDeleted
            name
          }
          message
          status
        }
      }`;

      const APPSYNC_MUTATION_URL = 'your-appsync-mutation-url';
      const signer = new SignatureV4({
        credentials: defaultProvider(),
        region: 'us-east-1',
        service: 'appsync',
        sha256: Sha256,
      });

      const { host, pathname } = new URL('your-appsync-mutation-url');
      const params = {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
          Host: host,
        },
        body: JSON.stringify({
          query,
          variables: {
            age,
            id,
            isDeleted,
            name,
          },
          operationName: 'MyMutation',
        }),
        hostname: host,
        path: pathname,
      };
      const requestToBeSigned = new HttpRequest(params);
      console.log('[INFO] requestToBeSigned', requestToBeSigned)

      // Sign the request to call mutation
      const signedRequest = await signer.sign(requestToBeSigned);

      // sending the signed request using Axios to call mutation.
      const response = await axios({
        url: APPSYNC_MUTATION_URL,
        method: signedRequest.method,
        data: signedRequest.body,
        headers: signedRequest.headers,
      });
      const { data } = response.data;
      console.log('[INFO] data from the mutation', data);
      return data;
    } catch (error) {
      console.log('[ERROR] Error while calling the mutation', JSON.stringify(error));  
      return error;
    }
};

module.exports.handler = async (event) => {
    try {
        const mutationData = callUserMutation({age: 22, id: '1', isDeleted: 'false', name: 'kishan'});
        console.log('[INFO] mutationData', mutationData);
    } catch (error) {
        console.log('[ERROR] Error',error);
        return error;
    }
}

Breaking Down the Code:

after seeing the code does it look little confusion?, let understand the source code by splitting it.

Initially, you must configure the parameters, such as the query, variables, method, and URL. which will be required to call the mutation.

      const { host, pathname } = new URL('your-appsync-mutation-url');
      const params = {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
          Host: host,
        },
        body: JSON.stringify({
          query,
          variables: {
            age,
            id,
            isDeleted,
            name,
          },
          operationName: 'MyMutation',
        }),
        hostname: host,
        path: pathname,
      };

Once you've made the parameters, the next step will be focusing on getting the authorisation token, that will allow the Lambda to invoke the mutation. To achieve this, use the SignatureV4() functionality, initiating the signature.sign() process. This will return an authorisation code which can be integrated with your parameters so that it as access to call the mutation.

      const signer = new SignatureV4({
        credentials: defaultProvider(),
        region: 'us-east-1',
        service: 'appsync',
        sha256: Sha256,
      });

      // Sign the request to call mutation
      const signedRequest = await signer.sign(requestToBeSigned);

After the IAM authentication token has been added to your parameters, the next step involves utilising Axios to initiate the mutation call and retrieve the response. The following code will be doing the task.

      // sending the signed request using Axios to call mutation.
      const response = await axios({
        url: APPSYNC_MUTATION_URL,
        method: signedRequest.method,
        data: signedRequest.body,
        headers: signedRequest.headers,
      });

Conclusion

By following these steps, now you are aware on how to call the appsync mutation from the Lambda with IAM security protection.

Solving the Puzzle: How to Pass Environment Variables in AWS AppSync Resolvers using Serverless

Kishan — Sat, 25 Nov 2023 05:29:04 +0000

Are you just diving into the world of AWS AppSync and puzzled about passing environment variables to the resolver using the Serverless Framework? If you've scoured countless blogs without finding a solution, you've landed on the right page!

Scenario

Imagine you have an AppSync endpoint triggering a mutation. In this scenario, you need to validate user information and perhaps trigger an SQS with a unique accountID for each environment or use the tableName in the resolver function to fetch data from DynamoDB. Whatever your use case, you can follow these steps to seamlessly pass environment variables to your resolver function.

Solution

Step1:- Add a property called substitutions to the resolver you've created. This is where you'll pass your environment variables as key-value pairs. In the example below, I'm using tableName as an environmental variable.

Step2:- Now that you've successfully passed your environment variables to the resolver, you'll need to access them within the resolver function. While in a Lambda function, you might use 'process.env.tableName', in the resolver function, you'll need to add "#" at the start and end of the name you passed in Step 1. Here's an example:

const tableName = '#tableName#';
const accountId = '#accountId#';

Ensure you add this before the request and response functions in your resolver.

Step3:- With the setup complete, you now have the knowledge to pass environment variables seamlessly to your AppSync resolver function.

This blog aims to simplify the process, ensuring you can efficiently manage and utilize environment variables in your AppSync projects. Feel free to explore, experiment, and elevate your serverless development experience!"