Forem: ajaydhungel23

AWS Step Functions: Creating a ‘Busy Waiting’ flow to wait for successful lambda executions.

ajaydhungel23 — Thu, 01 Jun 2023 18:45:43 +0000

Hi everyone, welcome to this article. In the past few days, I had to create a logic in AWS Step functions which involved waiting for a lambda that queries athena before going forward in the flow.

When I was discussing the logic, it felt like an easy thing to accomplish in minutes but I was surprised upon knowing that Step Functions does not have its native technique/solve to wait for a state to succeed before starting another. This made me search for hours before coming up with a solution.

Here, I will share with you a technique which will help you in the future.

First of all, I will create a simple lambda function. All we need to know from the lambda is whether it did its job. So, this simple function will make sure to return its status.

Lambda function:

import json

def lambda_handler(event, context):
    response = {
        'status': 'success',
        'message': 'Hello, World!'
    }

    return response

You can copy both the function and the Step function definition which I will share below to replicate all of this in your own environment.

Now, lets go ahead and create a state machine in AWS Step Functions.

Click on ‘Create state machine’.

Choose the option to write your own definition and choose ‘standard’.

Now, in the definition section, copy and paste the following definition.

Step Functions definition:

{
  "Comment": "A Hello World example of the Amazon States Language using Pass states",
  "StartAt": "Main",
  "States": {
    "Main": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "OutputPath": "$.Payload",
      "Parameters": {
        "Payload.$": "$",
        "FunctionName": "arn:aws:lambda:us-east-1:031342435657:function:stepfunctionlambda_test:$LATEST"
      },
      "Retry": [
        {
          "ErrorEquals": [
            "Lambda.ServiceException",
            "Lambda.AWSLambdaException",
            "Lambda.SdkClientException",
            "Lambda.TooManyRequestsException"
          ],
          "IntervalSeconds": 2,
          "MaxAttempts": 6,
          "BackoffRate": 2
        }
      ],
      "Next": "WaitforInvocation"
    },
    "WaitforInvocation": {
      "Type": "Wait",
      "Seconds": 2,
      "Next": "CheckInvocationsStatus"
    },
    "CheckInvocationsStatus": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.status",
          "StringEquals": "success",
          "Next": "Final"
        }
      ],
      "Default": "WaitforInvocation"
    },
    "Final": {
      "Type": "Succeed"
    }
  }
}

Save the definition and click next.

Specify the name of your state machine and then leave everything else as default. A new execution role will be created if you don’t specify any.

Save it and then you will have created a state machine which looks like this:

Visual flow:

To explain the definition above, I have pointed out how this state machine works:

The state machine starts at the "Main" state.
The "Main" state is a Task state that invokes a Lambda function specified by the "FunctionName" parameter. The input to the Lambda function is the entire input passed to the state machine.
The "OutputPath" parameter is set to "$.Payload" to capture the output of the Lambda function.
If the Lambda function invocation fails with certain exceptions specified in the "Retry" section (Lambda.ServiceException, Lambda.AWSLambdaException, Lambda.SdkClientException, Lambda.TooManyRequestsException), the state machine will retry the invocation up to 6 times with a backoff interval of 2 seconds.
After invoking the Lambda function, the state machine proceeds to the "WaitforInvocation" state.
The "WaitforInvocation" state is a Wait state that pauses the execution for 2 seconds.
After the wait period, the state machine goes to the "CheckInvocationsStatus" state.
The "CheckInvocationsStatus" state is a Choice state that checks the value of the "$.status" field in the output of the Lambda function. If the status is "success", the state machine proceeds to the "Final" state. Otherwise, it goes back to the "WaitforInvocation" state.
The "Final" state is a Succeed state that indicates the successful completion of the state machine.

Now, lets go ahead and execute it!

Click on ‘Start execution’ and leave the input field empty.

Incase of a lambda that requires input, you can insert your input here as a Json

After you start your execution, each state will turn into different colours which indicates the following.

Green: Indicates successful completion of a state or step.
Red: Indicates a failure or error occurred during the execution of a state or step.
Yellow: Indicates a state is in a waiting or paused state.
Blue: Represents a decision or choice point where the state machine evaluates conditions to determine the next state.
Gray: Represents a passive state that simply passes input to output without performing significant work.
White: Represents transitions or connections between states, indicating the flow of execution from one state to another.

Finally this is how the execution will look like:

Thank you. Hope you can now go ahead and apply the “Busy Waiting” logic in your step functions.

Leveraging AWS CodePipeline pipeline and Serverless Framework to deploy a Flask API.

ajaydhungel23 — Tue, 28 Mar 2023 03:30:52 +0000

For the past couple of days. I have been working with the Serverless framework. I came to know that you can actually host your flask API that integrates several AWS services such as Lambda, API Gateway, and DynamoDB using this framework. I had to spend a lot of time to understand the process at first since there is not a lot of content regarding this implementation. So, here’s me contributing a little.

To make this swift, I wanted to integrate the entire process with a pipeline that contains the code and a build environment. Here’s how everything is carried out. Please understand that this is not a beginner level tutorial and the whole process is written in assumption that the readers have knowledge and experience with AWS services and Python Flask.

First, you need to have a flask api. An API that works. I couldn’t share the actual code that I first used since it is confidential . So, I have decided to go with a template of serverless framework. You can also convert your flask api to be compatible with serverless using the serverless-wsgi plugin. For this, you may need to make some changes to your serverless.yaml configurations. You can check it here to achieve that

Now that you have your api , lets upload it again to an aws codecommit repository. You can clone an empty repo, add the files, commit and push. Check it here.

Next, we need to create a build project in AWS CodeBuild.

In this codebuild project, use the following buildspec.yaml file. You can choose an Ubuntu environment for the build project to run.

version: 0.2
phases:
  install:
    runtime-versions:
      nodejs: 14
    commands:
        - npm install -g serverless
        - npm install serverless-compose
        - serverless --version
        - npm install serverless-deployment-bucket
        - sls plugin install -n serverless-wsgi
        - sls plugin install -n serverless-python-requirements

  build: 
    commands: 
      - serverless deploy

Remember: In this case, Codebuild is only used to run the deploy command. All the resources are built by serverless by automatically running a cloudformation.

Now, we’re also gonna need another yaml file. This one is the configuration file for the serverless framework which will be deployed once we deploy it using the Aws pipeline.

Here’s the configuration code.

service: new12

frameworkVersion: '3'

custom:
  wsgi:
    app: app.hello

provider:
  name: aws
  runtime: python3.9

functions:
  api:
    handler: wsgi_handler.handler
    events:
      - httpApi: '*'

plugins:
  - serverless-wsgi
  - serverless-python-requirements

Also keep your flask api file in the same directory. Make sure to configure codebuild to look for the specific branch in the specific repo.

Once everything is created, we need to make sure that our Codebuild service role has enough permission to use cloudformation and all the other services associated with the resources built through the cloudformation stack.

In my case, I faced a lot of failed builds due to permission issues.Always make sure to check your build logs when your build fails, so that you can get a clear understanding of the underlying reasons and issues.

So, to make it quick I added some permission to the codebuild service role. I suggest to create your own permission policies based on the least privilege access policy rather than assigning managed policies with Full access.

Once the permissions are set, your pipeline will run swiftly. Here's how my pipeline looks like with the two phases.

Here are the outputs from the cloudformation stack.

You invoke URL will be generated in Amazon API Gateway. Here in the build logs, you can see an invoke URL for the api.

The serverless Framework will also help you monitor your resources and their performance through the console. All you need to do is login to the console and connect to your aws account. The console will run another cloudformation stack to integrate with the resources in our aws account. Here is how the dashboard looks like.

Thank you for making it to the end, hope You found what you were looking for!