Forem: Rahul Nath

5 Recommended Patterns When Using Cancellation Token in .NET

Rahul Nath — Wed, 03 Apr 2024 04:29:47 +0000

Are you passing around the CancellationToken to all your functions blindly?

Stop using Cancellation Tokens the wrong way in .NET!

CancellationTokens in .NET allows the caller to express lost interest in the result of an operation. This can be at a Function level, class level, API endpoint, etc.

Does that mean all requests are the same and can be canceled similarly?

Short answer - No!

In this post, let’s learn five good practices when using Cancellation Tokens in your application code.

Cancellation tokens are a great way to implement cooperative cancellation of asynchronous or long-running synchronous operations.

Cancellation tokens allow us to cancel out long-running processes and HTTP requests to other applications, thereby reducing the time our server spends on requests that no one cares about anymore.

A .NET Programmer's Guide to CancellationToken

rahulpnath.com

The Problem With Cancelling All Functions

Any time you chain multiple cancellable operations together by calling them together (either parallelly or sequentially), it's possible to leave the application in an invalid state.

Let's look at the same example as in the previous post, where we are uploading a file to Amazon S3.

Amazon S3 For the .NET Developer: How to Easily Get Started

Learn how to get started with Amazon S3 from a .NET Core application. We will learn how to store and retrieve data from the storage, important concepts to be aware of while using S3.

rahulpnath.com

This problem is not limited to Amazon S3 but applies to any cancellable operations chained together.

So, continue reading even if you don't understand how Amazon S3 works or how to use it from a .NET application. But if you want to learn more about Amazon S3, check out the post below.

app.MapPost("/upload-large-file", async ([FromForm] FileUploadRequest request, CancellationToken cancellationToken) =>
    {
        try
        {
            var s3Client = new AmazonS3Client();
            await s3Client.PutObjectAsync(new PutObjectRequest()
            {
                BucketName = "user-service-large-messages",
                Key = $"{Guid.NewGuid()} - {request.File.FileName}",
                InputStream = request.File.OpenReadStream()
            }, cancellationToken);

            await PerformAdditionalTasks(CancellationToken.None);
            return Results.NoContent();
        }
        catch (OperationCanceledException e)
        {
            return Results.StatusCode(499);
        }
    })
    .WithName("UploadLargeFile")
    .DisableAntiforgery()
    .WithOpenApi();

async Task PerformAdditionalTasks(CancellationToken cancellationToken)
{
    await Task.Delay(1000, cancellationToken);

    var snsClient = new AmazonSimpleNotificationServiceClient();
    await snsClient.PublishAsync(new PublishRequest()
    {
        TopicArn = "<SNS TOPIC ARN>",
        Message = "UserUploadedFileEvent"
    }, cancellationToken);

The API endpoint takes in the CancellationToken and passes that on to the Amazon S3 client PutObjectAsync method.

If the user cancels out of the upload process midway through uploading a large file, the file upload is also canceled and discarded from S3.

Let's say in our business functionality, we have to do some additional work right after uploading a file to S3, represented by the PerformASdditionalTasks function below.

The PerformAdditionalTasks for now, it simulates work with a Task.Delay and publishes a message into the Amazon SNS notifying anyone else interested in the UserUploadedFileEvent.

The PerformAdditionalTasks also takes in the same CancellationToken passed into the API endpoint.

When invoking the API endpoint, if the user cancels out before the file upload is complete or waits until the end, everything works as expected.

However, the operation will be canceled if the user cancels the API request right after the file is uploaded to the S3 bucket while the server processes the additional tasks.

This leaves the application in an invalid state.

If the file is uploaded but hasn't completed the additional tasks or raised the event on the file upload, it will be left dangling in that S3 bucket.

This brings us to our first recommended practice when using CancellationToken in your .NET applications.

1. Avoid canceling operations after side-effects

Once the application code has started making side effects don't cancel out of the operation.

In the above scenario, the application incurred a side effect after successfully uploading the file to the S3 bucket.

From this point on, canceling out the operation must be an intentional choice.

If it's decided to cancel the operation after that, necessary compensating action(s) must be taken. In our scenario, it will be to delete the uploaded file.

The application will continue processing and raise the events if not allowed to cancel. The function must return a success status code in this case and not throw an OperationCancelledException.

In this case, the function caller, which is any consumer of the API endpoint, must be ready to handle any cleanup activities in case the cancellation request is ignored and the processing is completed successfully.

2. Optional Cancellation Token on Public API and required elsewhere

The function composing the different functions to perform a bigger task should be able to decide whether an individual component should be canceled.

To enable this, it's recommended to make the CancellationToken mandatory on the internal/private functions.

At the same time, the callee of the public function should have the flexibility of passing in a CancellationToken or ignoring it if it does not intend to cancel.

So make CancellationToken optional on your Public functions and mandatory on the internal/private functions (that can be cancelled).

In ASP.NET Core, since the framework automatically injects the CancellationToken, which is available as part of the RequestContext, making it optional or mandatory does not make much difference.

But you can see this pattern of keeping it optional in almost all the SDKs/Nuget packages.

For example, look at the two different methods from the Amazon S3 client and the SNS client below

// S3 Client
public virtual Task<PutObjectResponse> PutObjectAsync(
  PutObjectRequest request, 
  System.Threading.CancellationToken cancellationToken = default(CancellationToken))

  // SNS Client
  public virtual Task<PublishResponse> PublishAsync(
    PublishRequest request, 
    System.Threading.CancellationToken cancellationToken = default(CancellationToken))

In both cases, the CancellationToken is optional and defaults CancellationToken.None.

3. Use CancellationToken.None after the point of 'no cancellation'

CancellationToken.None cannot be canceled; that is, its CanBeCanceled property is false.

Once the application has passed the point of 'no cancellation,' or in other words, has incurred side effects, you can pass on the CancellationToken.None after that point.

This ensures that even if the callee of the original public function cancels the token, the operation will run to completion/or a stable state.

Applying this to our original function means we do not pass on the cancellation token received from the ASP NET framework but pass on a CancellationToken.None after the file is uploaded to S3.

app.MapPost("/upload-large-file", async ([FromForm] FileUploadRequest request, CancellationToken cancellationToken) =>
    {
        try
        {
            var s3Client = new AmazonS3Client();
            await s3Client.PutObjectAsync(new PutObjectRequest()
            {
                BucketName = "user-service-large-messages",
                Key = $"{Guid.NewGuid()} - {request.File.FileName}",
                InputStream = request.File.OpenReadStream()
            }, cancellationToken);

            await PerformAdditionalTasks(CancellationToken.None);
            return Results.NoContent();
        }
        catch (OperationCanceledException e)
        {
            return Results.StatusCode(499);
        }
    })

In the updated code for our POST endpoint, once the file is uploaded to S3, it passes on the CancellationToken.None to the PerformAdditionalTasks function (and any other cancellable function after that point).

It's now the consumer of the API's responsibility to do necessary compensating work, if it requested cancellation but receives a successful response.

4. Check CancellationToken.CanBeCanceled

Suppose your functions can be made more efficient when they can't be canceled. In that case, checking if the CancellationToken can be canceled and having a different implementation for the function code is recommended.

The best example of this is the Task.Delay method.

 private static Task Delay(uint millisecondsDelay, TimeProvider timeProvider, CancellationToken cancellationToken) =>
            cancellationToken.IsCancellationRequested ? FromCanceled(cancellationToken) :
            millisecondsDelay == 0 ? CompletedTask :
            cancellationToken.CanBeCanceled ? new DelayPromiseWithCancellation(millisecondsDelay, timeProvider, cancellationToken) :
            new DelayPromise(millisecondsDelay, timeProvider);

Based on the CancellationToken.CanBeCancelled property, it switches between two different implementations using the DelayPromiseWithCancellation or DelayPromise.

It enables the function to be more efficient in scenarios where the token cannot be canceled, and it has to run till completion of the time passed to it.

5. Ignore CancellationToken if work is quick.

You can ignore the cancellation token if the work done inside your functions (public endpoints) is very short and quick.

Ignoring the token applies to your Public endpoint, not your internal functions.

So next time you pass around that Cancellation Token, think again,

Is this work cancellable?

Don't blindly pass around the token; it has consequences and can leave your application invalid.

References

Recommended patterns for CancellationToken

A .NET Programmer's Guide to CancellationToken

Rahul Nath — Fri, 22 Mar 2024 03:46:59 +0000

Imagine having a long-running request triggered by a user on your server.

But the user is no longer interested in the result and has navigated away from the page.

However, the server is still processing that request and utilizing resources until you come along and implement Cancellation Tokens in the application code.

In this blog post, let's learn

Problem with not using Cancellation Token
What is CancellationToken
Using Cancellation Token to Fix Long Running Processes
Using Cancellation Token in HTTP API Calls

The Problem with not using Cancellation Tokens

Below is a sample GET API endpoint. It simulates some long-running processes called in a loop a hundred times.

app.MapGet("/long-running-request", async () =>
    {
        var randomId = Guid.NewGuid();
        var results = new List<string>();

        for (int i = 0; i < 100; i++)
        {
            await Task.Delay(1000);
            var result = $"{randomId} - Result {i}";
            Console.WriteLine(result);
            results.Add(result);
        }

        return Results.Ok(results);
    })
    .WithName("GetAllData")
    .WithOpenApi();

The work is started as soon as the API endpoint is called.

Once invoked, if the user decides to stay at the other end of the request everything works as expected.

However, if the user navigates away from the API request or explicitly cancels out of the API request, the server will continue processing the request; in this case, it loops through the task a hundred times.

It's okay for one-off scenarios; however, if you have a busy website and the number of users increases on your website, this soon starts taking up the server resources that could otherwise be used to serve other user requests.

Let's learn how to fix this and put those server resources to better use.

What is CancellationToken in .NET?

.NET uses Cancellation Token for cooperative cancellation of asynchronous or long-running synchronous operations.

Cancellation tokens are particularly useful when long-running or asynchronous operations must be canceled under certain conditions, such as in UI applications where a user might decide to cancel an ongoing operation.

They allow for a cooperative approach to cancellation, where the executing code periodically checks for cancellation requests and responds accordingly.

Using CancellationToken in .NET

The CancellationToken object in itself is created and managed using the CancellationTokenSource.

The below source code shows a simple usage of the CancellationToken.

var source = new CancellationTokenSource();
Console.WriteLine("Press any key to cancel the operation...");
var longRunningTask = LongRunningOperationAsync(source.Token);

Console.ReadKey();
Console.WriteLine("Key Pressed");
source.Cancel();

await longRunningTask;

async Task LongRunningOperationAsync(CancellationToken cancellationToken)
{
    try
    {
        var i = 0;
        while (i++ < 10)
        {
            cancellationToken.ThrowIfCancellationRequested();
            // Simulate some work
            Console.WriteLine($"Working {i}");
            await Task.Delay(2000, cancellationToken);
            Console.WriteLine($"Completed {i}");
        }
    }
    catch (OperationCanceledException )
    {
        Console.WriteLine("User cancelled the operation");
    }
}

This code demonstrates how to use a cancellation token to cancel a long-running asynchronous operation in .NET.

It creates a CancellationTokenSource, starts a long-running task, and waits for a key press to cancel the operation.

The LongRunningOperationAsync method simulates work and periodically checks if cancellation has been requested, allowing for graceful cancellation handling.

When a key is pressed, the operation is canceled, and the program responds accordingly, providing a valid response to the calling method and the user.

If the user never cancels (in this case press the key), the long running operation will run to completion.

Cancelling Long Running API Requests in .NET

So, let's fix our long-running API request, which is still on its 88th loop, computing the work we requested.

In ASP NET APIs, when a user makes an HTTP request, the framework automatically creates a CancellationTokenSource and passes the token along with the HttpContext as the ReuestAborted property.

The framework also injects it into the Controller functions if it has a CancellationToken property, as shown in the code below.

app.MapGet("/long-running-request", async (CancellationToken cancellationToken) =>
    {
        var randomId = Guid.NewGuid();
        var results = new List<string>();

        for (int i = 0; i < 100; i++)
        {
            if (cancellationToken.IsCancellationRequested)
                return Results.StatusCode(499);

            await Task.Delay(1000);
            var result = $"{randomId} - Result {i}";
            Console.WriteLine(result);
            results.Add(result);
        }

        return Results.Ok(results);
    })

The updated code now looks at the Cancellation Token IsCancellationRequested property to check if the request has been canceled.

It continues with the work only if the request is not cancelled, otherwise returning a different response result.

So now any time a user navigates away from the page or cancels the request, our long-running process will stop immediately after and stop processing that request completely.

Using CancellationToken For HTTP Requests

You can also use Cancellation Tokens when making external calls to API's or databases.

Let's say we have a scenario in which the user can upload files, and we decide to store them in Amazon S3.

Amazon Simple Storage Service (S3) is an object storage service that provides a scalable and secure storage infrastructure.

Amazon S3 For the .NET Developer: How to Easily Get Started

Learn how to get started with Amazon S3 from a .NET Core application. We will learn how to store and retrieve data from the storage, important concepts to be aware of while using S3.

rahulpnath.com

This could also be any other service like Azure Blob Storage etc.

When uploading files, if they are large, it will take more time, but we can still be in a state where a user can navigate away.

Not handling this means we will have those large files unnecessarily uploaded to our S3 storage and our server performing those additional work for no reason.

We can use CancellationTokens for the rescue even in these scenarios.

app.MapPost("/upload-large-file", async (
        [FromForm] FileUploadRequest request, 
        CancellationToken cancellationToken) =>
    {
        try
        {
            var s3Client = new AmazonS3Client();
            await s3Client.PutObjectAsync(new PutObjectRequest()
            {
                BucketName = "user-service-large-messages",
                Key = $"{Guid.NewGuid()} - {request.File.FileName}",
                InputStream = request.File.OpenReadStream()
            }, cancellationToken);

            return Results.NoContent();
        }
        catch (OperationCanceledException e)
        {
            return Results.StatusCode(499);
        }
    })

The AmazonS3Client in this scenario, which is from the S3 Nuget package takes in a CancellationToken to it's PutObjectAsync method.

Any time the user cancels the request, the SDK automatically throws the OperationCanceledException and cancels the upload request.

Serverless Task Automation: Task Scheduling with AWS Lambda and Amazon EventBridge

Rahul Nath — Thu, 22 Feb 2024 07:14:50 +0000

Scheduling tasks using AWS Lambda is useful for automating repetitive, time-based operations. Some common use cases include data backups, report generation, data cleanup, and periodic application maintenance.

With Amazon EventBridge Scheduler, you can specify when and how often you want a Lambda Function to run. This means you can easily set up daily, weekly, or custom cron-like schedules.

In this post, let's learn how to use Amazon EventBridge and set up schedules to trigger a Lambda Function built using .NET. The concepts apply to other programming languages as well.

AWS EventBridge and .NET Lambda Trigger

EventBridge Scheduler (earlier referred to as CloudWatch Events) is highly customizable and scalable with a wide set of target API operations and AWS Services.

AWS Lambda For The .NET Developer

This blog post is a collection of other posts that covers various aspects of AWS Lambda and other services you can integrate with when building serverless applications on Lambda.

rahulpnath.com

To handle EventBridge Events from a Lambda Function we use the ScheduledEvent class from the Amazon.Lambda.CloudWatchEvents NuGet package

public string FunctionHandler(ScheduledEvent input, ILambdaContext context)
{
    Console.WriteLine($"Scheduled Task Run with {JsonSerializer.Serialize(input)}");
    return input.Source.ToUpper();
}

To run the AWS Lambda Function on a schedule, create a new Lambda Trigger and configure it to use 'EventBridge (CloudWatch Events)' trigger.

You can create a new rule and specify the name, description, and schedule expression for the recurring schedule.

In EventBridge you can create two types of scheduled rules

Rate Expression → Rules that run at a regular rate.
CRON Expression → Rules that run at specific times.

Let's learn how to use both of these to schedule and run AWS Lambda Functions.

Rate Expression Lambda Triggers

Rate expressions have two required fields - the value and unit - separated by white space.

rate(value unit)

EventBridge Lambda Trigger setup with a rate expression of 1 day

The above screenshot shows setting up an EventBridge Lambda Trigger with a rate expression of 1 day - rate(1 day).

EventBridge Rule Detail

Once created, you can navigate to the EventBridge rules by clicking the Trigger under the Lambda Triggers list or navigating directly to the Rules section under Amazon EventBridge.

The below screenshot shows the Rule I created for running a Lambda Function every minute.

Amazon EventBridge Rule detail that runs every minute.

You can further edit and customize the rule from Amazon EventBridge. You can also see the AWS Service that the rule triggers to run when it is executed.

The below screenshot shows that the rule triggers an AWS Lambda Function lambda-schedule when the rule is triggered.

Customizing EventBridge Trigger Payload For Lambda

In the current Lambda Function endpoint we are using the ScheduledEvent type to handle the incoming Event Bridge Event.

However, if you need to pass in custom information as part of the schedule trigger you can do that by specifying a Constant text in JSON format.

This is done under the AWS EventBridge Rule by setting the Additional Settings, and setting the Configure target input to be 'Constant (JSON text)'

In the above example, I have hardcoded the input to be a string input. In this case, the Lambda Function can take in a string as it's input type as shown below.

public string FunctionHandler(string input, ILambdaContext context)
{
  ...
}

CRON Expression Lambda Triggers

Rate Expressions allows you to specify how often you want a Rule to trigger. However, when you need more fine-grained control over when a rule should trigger you can use Cron Expression.

Cron expressions have six required fields, which are separated by white space.

The above screenshot sets up a CRON job to run the function every minute between 4-5 a.m. (UTC) on a Monday and Tuesday.

cron(0/1 4-5 ? * MON-TUE *)

All CRON jobs are set in UTC time, however, if you navigate to the CRON job rule in EventBridge, you can see the time in both UTC and local time zone.

The Event Schedule shows the next 10 upcoming triggers in both UTC and Local time to ensure the CRON job is set as expected.

Learn How to Easily Integrate Lambda Annotations and Other AWS Services

Rahul Nath — Mon, 19 Feb 2024 04:00:41 +0000

The Lambda Annotations Framework is a programming model that makes it easier to build AWS Lambda Functions using .NET.

In previous blog posts, we learned how to get started using the Lambda Annotations Framework, build a CRUD API Endpoint, and set up Dependency Injection when creating Lambda Functions using the Annotations Framework.

In all the above scenarios we used the Annotations Framework along with building Lambda Functions for the API Gateway.

However, you can also use the Annotations Framework when building Lambda Functions that integrate with other AWS Services.

In this blog post, let's see a quick example of using the Lambda Annotations Framework when building a Lambda Function with Amazon S3 service.

For E.g., we need to run a Lambda Function every time a new file is uploaded to an Amazon S3 bucket and have the function code read the file and process it. This could be an image post-processing Lambda Function, a report ingestion service, a bulk data upload service, etc.

AWS Lambda and Amazon S3 Integration

Amazon Simple Storage Service (S3) is an object storage service that allows you to store any data.

Any time an object is created or modified in S3, it raises event notifications. We can use these notifications in specific scenarios to perform additional business logic or application processing.

To learn more about this in detail check my Amazon S3 and AWS Lambda Triggers in .NET, blog post.

Amazon S3 and AWS Lambda Triggers in .NET

Amazon S3 raises event notifications when Objects are created and modified. Learn how to use this to trigger Lambda Functions in .NET and the different configuration associated with processing the notification messages.

rahulpnath.com

Lambda Functions that integrate with Amazon S3 services, take in a S3Event class type, to bind to the Amazon S3 event.

public async Task FunctionHandler(S3Event evnt, ILambdaContext context)
{
  ...
  var file = await this.S3Client
      .GetObjectAsync(s3Event.Bucket.Name, s3Event.Object.Key);
  ...
}

This happens by default using the basic Lambda JSON Serializer and there is nothing specific that the Annotations Framework can provide us here.

However, the features that are useful from the Annotations Framework when integrating with other AWS Services are Dependency Injection and Serverless Template for easy deployment.

Dependency Injection For Lambda Functions

Lambda Annotations Framework provides an easy out-of-the-box mechanism to set up and work with .NET Dependency Injection framework.

Instead of hard-coding the instance creation in the Function constructor, you can inject in the dependencies that the Function needs using the Function constructor or the Handler Function.

To enable the Annotations Framework we need to apply the LambdaFunction attribute to the Function Handler as shown below.

The below code shows examples of injecting the IAmazonS3 instance both using the constructor and also using the Function Handler.

public Function(IAmazonS3 s3Client)
{
    S3Client = s3Client;
}

[LambdaFunction(ResourceName = "MyLambdaFunction")]
public async Task FunctionHandler(
  [FromServices] IImageServices imageServices, S3Event evnt, ILambdaContext context)
{
  ...
}

To inject via the FunctionHandler method, we need to add FromServices attribute, to tell Annotations Framework to resolve the type from the DI container.

[Amazon.Lambda.Annotations.LambdaStartup]
public class Startup
{
  public void ConfigureServices(IServiceCollection services)
  {
      services.AddAWSService<Amazon.S3.IAmazonS3>();
  }
}

Serverless Template File For Deployment

Enable the Annotations Framework also generates the CloudFormation template file, which can be used to automate resource deployment.

In this scenario, since we are integrating with Amazon S3, we can further define the S3 bucket and also the required triggers to wire up the S3 event notification to trigger the Lambda Function.

"Bucket": {
      "Type": "AWS::S3::Bucket",
      "Properties": {
        "BucketName": {
          "Ref": "BucketName"
        },
        "NotificationConfiguration": {
          "LambdaConfigurations": [
            {
              "Event": "s3:ObjectCreated:*",
              "Filter": {
                "S3Key": {
                  "Rules": [
                    {
                      "Name": "prefix",
                      "Value": "test/"
                    },
                    {
                      "Name": "suffix",
                      "Value": ".txt"
                    }
                  ]
                }
              },
              "Function": {
                "Fn::GetAtt": [
                  "MyLambdaFunction",
                  "Arn"
                ]
              }
            }
          ]
        }
      }
    },
    "S3InvokeLambdaPermission": {
      "Type": "AWS::Lambda::Permission",
      "Properties": {
        "Action": "lambda:InvokeFunction",
        "FunctionName": {
          "Ref": "MyLambdaFunction"
        },
        "Principal": "s3.amazonaws.com",
        "SourceArn": {
          "Fn::Sub": "arn:aws:s3:::${BucketName}"
        }
      }
    },
    "LambdaRole": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "AssumeRolePolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": {
                "Service": "lambda.amazonaws.com"
              },
              "Action": [
                "sts:AssumeRole"
              ]
            }
          ]
        },
        "Path": "/",
        "ManagedPolicyArns": [
          "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
        ],
        "Policies": [
          {
            "PolicyName": "s3",
            "PolicyDocument": {
              "Statement": [
                {
                  "Effect": "Allow",
                  "Action": [
                    "s3:Get*"
                  ],
                  "Resource": [
                    {
                      "Fn::Sub": "arn:aws:s3:::${BucketName}"
                    },
                    {
                      "Fn::Sub": "arn:aws:s3:::${BucketName}/*"
                    }
                  ]
                }
              ]
            }
          }
        ]
      }
    }

The above template is in addition to the automatically generated template for the Lambda Function.

It sets up the S3 bucket (Bucket), the Lambda IAM Role (LambdaRole) giving it permission to read from the S3 bucket and also permission for the S3 bucket to invoke the Lambda Function (S3InvokeLambdaPermission).

This makes it easy to deploy the required resources and permissions together along with changes to the Lambda Function.

Similar to Amazon S3, you can also use the same approach to integrate with other AWS Services like SQS, SNS, DynamoDB, etc.

How To Effectively Manage Sensitive Information in AWS Lambda: Powertools Parameters

Rahul Nath — Fri, 16 Feb 2024 02:04:22 +0000

When building Lambda Functions, we often need to store configuration and sensitive information.

AWS Provides different services like Parameter Store, Secrets Manager, etc., to store sensitive information.

The AWS Lambda Powertools library makes it easy to work with these different services and retrieve one of their multiple parameter values.

In this blog post, let’s learn how to get started using the Lambda Powertools Parameters NuGet package, use it when building Lambda Functions, and connect quickly to Parameter Store and Secrets Manager using the library package.

AWS Powertools Parameters Package

Powertools Parameters utility is available as a NuGet package. To get started using it from the application, install AWS.Lambda.Powertools.Parameters NuGet package.

Once installed, we can use it to integrate with the various AWS Services to manage secrets and configuration.

AWS Lambda & Parameters Store

AWS Parameter Store is a centralized, secure store for your application configuration.

Parameter Store, a part of AWS Systems Manager, provides secure storage for application configuration and secret data. As parameter values, you can store passwords, database strings, Amazon Machine Image (AMI) IDs, API Keys, etc.

AWS Parameter Store For The .NET Developer: How to Easily Get Started

Learn how to get started with AWS Parameter Store and use it from a .NET application to store and retrieve configuration data. Understand how parameters are versioned and how to use labels and hierarchies to manage parameters better. Seamlessly use the built-in .NET Configuration capabilities to loa

rahulpnath.com

Parameter Store makes decoupling your code from configuration easy and acts as version control for your configuration data.

Retrieving Secrets From Parameter Store

You can either retrieve one or multiple secrets at a time from the Parameter Store.

The Powertools Parameter utility provides the ParametersManager.SsmProvider utility class to interact with the Parameter Store.

    var value = await ParametersManager.SsmProvider.GetAsync("/Value1");
    var multiple = await ParametersManager.SsmProvider
         .GetMultipleAsync("/weather-app/");

Use GetAsync method to retrieve one value and the GetMultipleAsync method to return multiple values given a key prefix.

In the example about, value represents the value of the key '/Value', and multiple has all the parameter key values that start with '/weather-app'.

Lambda Permissions for Parameter Store

For the Lambda Function to retrieve values from the Parameter Store, it needs appropriate permissions.

Let's update the IAM permission of our Lambda function and add the below policy to give it permission to retrieve the required keys.

To retrieve one parameter, we need the ssm:GetParameter and for multiple ssm:GetParametersByPath Action permissions. The below policy provides all actions starting with 'ssm:GetParameter', which is denoted by the '*' at the end.

{
  "PolicyName": "OrderApiParametersStorePolicy",
  "PolicyDocument": {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": "ssm:GetParameter*",
        "Resource": [
          "arn:aws:ssm:ap-southeast-2:189107071895:parameter/Value1",
          "arn:aws:ssm:ap-southeast-2:189107071895:parameter/weather-app/*"
        ]
      }
    ]
  }
}

Transforming Parameter Store Values

The Parameters utility supports transforming of values stored in Parameter Store.

For e.g., the parameter '/my-configuration' is stored as JSON in the Parameter Store.

When retrieving the value, we can use the WithTransformation and specify to use JSON Transformation. This allows us to automatically deserialize the configuration to a custom type that we specify (in this case MyConfiguration)

var myConfiguration = await ParametersManager.SsmProvider.WithTransformation(Transformation.Json)
     .GetAsync<MyConfiguration>("/my-configuration");
...
public class MyConfiguration
{
    public string Secret { get; set; }
    public string Url { get; set; }
}

The utility also supports Base64 transformation.

AWS Lambda & Secrets Manager

AWS Secrets Manager provides a centralized store to manage your application secrets.

Secrets can be information like passwords, credentials, connection strings, API keys, etc. Secrets Manager helps you protect access to your IT resources and data by enabling you to rotate and manage access to your secrets.

How Best To Secure Secrets When Building .NET Applications on AWS

Learn how to get started with using AWS Secrets Manager using a .NET Application. We will learn to connect to Secrets Manager from .NET using the client SDK and retrieve secrets. We will also see how to integrate Secrets Manager into built-in .NET Configuration and seamlessly use secrets from our ap

rahulpnath.com

Retrieving Secrets From Secrets Manager

The Parameters utility provides ParametersManager.SecretsProvider to retrieve secrets from the Secrets Manager.

Secrets Manager currently supports only retrieving one secret value at a time. Trying to use the GetMultipleAsync method will throw an exception at runtime.

var secret1 = await ParametersManager.SecretsProvider
       .GetAsync("weather-app/secret1");

Lambda Permissions for Secrets Manager

The Lambda Function requires secretsmanager:GetSecretValue permission on the Secret Values to return them successfully.

Let's update the IAM Permission also to include the required permission to retrieve the Secret from Secrets Manager, as shown below.

{
  "PolicyName": "OrderApiSecretsStorePolicy",
  "PolicyDocument": {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": "secretsmanager:GetSecretValue",
        "Resource": [
          "arn:aws:secretsmanager:ap-southeast-2:189107071895:secret:weather-app/secret1*"
        ]
      }
    ]
  }
}

DynamoDB Provider

The Parameters utility also supports using the DynamoDB table as a source of Parameter key values.

You can use the ParametersManager.DynamoDBProvider to interact with the configurated DynamoDB table to return key-value pair.

I'll leave that exercise to you to explore and use. You can read more about it here in the documentationkey-value.

Why Should You Care About Lambda Lifecycle As A .NET Developer?

Rahul Nath — Thu, 15 Feb 2024 02:25:24 +0000

I made a terrible mistake when I first started using AWS Lambda Functions.

I used an instance variable in my .NET class to store the Function state and reset it in the class Constructor code. I was expecting the state to reset for each function invocation.

But, the Function state was not getting reset on each invocation.

Understanding the AWS Lambda Lifecycle is essential when building Lambda Functions.

It would be best to avoid certain things when writing your Function code; we will go over those and why in this article.

AWS Lambda Function Lifecycle

AWS Lambda provides a secure and isolated runtime environment for our function code. This execution environment manages the resources required to run our Function and also provides lifecycle support for the Function’s runtime.

The Lambda Lifecycle mainly consists of three phases:

Init Phase → Limited to 10 seconds. Starts all extension, runs Function’s static code, etc
Invoke Phase → Invokes the Function handler configured when setting up Lambda Function and waits for the duration of the Functions timeout.
Shutdown Phase → Limited to 2 seconds.

You can read more about each phase and the related activities in the official documentation here.

Lambda Lifecycle and .NET Functions

The Lambda Lifecycle affects the way we write our Function code.

Especially when using .NET, where you define the Function code within a Class. In .NET, you can define instance variables - properties at your Class level scope.

Let’s consider a simple AWS Lambda Function code, as shown below.

public class Function
{
    public int Count { get; set; }
    public Function()
    {
        Count = 0;
    }

    public string FunctionHandler(string input, ILambdaContext context)
    {
        Count++;
        return $"{input} - {Count}";
    }
}

We have a class Function where we have the Lambda Function Handler (FunctionHandler) which takes in string input. The Function class also defines a Count property that is initialized inside the Class constructor.

Since the FunctionHandler increments the Count property every time it’s called, the return value depends on whether you are invoking the Function on the same instance or different instances

I was expecting the Count property to reset every time the Lambda Function was invoked because of the initialization code in the constructor. (Now, in my original scenario where I encountered this problem, I was doing much more than just incrementing a counter.)

But, like I said, it wasn’t. Why?

.NET Constructor and Lambda Init Phase

For .NET Functions, a new instance of the Function class gets created in the Init Phase. This instance is then stored and used in the Invoke phases, where all it does is pass on the request to the Function handler.

For any new Invoke requests, Lambda uses any free instances and invokes the function method on it.

AWS Lambda creates new instances of .NET class in the Init phase and reuses the instance in the Invoke Phase.

This means any initialization code in the class constructor is only called once when the instance is created. Any state maintained at the class/instance level must be reset appropriately inside the Function handler method.

So in our example before, if we invoke the Lambda Function multiple times, it keeps incrementing the Count property and returns the value accordingly.

What Triggers New .NET Instances To Be Created?

All existing instances are removed whenever you change your function code and deploy the changes to AWS Lambda.

Any new requests coming to the AWS Lambda will create a new instance with the newly deployed code.

New instances are also automatically created when Lambda receives more requests than what it can process.

Let’s simulate the Function doing some work by adding a Thread sleep to our code, as shown below.

public string FunctionHandler(string input, ILambdaContext context)
{
    Count++;
    Thread.Sleep(1000 * 5);
    return $"{input} - {Count}";
}

The Function now takes 5 seconds to respond once invoked.

If we make two parallel requests to our Lambda Function, the Lambda Runtime will automatically create 2 instances to serve the requests simultaneously. This behavior changes based on the Lambda concurrency setting.

To simulate two parallel requests, you can make requests from Visual Studio, AWS Console, or the command line.

Below is how you can use the aws cli to invoke a function in PowerShell.

aws lambda invoke --function-name lambda-lifecycle --cli-binary-format raw-in-base64-out --payload '\\"Console\\"' response.json && cat .\\response.json

If you make two parallel calls after freshly deploying the Lambda Function with the above code, you will see that both invocations return the Count as 1. Since Lambda creates multiple instances of the class to handle both requests happening at the same time.

What to Put In The .NET Constructor?

It would be best if you didn’t depend on when and how Lambda decides to create new instances. Your code should work regardless of how the internals of Lambda works.

So to ensure this, make sure your Function classes are stateless, and you don't depend on any particular class level variables within your Function Handler.

This brings us to the obvious question, What can I put in the .NET Constructor?

I typically have any connections to databases or external services, instantiated in the class constructor, and reuse them inside the Function handler, provided these connections are stateless and thread-safe.

Some examples are DynamoDB connections, Parameter Store connections, etc.

For anything else I prefer to have the variables scoped to the Function handler.

How To Set Up Dependency Injection in Lambda Functions Using Annotations Framework

Rahul Nath — Tue, 13 Feb 2024 03:37:15 +0000

The Lambda Annotations Framework provides a programming model for .NET developers to create AWS Lambda Functions.

In previous posts, we learned how to get started with the Annotations Framework and also how to build a CRUD API using the Annotations Framework.

We saw how the Annotations Framework makes the development experience with Lambda Functions very similar to building APIs using ASP NET Core Framework.

Another feature that the Annotations Framework provides is support for Dependency Injection. This takes the development experience with Lambda Functions a level up.

In this post let’s learn how to setup Dependency Injection when building Lambda Functions using the Annotations Framework.

Setting Up DI In .NET Lambda Annotations

Lamba Annotations provides the LambdaStartup attribute that can be applied on any class in the Lambda Function Project.

The class must have a method ConfigureServices taking in the IServiceCollection to set up the Dependency Injection container.

Below I have the Startup class which has the method specified and the LambdaStartup attribute applied.

[LambdaStartup]
public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddTransient<IDependency, MyDependency>();
    }
}

The function sets up and registers the different dependencies in the service collection.

Injecting Dependencies in .NET Lambda Function

The Annotations framework supports injecting dependencies through the constructor or through the Lambda Function entry point itself.

Constructor Level DI in Lambda Function

To inject the dependency through the constructor is very similar to how we do DI in .NET. All we need to do is specify the interface dependency in the constructor as shown below.

private readonly IMyDependency _myDependencyCtor;

public Function(IMyDependency myDependencyCtor)
{
    _myDependencyCtor = myDependencyCtor;
}

The above code injects the IDependency we registered in the Startup class to the Function class.

We can store this as a class-level property and use it in our Lambda Function code.

Function Level DI in Lambda Function

You can also inject a dependency through the Function parameters by adding the FromServices attribute.

[LambdaFunction]
[HttpApi(LambdaHttpMethod.Get, "/add/{a}/{b}")]
public List<string> Add(int a, int b, ILambdaContext lambdaContext, [FromServices] IDependency scopedDependencyFunc)
{
    var returnValues = new List<string>
    {
        _scopedDependencyCtor.Test("Constructor"),
        scopedDependencyFunc.Test("Function"),
        (a + b).ToString()
    };

    return returnValues;
}

The annotation framework uses this attribute to determine that it's a dependency to be resolved from the DI ServiceCollection container and uses it to resolve and pass the appropriate values when invoking the Function code.

DI Service Lifetimes and Lambda Lifecycle

The ServiceCollection supports three lifetimes when registering dependencies.

Transient → New instance created every request
Scoped → Instance created once per client request
Singleton → Single instance of dependency created

However, this is dependent on the lifecycle of the Lambda Functions.

The Lambda Function class is created only once, during the Init phase of the Lambda lifecycle. Once the instance is created it is reused for the same calls handled by the Lambda instance.

💡The Lambda Function class is instantiated only once when a new Lambda instance is created. The same class instance is reused to call the function entry point on the same Lambda instance.

It's up to the AWS Lambda infrastructure to decide when to create a new Lambda instance. When the function is just deployed, the Lambda runtime wipes out all existing instances and creates new instances for subsequent requests.

Learn more about Lambda Lifecycle in the below blog post.

%[https://www.rahulpnath.com/blog/lambda-lifecycle-and-net/]

The lifetime scope of the dependencies injected via the Service collection is dependent on the Lambda instance.

For example, a singleton instance is only singleton in the context of one Lambda instance. If you have multiple parallel calls coming to your API Gateway, Lambda might decide to spin up multiple Lambda instances and each of them will have its own Singleton instance.

💡Avoid depending on Singleton instances to maintain state or share data across multiple invocations.

All application state information must be managed outside of Lambda - like in a database or an external Cache service etc.

Under the Hoods of DI in Lambda Annotation Framework

As we have seen previously, the annotations framework generates a wrapper class around our actual Function code class for each LambdaFunction attributed function.

This wrapper class is where all of the boilerplate code is auto-generated using .NET Source Generators.

Below you can see in the generated class, it creates a new instance of the .NET ServiceCollection which is the DI container where we are registering the dependencies in the Startup class.

The actual Function class is registered as a Singleton instance in the DI container.

If needed, you can override this to a different lifetime scope inside the Startup class. Since the last registration wins, anything in the Startup class ConfigureServices method will override the auto-generated registration set up in the wrapper.

The wrapper Function code uses the ServiceCollection instance to resolve the actual function class and any Function parameter dependencies.

The generated function class wraps around the actual function and invokes the function we have written passing in the appropriate dependencies.

You can find the full source code here.

5 Ways To Query Data From Amazon DynamoDB using .NET

Rahul Nath — Fri, 09 Feb 2024 05:19:52 +0000

Querying is an essential operation in DynamoDB.

It allows you to filter and select items in your database based on your application and user needs.

When moving over to DynamoDB from more traditional relational databases like SQL Server, you must understand the different ways you can retrieve data in DynamoDB.

In this article, let’s explore the different ways you can query data from Amazon DynamoDB when building applications using .NET. We will learn

Loading Specific Items
Querying by Partition Key
Querying by Partition and Sort Key
Using Low-level .NET SDK API For Queries
Filtering Data By Other Properties

Quick Recap of DynamoDB

Before we get into how you can query data, let’s take a quick recap of AWS DynamoDB.

DynamoDB is a cloud-hosted NoSQL database provided by Amazon Web Services (AWS). DynamoDB provides reliable performance, a managed experience, and convenient API access to interact with it.

If you are new to DynamoDB, I highly recommend checking out my AWS DynamoDB For the .NET Developer article below to understand better.

AWS DynamoDB For The .NET Developer: How To Easily Get Started

Learn how to get started with AWS DynamoDB with .NET Core by updating the default ASP NET Web API template to use DynamoDB as it's data store. We will learn to do basic Creat, Read, Update and Delete operations from the API.

rahulpnath.com

One of the reasons why it can be a bit tricky with querying data is that you almost always need the partition key to get any data out of DynamoDB.

While there are ways to get data without it, which we will see towards the end of this article, it’s generally not recommended and not performant.

Let’s learn the different ways to filter and select data from DynamoDB using .NET.

.NET And DynamoDB Setup

The sample application I am using is the default ASP NET Core Web API Template that Visual Studio creates, with the WeatherForecastController.

I have updated the API to add a CityName property to the WeatherForecast class. I have also set up Dependency Injection to inject the IDynamoDBContext context into the Controller.

var dynamoDbClient = new AmazonDynamoDBClient(
    FallbackCredentialsFactory.GetCredentials(), RegionEndpoint.APSoutheast2);
builder.Services.AddSingleton<IAmazonDynamoDB>(dynamoDbClient);
builder.Services.AddSingleton<IDynamoDBContext, DynamoDBContext>();

If any of this is unclear, check the previous blog post on Getting Started With DynamoDB.

1. Load Specific Item

To load a specific item from the DynamoDB, we require both the Hash key and the Range Key, in this case, the CityName and DateTime.

The LoadAsync method, takes in both these as parameters and returns the WeatherForecast object.

[HttpGet("specific-date")]
public async Task<WeatherForecast> GetAsync(string cityName, DateTime date)
{
    return await _dynamoDbContext.LoadAsync<WeatherForecast>(cityName, date);
}

If an item with that Hash and Range Key does not exist, the method returns null. So make sure your application handles the scenario when the item does not exist.

For an API, you can return a 404 NotFound.

2. Query By Partition Key

Often applications and users need to get more data than just one specific item.

This is most commonly required when having a List/Table shown in the application UI.

Let's say we need an API endpoint to return all the WeatherData for a given City.

In this case, all we have is the CityName, which is the Hash key.

We can use the QueryAsync method and pass in the hash key to it, as shown below.

[HttpGet("city-all")]
public async Task<IEnumerable<WeatherForecast>>  GetAsync(string cityName)
{
    return await _dynamoDbContext.QueryAsync<WeatherForecast>(cityName).GetRemainingAsync();
}

The QueryAsync method returns an AsyncSearch<T> as the response, which has additional methods to retrieve data from DynamoDb.

In the sample above, I use the GetRemainingAsync method to fetch all the items.

Note: The GetRemainingAsync method on AsyncSearch, loops through and fetches all the items matching the specified condition. This could result in more than one request to DynamoDB.

3. Query By Partition and Sort Key

When using the QueryAsync method, you can further filter the items that get returned. For a given city, you might have weather data from a long time.

And the application might be interested only in the last month's data or three, for example.

In these scenarios, we can use the QueryOperator and pass in the range key along with it.

[HttpGet("city-date-filter")]
public async Task<IEnumerable<WeatherForecast>> GetAsync(string cityName, DateTime dateTime)
{
    return await _dynamoDbContext.QueryAsync<WeatherForecast>(cityName, QueryOperator.GreaterThan,
            new object[] {dateTime})
        .GetRemainingAsync();
}

Above the QueryOperator.GreaterThan, returns all the Weather Data items that are about the specified DateTime.

Since the greater than operator requires only one parameter, we need to pass in only one DateTime.

However, if you are using the QueryOperator.Between, which requires two dates, you need to pass them in order (start and end) to the object array, as shown below.

[HttpGet("city-date-filter")]
public async Task<IEnumerable<WeatherForecast>> GetAsync(
    string cityName, DateTime dateTime, DateTime? endDateTime)
{
    return await _dynamoDbContext
        .QueryAsync<WeatherForecast>(
           cityName,
           QueryOperator.Between,
           new object[] { dateTime, endDateTime })
        .GetRemainingAsync();
}

4. Using Low-level .NET SDK API For Queries

Now that we understand how to filter and select data based on the Hash and range keys let’s see how we can further filter the data based on other properties on our DynamoDB Items.

Let’s say we need to get only Weather data for a city since last month where the temperature was above 25°C.

The High level DynamoDBContext API in the .NET SDK does not support this level of data filtering. For this, we need to switch down to the Low-Level APIs in the DynamoDB .NET SDK.

The IAmazonDynamoDB can be used for advanced data filtering scenarios like this.

In Program.cs we have already set up the AmazonDynamoDBClientwhich implements IAmazonDynamoDB, to be dependency injected.

The IAmazonDynamoDB has a similar QueryAsync method, which takes in a more complex QueryRequest class.

Let’s first understand the QueryRequest class and its usage by using it in a simple use case to get all the data for a given city Name (Hash key) before we go into the more advanced scenarios.

Query with Hash using KeyConditions

The QueryRequest expects us to pass the TableName and the KeyConditions. The KeyConditions takes in a dictionary of property names and the associated conditions for those properties.

In our case, since we want to get all the weather forecast items for the city name, let’s pass in the CityName property and the Condition ComparisonOperator.EQ to the expected city name, as shown below.

[HttpGet("city-all-low-level")]
public async Task<IEnumerable<WeatherForecast>> GetLowLevelAsync(string cityName)
{
    var request = new QueryRequest()
    {
        TableName = nameof(WeatherForecast),
        KeyConditions = new Dictionary<string, Condition>()
        {
            {
                nameof(WeatherForecast.CityName),
                new Condition()
                {
                    ComparisonOperator = ComparisonOperator.EQ,
                    AttributeValueList = new List<AttributeValue>() {new(cityName)}
                }
            }
        }
    };

    var response = await _amazonDynamoDbClient.QueryAsync(request);
    return response.Items
        .Select(Document.FromAttributeMap)
        .Select(_dynamoDbContext.FromDocument<WeatherForecast>);
}

The QueryAsync being part of the Low-Level APIs does not return our custom-defined .NET class types (WeatherForecast in this case). It returns the Items as a List<Dictionary<string, AttributeValue>> type.

To convert it into a strongly typed .NET class, use the Document.FromAttributeMap helper method and then use the FromDocument method on the DynamoDBContext as shown above.

With the above code, we are now successfully using the .NET Low-Level APIs to query the DynamoDB directly.

Query with Hash using KeyConditionExpression

Using the KeyConditions adds in a lot more boilerplate code, which can be avoided by using the KeyConditionExpression property.

With this new property, you only need to specify the equality comparison condition as a plain string with a placeholder name for the actual value. This is similar to using a SQL query and passing parameters to it.

Below we have the KeyConditionExpression to get all CityName matching to the placeholder ‘:cityName’. To pass in the actual value for the city name, we use the ExpressionAttributeValues property.

Below we need a key-value pair for the cityName placeholder and its associated value, which is the actual city name we are looking to get the data for.

var request = new QueryRequest()
{
    TableName = nameof(WeatherForecast),
    KeyConditionExpression = "CityName = :cityName",
    ExpressionAttributeValues = new Dictionary<string, AttributeValue>()
    {
        {":cityName", new AttributeValue(cityName)}
    }
};

This is way less boilerplate code and much more readable.

Reserved Keywords in KeyConditionExpression and Attribute Names

If we want to further filter the data by the date as we did in the earlier examples, we can do that using the same KeyConditionExpression. All we need to do is append the condition on the Sort Key using the ‘and’ keyword and pass in the additional ExpressionAttributeValues.

For example: To filter the data by items after a given date, we can use the below expression.

    KeyConditionExpression = "CityName = :cityName and Date > :startDate",

However, in this particular case, this query will fail since the property name ‘Date’ conflicts with a DynamoDB Reserved Keyword.

In these scenarios, where the property names conflict with DynamoDB Reserved Keywords, you need to pass the property name also as placeholder properties and send the corresponding values using the ExpressionAttributeNames property.

As shown below, the KeyConditionExpression now specifies a placeholder ‘#Date’ for the Date property and adds the appropriate mapping for the placeholder to the actual property name in the ExpressionAttributeNames property.

var request = new QueryRequest()
{
    TableName = nameof(WeatherForecast),
    KeyConditionExpression = "CityName = :cityName and #Date > :startDate",
    ExpressionAttributeNames = new Dictionary<string, string>()
    {
        {"#Date", "Date"}
    },
    ExpressionAttributeValues = new Dictionary<string, AttributeValue>()
    {
        {":cityName", new AttributeValue(cityName)},
        {":startDate", new AttributeValue(startDate.ToString(AWSSDKUtils.ISO8601DateFormat))}
    }
};

This helps us to query the data using the Low-Level .NET DynamoDB APIs and filter data based on both the Hash and sort keys.

5. Filtering Data By Non-Key Properties

Finally, let’s filter the data based on the Temperature property, which is why we set down the path of exploring the Low-Level APIs.

Using the FilterExpression property, we can further filter down the data filtered by the KeyConditionExpression.

Since in this case, we need to get all the weather data items with a temperature greater than a specific value, let’s add that condition to FIlterExpression property as shown below.

This again uses the same placeholder name for the value, and it passes the actual value as part of the ExpressionAttributeValues as shown below.

var request = new QueryRequest()
{
    TableName = nameof(WeatherForecast),
    KeyConditionExpression = "CityName = :cityName and #Date > :startDate",
    FilterExpression = "TemperatureC >= :minTemp",
    ExpressionAttributeNames = new Dictionary<string, string>()
    {
        {"#Date", "Date"}
    },
    ExpressionAttributeValues = new Dictionary<string, AttributeValue>()
    {
        {":cityName", new AttributeValue(cityName)},
        {":startDate", new AttributeValue(startDate.ToString(AWSSDKUtils.ISO8601DateFormat))},
        {":minTemp", new AttributeValue() {N = minTemp.ToString()}}
    },
};

This filters the records that are greater than or equal to the minimum temperature passed in.

Filter Expression Using Between

Both the KeyConditionExpression and the FilterExpression supports using the between operator as well.

All we need to do is pass in multiple parameters for the between operator using the and operator.

Below is an example that uses the between operator in FilterExpression to filter all the items within a given temperature range.

var request = new QueryRequest()
{
    TableName = nameof(WeatherForecast),
    KeyConditionExpression = "CityName = :cityName and #Date > :startDate",
    FilterExpression = "TemperatureC between :minTemp and :maxTemp",
    ExpressionAttributeNames = new Dictionary<string, string>()
    {
        {"#Date", "Date"}
    },
    ExpressionAttributeValues = new Dictionary<string, AttributeValue>()
    {
        {":cityName", new AttributeValue(cityName)},
        {":startDate", new AttributeValue(startDate.ToString(AWSSDKUtils.ISO8601DateFormat))},
        {":minTemp", new AttributeValue() {N = minTemp.ToString()}},
        {":maxTemp", new AttributeValue() {N = maxTemp.ToString()}}
    },
};

6. Scan The Whole Table (Avoid Using This)

A Scan Operation does exactly as its name; it scans the entire table, looking for each item that matches the specified criteria.

I’ve kept this to the last because I was hoping you could try and avoid using this as much as possible.

Scans can also be costlier for the same reason. Since you are billed based on the data scanned and not on the date returned.

Below is an example of using Scan to find items where the temperature is greater than 30 degrees Celsius.

var scanItems = await _dynamoDbContext.ScanAsync<WeatherForecast>(
    new[]
    {
        new ScanCondition(nameof(WeatherForecast.TemperatureC), ScanOperator.GreaterThan, 30)
    }).GetRemainingAsync();

When running this, it loops through all the items in the DynamoDB and checks if the TemperatureC property is greater than the given value. If you have a database with a large number of items, this is going to be a slow and costly operation.

I hope you now can query data from DynamoDB without having to scan through it. If not, we will see how you could use Indexes to solve and make your use case faster in a future article.

Add to Apple Wallet from Your .NET Application: A Step-by-Step Guide

Rahul Nath — Thu, 08 Feb 2024 04:48:49 +0000

I recently worked on the 'Add to Apple Wallet' functionality at work, allowing the addition of Event tickets to Apple Wallet.

The iOS Wallet app allows users to manage payment cards, boarding passes, tickets, gift cards, and other passes.

The Wallet Pass is time and location enabled, so passes can be configured to display on the user’s device at the appropriate moment. Passes can also be updated with push notifications making it easy to notify users if details change.

In this blog post, let's learn how you can set up, build, and distribute Apple Wallet passes using .NET application.

I will use an AWS Lambda Function to host the API endpoint for distributing the pass files. However, you can use your existing application hosting mechanism for this.

Apple Wallet Passes

Apple Wallet Passes are created as a package/bundle, containing a pass.json file that defines the pass, and image assets such as the logo and the icon.

The pass.json file contains the information that identifies the pass, the text information, and details shown on the pass in the Wallet app.

The below image shows the files inside a sample pass file. It has the pass.json, logo and icon images, the manifest file, and also the signature file to avoid tampering with the pass file once it's generated.

Imaging showing the files inside a pass file. It contains the pass.json file, along with the icon and logo images, the manifest file and the signature file for the pass contents.

Generating Wallet Passes Using .NET

To generate Apple Wallet Pass files from .NET applications, you can use the dotnet-passbook NuGet package.

The NuGet package makes it very easy and straightforward to create Wallet Pass files. The NuGet package provides a PassGenerator class that takes in one or more instances of PassGeneratorRequest.

The PassGenerator converts these requests into a byte array, which represents the pass bundle that contains one or more pass files.

public async Task<APIGatewayHttpApiV2ProxyResponse> GetAppleWalletPasses()
{
    var eventId = Guid.NewGuid();
    var eventName = "YouTube Demo Event";
    var venueName = "YouTube Online";
    var eventDate = DateTime.Now.AddDays(55);

    var icon = await _appleWalletConfiguration.GetIcon();
    var logo = await _appleWalletConfiguration.GetLogo();

    var request = GeneratePassRequest(Guid.NewGuid().ToString(), eventId, icon, logo, eventName, eventDate, venueName, "Subscriber");

    var generator = new PassGenerator();
    var requests = new List<PassGeneratorRequest>() { request };
    var pass = generator.Generate(requests);
}

The PassGeneratorRequest identifies one pass that will be added to the Apple Wallet. If you have multiple passes to be added in the same bundle, create multiple PassGeneratorRequest.

The below code shows a sample PassGeneratorRequest that adds a pass with a EventTicket style. It sets the relevant properties to be shown on the Apple Wallet Pass.

var request = new PassGeneratorRequest
{
    Style = PassStyle.EventTicket,
    PassTypeIdentifier = _appleWalletConfiguration.PassTypeIdentifier,
    SerialNumber = serialNumber,
    GroupingIdentifier = eventId.ToString(),
    BackgroundColor = "#823EB7",
    LabelColor = "#000000",
    ForegroundColor = "#ffffff",
    Images =
    {
        {PassbookImage.Icon, icon},
        {PassbookImage.Icon2X, icon},
        {PassbookImage.Icon3X, icon},
        {PassbookImage.Logo, logo},
        {PassbookImage.Logo2X, logo},
        {PassbookImage.Logo3X, logo},
    },
    Description = eventName,
    OrganizationName = "Rahul",
    RelevantDate = eventDate,
    ExpirationDate = eventDate.AddDays(1),
    AppleWWDRCACertificate = _appleWalletConfiguration.AppleWWDRCACertificate(),
    PassbookCertificate = _appleWalletConfiguration.PassbookCertificate()
};
request.AddHeaderField(new StandardField("time", eventDate.ToShortTimeString(),
    eventDate.ToShortDateString()));
request.AddPrimaryField(new StandardField("name", null, eventName));
request.AddSecondaryField(new StandardField("venue", "Venue", venueName));
request.AddAuxiliaryField(new StandardField("ticketType", "Ticket Type", ticketType));
request.AddBackField(new StandardField("ticketHolderName-back", "Ticket holder", "Rahul Nath"));
request.AddBackField(new StandardField("ticketType-back", "Ticket Type", ticketType));

For information on the different fields, the pass styles available, and what each property represents check out the Wallet Guidelines and the documentation.

Setting Up Apple Certificates

To generate Apple Wallet Passes, you need to register with the Apple Developer Portal and have the appropriate certificates ready to sign the Wallet passes.

You need two certificates

Your application/company Passbook certificate along with its password
Apple WWDR (WorldWide Developer Relations) certificate

You can find the full instructions to generate the certificates in the dotnet-passbook NuGet package documentation here.

You can store these certificates and other related configurations for generating the Wallet as part of the application configuration - appsettings.json file*.*

I chose to store the certificate files as base64 encoded string which can be converted to a X509Certificate2 in .NET code as shown below.

public class AppleWalletConfiguration
{
    public string WWDRCertificateBase64 { get; set; }
    public string PassTypeIdentifier { get; set; }
    public string PassbookCertificateBase64 { get; set; }
    public string PassbookPassword { get; set; }
    public string IconUrl { get; set; }
    public string LogoUrl { get; set; }

    public X509Certificate2 AppleWWDRCACertificate() =>
        new(Convert.FromBase64String(WWDRCertificateBase64));

    public X509Certificate2 PassbookCertificate() =>
        new(Convert.FromBase64String(PassbookCertificateBase64), PassbookPassword);

    public async Task<byte[]> GetLogo() => await LogoUrl.GetBytesAsync();

    public async Task<byte[]> GetIcon() => await IconUrl.GetBytesAsync();
}

Distributing Wallet Pass Files Using .NET

To distribute the Wallet Pass Files generated, you can host this in an API endpoint and share the link via email, a button on a web page, or an app.

You can host this along with your other application code on the Cloud or your own servers.

For this post, I'll host this AWS Lambda, as it's a quick and easy way to host such kind of API endpoints. Using the latest Lambda Annotations Framework makes it even easier to create and host API endpoints.

%[https://www.rahulpnath.com/blog/lambda-annotation-framework-crud-api/]

All we need to do is add the Lambda Annotation Framework NuGet package and wrap our Pass Generation function with the LambdaFunction attribute. Since this is going to be exposed as an API will also add the HttpApi attribute and specify the route and the method Get.

[LambdaFunction(Policies = "AWSLambdaBasicExecutionRole")]
[HttpApi(LambdaHttpMethod.Get, "/apple-wallet")]
public async Task<APIGatewayHttpApiV2ProxyResponse> GetAppleWalletPasses()
{
    var eventId = Guid.NewGuid();
    var eventName = "YouTube Demo Event";
    ...
    var generator = new PassGenerator();
    var requests = new List<PassGeneratorRequest>() { request };
    var pass = generator.Generate(requests);

    return new APIGatewayHttpApiV2ProxyResponse()
    {
        Body = Convert.ToBase64String(pass),
        IsBase64Encoded = true,
        StatusCode = 200,
        Headers = new Dictionary<string, string>
        {
            { "Content-Type", "application/vnd.apple.pkpasses" },
            { "Content-Disposition", "attachment; filename=tickets.pkpasses.zip; filename*=UTF-8''tickets.pkpasses.zip" }
        }
    };
}

The HttpApi attribute allows us to expose the Lambda Function over a Lambda Function URL or an HTTP API Gateway endpoint.

Both of these mechanisms expect binary content to be encoded as base64 string, with the appropriate HTTP headers set. To achieve this we can use the APIGatewayHttpApiV2ProxyResponse object as shown above.

Once deployed, you can navigate the API Gateway URL endpoint or expose a Lambda Function URL, and navigate to the URL from an iPhone Safari browser, to add it to Apple Wallet.

Below is the screenshot of the pass generated using the above code, with the appropriate information we passed to create the pass.

Apple Wallet Passes added to phone showing the front and the back details of the pass.

For debugging pass files you can use the PKPassValidator service, where you can upload a pass file and it will show any issues with the file generated.

The Apple Wallet pass generated above is static and does not update itself.

However most of the time you would need to deliver live updates to these passes in case any information changes for example date changes, venue changes, status updates, etc. For this, we can configure the WebServiceUrl, on the pass that it will poll for pass updates

We will learn how to configure and set up the CallbackUrl in a future blog post. Until then, happy coding! 👋

The Quickest Way To Build Your APIs In AWS Lambda Functions Using .NET

Rahul Nath — Tue, 06 Feb 2024 05:16:42 +0000

The Lambda Annotations Framework is a programming model that makes it easier to build AWS Lambda Functions using .NET.

The framework uses C# custom attributes and Source Generators to translate annotated Lambda functions to the regular Lambda programming model.

In a previous post, we learned how to get started with using the Annotations Framework and how it compares to using the default Lambda Programming model. The Annotations Framework removes boilerplate code required to integrate with API Gateway.

In this post, let’s learn how to set up a basic CRUD (Create, Read, Update, and Delete) Order API Endpoint using the Lambda Annotation Framework.

We will use DynamoDB as our data store for this example and we will also learn how to set up the DynamoDB Table and Role Permissions via the same CloudFormation template that the Annotations framework generates.

Create and Update API Endpoint

Let's first create the Create Endpoint for our API.

Using the Annotations Framework, we can use the LambdaFunction and HttpApi attribute to decorate a .NET function to be a Lambda Entry point.

The endpoints accept an object of type Order which represents an Order in our domain.

We have a simple Order class with the OrderId as it's primary key.

To bind the Order item from the requests body, we can use the FromBody attribute on the function parameter. This is very similar to how we do when building ASP.NET Core APIs.

⚠️ Make sure the template path does not have a leading slash on it.

We also have specified a custom IAM Role Name as part of the Role parameter in the LambdaFunction attribute. More on this later.

[LambdaFunction(Role = "@OrdersApiLambdaExecutionRole")]
[HttpApi(LambdaHttpMethod.Post, template: "/orders")]
public async Task PostOrder([FromBody] Order order, ILambdaContext context)
{
    await _dynamodbContext.SaveAsync(order);
}

public class Order
{
    public string OrderId { get; set; }
    public decimal Total { get; set; }
    public DateTime CreatedDate { get; set; }
}

Using the DynamoDBContext we can save the Order into the table. If you are new to DynamoDB I highly recommend checking the below blog post to get started.

Amazon DynamoDB For The .NET Developer

For the DynamoDBContext save to work, we need to make sure the Table is set up and the Lambda Function has appropriate access to the table.

Setting up DynamoDB Table and IAM Roles via CloudFormation

As part of the Lambda Annotations Framework, it auto-generates a serverless.template CloudFormation template file.

However, you can manually add Resources to the template file.

In our case, we need to set up the DynamoDB Table and the Lambda Access Roles with the appropriate permission to talk to DynamoDB Table.

To set up the Order DynamoDB table, let's add the below to the Resources section of the template.

"OrderTable": {
  "Type": "AWS::DynamoDB::Table",
  "Properties": {
    "AttributeDefinitions": [
      {
        "AttributeName": "OrderId",
        "AttributeType": "S"
      }
    ],
    "KeySchema": [
      {
        "AttributeName": "OrderId",
        "KeyType": "HASH"
      }
    ],
    "TableName": "Order",
    "BillingMode": "PAY_PER_REQUEST"
  }
}

It sets up the DynamoDB table with a specific name and also the Keys for the table.

For setting up the Lambda IAM Role, let's use the below template in the Resources section.

It sets up full access to DynamoDB Order Table. Note I am using the CloudFormation function Fn::GetAtt to get the ARN of the DynamoDB Table created in the same template file.

"OrdersApiLambdaExecutionRole": {
  "Type": "AWS::IAM::Role",
  "Properties": {
    "AssumeRolePolicyDocument": {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Service": "lambda.amazonaws.com"
          },
          "Action": "sts:AssumeRole"
        }
      ]
    },
    "ManagedPolicyArns": [
      "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
    ],
    "Policies": [
      {
        "PolicyName": "OrderApiDynamoDBAccessPolicy",
        "PolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Action": "dynamodb:*",
              "Resource": {
                "Fn::GetAtt": ["OrderTable", "Arn"]
              }
            }
          ]
        }
      }
    ]
  }
}

The LambdaFunction attribute specifies the Role property to use the new IAM role that we just created above.

We use the name Role = "@OrdersApiLambdaExecutionRole" to refer to the IAM role created in the template file. The @ prefix refers to look up for the name from the CloudFormation template.

This generates the below section to the CloudFormation template for the Lambda Function

"LambdaAnnotationSampleOrderApiFunctionPostOrderGenerated": {
  "Type": "AWS::Serverless::Function",
  "Metadata": {
    "Tool": "Amazon.Lambda.Annotations",
    "SyncedEvents": [
      "RootPost"
    ]
  },
  "Properties": {
    "Runtime": "dotnet6",
    "CodeUri": ".",
    "MemorySize": 256,
    "Timeout": 30,
    "Role": {
      "Fn::GetAtt": [
        "OrdersApiLambdaExecutionRole",
        "Arn"
      ]
    },
    "PackageType": "Zip",
    "Handler": "LambdaAnnotationSample.OrderApi::LambdaAnnotationSample.OrderApi.Function_PostOrder_Generated::PostOrder",
    "Events": {
      "RootPost": {
        "Type": "HttpApi",
        "Properties": {
          "Path": "/order",
          "Method": "POST"
        }
      }
    }
  }
}

You can deploy the template from the IDE or using the AWS CLI, which will create the Lambda Function, the IAM Role, the API Endpoint and wire up the integration for the route to the Lambda Function on the API Gateway.

Read API Endpoint

To set up the Read API Endpoint, all we need to do is create another Function in the same class and add the LambdaFunction and HttpApi attribute. Since this is a GET, it specifies the appropriate LambdaHttpMethod on it.

[LambdaFunction(Role = "@OrdersApiLambdaExecutionRole")]
[HttpApi(LambdaHttpMethod.Get, "/orders/{orderId}")]
public async Task<Order> GetOrder(string orderId, ILambdaContext context)
{
    return await _dynamodbContext.LoadAsync<Order>(orderId);
}

The function takes in the orderId as the parameter, which is bound from the path parameter in the request URL.

You can use the DynamoDBContext to retrieve the item from the Table and return the Order item.

Lambda Annotations will automatically convert this into JSON representation and send it as part of the HTTP response body.

Delete API Endpoint

Very similar to the Get API Endpoint, we can add a new Function for the Delete endpoint.

[LambdaFunction(Role = "@OrdersApiLambdaExecutionRole")]
[HttpApi(LambdaHttpMethod.Delete, "/orders/{orderId}")]
public async Task DeleteOrder(string orderId, ILambdaContext context)
{
    await _dynamodbContext.DeleteAsync<Order>(orderId);
}

It takes the orderId from the Path parameter and uses that to delete the item from the DynamoDB table.

Annotations Framework Generated Code

The Lambda Annotations Framework uses .NET Source Generators to generate the boilerplate code to interact with the API Gateway.

Below is a screenshot of the DLL from JetBrains dotPeek application, showing the generated code.

Lambda Annotations Framework generated code using Source Generator.

As you can see, for each of the functions attributed with LambdaFunction it creates a new class, that wraps around the original function and adds the boilerplate code to talk with API Gateway.

This makes it easier for us developers to focus just on the business logic and not worry about how to integrate with the API Gateway.

You can find the full code sample here.

Amazon MQ RabbitMQ: A Reliable Messaging Solution for Your .NET Projects

Rahul Nath — Mon, 05 Feb 2024 07:19:54 +0000

RabbitMQ is a powerful open-source message broker facilitating communication between systems or applications.

It ensures seamless data exchange by enabling asynchronous messaging, making it an essential tool for building scalable and resilient distributed systems.

In this post, we will learn how to set up RabbitMQ and start using it from a .NET application to send and receive messages.

Setting Up RabbitMQ

RabbitMQ can be hosted through self-hosted deployments, major cloud providers, containerization with Docker and Kubernetes, managed RabbitMQ providers, etc, offering a wide range of options from complete control to fully managed solutions.

For this post, I will use Amazon MQ, a managed message broker service that supports ActiveMQ and RabbitMQ engine types.

However, you can use one of the various options that RabbitMQ provides to host your instance, including a local Docker instance.

Creating RabbitMQ broker on Amazon MQ

Amazon MQ is a fully managed message broker service on AWS that supports popular messaging protocols, providing compatibility, reliability, scalability, and security for distributed applications.

Amazon MQ supports Apache ActiveMQ and RabbitMQ engine types.

To create a RabbitMQ message broker, navigate to Amazon MQ in the AWS Console.

Create a new broker and choose RabbitMQ as the broker engine type, as shown below.

Create a new Amazon MQ broker using the RabbitMQ broker type.

It then prompts you to choose the deployment mode.

Single-instance broker → one broker in one Availability Zone, mainly used for development and testing.
Cluster deployment → logical grouping of three RabbitMQ broker nodes behind a Network Load Balancer, each sharing users, queues, and a distributed state across multiple Availability Zones

Choose the deployment mode for Amazon MQ Rabbit broker instance - choose between single instance and cluster.

For purposes of this blog post, I am using a Single instance. However, choose Cluster deployment if you are setting up a production workload.

Once done you can configure the broker name, the instance type, username/password and other advanced configuration required.

Configure RabbitMQ setting in Amazon MQ.

Review and create the RabbitMQ broker instance.

RabbitMQ broker instance up and running and the details as from the AWS console.

Creating .NET Application To Send/Receive Messages From RabbitMQ

RabbitMQ is a message broker.

This means you would typically have a sender/producer and a receiver/consumer when building applications on it. The sender sends messages, and the receiver receives and processes them.

To start using the RabbitMQ broker instance set up above, let's first create two console applications - Send and Receive.

To connect and send/receive messages from RabbitMQ, we will use a NuGet package - RabbitMQ.Client.

Below is a Powershell script to create the Send and Receive .NET console applications and add the required NuGet packages.

It creates a solution file, creates a Send console app with the start-up file as Send.cs , adds the NuGet package. It also creates a Receive console app with the start-up file Receive.cs and the required packages.

# Step 1: Create a new .NET solution
dotnet new sln -n rabbitmq-hello-world

# Step 2: Create the 'Send' console application
dotnet new console -n Send
dotnet add Send/Send.csproj package RabbitMQ.Client
dotnet sln rabbitmq-hello-world.sln add Send/Send.csproj
mv Send/Program.cs Send/Send.cs

# Step 3: Create the 'Receive' console application
dotnet new console -n Receive
dotnet add Receive/Receive.csproj package RabbitMQ.Client
dotnet sln rabbitmq-hello-world.sln add Receive/Receive.csproj
mv Receive/Program.cs Receive/Receive.cs

Core Connection Concepts With RabbitMQ

The two core concepts that you need to know in terms of connecting with RabbitMQ are

Connection: This represents the TCP connection between your application and the RabbitMQ broker.
Channels: A virtual connection within a connection*.* Used for specific tasks like publishing, consuming, and declaring queues and exchanges. Channels help organize and optimize communication between your application and the broker.

Since it's undesirable to have multiple TCP connections open at the same time from our applications to RabbitMQ, we can create one Connection and then use Channels to open lightweight connections with the broker.

Channels in RabbitMQ support efficient multiplexing, isolation, resource management, concurrency, and performance optimization while still sharing the same underlying TCP connection.

Connecting .NET Application with RabbitMQ

To connect to RabbitMQ, let's use the ConnectionFactory from the RabbitMQ.Client NuGet package to create a new TCP Connection.

var factory = new ConnectionFactory
{
    Uri = new Uri("amqps://b-71e1955a-1942-47e2-9e49-xxxxxxxxxxxx.mq.ap-southeast-2.amazonaws.com"),
    Port = 5671,
    UserName = "<USERNAME FROM CONFIGURATION FILE>",
    Password = "<PASSWORD FROM CONFIGURATION FILE>"
};
using var connection = factory.CreateConnection();
using var channel = connection.CreateModel();

The above code specifies the Uri to connect to the RabbitMQ instance in Amazon MQ and specify the username/password we used when configuring the RabbitMQ broker instance.

Using the Connection, it creates a Channel (referred to as Model in .NET speak), using the CreateModel function.

The Channel instance can then be used to send or receive messages from the RabbitMQ instance.

Creating a Queue in RabbitMQ From .NET Application

We first need to create a Queue to send or receive messages from RabbitMQ.

A Queue in RabbitMQ is a named message buffer that temporarily stores messages sent by producers and delivers them to consumers.

channel.QueueDeclare(queue: "hello",
    durable: false,
    exclusive: false,
    autoDelete: false,
    arguments: null);

The above code uses the QueueDeclare function on the channel instance to create a simple queue.

Creating a Queue is an idempotent operation.

It means a new Queue is created only if it does not already exist. So it's safe to have this same code in both our Send/Receive applications since we cannot guarantee which one will run first, and we need the Queue to exist regardless of the order in which they are run.

We can now use this Queue to send/receive messages from our .NET application.

Sending Messages to RabbitMQ From .NET Application

To send a message to RabbitMQ, we can use the BasicPublish method on the channel instance.

Below we get the message text to send from the console and send it as the message body.

string? message = null;
do
{
    Console.WriteLine("Enter Message. Press [enter] to exit.");
    message = Console.ReadLine();
    if (!string.IsNullOrEmpty(message))
        SendMessage(message, channel);
} while (!string.IsNullOrEmpty(message));

void SendMessage(string s, IModel channel)
{
    var body = Encoding.UTF8.GetBytes(s);

    channel.BasicPublish(exchange: string.Empty,
        routingKey: "hello",
        basicProperties: null,
        body: body);
    Console.WriteLine($" [x] Sent {s}");
}

Note that we specify the exchange as empty and routingKey as the queue name. This uses the default exchange that's available as part of RabbitMQ.

In a future blog post, we will learn more about Exchanges and the different types.

Receive Messages From RabbitMQ From .NET Application

With the message published successfully we are now ready to add in our first consumer/receiver for this message.

The channel instance provides a BasicConsumer method to register a consumer instance to a queue.

Console.WriteLine(" [*] Waiting for messages.");

var consumer = new EventingBasicConsumer(channel);
consumer.Received += (model, ea) =>
{
    var body = ea.Body.ToArray();
    var message = Encoding.UTF8.GetString(body);
    Console.WriteLine($" [x] Received {message}");
};
channel.BasicConsume(queue: "hello",
    autoAck: true,
    consumer: consumer);

Console.WriteLine(" Press [enter] to exit.");
Console.ReadLine();

The RabbitMQ.Client NuGet package provides a EventingBasicConsumer class we can use to handle the message processing.

The above code sets up the Received event handler with the function to process the message, in this case, writing the message body to the console.

Send and Receive applications in action, working with Amazon MQ RabbitMQ.

Send > 'Have a great day 😀'

How To Easily Make Your .NET AWS Lambda Function Idempotent

Rahul Nath — Fri, 02 Feb 2024 00:45:31 +0000

Idempotency refers to the property of a function that produces the same result regardless of how many times it is executed with the same input parameters.

Idempotency is particularly important in distributed systems, where messages may be retried due to failures, timeouts, or other issues. Idempotency ensures the system remains consistent without unintended side effects, even if a request is duplicated or repeated.

The Powertools Idempotency utility makes it easy to convert your Lambda functions into idempotent operations that are safe to retry.

In this blog post, let's learn how to get started using the Powertools Idempotency package, some key features, and how it easily fits into your existing Lambda Functions.

.NET Lambda Function and Powertools Idempotency Package

The Lambda Powertools Idempotency utility prevents the function handler from executing more than once for the same event payload during a specified period while ensuring it returns the same result.

The utility package is highly customizable, making it very easy to be tuned to work for your application-specific needs.

Setting Up Lambda Function Idempotency

To start using the Idempotency utility, let's first install the NuGet package - AWS.Lambda.Powertools.Idempotency

Once installed, we only need to add the Idempotent attribute to our Lambda Function handler endpoint, as shown below.

[LambdaFunction(Role = "@WeatherApiLambdaExecutionRole")]
[HttpApi(LambdaHttpMethod.Post, "/weather-forecast")]
[Idempotent]
public async Task AddWeatherData([FromBody] WeatherForecast weatherForecast)
{
    Console.WriteLine("Running time consuming process");
    await Task.Delay(TimeSpan.FromSeconds(5));
    await dynamoDBContext.SaveAsync(weatherForecast);
}

The Lambda Function sample uses Lambda Annotations Framework , which makes it extremely easy to build Lambda Functions in .NET.

The function is an API endpoint that takes in a WeatherForecast data from the body of the API Gateway API request object.

In the Function constructor, you can initialize the Idempotency configuration. The code below sets up the package to use the DynamoDB table IdempotencyTable , as it's storage mechanism to maintain request/response information.

public Functions(IDynamoDBContext dynamoDBContext)
{
    this.dynamoDBContext = dynamoDBContext;
    Idempotency.Configure(builder => builder.UseDynamoDb("IdempotencyTable"));
}

Ensure that the Lambda Function has appropriate rights to talk to the IdempotencyTable.

When making requests against the API endpoint, it creates a hash value based on the request object (in this case, the WeatherForecast object) and stores it in the DynamoDB table.

If the same request is made, it will find a matching item in the DynamoDB table with the hash key and return the recorded response from the earlier successful request.

The default timeout period is 1 hour, which means any request with the same payload during that period will get the recorded response. After the timeout, the record is deleted from the DynamoDB table, and the first new request with the same payload will invoke the Lambda Function again.

Idempotency DynamoDB Table with records populated as we make requests in the API endpoint.

Since the sample function above does not return any value, the data property in the DynamoDB table is empty.

Customizing Idempotency Key

The Idempotency package automatically used the first parameter In the above function, it automatically used the first parameter of the Lambda Function as the Idempotency key.

If there is more than one parameter for the Lambda Function, you can explicitly mark the item to use as the Idempotency Key using the IdempotencyKey attribute.

[LambdaFunction(Role = "@WeatherApiLambdaExecutionRole")]
[HttpApi(LambdaHttpMethod.Post, "/weather-forecast")]
[Idempotent]
public async Task AddWeatherData(APIGatewayHttpApiV2ProxyRequest request, [IdempotencyKey][FromBody] WeatherForecast weatherForecast)
{
    Console.WriteLine("Running time consuming process");
    await Task.Delay(TimeSpan.FromSeconds(5));
    await dynamoDBContext.SaveAsync(weatherForecast);
}

You can further customize the IdempotencyKey to use specific properties from the object payload.

This helps to ensure that the idempotency key is not affected by properties in the request object that might change over multiple requests (e.g., HTTP headers, etc.).

The code below specifies the CityName and Date properties to be used for the Idempotency key, ignoring all other properties on the request object (WeatherForecast).

public Functions(IDynamoDBContext dynamoDBContext)
{
    this.dynamoDBContext = dynamoDBContext;
    Idempotency.Configure(builder =>
        builder
        .WithOptions(optionsBuilder => 
              optionsBuilder.WithEventKeyJmesPath("[CityName, Date]"))
        .UseDynamoDb("IdempotencyTable"));
}

Any requests with the same CityName and Date property will be treated as an idempotent request during the timeout period.

Adding Payload Validation for Idempotency

Now, in the case of the WeatherForecast payload, what if the Temperature amount is different between two consecutive requests?

Since we chose not to include that as part of the Idempotency key, it will respond with the response from the initial request for any subsequent request.

In these cases, you can either add the Temperature property to the Idempotency key or add in request validation to ensure that the Temperature property has the same value as the initial request.

The initialization code below sets up the property validationTemperatureC using the WithPayloadValidationJmesPath function.

public Functions(IDynamoDBContext dynamoDBContext)
{
    this.dynamoDBContext = dynamoDBContext;
    Idempotency.Configure(builder =>
        builder
        .WithOptions(optionsBuilder => optionsBuilder
            .WithEventKeyJmesPath("[CityName, Date]")
            .WithPayloadValidationJmesPath("TemperatureC"))
        .UseDynamoDb("IdempotencyTable"));
}

Any subsequent requests with the same CityName and Date property, with a different TemperatureC property will throw an IdempotencyValidationException.

I hope this helps you to get started using the Idempotency Lambda package.