Forem: Donovan HOANG

SNS vs. SQS vs. EventBridge: Choosing the Right AWS Messaging Service

Donovan HOANG — Mon, 23 Dec 2024 14:30:00 +0000

TL; DR.

Messaging and events management in AWS allow for building scalable and decoupled applications. AWS offers three key messaging services:

Amazon SNS: Best for real-time notifications and broadcasting messages to multiple subscribers with high throughput.
Amazon SQS: Ideal for decoupling components and enabling reliable message queuing for asynchronous processing.
Amazon EventBridge: Perfect for creating complex event-driven architectures by connecting events from multiple sources, including SaaS applications.

Choosing the right service depends on your specific application needs—whether for notifications, queuing, or event orchestration.

The usage of messaging and events management allows our application to decouple and scale in a very efficient way. Messaging solutions play a major role in enabling these capabilities by providing communication between different components of an infrastructure. AWS offers several powerful messaging services, each designed to address specific needs and use cases: Amazon Simple Notification Service (SNS), Amazon Simple Queue Service (SQS), and Amazon EventBridge.

To take full benefits of the messages-based services of AWS, you need to choose the one that fits the best your context. Whether you need real-time notifications, reliable message queuing, or complex event-driven workflows, understanding the features and use cases of these services is essential.

This article aims to help you leverage AWS messaging services to build efficient applications in the cloud. We will explore all the features and typical use cases to compare the pros and cons of each service.

Understanding the AWS Messaging Services

Amazon Simple Notification Service (SNS): SNS is a fully managed publish/subscribe (pub/sub) messaging service that simplifies the decoupling and scaling of microservices, distributed systems, and serverless applications. It supports high-throughput message broadcasting to multiple subscribers via diverse delivery mechanisms, including HTTP/HTTPS endpoints, email, SMS, and AWS Lambda. SNS also offers seamless integration with other AWS services, enabling efficient notification and alerting workflows.
Amazon Simple Queue Service (SQS): SQS is a fully managed message queuing service designed to decouple application components, enhancing reliability and scalability. It supports two queue types: standard queues, which ensure at-least-once delivery and allow best-effort message ordering, and FIFO queues, which guarantee exactly-once processing and strict message order. SQS facilitates durable message retention, configurable visibility timeouts, and batch processing capabilities, making it ideal for asynchronous task handling. SQS also supports Dead-Letter Queues (DLQ), which allow for handling messages that can’t be processed successfully, enhancing fault tolerance in production systems.
Amazon EventBridge: EventBridge is a serverless event bus service tailored for event-driven architectures. It enables seamless connection and routing of events from AWS services, custom applications, and third-party SaaS providers to designated targets. EventBridge features advanced filtering, transformation options, and a schema registry to simplify event discovery and management. Its built-in scalability and support for complex workflows make it an excellent choice for modern, event-driven applications.

Key Features

Amazon SNS

Publish/Subscribe Model: SNS uses a pub/sub messaging model to send messages to multiple subscribers.
Message Delivery: Delivery mechanisms include HTTP/HTTPS, email, SMS, and Lambda functions.
Fan-out Architecture: Enables the ability to fan out messages to multiple endpoints simultaneously.

Amazon SQS

Queue-Based Messaging: SQS uses a queue-based system to store messages until they are processed.
Types of Queues: Includes standard queues (at-least-once delivery, best-effort ordering) and FIFO queues (exactly-once processing, ordered delivery).
Message Retention and Visibility Timeout: Concepts include the message retention period and visibility timeout.
Dead-Letter Queues: Allows for handling messages that cannot be processed successfully, enhancing reliability in production environments.

Amazon EventBridge

Event Bus Model: EventBridge uses an event bus to route events from different sources to target AWS services.
Schema Registry and discovery: Features a schema registry that allows you to discover, create, and manage event schemas.
Integrated SaaS Applications: Supports integration of events from SaaS applications directly into EventBridge.

Use Cases

Amazon SNS

Real-Time Notifications: Examples include sending real-time notifications to users via SMS, email, or mobile push notifications.
Fan-Out: Scenarios where the same message needs to be sent to multiple endpoints, such as updating multiple microservices.

Amazon SQS

Decoupling Microservices: Examples include decoupling application components to improve fault tolerance and scalability.
Batch Processing: Scenarios such as processing orders or logs where messages need to be processed in batches.

Amazon EventBridge

Event-Driven Architectures: Examples include building event-driven applications that react to changes in data or state.
SaaS Integration: Scenarios involving integrating and routing events from third-party SaaS applications to AWS services.

Integration Examples

E-commerce Application: Use SQS to queue orders for asynchronous processing, ensuring scalability and fault tolerance, while SNS can be used to send shipping notifications to customers via email or SMS.
Fan-Out Pattern: A common pattern is to use an SNS topic to fan out messages to multiple SQS queues. This allows parallel processing of the same event by different consumers, enabling flexible and scalable system architectures.

Cost Considerations

Service	Pricing Model
Amazon SNS	Charged based on the number of published messages and the protocol used for delivery (e.g., HTTP/S, SMS, email).
Amazon SQS	Charged based on the number of requests and message size. Standard queue: $0.40 per 1 million requests; FIFO queue: $0.50 per 1 million requests. Additional charges apply for long polling.
Amazon EventBridge	Charged based on the number of events published and the number of rules applied. $1.00 per 1 million events published. Other features incur other charges.

Understanding these cost models allows for better optimization of AWS expenses by aligning the service choice with architectural needs and budget constraints.

Understanding these cost models can help optimize overall AWS costs by selecting the right service and architecture for your needs.

Comparison

Feature	Amazon SNS	Amazon SQS	Amazon EventBridge
Delivery Model	Pub/Sub	Queue-based	Event bus
Use Case	Real-time notifications, fan-out	Decoupling systems, batching	Event-driven architectures
Scalability	High throughput	High throughput	High scalability
Message Ordering	Not guaranteed	FIFO queues provide ordering	Not guaranteed
Integration	Multi-protocol endpoints	Queue-based processing	SaaS and AWS service integration
Complexity Handling	Low	Medium	High

Decision help

When to Use SNS: Best for scenarios requiring real-time notifications and message broadcasting to multiple endpoints.
When to Use SQS: Ideal for decoupling components, ensuring reliable message processing, and handling message queues in fault-tolerant systems.
When to Use EventBridge: Optimal for building event-driven architectures that integrate multiple event sources, including SaaS applications, and require complex workflows.

Thanks for reading! Hope this helped you to understand how to take advantage of AWS messaging services.

Introducing fck-nat: Cost optimized alternative to AWS NAT Gateways

Donovan HOANG — Sun, 22 Dec 2024 21:21:32 +0000

TL;DR;

fck-nat is an open-source, cost-effective alternative to AWS Managed NAT Gateway. It dramatically reduces costs, eliminates data transfer charges, and supports high-bandwidth traffic. Built on Amazon Linux 2023, fck-nat offers features like high availability, custom monitoring with CloudWatch, and flexible deployment options using Terraform, CDK, or CloudFormation. Ideal for cost-sensitive setups and development environments, fck-nat has some limitations in production.

Managing Network Address Translation (NAT) in AWS can be expensive and tricky, but that's where fck-nat comes in. It's a simple, affordable, and customizable alternative to AWS Managed NAT Gateway that saves you money while keeping things running smoothly.

What is fck-nat?

fck-nat is an open-source project providing a modern, cost-effective alternative to AWS Managed NAT Gateway. Built on Amazon Linux 2023 and available as a pre-configured Amazon Machine Image (AMI), fck-nat supports high-bandwidth traffic at significantly lower costs than AWS's proprietary solutions. It also offers flexibility in deployment, including support for high-availability (HA) configurations, static IPs, and custom monitoring with CloudWatch.

AndrewGuenther / fck-nat

Feasible cost konfigurable NAT: An AWS NAT Instance AMI

fck-nat

Welcome to fck-nat. The (f)easible (c)ost (k)onfigurable NAT!

Overpaying for AWS Managed NAT Gateways? fck-nat.
Want to use NAT instances and stay up-to-date with the latest security patches? fck-nat.
Want to reuse your Bastion hosts as a NAT? fck-nat.

Full documentation is available at https://fck-nat.dev

fck-nat offers a ready-to-use ARM and x86 based AMIs built on Amazon Linux 2023 which can support up to 5Gbps burst NAT traffic on a t4g.nano instance. How does that compare to a Managed NAT Gateway?

Hourly rates:

Managed NAT Gateway hourly: $0.045
t4g.nano hourly: $0.0042

Per GB rates:

Managed NAT Gateway per GB: $0.045
fck-nat per GB: $0.00

Sitting idle, fck-nat costs 10% of a Managed NAT Gateway. In practice, the savings are even greater.

"But what about AWS' NAT Instance AMI?"

The official AWS supported NAT Instance AMI hasn't been updates since 2018, is still running Amazon Linux 1 which is now…

View on GitHub

Key Features

Cost Efficiency

Low Costs: Operating on a t4g.nano instance, fck-nat costs approximately $3.06/month compared to AWS Managed NAT Gateway's $32/month.
No Data Transfer Costs: Unlike Managed NAT Gateway, fck-nat does not charge for per-GB data transfer.

High Performance

Performance Tuning Tips: To maximize fck-nat's performance, ensure you choose the right EC2 instance type based on your traffic needs.
Scalable Bandwidth: Supports up to 5 Gbps egress on smaller instances and higher capacities on larger instances with over 32 vCPUs.
Optimized Configurations: Uses advanced routing and security configurations to maximize throughput.

Flexibility and Control

Custom Deployments: Supports deployment via CDK, Terraform, and CloudFormation.
High-Availability Mode: Automatically replaces instances in case of failure while maintaining consistent IP addresses.

Monitoring and Security

CloudWatch Integration: Provides detailed metrics for monitoring NAT traffic.
Enhanced Security: Leverages security groups and supports custom firewall rules using iptables.

Sustainability

ARM-based Instances: Optimized for ARM architecture to reduce costs and energy consumption.

Benefits of Using fck-nat

Cost Savings: fck-nat offers a highly affordable solution for managing NAT. Unlike AWS Managed NAT Gateway, which incurs steep costs for both usage and data transfer, fck-nat eliminates per-GB transfer charges and operates on low-cost instances, saving you hundreds of dollars monthly.

Customizability: You can adapt fck-nat to fit your requirements. Whether you need custom routing rules, specific security configurations, or tailored deployment strategies, fck-nat offers the flexibility to align with your infrastructure.

Simplicity: Deploying fck-nat is straightforward, even for users with minimal experience. Comprehensive guides and examples make the setup process quick and easy, ensuring you can get up and running without unnecessary complexity.

Reliability: High availability is built into fck-nat's design. With support for automatic failover and redundant instances, it guarantees consistent performance and minimal downtime, keeping your network running smoothly.

Use Cases

Enterprise Cost Optimization

Replace AWS Managed NAT Gateways in your Virtual Private Clouds (VPCs) to significantly cut costs. fck-nat maintains excellent performance, making it a perfect option for businesses aiming to optimize cloud expenses.

Hybrid Architectures

Seamlessly integrate fck-nat with private subnets in hybrid cloud setups. This ensures efficient management of egress traffic without the steep costs associated with other NAT solutions.

Development Environments

Simplify and reduce costs for NAT in development or test environments. fck-nat’s affordability and straightforward deployment make it ideal for temporary or experimental setups.

Step-by-Step Guide: Setting Up fck-nat

Prerequisites

Access to an AWS account.
Basic understanding of AWS VPCs, routing, and EC2 instances.

Deployment Methods

Using Terraform

module "fck-nat" {
  source               = "RaJiska/fck-nat/aws"
  name                 = "my-fck-nat"
  vpc_id               = "vpc-abc1234"
  subnet_id            = "subnet-abc1234"
  update_route_tables  = true
  route_tables_ids     = {
    "private-route-1" = "rtb-abc1234Foo"
    "private-route-2" = "rtb-abc1234Bar"
  }
}

Using CDK (TypeScript)

const natGatewayProvider = new FckNatInstanceProvider({
  instanceType: InstanceType.of(InstanceClass.T4G, InstanceSize.NANO),
});
const vpc = new Vpc(this, 'vpc', {
  natGatewayProvider,
});
natGatewayProvider.securityGroup.addIngressRule(Peer.ipv4(vpc.vpcCidrBlock), Port.allTraffic());

Manual Deployment

Launch the fck-nat AMI from the EC2 Console.
Disable source/destination checks on the instance.
Update private subnet route tables to point to the fck-nat instance.

Configuration Options

/etc/fck-nat.conf file controls behavior:
- eni_id: Elastic Network Interface ID for HA mode.
- eip_id: Elastic IP for consistent external IP.
- cwagent_enabled: Enable CloudWatch agent for detailed metrics.

Validating Deployment

SSH into a private instance routed through fck-nat.
Verify public IP matches the fck-nat Elastic IP.

Considerations for Production Environments

Scaling Limitations

fck-nat does not include native support for automatic scaling. This can be a challenge for applications with fluctuating or large-scale traffic requirements, as manual intervention might be needed to adjust resources.

Compliance and Security

Meeting strict compliance and security standards may require additional configurations. While fck-nat supports security groups and custom firewall rules, organizations may need to perform audits and implement tailored adjustments to ensure compliance.

High Availability

Deploying fck-nat in a high-availability setup requires careful planning and continuous monitoring. Failover events, while supported, depend on manual configurations or external tools to ensure seamless operations.

Before deploying fck-nat in critical systems, it is essential to evaluate these factors thoroughly and consider supplementary tools or processes to address potential gaps.

Conclusion

fck-nat bridges the gap between cost-efficiency and performance in AWS network configurations. However, it is important to note that this solution has some limitations for production environments. For instance, it does not natively support automatic scaling for high-traffic scenarios, and additional configuration may be required to meet compliance and security standards. With flexible deployment options, advanced features, and significant cost savings, fck-nat is best suited for cost-sensitive setups or development environments. To learn more and get started, visit the fck-nat GitHub page.

Getting started with Amazon SageMaker: Building your first machine learning model

Donovan HOANG — Mon, 08 Jul 2024 05:00:00 +0000

This article provides a guide on using Amazon SageMaker to build, train, and deploy a machine learning model for predicting house prices using the Ames Housing dataset. It covers the key features of SageMaker, data preprocessing steps, model training, and deployment, and demonstrates how to test the deployed model. The guide also includes important steps to clean up resources to avoid unnecessary costs.

Overview of Amazon Sagemaker

Amazon SageMaker is a fully managed service provided by AWS (Amazon Web Services) that enables developers and data scientists to build, train, and deploy machine learning models at scale. It simplifies the machine learning workflow by offering a suite of tools and services designed to handle various stages of the machine learning lifecycle, from data preparation to model deployment and monitoring.

Key Features

Integrated Development Environment:

SageMaker Studio: An integrated development environment (IDE) for machine learning that provides a web-based interface to build, train, and deploy models. It offers a collaborative environment with support for notebooks, debugging, and monitoring.
Data Preparation:
Data Wrangler: Simplifies data preparation and feature engineering with a visual interface that integrates with various data sources.
Feature Store: A repository to store, share, and manage features for machine learning models, ensuring consistency and reusability across projects.

Model Building:

Built-in Algorithms: Provides a collection of pre-built machine learning algorithms optimized for performance and scalability.
Custom Algorithms: Supports bringing your own algorithms and frameworks, including TensorFlow, PyTorch, and Scikit-learn.

Model Training:

Managed Training: Automatically provisions and manages the underlying infrastructure for training machine learning models.
Distributed Training: Supports distributed training for large datasets and complex models, reducing training time.
Automatic Model Tuning: Also known as hyperparameter optimization, it helps find the best version of a model by automatically adjusting hyperparameters.

Model Deployment:

Real-time Inference: Deploy models as scalable, secure, and high-performance endpoints for real-time predictions.
Batch Transform: Allows for batch processing of large datasets for inference.
Multi-Model Endpoints: Supports deploying multiple models on a single endpoint, optimizing resource utilization.

Model Monitoring and Management:

Model Monitor: Automatically monitors deployed models for data drift and performance degradation, triggering alerts and actions when necessary.
Pipelines: Enables the creation and management of end-to-end machine learning workflows, from data preparation to deployment and monitoring.

Step-by-Step guide

AWS Free Tier :

First let's talk about AWS Free Tier for SageMaker. What is interesting in our case is the "Studio notebooks, and notebook instances" and "Training" section. Based on what it offers, we are going to use ml.t2.medium as Notebook instance type and ml.m5.large as Training instance (check the availability of theses types in the region you will provision the resources, I am currently using eu-west-3 region).

Storing datasets

First, create a basic S3 bucket that will be used to store our raw dataset and then the training and test formatted datasets.

For this example, I will use the "Ames Housing Dataset" dataset which is a well-known dataset used in machine learning for predictive modeling. It contains information about various houses in Ames, Iowa, and includes features such as the size of the house, the year it was built, the type of roof, and the sale price. The goal is to predict the sale price of houses based on these features.

IAM Role

First, create an IAM role with the AmazonSageMakerFullAccess policy as well as the permission to get and create files on the dataset S3 bucket.

Notebook instance

Create a SageMaker Notebook instance with the following parameters :

type : ml.t2.medium
attach the previously created IAM role

Load and explore dataset

Once your instance appears as "InService" you can click on "Open Jupyter". (It can take minutes for your instance to become ready to use)

This will open a new page with the Jupyter Notebook interface. Now create a new notebook of type conda_python3.

Dependencies and dataset loading

Add this code in the first block and run it.

import boto3
import pandas as pd
import numpy as np
import sagemaker
from sagemaker import get_execution_role
from sagemaker.serializers import CSVSerializer
from sagemaker.deserializers import JSONDeserializer
from sklearn.model_selection import train_test_split

# Load Data from S3
s3 = boto3.client('s3')
bucket_name = 'dhoang-sagemaker-datasets'
file_key = 'AmesHousing.csv'
obj = s3.get_object(Bucket=bucket_name, Key=file_key)
df = pd.read_csv(obj['Body'])
df

It will import all the libraries that we need to access the S3 bucket, pre-process the dataset, train our model and deploy it. You should have a result like this :

Pre-processing the dataset

Because we want to avoid empty or non-numerical values, we need to pre-process the dataset. In addition, we need to split the formatted dataset into a train and a test part to validate the model.

Add this code and run it :

# Preprocess Data
print(df.isnull().sum())
numeric_cols = df.select_dtypes(include=['int64', 'float64']).columns
df[numeric_cols] = df[numeric_cols].fillna(df[numeric_cols].mean())
categorical_cols = df.select_dtypes(include=['object']).columns
df[categorical_cols] = df[categorical_cols].fillna(df[categorical_cols].mode().iloc[0])

# Encode Categorical Features
df = pd.get_dummies(df)

# Ensure all boolean columns are converted to numeric
df = df.applymap(lambda x: 1 if x is True else 0 if x is False else x)

# Split the Data
train, test = train_test_split(df, test_size=0.2, random_state=42)
train.to_csv('train.csv', index=False, header=False)
test.to_csv('test.csv', index=False, header=False)

# Upload the Processed Data to S3
s3_resource = boto3.resource('s3')
s3_resource.Bucket(bucket_name).upload_file('train.csv', 'train.csv')
s3_resource.Bucket(bucket_name).upload_file('test.csv', 'test.csv')

Train and deploy the model

Now this is the that we are going to explore SageMaker features. In our case, we use the linear-learner from SageMaker that allow us to make a linear regression on our dataset.

Next, we define the instance type that will be used for our SageMaker Training Jobs, here ml.m5.large to benefits of the Free Tier.

Then, after being trained, the model is deployed as a SageMaker Endpoint so we can use it afterward.

Add this code and run it :

# Train Model
role = get_execution_role()
sess = sagemaker.Session()
output_location = 's3://{}/output'.format(bucket_name)
container = sagemaker.image_uris.retrieve("linear-learner", sess.boto_region_name, "1.0-1")

linear = sagemaker.estimator.Estimator(
    container,
    role,
    instance_count=1,
    instance_type='ml.m5.large',
    output_path=output_location,
    sagemaker_session=sess
)

linear.set_hyperparameters(
    predictor_type='regressor',
    mini_batch_size=100
)

train_data = 's3://{}/train.csv'.format(bucket_name)
test_data = 's3://{}/test.csv'.format(bucket_name)

data_channels = {
    'train': sagemaker.inputs.TrainingInput(train_data, content_type='text/csv'),
    'validation': sagemaker.inputs.TrainingInput(test_data, content_type='text/csv')
}

linear.fit(inputs=data_channels)

# Deploy Model
predictor = linear.deploy(initial_instance_count=1, instance_type='ml.t2.medium')

It should take some minutes to execute. During this time, you can go to the console to see the Training Job status :

with the logs in your notebook :

and after the job finished, the Model Endpoint deployment :

Wait until the Endpoint status appears as InService.

Test the model

The following code aims to use the fresh new Endpoint to analyse and predict SalePrice on the test dataset. I will run it into the notebook, but you can actually run it from anywhere (if you have access to your endpoint) :

# Test Model
test_data_no_target = test.drop(columns=['SalePrice'])

# Ensure all data is numeric
assert test_data_no_target.applymap(np.isreal).all().all(), "Test data contains non-numeric values"

# Convert test data to CSV string
csv_input = test_data_no_target.to_csv(header=False, index=False).strip()

# Initialize the predictor with correct serializers
predictor = sagemaker.predictor.Predictor(
    endpoint_name=predictor.endpoint_name,
    sagemaker_session=sagemaker.Session(),
    serializer=CSVSerializer(),
    deserializer=JSONDeserializer()
)

# Make predictions
predictions = predictor.predict(csv_input)
print(predictions)

You should get a set of analysis results, here the precision is not mandatory as I just want to provide you a guide to use SageMaker.

Cleaning

To avoid unwanted cost and clean your account, do not forget to delete :

SageMaker Endpoint
SageMaker Notebook instance
S3 bucket
IAM role

Thanks for reading ! Hope this helped you to use or understand how to train a model with Amazon SageMaker from a raw dataset to a ready to use endpoint. Don’t hesitate to give me your feedback or suggestions.

Managing and Rotating Secrets with AWS Secrets Manager

Donovan HOANG — Mon, 24 Jun 2024 06:30:00 +0000

TL; DR;

Securely managing secrets and credentials is crucial for maintaining the integrity of your applications in the cloud. AWS Secrets Manager simplifies this process by providing a centralized, secure repository for storing, managing, and rotating secrets such as database credentials, API keys, and application credentials. This article explains how to use AWS Secrets Manager, highlights its key features, and provides a practical example using Terraform to set up secret rotation with a Lambda function. You'll learn how to securely store secrets, eliminate hard-coded credentials, automate secret rotation, and enhance your application's security.

Maintaining the security and integrity of your apps and data in today's cloud environments depends on securely managing secrets and credentials. You can securely store, manage and rotate secrets with the aid of AWS Secrets Manager. This post will walk you through the process of safely storing managing and rotating credentials and secrets using AWS Secrets Manager. Furthermore, I will provide an example case of AWS Secrets Manager with Terraform.

Understanding AWS Secrets Manager

AWS Secrets Manager enables you to manage, retrieve, and rotate database credentials, application credentials, OAuth tokens, API keys, and other sensitive secrets throughout their lifecycle. Many AWS services can store and use secrets directly from Secrets Manager.

By using Secrets Manager, you can enhance your security posture by eliminating the need to hard-code credentials in your application source code. Storing credentials in Secrets Manager helps prevent potential exposure to anyone who can access your application or its components. Instead of hard-coding credentials, you make a runtime call to Secrets Manager to retrieve them dynamically when needed.

Key features :

Secure Storage:

Encryption: Secrets are encrypted at rest using AWS Key Management Service (KMS) keys, ensuring that your sensitive data is protected.
Access Control: Fine-grained access control policies can be set using AWS IAM to restrict access to secrets based on roles and permissions.

Automatic Rotation:

Rotation Schedules: AWS Secrets Manager allows you to automatically rotate secrets according to a predefined schedule, helping to ensure that your secrets are regularly updated according to your security policies and without manual intervention.
Lambda Integration: Integrate genuinely with AWS Lambda to define custom rotation logic for your secrets.

Access Control and Auditing:

IAM Policies: Define who or what can access your secrets through IAM, ensuring that only authorized users and applications can retrieve sensitive data.
Audit Logging: Integration with AWS CloudTrail provides detailed logs of secrets access, enabling you to monitor and audit sensitive data usage for compliance and security purposes.

Secrets Management:

Versioning: AWS Secrets Manager supports versioning of secrets, allowing you to maintain multiple versions of a secret and revert to previous versions if necessary.
Secret Retrieval: Retrieve secrets programmatically using AWS SDKs, CLI, or HTTP-based requests, ensuring that your applications can securely access secrets when needed.

Integration with AWS Services:

Amazon RDS: Easily rotate database credentials for Amazon RDS instances with built-in support.
Other AWS Services: Seamless integration with other AWS services like Amazon EC2, AWS Lambda, and Amazon ECS to securely manage access to secrets across your application.

Setting Up AWS Secrets Manager with Terraform

First let's create a simple secret with two values : a username and a password :



resource "aws_secretsmanager_secret" "credentials" {
  name        = "sensitive-credentials"
}

resource "aws_secretsmanager_secret_version" "example_version" {
  secret_id     = aws_secretsmanager_secret.credentials.id
  secret_string = jsonencode({
    username = "example_user"
    password = "example_password"
  })
}

Then create the lambda with its specific permissions :




resource "aws_iam_role" "lambda_rotation_role" {
  name = "lambda_rotation_role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "lambda.amazonaws.com"
        }
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "lambda_rotation_policy_attachment" {
  role       = aws_iam_role.lambda_rotation_role.name
  policy_arn = aws_iam_policy.lambda_rotation_policy.arn
}

resource "aws_iam_policy" "lambda_rotation_policy" {
  name   = "lambda_rotation_policy"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = [
          "secretsmanager:GetSecretValue",
          "secretsmanager:PutSecretValue",
          "secretsmanager:UpdateSecretVersionStage",
          "secretsmanager:DescribeSecret"
        ]
        Resource = [ aws_secretsmanager_secret.credentials.arn ]
      }
    ]
  })
}

resource "aws_lambda_function" "rotation_lambda" {
  filename         = "lambda_rotation.zip"  # Path to your zip file
  function_name    = "SecretRotationFunction"
  role             = aws_iam_role.lambda_rotation_role.arn
  handler          = "rotation.lambda_handler"
  runtime          = "python3.8"
  source_code_hash = filebase64sha256("lambda_rotation.zip")

  environment {
    variables = {
      SECRET_ARN = aws_secretsmanager_secret.credentials.arn
    }
  }
}

With the corresponding python script that will handle the secret rotation logic depending on your need :



import boto3
import json
import os
import secrets
import string


def generate_random_password(length=16):
    # Define the characters to use for password generation
    alphabet = string.ascii_letters + string.digits + string.punctuation
    # Generate a random password
    password = ''.join(secrets.choice(alphabet) for i in range(length))
    return password


def lambda_handler(event, context):
    # Retrieve the secret details
    secretsmanager_client = boto3.client('secretsmanager')
    secret_arn = os.environ['SECRET_ARN']

    # Get the current secret value
    current_secret_value = secretsmanager_client.get_secret_value(
        SecretId=secret_arn
    )

    # Generate a new random password
    new_password = generate_random_password()

    # Update the secret with the new password
    secretsmanager_client.put_secret_value(
        SecretId=secret_arn,
        SecretString=json.dumps({
            "username": json.loads(current_secret_value['SecretString'])['username'],
            "password": new_password
        })
    )

    return {
        'statusCode': 200,
        'body': json.dumps('Secret rotated successfully')
    }

And finally, configure Secret Manager to use the Lambda to rotate the secret every 30 days :



resource "aws_secretsmanager_secret_rotation" "example_rotation" {
  secret_id           = aws_secretsmanager_secret.credentials.id
  rotation_lambda_arn = aws_lambda_function.rotation_lambda.arn
  rotation_rules {
    automatically_after_days = 30
  }
}

resource "aws_lambda_permission" "allow_secretsmanager" {
  statement_id  = "AllowSecretsManagerInvoke"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.rotation_lambda.function_name
  principal     = "secretsmanager.amazonaws.com"
  source_arn    = aws_secretsmanager_secret.credentials.arn
}

Deploy it 🚀



zip lambda_rotation.zip rotation.py

terraform init

terraform apply

Test the solution

In your AWS console, go to the AWS Secret Manager page and select the secret that we just created. Click on "Retrieve secret value", you should see the inital value that we set with Terraform :

Now on the "Rotation" tab click on "Rotate secret immediately" and confirm. After retrieving the secret value another time, you should now have a fresh new generated password:

This action will now be performed each 30 days. You can change the period of rotation and even define a cron schedule expression to fit more accurately your need.

Thanks for reading ! Hope this helped you to use or understand how to take advantages of Secret Manager thanks to the rotation of secrets. Don’t hesitate to give me your feedback or suggestions.

Invoke your Step Functions with Api Gateway

Donovan HOANG — Mon, 15 Jan 2024 19:11:47 +0000

TL; DR.

This article provides a comprehensive guide to leveraging AWS Step Functions for efficient orchestration of serverless application workflows. We’ll delve into integrating these workflows with your application backends, ensuring that inputs are correctly processed and outputs are accurately returned. The focus is on simplifying workflow management in complex serverless environments and enhancing the reliability of data exchange within these processes.

Introduction

Managing workflows in serverless applications can become increasingly challenging as processes proliferate. Fortunately, AWS Step Functions offer a robust solution for orchestrating and managing these serverless processes efficiently.

In this article, we’ll demonstrate how to integrate these workflows seamlessly with your application backends. More importantly, we’ll focus on ensuring accurate handling and return of both inputs and outputs within these orchestrated workflows.

You can find all the sources of this project on Github.

Donovan1905 / apigw-sfn-integration

AWS Api Gateway integration for Step Functions

apigw-sfn-integration

AWS Api Gateway integration for Step Functions

View on GitHub

The use cases

In modern cloud architectures, the need to efficiently manage complex workflows often leads to the integration of AWS Step Functions with API Gateway. This combination is particularly valuable when dealing with multi-step processes that require orchestration across various AWS services. By leveraging API Gateway to trigger Step Functions, developers can initiate intricate workflows through simple HTTP requests, streamlining the execution of tasks that are too complex or lengthy for traditional, single-step functions.

Because of the API Gateway timeout limit, you will have to choose between two possible approaches, the synchronous and asynchronous ways :

Synchronous

If you want your API response to give you the final result of your Step Function execution, you will need to use Express function type. This type will allow you to use the states:StartSyncExecution action and wait for the output of your workflow. Even if the Express Step Function timeout limit is much lower that the Standard ones (5 minutes instead of 1 year), it will be even more limited by the API Gateway timeout limit, which is 29 seconds.

So if your workflows will for sure have an execution time lower that 29 seconds and the output of your Step Function are important regarding your need, Express Step Functions are your solution !

Asynchronous

On the other hand, if your executions will, even though it is only some of those, you will need to use Standard Step Function which a timeout limit of 1 year with the states:StartExecution action (only possible invocation with Standard functions). Since this will execute asynchonously, you will get the execution Arn as an Api output, like this :



{
  "executionArn": "arn:aws:states:<region>:<account>:execution:<function_name>:<execution_name>",
  "startDate": <date>
}

With this Arn you will be able to use the states:DescribeExecution to get the state of the execution (Running, Succeeded, Failed, …) as well as the output of the state machine. You fetch this data or with the AWS SDK in your application of by creating another Api route that will retrieve the execution data of a given execution Arn.

Alternatively, you could publish messages on an SQS queue all along the execution and consume it by the concerned application directly. This will prevent your application to retry the DescribeExecution call and wait for the end of the process to get the result state. You could for example update a status of the workflow (for example an EC2 instance that starts, process some data and then stop) in live on a dashboard.

No matter which approach you choose, you will also need to define request and response mapping in the Api Gateway Integration. Theses are meant to define a template that will transform the data between Api Gateway and the Step Function.

Now that you have the keys to choose the type of the functions that fit your need, we will go further with the implementation of this infrastructure using Terraform.

Deploy with Terraform

The Step Function

Let's take a simple workflow for this one. Starting by checking the status of a given instance, if it is stopped we start it but if it is already running we stop it. There will be 2 Step Function like this except that one got a delay timer of 29 seconds so we can test the asynchronous approach.

First, create .tftpl files that will be loaded into Terraform as datasources to deploy the Step Function :



# ./express_state_machine_template.tftpl

{
   "StartAt":"CheckInstanceStatus",
   "States":{
      "CheckInstanceStatus":{
         "Type":"Task",
         "Resource":"arn:aws:states:::aws-sdk:ec2:describeInstanceStatus",
         "Parameters":{
            "InstanceIds.$": "States.Array($.instance_id)"
         },
         "ResultPath":"$.InstanceStatus",
         "Next":"InstanceStatusDecision"
      },
      "InstanceStatusDecision":{
         "Type":"Choice",
         "Choices":[
            {
               "Variable":"$.InstanceStatus.InstanceStatuses[0]",
               "IsPresent": false,
               "Next":"StartInstance"
            },
            {
               "Variable":"$.InstanceStatus.InstanceStatuses[0].InstanceState.Name",
               "StringEquals":"running",
               "Next":"StopInstance"
            }
         ],
         "Default":"EndState"
      },
      "StartInstance":{
         "Type":"Task",
         "Resource":"arn:aws:states:::aws-sdk:ec2:startInstances",
         "Parameters":{
            "InstanceIds.$": "States.Array($.instance_id)"
         },
         "End":true
      },
      "StopInstance":{
         "Type":"Task",
         "Resource":"arn:aws:states:::aws-sdk:ec2:stopInstances",
         "Parameters":{
            "InstanceIds.$": "States.Array($.instance_id)"
         },
         "End":true
      },
      "EndState":{
         "Type":"Succeed"
      }
   }
}

For ./standard_state_machine_template.tftpl simply add the following action :



...

"Wait": {
    "Type": "Wait",
    "Seconds": 29,
    "Next": "CheckInstanceStatus"
  }

...

Next, import the templates as datasources and define your Step Functions resources :



# ./sfn.tf

data "template_file" "express_state_machine_template" {

  template = file("${path.module}/express_state_machine_template.tftpl")

  vars = {

    instance_id = aws_instance.example.id

  }

}

resource "aws_sfn_state_machine" "express_sfn_state_machine" {

  name     = "${var.project_name}-express"

  role_arn = aws_iam_role.iam_for_sfn.arn

  type     = "EXPRESS"

definition = data.template_file.express_state_machine_template.rendered

}

data "template_file" "standard_state_machine_template" {

  template = file("${path.module}/standard_state_machine_template.tftpl")

  vars = {

    instance_id = aws_instance.example.id

  }

}

resource "aws_sfn_state_machine" "standard_sfn_state_machine" {

  name     = "${var.project_name}-standard"

  role_arn = aws_iam_role.iam_for_sfn.arn

  type     = "STANDARD"

definition = data.template_file.standard_state_machine_template.rendered

}

IAM roles

Then we define the two IAM roles that will be used, one for the Step Function execution and one to allow the Api Gateway to invoke the functions.



# ./iam.tf

./data "aws_iam_policy_document" "assume_role_policy_sfn" {

  statement {

    effect = "Allow"

    principals {

      identifiers = ["states.amazonaws.com"]

      type        = "Service"

    }

    actions = ["sts:AssumeRole"]

  }

}

data "aws_iam_policy_document" "role_policy_sfn" {

  statement {

    effect  = "Allow"

    actions = [

      "ec2:"

    ]

    resources = [

      "",

    ]

  }

statement {

    sid    = "LoggingPolicy"

    effect = "Allow"

    actions = [

      "logs:"

    ]

    resources = [

      ""

    ]

  }

}

resource "aws_iam_role" "iam_for_sfn" {

  name = "stepFunctionExecutionIAM"

inline_policy {

    name   = "PolicyForSfn"

    policy = data.aws_iam_policy_document.role_policy_sfn.json

  }

assume_role_policy = data.aws_iam_policy_document.assume_role_policy_sfn.json

}

data "aws_iam_policy_document" "assume_role_policy_apigw" {

  statement {

    sid    = ""

    effect = "Allow"

    principals {

      identifiers = ["apigateway.amazonaws.com"]

      type        = "Service"

    }

    actions = ["sts:AssumeRole"]

  }

}

data "aws_iam_policy_document" "policy_start_sfn" {

  statement {

    sid    = "ApiGwPolicy"

    effect = "Allow"

    actions = [

      "states:StartSyncExecution",

      "states:StartExecution"

    ]

    resources = [

      "*"

    ]

  }

}

resource "aws_iam_role" "iam_for_apigw_start_sfn" {

  name               = "${var.project_name}-apigw-exec-sfn"

  assume_role_policy = data.aws_iam_policy_document.assume_role_policy_apigw.json

}

resource "aws_iam_role_policy" "policy_start_sfn" {

  policy = data.aws_iam_policy_document.policy_start_sfn.json

  role   = aws_iam_role.iam_for_apigw_start_sfn.id

}

EC2 instance

We just need a simple EC2 instance, nothing particular here it just needs to be started and stopped.



# ./ec2.tf

data "aws_ami" "amazon_linux_2023" {

  most_recent = true

  owners      = ["amazon"]

filter {

    name = "architecture"

    values = ["x86_64"]

  }

filter {

    name   = "description"

    values = ["Amazon Linux 2023 *"]

  }

}

resource "aws_instance" "example" {

  ami           = data.aws_ami.amazon_linux_2023.id

  instance_type = "t3.nano"

tags = {

    Name = "HelloWorld"

  }

}

Finally, Api Gateway !

Our Api definition will contain all the following resources :

api_gateway_rest_api : the Api itself
api_gateway_resource : the resource name and path part (one per Step Function)
api_gateway_method : the associated resource HTTP verb (one per Step Function)
api_gateway_deployment : the content of Api definition
api_gateway_integration : the link between your Api route and your Step Function. This is where you define your request mapping template to transform and pass the input to the function (one per Step Function). In the integration, we need to define how we are going to invoke the function, states:StartSyncExecution or states:StartExecution by setting the uri parameter with the following structure arn:aws:apigateway:<region>:states:action/<action_type>. Also, the request mapping template will contain the input biding as well as the stateMachineArn.
api_gateway_method_response and api_gateway_integration_response : the link between your Step Function output and the Api Gateway. This is where you define your response mapping template to transform your Step Function result and pass it to the Api Gateway (one per Step Function)



# ./apigw.tf

resource "aws_api_gateway_rest_api" "apigw" {

  name = "${var.project_name}-apigw"

}

resource "aws_api_gateway_resource" "express_ec2" {

  parent_id   = aws_api_gateway_rest_api.apigw.root_resource_id

  path_part   = "express-instance"

  rest_api_id = aws_api_gateway_rest_api.apigw.id

}

resource "aws_api_gateway_method" "express_post" {

  authorization = "NONE"

  http_method   = "POST"

  resource_id   = aws_api_gateway_resource.express_ec2.id

  rest_api_id   = aws_api_gateway_rest_api.apigw.id

}

resource "aws_api_gateway_resource" "standard_ec2" {

  parent_id   = aws_api_gateway_rest_api.apigw.root_resource_id

  path_part   = "standard-instance"

  rest_api_id = aws_api_gateway_rest_api.apigw.id

}

resource "aws_api_gateway_method" "standard_post" {

  authorization = "NONE"

  http_method   = "POST"

  resource_id   = aws_api_gateway_resource.standard_ec2.id

  rest_api_id   = aws_api_gateway_rest_api.apigw.id

}

resource "aws_api_gateway_deployment" "example" {

  rest_api_id = aws_api_gateway_rest_api.apigw.id

triggers = {

    redeployment = sha1(jsonencode([

      aws_api_gateway_resource.express_ec2.id,

      aws_api_gateway_method.express_post.id,

      aws_api_gateway_integration.express_integration.id,

      aws_api_gateway_resource.standard_ec2.id,

      aws_api_gateway_method.standard_post.id,

      aws_api_gateway_integration.standard_integration.id,

    ]))

  }

lifecycle {

    create_before_destroy = true

  }

}

resource "aws_api_gateway_integration" "express_integration" {

  rest_api_id             = aws_api_gateway_rest_api.apigw.id

  resource_id             = aws_api_gateway_resource.express_ec2.id

  http_method             = aws_api_gateway_method.express_post.http_method

  integration_http_method = "POST"

  type                    = "AWS"

  uri = (

    "arn:aws:apigateway:${var.region}:states:action/StartSyncExecution"

  )

  credentials = aws_iam_role.iam_for_apigw_start_sfn.arn

request_templates = {

    "application/json" = <<EOF

  
  
  set($input = $input.json('$'))


{

   "input": "$util.escapeJavaScript($input).replaceAll("\'", "'")",

 "stateMachineArn": "${aws_sfn_state_machine.express_sfn_state_machine.arn}"

}

EOF

  }

}

resource "aws_api_gateway_integration" "standard_integration" {

  rest_api_id             = aws_api_gateway_rest_api.apigw.id

  resource_id             = aws_api_gateway_resource.standard_ec2.id

  http_method             = aws_api_gateway_method.standard_post.http_method

  integration_http_method = "POST"

  type                    = "AWS"

  uri = (

    "arn:aws:apigateway:${var.region}:states:action/StartExecution"

  )

  credentials = aws_iam_role.iam_for_apigw_start_sfn.arn

request_templates = {

    "application/json" = <<EOF

  
  
  set($input = $input.json('$'))


{

   "input": "$util.escapeJavaScript($input).replaceAll("\'", "'")",

 "stateMachineArn": "${aws_sfn_state_machine.standard_sfn_state_machine.arn}"

}

EOF

  }

}

resource "aws_api_gateway_method_response" "express_response_200" {

  rest_api_id = aws_api_gateway_rest_api.apigw.id

  resource_id = aws_api_gateway_resource.express_ec2.id

  http_method = aws_api_gateway_method.express_post.http_method

  status_code = "200"

}

resource "aws_api_gateway_integration_response" "express_response_200" {

  rest_api_id = aws_api_gateway_rest_api.apigw.id

  resource_id = aws_api_gateway_resource.express_ec2.id

  http_method = aws_api_gateway_method.express_post.http_method

  status_code = aws_api_gateway_method_response.express_response_200.status_code

response_templates = {

    "application/json" = <<EOF

  
  
  set ($parsedPayload = $util.parseJson($input.json('$.output')))


$parsedPayload

EOF

  }

}

resource "aws_api_gateway_method_response" "standard_response_200" {

  rest_api_id = aws_api_gateway_rest_api.apigw.id

  resource_id = aws_api_gateway_resource.standard_ec2.id

  http_method = aws_api_gateway_method.standard_post.http_method

  status_code = "200"

}

resource "aws_api_gateway_integration_response" "standard_response_200" {

  rest_api_id = aws_api_gateway_rest_api.apigw.id

  resource_id = aws_api_gateway_resource.standard_ec2.id

  http_method = aws_api_gateway_method.standard_post.http_method

  status_code = aws_api_gateway_method_response.standard_response_200.status_code

response_templates = {

    "application/json" = <<EOF

  
  
  set ($parsedPayload = $util.parseJson($input.json('$')))


$parsedPayload

EOF

  }

}

Let's test it !

Make sure to apply the terraform configuration with terraform apply.

Synchronous route

Go your Api Gateway, choose the /express-instance resource with the POST verb, in the "Test" tabs add the following input :

After waiting a few seconds, you will first see the input transformation :

And then the output of the workflow :

Now your instance is running.

Asynchronous route

Go your Api Gateway, choose the /standard-instance resource with the POST verb, in the "Test" tabs .
Enter the same input that we used. Now you will get the result instantly and the input transformation will remain the same.

But the output will now contain the executionArn :

If you want to check the status of the execution or the content you can use the states:DescribeExecution. Here with the cli but you will ideally use it with SDK in your application or in a Lambda exposed on your Api Gateway :

Go further…

In addition of the different approaches present in this article, you could integrate in your Step Function many SQS messages publication where you want your application to know the status changes. This will allow your application to react faster based on the internal function state changes.

To finish, you can read this article by AWS that treats about integrating Step Function and track its activity with the Api Gateway v2 WebSockets.

Thanks for reading ! Hope this helped you to use or understand how to take advantages of your Step Function thanks to the Api Gateway. Don’t hesitate to give me your feedback or suggestions.

Supercharge File Processing with AWS Lambda, S3 and DynamoDb

Donovan HOANG — Fri, 15 Dec 2023 08:55:14 +0000

TL; DR.

Unlock the potential of serverless architecture with AWS Lambda and DynamoDB to revolutionize your sales data processing:

Streamline data transformations: Say goodbye to complex ETL pipelines.
Enrich data dynamically: Calculate prices with taxes, totals, and unique IDs automatically.
Scalability and cost-efficiency: AWS Lambda adapts seamlessly to your workload.
Embrace serverless efficiency for transformative insights into your sales data.
Ready to transform your data? Dive into the article for a step-by-step guide!

Introduction

When a large volume of files needs to be processed, the costs can skyrocket as well as the capacity of handling this volume. Many options will not be suitable in this case as a server-centric architecture or manual scalability. Imagine being able to automate this process seamlessly, without the need for complex infrastructure. That’s the power of building a serverless file processing system using AWS Lambda and Amazon S3 Events.

Enter AWS Lambda and Amazon S3 Events — powerful tools that, when combined, offer a serverless solution for file processing. In this article, we’ll explore the step-by-step process of setting up a serverless file processing system using AWS Lambda and S3 Events. I will use Terraform as IaC tool.

You can find all the sources of this project on Github.

Donovan1905 / s3-serverless-file-processing

Serverless file processing using Lambda and S3 on AWS

s3-serverless-file-processing

Serverless file processing using Lambda and S3 on AWS

View on GitHub

Use case

In the realm of data-driven enterprises, processing and extracting insights from sales data is a pivotal challenge. Imagine a scenario where CSV files containing sales details—client IDs, products, quantities, and prices—arrive in an Amazon S3 bucket. Traditional methods might involve complex setups, but we’re about to redefine efficiency with the simplicity of serverless architecture.

The Challenge: Data Transformation Complexity:

Simplify the transformation of CSV data into a structured format.
Dynamic Data Enrichment: Automate tasks like calculating prices with taxes, determining total transaction values, and assigning unique command IDs.
Scalability and Cost Efficiency: Ensure scalability without the overhead of managing idle infrastructure.

The Solution:

AWS Lambda and DynamoDB take center stage. Lambda responds to CSV uploads, transforming data seamlessly, while DynamoDB becomes the canvas for storing and enriching sales information.

The Use Case:

CSV Upload to S3: Sales data lands in an S3 bucket, triggering serverless processing.
Lambda Takes the Stage: Responding to S3 events, Lambda transforms raw CSV data.
DynamoDB, the Canvas of Transformation: Enriched data is stored with additional computed information.
Scalability in Action: Lambda scales dynamically, ensuring efficiency and cost-effectiveness.

In the next sections, we delve into the technical details of implementing this serverless file processing system. From configuring Lambda functions to designing DynamoDB tables, let’s unlock the potential of AWS Lambda, DynamoDB and S3 to elevate sales data processing.

CSV processing with Python

To extract and process the CSV file in our Lambda, we are going to use Python runtime with csv and boto3 libraries.

from datetime import datetime

import boto3
import os
import botocore
import json
import csv
import uuid

s3 = boto3.client('s3')


def lambda_handler(event, context):
    local_filename = "/tmp/filename.csv"

    dynamodb = boto3.resource('dynamodb')
    item_table = dynamodb.Table(os.environ['ITEM_TABLE'])
    command_table = dynamodb.Table(os.environ['COMMAND_TABLE'])

    s3 = boto3.client('s3')
    BUCKET_NAME = event["Records"][0]["s3"]["bucket"]["name"]
    S3_KEY = event["Records"][0]["s3"]["object"]["key"]

    print(BUCKET_NAME)
    print(S3_KEY)

    try:
        s3.download_file(BUCKET_NAME, S3_KEY, local_filename)
    except botocore.exceptions.ClientError as e:
        if e.response['Error']['Code'] == "404":
            print("The object does not exist: s3://" + BUCKET_NAME + S3_KEY)

    with open(local_filename, mode='r') as csv_file:
        csv_reader = csv.DictReader(csv_file)
        total_price = 0
        command_id = str(uuid.uuid1())
        for row in csv_reader:
            total_price += float(row["unit_price"]) * float(row["quantity"])
            item_table.put_item(
                Item={
                    'id': str(uuid.uuid1()),
                    'command_id': command_id,
                    'item_name': row["item_name"],
                    'unit_price': row["unit_price"],
                    'quantity': row["quantity"],
                    'customer': row["customer"],
                    'salesman': row["salesman"]
                }
            )
        command_table.put_item(
            Item={
                'id': command_id,
                'date': str(datetime.now()),
                'status': "Pending"
            }
        )

    return {
        "statusCode": 200,
        "body": json.dumps({
            "message": "data processed with command_id " + command_id,
        }),
    }

Next, build the zip package that will be uploaded for Lambda with this script:

# lambda_build_script.sh 

python3 -m pip install --platform manylinux2014_x86_64 --implementation cp --only-binary=:all: --upgrade --target venv/lib/python3.11/site-packages/ -r requirements.txt
mkdir lambda_package
cp -r venv/lib/python3.11/site-packages/* lambda_package/
cp -r src lambda_package/
cd lambda_package && zip -r ../lambda.zip * && cd ../

Deploy with Terraform

S3 Buckets

We need two buckets here, one to collect the sales CSV files and one to store the Lambda code. Also, we define a bucket notification to trigger the Lambda function during certain event, in our case when a new file is uploaded in our S3.

# /tf/s3.tf

resource "aws_s3_bucket" "builds" {
  bucket = "${var.project_name}-lambda-builds"
}

resource "aws_s3_bucket" "bucket" {
  bucket = local.bucket_name
}

resource "aws_s3_bucket_notification" "s3-lambda-trigger" {
  bucket = aws_s3_bucket.bucket.id
  lambda_function {
    lambda_function_arn = module.lambda.lambda_function_arn
    events              = ["s3:ObjectCreated:*"]
  }
}

IAM roles and policies

In this part, you will need to create the appropriate role to allow your Lambda function to use the S3 bucket.

# /tf/iam.tf

data "aws_iam_policy_document" "assume_role_policy_lambda" {
  statement {
    sid    = ""
    effect = "Allow"
    principals {
      identifiers = ["lambda.amazonaws.com"]
      type        = "Service"
    }
    actions = ["sts:AssumeRole"]
  }
}

data "aws_iam_policy_document" "policy_s3" {
  statement {
    sid    = "S3AccessPolicy"
    effect = "Allow"
    actions = [
      "s3:*"
    ]
    resources = [
      "arn:aws:s3:::${local.bucket_name}",
      "arn:aws:s3:::${local.bucket_name}/*"
    ]
  }

  statement {
    sid    = "DynamoDbAccessPolicy"
    effect = "Allow"
    actions = [
      "dynamodb:PutItem",
      "dynamodb:GetItem"
    ]
    resources = [
      aws_dynamodb_table.command_table.arn,
      aws_dynamodb_table.item_table.arn
    ]
  }

  statement {
    sid    = "LambdaLoggingPolicy"
    effect = "Allow"
    actions = [
      "logs:CreateLogGroup",
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ]
    resources = [
      "arn:aws:logs:*:*:*"
    ]
  }
}

resource "aws_iam_role" "iam_for_lambda" {
  name               = "${var.project_name}-lambda-role"
  assume_role_policy = data.aws_iam_policy_document.assume_role_policy_lambda.json
}

resource "aws_iam_role_policy" "lambda_policy" {
  policy = data.aws_iam_policy_document.policy_s3.json
  role   = aws_iam_role.iam_for_lambda.id
}

Lambda function

Next, let’s define the Lambda function itself. Sending the lambda.zip as an S3 object and referencing it in the Lambda definition.

# /tf/lambda.tf

module "lambda" {
  source  = "terraform-aws-modules/lambda/aws"
  version = "5.2.0"

  function_name = "${var.project_name}-process-sales-data"
  runtime       = "python3.11"
  handler       = "src.file_processing_handler.lambda_handler"
  lambda_role   = aws_iam_role.iam_for_lambda.arn
  create_role   = false
  environment_variables = {
    "REGION"        = var.region,
    "ITEM_TABLE"    = aws_dynamodb_table.item_table.name,
    "COMMAND_TABLE" = aws_dynamodb_table.command_table.name
  }

  memory_size = 1028
  timeout     = 30

  create_package = false
  s3_existing_package = {
    bucket = aws_s3_bucket.builds.id
    key    = aws_s3_object.lambda_package.id
  }
}

resource "aws_s3_object" "lambda_package" {
  bucket = aws_s3_bucket.builds.id
  key    = "${filemd5(local.lambda_package)}.zip"
  source = local.lambda_package
}

resource "aws_lambda_permission" "s3_lambda_permission" {
  statement_id  = "AllowS3Invoke"
  action        = "lambda:InvokeFunction"
  function_name = module.lambda.lambda_function_name
  principal     = "s3.amazonaws.com"
  source_arn    = "arn:aws:s3:::${aws_s3_bucket.bucket.id}"
}

DynamoDB tables

In our scenario, we will use two DynamoDB table. One command table and one item table.

# /tf/dynamodb.tf

resource "aws_dynamodb_table" "command_table" {
  name         = "${var.project_name}-command-table"
  hash_key     = "id"
  billing_mode = "PAY_PER_REQUEST"

  attribute {
    name = "id"
    type = "S"
  }

  deletion_protection_enabled = true
}

resource "aws_dynamodb_table" "item_table" {
  name         = "${var.project_name}-item-table"
  hash_key     = "id"
  billing_mode = "PAY_PER_REQUEST"

  attribute {
    name = "id"
    type = "S"
  }

  deletion_protection_enabled = true
}

Let’s test it !

First you will need to build our lambda sources.

$ python3 -m venv venv
$ source ./venv/bin/activate
$ sh lambda_build_script.sh

Now you should have a lambda.zip folder.

Make sure to apply the terraform configuration with terraform apply.

Now, we will take this CSV example referencing all the product that a client bought (example/sales_sample.csv).

And now let the magic happen ! Just drop the CSV file into the S3 bucket and take a look into your DynamoDb tables to see your processed data.

Thanks for reading ! Hope this helped you to use or understand the power of AWS services. Don’t hesitate to give me your feedback or suggestions.