Forem: Shubham Murti

Build an automated video monitoring system with AWS IoT and AI/ML : AWS Project

Shubham Murti — Sat, 16 Nov 2024 11:25:28 +0000

Introduction

Amazon Web Services (AWS) offers a robust platform for developing cloud-based solutions, designed to cater to the growing demand for scalable, secure, and cost-effective technology. With AWS, professionals can build everything from simple applications to complex enterprise systems, leveraging a suite of globally recognized services such as EC2, S3, Lambda, and DynamoDB.

This blog introduces practical scenarios where AWS services can be utilized to solve real-world problems, empowering developers and cloud enthusiasts to sharpen their skills. The focus is on applying concepts effectively to deploy efficient, scalable, and reliable cloud solutions. Whether you’re a beginner exploring cloud computing or an experienced professional advancing your expertise, AWS’s comprehensive ecosystem serves as a foundation for innovation.

Prerequisites

AWS Account
Ensure you have an active AWS account to access and deploy resources. Proper billing permissions are essential for provisioning services.
Cloud Computing Basics
Familiarity with cloud concepts, including virtualization, networking, and basic AWS services like EC2 and S3, is helpful for a smoother experience.
Programming Fundamentals
Knowledge of programming concepts and languages (e.g., Python, JavaScript) is beneficial for working with AWS services and automation tools.
Technical Setup
Have a stable internet connection and tools like a modern browser, AWS CLI, and a text editor or IDE installed on your system.
Cost Awareness
Understand AWS pricing and usage limits. Utilize the Free Tier where possible to minimize costs while experimenting with services.

Technical Stack

Amazon EC2
Scalable virtual servers for hosting applications and workloads.
AWS Lambda
Serverless compute service for running code without managing servers.
Amazon S3
Secure and scalable storage for objects like files and backups.
Amazon RDS
Managed relational databases for engines like MySQL and PostgreSQL.
Amazon DynamoDB
Fully managed NoSQL database for high-speed and scalable applications.
Amazon VPC
Isolated cloud network for secure deployment of AWS resources.
AWS IAM
Access management to control permissions for users and resources.
AWS KMS
Encryption key management for secure data storage.
Amazon CloudWatch
Monitoring and observability service for AWS resources and applications.
AWS CloudFormation
Infrastructure automation using code templates.
AWS CLI
Command-line tool to manage AWS services programmatically.
AWS Route 53
Domain Name System (DNS) service for routing traffic to applications.

Architecture Diagram

The architecture diagram above illustrates a typical AWS-based web application setup

WebRTC Introduction

WebRTC is an open technology specification for enabling real-time communication (RTC) across browsers and mobile applications via simple APIs. It leverages peering techniques for real-time data exchange between connected peers and provides low media streaming latency required for human-to-human interaction. WebRTC specification includes a set of IETF protocols including Interactive Connectivity Establishment (ICE RFC5245), Traversal Using Relay around NAT (TURN RFC5766), and Session Traversal Utilities for NAT (STUN RFC5389) for establishing peer-to-peer connectivity, in addition to protocol specifications for real-time media and data streaming.

WebRTC Connection Flow

The following diagram illustrates the connection flow as it occurs using Kinesis Video Streams for WebRTC.

Kinesis Video Streams with WebRTC Components

Kinesis Video Streams provides a standards compliant WebRTC implementation as a fully-managed capability. You can use this capability to securely live stream media or perform two-way audio or video interaction between any camera IoT device and WebRTC compliant mobile or web players. As a fully-managed capability, you do not have to build, operate, or scale any WebRTC related cloud infrastructure such as signaling or media relay servers to securely stream media across applications and devices.

Kinesis Video Streams provides managed end-points for WebRTC signaling that allows applications to securely connect with each other for peer-to-peer live media streaming. Next, it includes managed end-points for TURN that enables media relay via the cloud when applications cannot stream peer-to-peer media. It also includes managed end-points for STUN that enables applications to discover their public IP address when they are located behind a NAT or a firewall. Additionally, it provides easy to use SDKs to enable camera IoT devices with WebRTC capabilities. Finally, it provides client SDKs for Android, iOS, and for Web applications to integrate Kinesis Video Streams WebRTC signaling, TURN, and STUN capabilities with any WebRTC compliant mobile or web player.

Amazon Rekognition — ML Video Analysis

Amazon Rekognition makes it easy to add image and video analysis to your applications. You just have to provide an image or video to the Amazon Rekognition API, and the service can:

Identify labels (objects, concepts, people, scenes, and activities) and text
Detect inappropriate content
Provide highly accurate facial analysis, face comparison, and face search capabilities

Working with stored video analysis

Amazon Rekognition Video is an API that you can use to analyze videos. With Amazon Rekognition Video, you can detect labels, faces, people etc. in videos that are stored in an Amazon Simple Storage Service (Amazon S3) bucket. Previously, scanning videos for objects or people would have taken many hours of error-prone viewing by a human being. Amazon Rekognition Video automates the detection of items and when they occur throughout a video.

Amazon Rekognition Video API overview

Amazon Rekognition Video processes a video that’s stored in an Amazon S3 bucket. The design pattern is an asynchronous set of operations. You start video analysis by calling a Start operation such as StartLabelDetection . The completion status of the request is published to an Amazon Simple Notification Service (Amazon SNS) topic. To get the completion status from the Amazon SNS topic, you can use an Amazon Simple Queue Service (Amazon SQS) queue or an AWS Lambda function. After you have the completion status, you call a Get operation, such as GetLabelDetection , to get the results of the request.

Amazon OpenSearch Service

Amazon OpenSearch Service makes it easy for you to perform interactive log analytics, real-time application monitoring, website search, and more. OpenSearch is an open source, distributed search and analytics suite derived from Elasticsearch. Amazon OpenSearch Service offers the latest versions of OpenSearch, support for 19 versions of Elasticsearch (1.5 to 7.10 versions), as well as visualization capabilities powered by OpenSearch Dashboards and Kibana (1.5 to 7.10 versions).

Step-by-Step Implementation

1. Getting Started

Let’s first review the resources that have already been created for you as part of this workshop:

Navigate to the AWS CloudFormation console . (you need to be in the US West (Oregon) region to be able to view the resources)
Click on the stack with name iot304-base-stack and go to the Outputs tab.

The Amazon OpenSearch Service Domain is pre-created in this workshop. This will be used to store the labels related to entities detected in your streaming videos.
Make a note of the Key (OpenSearchURL, OpenSearchARN) and Value (search-kvs-workshop-domain-..., arn...). You'll use them in later parts of the workshop.
You can also explore the Resources tab. In addition to the OpenSearch Domain, there is an AWS Cloud9 instance also created which you’ll use to host the Video Analytics application (front-end) in later half of this workshop.

3. Create Kinesis Video Stream Resources

As described in the ‘Kinesis Video Streams with WebRTC Components’ section, a signaling channel is required to establish peer-to-peer connection when you want to enable live-streaming of video. And a video stream is a KVS resource that enables you to transport live video data, store it, and make the data available for consumption both in real time and on a batch or ad hoc basis. In this section, you will create a signaling channel and a video stream in KVS, and initiate streaming from your laptop’s camera through Google Chrome browser. You will first create a peer-to-peer stream to see near real-time video, and then you’ll ingest video into a KVS stream and test the media playback via KVS Video Stream console.

Please make sure you are in the US West (Oregon) region for all steps.

Creating a signaling channel:

In the AWS Management Console, open the Kinesis Video Streams console
In the left navigation, click Signaling channels. And then click Create signaling channel.

On the Create a new signaling channel page, type the name for the signaling channel. For this workshop you can use StreamChannel. Leave the default Time-to-live (Ttl) value as 60 seconds.

Click Create signaling channel.
Once the signaling channel is created, review the details on the channel’s details page. Make note of the Signaling channel ARN.

Creating a video stream:

In the left navigation pane, select Video streams. Click Create video stream.

On the Create a new video stream page, type a name for this stream. For this workshop you can use WebRTCStream. Use the default configuration for other parameters.

Click Create video stream.
Once the stream is created, review the details on the Video Streams page. Make note of the Video stream ARN.

Stream video peer-to-peer in near real-time:

Open the Kinesis Video Streams WebRTC Test Page . You’ll use this page to initiate the video stream from your system. This page has been created for testing purposes using the Amazon Kinesis Video Streams WebRTC SDK for JavaScript .

Set the Region: us-west-2
Set the Access Key ID, Secret Access Key and Session Token.

If you are undertaking this outside of an AWS event and using your own AWS account, then please retrieve your credentials as you normally would, and paste them here.
If you are participating in an AWS event and using an AWS-provided account, please refer to Step 7 of this section and the Get AWS CLI credentials option. This is located on the bottom-left corner of the Workshop Studio Event Dashboard. Copy and paste the values within the double quotes (as shown for AWS_ACCESS_KEY_ID below).

Please Note
While Session Token is marked as an ‘Optional’ field in the KVS WebRTC Test Page, you must provide it if you’re participating in an AWS event, with an AWS-provided account.
Set the Channel Name: StreamChannel
or as described when creating the signaling channel.
Verify that Send Video and Send Audio options are enabled under Tracks. If not checked already then please make sure you enable both.

Click on the Start Master button located right above the Logs section. It may ask you to Allow Access for your browser to access your system’s camera and microphone. Please select Allow to proceed.

Vew the real-time peer-to-peer stream:

In the Kinesis Video Streams console , on the left navigation pane, click Signaling channels.
Click on the signaling channel you created earlier (StreamChannel if you gave the same name as suggested in steps above).
Expand the Media playback viewer option and press the play button.

The peer-to-peer video stream should appear within a few seconds. Put this viewer and the KVS WebRTC Test Page side-by-side and confirm that the latency is minimal. Congratulations, you have a real-time peer-to-peer stream!
Return to the KVS WebRTC Test Page and click on Stop Master.

Ingest video into the cloud:

Now you’ll reconfigure the KVS WebRTC Test Page to use the WebRTC ingest feature to store video in a stream, instead of having a peer-to-peer stream.

Expand ‘WebRTC Ingestion and Storage’ and provide WebRTCStream as the Stream name (or as described when creating the video stream).
Click on Update Media Storage Configuration. (You can verify if the configuration was updated successfully or not by scrolling down to the Logs section and looking for the success message as shown in screenshot below)

Scroll back to the ‘WebRTC Ingestion and Storage’ section and enable the Ingestion and storage peer joins automatically option by ticking the checkbox

Click on the Start Master button located right above the Logs section.

View the video stored in the stream:

In the Kinesis Video Streams console , on the left navigation pane, click Video streams.
Click on the video stream created before (WebRTCStream if you gave the same name as suggested in steps above).
Expand the Media playback option.

Wait for a few seconds and you should now be able to see the video stream coming in from your laptop’s camera. (Ignore the Download SDK pop-up and in case it persists then just refresh the KVS Video Streams Console page once). Note that since this video is playing back from storage, it has considerable latency. Congratulations, you have successfully played back video that you ingested and stored in a stream!
Return to the KVS WebRTC Test Page and click on Stop Master.

4. Provision supporting resources — Lambda, API Gateway & Step Function

You will now create the following resources which will be used further in the workshop (a CloudFormation script is provided in the next step which will be used to create all the resources mentioned below)-

Resource Identifier in CloudFormationUsagekvslambdaThis Lambda function would get the KVS video stream and cut them into smaller chunks and store into S3. This would also call the StartLabelDetection API for Rekognition to start asynchronous detection of labels in the stored video.LambdaRoleThis is the IAM role that will enable AWS Lambda to work with Rekognition, Kinesis Video Streams, OpenSearch cluster, SNS and S3.MyCloudFrontDistributionThis is the CloudFront Distribution URL which would serve the media in our front-end application.rekognitionlambdaSNS would trigger this Lambda function to get the detected labels, their confidence scores and timestamp from Rekognition and store them into Amazon OpenSearch Service.RekognitionRoleThis is the IAM role that will enable Amazon Rekognition to conduct stored video analysis.S3BucketVideoThis is an Amazon S3 Bucket that will be used for storing video snippets. The incoming video streams from KVS would be cut into smaller segments and stored into this S3 bucket.searchApiGatewayThis is a REST API within API Gateway to facilitate search option in the final front-end application.searchlambdaThis Lambda function would be invoked when an end user accesses the front-end application and searches for an entity. The search API would trigger this function which in turn would get the entities from OpenSearch.SnsTopicOnce Rekognition completes label detection for a video clip, it will trigger a notification via this SNS topic.KVSStateMachineStep Function to orchestrate the whole workflow.

4.1 Create CloudFormation Stack

Follow the steps mentioned below and pass the Parameter values as described carefully-

Click the following link to launch the CloudFormation stack: Launch CloudFormation stack in us-west-2 (Oregon)
Under the Parameters section of the Specify stack details page, provide the following values:

KVSStreamName: WebRTCStream. (This is the name of the KVS Video Stream created in Section 3 Step 9.)
OpenSearchARN: This is the ARN for the Amazon OpenSearch Service Domain. Paste the value noted in Step 4 of the ‘Getting Started’ section. For quick access, you can also navigate to the *Output **section of **iot304-base-stack *here and get the value.
OpenSearchApiUrl: This is the domain endpoint for the Amazon OpenSearch Service Domain. Paste the value noted in Step 4 of the ‘Getting Started’ section. For quick access, you can also navigate to the *Output **section of **iot304-base-stack *here and get the value.

Click Next till you get to the Review page (Step 4 in console). Scroll down till the end and check the box for I acknowledge that AWS CloudFormation might create IAM resources with custom names and click Submit.

Wait for the stack creation to reach CREATE_COMPLETE.
Once stack is provisioned, make a note of the Amazon API Gateway URL, AWS Lambda Role ARN, Amazon CloudFront Distribution and the AWS Step Function ARN from the Outputs section. You'll use these in the upcoming sections.

4.2 Mapping the IAM role for Lambda with Amazon OpenSearch Service domain

For the searchlambda function to be able to work with Amazon OpenSearch Service, for searching or analyzing data stored in OpenSearch, you’ll need to map the Lambda IAM role (LambdaRole) created in the section above, with the OpenSearch domain.

Navigate to the Amazon OpenSearch Service Domain console .
Click on the Domain Name kvs-workshop-domain.
Search for the OpenSearch Dashboards URL on the top-right corner and click on it. This will open a new tab for the OpenSearch Dashboards.

You’ll be prompted to enter the login credentials here. Copy and paste the following and click Log in-

Username: admin
Password: Amazon90!

Keep the tenant selection as Global and Confirm (if you get a pop-up asking you to add data, select Explore on my own).

Click on the hamburger menu option on the top-left side of the screen and select Security under Management section in the left navigation pane.

Click on Roles and copy-paste all_access in the Search option and hit return/enter.

Click on the all_access hyperlink and switch to the Mapped Users tab on top half of the screen. Click on Manage Mapping.

Scroll down to the Backend roles section.

You need to paste the IAM Role ARN for LambdaRole that you copied earlier in Step 6 here. Or, you can also go back to the CloudFormation Console and select iot304-lambda-stack created in the previous section. And navigate to the Outputs tab and copy the Value corresponding to the LambdaRole key.

Go back to the OpenSearch Dashboards console and paste this value under the Backend roles section.

Click on Map
Once the role has successfully been added you will see an entry named Backend role with LambdaRole’s ARN mapped to it

5. Set up the front-end search application

You’ll now host the Node.js based ‘Video Analytics App’ (front-end application) on an AWS Cloud9 IDE instance. AWS Cloud9 is a cloud-based integrated development environment (IDE) that lets you write, run, and debug your code with just a browser.

This instance was pre-created for you as part of the iot304-base-stack as mentioned in Step 5 of section 2. This application would be your interface to search for entities in video streams and verify the occurrences, timestamps and clips based on labels detected. Artifacts for this app already exist here . You just need to clone this to get started.

Navigate to AWS Cloud9 Console
Click on Open under Cloud9 IDE for the Environment named kvs-workshop-environment

You should see the Welcome page load up. The terminal is on the bottom part of the screen. On the left you have the directory options and on top the + sign gives you options to start new files, terminals, configurations etc. -

Head to the terminal and clone the Git repository. To do this, copy and paste the following into Cloud9 terminal

git clone https://github.com/aws-samples/aws-workshop-for-real-time-video-analysis

Go to the cloned directory aws-workshop-for-real-time-video-analysis in your Cloud9 IDE and expand it. Navigate to resources > code > frontend > src > config.js file. Double click on that to open it.

Take the value of the Amazon CloudFront Distribution URL and Amazon API Gateway URL copied earlier in Step 6 here. Alternatively, you can get them from the Outputs **section of the **iot304-lambda-stack **in CloudFormation . These would be the values corresponding to **MyCloudFrontDistribution and APIGWURL Keys (refer to the first screenshot, below). Paste the values in the CLOUDFRONT_URL and API_GW_URL parameter in config.js (refer to the second screenshot, below). And save the file.

In the terminal, cd into the frontend folder within the aws-workshop-for-real-time-video-analysis directory by running the following command:

cd aws-workshop-for-real-time-video-analysis/resources/code/frontend/
From this directory, run the following command in the terminal:

npm install

Now start the application by running the following command in the terminal once the previous step is completed:

npm run start
After a few seconds the compilation would complete. Post that, click on the Preview option on top.

Click on Preview Running Application. This would open a small window on the bottom-right part of the screen. Click on Pop Out Into New Window option.

This would open the application in a new tab in your browser.

6. Tying it all together

Refer to the solution components as listed below to verify all the components you have created till now-

Your laptop camera and the video feed via webRTC is setup via the Kinesis Video Streams WebRTC Test Page .
Kinesis Video Stream successfully streams video via webRTC (as confirmed in section 3 step 16).
kvslambda and the S3 bucket to store videos are created.
SNS topic and rekognitionlambda are created.
OpenSearch Domain is created and required IAM role is attached.
The front-end search app, CloudFront distribution to serve media, searchApiGateway and searchlambda are created and setup.

Now the only 2 steps that you have to do are-

Restart the Master Feed (as done earlier while testing in section 3 step 12): Open the Kinesis Video Streams WebRTC Test Page and confirm the values as selected in earlier steps. And click on Start Master button.
Trigger Step Function, created in section 4.1, to orchestrate KVS-Lambda-Rekognition flow. This is the workflow that connects 2nd and 3rd boxes in the diagram above. To do this, go to the Cloud9 environment *again. Open a new terminal by clicking on the *+ button.

Now update the field in the command mentioned below with the value of Step Function ARN obtained from section 4.1 Step 6. Or, you can also navigate to the CloudFormation Console and select iot304-lambda-stack created in the previous section and navigate to the Outputs tab to copy the Value corresponding to StepFunctionARN (refer to the image below the code snippet).

aws stepfunctions start-execution --state-machine-arn <STATE_MACHINE_ARN> --input "{\"doContinue\" : true}"

And run the command.

This will now initiate the whole workflow.

Verify the orchestration

To verify if the execution has started, you can go to the Step Functions Console .

Click on the KVSStateMachine- and scroll down to the Executions. Select the execution that would be in the Running Status and scroll to the Events section at the bottom.

TaskSucceeded would indicate that the invocations have started. After a few seconds you can go back to the Video Analytics front-end application to begin searching for entities.

7. Search Away!

Now that you have all the components created and configured, you can take this setup for a spin.

Head back to the front-end application

Refer to step 12 of section 5.12 where you had launched preview of the application in your browser. Head back to this preview.

Here you will see the following options-
Search Labels: This is where you type the entities to search for within your video streams. The application would fetch the values and video frames based on the the labels detected from your video stream.
Start and End Date: You can select the time period within which you’d want to search for the detected entities.
Try searching for something that could have been there in the video stream. If it was detected, you’ll get an option in form of a drop down menu that you can click on.

You should see the results in a similar format-

You can search for other entities similarly.

With this we have concluded the workshop!

In real-life scenarios, the webRTC stream that we simulated through our browser and system’s camera and microphone could be replaced with any other source that supports webRTC streaming. These could be mobile devices, Raspberry Pi with USB cameras or even CCTV or IP cameras that support WebRTC streaming.

You can also experiment with adding more functionalities such as setting alerts to highlight specific tagged objects once they are detected by Amazon Rekognition Video.

Refernce Output

8. Clean Up

If you undertook this workshop yourself, using your own AWS account, you should conclude by deleting resources to avoid incurring unnecessary costs. Please navigate to the AWS CloudFormation console and delete the following stacks:

iot304-lambda-stack
iot304-base-stack

Conclusion

In this project, I built an automated video monitoring system using AWS IoT and AI/ML services. By integrating AWS IoT Core, I connected video devices to the cloud for real-time data streaming.

Leveraging Amazon Rekognition, I implemented AI-driven video analytics to automatically detect objects, faces, and critical events in video feeds. The system used AWS Lambda and CloudWatch to trigger automated responses based on analysis results, creating an efficient and scalable solution for real-time monitoring.

This project demonstrated the potential of combining IoT and AI/ML to enhance video surveillance capabilities, providing practical benefits in security and operational efficiency.

Let’s connect: Linkdin, Twitter, Github

The Cloud Resume Challenge — AWS

Shubham Murti — Thu, 14 Nov 2024 11:48:08 +0000

Introduction to The Cloud Resume Challenge

Hello, I’m Shubham Murti, a cloud enthusiast and recent graduate with a Master’s in Computer Science. My journey into cloud computing began with a conversation that sparked a new career path. One of my college friends was talking about cloud technology and encouraged me to look into it. After researching, I realized that cloud computing was becoming essential across industries, with every major company moving to the cloud. It became clear to me that cloud computing would be a valuable field to enter.

I didn’t know where to start, but my friend suggested I try the Cloud Resume Challenge — a project designed to help people gain hands-on experience with cloud technologies. The challenge felt daunting initially since I was completely new to cloud. However, I decided to take it on as my first cloud project, determined to learn and complete it step by step. The challenge involves building a resume website while gaining experience with front-end development, creating a serverless backend, managing infrastructure with code, and automating deployments — all real-world cloud engineering skills.

The Cloud Resume Challenge taught me that each chunk of the project brings unique problems to solve, from setting up infrastructure to handling databases and APIs. I’m excited to share my journey through this challenge and how it helped me develop foundational cloud skills.

Chunk 0: Certification Prep

When I began the Cloud Resume Challenge, I hadn’t completed any cloud certifications. I wanted to dive into the hands-on work first to gain real-world experience before taking an exam. After finishing the project, I decided to pursue the AWS Certified Cloud Practitioner (CCP) certification to formalize my knowledge and gain a foundational credential in cloud computing.

Why Certification Matters

Completing the CCP certification after the project turned out to be an excellent choice. The practical experience from the challenge gave me a solid understanding of the core concepts, which made the certification process smoother and more meaningful. The CCP exam is widely recognized as an introductory certification, and having it on my resume helps demonstrate both my commitment to cloud and my understanding of basic cloud principles.

My Approach to Certification

To prepare for the CCP exam, I primarily used Andrew Brown’s CCP course on freeCodeCamp. It’s a comprehensive, 14-hour video course that covers all key topics in-depth. I found Andrew’s approach to be thorough, and it complemented my project experience well. Alongside the video course, I also used practice sets from AWSboy to test my knowledge. This combination of watching videos and taking practice exams really helped reinforce what I had learned during the Cloud Resume Challenge.

Resources and Tips for Success

For those preparing for the CCP exam, my advice is:

Watch Andrew Brown’s CCP Course: This 14-hour course covers all essential topics in a structured way. It’s ideal for beginners and aligns well with the exam content.
Practice as Much as Possible: Use practice sets like AWSboy or others you can find online. The more practice questions you do, the better prepared you’ll feel.
Hands-On Experience: If possible, try working on a project like the Cloud Resume Challenge before taking the exam. Practical experience can make concepts easier to understand and retain.

Completing the CCP certification after the project gave me a more well-rounded understanding of cloud fundamentals. This experience has prepared me for further cloud challenges and more advanced certifications.

Chunk 1: Building the Frontend

The first step in the Cloud Resume Challenge was to create and deploy a static website that hosts my resume. I used HTML and CSS to structure the site and hosted it on Amazon S3, followed by configuring a custom domain, DNS, and HTTPS using AWS CloudFront and Route 53.

Challenges and Solutions

Deploying a Static Site with HTTPS

Getting the static site deployed was relatively straightforward until I began configuring the domain and SSL certificate for HTTPS. I purchased a domain to personalize my site, and this required additional configuration steps to ensure secure communication (HTTPS) via AWS Certificate Manager and CloudFront. Setting up a secure HTTPS connection to my S3 bucket was particularly challenging, as I encountered issues with the SSL certificate integration and had to adjust CloudFront and Route 53 settings multiple times.

One consistent problem was establishing a secure HTTPS connection for my custom domain. Even after adding the SSL certificate, the site wouldn’t load securely. After digging into resources, I realized I needed to fine-tune the CloudFront distribution settings and confirm that the certificate was referenced correctly. The moment my site, murtishubham.click, finally loaded over HTTPS was a great relief and felt like a significant milestone.

Using My Third-Year Project Website

Since I didn’t have a dedicated portfolio site ready, I chose to use a website I built for a project during my third year of college. The idea was that once I completed automation in Chunk 4, I could refresh the deployment and replace the third-year project with a custom portfolio site, showcasing my own design. I liked the flexibility that automation brought to this, allowing me to swap out sites seamlessly. Although it was initially just an idea, the potential to automate future deployments and update the site continuously was exciting.

Looking Forward: Security Mods

I haven’t explored mods yet, but I’m interested in the security-focused mods, as my career goal is to become a cloud security engineer. I plan to delve into these mods in future chunks to further enhance my security skills.

Chunk 2: Building the API

The next phase of the Cloud Resume Challenge was to build a backend API that supports a visitor counter on my resume site. This required setting up an Amazon DynamoDB table to store the visitor count, creating an API endpoint using API Gateway, and connecting everything with AWS Lambda using Python.

Challenges and Solutions

Setting Up DynamoDB and API Gateway

Setting up the DynamoDB table to store the visitor count and configuring the API Gateway to handle requests was straightforward. Amazon’s documentation made it fairly easy to set up both services, and I was able to complete these parts without much trouble. Having a reliable storage solution like DynamoDB in place to record visitor data was a valuable introduction to using managed databases in the cloud.

Connecting DynamoDB and API Gateway with Python

The real challenge came when I needed to connect these components with Python code in AWS Lambda. Since I was still getting comfortable with Python, this part took a bit longer. Writing the backend code to communicate between the API Gateway and DynamoDB helped me gain hands-on experience with APIs and Python. Debugging and testing the function gave me a solid understanding of how data flows between different services in the cloud.

After a few days of trial and error, I was able to successfully connect the API to the database, retrieve visitor counts, and update the values in DynamoDB. Reflecting on this part of the project, I can say that working with APIs was a crucial learning experience, and I now understand that APIs are very important in cloud projects because they let different tools and services talk to each other and work together.

Key Takeaways

This chunk helped me understand the importance of APIs in cloud projects. Learning to connect multiple cloud services via API calls has given me the confidence to tackle similar tasks in the future. Thanks to this hands-on experience, I’m now much more comfortable working with APIs in the cloud.

Chunk 3: Frontend and Backend Integration

After completing the backend API and the frontend of my portfolio website, the next phase of the Cloud Resume Challenge was to integrate the two. This process involved creating a JavaScript visitor counter that would increment each time a user visited my site. Setting up the counter itself was a quick task, but designing the underlying logic took a bit more time. The real challenge was ensuring that the counter worked seamlessly with the backend.

Challenges and Solutions

Creating the Visitor Counter The first step was implementing a JavaScript counter to track the number of visitors on the site. This was a simple implementation, but the real work came when I had to figure out how to persist this data in a way that would work with the backend. This part required careful planning to ensure that the front-end could interact correctly with the server-side logic.

Testing and Debugging with Cypress Once the visitor counter was set up, it was time to test the functionality. Before starting this project, I had no experience with testing and didn’t understand its importance. However, the Cloud Resume Challenge guidebook provided me with the resources to learn how to use Cypress for testing. I spent some time experimenting with it, and soon, I was able to run basic tests to ensure the integration worked as expected. Learning to use Cypress was a valuable skill, and it helped me ensure the reliability of my site.

Dealing with CORS Issues One of the toughest hurdles I faced during the frontend and backend integration was dealing with CORS (Cross-Origin Resource Sharing) issues. CORS proved to be a tricky problem to solve in this context. It took several days of troubleshooting to get everything working correctly. However, in the end, I managed to overcome the issues, ensuring smooth communication between the frontend and backend.

Reference diagram I drew before performing the Cloud Resume Challenge

Key Takeaways

This chunk of the project helped me understand the importance of seamless integration between the frontend and backend. Learning how to implement real-time features like the visitor counter and dealing with CORS taught me valuable lessons in troubleshooting and debugging. The hands-on experience with Cypress also gave me a strong foundation in web testing, which I’ll be able to apply in future cloud projects.

Chunk 4: Automation and CI

From the beginning of the Cloud Resume Challenge (CRC), I was excited about automation, particularly the idea of using Infrastructure as Code (IaC). When I thought about how I could refresh my website and automatically see the changes after deployment, it really motivated me to dive deeper into this aspect. I was especially intrigued by using Terraform, so I eagerly began exploring how it could be integrated with AWS.

Challenges and Solutions

Infrastructure as Code (IaC)
The first step in my automation journey was learning Infrastructure as Code (IaC) using Terraform to automate resource provisioning in AWS. I spent time going through Terraform’s documentation, learning how to integrate it with AWS, and managing AWS Access Keys and Secret Keys using the AWS CLI for secure access.

CI/CD Pipeline with GitHub Actions
While working with Terraform, I set up a CI/CD pipeline using GitHub Actions to automate the deployment of both the frontend and backend of my portfolio site. This allowed me to automatically deploy updates whenever changes were made to the repository.

Automating Deployment for the Frontend and Backend
Terraform enabled me to automate the deployment of both the frontend and backend infrastructure, eliminating the need for manual configuration and deployment. This took about a week or two, but the effort was worthwhile, as it freed me up to focus more on coding.

Key Takeaways

This phase taught me the importance of automation in cloud development. Using Terraform and GitHub Actions streamlined my deployment process, making it more efficient and reliable. I now feel confident in managing cloud infrastructure using Infrastructure as Code.

Chunk 5: Writing the Blog Post

As I reach the final part of my Cloud Resume Challenge (CRC) journey, you’re reading Chunk 5, which marks the conclusion of this experience. Writing this blog post has been an opportunity to reflect on what I’ve learned and share the key takeaways from each chunk.

Key Learnings from Each Chunk

The CRC has taught me the value of hands-on practice, especially when learning AWS and cloud technologies. I’ve gained practical experience with Terraform, AWS CLI, CI/CD pipelines, and much more. While the theory was essential, it was the real-world implementation that truly deepened my understanding. Each chunk in this challenge has built on the last, and I now feel more confident working with cloud technologies.

The most rewarding part of the CRC was solving problems on my own. It pushed me to troubleshoot and find solutions, while also helping me discover what I enjoy most about cloud development, like automating deployments and working with infrastructure as code.

Conclusion and Reflection

To anyone new to the cloud or starting the Cloud Resume Challenge: don’t be discouraged if you feel overwhelmed. Stick with it, and in the end, you’ll complete the challenge and gain valuable skills that will serve you in the future. The guidebook has helped me tremendously, but the real learning came from doing.

Lastly, I’d like to thank Forrest Brazeal for creating the Cloud Resume Challenge. It has helped me in so many ways, and I highly recommend it to anyone looking to learn cloud technologies. Good luck to those starting out — you’ve got this!

Urls
Click here to view the portfolio developed.

Explore my GitHub repository.

Let’s connect: Linkdin, Twitter, Github

Serverless Data Processing on AWS : AWS Project

Shubham Murti — Wed, 13 Nov 2024 14:54:59 +0000

Introduction

Businesses looking for meaningful insights in the big data world depend on effective real-time stream processing and analysis. This project uses AWS services including Amazon Kinesis, AWS Lambda, Amazon S3, Amazon DynamoDB, Amazon Cognito, and Amazon Athena to construct a serverless data processing architecture. The solution provides a robust system for real-time data insights by utilizing these services to ingest, process, and store data with high scalability and low infrastructure administration.

Tech Stack

The serverless solution uses the following AWS services and technologies:

AWS Lambda: Triggers upon events from Kinesis streams, enabling real-time data processing.
Amazon Kinesis Data Analytics: Aggregates and transforms streaming data on the fly.
Amazon DynamoDB: Stores processed data in a fast, scalable NoSQL database.
Amazon S3: Serves as a scalable, durable storage location for raw and processed data.
Amazon Kinesis Data Firehose: Streams raw data directly to S3 for archival and further processing.
Amazon Athena: Allows ad-hoc querying directly on data stored in S3.
Amazon Cognito: Manages user authentication and authorization for secure access to resources.

Prerequisites

To follow along with this project, ensure you have:

Basic AWS Knowledge: Familiarity with Lambda, Kinesis, S3, and IAM services.
AWS SDKs and CLI: Installed and configured for command-line interactions with AWS services.
IAM Permissions: Set up permissions for Lambda, Kinesis, S3, and DynamoDB.

Problem Statement or Use Case

A fictive company Wild Rydes introduces an innovative transportation service that offers unicorn rydes to help people to get to their destination faster and hassle-free. Each unicorn is equipped with a sensor that reports its location and vital signs. During this workshop, we’ll build infrastructure to enable operations personnel at Wild Rydes to monitor the health and status of their unicorn fleet. We’ll use AWS to build applications that process and visualize the unicorn data in real-time.

Architecture Diagram

Below is a visual representation of the serverless data processing architecture, showing data flow from ingestion through Kinesis, processing with Lambda, and storage in DynamoDB and S3.

Component Breakdown

Amazon Kinesis Data Streams: Collects data from various sources, enabling the flow of real-time data.
AWS Lambda: Processes data in real-time when triggered by Kinesis events, transforming the data before it’s stored.
Amazon Kinesis Data Analytics: Provides on-the-fly data aggregation and transformation to gain insights before storage.
Amazon DynamoDB: Stores processed data for fast retrieval, especially suitable for structured data.
Amazon S3: Holds raw data and processed datasets, ensuring durability and scalability.
Amazon Kinesis Data Firehose: Streams data directly to S3 for long-term storage and archiving.
Amazon Athena: Allows for ad-hoc querying on the data stored in S3, ideal for data analysis without data movement.
Amazon Cognito: Ensures secure user authentication, managing permissions and access control.

Step-by-Step Implementation

AWS Cloud9 IDE

AWS Cloud9 is a cloud-based integrated development environment (IDE) that lets you write, run, and debug your code with just a browser. It includes a code editor, debugger, and terminal. Cloud9 comes pre-packaged with essential tools for popular programming languages and the AWS Command Line Interface (CLI) pre-installed so you don’t need to install files or configure your laptop for this workshop. Your Cloud9 environment will have access to the same AWS resources as the user with which you logged into the AWS Management Console.

Take a moment now and setup your Cloud9 development environment.

✅ Step-by-step Instructions

Go to the AWS Management Console, click Services then select Cloud9 under Developer Tools.
Click Create environment.
Enter Development into Name and optionally provide a Description.
Click Next step.
You may leave Environment settings at their defaults of launching a new t2.micro EC2 instance which will be paused after 30 minutes of inactivity.
Click Next step.
Review the environment settings and click Create environment. It will take several minutes for your environment to be provisioned and prepared.
Once ready, your IDE will open to a welcome screen. Below that, you should see a terminal prompt similar to:

If you are running this workshop at an AWS event: Paste here the Credentials/CLI Snippets you copied before to configure your environment with your credentials. If you are running it in your own account, you can use the default profile Cloud9 sets up for you. You can check it via cat ~/.aws/credentials.

Now you can run AWS CLI commands in here just like you would on your local computer. Verify that your user is logged in by running aws sts get-caller-identity.

aws sts get-caller-identity

You’ll see output indicating your account and user information:

Admin:~/environment $ aws sts get-caller-identity

{
    "Account": "123456789012",
    "UserId": "AKIAI44QH8DHBEXAMPLE",
    "Arn": "arn:aws:iam::123456789012:user/Alice"
}

Keep your AWS Cloud9 IDE opened in a tab throughout this workshop as you’ll use it for activities like building and running a sample app in a Docker container and using the AWS CLI.

Command Line Clients

The modules utilize two command-line clients to simulate and display sensor data from the unicorns in the fleet. These are small programs written in the Go Programming Language. The below instructions in the Installation section walks through downloading pre-built binaries, but you can also download the source and build it manually:

Producer

The producer generates sensor data from a unicorn taking a passenger on a Wild Ryde. Each second, it emits the location of the unicorn as a latitude and longitude point, the distance traveled in meters in the previous second, and the unicorn’s current level of magic and health points.

Consumer

The consumer reads and displays formatted JSON messages from an Amazon Kinesis stream which allow us to monitor in real-time what’s being sent to the stream. Using the consumer, you can monitor the data the producer and your applications are sending.

Installation

Switch to the tab where you have your Cloud9 environment opened.
Download and unpack the command line clients by running the following command in the Cloud9 terminal:

curl -s https://data-processing.serverlessworkshops.io/client/client.tar | tar -xv

This will unpack the consumer and producer files to your Cloud9 environment.

⭐ Tips

💡 Keep an open scratch pad in Cloud9 or a text editor on your local computer for notes. When the step-by-step directions tell you to note something such as an ID or Amazon Resource Name (ARN), copy and paste that into the scratch pad.

⭐ Recap

🔑 Use a unique personal, development AWS Account, event engine or EventBox

🔑 Use one of the US East (N. Virginia), US West (Oregon), EU* (Ireland, London, Frankfurt) Regions if using your own AWS account.

🔑 Keep your AWS Cloud9 IDE opened in a tab

Real-time Data Streaming

In this module, you’ll create an Amazon Kinesis stream to collect and store sensor data from our unicorn fleet. Using the provided command-line clients, you’ll produce sensor data from a unicorn on a Wild Ryde and read from the stream. Lastly, you’ll use the unicorn dashboard to plot our unicorns on a map and watch their status in real-time. In subsequent modules you’ll add functionality to analyze and persist this data using Amazon Kinesis Data Analytics, AWS Lambda, and Amazon DynamoDB.

Overview

The architecture for this module involves an Amazon Kinesis stream, a producer, and a consumer.

Our producer is a sensor attached to a unicorn currently taking a passenger on a ride. This sensor emits data every second including the unicorn’s current location, distance traveled in the previous second, and magic points and hit points so that our operations team can monitor the health of the unicorn fleet from Wild Rydes headquarters.

The Amazon Kinesis stream stores data sent by the producer and provides an interface to allow consumers to process and analyze those data. Our consumer is a simple command-line utility that tails the stream and outputs the data points from the stream in effectively real-time so we can see what data is being stored in the stream. Once we send and receive data from the stream, we can use the unicorn dashboard to view the current position and vitals of our unicorn fleet in real-time.

Implementation

❗ Ensure you’ve completed the setup guide before beginning the workshop.

1. Create an Amazon Kinesis stream

Use the Amazon Kinesis Data Streams console to create a new provisioned stream named wildrydes with 1 shard.

A Shard is the base throughput unit of an Amazon Kinesis data stream. One shard provides a capacity of 1MB/sec data input and 2MB/sec data output. One shard can support up to 1000 PUT records per second. You will specify the number of shards needed when you create a data stream. For example, if we create a data stream with four shards then this data stream has a throughput of 4MB/sec data input and 8MB/sec data output, and allows up to 4000 PUT records per second. You can monitor shard-level metrics in Amazon Kinesis Data Streams and add or remove shards from your data stream dynamically as your data throughput changes by resharding the data stream.

At re:Invent 2021, AWS introduced a new capacity mode for Kinesis Data Streams called Kinesis Data Streams On-Demand. The new mode is capable of serving gigabytes of write and read throughput per minute without capacity planning. During this workshop, we will use the provisioned mode with shard capacity planning for educational purposes. For your environments, you should consider using on-demand capacity mode based on your needs and cost considerations.

✅ Step-by-step Instructions

Go to the AWS Management Console, click Services then select Kinesis under Analytics.
Click Get started if prompted with an introductory screen.
Click Create data stream.
Enter wildrydes into Kinesis stream name
Select the Provisioned option and enter 1 into Number of shards, then click Create Kinesis stream.
Within 60 seconds, your Kinesis stream will be ACTIVE and ready to store real-time streaming data.

2. Produce messages into the stream

Use the command-line producer to produce messages into the stream.

✅ Step-by-step Instructions

Switch to the tab where you have your Cloud9 environment opened.
In the terminal, run the producer to start emitting sensor data to the stream.

./producer

The producer emits a message a second to the stream and prints a period to the screen.

$ ./producer ..................................................

In the Amazon Kinesis Streams console, click on wildrydes and click on the Monitoring tab.
After several minutes, you will see the Put Record Success (percent) — Average graph begin to record a single put a second. Keep the producer running till end of the Lab 2, so that you can see the unicorns flying in action.

3. Read messages from the stream

✅ Step-by-step Instructions

While the producer is running, switch to the tab where you have your Cloud9 environment opened.
Hit the (+) button and click New Terminal to open a new terminal tab.
Run the consumer to start reading sensor data from the stream.

./consumer

The consumer will print the messages being sent by the producer:

{ "Name": "Shadowfax", "StatusTime": "2017-06-05 09:17:08.189", "Latitude": 42.264444250051326, "Longitude": -71.97582884770408, "Distance": 175, "MagicPoints": 110, "HealthPoints": 150 } { "Name": "Shadowfax", "StatusTime": "2017-06-05 09:17:09.191", "Latitude": 42.265486935100476, "Longitude": -71.97442977859625, "Distance": 163, "MagicPoints": 110, "HealthPoints": 151 }

4. Create an identity pool for the unicorn dashboard

Create an Amazon Cognito identity pool to grant unauthenticated users access to read from your Kinesis stream. Note the identity pool ID for use in the next step.

✅ Step-by-step directions

Go to the AWS Management Console, click Services then select Cognito under Security, Identity & Compliance.
Click Manage Identity Pools.
Click Create new identity pool. [This is not necessary if you do not have any identity pool yet.]
Enter wildrydes into Identity pool name.
Tick the Enable access to unauthenticated identities checkbox.
Click Create Pool.
Click Allow which will create authenticated and unauthenticated roles for your identity pool.
Click Go to Dashboard.
Click Edit identity pool in the upper right hand corner.
Note the Identity pool ID for use in a later step.

5. Grant the unauthenticated role access to the stream

Add a new policy to the unauthenticated role to allow the dashboard to read from the stream to plot the unicorns on the map.

✅ Step-by-step directions

Go to the AWS Management Console, click Services then select IAM under Security, Identity & Compliance.
Click on Roles in the left-hand navigation.
Click on the Cognito_wildrydesUnauth_Role.
Click Add inline policy (Add permissions -> Create inline policy).
Click on Choose a service and click Kinesis.
Click on Actions.
Tick the Read and List permissions checkboxes.
Under Resources you will limit the role to the wildrydes stream and consumer.
Click Add ARN next to consumer.
In the Add ARN(s) dialog box, enter the following information:

the region you’re using in Region (e.g. us-east-1)
your Account ID in Account
- in Stream type
wildrydes in Stream name
- in Consumer name
- in Consumer creation timestamp

Click Add.
Click Add ARN next to stream.
In the Add ARN(s) dialog box, enter the following information:

the region you’re using in Region (e.g. us-east-1)
your Account ID in Account
wildrydes in Stream name

Click Add.

Click Review policy.
Enter WildrydesDashboardPolicy in Name.
Click Create policy.

6. View unicorn status on the dashboard

Use the Unicorn Dashboard to see the unicorn on a real-time map. You may need to zoom out to find the unicorn. Please double check that producer and consumer are both running if you can’t find it.

✅ Step-by-step directions

Open the Unicorn Dashboard.
Enter the Cognito Identity Pool ID you noted in step 4 and click Start.

Validate that you can see the unicorn on the map.If you can not see the unicorn, please go back to Cloud9 and run ./producer.

Click on the unicorn to see more details from the stream and compare with the consumer output.

The speed is calculated internally based on the difference of the longitude and latitude values.

7. Experiment with the producer

Stop and start the producer while watching the dashboard and the consumer. Start multiple producers with different unicorn names.

Stop the producer by pressing Control + C and notice the messages stop and the unicorn disappear after 30 seconds.
Start the producer again and notice the messages resume and the unicorn reappear.
Hit the (+) button and click New Terminal to open a new terminal tab.
Start another instance of the producer in the new tab. Provide a specific unicorn name and notice data points for both unicorns in consumer’s output:

./producer -name Bucephalus

Check the dashboard and verify you see multiple unicorns.

⭐ Recap

🔑 Amazon Kinesis makes it easy to collect, process, and analyze real-time, streaming data so you can get timely insights and react quickly to new information.

🔧 In this module, you’ve created an Amazon Kinesis stream and used it to store and visualize data from a simulated fleet of unicorns.

Stream Processing and analytics with AWS Lambda

In this module, you’ll use AWS Lambda to process data from the wildrydes Amazon Kinesis stream created earlier. We’ll create and configure a Lambda function to read from the stream and write records to an Amazon DynamoDB table as they arrive. We will also explore a few error handling mechanisms when there are poison pill messages in the stream. Finally, We will learn how to do stream analytics with AWS Lambda.

Our target architecture looks as follows:

Setup

Implementation

In this module, you’ll setup all of the resources needed to support processing records from the Kinesis Data Stream wildrydes including a Dynamo DB table, a Lambda function, an IAM role, a Kinesis Data Stream, and a SQS queue.

Resources

1. Create an Amazon DynamoDB table

Use the Amazon DynamoDB console to create a new DynamoDB table. This table is used to store the unicorn data from the AWS Lambda function. We will call your table UnicornSensorData and give it a Partition key called Name of type String and a Sort key called StatusTime of type String. Use the defaults for all other settings.

After you’ve created the table, note the Amazon Resource Name (ARN) for use in the next section.

✅ Step-by-step Instructions

Go to the AWS Management Console, choose Services then select DynamoDB under Database. Alternatively, you can use the search bar and type DynamoDB in the search dialog box.
Click Create table.
Enter UnicornSensorData for the Table name.
Enter Name for the Partition key and select String for the key type.
Enter StatusTime for the Sort key and select String for the key type.
Leave the Use default settings box checked and choose Create.

Once the table is created, Click on the hyperlink on the table name. This will take you to the new screen. Under General information, You will see Amazon Resource Name (ARN). Copy and save the Amazon Resource Name (ARN) in the scratchpad. You will use this when creating IAM role.

2. Create an SQS On-Error Destination Queue

Use the Amazon SQS console to create a new queue nammed wildrydes-queue. Your Lambda function will send messages to this queue when the processing is failed based on the retry settings.

After you’ve created the queue, note the Amazon Resource Name (ARN) for use in later sections.

✅ Step-by-step Instructions

Go to the AWS Management Console, choose Services then select Simple Queue Service under Application Integration. Alternatively, you can use the search bar and type Simple Queue Service in the search dialog box.
Click the orange Create queue button
For the Name field enter wildrydes-queue
Leave the rest of the options as the defaults and click “Create queue”

Copy and save the ARN of the SQS queue in the scratchpad as you will need it for Lambda configuration

3. Create an IAM role for your Lambda function

Use the IAM console to create a new role. Name it WildRydesStreamProcessorRole and select Lambda for the role type. Create a policy that allows dynamodb:BatchWriteItem access to the DynamoDB table created in the last section and sqs:SendMessage to send failed messages to DLQ and attach it to the new role. Also, Attach the managed policy called AWSLambdaKinesisExecutionRole to this role in order to grant permissions for your function to read from Amazon Kinesis streams and to log to Amazon CloudWatch Logs.

✅ Step-by-step Instructions

From the AWS Console, click on Services and then select IAM in the Security, Identity & Compliance section. Alternatively, you can use the search bar and type IAM in the search dialog box.
Select Policies from the left navigation and then click Create policy.
Using the Visual editor, we’re going to create an IAM policy to allow our Lambda function access to the DynamoDB table created in the last section. To begin, click Service, begin typing DynamoDB in Find a service, and click DynamoDB.
Click Action, begin typing BatchWriteItem in Filter actions, and tick the BatchWriteItem checkbox.
Click Resources, click Add ARN in table, Copy the ARN of the DyanamoDB table from the scratchpad and paste it in Specify ARN for table. Alternatively, you can construct the ARN of the DynamoDB table you created in the previous section by specifying the Region, Account, and Table Name.
In Region, enter the AWS Region in which you created the DynamoDB table in the previous section, e.g.: us-east-1.
In Table Name, enter UnicornSensorData.
You should see your ARN in the Specify ARN for table field and it should look similar to:

Click Add.
When completed, your console should look similar to this:

Next we are going to add permissions to allow the Lambda function access to the SQS On-Error Destination Queue. Click Add additional permissions click Service begin typing SQS in Find a service and click SQS.
Click Action begin typing SendMessage in Filter actions, and tick the SendMessage checkbox.
Click Resources, click Add ARN in queue. Copy the ARN of the SQS queue from the scratchpad and paste it in Specify ARN for queue. Alternatively, you can construct the ARN of the SQS queue you created in the previous section by specifying the Region, Account, and Queue Name.

In Region, enter the AWS Region in which you created the SQS queue in the previous section, e.g.: us-east-1. In Queue Name, enter wildrydes-queue. You should see your ARN in the Specify ARN for queue field and it should look similar to:

Click Next: Tags**.
Click Next: Review**.
Enter WildRydesDynamoDBWritePolicy in the Name field.
Click Create policy.
Select Roles from the left navigation and then click Create role.
Click Lambda for the role type from the AWS service section.
Click Next: Permissions.
Begin typing AWSLambdaKinesisExecutionRole in the Filter text box and check the box next to that role.
Begin typing WildRydesDynamoDBWritePolicy in the Filter text box and check the box next to that policy.
Click Next.
Enter WildRydesStreamProcessorRole for the Role name.
Click Create role.
Begin typing WildRydesStreamProcessorRole in the Search text box

Click on WildRydesStreamProcessorRole and it should look similar to:

4. Create a Lambda function to process the stream

Create a Lambda function called WildRydesStreamProcessor that will be triggered whenever a new record is avaialble in the wildrydes stream. Use the provided index.js implementation for your function code. Create an environment variable with the key TABLE_NAME and the value UnicornSensorData. Configure the function to use the WildRydesStreamProcessor role created in the previous section.

✅ Step-by-step Instructions

Go to the AWS Management Console, choose Services then select Lambda under Compute. Alternatively, you can use the search bar and type Lambda in the search dialog box.
Click Create function.
Enter WildRydesStreamProcessor in the Function name field.
Select Node.js 14.x from Runtime.
Select WildRydesStreamProcessorRole from the Existing role dropdown.

Click Create function.
Scroll down to the Code source section.
Copy and paste the JavaScript code below into the code editor.

"use strict"; const AWS = require("aws-sdk"); const dynamoDB = new AWS.DynamoDB.DocumentClient(); const tableName = process.env.TABLE_NAME; // Entrypoint for Lambda Function exports.handler = function (event, context, callback) { const requestItems = buildRequestItems(event.Records); const requests = buildRequests(requestItems); Promise.all(requests) .then(() => callback(null, Delivered ${event.Records.length} records) ) .catch(callback); }; // Build DynamoDB request payload function buildRequestItems(records) { return records.map((record) => { const json = Buffer.from(record.kinesis.data, "base64").toString( "ascii" ); const item = JSON.parse(json); return { PutRequest: { Item: item, }, }; }); } function buildRequests(requestItems) { const requests = []; // Batch Write 25 request items from the beginning of the list at a time while (requestItems.length > 0) { const request = batchWrite(requestItems.splice(0, 25)); requests.push(request); } return requests; } // Batch write items into DynamoDB table using DynamoDB API function batchWrite(requestItems, attempt = 0) { const params = { RequestItems: { [tableName]: requestItems, }, }; let delay = 0; if (attempt > 0) { delay = 50 * Math.pow(2, attempt); } return new Promise(function (resolve, reject) { setTimeout(function () { dynamoDB .batchWrite(params) .promise() .then(function (data) { if (data.UnprocessedItems.hasOwnProperty(tableName)) { return batchWrite( data.UnprocessedItems[tableName], attempt + 1 ); } }) .then(resolve) .catch(reject); }, delay); }); }

Now, You will be adding the DynamoDB table name as an environment variable. In the Configuration tab, select the Environment variables section.

Click Edit and then Add environment variable

Enter TABLE_NAME in Key and UnicornSensorData in Value.
Add another environment variable called AWS_NODEJS_CONNECTION_REUSE_ENABLED in Key and 1 in Value. This setting helps to reuse TCP connection. You can learn more here
click Save.

Now, You will add the event source mapping(ESM) for AWS Lambda to integrate with Kinesis. Scroll up and click on Add Trigger from the Function overview section.

In the Trigger configuration section, select Kinesis.
Select wildrydes from Kinesis stream.
Leave Batch size set to 100 and Starting position set to Latest.
Open the Additional settings section
Under On-failure destination add the ARN of the wildrydes-queue SQS queue
Make sure Enable trigger is checked.
Click Add.

Go back to Code Tab and Deploy the Lambda function — the screen will provide a message on successful deployment.

⭐ Recap

🔑 You can subscribe Lambda functions to automatically read batches of records off your Kinesis stream and process them if records are detected on the stream.

🔧 In this module, you’ve setup a DynamoDB table for storing unicorn data, a Dead Letter Queue(DLQ) for recieving failed messages and a Lambda function to read data from Kinesis Data Stream and store in DynamoDB.

Stream Processing

Implementation

In the SetUp section, we set up the all the necessary services and roles required to read a message from Amazon Kinesis Data Stream wildrydes by the Lambda function WildRydesStreamProcessor. This function processes the records and inserts the data into Amazon DynamoDB table UnicornSensorData.

Lambda reads records from the data stream and invokes your function synchronously with an event that contains stream records. Lambda reads records in batches and invokes your function to process records from the batch. Each batch contains records from a single shard/data stream. Follow this link to learn more about this integration

In this module, you’ll send streaming data to Amazon Kinesis Data Stream, wildrydes using the producer.go library and use the AWS Console to monitor Lambda’s processing of records from the wildrydes stream and query the results in Amazon DynamoDB.

1. Produce streaming data

Return to you AWS Cloud9 instance and run the producer to start emitting sensor data to the stream with a unique unicorn name.

./producer -name Shadowfax -stream wildrydes -msgs 20

2. Verify Lambda execution

Verify that the trigger is properly executing the Lambda function. View the metrics emitted by the function and inspect the output from the Lambda function.

✅ Step-by-step Instructions

Return to the AWS Lambda function console. Click on the Monitor tab and explore the metrics available to monitor the function.

Click on Logs to explore the function’s log output.

Click on View logs in CloudWatch to explore the logs in CloudWatch for the log group /aws/lambda/WildRydesStreamProcessor

The log groups can take a while to create so if you see “Log Group Does Not Exist” when pressing “View Logs” in CloudWatch then just wait a few more minutes before hitting refresh.

3. Query the DynamoDB table

Using the AWS Management Console, query the DynamoDB table for data for a specific unicorn. Use the producer to create data from a distinct unicorn name and verify those records are persisted.

✅ Step-by-step Instructions

Click on Services then select DynamoDB in the Database section.
Click Tables from the left-hand navigation
Click on UnicornSensorData.
Click on the Explore table items button at the top right. Here you should see the Unicorn data for which you’re running a producer.

By default, There is one to one mapping between Kinesis shard to Lambda function invocation. You can configure the ParallelizationFactor setting to process one shard of a Kinesis with more than one Lambda invocation simultaneously. If you increase the number of concurrent batches per shard, Lambda still ensures in-order processing at the partition-key level. Follow the link to learn more about parallelization.

⭐ Recap

🔑 You can subscribe Lambda functions to automatically read batches of records off your Kinesis stream and process them if records are detected on the stream.

🔧 In this module, you’ve created a Lambda function that reads from the Kinesis Data Stream wildrydes and saves each row to DynamoDB.

Error Handling

Implementation

In this module, you’ll setup AWS Lambda to process data and handle errors when processing data from the wildrydes created earlier. There are couple of approaches for error handling.

Resources

1. Error Handling with Retry Settings

AWS Lambda can reprocess batches of messages from Kinesis Data Streams when an error occurs in one of the items in the batch. You can configure the number of retries by configuring Retry attempts and/or Maximum age of record. The batch will be retried until the number of retry attempts or until the expiration of the batch. You can also configure On-failure destination which will be used by Lambda to send metadata of your failed invocation. You can send this metadata of the failed invocation to either an Amazon SQS queue or an Amazon SNS topic. Typically there are two kinds of errors in the data stream. One category belongs to transient errors which are temporary in nature and are successfully processed with retry logic. Second category belongs to Poison Pill (either data quality / data that generates an exception in Lambda code) which are permanent in nature. In this case Lambda retries for the configured retry attempts and then discards the records to the On-failure destination.

In order to simulate poison pill message, We will introduce an error data in streaming data and throw an error when it is found in the message. In real world, this may be a validation or a call to another service which expects certain information in the record. This is the code change that will be introduced in the Lambda function

if (item.InputData.toLowerCase().includes(errorString)) {
    console.log("Error record is = ", item);
    throw new Error("kaboom");
}

✅ Step-by-step Instructions

Go to the AWS Management Console, choose Services then select Lambda under Compute. Alternatively, you can use the search bar and type Lambda in the search dialog box.
Click the WildRydesStreamProcessor function
Scroll down to the Function Code section.
Double click the index.js file to open it in the editor
Copy and paste the JavaScript code below into the code editor, replacing all of the existing code.

"use strict"; const AWS = require("aws-sdk"); const dynamoDB = new AWS.DynamoDB.DocumentClient(); const tableName = process.env.TABLE_NAME; // This is used to mimic poison pill messages const errorString = "error"; // Entrypoint for Lambda Function exports.handler = function (event, context, callback) { console.log( "Number of Records sent for each invocation of Lambda function = ", event.Records.length ); const requestItems = buildRequestItems(event.Records); const requests = buildRequests(requestItems); Promise.all(requests) .then(() => callback(null, Delivered ${event.Records.length} records) ) .catch(callback); }; // Build DynamoDB request payload function buildRequestItems(records) { return records.map((record) => { const json = Buffer.from(record.kinesis.data, "base64").toString( "ascii" ); const item = JSON.parse(json); //Check for error and throw the error. This is more like a validation in your usecase if (item.InputData.toLowerCase().includes(errorString)) { console.log("Error record is = ", item); throw new Error("kaboom"); } return { PutRequest: { Item: item, }, }; }); } function buildRequests(requestItems) { const requests = []; // Batch Write 25 request items from the beginning of the list at a time while (requestItems.length > 0) { const request = batchWrite(requestItems.splice(0, 25)); requests.push(request); } return requests; } // Batch write items into DynamoDB table using DynamoDB API function batchWrite(requestItems, attempt = 0) { const params = { RequestItems: { [tableName]: requestItems, }, }; let delay = 0; if (attempt > 0) { delay = 50 * Math.pow(2, attempt); } return new Promise(function (resolve, reject) { setTimeout(function () { dynamoDB .batchWrite(params) .promise() .then(function (data) { if (data.UnprocessedItems.hasOwnProperty(tableName)) { return batchWrite( data.UnprocessedItems[tableName], attempt + 1 ); } }) .then(resolve) .catch(reject); }, delay); }); }

Click Deploy to deploy the changes to the WildRydesStreamProcessor Lambda function.
Remove the existing Kinesis Data Stream mapping by clicking the Configuration Tab above the code editor. (This step is needed only if there is an existing Kinesis Data Stream mapping or any other Event Source Mapping present for the Lambda function). In the Configuration Tab, Select the Kinesis:wildrydes (Enabled). If the trigger is not enabled, press refresh. Delete the trigger.

Add a new Kinesis Data Stream mapping by clicking the Configuration Tab.
Select the Triggers section and Add trigger button.
Select Kinesis for the service and wildrydes from Kinesis Stream.
Set the Batch size set to 10 and Starting position set to Latest. This small batch size will help us monitor the AWS Lambda error handling clearly from Cloudwatch logs
Set the Batch window to 15. This window will help you batch the incoming messages by waiting for 15 seconds. By default, AWS Lambda will poll messages from Amazon Kinesis Data Stream every 1 sec.
Open the Additional settings section.
Under On-failure destination add the ARN of the wildrydes-queue SQS queue.
Change Retry attempts to 2 and Maximum age of record to 60.

Leave the rest of the fields to default values.
Click Add to create the trigger.
Click the refresh button until creation is complete and the function shows as Enabled. (You may need to hit the refresh button to refresh the status)

Return to AWS Cloud9 environment and Insert data into Kinesis Data Stream by running producer binary.

./producer -stream wildrydes -error yes -msgs 9

Return to the AWS Lambda function console. Click on the Monitor tab and explore the metrics available to monitor the function.
Click on View logs in CloudWatch to explore the logs in CloudWatch for the log group /aws/lambda/WildRydesStreamProcessor
In the logs you can observe that, there will be error and the same batch will be retried twice ( as we configured retry-attempts to 2)

Since the entire batch failed, you should not notice any new records in the DynamoDB table UnicornSensorData. You will see only the 20 records that are inserted from the previous section.
Optionally you can Monitor SQS Queue by following the following steps.
Go to the AWS Management Console, choose Services then search for Simple Queue Service and select the Simple Queue Service.
There will be a message in the wildrydes-queue. This is the discarded batch that had one permanent error in the batch.
Click wildrydes-queue.
Click Send and recieve messages on the top right corner
Click Poll for messages on the bottom right corner
You’ll observe one message. Click the message ID and choose the Body tab. You can see all the details of the discarded batch. Notice that entire batch of messages(Size:9) is discarded even through there was only one error message.
Check the checkbox beside the message ID
Click Delete button. This will delete the message from the SQS queue. This step is optional and is needed only to keep the SQS queue empty.

2. Error Handling with Bisect On Batch settings

The retry setting we had before discards entire batch of records even if there is one bad record in the batch. Bisect On Batch error handling feature of AWS Lambda splits the batch into two and retries the half-batches separately. The process continues recursively until there is a single item in a batch or messages are processed successfully.

There are no code changes to the WildRydesStreamProcessor Lambda function. The only change is around setting Kinesis Data Stream configuration. Follow the below steps to remove and add a new Kinesis Data Stream mapping.
Remove the existing Kinesis Data Stream mapping by clicking the Configuration Tab above the code editor. (This step is needed only if there is an existing Kinesis Data Stream mapping or any other Event Source Mapping present for the Lambda function). In the Configuration Tab select the Kinesis:wildrydes (Enabled) and Delete the trigger.

Add a new Kinesis Data Stream mapping by clicking the Configuration Tab.
Select the Triggers section and Add trigger button.
Select Kinesis for the service and wildrydes from Kinesis Stream.
Set the Batch size set to 10 and Starting position set to Latest. This small batch size will help us monitor the AWS Lambda error handling clearly from Cloudwatch logs
Set the Batch window to 15. Again, this window will help you batch the incoming messages by waiting for 15 seconds.
Open the Additional settings section.
Under On-failure destination add the ARN of the wildrydes-queue SQS queue.
Change Retry attempts to 2 and Maximum age of record to 60.
Check the box Split batch on error.

Leave the rest of the fields to default values.
Click Add to create the trigger.
Click the refresh button until creation is complete and the function shows as Enabled. (You may need to hit the refresh button to refresh the status)

Return to AWS Cloud9 environment and Insert data into Kinesis Data Stream by running producer binary.

./producer -stream wildrydes -error yes -msgs 9
Return to the AWS Lambda function console. Click on the Monitor tab and explore the metrics available to monitor the function.
Click on View logs in CloudWatch to explore the logs in CloudWatch for the log group /aws/lambda/WildRydesStreamProcessor
In the logs you can observe that, there will be error and the same batch will be split into two halves and processed. This splitting continues recursively until there is a single item or messages are processed successfully.

Since Bisect-On-Batch splits the batch and processes records,you should notice new records in the DynamoDB table UnicornSensorData. There should be a total of 28 items in UnicornSensorData ( 1 record is an error record ).
Optionally you can Monitor SQS Queue by following the below steps.
Go to the AWS Management Console, choose Services then search for Simple Queue Service and select the Simple Queue Service.
There will be a message in the wildrydes-queue. This is the discarded batch that had one permanent error in the batch.
Click wildrydes-queue.
Click Send and recieve messages on the top right corner
Click Poll for messages on the bottom right corner
You’ll observe one message. Click the message ID and choose the Body tab. You can see all the details of the discarded batch. Notice that this time only one message is discarded.
Check the checkbox beside the message ID
Click Delete button. This will delete the message from the SQS queue. This step is optional and is needed only to keep the SQS queue empty.

Analytics with Tumbling Windows

In this module, you’ll use the tumbling window feature of AWS Lambda to aggregate sensor data from a unicorn in the fleet in real-time. The Lambda function will read from the Amazon Kinesis stream, calculate the total distance traveled per minute for a specific unicorn, and store the results in an Amazon DynamoDB table.

Tumbling windows are distinct time windows that open and close at regular intervals. By default, Lambda invocations are stateless — you cannot use them for processing data across multiple continuous invocations without an external database. However, with tumbling windows, you can maintain your state across invocations. This state contains the aggregate result of the messages previously processed for the current window. Your state can be a maximum of 1 MB per shard. If it exceeds that size, Lambda terminates the window early. Each record of a stream belongs to a specific window. A record is processed only once, when Lambda processes the window that the record belongs to. In each window, you can perform calculations, such as a sum or average, at the partition key level within a shard.

Resources

Implementation

1. Create a DynamoDB table

Use the Amazon DynamoDB console to create a new DynamoDB table. Call your table UnicornAggregation and give it a Partition key called name of type String and a Sort key called windowStart of type String. Use the defaults for all other settings.

✅ Step-by-step Instructions

Go to the AWS Management Console, choose Services then select DynamoDB under Database.
Click Create table.
Enter UnicornAggregation for the Table name.
Enter name for the Partition key and select String for the key type.
Enter windowStart for the Sort key and select String for the key type.
Leave the Default settings box checked and choose Create table.

2. Create an IAM Role for the Lambda function

Use the IAM console to create a new role. Name it unicorn-aggregation-role and select Lambda for the role type. Attach the managed policy called AWSLambdaKinesisExecutionRole to this role in order to grant permissions for your function to read from Amazon Kinesis streams and to log to Amazon CloudWatch Logs. Create a policy that allows dynamodb:PutItem access to the DynamoDB table created in the last section and attach it to the new role.

✅ Step-by-step Instructions

From the AWS Console, click on Services and then select IAM in the Security, Identity & Compliance section.
Select Policies from the left navigation and then click Create Policy.
Using the Visual editor, we’re going to create an IAM policy to allow our Lambda function access to the DynamoDB table created in the last section. To begin, click Service, begin typing DynamoDB in Find a service, and click DynamoDB.
Type PutItem in Filter actions, and tick the PutItem checkbox.
Click Resources, click Add ARN in table, and construct the ARN of the DynamoDB table you created in the previous section by specifying the Region, Account, and Table Name.

In Region, enter the AWS Region in which you created the DynamoDB table in the previous section, e.g.: us-east-1. In Account, enter your AWS Account ID which is a twelve digit number, e.g.: 123456789012. To find your AWS account ID number in the AWS Management Console, click on Support in the navigation bar in the upper-right, and then click Support Center. Your currently signed in account ID appears in the upper-right corner below the Support menu. In Table Name, enter UnicornAggregation. You should see your ARN in the Specify ARN for table field and it should look similar to:

Click Add.

Click Next: Tags.
Click Next: Review.
Enter unicorn-aggregation-ddb-write-policy in the Name field.
Click Create policy.
Select Roles from the left navigation and then click Create role.
Click Lambda for the role type from the AWS service section.
Click Next: Permissions.
Begin typing AWSLambdaKinesisExecutionRole in the Filter text box and check the box next to that role.
Begin typing unicorn-aggregation-ddb-write-policy in the Filter text box and check the box next to that role.
Click Next.
Enter unicorn-aggregation-role for the Role name.
Click Create role.
Begin typing unicorn-aggregation-role in the Search text box.

Click on unicorn-aggregation-role and it should look similar to:

3. Create a Lambda function

Use the Lambda console to create a Lambda function called WildRydesAggregator that will be triggered whenever a new record is available in the wildrydes stream. Use the provided index.js implementation for your function code. Create an environment variable with the key TABLE_NAME and the value UnicornAggregation. Configure the function to use the unicorn-aggregation-role role created in the previous sections.

Go to the AWS Management Console, choose Services then select Lambda under Compute.
Click the three lines icon to expand the service menu.
Click Functions.
Click Create function.
Enter WildRydesAggregator in the Function name field.
Select Node.js 14.x from Runtime.
Select unicorn-aggregation-role from the Existing role dropdown.

Click Create function.
Scroll down to the Code source section.
Double click the index.js file to open it in the editor.
Copy and paste the JavaScript code below into the code editor, replacing all of the existing code:

const AWS = require("aws-sdk"); AWS.config.update({ region: process.env.AWS_REGION }); const docClient = new AWS.DynamoDB.DocumentClient(); const TableName = process.env.TABLE_NAME; function isEmpty(obj) { return Object.keys(obj).length === 0; } exports.handler = async (event) => { // Save aggregation result in the final invocation if (event.isFinalInvokeForWindow) { console.log("Final: ", event); const params = { TableName, Item: { windowStart: event.window.start, windowEnd: event.window.end, distance: Math.round(event.state.distance), shardId: event.shardId, name: event.state.name, }, }; console.log({ params }); await docClient.put(params).promise(); } console.log(JSON.stringify(event, null, 2)); // Create the state object on first invocation or use state passed in let state = event.state; if (isEmpty(state)) { state = { distance: 0, }; } console.log("Existing: ", state); // Process records with custom aggregation logic event.Records.map((record) => { const payload = Buffer.from(record.kinesis.data, "base64").toString( "ascii" ); const item = JSON.parse(payload); let value = item.Distance; console.log("Adding: ", value); state.distance += value; let unicorn = item.Name; console.log("Name: ", unicorn); state.name = unicorn; }); // Return the state for the next invocation console.log("Returning state: ", state); return { state: state }; };

In the Configuration tab, select the Environment variables section.
Click Edit and then Add environment variable.
Enter TABLE_NAME in Key and UnicornAggregation in Value and click Save.

Scroll up and click on Add trigger from the Function overview section.

In the Trigger configuration section, select Kinesis.
Under Kinesis stream, select wildrydes.
Leave Batch size set to 100 and Starting position set to Latest.
Open the Additional settings section.
It is a best practice to set the retry attempts and bisect on batch settings when setting up your trigger. Change Retry attempts to 2 and select the checkbox for Split batch on error.
For Tumbling window duration, enter 60. This sets the time interval for your aggregation in seconds.

Click Add and the trigger will start creating.

Click the refresh button until creation is complete and the trigger shows as Enabled.
Go back to the Code tab and Deploy the Lambda function. You will see a confirmation that the changes were deployed.

4. Monitor the Lambda function

Verify that the trigger is properly executing the Lambda function and inspect the output from the Lambda function.

✅ Step-by-step Instructions

Return to your Cloud9 environment, and run the producer to start emitting sensor data to the stream.

./producer

Click on the Monitor tab. Next, click on View logs in CloudWatch to explore the logs in CloudWatch for the log group /aws/lambda/WildRydesAggregator.
Select the most recent Log stream.

You can use the Filter events bar at the top to quickly search for matching values within your logs. Use the filter bar, or scroll down, to examine the log events showing the Existing: distance state, Adding: a new distance count, and the Returning state: after the Lambda function is invoked.

Because we set the tumbling window to 60 seconds, every minute the Final: distance state is aggregated and passed to the DynamoDB table. To see the final distance count, use the filter bar to search for "Final:".
After expanding this log, you will see isFinalInvokeForWindow is set to true, along with the state data that will be passed to DynamoDB.

5. Query the DynamoDB table

Using the AWS Management Console, query the DynamoDB table to verify the per-minute distance totals are aggregated for the specified unicorn.

✅ Step-by-step Instructions

Click on Services then select DynamoDB in the Database section.
Click Tables from the left-hand navigation.
Click on UnicornAggregation.
Click on the View Items button on the top right.
Select Run to return the items in the table. Here you should see each per-minute data point for the unicorn for which you’re running a producer. Verify the state information from the CloudWatch log you viewed was successfully passed to the DynamoDB table.

⭐ Recap

🔑 The tumbling window feature allows a streaming data source to pass state between multiple Lambda invocations. During the window, a state is passed from one invocation to the next, until a final invocation at the end of the window. This allows developers to calculate aggregates in near-real time, and makes it easier to calculate sums, averages, and counts on values across multiple batches of data. This feature provides an alternative way to build analytics in addition to services like Amazon Kinesis Data Analytics.

🔧 In this module, you’ve created a Lambda function that reads from the Kinesis stream of unicorn data, aggregates the distance traveled per-minute, and saves each row to DynamoDB.

Error Handling with Checkpoint and Bisect On Batch

While Bisect On Batch is helpful in narrowing down to the problematic messages, it can result in processing previously successful messages more than once. With Checkpoint feature you can return the sequence identifier for the failed messages. This provides more precise control over how to choose to continue processing the stream. For example in a batch of 9 messages where the fifth message fails — Lambda process the batch of messages, items 1–9. The fifth message fails and the function returns the failed sequence identifier. The batch is only retried from message 5 to 9

Go to the AWS Management Console, choose Services then select Lambda under Compute. Alternatively, you can use the search bar and type Lambda in the search dialog box.
Figure out the changes to the WildRydesStreamProcessor function. The changes involve storing the kinesis sequence number (kinesis.sequenceNumber) of the error record and returning the sequence number in the catch block

*

javascript 'use strict'; const AWS = require('aws-sdk'); const dynamoDB = new AWS.DynamoDB.DocumentClient(); const tableName = process.env.TABLE_NAME; const errorString = 'error'; exports.handler = function(event, context, callback) { console.log('Number of Records sent for each invocation of Lambda function = ', event.Records.length) const requestItems = buildRequestItems(event.Records); const requests = buildRequests(requestItems); Promise.all(requests) .then(() => callback(null, `Delivered ${event.Records.length} records`)) .catch(callback); }; function buildRequestItems(records) { let sequenceNumber = 0; try { return records.map((record) => { sequenceNumber = record.kinesis.sequenceNumber; console.log('Processing record with Sequence Number = ', sequenceNumber); const json = Buffer.from(record.kinesis.data, 'base64').toString('ascii'); const item = JSON.parse(json); if(item.InputData.toLowerCase().includes(errorString)) { console.log ('Error record is = ', item); throw new Error('kaboom') } return { PutRequest: { Item: item, }, }; }); }catch (err) { console.log('Returning Failure Sequence Number =', sequenceNumber) return { "batchItemFailures": [ {"itemIdentifier": sequenceNumber} ] } } } function buildRequests(requestItems) { const requests = []; while (requestItems.length > 0) { const request = batchWrite(requestItems.splice(0, 25)); requests.push(request); } return requests; } function batchWrite(requestItems, attempt = 0) { const params = { RequestItems: { [tableName]: requestItems, }, }; let delay = 0; if (attempt > 0) { delay = 50 * Math.pow(2, attempt); } return new Promise(function(resolve, reject) { setTimeout(function() { dynamoDB.batchWrite(params).promise() .then(function(data) { if (data.UnprocessedItems.hasOwnProperty(tableName)) { return batchWrite(data.UnprocessedItems[tableName], attempt + 1); } }) .then(resolve) .catch(reject); }, delay); }); }

Do not forget to click Deploy to deploy the changes to the WildRydesStreamProcessor Lambda function.
Do not forget to Remove the existing Kinesis Data Stream mapping by clicking the Configuration Tab above the code editor. (This step is needed only if there is an existing Kinesis Data Stream mapping or any other Event Source Mapping present for the Lambda function). In the Configuration Tab select the Kinesis:wildrydes (Enabled) and Delete the trigger.
Add a new Kinesis Data Stream mapping by clicking the Configuration Tab. The mapping configuration is same except that you can check mark an additional field Report batch item failures.
Test your changes by inserting data into Kinesis Data Stream.
Monitor CloudWatch logs and Query DynamoDB by repeating steps in the sections Monitor and Query

Enhanced Fan Out

Enhanced fan-out enables consumers to receive records from a stream with throughput of up to 2 MB of data per second per shard. This throughput is dedicated, which means that consumers that use enhanced fan-out don’t have to contend with other consumers that are receiving data from the stream. Kinesis Data Streams pushes data records from the stream to consumers that use enhanced fan-out. Therefore, these consumers don’t need to poll for data.

For standard iterators, Lambda polls each shard in your Kinesis stream for records at a base rate of once per second. When more records are available, Lambda keeps processing batches until the function catches up with the stream. The event source mapping shares read throughput with other consumers of the shard.

To minimize latency and maximize read throughput, create a data stream consumer with enhanced fan-out. Enhanced fan-out consumers get a dedicated connection to each shard that doesn’t impact other applications reading from the stream. Stream consumers use HTTP/2 to reduce latency by pushing records to Lambda over a long-lived connection and by compressing request headers. You can create a stream consumer with the Kinesis RegisterStreamConsumer API.

aws kinesis register-stream-consumer --consumer-name con1 \
--stream-arn arn:aws:kinesis:us-east-2:123456789012:stream/lambda-stream

When you configure Event source mapping, use the consumer name created above as the consumer.

You can also try the template from serverlessland

Stream Aggregation

Streaming Aggregation

In this module, you’ll create an Amazon Kinesis Data Analytics application to aggregate sensor data from the unicorn fleet in real-time. The application will read from the Amazon Kinesis stream, calculate the total distance traveled and minimum and maximum health and magic points for each unicorn currently on a Wild Ryde and output these aggregated statistics to an Amazon Kinesis stream every minute. In the first section, you’ll run the application from a Flink Studio notebook. In the second, optional step, you’ll learn how to deploy the application to run outside the notebook.

Overview

The architecture for this module involves an Amazon Kinesis Data Analytics application, source and destination Amazon Kinesis streams, and the producer and consumer command-line clients.

The Amazon Kinesis Data Analytics application processes data from the source Amazon Kinesis stream that we created in the previous module and aggregates it on a per-minute basis. Each minute, the application will emit data including the total distance traveled in the last minute as well as the minimum and maximum readings from health and magic points for each unicorn in our fleet. These data points will be sent to a destination Amazon Kinesis stream for processing by other components in our system.

During the workshop, we will use the consumer.go application to consume the resulting stream. To do so, the application leverages the AWS SDK and acts as Kinesis Consumer.

Implement the Stream Aggregation

Implement the Streaming Aggregation

1. Create a Kinesis Data Stream for the summarized events

Use the Amazon Kinesis Data Streams console to create a new stream named wildrydes-summary with 1 shard. This stream will serve as destination for our Kinesis Data Analytics application.

✅ Step-by-step Instructions

Go to the AWS Management Console, click Services then select Kinesis under Analytics.
Click Get started if prompted with an introductory screen.
Click Create data stream.
Enter wildrydes-summary into Data stream name, select the Provisioned mode, and enter 1 into Number of open shards, then click Create Kinesis stream.
Within 60 seconds, your Kinesis stream will be ACTIVE and ready to store real-time streaming data.

2. Create an Amazon Kinesis Data Analytics application

In this step, we are going to build an Amazon Kinesis Data Analytics application which reads from the wildrydes stream built in the Real-time Data Streaming module and emits a JSON object with the following attributes for each minute:

NameUnicorn nameStatusTimeROWTIME provided by Amazon Kinesis Data AnalyticsDistanceThe sum of distance traveled by the unicornMinMagicPointsThe minimum data point of the MagicPoints attributeMaxMagicPointsThe maximum data point of the MagicPoints attributeMinHealthPointsThe minimum data point of the HealthPoints attributeMaxHealthPointsThe maximum data point of the HealthPoints attribute

To do so, we will use Kinesis Data Analytics to run an Apache Flink application. To enhance our development experience, we will use Studio notebooks for Kinesis Data Analytics that are powered by Apache Zeppelin.

Kinesis Data Analytics provides the underlying infrastructure for your Apache Flink applications. It handles core capabilities like provisioning compute resources, parallel computation, automatic scaling, and application backups (implemented as checkpoints and snapshots). You can use the high-level Flink programming features (such as operators, functions, sources, and sinks) in the same way that you use them when hosting the Flink infrastructure yourself. You can learn more about Amazon Kinesis Data Analytics for Apache Flink by checking out the corresponding developer guide.

✅ Step-by-step directions

In the AWS Management Console, click Services, select the Kinesis service under Analytics
Choose the Analytics Applications tab in sidebar on the left. If you cannot see a sidebar on the left, please click the Hamburger icon (three vertical lines) to open it.

Select the Studio tab
Click the Create Studio Notebook button. Keep the creation method to the default value.

Name the notebook flink-analytics
In the Permissions panel, click the Create button to create a new AWS Glue database. The AWS Glue service console will open in a new browser tab.

In the AWS Glue console, create a new database named default.
Switch back to your tab with the Studio notebook creation process and click Create Studio notebook.

Once the notebook is created, click the Edit IAM permissions button in the Studio notebook details section.

Make sure to select the default database in the AWS Glue database dropdown. Click the Choose source button in the Included sources in IAM policy section.

Click Browse to select the wildrydes Kinesis data stream as a source. Afterwards, click Save changes.

Click Choose destination followed by Browse to select the wildrydes-summary Kinesis data stream as a output. Afterwards, click Save changes.

Click Save changes the second time to update the IAM policy.
Click on IAM Role link to open it in separate tab. We need to add an additional managed policy to the role that would allow us to delete Glue tables. And we will reuse this policy when deploying the notebook as Flink application later.

Click Add permissions, then Attach Policies, and then Create Policy. Use the Visual editor. Choose Glue as Service.
For the Actions, select GetPartitions from the Read subsection and DeleteTable from the Write subsection.
For the Resources, click Add ARN for catalog and enter your region (e.g. eu-west-1)
For the database, click Add ARN, enter your region and specify default for database name
Finally, for the table, click Add ARN, enter your region, specify default as database, and select Any for the Table name.

Click Next: Tags button and then Next: Review. Specify kinesis-analytics-service-flink-analytics-glue as the policy name and click Create Policy button.
Attach the policy you have just created to the notebook IAM role, by selecting it from the list and clicking Attach Policy button.
Switch back to KDA Studio notebook tab and click Run to run the notebook. As soon it is in the Running state (takes a few minutes), click the Open in Apache Zeppelin button.
In the Apache Zeppelin notebook, choose create a new note. Name it flinkagg and insert the following SQL command. Replace the placeholder with the actual region (e.g. eu-west-1).

%flink.ssql(type=update) DROP TABLE IF EXISTS wildrydes; CREATE TABLE wildrydes ( Distance double, HealthPoints INT, Latitude double, Longitude double, MagicPoints INT, Name VARCHAR, StatusTime AS proctime() ) WITH ( 'connector' = 'kinesis', 'stream' = 'wildrydes', 'aws.region' = '', 'scan.stream.initpos' = 'LATEST', 'format' = 'json' ); DROP TABLE IF EXISTS wildrydes_summary; CREATE TABLE wildrydes_summary ( Name VARCHAR, StatusTime TIMESTAMP, Distance double, MinMagicPoints INT, MaxMagicPoints INT, MinHealthPoints INT, MaxHealthPoints INT ) WITH ( 'connector' = 'kinesis', 'stream' = 'wildrydes-summary', 'aws.region' = '', 'scan.stream.initpos' = 'LATEST', 'format' = 'json' ); INSERT INTO wildrydes_summary SELECT Name, TUMBLE_START(StatusTime, INTERVAL '1' MINUTE) AS StatusTime, SUM(Distance) AS Distance, MIN(MagicPoints) AS MinMagicPoints, MAX(MagicPoints) AS MaxMagicPoints, MIN(HealthPoints) AS MinHealthPoints, MAX(HealthPoints) AS MaxHealthPoints FROM wildrydes GROUP BY TUMBLE(StatusTime, INTERVAL '1' MINUTE), Name;

Execute the code by clicking the Play button next to the READY statement on the right side of the cell.
In the Cloud9 development environment, run the producer to start emiting sensor data to the stream.

./producer

3. Read messages from the stream

Use the command line consumer to view messages from the Kinesis stream to see the aggregated data being sent every minute.

✅ Step-by-step directions

Switch to the tab where you have your Cloud9 environment opened.
Run the consumer to start reading sensor data from the stream. It can take up to a minute for the first message to appear, since the Analytics application aggregates the messages in one minute intervals.

./consumer -stream wildrydes-summary

The consumer will print the messages being sent by the Kinesis Data Analytics application every minute:

{ "Name": "Shadowfax", "StatusTime": "2018-03-18 03:20:00.000", "Distance": 362, "MinMagicPoints": 170, "MaxMagicPoints": 172, "MinHealthPoints": 146, "MaxHealthPoints": 149 }

4. Experiment with the producer

Stop and start the producer while watching the dashboard and the consumer. Start multiple producers with different unicorn names.

✅ Step-by-step directions

Switch to the tab where you have your Cloud9 environment opened.
Stop the producer by pressing Control + C and notice how the consumer stops receiving messages.
Start the producer again and notice the messages resume.
Hit the (+) button and click New Terminal to open a new terminal tab.
Start another instance of the producer in the new tab. Provide a specific unicorn name and notice data points for both unicorns in consumer’s output:

./producer -name Bucephalus

Verify you see multiple unicorns in the output:

{ "Name": "Shadowfax", "StatusTime": "2018-03-18 03:20:00.000", "Distance": 362, "MinMagicPoints": 170, "MaxMagicPoints": 172, "MinHealthPoints": 146, "MaxHealthPoints": 149 } { "Name": "Bucephalus", "StatusTime": "2018-03-18 03:20:00.000", "Distance": 1773, "MinMagicPoints": 140, "MaxMagicPoints": 148, "MinHealthPoints": 132, "MaxHealthPoints": 138 }

⭐ Recap

🔑 Amazon Kinesis Data Analytics enables you to query streaming data or build entire streaming applications using SQL, so that you can gain actionable insights and respond to your business and customer needs promptly.

🔧 In this module, you’ve created a Kinesis Data Analytics application that reads from the Kinesis stream of unicorn data and emits a summary row each minute.

Challenges Faced and Solutions

Challenge 1: Handling High Volumes of Real-Time Data

Solution: Utilized Kinesis and Lambda to process data in small batches, ensuring efficient handling of high-throughput streams.

Challenge 2: Ensuring Data Durability and Query Flexibility

Solution: Stored raw data in S3 for archiving, leveraging Athena for querying, and DynamoDB for structured data storage.

Challenge 3: Securing Access Across Services

Solution: Implemented Cognito for secure, role-based access management, enforcing strict access control for users and applications.

Conclusion

The ability of serverless architecture to manage real-time data streams is demonstrated by the serverless data processing solution developed on AWS. This project offers a scalable, effective, and secure platform for processing large amounts of data with less infrastructure administration by combining services like Kinesis, Lambda, DynamoDB, and S3. This architecture is flexible and can be used for a variety of purposes, including online monitoring, financial transactions, and the Internet of Things. It is a prime example of how serverless services may streamline intricate data processing requirements while preserving cost-effectiveness and scalability.

Explore my GitHub repository.

Let’s connect: Linkdin, Twitter, Github

Deploying a Complete Machine Learning Fraud Detection Solution Using Amazon SageMaker : AWS Project

Shubham Murti — Wed, 13 Nov 2024 12:55:03 +0000

Introduction

This project leverages Amazon SageMaker and key AWS services to build a scalable, real-time fraud detection solution. By utilizing Amazon SageMaker’s machine learning capabilities, combined with services such as AWS Lambda, S3, and API Gateway, this setup processes transaction data to identify fraudulent patterns efficiently. It provides a robust framework for secure, automated, and reliable fraud detection, designed for seamless integration into production environments where real-time insights are essential.

Tech Stack

This solution leverages a range of AWS services, each playing a crucial role in creating a scalable, secure, and responsive machine learning infrastructure:

Amazon SageMaker: Core platform for model training, deployment, and hosting.
AWS Lambda: Automates resource creation, orchestrates SageMaker integrations, and handles infrastructure setup.
Amazon S3: Stores training data, model artifacts, and log files.
AWS IAM: Manages access control with defined roles and policies.
Amazon EC2 and VPC: Provides network isolation and backend processing capabilities.
Amazon CloudWatch: Enables monitoring and alerting for various system components.
Amazon SQS: Manages asynchronous task queues for inter-service communication.

Additional Services:

AWS Secrets Manager: Safeguards sensitive information such as API keys.
AWS CloudTrail: Tracks account activity and resource changes.
Amazon Route 53: Manages domain name resolution for the API endpoint.
AWS Systems Manager (SSM): Provides parameter management and infrastructure automation.
Amazon API Gateway: Exposes model predictions as RESTful APIs.
Amazon SNS: Sends alerts and notifications based on specific events.
Amazon CloudFormation: Automates infrastructure provisioning.

Prerequisites

Before beginning, ensure you have:

Basic AWS Knowledge: Familiarity with Amazon SageMaker, IAM, Lambda, and S3.
Python and ML Basics: Knowledge of Python for model training and AWS SDK integration.
AWS CLI and SDKs: Installed and configured for seamless AWS service management.
IAM Permissions: Appropriate permissions to interact with SageMaker, S3, Lambda, and other services used in the project.

Problem Statement or Use Case

Detecting fraudulent transactions is a challenge for financial institutions due to the high volume and complexity of real-time data. Fraud detection models need to be scalable, secure, and capable of integrating seamlessly with backend systems to process transactions in real-time. This project addresses these needs by developing a machine learning-based fraud detection model that:

Learns and Identifies Fraud Patterns: Uses machine learning to analyze transaction data and classify transactions as legitimate or suspicious.
Ensures Scalability and Efficiency: Deploys a highly scalable, serverless architecture using SageMaker and Lambda.
Enables Real-Time Monitoring and Notifications: Implements CloudWatch, CloudTrail, and SNS for tracking and alerting on model performance and anomalies.

The solution is ideal for large-scale fraud detection in production environments, enabling real-time insights with minimal manual intervention.

Architecture Diagram

The architecture diagram below shows the interaction between various AWS services, highlighting the flow of data from transaction storage to model inference and result distribution.

Step-by-Step Implementation

Launch the CloudFormation Stack

Local Computer

Download the CloudFormation template to your local computer:

curl 'https://static.us-east-1.prod.workshops.aws/public/aed5dc57-f15e-4afa-bbf4-9ff167491648/static/fraud-detection-workshop-selfpaced.yaml' --output fraud-detection-workshop-selfpaced.yaml

Within your AWS account

Navigate to AWS CloudFormation (AWS Console) to create a new stack.
On Create stack screen, under the Specify a template section, select Upload a template file option and navigate to select fraud-detection-workshop-selfpaced.yml file you downloaded earlier. Click Next.

On Specify stack details screen, under the Stack name section, provide a name for the CloudFormation stack such as fraud-detection-workshop.
Leave the rest of the parameters unchanged, and click Next

On the Configure stack options screen, leave default parameters unchanged, scroll to the bottom of the page, and click Next
On the Review fraud-detection-workshop screen, scroll to the bottom of the page and check off the box “I acknowledge that AWS CloudFormation might create IAM resources.”. Click Create Stack.

STOP: Cloudformation will take a few minutes to run and set up your environment. Please wait for this step to finish.

Congratulations! You have successfully deployed AWS environment for this workshop. Next, we will verify this environment.

Click “Next” to go to the next section.

Overview of the Environment

In this section, we’re going to go through a quick overview of the workshop and check if everything is set up correctly.

Workshop Overview

In this workshop we will -

Explore, clean, visualize and prepare the data: This step is all about understanding the auto insurance claims data.
Select & engineer features: Here we will get acquainted with Amazon SageMaker Feature Store.
Build and train a model: Train your model through the SageMaker API.
Deployment & Inference: Learn to deploy your model through quick commands for Real-time inference.
(Bonus) Transform data visually: Learn to transform and visualize data through Amazon SageMaker DataWrangler.
(Bonus) Detect bias in the dataset: Learn to use Amazon SageMaker Clarify to detect bias in a bonus lab.
(Bonus) Batch transforms: Learn to batch inference requests and use SageMaker Batch Transforms.
Finally, put everything together into a production CI/CD pipeline using Amazon SageMaker Pipelines

We are going to make use of three core jupyter notebooks.

The first notebook (Lab_1_and_2-Data-Exploration-and-Features.ipynb) demonstrates Exploratory Data Analysis (EDA). Specifically, data visualization, manipulation and transformation through Pandas and Seaborn python libraries. It will then walk you through feature engineering and getting the data ready for training.
The second notebook (Lab_3_and_4-Training_and_Deployment.ipynb) demonstrates training and deployment of the model followed by validation of the predictions using a subset of the data. Once deployed, the next step shows how to get predictions from the model.
The third notebook (Lab_5-Pipelines.ipynb) showcases a pipeline which integrates all previous steps. This is a good example on how to operationalize a Machine Learning Model into a production pipeline. This is a stand-alone lab which doesn't require executing the first two notebooks.

Verify SageMaker Studio is ready

Navigate to AWS console browser window and type SageMaker in the search bar at the top of the AWS console home page. Click on Amazon SageMaker service page in the AWS console.
Click on the Studio link in the left navigation pane under Control Panel.

Next, you should see a user is already setup for you. Click on “Open Studio”.

This will open SageMaker Studio UI in new browser window.

Attention

Amazon SageMaker Studio and Amazon SageMaker Studio Classic are two of the machine learning environments that you can use to interact with SageMaker. In this workshop, we will use SageMaker Studio Classic experience.

The Amazon SageMaker Studio UI extends the SageMaker Studio Classic interface. Click on “Studio Classic” icon under Applications.

The Amazon SageMaker Studio Classic interface is based on JupyterLab, which is a web-based interactive development environment for notebooks, code, and data. Keep this window open.

Now let’s walk through the various files and resources pre-provisioned for you.

Explanation of the code files

There are five notebooks in the folder FraudDetectionWorkshop:

Explanation of the data files

The target data exists in the data directory. Below is a list of files and their brief description.

Explanation of output files

The outputs directory contains two files that contain data transformations. We will use these files in Lab 5.

claims_flow_template
customer_flow_template

Explanation of helper scripts

There are seven helper scripts in scripts directory:

Attention

If for some reason, you are unable to complete the workshop, please head to the clean up steps. Resources created during the workshop may incur minor charges. It is best practice to spin down resources when they’re not in use.

Congratulations! You have successfully completed the Overview of the Environment.

Click “Next” to go to the next section.

Running Jupyter Notebooks

Note

If you already know how to execute Jupyter notebooks, skip this section.

Running Notebook cells

When you open a notebook, you’ll see a popup that requires you to select a kernel and instance type. Please make sure that the Image is Data Science, Kernel is Python3 and Instance Type is ml.t3.medium as shown in the screenshot below.

Note: If for some reason, you see an error on capacity for this particular instance type, it’s okay to scale up and choose the next available instance type.

If you haven’t worked with Jupyter notebooks before, the following screenshots explains how to execute and run different cells.

Clicking on the play button will execute the code within a selected cell.

If you see a * sign next to a cell, it means that cell is still being executed, and you should wait. Once it finishes it will show a number where the * was.

Congratulations! You have successfully completed the steps of how to run Jupyter Notebooks.

Click “Next” to go to the next section.

Data Preparation

Overview

In this section, you will learn about the highlighted steps of the machine learning process.

1 — Ingest, Transform And Preprocess Data

Note

The following material provides contextual information about this lab. Please read through this information before you refer jupyter notebook for step-by-step code block instructions.

Overview

Exploratory Data Analysis (EDA) is an unavoidable step in the Machine Learning process. Raw data cannot be consumed directly to create a model. Data stakeholders understand, visualize and manipulate data before using it. Common transforms include (but aren’t limited to): removing symbols, one-hot encoding, removing outliers, and normalization.

Instructions

You will be working on the first notebook in the series Lab_1_and_2-Data-Exploration-and-Features.ipynb. Please scroll down for important context before starting with the notebooks.

The steps are outlined below:

Data visualization ~5m
Data transformation ~8m

Total run time ~13 minutes

Data Transformation

For our use case we have been provided with two datasets claims.csv and customers.csv containing the auto-insurance claims and customers' information respectively. This dataset was generated synthetically. However, the raw dataset can be non-numeric which is hard to visualize and cannot be used for the Machine Learning process.

Consider the columns driver_relationship or incident_type in claims.csv. The data type for values under these columns is known as an Object. It's a string that represents a feature. It's hard to use this kind of data directly since machines don't understand strings or what they represent. Instead, it would be a lot easier to just mark a feature as a one or zero.

So instead of saying:

driver_relationship = 'Spouse'

It’s better to break it out into another feature like so:

driver_relationship_spouse = 1

We’ve elected to transform the data to get it ready for Machine Learning.

These columns will be one-hot encoded so that every type of collision can be a separate column.

Similarly, many transformations are required before the data can be used for Machine Learning. Data stakeholders often iterate over datasets multiple times before they can be used. In this case, transformations are created using Amazon SageMaker Data Wrangler (see the hint below). With this context in mind the following files are available:

The .flow templates named customer_flow_template and claims_flow_template. These templates contain the transformations on customer and claims dataset created through SageMaker Data Wrangler. These files are in the standard JSON format and can be read in using the python json module.
These transformations are applied to the raw datasets. The final processed datasets are claims_preprocessed.csv and customers_preprocessed.csv. The notebook starts off with these preprocessed datasets.

Hint

If you wish to learn how to make these transformations yourself, you can go through the Bonus Labs section of this workshop titled — Data exploration using Amazon SageMaker Data Wrangler

Data Visualization

At this point, let’s head over to the first notebook. Navigate to the SageMaker Studio UI and click on the folder icon on the left navigation panel. Open the folder FraudDetectionWorkshop. Finally, open the first notebook titled Lab_1_and_2-Data-Exploration-and-Features.ipynb.

Note

Follow the jupyter notebook instructions till you complete Lab 1 and navigate back here when done.

Conclusion

Congratulations! You’ve successfully learned how to visualize and pre-process the data and gather insights.

In this lab we learned how to transform data easily using Amazon SageMaker Studio Notebooks.

Click “Next” to go to the next section.

2 — Feature Engineering

Note

The following material provides contextual information about this lab. Please read through this information before you refer jupyter notebook for step-by-step code block instructions.

Prerequisite

Please make sure Lab 1 is executed successfully before you proceed with this lab.

Overview

Amazon SageMaker Feature Store provides a central repository for data features with low latency (milliseconds) reads and writes. Features can be stored, retrieved, discovered, and shared through SageMaker Feature Store for easy reuse across models and teams with secure access and control.

SageMaker Feature Store keeps track of the metadata of stored features (e.g. feature name or version number) so that you can query the features for the right attributes in batches or in real time using Amazon Athena , an interactive query service.

In this lab, you will learn how to use Amazon SageMaker Feature Store to store and retrieve machine learning (ML) features.

Instructions

The steps are outlined below:

Creating the Feature Store ~6 min (including time to create and ingest data on Feature Store).
Visualize Feature Store ~2 min.
Upload data to S3 ~1 min.

Total run time ~10 minutes.

Creating the Feature Store

The collected data, we refer to it as raw data is typically not ready to be consumed by ML Models, The data needs to transformed e.g. encoding, dealing with missing values, outliers, aggregations. This process is known as feature engineering and the signals that are extracted as part of this data prep are referred to as features.

A feature group is a logical grouping of features and these groups consist of features that are computed together, related by common parameters, or are all related to the same business domain entity.

In this step, you are going to create two feature groups: customer and claims.

After the Feature Groups have been created, we can put data into each store by using the PutRecord API . This API can handle high TPS (Transactions Per Second) and is designed to be called concurrently by different streams. The data from PUT requests is written to the offline store within few minutes of ingestion.

Hint

It is possible to verify that the data is available offline by navigating to the S3 Bucket.

Split the Dataset and upload to S3

Once the data is available in the offline store, it will automatically be cataloged and loaded into an Amazon Athena table (this is done by default, but can be turned off). In order to build our training and test datasets, you will submit a SQL query to join the Claims and Customers tables created in Athena.

The last step in this notebook is to upload newly created datasets into S3.

At this point, let’s navigate back to the first notebook (Lab_1_and_2-Data-Exploration-and-Features.ipynb) and scroll down to Lab 2: Feature Engineering

Note

Follow the jupyter notebook instructions till you complete Lab 2 and navigate back here when done.

Conclusion

Congratulations! You have successfully prepared the data to train an XGBoost model.

In this lab we learned how to ingest features into Amazon SageMaker Feature Store and prepare our data for training.

Click “Next” to go to the next section.

Training and Deployment

Overview

In this section, you will learn about the following highlighted step of the Machine Learning process.

3 — Train a Model using XGBoost

Note

The following material provides contextual information about this lab. Please read through this information before you refer jupyter notebook for step-by-step code block instructions.

Prerequisite

Please make sure Lab 2 is executed successfully before you proceed with this lab.

Overview

In this lab, you will learn how to use Amazon SageMaker Training Job to build, and train the ML model.

To train a model using SageMaker, you create a training job. The training job includes the following information:

The URL of the Amazon Simple Storage Service (Amazon S3) bucket where you’ve stored the training data.
The compute resources that you want SageMaker to use for model training. Compute resources are ML compute instances that are managed by SageMaker.
The URL of the S3 bucket where you want to store the output of the job.
The Amazon Elastic Container Registry path where the docker container image is stored.

For this tutorial, you will use the XGBoost Open Source Framework to train your model. This estimator is accessed via the SageMaker SDK, but mirrors the open source version of the XGBoost Python package . Any functionality provided by the XGBoost Python package can be implemented in your training script. XGBoost is an extremely popular, open-source package for gradient boosted trees. It is computationally powerful, fully featured, and has been successfully used in many machine learning competitions.

Instructions

The steps are outlined below:

Data handling ~1m
Train a model using XGBoost ~8m (including running the training code ~4m)
Deposit the model in SageMaker Model Registry ~3m

Total run time ~ 12 mins.

Data handling

There are two ways to obtain the dataset:

Use the dataset you uploaded to Amazon S3 bucket in the previous Lab (Lab 2 — Feature Engineering).
Upload the following datasets from data folder to Amazon S3: train.csv, test.csv

The following code upload the datasets from data folder to Amazon S3:

Train a model using XGBoost

You will define SageMaker Estimator using XGBoost Open Source Framework to train your model. The following code will create the Estimator object and start the training job using xgb_estimator.fit() function call.

For this example, we will use the following parameters for the XGBoost estimator:

entry_point - Path to the Python source file which should be executed as the entry point to training.
hyperparameters - Hyperparameters that will be used for training. The hyperparameters are made accessible as a dict[str, str] to the training code on SageMaker.
output_path - S3 location for saving the training result (model artifacts and output files).
framework_version - XGBoost version you want to use for executing your model training code.
instance_type - Type of EC2 instance to use for training.

If you want to explore the breadth of functionality offered by the SageMaker XGBoost Framework you can read about all the configuration parameters by referencing the inheriting classes. The XGBoost class inherits from the Framework class and Framework inherits from the EstimatorBase class:

Launching a training job and storing the trained model into S3 should take ~4 minutes. Notice that the output includes the value of Billable seconds, which is the amount of time you will be actually charged for.

Deposit the model in SageMaker Model Registry

After the successful training job, you can register the trained model in SageMaker Model Registry . SageMaker’s Model Registry is a metadata store for your machine learning models. Within the model registry, models are versioned and registered as model packages within model groups. Each model package contains an Amazon S3 URI to the model files associated with the trained model and an Amazon ECR URI that points to the container used while serving the model.

At this point, let’s navigate back to the training notebook (Lab_3_and_4-Training_and_Deployment.ipynb) and scroll down to Lab 3: Prerequisites

Note

Follow the jupyter notebook instructions till you complete Lab 3 and navigate back here when done.

Conclusion

Congratulations! You have successfully built and trained your model.

In this lab you have walked through the process of building, and training XGBoost model using Amazon SageMaker Estimator. You also used the SageMaker Python SDK to train the model.

Click “Next” to go to the next section.

4 — Deploy and Serve the Model

Note

The following material provides contextual information about this lab. Please read through this information before you refer jupyter notebook for step-by-step code block instructions.

Prerequisite

Please make sure Lab 3 is executed successfully before you proceed with this lab.

Overview

After you train your machine learning model, you can deploy it using Amazon SageMaker to get predictions in any of the following ways, depending on your use case:

For persistent, real-time endpoints that make one prediction at a time, use SageMaker real-time hosting services.
Workloads that have idle periods between traffic spurts and can tolerate cold starts, use Serverless Inference.
Requests with large payload sizes up to 1GB, long processing times, and near real-time latency requirements, use Amazon SageMaker Asynchronous Inference.
To get predictions for an entire dataset, use SageMaker batch transform.

Following image describes different deployment options and their use cases.

Instructions

The steps are outlined below:

Evaluate trained model and update status in the model registry: ~3 mins
Model deployment: ~1 min
Create/update endpoint: 5 mins
Predictor interface: 1 mins

Total run time ~ 10 mins.

Evaluate trained model

After you create a model version, you typically want to evaluate its performance before you deploy the model in production. If it performs to your requirements, you can update the approval status of the model version to Approved. In the real-life MLOps lifecycle, a model package gets approved after evaluation by data scientists, subject matter experts, and auditors.

For the purpose of this lab, we will evaluate the model with test dataset that was created during training process. The lab contains evaluate.py script that calculates AUC (Area under the ROC Curve) on the test dataset. The AUC threshold is set at 0.7. If the test dataset AUC is below the threshold, then the approval status should be “Rejected” for that model version.

Model deployment

To prepare the model for deployment, you will conduct following steps:

Query the model registry and list all the model versions:
Hint
For the purpose of this lab, we will get the latest version of the model from the model registry. However, you can apply different filtering criterion such as listing approved models or get specific version of the model. Please refer to the Model Registry documentation.
Define the endpoint configuration: Specify the name of one or more models in production (variants) and the ML compute instances that you want SageMaker to launch to host each production variant.

When hosting models in production, you can configure the endpoint to elastically scale the deployed ML compute instances. For each production variant, you specify the number of ML compute instances that you want to deploy. When you specify two or more instances, SageMaker launches them in multiple Availability Zones, this ensures continuous availability. SageMaker manages deploying the instances.

Create/update endpoint

Once you have your model and endpoint configuration, use the CreateEndpoint API to create your endpoint. Provide the endpoint configuration to SageMaker. The service launches the ML compute instances and deploys the model or models as specified in the configuration. Please refer to the documentation .

Predictor interface

In this part of the workshop, you will use the data from dataset.csv to run inference against the newly deployed endpoint.

At this point, let’s navigate back to the training notebook (Lab_3_and_4-Training_and_Deployment.ipynb) and scroll down to Lab 4: Deploy and serve the model

Note

Follow the jupyter notebook instructions till you complete Lab 4 and navigate back here when done.

Conclusion

Congratulations! You have successfully deployed an endpoint to get predictions from your model.

In this lab, you created a low latency Endpoint using Amazon SageMaker and deployed your model to get predictions. In the next lab, you will learn how to integrate all of the steps you’ve learnt so far using SageMaker Pipelines.

Click “Next” to go to the next section.

Machine Learning Workflow

Overview

In this section, you will learn about the following highlighted step of the Machine Learning process.

5 — Pipelines

Note

The following material provides contextual information about this lab. Please read through this information before you refer jupyter notebook for step-by-step code block instructions.

Attention

This lab demonstrates how to build an end-to-end machine learning workflow using Sagemaker Pipeline. This is a stand-alone lab and can be run independently of the previous labs.

If you have already executed the previous labs (Lab 1 and Lab 2) then you don’t need to run the Step 0 on juypter notebook.

Content

Overview

In previous labs, you built separate processes for data preparation, training, and deployment. In this lab, you will build a machine learning workflow using SageMaker Pipelines that automates end-to-end process of data preparation, model training, and model deployment to detect fraudulent automobile insurance claims.

Instructions

The machine learning workflow steps are outlined below:

Step 1 — Data Wrangler Preprocessing ~2 min
Step 2 — Create Dataset and Train/Test Split ~1 min
Step 3 — Train XGBoost Model ~1 min
Step 4 — Model Pre-Deployment Step ~1 min
Step 5 — Register Model ~1 min
Step 6 — Model deployment ~1 min
Step 7 — Combine and Run the Pipeline Steps ~1 min
Run the pipeline ~15 mins

Total run time ~ 23 mins.

Create Automated Machine Learning Pipeline

SageMaker Pipelines service is composed of following steps. These steps define the actions that the pipeline takes and the relationships between steps using properties.

Step 1 — Data Wrangler Preprocessing

Define Data Wrangler inputs using ProcessingInput, outputs using ProcessingOutput, and Processing Step to create a job for data processing for "claims" and "customer" data.

Step 2 — Create Dataset and Train/Test Split

Next you will create an instance of a SKLearnProcessor processor. You can split the dataset without using SKLearnProcessor as well, but if the dataset is larger than the one provided, it will takes more time and requires local compute resources. Hence it is recommended to use manage processing job.

Step 3 — Train XGBoost Model

You will use SageMaker’s XGBoost algorithm to train the dataset using the Estimator interface. A typical training script loads data from the input channels, configures training with hyperparameters, trains a model, and saves the model to “model_dir”.

Step 4 — Model Pre-Deployment

sagemaker.model.Model denotes a SageMaker Model that can be deployed to an Endpoint.

Step 5 — Register Model

Typically, customers can create a ModelPackageGroup for SageMaker Pipelines so that model package versions are added for every iteration.

Step 6 — Model deployment

Once the model is registered, the next step is deploying the model. You will use Lambda function step to deploy the model as real time endpoint. The SageMaker SDK provides a Lambda helper class that can be used to create a Lambda function. This function is provided to the Lambda step for invocation via the pipeline. Alternatively, a predefined Lambda function can also be provided to the Lambda step.

Attention

Please open CloudFormation console and copy Lambda ARN of the lambda function (under the Outputs tab).

Copy lambda function ARN value and add it to cell #21

Step 7 — Combine the Pipeline Steps

SageMaker Pipelines is a series of interconnected workflow steps that are defined using the Pipelines SDK . This Pipelines definition encodes a pipeline using a directed acyclic graph (DAG) that can be exported as a JSON definition.

Create the pipeline definition

Submit the pipeline definition to the SageMaker Pipelines to either create a new pipeline if it doesn’t exist, or update the existing pipeline if it does.

Review the pipeline definition

Describing the pipeline status ensures that it has been created successfully. Viewing the pipeline definition with all the string variables interpolated may help debug pipeline bugs.

Run the pipeline

Start a pipeline execution. Note this will take about 15 minutes to complete. You can watch the progress of the Pipeline Job on your SageMaker Studio Pipelines panel.

Click the Home folder pointed by the arrow and click on Pipelines.
You will see the available pipelines in the table on the right.
Click on FraudDetectDemo.

Next, you will see the executions listed on the next page. Double-click on the Status executing to be taken to the graph representation.

You will see the nodes turn green when the corresponding steps are complete.

Note

Follow the jupyter notebook instructions till you complete Lab 5 and navigate back here when done.

Conclusion

Congratulations! You have successfully created an end to end machine learning workflow using SageMaker Pipelines.

Click “Next” to go to the next section.

Summary

What you have learned

In this workshop, you have learned how to:

Inspect, analyze and transform an auto insurance fraud dataset
Ingest transformed data into SageMaker Feature Store using the SageMaker Python SDK
Train an XGBoost model using SageMaker Training Jobs
Create a realtime endpoint for low latency requests using SageMaker
Integrate all previous steps into an MLOps workflow with SageMaker Pipelines

Thank you!

Clean Up

Congratulations on developing an ML fraud detection solution and deploying automated pipelines!

Attention

If you are at an AWS event, such as re:Invent or an Immersion Day, and are using an AWS provided account, then you don’t need to worry about cleaning up the resources.

To clean up resources, please do the following:

Delete the CloudFormation stack to clean up the environment.
Go to CloudFormation home page, click Stacks **under the left hand side menu, select stack **fraud-detection-workshop **stack, and click **Delete button to delete the stack.

Delete the Model.
During Lab 4, we deleted the SageMaker hosted endpoint but not the model. We needed that model for Detect Bias bonus lab execution. Go to SageMaker home page and expand Inference **in SageMaker dashboard section on left hand side menu. Click **Models, and select fraud-detect-model-xxxxxxxxxxxx. Click Action **button and select **Delete option to delete the model.

Delete the Endpoint Configuration.
In SageMaker home page left hand side menu, expand Inference and click on Endpoint configurations. Select fraud-detect-demo-endpoint-config-xxxxxxxxxxxx configuration, click Actions button, and select Delete option to delete the configuration.

Delete the lifecycle configuration.
In SageMaker home page left hand side menu, click on Lifecycle configurations. Select git-clone-step lifecycle configuration and click Delete button to delete the lifecycle configuration.

And, finally delete the S3 bucket.
Go to S3 . To delete the bucket, you need to first delete the objects inside the bucket. Click on sagemaker — xxxxxxxxxxxx bucket and select checkbox on the top of the object table to select all objects.

Click Delete button to delete all objects.
Navigate back to S3 buckets list, select sagemaker — xxxxxxxxxxxx bucket and click Delete button to delete the bucket.

Congratulations! You have successfully cleaned up the environment.

This brings us to the end of this workshop.

Thank you.

Challenges Faced and Solutions

Challenge 1: Real-Time Model Inference and Latency Optimization

Solution: Leveraged API Gateway and Lambda to manage requests, reducing latency by preprocessing data in Lambda and only sending necessary data to SageMaker.

Challenge 2: Managing Security for Sensitive Data

Solution: Used AWS Secrets Manager to secure sensitive information such as database credentials, API keys, and integrated with IAM to enforce role-based access control.

Challenge 3: Monitoring and Troubleshooting Complex Workflows

Solution: Integrated CloudWatch, CloudTrail, and X-Ray to gain visibility into all workflow steps, allowing for efficient troubleshooting and resource optimization.

Conclusion

This end-to-end solution highlights the power of Amazon SageMaker in handling real-time fraud detection and anomaly classification. By integrating AWS tools for automation, monitoring, and security, this project demonstrates an adaptable, high-performance architecture that can scale to meet growing data demands. The setup is versatile and supports businesses in proactive fraud management, ensuring fast, accurate, and secure anomaly detection for production-grade applications.

Explore my GitHub repository.

Let’s connect: Linkdin, Twitter, Github

Large-scale Data Processing with Step Functions : AWS Project

Shubham Murti — Wed, 13 Nov 2024 09:42:06 +0000

Introduction

In this project, we tackle the challenge of orchestrating large-scale data processing using AWS Step Functions. By integrating key AWS services like Amazon S3, IAM, CloudWatch, and AWS X-Ray, we build a scalable, secure, and optimized workflow for handling vast amounts of data. This setup is designed to reduce operational complexity, enhance scalability, and provide robust monitoring, making it ideal for data-intensive industries requiring high levels of efficiency and control.

Tech Stack

Key AWS services, tools, and technologies used in this project include:

AWS Step Functions: Orchestrates the data processing workflow
Amazon S3: Provides centralized storage for input and output data
IAM (Identity and Access Management): Ensures secure permissions and data access policies
CloudWatch: Monitors each step of the workflow, logging actions and errors
AWS X-Ray: Provides detailed tracing and performance insights into the workflow’s execution

Prerequisites

To follow along with this project, you’ll need the following prerequisites:

AWS Knowledge: Familiarity with AWS services like S3, IAM, and CloudWatch.
Serverless and Workflow Concepts: Basic understanding of serverless architecture and workflow orchestration.
AWS CLI and SDKs: Installed and configured for managing AWS resources.
IAM Permissions: Ensure the necessary IAM roles and policies for accessing and executing Step Functions.

Problem Statement or Use Case

Organizations often deal with complex workflows that require efficient processing of large data sets. This project addresses the need for a scalable and automated solution to manage such workflows reliably. Key challenges include:

Coordination of Multiple Services: Processing large datasets often involves multiple tasks and services that need to be coordinated systematically.
Scalability: As data size grows, the system must scale to handle the increased load.
Visibility and Monitoring: Detailed logging, tracing, and monitoring are essential for troubleshooting and optimizing the workflow.

AWS Step Functions, combined with other AWS services, provides an ideal solution for orchestrating complex workflows in a serverless environment. This architecture not only improves processing efficiency but also enhances visibility and simplifies troubleshooting.

Architecture Diagram

Below is the architecture diagram for the data processing workflow. The visual aid below illustrates how AWS Step Functions orchestrates interactions among S3, IAM, CloudWatch, and AWS X-Ray to create a reliable and traceable workflow.

Component Breakdown

Each component within this solution has a critical role:

AWS Step Functions: Manages the flow of data processing tasks, coordinating AWS services and monitoring the progress of each step.
Amazon S3: Acts as a centralized storage solution for data inputs and outputs, making it easy to retrieve and store processed data.
IAM: Controls access to resources, ensuring only authorized services and users can interact with sensitive data.
CloudWatch: Provides insights into each task’s status, allowing us to set up alarms for failure states or performance bottlenecks.
AWS X-Ray: Traces and analyzes the performance of the workflow, helping to optimize and troubleshoot issues effectively.

Step-by-Step Implementation

Introduction to Distributed Map

Setup

Important

Follow the instructions on this page only if you are executing this workshop in your own account. To skip these instructions click here.

Click on the Launch link against any of the regions in the table below to start the deployment.

RegionDeployment

US East (Northern Virginia) us-east-1 Launch
Europe (Ireland) eu-west-1 Launch
Asia Pacific (Singapore) ap-southeast-1 Launch
Asia Pacific (Sydney) ap-southeast-2 Launch

The location of the CloudFormation template will be auto-populated in the Amazon S3 URL field, as shown in the image below. Click Next.

On the Specify stack details page, Stack name will be auto-populated to sfw-hello-distributed-map. (You can enter a different name if you want.) Click Next two times.

On the Review page, scroll to the bottom, check the Capabilities box if shown, then click Create stack.

Wait until the stack shows CREATE_COMPLETE status.

Building a Distributed Map Workflow

Setup

Important

Follow the instructions on this page only if you are executing this workshop in your own account. To skip these instructions click here.

Click on the Launch link against any of the regions in the table below to start the deployment.

Region Deployment

US East (Northern Virginia) us-east-1 Launch
Europe (Ireland) eu-west-1 Launch
Asia Pacific (Singapore) ap-southeast-1 Launch
Asia Pacific (Sydney) ap-southeast-2 Launch

The location of the CloudFormation template will be auto-populated in the Amazon S3 URL field, as shown in the image below. Click Next.

On the Specify stack details page, Stack name will be auto-populated to sfw-processmulti-distributed-map. (You can enter a different name if you want.) Click Next two times.

-On the Review page, scroll to the bottom, check the Capabilities box if shown, then click Create stack.

Wait until the stack shows CREATE_COMPLETE status.

Advanced — Optimization

Setup

Important

Follow the instructions on this page only if you are executing this workshop in your own account. To skip these instructions click here.

Click on the Launch link against any of the regions in the table below to start the deployment.

Region Deployment

US East (N. Virginia) us-east-1 Launch
Europe (Ireland) eu-west-1 Launch
Asia Pacific (Singapore) ap-southeast-1 Launch
Asia Pacific (Sydney) ap-southeast-2 Launch

Location of the CloudFormation template will be auto populated in the Amazon S3 URL field as shown in the diagram below. Click Next

On the Specify stack details page, Stack name would be auto populated to sfw-optimization-distributed-map. You can specify a different name if you want.

Click Next two times and on the last Review page, scroll to the bottom. Click the checkbox if shown and then click Create stack.

Wait till the stack shows CREATE_COMPLETE status.

Use Case — Healthcare Claims Processing

Setup

Important

Follow the instructions on this page only if you are executing this workshop in your own account. To skip these instructions click here.

Select Launch link against any of the regions in the table below to start the deployment.

Region Deployment

US East (N. Virginia) us-east-1 Launch
Europe (Ireland) eu-west-1 Launch
Asia Pacific (Singapore) ap-southeast-1 Launch
Asia Pacific (Sydney) ap-southeast-2 Launch

Location of the CloudFormation template will be auto populated in the Amazon S3 URL field as shown in the diagram below. Select Next

On the Specify stack details page, Stack name would be auto populated to sfw-healthcare-processing. You can specify a different name if you want.

Select Next two times and on the last Review page, scroll to the bottom. Select the checkbox if shown and then Select Create stack.

Wait till the stack shows CREATE_COMPLETE status.

Use Case — Security Vulnerability Scanning

Setup

Important

Follow the instructions on this page only if you are executing this workshop in your own account. To skip these instructions click here.

Click on the Launch link against any of the regions in the table below to start the deployment.

Region Deployment

US East (Northern Virginia) us-east-1 Launch
Europe (Ireland) eu-west-1 Launch
Asia Pacific (Singapore) ap-southeast-1 Launch
Asia Pacific (Sydney) ap-southeast-2 Launch

The location of the CloudFormation template will be auto-populated in the Amazon S3 URL field. Click Next.

On the Specify stack details page, Stack name will be auto-populated to vulnerability-scanning-module. (You can enter a different name if you want.) Click Next two times.
On the Review page, scroll to the bottom then select the Capabilities and transforms boxes if shown.
Click Submit and wait until the stack shows CREATE_COMPLETE status.

Use Case — Monte Carlo Simulation

Setup

Important

Follow the instructions on this page only if you are executing this workshop in your own account. To skip these instructions click here.

About The Stacks

The solution is deployed in two sets. The first stack will deploy a stack of resources that will generate our simulated dataset. It is orchestrated by Step Functions and consists of three Lambda Functions that will generate the data as well as generate a simulated S3 Inventory Report. The second stack will execute the first state machine as well as deploy the components for the module.

Deploy The Data Generation Stack

Click on the Launch link against any of the regions in the table below to start the deployment.
RegionDeployment*US East (N. Virginia)* us-east-1Launch Europe (Ireland) eu-west-1Launch Asia Pacific (Singapore) ap-southeast-1Launch Asia Pacific (Sydney) ap-southeast-2Launch
Take Defaults — Click Next 3 times and Submit
Deployment Screenshots

Deploy The Data Processing Stack

Click on the Launch link against any of the regions in the table below to start the deployment.

Region Deployment

US East (N. Virginia) us-east-1 Launch
Europe (Ireland) eu-west-1 Launch
Asia Pacific (Singapore) ap-southeast-1 Launch
Asia Pacific (Sydney) ap-southeast-2 Launch
Take Defaults — Click Next 3 times and Submit

Deployment Screenshots

Step 1

Step 2

Step 3

Step 4

Module 1 — Basics

Introduction to Distributed Map

What is Distributed Map?

Distributed map is a map state that executes the same processing steps for multiple entries in a dataset at 10,000 concurrency. This means you can run a large-scale parallel data processing workload without worrying about how to parallelize the executions, workers, and data. Distributed map can iterate over millions of objects such as logs, images or records inside .csv or json files stored in Amazon S3. It can launch up to 10,000 parallel child workflows to process the data. You can include any combination of AWS services like AWS Lambda functions, Amazon ECS tasks, or AWS SDK calls in the child workflow.

Distributed map is the new addition to the two types of maps available in Step Functions. Primary difference between the inline map and distributed map are -

Distributed map has higher concurrency compared to inline map’s 40 concurrency
Distributed map can directly iterate on the data from Amazon S3.
Each sub workflow of Distributed map runs as a separate child workflow that avoids 25K execution history limit of Step Functions.

What you do in the module

In this module, you will explore a pre-created workflow using distributed map. The sample workflow processes amazon reviews data from repo . It is 83MB CSV file with 1,292,954 records.

The workflow iterates on the electronics review data and filters highly rated reviews. You will download the data to S3, run the workflow, and verify the results. In the process, you will learn how to build and run a simple distributed map workflow yourself.

Services in this module

AWS Step Functions — Serverless visual workflow service

Reviewing the Workflow

Open the Step Functions console then select the state machine containing “HelloDmap” in its name.

Choose Edit to edit the workflow in Workflow Studio.

Review the definition by selecting the Definition toggle at the right side of the page.

Select each state in the design to review its definition.

Take a closer look at the map definition. You define it as DISTRIBUTED to tell Step Functions to run the map state in distributed mode.

Did you notice ItemReader in the definition? This is how you tell distributed map to process a csv or JSON file. Notice that the reader uses the s3.getObject API to read the object. Yes! It reads the content of the csv file and distributes the data to child workflows in batches, running at a concurrency of 10,000!

Read ahead and you will notice that there are a few more settings.

Firstly, you can do batching. Do you see MaxItemsPerBatch set as 1000?

You can not only run 10,000 (10K) workflows, you can also batch the data to each workflow which means, in a single iteration, you can process 10K * 1K = 10M records from the csv file!

Secondly, you can write the output of the distributed map or the child workflow execution results to an S3 location in an aggregated fashion.

Thirdly, you can set failure toleration. What does that mean? You don’t want to run 100M records when half of them are bad data. It is a waste of time and money to process those records. By default, the failure toleration is set to 0. Any single child workflow failure will result in the failure of the workflow.

Data quality is a big challenge with large data processing. So, you can set a percentage or number of items that can be tolerated as failures. When failures exceed that tolerance, the Step Functions workflow fails, saving you time and money.

Next, you can see what is inside of the distributed map. In this introduction module, we did not use any compute services like Lambda to process the records. You can see a pass state filtering highly rated reviews. Pass state is really useful to transform and filter input. It is simple and easy to demonstrate the 10K concurrency, without worrying about the scale of a downstream service. However, in a real-world scenario, you include any combination of AWS services like AWS Lambda functions, Amazon ECS tasks, or AWS SDK calls in the child workflow.

The states inside the distributed map are run as separate child workflows. The number of child workflows is dependent on the concurrency setting and the volume of the records to process. For example, you might set the concurrency to 1000 and batch size to 100, but if the total number of records in the file is just 20K, Step Functions only needs 200 child workflows (20,000 / 100 = 200). On the other hand, if the file has 200K records, Step Functions will spin up 1000 child workflows to reach the max concurrency and as child workflows complete, Step Functions will spin up child workflows until all 2000 (200,000 /100 = 2000) child workflows are completed.

Running the Workflow

Prepare data set

You are going to download the data file to the local computer and upload to S3.
Run the following command in your local machine.
Linux or macOS (bash)

curl https://raw.githubusercontent.com/MengtingWan/marketBias/master/data/df_electronics.csv --output df_electronics.csv
Windows (PowerShell)

Invoke-WebRequest https://raw.githubusercontent.com/MengtingWan/marketBias/master/data/df_electronics.csv -OutFile df_electronics.csv
Open the S3 console then select the bucket containing “hellodmapdatabucket” in its name.
Copy the downloaded file “df_electronics.csv” to the S3 bucket.

Run the workflow

Open the Step Functions Console , select and open the state machine containing HelloDmapStateMachine in its name.
Choose Start execution in the top right corner.
In the popup, enter the following input:

{
"key":"df_electronics.csv",
"output":"results"
}
You are providing the name of the file and S3 prefix where you want the results of the distributed map to be stored.
Click Start execution.
In a few seconds, you will see the execution start to run. It takes a couple of minutes to complete the processing.

Congratulations! You have successfully run the hello distributed map workflow.

Viewing the Workflow Results

View map run

Navigate to the bottom of the workflow execution page and click Map Run.

You can see the status of the distributed processes in the Item processing status card. It shows the number of records processed and the duration. 1,292,954 records processed in less than 90 seconds! At the bottom of the page, you can see the links to all the child workflow executions.

Open one of the child workflows and view the execution input/output. You see in the output window that the pass state filtered the records with ratings of 4 and above.

Verifying S3 results bucket

Recall you also stored the results of the workflow to an S3 bucket! Open the S3 console then select the bucket containing hellodmapresultsbucket in its name.

Navigate to contents inside results prefix, select "SUCCEEDED_0.json", and download the file to view the results. You notice that the content of the file is the aggregated result of all the child workflows. If you are building map reduce use cases, this content can be used for the downstream. To learn more about how result writer works, follow the link here .

Fantastic! You processed 1.3M records in less than 90 seconds without using servers and running complex code. Distributed map is exciting, right!?!

Summary

You have now reviewed a pre-created workflow with distributed map, learned some important attributes of the distributed map definition, and ran the workflow yourself using the AWS console.

While the AWS console provides a convenient way to execute workflows at the click of a button, in real-world production environments, large scale data processing workflows are commonly invoked per schedule or the occurrence of an event (ex. a file upload).

Two common patterns to invoke AWS Step Functions state machines are Amazon EventBridge schedules and S3 event notifications . You can use the following links for instructions to set up event-driven or scheduled executions for the workflows.

In the next module, you will build a distributed map workflow yourself. Get ready for some challenging questions as well!

Building a Distributed Map Workflow

Introduction

In the previous module, you saw an example of distributed processing with a single S3 object. Distributed map not only iterates on a single large object located in S3, it can also iterate on a collection of objects in S3. You can iterate through and process each object in parallel and aggregate the results. This supports various use cases, such as processing thousands of log files, a Monte Carlo simulation which runs the same processing for multiple inputs, running a backfill process that scans millions of files for security vulnerability for past dates.

Below diagram shows how distributed map works for multiple S3 objects. Notice that it uses S3.listObjectV2 instead of S3.GetObject which is used in the previous sub module. When processing multiple objects, distributed map lists the metadata of the objects, distributes batches of the metadata to the child workflows. This means you can process any file format; structured, unstructured, and semi-structured.

What you do in the module

In this module, you are going to build a distributed map workflow that processes thousands of weather data files from NOAA climatology data . The workflow is going to find the highest precipitation for each weather station and store the results in DynamoDB table. Each weather station is an individual S3 object containing various weather data and there are about 1,000 stations.

Services used

AWS Step Functions — Serverless visual workflow service
AWS Lambda — Compute service; functions in serverless runtimes
Amazon DynamoDB is a fully managed, serverless, key-value NoSQL database designed to run high-performance applications. DynamoDB offers built-in security, continuous backups, automated multi-Region replication, in-memory caching, and data export tools.

Pre-created resources

To quickly build the workflow, we have created a few resources ahead of time.

Lambda function to find the highest precipitation for the station
One S3 bucket for the dataset and another S3 bucket for storing distributed map results
Sample dataset of 1,000 S3 objects from NOAA climatology data
Amazon DynamoDB table to store the precipitation data

Building the Workflow

Workflow Studio

Open the Step Functions console then choose Create state machine button.

Choose the Blank card and choose Select.

You are in Workflow Studio. Take a moment to explore it. You will see the actions, flows, and patterns on the left side. You can drag the states on the left to the center of the page where you see the workflow design. You can configure the input, output, errors, etc. on the right side of the UI. If you click the Definition toggle, you can view the ASL definition of the workflow.

Take some time to explore the menus.

Alright! Let’s start building the workflow.

Select the Flow tab under the search text box on the left side.

Drag the Map state between the Start and End states.

Change the Processing mode to Distributed — new.

Alright. You are now going to configure additional attributes of the distributed map. First, you will configure where to read the dataset from. You will pass the location of the precreated dataset in S3 as input.

Select Amazon S3 as the Item source.

Select S3 object list in S3 item source dropdown.

Select Get bucket and prefix at runtime from state input in S3 bucket dropdown.

For Bucket Name, enter $.bucket

For Prefix, enter $.prefix

Enable batching and set the Max items per batch to 100.

Leave everything else as default and move on to adding the child workflow components.

Enter Lambda

in the search textbox at the top left.

Drag and drop the Lambda — Invoke action to the center.

Click on the Function name dropdown and select function with HighPrecipitation in its name.

Review the definition in the workflow studio by clicking the Definition toggle at the right.

Notice that the ItemReader object uses listobjectV2. In sub-module 1, you saw GetObject in ItemReader. The reason is that you processed a single S3 object in sub-module 1 and you are processing multiple S3 objects here in this sub-module. You can nest both patterns to read multiple csv/json files in a highly parallel fashion.

      "ItemReader": {
        "Resource": "arn:aws:states:::s3:listObjectsV2",
        "Parameters": {
          "Bucket.$": "$.bucket",
          "Prefix.$": "$.prefix"
        }
      },

Select the Config tab next to the state machine name at the top of the page and change the State machine name to FindHighPrecipitationWorkflow.

Choose an existing IAM role with the name StatesHighPrecipitation

Choose Create button at the top.

Run the workflow by choosing Start execution button.

Wait! You need the bucket and prefix as input to the workflow.

Open the S3 console then copy the full name of the bucket containing MultiFileDataBucket in its name.

Return to the Start execution popup and enter the following json as input, replacing the bucket name with your bucket name from S3:

{
  "bucket": "bucketname",
  "prefix":"csv/by_station"
}

To verify the ASL definition

{
  "Comment": "A description of my state machine",
  "StartAt": "Map",
  "States": {
    "Map": {
      "Type": "Map",
      "ItemProcessor": {
        "ProcessorConfig": {
          "Mode": "DISTRIBUTED",
          "ExecutionType": "STANDARD"
        },
        "StartAt": "Lambda Invoke",
        "States": {
          "Lambda Invoke": {
            "Type": "Task",
            "Resource": "arn:aws:states:::lambda:invoke",
            "OutputPath": "$.Payload",
            "Parameters": {
              "Payload.$": "$",
              "FunctionName": "arn:aws:lambda:{region}:{account}:function:sfw-processmulti-distribu-HighPrecipitationFunctio-vH7XVagF8llI:$LATEST"
            },
            "Retry": [
              {
                "ErrorEquals": [
                  "Lambda.ServiceException",
                  "Lambda.AWSLambdaException",
                  "Lambda.SdkClientException",
                  "Lambda.TooManyRequestsException"
                ],
                "IntervalSeconds": 1,
                "MaxAttempts": 3,
                "BackoffRate": 2
              }
            ],
            "End": true
          }
        }
      },
      "End": true,
      "Label": "Map",
      "MaxConcurrency": 1000,
      "ItemReader": {
        "Resource": "arn:aws:states:::s3:listObjectsV2",
        "Parameters": {
          "Bucket.$": "$.bucket",
          "Prefix.$": "$.prefix"
        }
      },
      "ItemBatcher": {
        "MaxItemsPerBatch": 100,
        "BatchInput": {
          "Bucket.$": "$.bucket"
        }
      }
    }
  }
}

Fix the workflow

Expand the error ribbon to see the reason for failure.

By default, distributed map fails even when a single child fails. Let’s explore what actually caused the child workflow to fail.

Click Map Run to check the child workflow executions.

Select a child workflow execution to see the error.

It looks like Lambda is expecting event[BatchInput][Bucket] and it is not found. Explore the input to the Lambda function by selecting Execution input and output at the top.

The input only contains the S3 key. It is missing the bucket name.

Alright! You are going back to Workflow Studio to pass the bucket name as input to the Lambda function.

Close this tab and return to the previous tab, or you can browse to the workflow by following the breadcrumb at the top.

Choose Edit button.

Select the map state in the workflow design.

Modify the Batch input under Item batching to include the bucket name.

{ 
  "Bucket.$": "$.bucket"
}

Batch Input enables you to send additional input to the child workflow steps as global data. For ex. Bucket name is a global data and does not have to be in individual line item.

Save the workflow and choose Execute button.

Do not forget to execute the workflow with the proper bucket and prefix input. If you would like, you can get the input by navigating to the previous execution of the workflow.

Voila!! It is a success now!

Viewing the Workflow Results

Verify Results

Open the Lambda console and select the function containing HighPrecipitation in its name.

Explore the Code.

The Lambda function writes the calculated value to DynamoDB table.

def _write_results_to_ddb(high_by_station: Dict[str, Dict]):
    dynamodb = boto3.resource("dynamodb")
    table = dynamodb.Table(os.environ["RESULTS_DYNAMODB_TABLE_NAME"])

The table name comes from the env variable RESULTS_DYNAMODB_TABLE_NAME.

Click Configuration and select Environment Variable to view the table name.

Open the DynamoDB console then select Tables from left side menu.

Select the table name that you saw in the Lambda configuration and choose Explore table items. You can now view the calculated highest precipitation across stations.

Summary

In this module, you built a data processing workflow with distributed map, configuring Step Functions to distribute the S3 objects across multiple child workflows to process them in parallel.

Important

When processing large numbers of objects in S3 with Distributed Map, you have a couple of different options for listing those objects: S3 listObjectsV2 and S3 Inventory List . With S3 listObjectsV2, Step Functions is making S3 listObjectsV2 *API calls on your behalf to retrieve all of the items needed to run the Distributed Map. Each call to *listObjectsV2 can only return a maximum of 1000 S3 objects. This means that if you have 2,000,000 objects to process, Step Functions has to make at least 2000 API calls. This API is fast and it won’t take too long, but if you have an S3 Inventory file that has all the objects listed in it that you need to process, you can use that as the input.

Using an S3 Inventory file as the input for a Distributed Map when processing large numbers of files is faster than S3 listObjectsV2. This is because, for S3 Inventory ItemReaders, there is a single S3 getObject call to get the manifest file and then one call for each Inventory file. If you know that your Distributed Map is going to run on a set schedule you can schedule the S3 Inventory to be created ahead of time.

Module 2 — Advanced

Welcome to the Advanced module of the data processing workshop!

In the basics module, you learnt about how to use distributed map to build large scale data processing solution. For a production ready application, you need to make sure the solution is optimized for performance, cost etc. We will focus on optimizing cost and performance of the distributed map in this module.

Optimization

Introduction

In this module, we use the same example workflow from earlier sub module Building a distributed map workflow. But we pre-created the workflow to make it easier for you.

Services used

AWS Step Functions — Serverless visual workflow service
AWS Lambda — compute service; functions in serverless runtimes
Amazon DynamoDB is a fully managed, serverless, key-value NoSQL database designed to run high-performance applications. DynamoDB offers built-in security, continuous backups, automated multi-Region replication, in-memory caching, and data export tools.

Pre-created resources

A Lambda function to find the highest precipitation for the station.
One S3 bucket for data set and another S3 bucket for storing distributed map results.
Sample data set of 1000 S3 objects from NOAA climatology data .
Amazon DynamoDB table to store the precipitation data.
A Step Functions workflow that processes the sample data.

What you do in the module

Using the precreated Step Functions workflow, you will tune some attributes/fields of distributed map and understand the performance and cost impact of the change.

Choosing the Workflow Type

Workflow types

Step Functions offers two types of workflows — Standard and Express. Standard workflows are ideal for long running workflow; it can run for 365 day whereas express workflows can run only for 5 minutes. Another important distinction is how pricing works. Standard workflows are priced by state transition while express workflows are priced by number of request and the duration. To learn more about the difference, click here .

When you use distributed map, Step Functions spins up child workflows to run the states inside the distributed map. The number of child workflows to spin up is dependent on the number of objects or records to process, batch size and concurrency. You can define the child workflow to run as either standard or express based on your use case.

In the following sections, you will learn how to change workflow type of distributed map child workflows, try out a technique to find if express workflow suits your use case, and the cost impact of running standard vs express.

Workflow studio

Navigate to Step Functions Console , select State machines from the right menu.

Select the workflow that starts with OptimizationStateMachine.

Choose edit button to edit the workflow in workflow studio.

Review the definition in the workflow studio by enabling definition at the right.

Highlight the Distributed map high precipitation step in the workflow graphic.

Check the child workflow ExecutionType. It is set as STANDARD.

      "ItemProcessor": {
        "ProcessorConfig": {
          "Mode": "DISTRIBUTED",
          "ExecutionType": "STANDARD"
        },

Also, explore the batch setting. Each child workflow receives a batch of 100 objects.

      "ItemBatcher": {
        "MaxItemsPerBatch": 100
      },

Identify if workflow can be Express

Child workflow can be run either STANDARD or EXPRESS. Express workflows are generally less expensive and run faster than Standard workflows.

Sometimes, you may not be sure if your workflow runs within 5 minutes. In this section, you are going to use a feature of distributed map that allows you to test your data with small number of items. This technique is helpful in couple of ways

To determine the duration of the child workflow
To gain confidence that the child workflow logic will run fine when running with full data set.

Start making the changes

Toggle Definition button to edit the configuration.
Expand Additional configuration and select Limit number of items.
Type 1 in Max Items textbox.

Save and execute with default input.
Select Map Run from the execution page.

Observe the duration for single item. It is around 3 seconds.

Repeat the above steps with 100 items.
Explore the Map Run page to find the duration for 100 items. It is less than 30 seconds.
Now you know it takes 3 seconds to run 1 item and 26 seconds for 100 items, you can even run all the 1000 items in a single child workflow with 1 concurrency. But, you don’t utilize any parallelism to speed up the process.

This simple test is really handy in finding the right batch size and choosing the workflow type without running the entire dataset!!!

Changing the workflow type to Express

Return to workflow studio.
Change the workflow type to EXPRESS.

Unselect Limit number of items under Additional configuration to revert back to the setting’s pre-test state.
Save the workflow and run.
Explore the Map Run page and note down the duration.

If you now run this workflow as a Standard workflow type and compare the durations, you will notice the Express workflow execution was faster.

Review Cost impact

Consider you are processing 500K objects and set the batch size to 500.

500k objects / 500 objects per workflow = 1000 child workflows

Distributed map runs a total of 1000 child workflows to process 500K objects.

Standard child workflow execution cost

In the example we used in this module, we have one Lambda function inside the child workflow. so, the number of state transition per child workflow is 2:- one transition for starting the child workflow and one for Lambda function.

Total cost = (number of transitions per execution x number of executions) x $0.000025

Total cost = (2 * 1000) x $0.000025 = $0.05

Let’s assume you have 5 steps inside your child workflow, the number of state transitions per child workflow is 6.

Total cost = (number of transitions per execution x number of executions) x $0.000025

Total cost = (6 * 1000) x $0.000025 = $0.15

Let’s take another scenario. You have 2 steps inside your child workflow, the number of state transitions per child workflow is 3. Let’s assume you can not utilize batching, the number of child workflows to complete the work = 500K

Total cost = (number of transitions per execution x number of executions) x $0.000025

Total cost = (3 * 500K) x $0.000025 = $37.4

Express child workflow execution cost

With Express workflows, you pay for the number of requests for your workflow and the duration. With scenario outlined earlier under Review cost impact, we need additional dimension of how long workflow runs to calculate the express workflow cost. Let's assume express child workflow runs for an average of 100sec to process 500 objects using 64-MB memory.

Duration cost = (Avg billed duration ms / 100) * 0.0000001042

Duration cost = (100,000 MS /100) * $ 0.0000001042 = $0.0001042

Express request cost = $0.000001 per request ($1.00 per 1M requests)

workflow cost = (Express request cost + Duration cost) x Number of Requests

workflow cost = ($0.000001 + $0.0001042) x 1000 = $0.10

Express child workflow introduces 1 state transition per child workflow regardless of how many states you have inside the workflow. This is to start each child execution.

Transition cost = (1 * 1000) x $0.000025 = $0.025

Total cost = $0.10 + $0.025 = $0.125

If we repeat the calculation for an express workflow that runs for 30 seconds, the total cost = $0.057

If we repeat the calculation for an express workflow that runs for 1 second to process 1 object because you cannot utilize batching, the total cost = $13.42

What did you observe?

Express workflows are cheaper when the duration is lesser. They are also cost effective if there are more steps in the child workflow or your distributed map cannot make use of batching. Remember, Standard workflows are priced by state transitions meaning when number of steps and number of child workflow executions increase, cost increases.

You can look at from the below chart how express workflow duration affect the cost.

Balancing Cost and Performance Using Concurrency and Batching

Higher parallelism generally means you can run the workflows faster. Higher parallelism with no or sub optimal batching results in higher cost for couple of reasons;

There is more state transitions for standard workflows and request cost for express workflows
Services you use inside the child workflow might have cost for requests.

Higher parallelism also causes scaling bottlenecks for downstream services inside the child workflow. You can use distributed map concurrency control to control the number of parallel workflow. If you have multiple workflows and need to manage downstream scaling then you can use techniques such as queueing and activities .

In the following sections, you will run a few experiments with batch size and understand the performance and cost impact of batch size between standard and express workflows.

Review the workflow

Navigate to Step Functions Console , select State machines from the right menu.

Select the workflow that starts with OptimizationStateMachine.

Choose edit to edit the workflow in workflow studio.

Review the definition in the workflow studio by enabling definition at the right.

Highlight the Distributed map high precipitation step in the workflow graphic.

Each child workflow receives a batch of 100 objects. The concurrency or the parallelism of workflow is set as 1000.

      "ItemBatcher": {
        "MaxItemsPerBatch": 100
      },
      "MaxConcurrency": 1000,

Run the workflow

Execute the workflow with default input.
Select Map Run from the Execution page
Examine the map run results.
Note down the duration. You can see there is 10 child workflow executions. As there are only 1000 objects, with the batch setting of 100, only 10 parallel workflows are triggered.
Close all the tabs

Change batch setting

Navigate to Step Functions Console , select State machines from the right menu.
Select the workflow that starts with OptimizationStateMachine.
Choose edit to edit the workflow in workflow studio.
Highlight the Distributed map high precipitation step and view the configurations
Modify the MaxItemsPerBatch to 1

Save and Execute the workflow with default input
Explore the map run results. You can see 1000 child workflow executions.
All the child workflows are completed in little under 25 seconds

Repeat the exercise with different batch settings. What did you observe?

Yes, The total duration increases when you increase the batch size since a single Lambda is looping through an array passed to it, thus increasing the duration of Lambda execution.

Review the cost impact

You are now going to review the impact of cost with different batch sizes. Assume your workflow is processing 50M S3 objects. Now, let’s look at the cost of distributed map if you define the execution type as standard against express.

Standard child workflow

Assume you need to process 50M objects. You have 2 steps inside your child workflow. Each child workflow processes 100 objects in a batch. The number of state transition per child workflow is 3. Total number of child workflows to process 50M objects is 500,000.

Total cost = (number of transitions per execution x number of executions) x $0.000025

Total number of child workflows to process 50M objects = 500,000

Total cost = (3 * 500000) x $0.000025 = $37.5

Express vs Standard vs Batch size

Here is a visual representation of the same information

What did you observe?

Important

The degree of parallelism is determined by the Concurrency setting in Distributed Map, which determines the maximum number of parallel child workflows you want to execute at once. A key consideration here is the service quotas of the AWS services called in your child workflow. For example AWS Lambda in most large regions has a default concurrency quota of 1500 and a default burst limit of 3000, other services such as AWS Rekognition or AWS Textract have much lower default quotas.

The other thing to keep in mind is any performance limitations of other systems that your child workflow interacts with. An example here would be an on-prem relational database that Lambda within a child workflow connects to. This database might have a limit to the number of connections it can support, so you would need to limit your concurrency accordingly. Once you identify all of the AWS Service quotas and any additional concurrency limitation you’ll want to test various combinations of batch size and concurrency to find the best performance within your concurrency constraints

Review the documentation:

Lambda performance optimization

Module 3 — Use Cases

HealthCare Claims Processing

In the US healthcare system, claims are typically categorized as professional, institutional, or dental claims when they are submitted to health insurance payers. Health plans are responsible for validating these claims, responding to the provider, assessing the claims, making payments to the provider, and providing an explanation of benefits to the member. In this module, we focus on the validation phase of the claims process, which occurs after the claims data has already been converted to comply with the FHIR specification. During the validation phase, various business rules are applied to validate and enrich the claims. This represents the final step in the incoming flow of claims before they are transformed into custom data formats required by backend claims adjudication systems.
Vulnerability Scanning

Discovering and reporting vulnerabilities and security issues by scanning documents is a common process. If you are a security partner, when a new customer is onboarded, there can be hundreds of thousands of files to scan. Similarly, if a security procedure changes, previously scanned files may need to be rescanned. Scanning a large number of files is a both time consuming and expensive process. In this module, you will learn to scale a vulnerability scanning application to quickly and efficiently handle hundreds of thousands of files.
Monte Carlo Simulation

A Monte Carlo simulation is a mathematical technique that allows us to predict different outcomes for various changes to a given system. In financial portfolio analysis the technique can be used to predict likely outcomes for aggregate portfolio across a range of potential conditions such as aggregate rate of return or default rate in various market conditions. The technique is also valuable in scenarios where your business case requires predicting the likely outcome of individual portfolio assets such detailed portfolio analysis or stress tests.

Healthcare Claims Processing

Estimated Duration: 30 minutes

Introduction

In the US healthcare system, claims are typically categorized as professional, institutional, or dental claims when they are submitted to health insurance payers. Health plans are responsible for validating these claims, responding to the provider, assessing the claims, making payments to the provider, and providing an explanation of benefits to the member. In this module, we focus on the validation phase of the claims process, which occurs after the claims data has already been converted to comply with the FHIR specification. During the validation phase, various business rules are applied to validate and enrich the claims. This represents the final step in the incoming flow of claims before they are transformed into custom data formats required by backend claims adjudication systems.

You will build a Step Functions workflow that processes healthcare claims data in a highly parallel fashion. The workflow uses the Distributed Map state that runs multiple child workflows, each processing a batch of the overall claims data. Each child workflow picks a set of individual claims files and processes them using AWS Lambda functions that load the data to an Amazon DynamoDB table and then apply rules to determine validity of the claims. Upon processing the claims, the functions returns the output back to the workflow.

What you will accomplish

Learn how to use and configure a Distributed Map state for Healthcare claims data processing
Analyze the results of Distributed Map run
Challenge yourself to optimize the solution

Services in this module

Amazon S3 — Object storage built to retrieve any amount of data from anywhere
AWS Step Functions — Visual workflows for distributed applications
AWS Lambda — Serverless compute service; Run code without thinking about servers or clusters
Amazon DynamoDB — Fully managed, serverless, key-value NoSQL database designed to run high-performance applications

What’s included in this module

Data processing code in the following AWS Lambda functions:

DMapHealthCareClaimProcessingFunction: This function reads a claims file and stores data in an Amazon DynamoDB table (DMapHealthCareClaimTable).
DMapHealthCareRuleEngineLambdaFunction: This function reads data from the Amazon DynamoDB table and applies rules to determine whether the claims need to be accepted or rejected and returns the output. If it is rejected, it will return the reason as well.

Exploring the Dataset

Navigate to the S3 Bucket. Search for dmapworkshophealthcare bucket. This bucket contains 1026 JSON files with around 60,000 records, with a total size of 270MB. These JSON files are generated using Synthea Health Library to simulate patient claims data in FHIR format.

The excerpt below shows one such daily claim record.

Each claim has a claim coding, the service dates, the provider details, the procedure details, an item-wise bill, the total amount and any additional information required for processing.

{
        "resourceType": "Claim",
        "id": "d6c1872c-9b97-0fc1-161a-5c2c9b3f54bf",
        "status": "active",
        "type": {
            "coding": [
                {
                    "system": "http://terminology.hl7.org/CodeSystem/claim-type",
                    "code": "institutional"
                }
            ]
        },
        "use": "claim",
        "patient": {
            "reference": "urn:uuid:86ad439b-3df7-8882-2458-af0d0743d12b",
            "display": "Alvin56 Zulauf375"
        },
        "billablePeriod": {
            "start": "2013-09-09T11:01:44+00:00",
            "end": "2013-09-09T12:01:44+00:00"
        },
        "created": "2013-09-09T12:01:44+00:00",
        "provider": {
            "reference": "Organization?identifier=https://github.com/synthetichealth/synthea|5c896155-eb9a-383e-9162-a43ebb7f1cc5",
            "display": "LINDEN PONDS"
        },
        "priority": {
            "coding": [
                {
                    "system": "http://terminology.hl7.org/CodeSystem/processpriority",
                    "code": "normal"
                }
            ]
        },
        "facility": {
            "reference": "Location?identifier=https://github.com/synthetichealth/synthea|de4402eb-c9e7-3723-9584-345f665c5f5c",
            "display": "LINDEN PONDS"
        },
        "procedure": [
            {
                "sequence": 1,
                "procedureReference": {
                    "reference": "urn:uuid:ece5738e-95ce-4dc6-1df0-95f4dcccce9d"
                }
            }
        ],
        "insurance": [
            {
                "sequence": 1,
                "focal": true,
                "coverage": {
                    "display": "Medicaid"
                }
            }
        ],
        "item": [
            {
                "sequence": 1,
                "productOrService": {
                    "coding": [
                        {
                            "system": "http://snomed.info/sct",
                            "code": "182813001",
                            "display": "Emergency treatment (procedure)"
                        }
                    ],
                    "text": "Emergency treatment (procedure)"
                },
                "encounter": [
                    {
                        "reference": "urn:uuid:c6f74fed-bdb9-6f50-29c5-3519fd948936"
                    }
                ]
            },
            {
                "sequence": 2,
                "procedureSequence": [
                    1
                ],
                "productOrService": {
                    "coding": [
                        {
                            "system": "http://snomed.info/sct",
                            "code": "65546002",
                            "display": "Extraction of wisdom tooth"
                        }
                    ],
                    "text": "Extraction of wisdom tooth"
                },
                "net": {
                    "value": 14150.92,
                    "currency": "USD"
                }
            }
        ],
        "total": {
            "value": 14297.1,
            "currency": "USD"
        }
    }

In the next step, you will build a workflow with a Distributed Map state to analyze this dataset.

Creating the Data Analysis Workflow

In this step, you’ll create a workflow in Workflow Studio to analyze the Healthcare claims data using a Distributed Map state.

Creating the Workflow

Navigate to AWS Step Functions in your AWS console. Make sure you are in the correct region.
If you are not on the State machines page, choose State machines on the left side hamburger menu icon and then select Create state machine
On the Choose a template overlay, choose the Blank template, and select Select.

Select patterns tab and drag Process S3 objects onto the Workflow Studio canvas.

Configure the Distributed Map state with the following values:

Select the Lambda Invoke state within the Distributed Map state. Configure the state with the following values.

Search for AWS Lambda and drag the Invoke state onto the canvas within the Distributed Map under the existing Lambda state. Configure the state with the following values:

Select the Config tab next to the state machine name at the top of the page and Edit the State machine name: HealthCareClaimProcessingStateMachine

If you don’t use this exact name, you may receive an IAM error in the next step.

For the Execution role, choose an existing role containing HealthCareClaimProcessingStateMachineRole in its name
Leave the rest of the defaults and select Create.

In the next step, you’ll execute the workflow and view the results of the data processing job.

Executing the Workflow and Viewing the Results

View map run

Select Start execution and use the default input payload.
The execution will take up to 5 minutes to complete successfully.
In the execution details page, select Distributed Map state in the Graph View, then select Details tab.

Select Map Run link to view details of the Distributed Map execution.
This page provides a summary of the Distributed Map job.

We can see that 21 child workflow executions completed successfully with 0 failures. Each child workflow processed 50 files.
We can view the duration of each child workflow execution. You can see overlapping timestamps for the start and end times, indicating that the data was processed in parallel.
If you select the execution name, you can use the Execution Input and output tab to view the input files for a child workflow execution and the execution output with details.

Verifying DynamoDB results

The Validate Claim function will apply the rules on the claims and store the claim status (Approved / Rejected) along with the rejected reason in the DynamoDB table DMapHealthCareClaimTable. You can use the gear icon on the right side of the screen to select which columns you want to view.

Summary

You have come to the end of the Healthcare Claim Processing Module. In this module, you created a Workflow with distributed map, learnt some important attributes of distributed map definition and run the workflow yourself.

Congratulations! You used Distributed Map state to quickly process a large dataset using parallel processing.

Extra Credits

Great! You have now executed and analyzed the results of the workflow. Well done!

But you cannot stop there! You will need to optimize for performance and cost!

Here is a list of things to try in order to understand the various handles you have at your disposal to optimize a workflow:

Increase the concurrency limit to 1000 and execute it again. Does it change the duration of the execution?
What happens if you decrease the Item Batching size to 25 and execute the workflow? What is the impact on duration as well as cost?
What combination of concurrency limit and batching size would be optimal?
What happens if you change the type of the workflow to ‘Express’ and execute it? What is the impact on cost? Would this workflow type work for any batching size of the provided data set?

Review what you learnt previously in this workshop on concurrency and batching

Security Vulnerability Scanning

Estimated Duration: 30 minutes

Introduction

You are developing a security vulnerability scanning application that alerts you of sensitive information in plain text files. The application you have built executes a workflow that scans a single file for exposed social security numbers (SSNs) and, if one is detected, sends a message to a queue for further downstream processing.

You intended to scan files singly as they got uploaded to your S3 bucket, but, due to delays in development, you have accumulated a large backlog of unscanned, potentially vulnerable files in S3.

What you do in the module

In this module, you will scale your security vulnerability scanning application using Step Functions Distributed Map to quickly address this backlog by processing multiple files concurrently.

Services in this module

Amazon S3 — Object storage built to retrieve any amount of data from anywhere
AWS Step Functions — Visual workflows for distributed applications
AWS Lambda — Run code without thinking about servers or clusters
Amazon SQS — Fully managed message queuing for microservices, distributed systems, and serverless applications

Reviewing the Workflow

Open the Step Functions console .
Select the state machine containing “VulnerabilityScanningStateMachine” in its name.
Click Edit to review the design in Workflow Studio.

Select the Lambda state titled “Scan”.
Under API Parameters, click View function to review the Code.

Reference

Return to Workflow Studio.
Select the Choice state.
Under Choice Rules, click the edit icons to expand the rule logic.

The state machine orchestrates a Lambda function which scans a single file and, if an exposed SSN is detected, sends an SQS message with the location of the SSN and its serial number (the last four numbers).

Executing the Workflow

Open the S3 console .
Click the name of the bucket containing “vulnerabilitydatabucket”.
Copy both the full name of the bucket and the name of a file in that bucket.
Return to Workflow Studio.
Click Execute.
Enter the following input, replacing [bucket name] and [file name] with the names you copied:

{
"detail": {
"bucket": {
"name": "[bucket name]"
},
"object": {
"key": "[file name]"
}
}
}

Click Start execution then wait a moment for the execution to succeed.

Your successful graph will vary depending on the file name you copied. Only executions that process a file with an exposed SSN will send a message to the queue.

Scaling the Workflow with Distributed Map

In this section, you will add a Distributed Map state around the existing workflow to process multiple files concurrently.

Click Edit state machine.
Select the Code tab.
Overwrite the JSON definition with the following:

{
"StartAt": "Map",
"States": {
"Map": {
"Type": "Map",
"ItemProcessor": {
"ProcessorConfig": {
"Mode": "DISTRIBUTED",
"ExecutionType": "STANDARD"
},
"StartAt": "Scan",
"States": {
"Scan": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"OutputPath": "$.Payload",
"Parameters": {
"Payload.$": "$",
"FunctionName": ""
},
"Retry": [
{
"ErrorEquals": [
"Lambda.ServiceException",
"Lambda.AWSLambdaException",
"Lambda.SdkClientException",
"Lambda.TooManyRequestsException"
],
"IntervalSeconds": 2,
"MaxAttempts": 6,
"BackoffRate": 2
}
],
"Next": "Choice"
},
"Choice": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.ssns",
"IsPresent": true,
"Next": "Queue"
}
],
"Default": "Pass"
},
"Queue": {
"Type": "Task",
"Resource": "arn:aws:states:::sqs:sendMessage",
"Parameters": {
"MessageBody.$": "$",
"QueueUrl": ""
},
"End": true
},
"Pass": {
"Type": "Pass",
"End": true
}
}
},
"End": true,
"Label": "Map",
"MaxConcurrency": 1000,
"ItemReader": {
"Resource": "arn:aws:states:::s3:listObjectsV2",
"Parameters": {
"Bucket.$": "$.bucket",
"Prefix.$": "$.prefix"
}
}
}
}
}
Select the Design tab.

Select the Lambda state titled “Scan”.
Under API Parameters, click on the Enter function name dropdown then select the name containing “VulnerabilityScanning”.
Select the SQS state titled “Queue”.
Under API Parameters, click on the Enter queue URL dropdown then select the URL containing “VulnerabilitiesQueue”.
Select the Map state.
Under Item source, expand Additional configuration.

Limiting the number of items that are sent to the Map state is useful for performing trial executions.

Select Limit number of items then enter 1000 under Max items.

Modify items before they are passed on to child executions, selecting only the relevant items and adding input where needed with ItemSelector.

Select Modify Items with ItemSelector then enter the following JSON:

{
"detail": {
"bucket": {
"name.$": "$.bucket"
},
"object": {
"key.$": "$$.Map.Item.Value.Key"
}
}
}

Express Workflows are ideal for quick, high-volume workloads like this. They can run for up to five minutes.

Under Child execution type, choose Express.
Select the Code tab again to review the JSON definition.

Your FunctionName and QueueUrl will be different.

{
  "StartAt": "Map",
  "States": {
    "Map": {
      "Type": "Map",
      "ItemProcessor": {
        "ProcessorConfig": {
          "Mode": "DISTRIBUTED",
          "ExecutionType": "EXPRESS"
        },
        "StartAt": "Scan",
        "States": {
          "Scan": {
            "Type": "Task",
            "Resource": "arn:aws:states:::lambda:invoke",
            "OutputPath": "$.Payload",
            "Parameters": {
              "Payload.$": "$",
              "FunctionName": "arn:aws:lambda:us-east-1:172187416625:function:vulnerability-scanning-mo-VulnerabilityScanningFun-G786d6ZNTaPI:$LATEST"
            },
            "Retry": [
              {
                "ErrorEquals": [
                  "Lambda.ServiceException",
                  "Lambda.AWSLambdaException",
                  "Lambda.SdkClientException",
                  "Lambda.TooManyRequestsException"
                ],
                "IntervalSeconds": 2,
                "MaxAttempts": 6,
                "BackoffRate": 2
              }
            ],
            "Next": "Choice"
          },
          "Choice": {
            "Type": "Choice",
            "Choices": [
              {
                "Variable": "$.ssns",
                "IsPresent": true,
                "Next": "Queue"
              }
            ],
            "Default": "Pass"
          },
          "Queue": {
            "Type": "Task",
            "Resource": "arn:aws:states:::sqs:sendMessage",
            "Parameters": {
              "MessageBody.$": "$",
              "QueueUrl": "https://sqs.us-east-1.amazonaws.com/172187416625/vulnerability-scanning-module-VulnerabilitiesQueue-UtFIvCLDH5Lh"
            },
            "End": true
          },
          "Pass": {
            "Type": "Pass",
            "End": true
          }
        }
      },
      "End": true,
      "Label": "Map",
      "MaxConcurrency": 1000,
      "ItemReader": {
        "Resource": "arn:aws:states:::s3:listObjectsV2",
        "Parameters": {
          "Bucket.$": "$.bucket",
          "Prefix.$": "$.prefix"
        },
        "ReaderConfig": {
          "MaxItems": 1000
        }
      },
      "ItemSelector": {
        "detail": {
          "bucket": {
            "name.$": "$.bucket"
          },
          "object": {
            "key.$": "$$.Map.Item.Value.Key"
          }
        }
      }
    }
  }
}

}
}

If the changes to the JSON definition look accurate, click Save.

Executing the Map Workflow

In this section, you will test last section’s saved workflow. You will execute the workflow with the necessary input, explore child workflow executions, and verify the outputs by polling messages in the SQS queue.

Click Execute.
Enter the following input, replacing [bucket name] with the bucket name you copied:

{
"bucket": "[bucket name]",
"prefix": ""
}

Click Start execution.

Under Events, click Map Run.

Because you added a limitation on the number of items processed, under Executions, there are only 1000 workflows running despite there being more files in S3.

Click a given child execution.
After a successful execution, select the Execution input and output tab.

Again, your output will vary depending on the file that was executed on. Executions that process a clean file with no exposed SSN will return an empty object.

Return to the parent execution view.

The final graph view should look like this upon completion.

Open the SQS console .
Select the queue containing “VulnerabilitiesQueue” in its name.
Click Send and receive messages.
Click Poll for messages.
Click a given message to view the Body.

The body of the message contains the location and serial number of the SSN(s).

Click Done.

Notice the number of Messages available in the queue. You will now purge the queue in anticipation of the next full state machine execution.

Click the queue name.

Under the queue name, click Purge.
Enter purge then click Purge.

Optimizing the Workflow with Batching

In this section, you will modify your Step Functions workflow and Lambda function to enable batch processing, optimizing the performance and cost of your workflow.

Modify the Workflow

Return to the Step Functions execution page.
Click Edit state machine.
Select the Map state.

You can optimize the performance and cost of your workflow by selecting a batch size that balances the number of items against the items processing time. If you use batching, Step Functions adds the items to an Items array. It then passes the array as input to each child workflow execution.

Under Item batching, select Enable batching.
Set the Max MBs per batch to 50 KBs.

With batch input, you can pass a global JSON input to each child execution, merged with the inputs for items. In the last section, the bucket name was included in every single item, increasing the total input size. Instead of including the bucket name repeatedly with ItemSelector, you will include it once in the batch input.

Enter the following Batch input:

{
"bucket.$": "$.bucket"
}
Under Item source, expand Additional configuration.
Under Modify items with ItemSelector, overwrite the JSON with the following, removing details of the bucket:

{
"key.$": "$$.Map.Item.Value.Key"
}
Unselect Limit number of items to process all your files.

Express Workflows only run for up to five minutes and batch processing increases the number of files processed per child execution. If you configure a sufficiently large batch size, you may need to use a Standard Workflow.

Under Child execution type, select Standard.
Select the Code tab to review the JSON definition.

Your FunctionName and QueueUrl will be different.

{
  "StartAt": "Map",
  "States": {
    "Map": {
      "Type": "Map",
      "ItemProcessor": {
        "ProcessorConfig": {
          "Mode": "DISTRIBUTED",
          "ExecutionType": "STANDARD"
        },
        "StartAt": "Scan",
        "States": {
          "Scan": {
            "Type": "Task",
            "Resource": "arn:aws:states:::lambda:invoke",
            "OutputPath": "$.Payload",
            "Parameters": {
              "Payload.$": "$",
              "FunctionName": "arn:aws:lambda:us-east-1:172187416625:function:vulnerability-scanning-mo-VulnerabilityScanningFun-G786d6ZNTaPI:$LATEST"
            },
            "Retry": [
              {
                "ErrorEquals": [
                  "Lambda.ServiceException",
                  "Lambda.AWSLambdaException",
                  "Lambda.SdkClientException",
                  "Lambda.TooManyRequestsException"
                ],
                "IntervalSeconds": 2,
                "MaxAttempts": 6,
                "BackoffRate": 2
              }
            ],
            "Next": "Choice"
          },
          "Choice": {
            "Type": "Choice",
            "Choices": [
              {
                "Variable": "$.ssns",
                "IsPresent": true,
                "Next": "Queue"
              }
            ],
            "Default": "Pass"
          },
          "Queue": {
            "Type": "Task",
            "Resource": "arn:aws:states:::sqs:sendMessage",
            "Parameters": {
              "MessageBody.$": "$",
              "QueueUrl": "https://sqs.us-east-1.amazonaws.com/172187416625/vulnerability-scanning-module-VulnerabilitiesQueue-UtFIvCLDH5Lh"
            },
            "End": true
          },
          "Pass": {
            "Type": "Pass",
            "End": true
          }
        }
      },
      "End": true,
      "Label": "Map",
      "MaxConcurrency": 1000,
      "ItemReader": {
        "Resource": "arn:aws:states:::s3:listObjectsV2",
        "Parameters": {
          "Bucket.$": "$.bucket",
          "Prefix.$": "$.prefix"
        },
        "ReaderConfig": {}
      },
      "ItemSelector": {
        "key.$": "$$.Map.Item.Value.Key"
      },
      "ItemBatcher": {
        "MaxInputBytesPerBatch": 51200,
        "BatchInput": {
          "bucket.$": "$.bucket"
        }
      }
    }
  }
}

If the changes to the JSON definition look accurate, click Save.

Modify the Lambda Function

Since you have enabled batching and changed the input to each child execution, you will need to refactor your Lambda function code to read in the bucket name from BatchInput and process multiple files.

Select the Design tab.
Select the Lambda state titled “Scan”.
Under API Parameters, click View function to edit the Code.

Overwrite the code with the following:

import json
import boto3
import re

def handler(event, context):

bucket = event["BatchInput"]["bucket"]

ssns = []
for item in event["Items"]:

key = item["key"]

obj = boto3.client('s3').get_object(
  Bucket=bucket, 
  Key=key
)
body = obj['Body'].read().decode()

searches = re.findall("ssn=[^\s]+", body)
if searches:
  ssns.extend([{"key": key, "serial": number[-4:]} 
    for ssn, number in (search.split("=") for search in searches)
])

if ssns:
return {"ssns": ssns}
else:
return {}

Click Deploy.

Executing the Batch Workflow

Return to Workflow Studio.
Click Execute.
Enter the following input, replacing [bucket name] with the bucket name you copied::

{
"bucket": "[bucket name]",
"prefix": ""
}

Click Start execution.

Under Events, click Map Run.
Wait a moment, if necessary, then click a given child execution.

Under Events, expand the ID 3 dropdown to view the new input to the Lambda function titled “Payload”.

After a successful invocation, expand the ID 6 dropdown to see the output of the Lambda function.
Return to the parent execution view.

The final graph view should look like this upon completion.

Open the SQS console .
Click the refresh icon, if necessary.

Notice the number of Messages available in the queue. Since you removed the limitation on the number of items processed, it is much higher than before.

Summary

In this module, you added a Distributed Map state to your workflow to scale your security vulnerability scanning application across multiple files concurrently. By refactoring your application and enabling batch processing, you further optimized the performance and cost of your workflow.

Beyond security vulnerability scanning, AWS Step Functions can be used for other data processing use cases, from typical cleaning and normalization to healthcare claims processing.

Monte Carlo Simulation

Estimated Duration: 30 minutes

Introduction

A Monte Carlo simulation is a mathematical technique that allows us to predict different outcomes for various changes to a given system. In financial portfolio analysis the technique can be used to predict likely outcomes for aggregate portfolio across a range of potential conditions such as aggregate rate of return or default rate in various market conditions. The technique is also valuable in scenarios where your business case requires predicting the likely outcome of individual portfolio assets such detailed portfolio analysis or stress tests.

For this fictitious use case we will be working with a portfolio of personal and commercial loans owned by our company. Each loan is represented by a subset of data housed in individual S3 objects. Our company has tasked us with trying to predict which loans will default in the event of a Federal Reserve rate increase.

Loan defaults occur when the borrower fails to repay the loan. Predicting which loans in a portfolio would default in various scenarios helps companies understand their risk and plan for future events.

What is a Worker?

In this solution we are distributing the data using a Step Functions Activity . Activities are an AWS Step Functions feature that enables you to have a task in your state machine where the work is performed by a worker that can be hosted on Amazon Elastic Compute Cloud (Amazon EC2), Amazon Elastic Container Service (Amazon ECS), mobile devices — basically anywhere. Think of activity as a Step Functions managed internal queue. You use an activity state to send data to the queue, one or more workers will consume the data from the queue. For this solution we utilize Amazon ECS on Amazon Fargate to run our Activity Workers (Worker).

Solution Overview

The solution uses AWS Step Functions to provides end to end orchestration for processing billions of records with your simulation or transformation logic using AWS Step Functions Distributed Map and Activity features. At the start of the workflow, Step Functions will scale the number of workers to a (configurable) predefined number. It then reads in the dataset and distributes metadata about the dataset in batches to the Activity. The workers are polling the Activity looking for data to process. Upon receiving a batch, the worker will process the data and report back to Step Functions that the batch has been completed. This cycle continues until all records from the dataset have been processed. Upon completion, Step Functions will scale the workers back to zero.

The workers in this example are containers, running in Amazon Elastic Container Service (ECS) with an Amazon Fargate Capacity Provider . Though the workers could potentially run almost anywhere so long as they had access to poll the Step Functions Activity and report SUCCESS/FAILURE back to Step Functions.

Module Goals

Learn how Step Functions Distributed Map can use a Step Functions Activity to distribute work to workers almost anywhere
Learn how Step Functions can manage workers through its built-in Amazon ECS integrations

Updating the ECS Service — Part 1

The solution uses Amazon ECS to run the workers that handle the actual data processing. In this example we have created an ECS Service that will run a variable number of ECS Tasks, controlled by our Step Functions workflow. The workers run asynchronously from the Distributed Map, which uses an Activity to distribute the dataset. In this step you will configure that ECS Service to use a Task Definition that was predefined in CloudFormation. Let’s get started.

Important

A task definition is a blueprint for your application. It is a text file in JSON format that describes the parameters and one or more containers that form your application. You can learn more here

Navigate to the ECS Console Page
Choose the ECS Cluster named “sfn-fargate-dataproc-xxxxxxxx” (note the similar Cluster named sfn-fargate-datagen-xxxxxxxx, please choose the one ending in dataproc)
Choose ECS Cluster

Choose the ECS Service named “sfn-fargate-dataproc-xxxxxxxx”
Choose ECS Service

Click the Update Service button in the top right of the ECS Service page
Update Service

In the field “Family”, click the Dropdown menu and change the current Task Definition, sfn-fargate-dataproc-placeholder-xxxxxxxx, and choose the Task Definition named sfn-fargate-dataproc-small-xxxxxxxx
Choose Task Definition

Leave all other fields default. Scroll to the bottom of the page and click Update.

That’s it! You have successfully updated your ECS Service to use a new Task Definition. Now lets run our Step Functions State Machine.

Executing the Workflow

Let’s go ahead and run the Step Function and then we will walk through each step as well as some optimizations for you to consider….

In your AWS Console navigate to the AWS Step Functions Console by using the search bar in the upper left corner of your screen, typing “step functions” and clicking the Step Functions icon.

Console Navigation

Open the “sfn-fargate-dataproc-xxxxxxxxxxxx” Step Function by clicking on the Link and then click the “Start Execution” button in the upper right-hand corner of the details screen.

Step Function Selection

Click the “Start execution” button to start the workflow.

Step Function Execution

3a. You will be prompted for JSON input, leave default and click “Start Execution”

Step Function Execution — Take Defaults

We can then monitor the process of the Step Function State Machine process from the execution status screen. Processing of the records in the simulated dataset takes just a few minutes.

Now, that will take about 5 minutes to complete. While it’s running, lets dive a little deeper into each step in the workflow.

Reviewing the Workflow

Distributed Map

The Processing DMap step is an AWS Step Functions Distributed Map step that reads the S3 Inventory manifest provided by the Parent Map and processes the referenced S3 Inventory files. For this use case each line of the S3 Inventory file contains metadata referencing an object is S3 containing a single customer loan. The Distributed Map feature creates batches of 400 loan files per our configuration for concurrent distribution to the Step Functions Activity step. Each Distributed Map step supports up to ten thousand concurrent workers. For this example, the Runtime Concurrency is set to 1000. As Step Functions adds messages to the Activity, the workers are polling to pull batches for processing. Once complete, the worker reports back to Step Functions to acknowledge the batch has been completed. Step Functions removes that message from the Activity and adds a new batch, it will repeat this process until all batches have been completed.

Distributed Map Configuration Example

As Step Functions Distributed Map can scale to massive concurrency very quickly it is advisable to configure back off and retries within our process to allow downstream systems to scale to meet the processing needs. We utilize Step Functions Distributed Map retry feature to implement graceful back offs without any code. In this example we have configured retry logic for S3 bucket. Each new S3 bucket allocates a single throughput partition of 5,500 reads and 3,500 writes per second which auto-scales based on usage patterns. To allow S3 to auto-scale to meet our workloads write demands the we configure the base retry delay, maximum retires, and back-off within the step function.

Step Functions Retry Configuration

Amazon ECS Workers

The Amazon Elastic Container Service (ECS) Cluster is using a Fargate Spot Capacity Provider to reduce costs and eliminate maintaining EC2 instances. Fargate provides us with AWS managed compute for scheduling our containers.

ECS Cluster / Capacity Provider

The Task Definition in the solution is where we define resources required for our containers to complete. Resources such as network requirements, number of vCPU’s, amount of RAM, etc. In this solution, to avoid building a container, we’re simply using an unmodified Amazon Linux 2023 container but specifying a bootstrap sequence. Bootstrapping lets us give the container a series of commands to execute on start. When the container comes up, it will download a pre-generated python script from our S3 bucket and execute it. This python script has the sample logic we want to run against our dataset.

Task Definition

{
    "taskDefinitionArn": "arn:aws:ecs:us-east-2:123456789101:task-definition/sfn-fargate-dataproc-06f7c24a2819:1",
    "containerDefinitions": [
        {
            "name": "sfn-fargate-dataproc-06f7c24a2819",
            "image": "public.ecr.aws/amazonlinux/amazonlinux:2023",
            "cpu": 256,
            "memory": 512,
            "links": [],
            "portMappings": [],
            "essential": true,
            "entryPoint": [],
            "command": [
                "/bin/sh",
                "-c",
                "yum -y update && yum -y install awscli python3-pip && python3 -m pip install boto3 && aws s3 cp s3://${SOURCEBUCKET}/script/fargate.py . && python3 fargate.py"
            ],
            "environment": [
                {
                    "name": "ACTIVITY_ARN",
                    "value": "arn:aws:states:us-east-2:123456789101:activity:sfn-fargate-activity-06f7c24a2819"
                },
                {
                    "name": "DESTINATIONBUCKET",
                    "value": "sfn-datagen-destination-060aa4adf045"
                },
                {
                    "name": "RECORDCOUNT",
                    "value": "105000"
                },
                {
                    "name": "SOURCEBUCKET",
                    "value": "sfn-datagen-source-060aa4adf045"
                },
                {
                    "name": "REGION",
                    "value": "us-east-2"
                }
            ],
            "environmentFiles": [],
            "mountPoints": [],
            "volumesFrom": [],
            "secrets": [],
            "dnsServers": [],
            "dnsSearchDomains": [],
            "extraHosts": [],
            "dockerSecurityOptions": [],
            "dockerLabels": {},
            "ulimits": [],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "sfn-fargate-dataproc-06f7c24a2819",
                    "awslogs-region": "us-east-2",
                    "awslogs-stream-prefix": "sfn-fargate"
                },
                "secretOptions": []
            },
            "systemControls": []
        }
    ],
    "family": "sfn-fargate-dataproc-06f7c24a2819",
    "taskRoleArn": "arn:aws:iam::123456789101:role/sfn-fargate-ecs-task-role-06f7c24a2819",
    "executionRoleArn": "arn:aws:iam::123456789101:role/sfn-fargate-ecs-exec-role-06f7c24a2819",
    "networkMode": "awsvpc",
    "revision": 1,
    "volumes": [],
    "status": "ACTIVE",
    "requiresAttributes": [
        {
            "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
        },
        {
            "name": "ecs.capability.execution-role-awslogs"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.17"
        },
        {
            "name": "com.amazonaws.ecs.capability.task-iam-role"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
        },
        {
            "name": "ecs.capability.task-eni"
        }
    ],
    "placementConstraints": [],
    "compatibilities": [
        "EC2",
        "FARGATE"
    ],
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "cpu": "256",
    "memory": "512",
    "registeredAt": "2023-09-02T15:36:54.377Z",
    "registeredBy": "arn:aws:sts::123456789101:assumed-role/haughtmx-vscode-remote-ssh-role/i-00c5182010adb45a3",
    "tags": [
        {
            "key": "Name",
            "value": "sfn-fargate-dataproc-06f7c24a2819"
        }
    ]
}

The tasks are managed with an ECS Service. A Service can allow for integration with other services such as Elastic Load Balancers, but in our case, we’re using it to manage how many tasks are running at any given time. In the “Scale Out Workers” step of the workflow we are using Step Functions built-in integration with ECS to scale out the Service to 50. This way, once Distributed Map begins filling the Activity with batches of work, the containers are already running and immediately begin picking up batches for processing.

ECS Service

Data Processing

The Run Activity step consists of a single Step Functions Activity which accepts a JSON payload from Distributed Map containing a batch of Loan objects stored in S3. The Run Activity step will continuously add/remove batches of loans by reading the contents of the S3 object. These batches are picked up by our workers and processed. After the worker reports that a batch was successfully processed, Step Functions will remove the batch from the Activity and adds a new one in its place, maintaining our concurrency limit of batches until all batches are processed. In this example the output files contain batched loans to facilitate more efficient reads for analytics and ML workloads.

Step Functions error handling features removes the need to implement catch and retry logic within the python code. If a batch fails processing or a worker fails to report the batch as complete, Step Functions will wait the allotted time and re-add the batch for another worker to pick up and process.

Example Processing Python Code

import boto3

import json

import csv

from io import StringIO

import os

import time

from random import randint

from botocore.client import Config

  
  
  set a few variables we'll use to get our data


activity_arn = os.getenv('ACTIVITY_ARN')

worker_name = os.getenv('HOSTNAME')

region = os.getenv('REGION')

print('starting job...')

  
  
  setup our client


config = Config(

  connect_timeout=65,

  read_timeout=65,

  retries={'max_attempts': 0}

)

client = boto3.client('stepfunctions', region_name=region, config=config)

s3_client = boto3.client('s3', region_name=region)

s3 = boto3.resource('s3')

  
  
  now we start polling until we have nothing left to do. i realize this should


  
  
  be more functions and it's pretty gross but it works for a demo :)


while True:

  response = client.get_activity_task(

    activityArn = activity_arn,

    workerName = worker_name

  )

if 'input' not in response.keys() or 'taskToken' not in response.keys():

    print('no tasks to process...waiting 30 seconds to try again')

    time.sleep(30)

    continue

    # break

token = response['taskToken']

  data = json.loads(response['input'])

  items = data['Items']

  other = data['BatchInput']

  rndbkt = other['dstbkt'] 

  success = True

  cause = ""

  error = ""

  results = ["NO", "NO", "NO", "NO", "YES", "NO", "NO", "NO", "NO", "NO", "NO", "NO", "NO", "NO", "NO", "NO", "NO", "NO", "NO", "NO"]

  for item in items:

    try:

      source = s3_client.get_object(Bucket=other['srcbkt'], Key=item['Key'])

      content = source.get('Body').read().decode('utf-8')

      buf = StringIO(content)

      reader = csv.DictReader(buf)

      objects = list(reader)

  # just randomly assign a value with a theoretical ballpark of 5% of the values being 'YES'
  objects[0]['WillDefault'] = results[randint(0,19)]

  stream = StringIO()
  headers = list(objects[0].keys())
  writer = csv.DictWriter(stream, fieldnames=headers)
  writer.writeheader()
  writer.writerows(objects)
  body = stream.getvalue()

  dst = s3.Object(rndbkt, other['dstkey'] + '/' + item['Key'].split('/')[1])
  dst.put(Body=body)

except Exception as e:
  cause = "failed to process object " + item['Key'],
  error = str(e)
  success = False
  break



if success:

    client.send_task_success(

      taskToken = token,

      output = "{\"message\": \"success\"}"

    )

  else:

    client.send_task_failure(

      taskToken = token,

      cause = cause,

      error = error

    )

Updating the ECS Service — Part 2

The update you made in the “Updating the ECS Service — part 1” step used a task definition with a relatively low resource setting, 0.25vcpu and 0.5GB of memory. In this step you will choose a Task Definition that uses the same container as the previous step, but with double the previous resources. Then you will re-run the State Machine and check the differences between the executions. Let’s get started.

Navigate to the ECS Console Page
Choose the ECS Cluster named “sfn-fargate-dataproc-xxxxxxxx” (note the similar Cluster named sfn-fargate-datagen-xxxxxxxx, please choose the one ending in dataproc)
Choose ECS Cluster

Choose the ECS Service named “sfn-fargate-dataproc-xxxxxxxx”
Choose ECS Service

Click the Update Service button in the top right of the ECS Service page
Update Service

In the field “Family”, click the Dropdown menu and change the current Task Definition, sfn-fargate-dataproc-small-xxxxxxxx, and choose the Task Definition named sfn-fargate-dataproc-large-xxxxxxxx
Choose Task Definition

Leave all other fields default. Scroll to the bottom of the page and click Update.
Now that you have updated the Service, lets run the State Machine again. If you need instructions please refer back to Execute The State Machine

Give that approximately 4 minutes to complete

Reviewing the Results

In the first execution you used a Task Definition that allocated .25 vCPU’s and .5GB of RAM to each container. In the second execution you used a Task Definition that allocated .5 vCPU’s and 1GB of RAM to each container. Let’s check out the results and see how they differ.

Go to the Step Functions Console
Choose the State Machine named “sfn-fargate-dataproc-xxxxxxxx”
Step Function Selection

On the State Machine Details page you will see 2 executions, open each in a separate tab. (right-click each link and choose “Open Link in New Tab”)
Open Executions

On each tab view the details of the execution and find the Duration field to see how long each execution required.
Find Duration

Do they differ? For this simulated dataset, which would be considered small for a Monte Carlo Simulation, the difference is typically less than a minute but overall the increased vCPU/RAM will lead to ~25% time savings. However, to get this ~25% savings you had to double the resources for each container, effectively doubling your Fargate costs. If time is your most critical factor, this may be worth it. If cost is your most critical factor, perhaps not. The variation for both cost and time are minimal with a dataset this small, however if you extrapolate this to a dataset consisting of millions or even billions of objects, both time and cost variations are considerable. You will want to experiment with your actual workloads on what settings work best.

Cleanup

Login to the AWS account where you deployed the module.
Open the S3 console then empty both buckets containing “hellodmap” in the name.
Open the CloudFormation console , select the stack with a name containing “sfw-hello-distributed-map” (or with the name you entered earlier), then click Delete.
Make sure the stack deletion completes.
Important
Follow the instructions on this page only if you are executing this workshop in your own account.
Navigate to S3 . Search for sfw-optimization. Empty both buckets.
Navigate to CloudFormation console. Search for sfw-optimization-distributed-map. Delete the stack.
Navigate to S3 . Search for dmapworkshophealthcare. Empty the bucket.
Navigate to CloudFormation console. Search for sfw-healthcare-processing. Delete the stack.
Open the CloudFormation console then select the stack with a name containing “vulnerability-scanning-module” (or with the name you entered earlier).
Click Delete.
Make sure the stack deletion completes.
Navigate to S3 . Search for sfn-datagen. Empty both buckets.
Navigate to CloudFormation console.
Search for sfn-fargate. Delete the stack.
Search for sfn-datagen. Delete the stack.

Challenges Faced and Solutions

Challenge 1: Complex Workflow Management

Solution: Leveraged AWS Step Functions’ visual editor to design and troubleshoot each state transition, ensuring a seamless workflow.

Challenge 2: Detailed Monitoring Across Multiple Services

Solution: Integrated AWS CloudWatch and X-Ray to gain a comprehensive view of workflow execution, enabling more effective troubleshooting.

Challenge 3: Ensuring Data Security

Solution: Implemented strict IAM policies to ensure secure access control, preventing unauthorized access to sensitive data.

Conclusion

This project showcases how AWS Step Functions can effectively orchestrate complex data workflows across multiple services, enabling seamless scalability and enhanced reliability. By combining automated error handling, real-time monitoring, and optimized processing techniques, this architecture demonstrates a highly adaptable solution for data-driven organizations. The end result is a streamlined, resilient system capable of handling large datasets efficiently, supporting businesses in making data-informed decisions with minimal operational overhead.

Explore my GitHub repository.

Let’s connect: Linkdin, Twitter, Github

Building Web Applications Using Amazon EKS : AWS Project

Shubham Murti — Wed, 13 Nov 2024 08:07:42 +0000

Introduction

This project focuses on building a highly scalable, resilient web application using Amazon EKS (Elastic Kubernetes Service) for container orchestration. By leveraging key AWS services like Amazon ECR (Elastic Container Registry), Cloud9, and AWS Fargate, along with CI/CD automation and monitoring tools like CloudWatch Container Insights, the setup ensures efficient deployment, auto-scaling, and robust application management. Ideal for DevOps teams, this architecture integrates Kubernetes best practices, providing automated deployments, scaling, and resource management to meet dynamic application demands with high reliability.

Tech Stack

Here’s a rundown of the AWS services, tools, and technologies used:

AWS Cloud9: For an integrated development environment (IDE) on the cloud
Amazon Elastic Container Registry (ECR): To store Docker images for easy deployment
Amazon Elastic Kubernetes Service (EKS): The primary Kubernetes cluster manager
Container Insights: For real-time monitoring and insights into the application
AWS Fargate: For serverless deployment and resource optimization
CI/CD Pipeline: To automate code deployment, enhancing release efficiency

Prerequisites

Before diving in, ensure you meet these prerequisites:

AWS Basic Knowledge: Familiarity with AWS core services and IAM (Identity and Access Management) permissions.
Docker & Kubernetes Basics: Knowledge of containerization and Kubernetes concepts will be helpful.
AWS CLI & eksctl Installed: Both AWS CLI and eksctl should be configured on your local environment.
IAM Setup: Permissions to access AWS resources, especially EKS, ECR, Cloud9, and IAM roles.

Problem Statement or Use Case

Person A, a member of the DevOps team of a famous Korean company, will be in charge of a project to develop a new web applications. The application should be satisfied with the following:

Quickly reflect changes when new requests are occurred
Easy scaling
Operate and develop application with fewer people

After confirming the above requirements, Person A decided to build Modern Application, and after discussing with team members, A wants to build the web application through MSA, container, CI/CD. And they choose kubernetes as container orchestration tool based on majority opinion.

There are few team members and not enough time to work directly to build everything by using open source. Expanding infrastructure is also a pain point. And there are some people who don’t know Kubernetes precisely.

At this point, Person A found out managed kubernetes service, Amazon Elastic Kubernetes Service . And to understand the advantages and characteristics of the Amazon EKS, A determined to do simple PoC(Proof of Concept)!

Architecture Diagram

Below is the architecture diagram for the web application built on Amazon EKS. This high-level view showcases how different AWS services interact to deliver a cohesive deployment environment.

Component Breakdown

Each component plays a vital role in the solution architecture:

Amazon ECR: Stores Docker container images for EKS to pull from and deploy.
Amazon EKS: Manages Kubernetes clusters that house the application, making it scalable and robust.
Container Insights: Monitors the health and performance of the application and clusters.
AWS Fargate: Enables serverless container deployment, reducing infrastructure management needs.
CI/CD Pipeline: Automates code deployment to EKS, enhancing deployment frequency and reliability.

Step-by-Step Implementation

Setting workspace

Build a workspace with AWS Cloud9

This workshop is conducted through AWS Cloud9, a cloud-based integrated development environment(IDE).

AWS Cloud9 is a cloud-based integrated development environment (IDE) that lets you write, run, and debug your code with just a browser. It includes a code editor, debugger, and terminal. Cloud9 comes prepackaged with essential tools for popular programming languages, including JavaScript, Python, PHP, and more, so you don’t need to install files or configure your development machine to start new projects.

Click here to learn more about the features and characteristics of AWS Cloud9.

AWS Cloud9

The order in which you build a workspace with AWS Cloud9 is as follows:

IDE configuration with AWS Cloud9
Create IAM Role
Grant IAM Role to an AWS Cloud9 instance
Update IAM settings in IDE

In case of AWS Event, accounts are already prepared with belows configuration. You can skip this.

IDE configuration with AWS Cloud9

Access AWS Cloud9 console and click the Create environment button.
Write down the IDE name and click Next step. In this lab, type eks-workspace.
Click the other instance type radio button, select t3.medium. In case of platform, select Amazon Linux 2 (recommended) and click Next step button. Check the property value you set, then click Create environment.
When the creation is completed, the screen below appears.

AWS Cloud9 requires third-party-cookies. If the screen above does not appear, refer to here .

Create IAM Role

An IAM role is an IAM identity that you can create in your account that has specific permissions. For the IAM role, it is available for IAM users and services provided by AWS. If you grant an IAM Role to the AWS service, the service performs the delegated role on your behalf.

In this lab, we create an IAM Role with Administrator access policy and attach it to AWS Cloud9.

Click here to enter into IAM Role console.
Check that AWS service and EC2 are selected and click Next: Permissions.
Check that the AdministratorAccess policy is selected and click Next: Tags.
In the Add Tag (optional) step, click Next: Review.
In Role name, type eksworkspace-admin, confirm that the AdministratorAccess managed policy has been added, and click Create role.

For this workshop, the AdministratorAccess policy is used to facilitate the workshop, but it is appropriate to grant minimum privileges when running a production environment.

Grant IAM Role to an AWS Cloud9 instance

The AWS Cloud9 environment is powered by an EC2 instance. Therefore, grant the IAM Role that you just created to the AWS Cloud9 instance in EC2 console.

Click here to enter into EC2 instnace console.
Select AWS Cloud9 instance, then click Actions > Security > Modify IAM Role.

Select eksworkspace-admin in IAM Role section, then click the Save button.

Update IAM settings in IDE

In AWS Cloud9, it dynamically manage IAM credits. Disable these credentials because they are not compatible with EKS IAM authentication. After that attach the IAM Role.

Reconnect to the IDE you created in AWS Cloud9 console, click the gear icon in the upper right corner, and click AWS SETTINGS in the sidebar.
Disable the AWS managed temperature credits setting in the Credentials topic.
Exit Preference tab.

Remove existing credential files to ensure Temporary credentials are not present.

rm -vf ${HOME}/.aws/credentials
Use the GetCallerIdentity CLI command line to check that Cloud9 IDE is using the correct IAM Role. If the result value is shown it is set correctly.

aws sts get-caller-identity --query Arn | grep eksworkspace-admin

kubectl

Install kubectl

kubectl is the CLI that commands a Kubernetes cluster.

Kubernetes uses Kubernetes API to perform actions related to creating, modifying, or deleting objects. When you use the kubectl CLI, the command invokes the Kubernetes API to perform the associated actions.

Click here to install the corresponding kubectl to the Amazon EKS version you want to deploy. In this workshop, we will install the latest kubectl binary(as of August 2023).

sudo curl -o /usr/local/bin/kubectl  \
   https://s3.us-west-2.amazonaws.com/amazon-eks/1.27.4/2023-08-16/bin/linux/amd64/kubectl

sudo chmod +x /usr/local/bin/kubectl

Use the command below to check that the latest kubectl is installed.

kubectl version --client=true --short=true

# Output Sample
Client Version: v1.27.4-eks-8ccc7ba
Kustomize Version: v5.0.1

ETC

Install jq

jq is a command line utility that deals with data in JSON format. Using the command below, install jq.

sudo yum install -y jq

Install bash-completion

In the Bash shell, the kubectl completion script can be created using the kubectl completion bash command. Sourcing the completion script to the shell enables automatic completion of the kubectl command. However, because these completion scripts rely on bash-completion, you must install bash-completion through the command below.

sudo yum install -y bash-completion

Below is only necessary for exploring CI/CD for EKS Cluster

Install Git

Click the Git Downloader link and install the git.

Install Python

Install python because CDK for Python is used. Python is installed by default in the Cloud9 environment. Python Installer Select the appropriate package from the link to download and install it.

python --version
python3 --version

Check PIP

Checks whether PIP, a manager that installs and manages Python packages, is installed. It is installed by default in certain versions of Python and above.

pip
pip3

Since pip version 9.0.3 or higher is required to use CodeCommit, update pip by executing the command below.

curl -O https://bootstrap.pypa.io/get-pip.py
python3 get-pip.py --user

If not installed pip install page recommended to proceed with the installation according to the guide or install with the latest version of Python.

Install eksctl

There are various ways to deploy an Amazon EKS cluster. AWS console, CloudFormation, CDK, eksctl, and Terraform are examples.

In this lab, we will deploy the cluster using eksctl.

eksctl is a CLI tool for easily creating and managing EKS clusters. It is written in Go language and deployed in CloudFormation form.

Download the latest eksctl binary using the command below.

curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp

Move the binary to the location /usr/local/bin.

sudo mv -v /tmp/eksctl /usr/local/bin

Use the command below to check the installation.

eksctl version

AWS Cloud9 Additional Settings

On the previous chapter, we deploy AWS Cloud9 IDE and install the required tools, then proceed to the additional setup below.

Set default value to AWS Region that is currently using.

TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

export AWS_REGION=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r '.region')

echo "export AWS_REGION=${AWS_REGION}" | tee -a ~/.bash_profile

aws configure set default.region ${AWS_REGION}

Check AWS region.

aws configure get default.region

Register the account ID you are currently working on as an environment variable.

export ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text)

echo "export ACCOUNT_ID=${ACCOUNT_ID}" | tee -a ~/.bash_profile
During the docker image build, the AWS Cloud9 environment may experience a capacity shortage issue. To resolve this, run a shell script that extends the disk size.

wget https://gist.githubusercontent.com/joozero/b48ee68e2174a4f1ead93aaf2b582090/raw/2dda79390a10328df66e5f6162846017c682bef5/resize.sh

sh resize.sh

After completion, use the command below to check that the increased volume size is reflected in the file system.

df -h

If you do not set AWS Region, an error occurs when you deploy the cluster and check the relevant information.

Container Image

In this lab, you will learn how to create a container image using a docker platform.

Docker

Docker is a software platform **that allows you to build, test and deploy **containerized applications. Docker packages software into standardized units called containers, which contain everything you need to run the software, including libraries, system tools, code, runtime, and so on.

To learn more about Docker, click here .

Container Image

Container image is a combination of the files and settings required to run the container. These images can be uploaded and downloaded in the repository. And the state in which the image was executed is container. Container images can be downloaded and used by official image repositories such as Docker Hub or created directly.

Build Container Image

This part is an independent lab for those who want to learn more about containers and container images. If you don’t proceed with this lab, you won’t have any problem with configuring web application with Amazon EKS. If you don’t want this lab, move onto the Upload Image to Amazon ECR chapter.

Create Container Image Yourself

Docker File is a setup file for building container images. That is, think of it as a blueprint for the image to be built. When these images become containers, the application is actually running.

Paste the values below in the root folder(/home/ec2-user/environment).

cd ~/environment/

cat << EOF > Dockerfile
FROM nginx:latest
RUN echo '

test nginx web page
' >> index.html RUN cp /index.html /usr/share/nginx/html EOF

Instruction component in the Docker File is as follows.

FROM: Set the Base Image(Specify OS or version)
RUN: Execute any commands in a new layer on top of the current image and commit the results
WORKDIR: Where to perform instructions such as RUN, CMD, ENTRYPOINT, COPY, ADD in the Docker File
EXPOSE: Specify the port number to connect to the host
CMD: Commands for running application
Create an image with the docker build command. In name, enter the name of the container image and in case of tag, if not named, you will have a value called latest. In this lab, you will write test-image by container image name.

docker build -t test-image .

Check the images created with the docker images command.

docker images

Run the image as a container with the docker run command. The command below uses a container image named test-image to run a container named test-nginx, which means that 8080 ports of the host and 80 ports of the container are mapped.

docker run -p 8080:80 --name test-nginx test-image

In other words, information passed to 8080 ports on the host is forwarded through the docker to 80 ports on the container.

You can use the docker ps command to check which containers are running on the current host. Open a new terminal on AWS Cloud9 and type the command below.

docker ps

You can check the status by outputting logs from the container with the docker logs command.

docker logs -f test-nginx

You can access into the inside shell environment of the container with docker exec command. After access, you can apprehend the internal structure and exit through the exit command.

docker exec -it test-nginx /bin/bash

In AWS cloud9, you can see which applications are currently running by clicking the top Tools > Preview > Preview Running Application.

Stop running containers with docker stop command.

docker stop test-nginx

If you type the docker ps command again, you can see that the container that was just running has disappeared from the list.

Delete the container with the docke rm command. The container deletion is possible only when the container is stopped.

docker rm test-nginx

Delete the container image with docke rmi command.

docker rmi test-image

When you type the docker images command, you can check that the container image that was created is not listed.

You can also use the Docker command to perform operations such as cpu or memory restrictions, and sharing directory with hosts.

Upload container image to Amazon ECR

Create Amazon ECR Repository and Upload Image

Create repositories and upload container images in the docker container registry Amazon Elastic Container Registry (ECR).

Amazon Elastic Container Registry(**Amazon ECR **) is an AWS managed container image registry service that is secure, scalable, and reliable. Amazon ECR supports private container image repositories with resource-based permissions using AWS IAM. This is so that specified users or Amazon EC2 instances can access your container repositories and images. You can use your preferred CLI to push, pull, and manage Docker images, Open Container Initiative (OCI) images, and OCI compatible artifacts.

Download the source code to be containerized through the command below.

git clone https://github.com/joozero/amazon-eks-flask.git

Through the AWS CLI, create an image repository. In this lab, we will set the repository name to demo-flask-backend. Also, specify AWS Region code(for example, ap-northeast-2) to deploy the EKS cluster in — region’s value.

aws ecr create-repository \ --repository-name demo-flask-backend \ --image-scanning-configuration scanOnPush=true \ --region ${AWS_REGION}

When you enter this CLI, information about the repository is derived from the resulting value. You can also find the repositories created in the Amazon ECR Console .

For the tasks below, your personal account information will be included. Click on the repository you just created on the Amazon ECR Console , then click View push commands in the upper right corner to find the guide below.

To push the container image to the repository, bring the authentication token and pass the authentication to the docker login command. At this point, specify the user name as AWS and specify the Amazon ECR registry URI that you want to authenticate with.

aws ecr get-login-password --region ${AWS_REGION} | docker login --username AWS --password-stdin $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com

[!] If the above command does not work properly, check whether environment variable named ACCOUNT_ID is called in the terminal.

Input the downloaded source code location(for example, /home/ec2-user/environment/amazon-eks-flask) and enter the command below to build the docker image.

cd ~/environment/amazon-eks-flask docker build -t demo-flask-backend .

When the image is built, use the docker tag command to enable it to be pushed to a specific repository.

docker tag demo-flask-backend:latest $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/demo-flask-backend:latest

Push the image into the repository via the docker push command.

docker push $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/demo-flask-backend:latest

In the Amazon ECR Console, click on the repository you just created to see the uploaded image as shown in the screen below.

We have created container images to deploy on EKS clusters and pushed it into the repository.

Create EKS Cluster

Amazon EKS clusters can be deployed in various ways.

Deploy by clicking on AWS console
Deploy by using IaC(Infrastructure as Code) tool such as AWS CloudFormation or AWS CDK
Deploy by using eksctl
Deploy to Terraform, Pulumi, Rancher, etc.

In this lab, we will create an EKS cluster using eksctl.

Create EKS Cluster with eksctl

If you use eksctl to execute this command (eksctl create cluster) without giving any setting values, the cluster is deployed as a default parameter.

However, we will create configuration files to customize some values and deploy it. In later labs, when you create Kubernetes’ objects, you create a configuration file that is not just created with the kubectl CLI. This has the advantage of being able to easily identify and manage the desired state of the objects specified by the individual.

Paste the values below in the root folder(/home/ec2-user/environment) location.

cd ~/environment

cat << EOF > eks-demo-cluster.yaml

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
name: eks-demo # EKS Cluster name
region: ${AWS_REGION} # Region Code to place EKS Cluster
version: "1.27"

vpc:
cidr: "10.0.0.0/16" # CIDR of VPC for use in EKS Cluster
nat:
gateway: HighlyAvailable

managedNodeGroups:
- name: node-group # Name of node group in EKS Cluster instanceType: m5.large # Instance type for node group desiredCapacity: 3 # The number of worker node in EKS Cluster volumeSize: 20 # EBS Volume for worker node (unit: GiB) privateNetworking: true iam: withAddonPolicies: imageBuilder: true # Add permission for Amazon ECR albIngress: true # Add permission for ALB Ingress cloudWatch: true # Add permission for CloudWatch autoScaler: true # Add permission Auto Scaling ebs: true # Add permission EBS CSI driver
cloudWatch:
clusterLogging:
enableTypes: ["*"]
EOF

If you look at the cluster configuration file, you can define policies through iam.attachPolicyARNs and through iam.withAddonPolicies, you can also define add-on policies. After the EKS cluster is deployed, you can check the IAM Role of the worker node instance in EC2 console to see added policies.

Click here to see the various property values that you can give to the configuration file.

Using the commands below, deploy the cluster.

eksctl create cluster -f eks-demo-cluster.yaml

The cluster takes approximately 15 to 20 minutes to fully be deployed. You can see the progress of your cluster deployment in AWS Cloud9 terminal and also can see the status of events and resources in AWS CloudFormation console.

When the deployment is completed, use command below to check that the node is properly deployed.

kubectl get nodes
Also, you can see the cluster credentials added in ~/.kube/config.

The architecture as of now

After creating a Kubernetes cluster with eksctl, the architecture of the services configured as of now is shown below.

Add Console Credential

Attach Console Credential

The EKS cluster uses IAM entity(user or role) for cluster access control. The rule runs in a ConfigMap named aws-auth. By default, IAM entities used to create clusters are automatically granted system:masters privilege of the cluster RBAC configuration in the control plane.

If you access the Amazon EKS console in the current state, you cannot check any information as below.

When you created the cluster through IAM credentials on Cloud9 in **Create EKS Cluster with eksctl** chapter, so you need to determine the correct credential(such as your IAM Role not Cloud9 credentials) to add for your AWS EKS Console access.

Use the command below to define the role ARN(Amazon Resource Number).

rolearn=$(aws cloud9 describe-environment-memberships --environment-id=$C9_PID | jq -r '.memberships[].userArn')

echo ${rolearn}

[!] If echo command’s result contains assumed-role, perform the additional actions below.

assumedrolename=$(echo ${rolearn} | awk -F/ '{print $(NF-1)}')
rolearn=$(aws iam get-role --role-name ${assumedrolename} --query Role.Arn --output text)

Create an identity mapping.

eksctl create iamidentitymapping --cluster eks-demo --arn ${rolearn} --group system:masters --username admin
You can check aws-auth config map information through the command below.

kubectl describe configmap -n kube-system aws-auth
When the above operations are completed, you will be able to get information from the control plane, the worker node, logging activation, and update information in Amazon EKS console.

On the Workloads tab, you can see the applications placed in the Kubernetes cluster.

On the Configuration tab, you can get cluster configuration detail.

Create Ingress Controller

Ingress Controller

In this lab, we will use AWS Load Balancer Controller for Ingress Controller.

The AWS ALB Ingress Controller has been rebranded to AWS Load Balancer Controller.

Ingress is a rule and resource object that defines how to handle requests, primarily when accessing from outside the cluster to inside the Kubernetes cluster. In short, it serve as a gateway for external requests to access inside of the cluster. You can set up it for load balancing for external requests, processing TLS/SSL certificates, routing to HTTP routes, and so on. Ingress processes requests from the L7.

In Kubernetes, you can also externally expose to NodePort or LoadBalancer type in Service object, but if you use a Serivce object without any Ingress, you must consider detailed options such as routing rules and TLS/SSL to all services. That’s why Ingress is needed in Kubernetes environment.

Ingress means the object that you have set up rules for handling external requests, and Ingress Controller is needed for these settings to work. Unlike other controllers that run as part of the kube-controller-manager, the ingress controller is not created with the cluster by nature. Therefore, you need to install it yourself.

Create AWS Load Balancer Controller

The AWS Load Balancer Controller manages AWS Elastic Load Balancers for a Kubernetes cluster. The controller provisions the following resources.

It satisfies Kubernetes Ingress resources by provisioning Application Load Balancers.
It satisfies Kubernetes Service resources by provisioning Network Load Balancers.

The controller was formerly named the AWS ALB Ingress Controller. There are two traffic modes supported by each type of AWS Load Balancer controller:

Instance(default): Register nodes in the cluster as targets for ALB. Traffic reaching the ALB is routed to NodePort and then proxied to the Pod.
IP: Register the Pod as an ALB target. Traffic reaching the ALB is routed directly to the Pod. In order to use that traffic mode, you must explicitly specify it in the ingress.yaml file with comments.

Create a folder named manifests in the root folder (for example, /home/ec2-user/environment/) to manage manifests. Then, inside the manifests folder, create a folder alb-controller to manage the manifest associated with the ALB Ingress Controller.

cd ~/environment

mkdir -p manifests/alb-ingress-controller && cd manifests/alb-ingress-controller

# Final location
/home/ec2-user/environment/manifests/alb-ingress-controller

Before deploying the AWS Load Balancer controller, we need to do some things. Because the controller operates over the worker node, you must make it accessible to AWS ALB/NLB resources through IAM permissions. IAM permissions can install IAM Roles for ServiceAccount or attach directly to IAM Roles on the worker node.

First, create IAM OpenID Connect (OIDC) identity provider for the cluster. IAM OIDC provider must exist in the cluster(in this lab, eks-demo) in order for objects created by Kubernetes to use *service account *which purpose is to authenticate to API Server or external services.

eksctl utils associate-iam-oidc-provider \
--region ${AWS_REGION} \
--cluster eks-demo \
--approve

[!] Let’s find out a little more here.

The IAM OIDC identity provider you create can be found in Identity providers menu on IAM console or in the commands below.
Check the OIDC provider URL of the cluster through the commands below.

aws eks describe-cluster --name eks-demo --query "cluster.identity.oidc.issuer" --output text

Result from the command have the following format:

https://oidc.eks.ap-northeast-2.amazonaws.com/id/8A6E78112D7F1C4DC352B1B511DD13CF

Copy the value after /id/ from the output above, then execute the command as shown below.

aws iam list-open-id-connect-providers | grep 8A6E78112D7F1C4DC352B1B511DD13CF
If the result appears, IAM OIDC identity provider is created in the cluster, and if no value appears, you must execute the creation operation again.

Create an IAM Policy to grant to the AWS Load Balancer Controller.

curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.5.4/docs/install/iam_policy.json

aws iam create-policy \
--policy-name AWSLoadBalancerControllerIAMPolicy \
--policy-document file://iam_policy.json
Create ServiceAccount for AWS Load Balancer Controller.

eksctl create iamserviceaccount \
--cluster eks-demo \
--namespace kube-system \
--name aws-load-balancer-controller \
--attach-policy-arn arn:aws:iam::$ACCOUNT_ID:policy/AWSLoadBalancerControllerIAMPolicy \
--override-existing-serviceaccounts \
--approve

When deploying an EKS cluster, you can also add the IAM policy associated with the AWS Load Balancer Controller to the Worker node in the form of Addon. However, in this lab, we will conduct with the reference, here .

Also, please refer simple hands on lab about IAM roles for service accounts(IRSA ) in here .

Add Controller to Cluster

Add AWS Load Balancer controller to the cluster. First, install *cert-manager *to insert the certificate configuration into the Webhook. Cert-manager is an open source that automatically provisions and manages TLS certificates within a Kubernetes cluster.

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.12.0/cert-manager.yaml
Download Load balancer controller yaml file.

curl -Lo v2_5_4_full.yaml https://github.com/kubernetes-sigs/aws-load-balancer-controller/releases/download/v2.5.4/v2_5_4_full.yaml
Run the following command to remove the ServiceAccount section in the manifest. If you don’t remove this section, the required annotation that you made to the service account in a previous step is overwritten.

sed -i.bak -e '596,604d' ./v2_5_4_full.yaml

Replace cluster name in the Deployment spec section of the file with the name of your cluster by replacing my-cluster with the name of your cluster.

sed -i.bak -e 's|your-cluster-name|eks-demo|' ./v2_5_4_full.yaml

Deploy AWS Load Balancer controller file.

kubectl apply -f v2_5_4_full.yaml

Download the IngressClass and IngressClassParams manifest to your cluster. And apply the manifest to your cluster.

curl -Lo v2_5_4_ingclass.yaml https://github.com/kubernetes-sigs/aws-load-balancer-controller/releases/download/v2.5.4/v2_5_4_ingclass.yaml

kubectl apply -f v2_5_4_ingclass.yaml

Check that the deployment is successed and the controller is running through the command below. When the result is derived, it means success.

kubectl get deployment -n kube-system aws-load-balancer-controller

In addition, the command below shows that service account has been created.

kubectl get sa aws-load-balancer-controller -n kube-system -o yaml

Pods running inside the cluster for the necessary functions are called Addon. Pods used for add-on are managed by the Deployment, Replication Controller, and so on. And the namespace that this add-on uses is kube-system. Because the namespace is specified as kube-system in the yaml file, it is successfully deployed when the pod name is derived from the command above. You can also check the relevant logs with the commands below.

kubectl logs -n kube-system $(kubectl get po -n kube-system | egrep -o "aws-load-balancer[a-zA-Z0-9-]+")

Detailed property values are available with the commands below.

ALBPOD=$(kubectl get pod -n kube-system | egrep -o "aws-load-balancer[a-zA-Z0-9-]+")

kubectl describe pod -n kube-system ${ALBPOD}

Deploy Microservices

In this lab, you will learn how to deploy the backend, frontend to Amazon EKS, which makes up the web service. The order in which each service is deployed is as follows.

Download source code from git repository
Create a repository for each container image in Amazon ECR
Build container image from source code location, including Dockerfile, and push to repository
Create and deploy Deployment, Service, Ingress manifest files for each service.

The figure below shows the order in which end users access the web service.

Deploy First Backend Service

Deploy flask backend

To proceed with this lab, Upload container image to Amazon ECR part must be preceded.

Move on to manifests folder(/home/ec2-user/environment/manifests).

cd ~/environment/manifests/

Create deploy manifest.

cat < flask-deployment.yaml --- apiVersion: apps/v1 kind: Deployment metadata: name: demo-flask-backend namespace: default spec: replicas: 3 selector: matchLabels: app: demo-flask-backend template: metadata: labels: app: demo-flask-backend spec: containers: - name: demo-flask-backend image: $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/demo-flask-backend:latest imagePullPolicy: Always ports: - containerPort: 8080 EOF

Next, create service manifest.

cat < flask-service.yaml --- apiVersion: v1 kind: Service metadata: name: demo-flask-backend annotations: alb.ingress.kubernetes.io/healthcheck-path: "/contents/aws" spec: selector: app: demo-flask-backend type: NodePort ports: - port: 8080 targetPort: 8080 protocol: TCP EOF

Finally, create ingress manifest.

cat < flask-ingress.yaml --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: "flask-backend-ingress" namespace: default annotations: alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip alb.ingress.kubernetes.io/group.name: eks-demo-group alb.ingress.kubernetes.io/group.order: '1' spec: ingressClassName: alb rules: - http: paths: - path: /contents pathType: Prefix backend: service: name: "demo-flask-backend" port: number: 8080 EOF

Deploy the manifest created above in the order shown below. Ingress provisions Application Load Balancer(ALB).

kubectl apply -f flask-deployment.yaml kubectl apply -f flask-service.yaml kubectl apply -f flask-ingress.yaml

Paste the results of the following command into the Web browser or API platform(like Postman) to check:

echo http://$(kubectl get ingress/flask-backend-ingress -o jsonpath='{.status.loadBalancer.ingress[*].hostname}')/contents/aws

It will take a few time for the ingress object to be deployed. Wait for the Load Balancers status to be active in EC2 console .

The architecture as of now is shown below.

Deploy Second Backend Service

Deploy Express backend

Deploy the express backend in the same order as the flask backend.

The lab below will deploy pre-built container images to skip the image build and repository push process conducted in Upload container image to Amazon ECR.

Move on to manifests folder(/home/ec2-user/environment/manifests).

cd ~/environment/manifests/

Create deploy manifest which contains built pre-built container image.

cat <<EOF> nodejs-deployment.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-nodejs-backend
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: demo-nodejs-backend
  template:
    metadata:
      labels:
        app: demo-nodejs-backend
    spec:
      containers:
        - name: demo-nodejs-backend
          image: public.ecr.aws/y7c9e1d2/joozero-repo:latest
          imagePullPolicy: Always
          ports:
            - containerPort: 3000
EOF

And then, create service manifest file.

cat <<EOF> nodejs-service.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: demo-nodejs-backend
  annotations:
    alb.ingress.kubernetes.io/healthcheck-path: "/services/all"
spec:
  selector:
    app: demo-nodejs-backend
  type: NodePort
  ports:
    - port: 8080
      targetPort: 3000
      protocol: TCP
EOF

create ingress manifest.

cat <<EOF> nodejs-ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: "nodejs-backend-ingress"
  namespace: default
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/group.name: eks-demo-group
    alb.ingress.kubernetes.io/group.order: '2'
spec:
  ingressClassName: alb
  rules:
  - http:
        paths:
          - path: /services
            pathType: Prefix
            backend:
              service:
                name: "demo-nodejs-backend"
                port:
                  number: 8080
EOF

Deploy the manifest files.

kubectl apply -f nodejs-deployment.yaml
kubectl apply -f nodejs-service.yaml
kubectl apply -f nodejs-ingress.yaml
Paste the results of the following command into the Web browser or API platform(like Postman) to check.

echo http://$(kubectl get ingress/nodejs-backend-ingress -o jsonpath='{.status.loadBalancer.ingress[*].hostname}')/services/all
The architecture as of now is shown below.

Deploy Frontend Service

Deploy React Frontend

Once you have deployed two backend services, you will now deploy the frontend to configure the web page’s screen.

Download the source code to be containerized through the command below.

cd /home/ec2-user/environment
git clone https://github.com/joozero/amazon-eks-frontend.git
Through AWS CLI, create an image repository. In this lab, we will set the repository name to demo-frontend.

aws ecr create-repository \
--repository-name demo-frontend \
--image-scanning-configuration scanOnPush=true \
--region ${AWS_REGION}
To spray two backend API data on the web screen, we have to change source code. Change the url values in App.js file and page/upperPage.js file from the frontend source code(location: /home/ec2-user/environment/amazon-eks-frontend/src).

In the above source code, paste the values derived from the result (ingress addresses) below.

echo http://$(kubectl get ingress/flask-backend-ingress -o jsonpath='{.status.loadBalancer.ingress[*].hostname}')/contents/'${search}'

In the above source code, paste the values derived from the result (ingress addresses) below.

echo http://$(kubectl get ingress/nodejs-backend-ingress -o jsonpath='{.status.loadBalancer.ingress[*].hostname}')/services/all

Execute the following command in the location of the amazon-eks-frontend folder.

cd /home/ec2-user/environment/amazon-eks-frontend
npm install
npm run build

[!] After npm install, if severity vulnerability comes out, perform the npm audit fix command and apply npm run build.

Refer Upload container image to Amazon ECR guide and proceed to create container image repository and push image. In this lab, set the image repository name to demo-frontend.

docker build -t demo-frontend .

docker tag demo-frontend:latest $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/demo-frontend:latest

docker push $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/demo-frontend:latest

[!] During applying above CLI, if you receive message which said denied: Your authorization token has expired. Reauthenticate and try again., then applying bottom command line and do this again.

aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com

Move to manifests folder. At this point, type image value to demo-frontend repository URI.

cd /home/ec2-user/environment/manifests

cat < frontend-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
name: demo-frontend
namespace: default
spec:
replicas: 3
selector:
matchLabels:
app: demo-frontend
template:
metadata:
labels:
app: demo-frontend
spec:
containers:
- name: demo-frontend
image: $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/demo-frontend:latest
imagePullPolicy: Always
ports:
- containerPort: 80
EOF

cat < frontend-service.yaml

apiVersion: v1
kind: Service
metadata:
name: demo-frontend
annotations:
alb.ingress.kubernetes.io/healthcheck-path: "/"
spec:
selector:
app: demo-frontend
type: NodePort
ports:
- protocol: TCP
port: 80
targetPort: 80
EOF

cat < frontend-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: "frontend-ingress"
namespace: default
annotations:
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/group.name: eks-demo-group
alb.ingress.kubernetes.io/group.order: '3'
spec:
ingressClassName: alb
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: "demo-frontend"
port:
number: 80
EOF
Deploy manifest file.

kubectl apply -f frontend-deployment.yaml
kubectl apply -f frontend-service.yaml
kubectl apply -f frontend-ingress.yaml
Copy and paste the result of the command below into the web browser.

echo http://$(kubectl get ingress/frontend-ingress -o jsonpath='{.status.loadBalancer.ingress[*].hostname}')
If the screen below comes out same, all the containers are working successfully.

The architecture as of now

After deploying Ingress Controller and Service objects, the architecture configured is shown below.

AWS Fargate

AWS Fargate is a serverless compute engine for containers that works with both Amazon Elastic Container Service (ECS) and Amazon Elastic Kubernetes Service (EKS). Fargate makes it easy for you to focus on building your applications. Fargate removes the need to provision and manage servers, lets you specify and pay for resources per application, and improves security through application isolation by design.

Deploy service with AWS Fargate

Deploy pod with AWS Fargate

To deploy pods to Fargate in a cluster, you must define at least one fargate profile that the pod uses when it runs. In other words, the fargate profile is a profile that specifies the conditions for creating pods with AWS Fargate type.

cd /home/ec2-user/environment/manifests

cat < eks-demo-fargate-profile.yaml

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: eks-demo
region: ${AWS_REGION}
fargateProfiles:
- name: frontend-fargate-profile selectors:
  - namespace: default labels: app: frontend-fargate EOF

For pods that meet the conditions listed in selectors in the yaml file above, it will be deployed as AWS Fargate type.

Deploy fargate profile.

eksctl create fargateprofile -f eks-demo-fargate-profile.yaml
Check whether fargate profile was deployed successfully.

eksctl get fargateprofile --cluster eks-demo -o json

Example of command result

{
"name": "frontend-fargate-profile",
"podExecutionRoleARN": "arn:aws:iam::account-id:role/eksctl-eks-demo-test-farga-FargatePodExecutionRole-OLC3P21AD5DX",
"selectors": [
{
"namespace": "default",
"labels": {
"app": "frontend-fargate"
}
}
],
"subnets": [
"subnet-07e2d55650225419c",
"subnet-0ac4a7fdbd803039c",
"subnet-046a3dcfabce11b5f"
],
"status": "ACTIVE"
}
In this lab, we will provision frontend pods to Fargate type. First, delete the existing frontend pod. Work with the command below in the folder where the yaml file is located.

kubectl delete -f frontend-deployment.yaml
Modify frontend-deployment.yaml file. Compared with previous yaml file, you can see that the value of label changed from demo-frontend to frontend-fargate. In step 1, when the pod meet the condition that key=app, value=frontend-fargate and namespace=default, eks cluster deploy pod to Fargate type.

cd /home/ec2-user/environment/manifests

cat < frontend-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
name: demo-frontend
namespace: default
spec:
replicas: 3
selector:
matchLabels:
app: frontend-fargate
template:
metadata:
labels:
app: frontend-fargate
spec:
containers:
- name: demo-frontend
image: $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/demo-frontend:latest
imagePullPolicy: Always
ports:
- containerPort: 80
EOF
Modify frontend-service.yaml file.

cat < frontend-service.yaml

apiVersion: v1
kind: Service
metadata:
name: demo-frontend
annotations:
alb.ingress.kubernetes.io/healthcheck-path: "/"
spec:
selector:
app: frontend-fargate
type: NodePort
ports:
- protocol: TCP
port: 80
targetPort: 80
EOF
Deploy manifest file.

kubectl apply -f frontend-deployment.yaml
kubectl apply -f frontend-service.yaml
With the command below, you can see that demo-frontend pods are provisioned at NOMINATED NODE.

kubectl get pod -o wide

Or, you can check the list of Fargate worker nodes by following command.

kubectl get nodes -l eks.amazonaws.com/compute-type=fargate

You can also paste the results of the command below into the web browser to see the same screen as before.

echo http://$(kubectl get ingress/frontend-ingress -o jsonpath='{.status.loadBalancer.ingress[*].hostname}')

Explore Container Insights

Amazon CloudWatch Container Insight

Use CloudWatch Container Insights to collect, aggregate, and summarize metrics and logs from your containerized applications and microservices. Container Insights is available for Amazon Elastic Container Service (Amazon ECS), Amazon Elastic Kubernetes Service (Amazon EKS), and Kubernetes platforms on Amazon EC2. Amazon ECS support includes support for Fargate.

CloudWatch automatically collects metrics for many resources, such as CPU, memory, disk, and network. Container Insights also provides diagnostic information, such as container restart failures, to help you isolate issues and resolve them quickly. You can also set CloudWatch alarms on metrics that Container Insights collects.

Explorer EKS CloudWatch Container Insights

In this lab, you will use *Fluent Bit *to route logs. The lab order will install CloudWatch Agent **to collect metric of the cluster and **Fluent Bit to send logs to CloudWatch Logs in DaemonSet type.

First, create a folder to manage manifest files.

cd ~/environment
mkdir -p manifests/cloudwatch-insight && cd manifests/cloudwatch-insight

Install CloudWatch agent, Fluent Bit

In Create EKS Cluster with eksctl, cloudWatch related permissions were placed in the worker node.

Create namespace named amazon-cloudwatch by following command.

kubectl create ns amazon-cloudwatch

[!] If namespace created successfully, the namespace will exist in the list using the command below.

kubectl get ns

After specifying some settings values, install CloudWatch agent and Fluent Bit. Copy and paste one line at a time.

ClusterName=eks-demo
RegionName=$AWS_REGION
FluentBitHttpPort='2020'
FluentBitReadFromHead='Off'
[[ ${FluentBitReadFromHead} = 'On' ]] && FluentBitReadFromTail='Off'|| FluentBitReadFromTail='On'
[[ -z ${FluentBitHttpPort} ]] && FluentBitHttpServer='Off' || FluentBitHttpServer='On'

For RegionName, make sure it contains AWS Region code that you are currently working on. For instance, ap-northeast-2 if you are working on in Seoul Region.

Then, through the command below, download yaml file.

wget https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/quickstart/cwagent-fluent-bit-quickstart.yaml

And then, apply environment variables into this yaml file.

sed -i 's/{{cluster_name}}/'${ClusterName}'/;s/{{region_name}}/'${RegionName}'/;s/{{http_server_toggle}}/"'${FluentBitHttpServer}'"/;s/{{http_server_port}}/"'${FluentBitHttpPort}'"/;s/{{read_from_head}}/"'${FluentBitReadFromHead}'"/;s/{{read_from_tail}}/"'${FluentBitReadFromTail}'"/' cwagent-fluent-bit-quickstart.yaml

Open this yaml file, find DaemonSet object which name is fluent-bit and add the values below the spec.

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: eks.amazonaws.com/compute-type
          operator: NotIn
          values:
          - fargate

The results of extracting some of the pasted ones are as follows. Take care indentation.

Deploy yaml file.

kubectl apply -f cwagent-fluent-bit-quickstart.yaml

Use the command below to check that the installation was succeeded. As a result, three cloudwatch-agent pods and three fluid-bit pods are available.

kubectl get po -n amazon-cloudwatch

You can also check it through the command below. You can see that two Daemonsets are output.

kubectl get daemonsets -n amazon-cloudwatch

Dive into the CloudWatch console

When you click Map View in the upper right corner, the cluster’s resources are displayed in tree form. You can also click on a particular object to see the associated metric values as shown below.

Then select Performance monitoring in the above select bar. And if you click EKS Services at the top, you can see the metric values in terms of service as shown below.

Also, select a specific pod from Pod performance section, then click View performance logs in the dropbox to the right.

You will be redirected to the CloudWatch Logs Insights page as shown below. The query allows you to view the logs you want.

Autoscaling Pod & Cluster

Kubernetes Auto Scaling

Auto scaling service means that the ability to automatically create or delete servers based on user-defined cycles and events. Auto scaling enables applications to respond flexibly to traffic.

Kubernetis has two main auto-scaling capabilities.

HPA(Horizontal Pod AutoScaler)
Cluster Autoscaler

HPA automatically scales the number of pods by observing CPU usage or custom metrics. However, if you run out of EKS cluster’s own resources to which the pod goes up, consider Cluster Autoscaler.

Applying these auto-scaling capabilities to a cluster allows you to configure a more resilient and scalable environment.

Apply HPA

Applying Pod Scaling with HPA

The HPA(Horizontal Pod Autoscaler) controller allocates the number of pods based on metric. To apply pod scaling, you must specify the amount of resources required for the container and create conditions to scale through HPA.

Create metrics server. Metrics Server aggregates resource usage data across the Kubernetes cluster. Collect metrics such as the CPU and memory usage of the worker node or container through kubelet installed on each worker node.

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Use the command below to check that the metrics server is created successfully.

kubectl get deployment metrics-server -n kube-system

And then, modify flask deployment yaml file that you created in Deploy First Backend Service. Change replicas to 1 and set the amount of resources required for the container.

cd /home/ec2-user/environment/manifests cat < flask-deployment.yaml --- apiVersion: apps/v1 kind: Deployment metadata: name: demo-flask-backend namespace: default spec: replicas: 1 selector: matchLabels: app: demo-flask-backend template: metadata: labels: app: demo-flask-backend spec: containers: - name: demo-flask-backend image: $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/demo-flask-backend:latest imagePullPolicy: Always ports: - containerPort: 8080 resources: requests: cpu: 250m limits: cpu: 500m EOF

1vCPU = 1000m(milicore)

Apply the yaml file to reflect the changes.

kubectl apply -f flask-deployment.yaml

To set up HPA, create the yaml file below as well.

cat < flask-hpa.yaml --- apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler metadata: name: demo-flask-backend-hpa namespace: default spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: demo-flask-backend minReplicas: 1 maxReplicas: 5 targetCPUUtilizationPercentage: 30 EOF

Deploy yaml file.

kubectl apply -f flask-hpa.yaml

You can set to this with simple kubectl command.

kubectl autoscale deployment demo-flask-backend --cpu-percent=30 --min=1 --max=5

After creating HPA, you can check the HPA status with the command below. If the target shows CPU usage as unknown, wait a moment and check.

kubectl get hpa

Perform a simple load test to check that the autoscaling functionality is working properly. First, enter the command below to understand the amount of change in the pod.

kubectl get hpa -w

In addition, create additional terminals for load testing in AWS Cloud9. HTTP load testing through siege tool.

sudo yum -y install siege

export flask_api=$(kubectl get ingress/flask-backend-ingress -o jsonpath='{.status.loadBalancer.ingress[*].hostname}')/contents/aws

siege -c 200 -i [http://$flask_api](http://$flask_api)

As shown in the screen below, you can load one side terminal and observe the amount of change in the pod accordingly on the other side. You can see that the REPLICAS value changes by up to 5 depending on the load.

Apply Cluster Autoscaler

Applying Cluster Scaling with Cluster Autoscaler

Auto scaling was applied to the pod on the previous chapter. However, depending on the traffic, there may be insufficient Worker Node resources for the pod to increase. In other words, it’s full of Worker Nodes’ capacity and no more pod can’t be scheduled. At this point, what we use is Cluster Autoscaler(CA).

Cluster Autoscaler(CA) scales out the worker node if a pod in the pending state exists. Perform scale-in/out by checking utilization at intervals of a particular time. AWS also uses Auto Scaling Group to apply Cluster Autoscaler.

! To visualize the status of the current cluster, see kube-ops-view.

Use the command below to check the value of ASG(Auto Scaling Group) applied to the current cluster’s worker nodes.

aws autoscaling \ describe-auto-scaling-groups \ --query "AutoScalingGroups[? Tags[? (Key=='eks:cluster-name') && Value=='eks-demo']].[AutoScalingGroupName, MinSize, MaxSize,DesiredCapacity]" \ --output table

In this lab, when deploying an EKS cluster, we performed the task of attaching IAM policies related to autoscaling. However, if you have not done so, click the hidden folder below to create the relevant IAM policy and attach it to the IAM role.

Creating an Auto Scaling IAM policy and attaching it to a worker nodes’ IAM Role

In Auto Scaling Groups page, click ASG applied in worker node, and update Group details value same as below.
Return to your Cloud9 environment and download the deployment example file provided by the Cluster Atuoscaler project.

cd /home/ec2-user/environment/manifests wget https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml

Open the downloaded yaml file, set the cluster name from to eks-demo and deploy it.

... command: - ./cluster-autoscaler - --v=4 - --stderrthreshold=info - --cloud-provider=aws - --skip-nodes-with-local-storage=false - --expander=least-waste - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/eks-demo ...

kubectl apply -f cluster-autoscaler-autodiscover.yaml

Perform a simple load test to check that the autoscaling functionality is working properly. First, enter the command below to understand the change in the number of worker nodes.

kubectl get nodes -w

Then turn on the new terminal, and then perform a command to deploy 100 pods to increase the worker node.

kubectl create deployment autoscaler-demo --image=nginx
kubectl scale deployment autoscaler-demo --replicas=100

To check the progress of the pod’s deployment, perform the following command.

kubectl get deployment autoscaler-demo --watch

If kube-ops-view is installed, you can visually see the results below. This shows that two additional worker nodes were created and 100 pods were deployed.
If you delete a previously created pods with the command below, you can see that the worker node will be scaled in.

kubectl delete deployment autoscaler-demo

Install Kubernetes Operational View

Kubernetes Operational View is a simple web page that provides a visual view of the health of multiple Kubernetes clusters. Although not used for monitoring and operations management purposes, you can visually observe the process of scale-in/out during cluster autoscaling operations, such as Cluster Autoscaler.

In this lab, we deploy kube-ops-view via *Helm *. Helm is a tool for managing Kubernetes charts, which means a preconfigured Kubernetes resource package. The purpose of managing charts with Helm is to manage various manifest files easily.

Install Helm

Before configuring Helm, install the helm cli tool.

curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash

Check the current version through the command below.

helm version --short

Use the command below to add a stable repository.

helm repo add stable https://charts.helm.sh/stable

helm repo add k8s-at-home https://k8s-at-home.com/charts/

helm repo update
(Optional) Configure Bash completion for the helm command.

helm completion bash >> ~/.bash_completion
. /etc/profile.d/bash_completion.sh
. ~/.bash_completion
source <(helm completion bash)

Install kube-ops-view

Install kube-ops-view through helm.

helm install kube-ops-view k8s-at-home/kube-ops-view
Check the chart was installed successfully.

helm list
Get the application URL by running these commands.

export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=kube-ops-view,app.kubernetes.io/instance=kube-ops-view" -o jsonpath="{.items[0].metadata.name}")

echo "Visit http://127.0.0.1:8080 to use your application"

kubectl port-forward $POD_NAME 8080:8080
In case of Cloud9, click the preview > preview running application which placed in the upper end. You can observe screen like below.

CI/CD for EKS cluster

CI/CD pipeline for EKS Cluster / Kubernetes Cluster

Shape of CI/CD pipeline for either EKS or Kubernetes varies. This is because; 1/many different kinds of tools for CI/CD are out there, 2/each of dev/ops team’s culture that embraces and uses those tools is also diverse.

Given that situation, this tutorial amis to introduce the shape of CI/CD pipeline that is

easy and speedy to implement
to minimize manual task rather than automation

On top of that, the goal CI/CD pipeline in this tutorial will automatically detect application code changes in Github, and then trigger Github Action to integrate and buid the code changes. At the end, ArgoCD will be subsequently executed to deploy built artifacts to the target, EKS cluster. For pieces of block helping to automate this flow, we will introduce Kustomize that is tool to package kubernetes manifest up, Checkov and Tryvy for static analysis to secure EKS cluster running on.

GitHub
GitHub Actions
Kustomize
ArgoCD
Checkov
Trivy

The goal of CI/CD pipeline will like below, which is also called as gitops flow.

CI/CD pipeline for EKS Cluster / Kubernetes Cluster with cdk helm

Given that situation, this tutorial amis to introduce the shape of CI/CD pipeline that is

easy and speedy to implement
to minimize manual task rather than automation

On top of that, the goal CI/CD pipeline in this tutorial will automatically detect application code changes in CodeCommit, and then trigger Codebuild to integrate and buid the code changes. At the end, ArgoCD will be subsequently executed to deploy built artifacts to the target, EKS cluster. For pieces of block helping to automate this flow, we will introduce Helm that is tool to package kubernetes manifest up, Checkov and Tryvy for static analysis to secure EKS cluster running on.

CodeCommit
CodeBuild
CodePipeline
Helm
ArgoCD
Checkov
Trivy

The goal of CI/CD pipeline will like below, which is also called as gitops flow.

Create CI/CD pipeline

Build up CI/CD pipeline

Target CI/CD pipeline looks like this.

1. Create two git repository for application, kubernetes manifest each

We need to have two github repository in place.

front-app-repo: located front-end application source code in
k8s-manifest-repo: located kubernetes manifest files in

(1) Create front-app-repo

(2) initialize local code directory

You should change “your-github-username” to your own git user name.

cd ~/environment/amazon-eks-frontend
rm -rf .git
export GITHUB_USERNAME=your-github-username

(3) Configure git remote repository locally

Push front-end source code to front-app-repo you just created.

cd ~/environment/amazon-eks-frontend
git init
git add .
git commit -m "first commit"
git branch -M main
git remote add origin https://github.com/$GITHUB_USERNAME/front-app-repo.git
git push -u origin main

Confirm that it is all set.

If you feel reluctant to submit username and password every single time you login, you can set cache to avoid it as below.

git config --global user.name USERNAME
git config --global user.email EMAIL
git config credential.helper store
git config --global credential.helper 'cache --timeout TIME YOU WANT'

If you are already using github MFA authentication, you are asked to use personal access token for password. To generate personal access token, you can follow this github guidance on it. Once you get the token from there, you can use the token when asked to submit password.

2. Prepare least privilege IAM to use in CI/CD pipeline

gitHub Action plays main role to build front-end application, build its docker container image and push it to ECR at the end. So, to make gitHub Action access ECR securly, we are strongly recommended to use seperate IAM with least privilege, and which will limit gitHub Action to access to ECR only.

(1) Create IAM user

aws iam create-user --user-name github-action

(2) Create ECR policy

Make json file containg policy to ECR

cd ~/environment
cat <<EOF> ecr-policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowPush",
            "Effect": "Allow",
            "Action": [
                "ecr:GetDownloadUrlForLayer",
                "ecr:BatchGetImage",
                "ecr:BatchCheckLayerAvailability",
                "ecr:PutImage",
                "ecr:InitiateLayerUpload",
                "ecr:UploadLayerPart",
                "ecr:CompleteLayerUpload"
            ],
            "Resource": "arn:aws:ecr:${AWS_REGION}:${ACCOUNT_ID}:repository/demo-frontend"
        },
        {
            "Sid": "GetAuthorizationToken",
            "Effect": "Allow",
            "Action": [
                "ecr:GetAuthorizationToken"
            ],
            "Resource": "*"
        }
    ]
}
EOF

Using the file you made, create IAM policy. Policy name is ecr-policy recommended.

aws iam create-policy --policy-name ecr-policy --policy-document file://ecr-policy.json

(3) Attach ECR policy to IAM user

Attach ecr-policy to IAM user you create previously.

aws iam attach-user-policy --user-name github-action --policy-arn arn:aws:iam::${ACCOUNT_ID}:policy/ecr-policy

3. Create githup secrets(AWS Credential, githup token)

gitHub Action we make will store and use AWS credential, github token in githup secrets. This way, we can secure those secrets without being exposed unintentinally.

(1) Generate AWS Credential

When gitHub Action push the docker image of front-end application, it uses AWS credential. For this working, we have created github-action, IAM user with least privilege. Now, create Access Key, Secret Key for the IAM User.

aws iam create-access-key --user-name github-action

Make a note of values of "SecretAccessKey", "AccessKeyId" out of output. Those will be used going forward in this tutorial.

{
  "AccessKey": {
    "UserName": "github-action",
    "Status": "Active",
    "CreateDate": "2021-07-29T08:41:04Z",
    "SecretAccessKey": "***",
    "AccessKeyId": "***"
  }
}

(2) Generate gitHub personal token

Once log in github.com, naviate profile > Settings > Developer settings > Personal access tokens. Finally, click on Generate new token in the top right corner.

Type access token for github action in Note, and then select repo in Select scopes. Finally, click Generate token

Copy value of token in the output.

(3) Set up gitHub secret

Go back to front-app-repo repository and navigate Settings > Secrets. And click New repository secret in the top right corner.

As below screen shot, put ACTION_TOKEN, personal access token in Name, Value respectively.(You must have copied personal access token in the previous step). Finally click **Add secret*

Similar way, store both AccessKeyId and SecretAccessKey that github-action will use in gitHub secret. Note that Name of AccessKeyId and SecretAccessKey must be AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY each.

4. Make build script for gitHub Action to use

(1) Make .github directory

cd ~/environment/amazon-eks-frontend
mkdir -p ./.github/workflows

(2) Make build.yaml for gitHub Action to use

gitHub Action will be running based on tasks that build.yaml contains. So we need to declare tasks that gitHub Action execute for us in build.yaml. build.yamlwill include checkout, build front-end application, make docker image, and push it to ECR.

Most eye-catching part in the script is procedure to dynamically put docker image tag. We intend to have **$IMAGE_TAG **that is dynamically and randomly created to attach docker image built.

cd ~/environment/amazon-eks-frontend/.github/workflows
cat > build.yaml <<EOF
name: Build Front
on:
  push:
    branches: [ main ]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout source code
        uses: actions/checkout@v2
      - name: Check Node v
        run: node -v
      - name: Build front
        run: |
          npm install
          npm run build
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: \${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: \${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: $AWS_REGION
      - name: Login to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v1
      - name: Get image tag(verion)
        id: image
        run: |
          VERSION=\$(echo \${{ github.sha }} | cut -c1-8)
          echo VERSION=\$VERSION
          echo "::set-output name=version::\$VERSION"
      - name: Build, tag, and push image to Amazon ECR
        id: image-info
        env:
          ECR_REGISTRY: \${{ steps.login-ecr.outputs.registry }}
          ECR_REPOSITORY: demo-frontend
          IMAGE_TAG: \${{ steps.image.outputs.version }}
        run: |
          echo "::set-output name=ecr_repository::\$ECR_REPOSITORY"
          echo "::set-output name=image_tag::\$IMAGE_TAG"
          docker build -t \$ECR_REGISTRY/\$ECR_REPOSITORY:\$IMAGE_TAG .
          docker push \$ECR_REGISTRY/\$ECR_REPOSITORY:\$IMAGE_TAG
EOF

(3) Run gitHub Action workflow

Push code to front-app-repo to trigger gitHub Action workflow. gitHub Action workflow will automatically be triggered on push of code. gitHub Action workflow will run step-by-step based on build.yaml we’ve declared.

cd ~/environment/amazon-eks-frontend
git add .
git commit -m "Add github action build script"
git push origin main

Return to gitHub page and comfirm push is done. Also comfirm that gitHub Action workflow works exepectedly as below screen shots.

If you confirmed the workflow finished successfully, go to ECR repository, demo-frontend to see if new docker imgage is pushed with new $IMAGE_TAG.

Check if the image tag contains part of sha value.

5. Kustomize Overview

In this tutorial, we are going to use Kustomize to simply inject same value of label, metadata, etc. into kubernetes Deployment objects. This helps us to avoid hassle job to modify mannually value in each of kubernetes objects. Most importantly, we are to use Kustomize to not only automatically, but also dynamically assign image tag to kubernetes Deployment.

Please discover more details on Kustomize here, Kustomize official document

6. Structure directories for Kustomize

(1) Make directories

Now that kubernetes manifest owns seperated github repository. On top of that, we are going to package it to deploy using Kustomize. For this we need to make directories that Kustomize can run accordingly. The struction of directories should follow predefined naming rule.

cd ~/environment
mkdir -p ./k8s-manifest-repo/base
mkdir -p ./k8s-manifest-repo/overlays/dev
cd ~/environment/manifests
cp *.yaml ../k8s-manifest-repo/base
cd ../k8s-manifest-repo/base
ls -rlt

Outcome of directories has base and overlays/dev under k8s-manifest-repo.

base : raw kubernetes manifest files are here. During kustomize build process, those files in here will be automatically modified along with customized content by users in kustomize.yaml under overlays.
overlays : customized content by users is in kustomize.yaml under this directory. Also note that dev directory is to put all relevant files for deploying to dev environment. In this tutorial, we assume that we deploy to dev environment accordingly.

(2) Make Kustomize manifest files

Remember the goal of this tutorial is to make deployment pipeline for front-end application. So, we will change and replace some values in frontend-deployment.yaml and frontend-service.yaml with values we intend to inject during deployment step(e.g. image tag). These are values we are definitely to inject dynamically into associated kubernetes manifest files.

metadata.labels*:* "env: dev" will be reflected to frontend-deployment.yaml, frontend-service.yaml
spec.selector : "select.app: frontend-fargate" will be reflected to frontend-deployment.yaml, frontend-service.yaml
spec.template.spec.containers.image : "image: " with newly created image tag will be reflected to frontend-deployment.yaml

Make kustomize.yamlas below. Main purpose of this file is to define target files to be automatically injected by kustomize.

cd ~/environment/k8s-manifest-repo/base
cat <<EOF> kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - frontend-deployment.yaml
  - frontend-service.yaml
EOF

Next, it’s time to make files that contain what to inject into target files that kustomize.yamldefines in the previous step. First, make a file to inject into frontend-deployment.yaml.

cd ~/environment/k8s-manifest-repo/overlays/dev
cat <<EOF> front-deployment-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-frontend
  namespace: default
  labels:
    env: dev
spec:
  selector:
    matchLabels:
      app: frontend-fargate
  template:
    metadata:
      labels:
        app: frontend-fargate
EOF

Also, make make a file to inject into frontend-service.yaml.

cd ~/environment/k8s-manifest-repo/overlays/dev
cat <<EOF> front-service-patch.yaml
apiVersion: v1
kind: Service
metadata:
  name: demo-frontend
  annotations:
    alb.ingress.kubernetes.io/healthcheck-path: "/"
  labels:
    env: dev
spec:
  selector:
    app: frontend-fargate
EOF

Lastly, improve kustomization.yaml to replace the name of image kubernetes deployment object refers to. The name of image will have to be with image tag that is randomly generated during build process of front-end application in gitHub Action workflow.

To be specific, value assigned to name will be replaced with value of a combination of newNameand newTag

Run this code.

cd ~/environment/k8s-manifest-repo/overlays/dev
cat <<EOF> kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
images:
- name: ${ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/demo-frontend
  newName: ${ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/demo-frontend
  newTag: abcdefg
resources:
- ../../base
patchesStrategicMerge:
- front-deployment-patch.yaml
- front-service-patch.yaml
EOF

As a result, content in XXXX-patch.yaml and value of images in kustomization.yaml will be automatically applied to kubernetes manifest on deployment to EKS cluster.

7. Create gitHub repository for kubernetes manifest

(1) gitHub repo for kubernetes manifest

Create respository in gitHub with name as k8s-manifest-repo. This will persist kubernetes manifest files we’ve created so far.

Push kubernetes manifest files to k8s-manifest-repo

cd ~/environment/k8s-manifest-repo/
git init
git add .
git commit -m "first commit"
git branch -M main
git remote add origin https://github.com/$GITHUB_USERNAME/k8s-manifest-repo.git
git push -u origin main

8. Set up ArgoCD

(1) Install ArgoCD in EKS cluster

Run this code to install ArgoDC in EKS cluster.

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

ArgoCD also provides CLI to users. Install ArgoCD CLI, we are not using it in this tutorial going on though.

cd ~/environment
VERSION=$(curl --silent "https://api.github.com/repos/argoproj/argo-cd/releases/latest" | grep '"tag_name"' | sed -E 's/.*"([^"]+)".*/\1/')

sudo curl --silent --location -o /usr/local/bin/argocd [https://github.com/argoproj/argo-cd/releases/download/$VERSION/argocd-linux-amd64](https://github.com/argoproj/argo-cd/releases/download/$VERSION/argocd-linux-amd64)

sudo chmod +x /usr/local/bin/argocd

Basically ArgoCD is not directly exposed to external, so we need to set up ELB in front of ArgoCD for incoming transactions.=

kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'

It may take 3~4 mins to be reachable via ELB. Run this code to get the URL of ELB.

export ARGOCD_SERVER=`kubectl get svc argocd-server -n argocd -o json | jq --raw-output .status.loadBalancer.ingress[0].hostname`
echo $ARGOCD_SERVER

ArgoCD default username is admin. Get password against it with this command.

ARGO_PWD=`kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d`
echo $ARGO_PWD

Open $ARGOCD_SERVER in your local web browser with Username = adminand Password = $ARGO_PWD.

9. Configure ArgoCD

(1) Configure ArgoCD

After logging in, Click Applicaions to configure in the top left corner.

Next, input basic information about target deployment of application. Application Name and Project should beeksworkshop-cd-pipeline and default each.

Repository URL , Revision, Path in section of SOURCE must be **git address of **k8s-manifest-repo, main, and overlays/dev each.

Cluster URL and Namespace in section of DESTINATION must be https://kubernetes.default.svc and default each. After input, click Create

Appropriate result will be named eksworkshop-cd-pipeline as below.

10. Add Kustomize build step

(1) Improve gitHub Action build script Add this code in build.yamlfor front-app-repo. This code will update container image tag in kubernetes manifest files using kustomize. Afte that, it will commit and push those files to k8s-manifest-repo.

When it successfully finishes, ArgoCD watching k8s-manifest-repo will catch the new update and start deployment process afterward.

cd ~/environment/amazon-eks-frontend/.github/workflows
cat <<EOF>> build.yaml
      - name: Setup Kustomize
        uses: imranismail/setup-kustomize@v1
      - name: Checkout kustomize repository
        uses: actions/checkout@v2
        with:
          repository: $GITHUB_USERNAME/k8s-manifest-repo
          ref: main
          token: \${{ secrets.ACTION_TOKEN }}
          path: k8s-manifest-repo
      - name: Update Kubernetes resources
        run: |
          echo \${{ steps.login-ecr.outputs.registry }}
          echo \${{ steps.image-info.outputs.ecr_repository }}
          echo \${{ steps.image-info.outputs.image_tag }}
          cd k8s-manifest-repo/overlays/dev/
          kustomize edit set image \${{ steps.login-ecr.outputs.registry}}/\${{ steps.image-info.outputs.ecr_repository }}=\${{ steps.login-ecr.outputs.registry}}/\${{ steps.image-info.outputs.ecr_repository }}:\${{ steps.image-info.outputs.image_tag }}
          cat kustomization.yaml
      - name: Commit files
        run: |
          cd k8s-manifest-repo
          git config --global user.email "github-actions@github.com"
          git config --global user.name "github-actions"
          git commit -am "Update image tag"
          git push -u origin main
EOF

(2) Commit&push to front-app-repo

Commit and push newly improved build.yamlto front-app-repo to run gitHub Action workflow.

cd ~/environment/amazon-eks-frontend
git add .
git commit -m "Add kustomize image edit"
git push -u origin main

(3) Check github action

Check if gitHub Action workflow works fine.

(4) Check k8s-manifest-repo

Check if k8s-manifest-repo’s latest commit is derived from gitHub Action workflow of *front-app-repo*.

(5) Check ArgoCD

Return to ArgoCD UI. Navigate Applications > eksworkshop-cd-pipeline. Now CURRENT SYNC STATUS is Out of Synced.

To run sync job automatically, we need to enable Auto-Sync. To do so, go to APP DETAILS and click ENABLE AUTO-SYNC.

As a result, k8s-manifest-repo’s commit will be deployed to EKS Cluster.

To confirm that new image tag is deployed, check out k8s-manifest-repo commit history to get image tag information. And then, compare it with image tag that frontend-deploymentin ArgoCD used.

{{% notice info %}} You can see detail information when you click pod that starts with demo-frontend-. To get there, you might need to first navigate Applications > eksworkshop-cd-pipeline >. {{% /notice %}}

From now on, on commit in k8s-manifest-repo, ArgoCD automatically deploy the commit to EKS Cluster.

11. Check CI/CD pipeline working from end to end

Let’s test out whole gitops pipeline we’ve built by making code changes in front-end application.

(1) Change code

Go to Cloud9 first, and then move to amazon-eks-frontend/src/ and open App.js in folder tree of the left pane.

Replace code at line 67 with EKS DEMO Blog version 1 and save it.

  return (
    <div className={classes.root}>
      <AppBar position="static" style={{ background: '#2E3B55' }}>
        <Toolbar>
          <IconButton edge="start" className={classes.menuButton} color="inherit" aria-label="menu">
            <CloudIcon />
          </IconButton>
          <Typography
            variant="h6"
            align="center"
            className={classes.title}
          >
            EKS DEMO Blog version 1
          </Typography>
          {new Date().toLocaleTimeString()}
        </Toolbar>
      </AppBar>
      <br/>

(2) Commit and push

Commit and push changed code to git repository.

cd ~/environment/amazon-eks-frontend
git add .
git commit -m "Add new blog version"
git push -u origin main

(3) Check CI/CD pipeline and application

Wait until ArgoCD sync job completes as below.

When it is all set, hit the application via URL from the below command.

echo http://$(kubectl get ingress/frontend-ingress -o jsonpath='{.status.loadBalancer.ingress[*].hostname}')

It must show new code change as below.

CI/CD with security

Improve CI/CD pipeline with security implementation

Prior to deploying kubernetes manifest files to EKS Cluster, supplementary steps need to be added to prevent security and misconfiguration issue by using both *Checkov *and Trivy . Also, we will use seperate ArgoCD account from admin user that we’ve used in the previous lab. This will follow ArgoCD RBAC rule to secure ArgoCD and EKS cluster ultimately.

For this, we will need to improve CD (Continuous Deploy) process as follows.

On application code change, new docker image with new image tag is created
Trivy inspects security vulnerability of the new image
Kustomize starts making kubernetes manifest files with the new image information
Checkov inspects security vulnerability and misconfiguration of kubernetes manifest files
If no issue out there, ArgoCD starts sync job to deploy

Each of steps above is ran in the different gitHub Action workflow

1~2 : github Action workflow of application repository
3~5 : github Action workflow of k8s manifest repository

We will conduct followings to build it up.

Improve gitHub Action build script in frontend application repository
Improve gitHub Action build script in k8s manifest repository
Deactivate ArgoCD AUTO_SYNC (Manual)
Create new ArgoCD account
Create auth-token for new ArgoCD account
Configure Argo RBAC for new ArgoCD account

1. Improve gitHub Action build script in frontend application repository

We need additional step to ensure that newly created docker image has no security vulnerability prior to pushing it to ECR. For this we will modify build.yaml to add image scanning process using **Trivy**

Run this code to change build.yaml for frontend application repo.

cd ~/environment/amazon-eks-frontend/.github/workflows
cat <<EOF> build.yaml
name: Build Front

on:
  push:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout source code
        uses: actions/checkout@v2

      - name: Check Node v
        run: node -v

      - name: Build front
        run: |
          npm install
          npm run build

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: \${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: \${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: $AWS_REGION

      - name: Login to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v1

      - name: Get image tag(verion)
        id: image
        run: |
          VERSION=\$(echo \${{ github.sha }} | cut -c1-8)
          echo VERSION=\$VERSION
          echo "::set-output name=version::\$VERSION"

      - name: Build, tag, and push image to Amazon ECR
        id: image-info
        env:
          ECR_REGISTRY: \${{ steps.login-ecr.outputs.registry }}
          ECR_REPOSITORY: demo-frontend
          IMAGE_TAG: \${{ steps.image.outputs.version }}
        run: |
          echo "::set-output name=ecr_repository::\$ECR_REPOSITORY"
          echo "::set-output name=image_tag::\$IMAGE_TAG"
          docker build -t \$ECR_REGISTRY/\$ECR_REPOSITORY:\$IMAGE_TAG .

      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: '\${{ steps.login-ecr.outputs.registry}}/\${{ steps.image-info.outputs.ecr_repository }}:\${{ steps.image-info.outputs.image_tag }}'
          format: 'table'
          exit-code: '0'
          ignore-unfixed: true
          vuln-type: 'os,library'
          severity: 'CRITICAL,HIGH'

      - name: Push image to Amazon ECR
        run: |
          docker push \${{ steps.login-ecr.outputs.registry}}/\${{ steps.image-info.outputs.ecr_repository }}:\${{ steps.image-info.outputs.image_tag }}

      - name: Setup Kustomize
        uses: imranismail/setup-kustomize@v1

      - name: Checkout kustomize repository
        uses: actions/checkout@v2
        with:
          repository: $GITHUB_USERNAME/k8s-manifest-repo
          ref: main
          token: \${{ secrets.ACTION_TOKEN }}
          path: k8s-manifest-repo

      - name: Update Kubernetes resources
        run: |
          echo \${{ steps.login-ecr.outputs.registry }}
          echo \${{ steps.image-info.outputs.ecr_repository }}
          echo \${{ steps.image-info.outputs.image_tag }}
          cd k8s-manifest-repo/overlays/dev/
          kustomize edit set image \${{ steps.login-ecr.outputs.registry}}/\${{ steps.image-info.outputs.ecr_repository }}=\${{ steps.login-ecr.outputs.registry}}/\${{ steps.image-info.outputs.ecr_repository }}:\${{ steps.image-info.outputs.image_tag }}
          cat kustomization.yaml

      - name: Commit files
        run: |
          cd k8s-manifest-repo
          git config --global user.email "github-actions@github.com"
          git config --global user.name "github-actions"
          git commit -am "Update image tag"
          git push -u origin main

EOF

Commit&push

git add .
git commit -m "Add Image Scanning in build.yaml"
git push -u origin main

Afterwards, gitHub Action workflow will be executed and it shows the result of image scan.

We intended to have gitHub Action workflow move forward and complete even though image scan step fails. In real world, you might want to stop the workflow when image scan fails. For this you can set exit-code: '1'instead of exit-code: '0'in part of trivy step in build.yaml. For more details on this, please refer to Trivy doc .

2. Improve gitHub Action build script in k8s manifest repository

cd ~/environment/k8s-manifest-repo
mkdir -p ./.github/workflows
cd ~/environment/k8s-manifest-repo/.github/workflows
cat <<EOF> build.yaml
name: "ArgoCD sync"
on: "push"

jobs:
  build:
    runs-on: ubuntu-latest
    steps:

      - name: Checkout source code
        uses: actions/checkout@v2

      - name: Setup Kustomize
        uses: imranismail/setup-kustomize@v1

      - name: Build Kustomize
        run: |
          pwd
          mkdir kustomize-build
          kustomize build ./overlays/dev > ./kustomize-build/kustomize-build-output.yaml
          ls -rlt
          cd kustomize-build
          cat kustomize-build-output.yaml

      - name: Run Checkov action
        id: checkov
        uses: bridgecrewio/checkov-action@master
        with:
          directory: kustomize-build/
          framework: kubernetes

      - name: Install ArgoCD and execute Sync in ArgoCD
        run: |
          curl -sSL -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
          chmod +x /usr/local/bin/argocd
          ARGO_SERVER=$ARGOCD_SERVER
          argocd app sync eksworkshop-cd-pipeline --auth-token \${{ secrets.ARGOCD_TOKEN }} --server $ARGOCD_SERVER --insecure

EOF

3. Deactivate ArgoCD AUTO_SYNC *(*Manual)

Go to Applicaiton > eksworkshop-cd-pipeline and then click APP DETAIL. Next, change SYNC_POLICY with DISABLE AUTO-SYNC

4. Create new ArgoCD account

To increase security of ArgoCD, we will use seperate ArgoCD account from admin user we used. Also we add role on top of the account.

New account name of ArgoCD for CI/CD pipeline is devops

ArgoCD allows us to add account via Configmap tha ArgoCD is using in the cluster.

Run this code.

kubectl -n argocd edit configmap argocd-cm -o yaml

Next, add this code.

data:
  accounts.devops: apiKey,login

Final code must be like this. (* createTimestamp, resourceVersion, etc differ depending on users environment)

apiVersion: v1
data:
  accounts.devops: apiKey,login
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"argocd-cm","app.kubernetes.io/part-of":"argocd"},"name":"argocd-cm","namespace":"argocd"}}
  creationTimestamp: "2021-07-28T07:45:53Z"
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
  name: argocd-cm
  namespace: argocd
  resourceVersion: "153620981"
  selfLink: /api/v1/namespaces/argocd/configmaps/argocd-cm
  uid: a8bb80e7-577c-4f10-b3de-359e83ccee20

Finally, type :wq! to save and exit.

5. Create *auth-token *for new ArgoCD account

Let’s generate auth-token that new ArgoCD account, devops is using. This will be used for authentication token when we make api call to ArgoCD. So this is different credential from login password for ArgoCD UI.

Run this code and make a note of the output so that we can continue to use it

argocd account generate-token --account devops

Failed to establish connection

If you are encountered “Failed to establish connection”,you should log in ArgoCD with following command.

argocd login $ARGOCD_SERVER

To save token value from the output in Secrets of kubernetes maniffest repository, go to Settings > Secrets, and then click New repository secret. Finally, input ARGOCD_TOKEN and saved token value into Name and Values each, and then click Add Secret

6. Configure Argo RBAC for new ArgoCD account

The new ArgoCD account we’ve created has no permission to make API call to sync. So, we need to grant it permission according to RBAC of ArgoCD.

To grant permission, run this code to modify argocd-rbac-cm, ArgoCD Configmap.

kubectl -n argocd edit configmap argocd-rbac-cm -o yaml

Add this content. For lab purpose only, we intend to allow many of permissions. So please be mindful when you do this in production environment.

data:
  policy.csv: |
    p, role:devops, applications, *, */*, allow
    p, role:devops, clusters, get, *, allow
    p, role:devops, repositories, get, *, allow
    p, role:devops, repositories, create, *, allow
    p, role:devops, repositories, update, *, allow
    p, role:devops, repositories, delete, *, allow

    g, devops, role:devops
  policy.default: role:readonly

Final content must be like this after adding the content. (* createTimestamp, resourceVersion, etc differ depending on users environment)

apiVersion: v1
data:
  policy.csv: |
    p, role:devops, applications, *, */*, allow
    p, role:devops, clusters, get, *, allow
    p, role:devops, repositories, get, *, allow
    p, role:devops, repositories, create, *, allow
    p, role:devops, repositories, update, *, allow
    p, role:devops, repositories, delete, *, allow

    g, devops, role:devops
  policy.default: role:readonly
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"argocd-rbac-cm","app.kubernetes.io/part-of":"argocd"},"name":"argocd-rbac-cm","namespace":"argocd"}}
  creationTimestamp: "2021-07-28T07:45:53Z"
  labels:
    app.kubernetes.io/name: argocd-rbac-cm
    app.kubernetes.io/part-of: argocd
  name: argocd-rbac-cm
  namespace: argocd
  resourceVersion: "153629591"
  selfLink: /api/v1/namespaces/argocd/configmaps/argocd-rbac-cm
  uid: 1fe0d735-f3a0-4867-9357-7a9e766fef22

7. Check new implementation working

Commit and push the code

cd ~/environment/k8s-manifest-repo
git add .
git commit -m "Add github action with ArgoCD"
git push -u origin main

See if gitHub Action workflow completes and trigger ArgoCD deployment process.

gitHub Action workflow run into failure error as below. This is the result from Checkov ‘s static analysis on kubernetes manifest files. The result comes along with warning messages based on security best practice which is predefined in Checkov.

Since we confirmed Checkov works expectedly, we will narrow down the scope of analysis for lab purpose.

Run this code to scope Checkov analysis to CKV_K8S_17.

cd ~/environment/k8s-manifest-repo/.github/workflows
cat <<EOF> build.yaml
name: "ArgoCD sync"
on: "push"

jobs:
  build:
    runs-on: ubuntu-latest
    steps:

      - name: Checkout source code
        uses: actions/checkout@v2

      - name: Setup Kustomize
        uses: imranismail/setup-kustomize@v1

      - name: Build Kustomize
        run: |
          pwd
          mkdir kustomize-build
          kustomize build ./overlays/dev > ./kustomize-build/kustomize-build-output.yaml
          ls -rlt
          cd kustomize-build
          cat kustomize-build-output.yaml

      - name: Run Checkov action
        id: checkov
        uses: bridgecrewio/checkov-action@master
        with:
          directory: kustomize-build/
          framework: kubernetes
          check: CKV_K8S_17

      - name: Install ArgoCD and execute Sync in ArgoCD
        run: |
          curl -sSL -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
          chmod +x /usr/local/bin/argocd
          ARGO_SERVER=$ARGOCD_SERVER
          argocd app sync eksworkshop-cd-pipeline --auth-token \${{ secrets.ARGOCD_TOKEN }} --server $ARGOCD_SERVER --insecure

EOF

Commit and push code.

cd ~/environment/k8s-manifest-repo
git add .
git commit -m "Chage Checkov check scope"
git push -u origin main

See if gitHub Action workflow completes and trigger ArgoCD deployment process. This time, ArgoCD will be successfully completed without interruption.

8. Test out end-to-end pipeline working

Let’s test out end-to-end pipeline working with code change on front-end application first.

(1) front-end application code change

Go to Cloud9 first, and then move to amazon-eks-frontend/src/ and open App.js in folder tree of the left pane.

Replace code at line 67 with EKS DEMO Blog version 2 and save it.

  return (
    <div className={classes.root}>
      <AppBar position="static" style={{ background: '#2E3B55' }}>
        <Toolbar>
          <IconButton edge="start" className={classes.menuButton} color="inherit" aria-label="menu">
            <CloudIcon />
          </IconButton>
          <Typography
            variant="h6"
            align="center"
            className={classes.title}
          >
            EKS DEMO Blog version 2
          </Typography>
          {new Date().toLocaleTimeString()}
        </Toolbar>
      </AppBar>
      <br/>

(2) commit and push

Commit and push changed code to git repository.

cd ~/environment/amazon-eks-frontend
git add .
git commit -m "Add new blog version 2"
git push -u origin main

See if gitHub Action workflow completes and trigger ArgoCD deployment process. This time, ArgoCD will be successfully completed without interruption.

After end-to-end pipeline finishes successfully, open your local browser with URL of the application from this command.

echo http://$(kubectl get ingress/backend-ingress -o jsonpath='{.status.loadBalancer.ingress[*].hostname}')

It must show you EKS DEMO Blog version 2 at the top of page.

Create CI/CD with HELM & CDK

Build up CI/CD pipeline

1. Deploy CDK stack

Deploy pipeline with AWS CDK for workshop. Below resources will be deployed.

CodeCommit-App-Repo: App Source repo
CodeCommit-Helm-Repo: Helm source
CodeBuild-App: Perform app build and push image to ecr
CodeBuild-Helm: Update image tag and push to helm repo
CodePipeline: Pipeline
ECR-Repo: Container Image Repo for save App Container Image

(0) Prerequisite

To install AWS CDK, follow this instruction . Latest version is recommended.
Check AWS Account

aws sts get-caller-identity
Set AWS Region

export AWS_REGION=$(curl -s 169.254.169.254/latest/dynamic/instance-identity/document | jq -r '.region')

If aws profile setted account and workshop account are different, please re-configure account id to workshop account

(1) Download CDK source for deploy infra and install python packages.

curl 'https://static.us-east-1.prod.workshops.aws/public/d0e72c6e-904d-4933-beec-4c908d928217/static/images/110-cicd/code-pipeline-cdk.zip' --output cdk.zip
unzip cdk.zip
cd cdk
pip install .

If you use Windows OS, please download source with followed link. CDK Source

(2) CDK Bootstrap and deploy

cdk bootstrap
cdk synth
cdk deploy --require-approval never

(3) Check deployed resource.

aws codecommit list-repositories --region ${AWS_REGION}
aws codepipeline list-pipelines --region ${AWS_REGION}
aws codebuild list-projects --region ${AWS_REGION}

2. Deploy Application

(0) Prerequirement for CodeCommit.

pip install git-remote-codecommit

If you use the Instance Profile in Cloud9, you can access codecommit through credential.helper.

git config --global user.email "test@aaa.com" # put your email git config --global --replace-all credential.helper '!aws codecommit credential-helper $@' git config --global credential.UseHttpPath true
If you access with AWS IAM User , get User name and password from IAM User’s Generate Credentials.

(1) Download application source and unzip the source.

curl 'https://static.us-east-1.prod.workshops.aws/public/d0e72c6e-904d-4933-beec-4c908d928217/static/images/110-cicd/app.zip' --output app.zip
unzip app.zip

If you use Windows OS, please download source with followed link. App Source

(2) Push source codes into eks-workshop-app repository.

export APP_CODECOMMIT_URL=$(aws codecommit get-repository --repository-name eks-workshop-app --region ${AWS_REGION} | grep -o '"cloneUrlHttp": "[^"]*'|grep -o '[^"]*$')

git clone $APP_CODECOMMIT_URL
cd eks-workshop-app

cp -R ../app/* ./
git add .
git commit -m "init app Source"
git push origin master

3. Deploy Helm

(1) Download helm source and unzip the source.

curl 'https://static.us-east-1.prod.workshops.aws/public/d0e72c6e-904d-4933-beec-4c908d928217/static/images/110-cicd/helm.zip' --output helm.zip
unzip helm.zip

If you use Windows OS, please download source with followed link. Helm Source

(2) Helm CodeCommit Repo push

export HELM_CODECOMMIT_URL=$(aws codecommit get-repository --repository-name eks-workshop-helm --region ${AWS_REGION} | grep -o '"cloneUrlHttp": "[^"]*'|grep -o '[^"]*$')
cd helm
git init
git checkout -b master
git add .
git commit -m "init"
git remote add origin $HELM_CODECOMMIT_URL
git push origin master

4. Check Code Pipeline

5. Install ArgoCD

(1) ArgoCD install

ArgoCD install to EKS cluster with below commands

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Below commands for argocd cli

cd ~/environment
VERSION=$(curl --silent "https://api.github.com/repos/argoproj/argo-cd/releases/latest" | grep '"tag_name"' | sed -E 's/.*"([^"]+)".*/\1/')

sudo curl --silent --location -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/download/$VERSION/argocd-linux-amd64
sudo chmod +x /usr/local/bin/argocd

Mac install command

brew install argocd

Argocd is not recommanded expose public, but we expose argocd to public for workshop.

kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'

check argocd server host

export ARGOCD_SERVER=`kubectl get svc argocd-server -n argocd -o json | jq --raw-output .status.loadBalancer.ingress[0].hostname`
echo $ARGOCD_SERVER

Argocd username is admin and get pw from below command.

ARGO_PWD=`kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d`
echo $ARGO_PWD

open $ARGOCD_SERVER from browser and input admin and $ARGO_PWD

6. Configure ArgoCD

(0) Argocd access iam user configure

Create user from IAM Console

Set Username and check Access Key ckeckbox

Choose AWSCodeCommitPoserUser Policy

And reaccess to IAM user console and click the Security credentials tab and generate credenitals from HTTPS Git credentials for AWS CodeCommit

Download credentials or note

(1) Configure ArgoCD

Click Connect Repo Button Method -> VIA HTTPS, Project -> default

Repository URL -> CodeCommit Helm Repo’s HTTPS Address, Put UserName Password

NewApp button click in Application tab Application Name -> random , Project -> default

Sync policy -> AUTOMATIC, RepoURL -> before generated Repository, PATH -> . , DESTINATION section’s Cluster URL -> https://kubernetes.default.svc, Namespace -> default and click Create

(5) Check ArgoCD

Check deployment status in Argocd console

Applications > APP Name

Check new pipeline check image tag from Helm’s commit history

And check Argocd console’s eks-workshop pod Image Tag

Move to Applications > eks-workshop > and check eks-workshop pod click and check status

It will be continous synced Helm Repo and Argocd, when Helm Repo commit occurred.

7. Check Pipeline

Commit Java code and check deployment app.

(1) Create source code

Create app/src/main/java/com/aws/sample/HelloWorldController.java from cloud9 console.

package com.aws.samples;

import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.servlet.ModelAndView;

@Controller

public class HelloWorldController {

    @GetMapping("/hello-world")

    public ModelAndView hello() {

        ModelAndView modelAndView = new ModelAndView("hello-world");
        modelAndView.addObject("str", "Hello World");
        return modelAndView;
    }
}

Create app/src/main/resources/templates/hello-world.html

<!DOCTYPE html>
<html lang="en"
      xmlns="http://www.w3.org/1999/xhtml"
      xmlns:th="http://www.thymeleaf.org">
<head>
    <meta charset="utf-8">
    <meta content="width=device-width, initial-scale=1" name="viewport"/>
    <title>SampleApp</title>
    <link href="/favicon.ico" rel="icon">
    <link href="https://cdn.jsdelivr.net/npm/bootstrap@4.6.1/dist/css/bootstrap.min.css" rel="stylesheet">
    <link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.3/css/all.css" rel="stylesheet"
          type="text/css"/>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.6.0/jquery.min.js"
            integrity="sha512-894YE6QWD5I59HgZOGReFYm4dnWc1Qt5NtvYSaNcOP+u1T9qYdvdihz0PPSiiqn/+/3e7Jo4EaG7TubfWGUrMQ=="
            crossorigin="anonymous" referrerpolicy="no-referrer"></script>
    <script src="https://cdn.jsdelivr.net/npm/bootstrap@4.6.1/dist/js/bootstrap.min.js"
            integrity="sha256-SyTu6CwrfOhaznYZPoolVw2rxoY7lKYKQvqbtqN93HI=" crossorigin="anonymous"></script>
</head>
<body>
<div>
    <div class="container">
        <h1 style="text-align: center; margin-top: 10px" th:text=" ${str} + '&#127811;'"></h1>
    </div>
</div>
</body>
</html>

(2) Commit and push

cd app
git add .
git commit -m "Add hello-world page"
git push origin master

(3) check result

Check Codepipeline status in AWS console.

Check ArgoCd console

Access to Sample App

echo http://$(kubectl get ingress/eks-workshop-workshop-example-app -o jsonpath='{.status.loadBalancer.ingress[*].hostname}')

If the deployment successed, Hello-world menu print nomally

Optional : IRSA setting for APP Cluster Menu

Policy Generate

cat <<EOF> eks-workshop-test-policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "eks:DescribeCluster",
                "eks:ListClusters"
            ],
            "Resource": "*"
        }
    ]
}
EOF

create Policy

aws iam create-policy \
    --policy-name EKSWorkshopTestPolicy \
    --policy-document file://eks-workshop-test-policy.json

check OIDC Url

export OIDC_URL=$(aws eks describe-cluster --name [my-cluster] --query "cluster.identity.oidc.issuer" --output text)
export ACCOUNT_ID=$(curl -s 169.254.169.254/latest/dynamic/instance-identity/document | jq -r '.accountId')
export FEDERATED=arn:aws:iam::$ACCOUNT_ID:oidc-provider/$OIDC_URL
create Trust Policy
cat >eks-workshop-test-trust-policy.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "$FEDERATED"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com",
                    "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:default:eks-workshop-test-role"
                }
            }
        }
    ]
}
EOF

Create IAM Role

aws iam create-role \
  --role-name eks-workshop-test-role \
  --assume-role-policy-document file://"eks-workshop-test-trust-policy.json"

Attach policy to role

aws iam attach-role-policy \
  --policy-arn arn:aws:iam::$ACCOUNT_ID:policy/EKSWorkshopTestPolicy \
  --role-name eks-workshop-test-role

Edit values yaml role-arn and push to helm repo

serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # Annotations to add to the service account
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT_ID:role/eks-workshop-test-role
  name: workshop-example-app

Pull recent commit from helm repo

cd helm
git pull
git add .
git commit -m "update irsa role arn"
git push origin master

If conflict occurred, solve the conflict.

git pull
git merge origin/master
git push origin master

check deployment in argocd

Clean up resources

At the end of this workshop, you need to delete used resources to avoid additional costs to your AWS account.

Delete the Ingress resources. At this point, perform the command in the folder where the yaml file is located(/home/ec2-user/environment/manifests).

cd ~/environment/manifests/

kubectl delete -f flask-ingress.yaml
kubectl delete -f nodejs-ingress.yaml
kubectl delete -f frontend-ingress.yaml
kubectl delete -f alb-ingress-controller/v2_5_4_full.yaml
Delete EKS cluster.

eksctl delete cluster --name=eks-demo

[!] Check that all related stacks have been deleted from AWS CloudFormation console.

Remove Amazon ECR repository. With the command below, load the list of repository that you created.

aws ecr describe-repositories

aws ecr delete-repository --repository-name demo-flask-backend --force
aws ecr delete-repository --repository-name demo-frontend --force
Delete the collected metrics.

aws logs describe-log-groups --query 'logGroups[*].logGroupName' --output table | \
awk '{print $2}' | grep ^/aws/containerinsights/eks-demo | while read x; do echo "deleting $x" ; aws logs delete-log-group --log-group-name $x; done

aws logs describe-log-groups --query 'logGroups[*].logGroupName' --output table | \
awk '{print $2}' | grep ^/aws/eks/eks-demo | while read x; do echo "deleting $x" ; aws logs delete-log-group --log-group-name $x; done
Delete the Cloud9 IDE environment you created.

Challenges Faced and Solutions

Challenge 1: Managing Kubernetes Configurations

Solution: Used eksctl and pre-configured YAML templates to manage and deploy configurations easily.

Challenge 2: Monitoring Application and Cluster Performance

Solution: AWS Container Insights was critical in providing visibility into resource utilization and application health.

Challenge 3: Automating Deployment without Downtime

Solution: CI/CD with blue-green deployment strategies helped ensure smooth transitions with minimal downtime during updates.

Conclusion

This project demonstrates the power and flexibility of Amazon EKS and AWS services in building a scalable, high-performance web application infrastructure. By using EKS for container orchestration and integrating tools like Amazon ECR, AWS Fargate, and CloudWatch Container Insights, the application achieves automated deployment, efficient resource management, and robust monitoring. This setup is ideal for production environments where scalability, resilience, and operational efficiency are crucial, equipping DevOps teams with an automated and adaptable solution that can seamlessly handle dynamic workloads and application demands.

Explore my GitHub repository.

Let’s connect: Linkdin, Twitter, Github

Create a Continuous Delivery Pipeline : AWS Project

Shubham Murti — Tue, 12 Nov 2024 12:40:25 +0000

Introduction

This project implements a Continuous Delivery (CD) pipeline using AWS CodePipeline, AWS CodeBuild, and AWS Elastic Beanstalk to automate deployments. The setup provides a structured approach to code deployment, enhancing reliability and minimizing manual processes—ideal for agile development teams aiming for efficient, high-frequency deployments.

Tech Stack

AWS CodePipeline: Manages the entire deployment flow, coordinating the build, test, and deployment stages.
AWS CodeBuild: Automates the building and testing of code, ensuring that each change is thoroughly validated before deployment.
AWS Elastic Beanstalk: Manages the deployment of the web application, ensuring it is hosted in a high-availability, scalable environment.
Amazon EC2 with Auto Scaling: Ensures scalable, fault-tolerant infrastructure to support application load, with ALB (Application Load Balancer) for even traffic distribution.

Prerequisites

AWS Account: Required for configuring the CD pipeline and associated services.
Code Repository: A source code repository such as GitHub or CodeCommit that integrates with CodePipeline.
Basic CI/CD Knowledge: Familiarity with the principles of continuous integration and continuous delivery.
AWS CLI: To facilitate configuration and command-line management.

Problem Statement or Use Case

Problem: Manually deploying updates to web applications is error-prone, time-consuming, and can lead to inconsistent deployment practices, especially in fast-paced development environments.

Solution: The Continuous Delivery Pipeline ensures that each code change is automatically built, tested, and deployed to a managed environment, reducing manual intervention. This setup allows developers to push code more frequently, get faster feedback, and minimize deployment risks.

Real-World Relevance: In production settings, a CD pipeline is crucial for agile teams who need reliable, frequent deployments without impacting application uptime or performance. This project demonstrates how AWS can automate and manage application deployments, making it ideal for high-availability and fast-paced development scenarios.

Architecture Diagram

Component Breakdown

Code Repository: A source control repository (GitHub or CodeCommit) triggers the pipeline when new code is committed, enabling continuous integration.
AWS CodePipeline: Automates the entire CI/CD process, orchestrating each stage from source to build to deployment.
AWS CodeBuild: Builds and tests the code. CodeBuild compiles, runs unit tests, and verifies the application to ensure it is ready for deployment.
AWS Elastic Beanstalk: Deploys the built application onto a highly available environment with Auto Scaling capabilities, which provides a robust infrastructure layer for the application.

Step-by-Step Implementation

Module 1: Set Up Git Repo

Fork the starter repo

This tutorial assumes you have an existing GitHub account and Git installed on your computer. If you don’t have either of these two installed, you can follow these step-by-step instructions.

In a new browser tab, navigate to GitHub and make sure you are logged into your account.
In that same tab, open the aws-elastic-beanstalk-express-js-sample repo.
Choose the white Fork button on the top right corner of the screen. Next, you will see a small window asking you where you would like to fork the repo.
Verify it is showing your account and choose Create a fork. After a few seconds, your browser will display a copy of the repo in your account under Repositories.

Push a change to your new repo

Go to the repository and choose the green Code button near the top of the page.
To clone the repository using HTTPS, confirm that the heading says Clone with HTTPS. If not, select the Use HTTPS link.
Choose the white button with a clipboard icon on it (to the right of the URL).

If you’re on a Mac or Linux computer, open your terminal. If you’re on Windows, launch Git Bash.
In the terminal or Bash platform, whichever you are using, enter the following command and paste the URL you just copied in Step 2 when you clicked the clipboard icon. Be sure to change “YOUR-USERNAME” to your GitHub username. You should see a message in your terminal that starts with Cloning into. This command creates a new folder that has a copy of the files from the GitHub repo.

git clone https://github.com/YOUR-USERNAME/aws-elastic-beanstalk-express-js-sample
In the new folder there is a file named app.js. Open app.js in your favorite code editor.
Change the message in line 5 to say something other than “Hello World!” and save the file.
Go to the folder created with the name aws-elastic-beanstalk-express-js-sample/ and Commit the change with the following commands:

git add app.js
git commit -m "change message"
Push the local changes to the remote repo hosted on GitHub with the following command. Note that you need to configure Personal access tokens (classic) under Developer Settings in GitHub for remote authentication.

git push

Test your changes

In your browser window, open GitHub.
In the left navigation panel, under Repositories, select the one named aws-elastic-beanstalk-express-js-sample.
Choose the app.js file. The contents of the file, including your change, should be displayed.

Application architecture

Here is what our architecture looks like right now:

We have created a code repository containing a simple web app. We will be using this repository to start our continuous delivery pipeline. It’s important to set it up properly so we push code to it.

Module 2: Deploy Web App

Implementation

Configure an AWS Elastic Beanstalk app

In a new browser tab, open the AWS Elastic Beanstalk console.
Choose the orange Create Application button.
Choose Web server environment under the Configure environment heading.
In the text box under the heading Application name, enter DevOpsGettingStarted*.*
In the Platform dropdown menu, under the Platform heading, select Node.js . Platform branch and Platform version will automatically populate with default selections.
Confirm that the radio button next to Sample application under the Application code heading is selected.
Confirm that the radio button next to Single instance (free tier eligible) under the Presets heading is selected.
Select Next.

On the Configure service access screen, choose Use an existing service role for Service Role.
For EC2 instance profile dropdown list, the values displayed in this dropdown list may vary, depending on whether you account has previously created a new environment.
Choose one of the following, based on the values displayed in your list.

If aws-elasticbeanstalk-ec2-role displays in the dropdown list, select it from the EC2 instance profile dropdown list.
If another value displays in the list, and it’s the default EC2 instance profile intended for your environments, select it from the EC2 instance profile dropdown list.
If the EC2 instance profile dropdown list doesn’t list any values to choose from, expand the procedure that follows, Create IAM Role for EC2 instance profile.
Complete the steps in Create IAM Role for EC2 instance profile to create an IAM Role that you can subsequently select for the EC2 instance profile. Then, return back to this step.
Now that you’ve created an IAM Role, and refreshed the list, it displays as a choice in the dropdown list. Select the IAM Role you just created from the EC2 instance profile dropdown list.

Choose Skip to Review on the Configure service access page.

This will select the default values for this step and skip the optional steps.

The Review page displays a summary of all your choices.
Choose Submit at the bottom of the page to initialize the creation of your new environment.

While waiting for deployment, you should see:

A screen that will display status messages for your environment.
After a few minutes have passed, you will see a green banner with a checkmark at the top of the environment screen.

Once you see the banner, you have successfully created an AWS Elastic Beanstalk application and deployed it to an environment.

Test your web app

To test your sample web app, select the link under the name of your environment.

2. Once the test has completed, a new browser tab should open with a webpage congratulating you!

Application architecture

Now that we are done with this module, our architecture will look like this:

We have created an AWS Elastic Beanstalk environment and sample application. We will be using this environment and our continuous delivery pipeline to deploy the Hello World! web app we created in the previous module.

Module 3: Create Build Project

Configure the AWS CodeBuild project

In a new browser tab, open the AWS CodeBuild console.
Choose the orange Create project button.
In the Project name field, enter Build-DevOpsGettingStarted.
Select GitHub from the Source provider dropdown menu.
Confirm that the Connect using OAuth radio button is selected.
Choose the white Connect to GitHub button. A new browser tab will open asking you to give AWS CodeBuild access to your GitHub repo.
Choose the green Authorize aws-codesuite button.
Enter your GitHub password.
Choose the orange Confirm button.
Select Repository in my GitHub account.
Enter aws-elastic-beanstalk-express-js-sample in the search field.
Select the repo you forked in Module 1. After selecting your repo, your screen should look like this:

Confirm that Managed Image is selected.
Select Amazon Linux 2 from the Operating system dropdown menu.
Select Standard from the Runtime(s) dropdown menu.
Select aws/codebuild/amazonlinux2-x86_64-standard:3.0 from the Image dropdown menu.
Confirm that Always use the latest image for this runtime version is selected for Image version.
Confirm that Linux is selected for Environment type.
Confirm that New service role is selected.

Create a Buildspec file for the project

Select Insert build commands.
Choose Switch to editor.
Replace the Buildspec in the editor with the code below:

version: 0.2
phases:
build:
commands:
- npm i --save
artifacts:
files:
- '*/'
1. Choose the orange Create build project button. You should now see a dashboard for your project.

Test the CodeBuild project

Choose the orange Start build button. This will load a page to configure the build process.
Confirm that the loaded page references the correct GitHub repo.
Choose the orange Start build button.
Wait for the build to complete. As you are waiting you should see a green bar at the top of the page with the message Build started, the progress for your build under Build log, and, after a couple minutes, a green checkmark and a Succeeded message confirming the build worked.

Application architecture

Here’s what our architecture looks like now:

We have created a build project on AWS CodeBuild to run the build process of the Hello World! web app from our GitHub repository. We will be using this build project as the build step in our continuous delivery pipeline, which we will create in the next module.

Module 4: Create Delivery Pipeline

Create a new pipeline

In a browser window, open the AWS CodePipeline console.
Choose the orange Create pipeline button. A new screen will open up so you can set up the pipeline.
In the Pipeline name field, enter Pipeline-DevOpsGettingStarted.
Confirm that New service role is selected.
Choose the orange Next button.

Configure the source stage

Select GitHub version 1 from the Source provider dropdown menu.
Choose the white Connect to GitHub button. A new browser tab will open asking you to give AWS CodePipeline access to your GitHub repo.
Choose the green Authorize aws-codesuite button. Next, you will see a green box with the message You have successfully configured the action with the provider.
From the Repository dropdown, select the repo you created in Module 1.
Select main from the branch dropdown menu.
Confirm that GitHub webhooks is selected.
Choose the orange Next button.

Configure the build stage

From the Build provider dropdown menu, select AWS CodeBuild.
Under Region confirm that the US West (Oregon) Region is selected.
Select Build-DevOpsGettingStarted under Project name.
Choose the orange Next button.

Configure the deploy stage

Select AWS Elastic Beanstalk from the Deploy provider dropdown menu.
Under Region, confirm that the US West (Oregon) Region is selected.
Select the field under Application name and confirm you can see the app DevOpsGettingStarted created in Module 2.
Select DevOpsGettingStarted-env from the Environment name textbox.
Choose the orange Next button. You will now see a page where you can review the pipeline configuration.
Choose the orange Create pipeline button.

Watch first pipeline execution

While watching the pipeline execution, you will see a page with a green bar at the top. This page shows all the steps defined for the pipeline and, after a few minutes, each will change from blue to green.

Once the Deploy stage has switched to green and it says Succeeded, choose AWS Elastic Beanstalk. A new tab listing your AWS Elastic Beanstalk environments will open.
Select the URL in the Devopsgettingstarted-env row. You should see a webpage with a white background and the text you included in your GitHub commit in Module 1.

Application architecture

Here’s what our architecture looks like now:

We have created a continuous delivery pipeline on AWS CodePipeline with three stages: source, build, and deploy. The source code from the GitHub repo created in Module 1 is part of the source stage. That source code is then built by AWS CodeBuild in the build stage. Finally, the built code is deployed to the AWS Elastic Beanstalk environment created in Module 3.

Module 5: Finalize Pipeline and Test

Create a review stage in pipeline

Open the AWS CodePipeline console.
You should see the pipeline we created in Module 4, which was called Pipeline-DevOpsGettingStarted. Select this pipeline.
Choose the white Edit button near the top of the page.
Choose the white Add stage button between the Build and Deploy stages.
In the Stage name field, enter Review.
Choose the orange Add stage button.
In the Review stage, choose the white Add action group button.
Under Action name, enter Manual_Review.
From the Action provider dropdown, select Manual approval.
Confirm that the optional fields have been left blank.
Choose the orange Done button.
Choose the orange Save button at the top of the page.
Choose the orange Save button to confirm the changes. You will now see your pipeline with four stages: Source, Build, Review, and Deploy.

Push a new commit to your repo

In your favorite code editor, open the app.js file from Module 1.
Change the message in Line 5.
Save the file.
Open your preferred Git client.
Navigate to the folder created in Module 1.
Commit the change with the following commands:

git add app.js
git commit -m "Full pipeline test"
1. Push the local changes to the remote repo hosted on GitHub with the following command:
git push

Monitor the pipeline and manully approve the change

Navigate to the AWS CodePipeline console.
Select the pipeline named Pipeline-DevOpsGettingStarted. You should see the Source and Build stages switch from blue to green.
When the Review stage switches to blue, choose the white Review button.
Write an approval comment in the Comments textbox.
Choose the orange Approve button.
Wait for the Review and Deploy stages to switch to green.
Select the AWS Elastic Beanstalk link in the Deploy stage. A new tab listing your Elastic Beanstalk environments will open.
Select the URL in the Devopsgettingstarted-env row. You should see a webpage with a white background and the text you had in your most recent GitHub commit.

Congratulations! You have a fully functional continuous delivery pipeline hosted on AWS.

Application architecture

With all modules now completed, here is the architecture of what you built:

We have used AWS CodePipeline to add a review stage with manual approval to our continuous delivery pipeline. Now, our code changes will have to be reviewed and approved before they are deployed to AWS Elastic Beanstalk.

Clean up resources

Delete AWS Elastic Beanstalk application

In a new browser window, open the AWS Elastic Beanstalk Console.
In the left navigation menu, click on “Applications.” You should see the “DevOpsGettingStarted” application listed under “All applications.”
Select the radio button next to “DevOpsGettingStarted.”
Click the white dropdown “Actions” button at the top of the page.
Select “Delete application” under the dropdown menu.
Type “DevOpsGettingStarted” in the text box to confirm deletion.
Click the orange “Delete” button.

Delete pipeline in AWS CodePipeline

In a new browser window, open the AWS CodePipeline Console.
Select the radio button next to “Pipeline-DevOpsGettingStarted.”
Click the white “Delete pipeline” button at the top of the page.
Type “delete” in the text box to confirm deletion.
Click the orange “Delete” button.

Delete pipeline resources from Amazon S3 bucket

In a new browser window, open the Amazon S3 Console.
You should see a bucket named “codepipeline-us-west-2” followed by your AWS account number. Click on this bucket. Inside this bucket, you should see a folder named “Pipeline-DevOpsGettingStarted.”
Select the checkbox next to the “Pipeline-DevOpsGettingStarted” folder.
Click the white “Actions” button from the dropdown menu.
Select “Delete” under the dropdown menu.
Click the blue “Delete” button.

Delete build project in AWS CodeBuild

In a new browser window, open the AWS CodeBuild Console.
In the left navigation, click on “Build projects” under “Build.” You should see the “Build-DevOpsGettingStarted” build project listed under “Build project.”
Select the radio button next to “Build-DevOpsGettingStarted.”
Click the white “Delete build project” button at the top of the page.
Type “delete” in the text box to confirm deletion.
Click the orange “Delete” button.

Congratulations!

You successfully built a continuous delivery pipeline on AWS! As a great next step, dive deeper into specific AWS technologies and take your application to the next level.

Challenges Faced and Solutions

Deployment Failures Due to Environment Variables: At times, missing environment variables caused builds to fail.
Solution: Ensured that the required environment variables were securely stored in Elastic Beanstalk and made accessible to the application during deployment.
CodePipeline Integration with External Repositories: Faced issues when integrating CodePipeline with GitHub.
Solution: Configured OAuth permissions carefully and verified GitHub webhook functionality to trigger pipeline events.

Conclusion

The project demonstrates an efficient AWS Continuous Delivery pipeline that automates deployments from commit to production. By integrating CodePipeline, CodeBuild, and Elastic Beanstalk, development teams can focus on code quality and agility, resulting in faster releases and robust application performance.

Explore my GitHub repository.

Let’s connect: Linkdin, Twitter, Github

Create a Highly Available WordPress Web Application : AWS Project

Shubham Murti — Tue, 12 Nov 2024 10:57:39 +0000

Introduction

This project focuses on designing a resilient, scalable WordPress web application on AWS. Using key AWS services like Amazon VPC, Amazon RDS, Amazon EFS, EC2, and Application Load Balancer (ALB), it establishes a robust architecture that ensures high availability, scalability, and fault tolerance for a WordPress site, making it suitable for high-traffic applications.

By deploying WordPress in a multi-tier architecture, I learned how to leverage AWS infrastructure to meet the needs of high-traffic applications, ensuring both availability and fault tolerance.

Tech Stack

Amazon VPC: Provides isolated network environments and security controls.
Amazon RDS: Hosts a reliable, highly available MySQL database for WordPress.
Amazon EFS: A scalable, shared storage solution for dynamic content, enabling multiple EC2 instances to access the same data.
Amazon EC2: Hosts WordPress instances and dynamically scales with Auto Scaling groups.
Application Load Balancer (ALB): Distributes incoming traffic across multiple instances for enhanced availability.

Prerequisites

AWS Account: Required to access and configure AWS services.
AWS CLI: For resource management and deployment tasks.
Basic AWS Networking Knowledge: Understanding of VPCs, subnets, and security groups.
WordPress Setup Knowledge: Familiarity with WordPress installation and configuration.

Problem Statement or Use Case

Problem: WordPress applications often face challenges around scalability and availability, especially with traditional hosting environments where capacity may not automatically adjust to traffic demands.

Solution: Using AWS, this project demonstrates how to set up WordPress in a highly available, scalable architecture that can handle fluctuations in traffic and ensure minimal downtime. AWS’s managed services enable the environment to scale automatically, handle failover, and improve the user experience.

Real-World Relevance: This approach is ideal for production-grade WordPress applications with high traffic, such as e-commerce sites, news platforms, and corporate blogs. By implementing this architecture, companies can reduce manual management, lower costs, and improve their WordPress application’s reliability and speed.

Step-by-Step Implementation

Configure the network

Create a new Virtual Private Cloud (VPC)

As a starting point for the workshop you will need to login to your AWS account, select the region of your choice and create a new VPC.

To do this click on Your VPCs on the left hand side of the console and click Create VPC. Enter wordpress-workshop as name for your VPC and a CIDR range such as the one below. When you're fiinished click Create.

After creating the VPC, on the VPC details page click on Actions and then select Edit VPC Settings.
Make sure to enable both DNS resolution and DNS hostnames under DNS Settings and click Save.

Create public and private subnets in the new VPC

Once the VPC has been created, the next step is to create the subnets that will be used to host the application across two different Availability Zones. We are going to create six subnets in total, three for each AZ, as shown in the following diagram:

The first pair of subnets, Public, will be accessible from the Internet and contain load balancers and NAT gateways. The second pair, Application, will contain application servers and your shared EFS filesystem. Your application servers will be able to communicate with the Internet via the NAT gateways but will only be addressable from the load balancers. Finally the Database pair of subnets will hold your active / passive relational database. It will be accessible to other resources in the VPC but will have no access to the Internet and cannot be addressed by the Internet or the load balancers.

To create each of the six subnets please select Subnets on the left of the AWS VPC console, then click on Create subnet and use the details in the table below to define the characteristics of each of your subnets. Make sure to always select the Wordpress-workshop VPC when creating the subnets.

The screenshots in this lab were taken from a deployment in the Ireland (eu-west-1) region, if you are building in a different AWS region please just ensure that you create your subnets in 2 different availability zones in the same region, such as us-west-2a and us-west-2b.

For each subnet specify a name and a CIDR range for the subnet. Be sure and create a public, application, and data subnet in each of two availability zones as detailed in the table below.

At this point all the correct subnets have been created and they can route network traffic between them. In the next set of steps you will create an Internet Gateway, allowing communication between your VPC and the Internet. You will also configure your routing tables to only allow Internet communication with your public subnets and not the private application or data subnets.

Create an Internet Gateway and set up routing

The following steps will allow connectivity from the Internet to the public subnets and also connectivity from the private subnets to the Internet via NAT gateways.

First you need to create a new Internet Gateway (IGW) from your VPC dashboard and attach it to the wordpress-workshop VPC. Start by clicking Internet gateways on the left hand side of the VPC console and then click the Create Internet gateway button. Enter a name for your IGW such as WP Internet Gateway and click Create Internet gateway.

After the IGW has been created you need to associate it with your VPC by attaching it to your VPC.
Select Attach to VPC from the Actions drop-down menu then select the wordpress-workshop VPC from drop-down list of available VPCs and click on Attach Internet gateway.

The gateway will be used by instances and services running in the public subnets (e.g. Public Subnet A and Public Subnet B) to communicate to the Internet.

Once the gateway is created you will need to create a new routing table and associate it with the public subnets.
Create a new route table by selecting Route tables in the left-hand menu of the console and then clicking on the Create route table button.
Give it a Name and select the wordpress-workshop from the drop-down, then click on Create route table.

After creating the route table, select it from the Route tables section of your VPC dashboard, then click on Actions -> Edit routes and add a default route via the Internet Gateway created in the previous step and click on Save changes.

Finally, you need to associate the newly created route table with the public subnets. To do that, click on the public route table, then click on Subnet Associations, edit by clicking on Edit subnet associations and select the two public subnets created earlier and click on Save associations.

Create one NAT gateway in each public subnet

The Wordpress instances will need to be able to connect to the Internet and download application and OS updates. To avoid dependencies across Availability Zones, you are going to create two NAT gateways, one for each Availability Zone where the application is deployed.

To do this, you will create one NAT Gateway in each Availability Zone, then create one route table for each application subnet, update the route table with a default route through the NAT gateway in the same AZ, and then associate the route table to the respective application subnet.

Go to the VPC dashboard in your account, select NAT gateways and create one gateway in each of the two public subnets (i.e. Public Subnet A and Public Subnet B) Always make sure you have selected the correct public subnet when creating the gateway.

Now we need to create route tables for each of the two Application subnets and use the NAT gateways created earlier as the default gateway:

Edit the route table and add the default route via the NAT gateway in Application subnet A:

Associate the route table with Application Subnet A:

Repeat the last three steps to also create a route table for Application Subnet B which uses the NAT gateway deployed in the second availability zone.

Verify your configuration

You have now created a virtual private cloud network across two availability zones within an AWS region. You have created six subnets, three in each availability zone, and have configured a route so that the Internet can communicate with resources in the public subnets and vice versa. The application subnets have been configured, via routing table, to communicate with the Internet via NAT gateways in the public subnets, and the data subnets can only communicate with resources in the six subnets, but not the Internet.

Please note that the information below is based on a VPC deployed in the Ireland (eu-west-1) region. If you had choosen a different region for your setup you need to adjust the region name accordingly.

You can compare your own configuration based on the screenshot below and move along when you have verified your setup.

Check the Resource map section of your VPC which shows your VPC, subnets, route tables, Internet gateways, NAT gateways which helps you visualise the resources.

Building the Data Tier

Set up the RDS database

Create database security groups

You will create 2 security groups:

WP Database Clients will be attached to the EC2 instances running the web servers
WP Database will be attached to the RDS DB instance

Visit the Amazon VPC console and create 2 security groups.

First, create the WP Database Clients security group:

Click on Create security group
Fill in the Security group name and Description fields
Select the wordpress-workshop from the drop-down
Scroll to the bottom of the page and click on Create security group

Now create the WP Database security group:

In the Inbound Rules section, click on Add rule
Select Type MySQL/Aurora which allows traffic on port 3306 from Custom source WP Database Clients security group.

Please note, that you can search for the security group by name in the source field of the security group rule.

Now you are ready to create your RDS database.

Create an RDS subnet group

Amazon RDS is an easy to manage relational database service. When you use Amazon RDS to deploy a database in a highly available setup, it will create 2 instances in 2 different availability zones. To do this, when you create a database you specify a subnet group which tells RDS in which subnets it can deploy your database instances.

To create a DB subnet group browse to the Amazon RDS console , click on Subnet groups **in the panel on your left, click on the **Create DB Subnet Group button and use the following details:

Name: Aurora-Wordpress
Description: RDS subnet group used by Wordpress
VPC: wordpress-workshop

Scroll down and add the two Data subnets created earlier (one for each AZ) to your new subnet group and click Create.

Please note, in order to get the ID of the data subnets, you can open a second tab and navigate to the **Subnets **section of the VPC console. From the list of subnets select the one you are interested in. On the bottom of the screen you will then be able to copy the Subnet ID by clicking the Copy to clipboard icon beside the id.

Create the Aurora database instance

Once the subnet group has been created you are ready to launch the RDS-managed database.
Go to the Amazon RDS Console , select Databases **from the menu on the left and click **Create database.

Enter the following details:

Database creation method: Standard create
Engine options: Aurora (MySQL Compatible)
Keep the default Engine Version

Use wpadminas Master username
When prompted for the Master password, you can either click on Auto generate a password or create your own — in either case please make sure you write down the password as it will be required a few steps later when setting up the connectivity of the Wordpress instances to the database.

Select the DB instance size together with a Multi-AZ deployment, required for high availability. To keep costs low, for this workshop we recommend using a burstable instance class (db.t4g.medium or similar). Burstable instances might not be suited for production environments.

In the connectivity section, make sure you select the wordpress-workshop VPC, together with the aurora-wordpress DB subnet group, and WP Database security group created earlier:

In the Monitoring section, uncheck Turn on DevOps Guru

Expand Additional configuration section and specify an Initial database name of wordpress.

Finally, click on Create database to start building the cluster.
The database will take a few minutes to be provisioned and made available.

Verify your configuration

The active / passive database should now be available and running in two different availability zones, waiting for connections from any EC2 resource with the client security group associated to it.
Please compare your own configuration based on the screenshots below and move along when you have verified your setup.

Security Groups

Subnet group

Database setup

Create the shared filesystem

Amazon Elastic File System (Amazon EFS) provides a simple, scalable, elastic file system for general purpose workloads for use with AWS Cloud services and on-premises resources. In this lab you will create an EFS cluster that will provide a shared filesystem for your Wordpress content.

Create filesystem security groups

When using Amazon EFS, you specify Amazon EC2 security groups for your EC2 instances and security groups for the EFS mount targets associated with the file system. A security group acts as a firewall, and the rules that you add define the traffic flow.

In this workshop, you will create 2 security groups:

WP EFS Clients will be attached to the EC2 instances running the web servers
WP EFS will be attached to the EFS mount targets

Visit the Amazon VPC console to create the 2 security groups.

First, create the WP EFS Clients security group:

Click on Create security group
Fill in the Security group name and Description fields
Select the wordpress-workshop VPC from the drop-down
Scroll to the bottom of the page and click on Create security group

Amazon EFS creates a shared file system and exposes it as a NFS share. The security group attached to the EFS mount points will need to allow inbound connections on the NFS TCP port 2049.

Create the WP EFS security group:

In the Inbound Rules section, click on Add rule
Select Type NFS and specify Custom source WP EFS Clients security group.

Please note, that you can search for the security group by name in the source field of the security group rule.

Now you are ready to create the EFS file system.

Create the EFS file system

To create an EFS file system visit the Amazon EFS console and click Create file system.

Enter Wordpress-EFSin the Name field.
From the VPC drop down select the wordpress-workshop VPC and click Customize.

On the creation page, uncheck Enable automatic backups to avoid backing up the contents of the file system. It’s recommended to keep it enabled when deploying a production environment.

Keep all other settings unchanged and click Next.

On the Network access page, under Mount targets, choose the two subnets created for the Data tier (Data subnet A and B). On the right side, under Security groups, associate the WP EFS security group created above to each mount target and remove the association with the Default security group.
Click Next.

Accept the defaults on the next screen for File system policy

Click Next, review and confirm the file system creation by clicking Create.

This will create two mount targets in the Data subnets and after a few moment the file system will become Available.

Build the Application Tier

Create the load balancer

To distribute traffic across your Wordpress application servers you will need a load balancer. In this lab you will create an Application Load Balancer.

Application Load Balancer operates at the request level (layer 7), routing traffic to targets (EC2 instances, containers, IP addresses, and Lambda functions) based on the content of the request

Create load balancer and application security groups

Visit the Amazon VPC console to create the security groups.

First, create the WP Load Balancer security group:

Click on Create security group
Fill in the Security group name and Description fields
Select the wordpress-workshop from the drop-down
In the Inbound Rules section, click on Add rule
Select Type HTTP which allows traffic on port 80 from My IP source to limit access to your current public IP.
Scroll to the bottom of the page and click on Create security group

Outside of a workshop environment you would likely want to modify the security group to allow access from any IP address. To learn more about security in and of the cloud please visit the AWS Cloud Security website.

Now create the WP Web Servers security group:

In the Inbound Rules section, click on Add rule
Select Type HTTP which allows traffic on port 80 from Custom source WP Load Balancer security group.

Please note, that you can search for the security group by name in the source field of the security group rule.

Create a load balancer

A load balancer distributes incoming application traffic across multiple targets, such as EC2 instances, in multiple Availability Zones, increasing the availability of the Wordpress platform.

From the EC2 console click Load Balancers **on the left-hand menu and then click **Create load balancer.
Click Create **under **Application Load Balancer.

Give your load balancer a name and under Network mapping select the wordpress-workshop VPC.
Then tick the checkbox for both availability zones and select the public subnets created in the first lab.

Under the Security groups select the WP Load Balancer created earlier and remove any default security group.

Under Listeners and routing click on the link Create target group.

This opens a new window for you to create a new target group. Use the following details

Wordpress-TargetGroup as Traget group name
wordpress-workshopas VPC

Then, in the Health checks section, enter the following path:

/phpinfo.php In the next lab you will create the Launch Template for the web application servers which will create the phpinfo.php as part of the UserData script execution at instance boot. If health checks fail the User Data script has most likely failed.

Click on Next and then on Create target group without defining any targets.

Close the window, refresh the Listener to choose the created target group.

In the Summary section, review and click Create load balancer to create the load balancer.

Make a note of the DNS name created for your load balancer as you will need this in the following steps.

Create a launch Template

You have created a software-defined network across multiple fault-isolated Availability Zones, deployed a highly-available Aurora MySQL database and an EFS file system for shared storage. In this lab you will define the templates for the application servers running PHP as part of a scalable Wordpress installation.

Create a launch template for the Auto Scaling Groups (ASG)

A launch template is an instance configuration information, that allows you to create a saved instance configuration that can be used to launch instances at a later time. It includes the ID of the Amazon Machine Image (AMI), the instance type, a key pair, security groups, and other parameters used to launch EC2 instances. Additionally, it allows you to have multiple versions of a launch template.

Select Launch Templates on the left panel of your EC2 console , then click on Create launch template.

Give the Launch template the name WP-WebServers-LT

Choose the Amazon Linux AMI after selecting Quick Start in the Application and OS Images (Amazon Machine Image) section

Select the t3.micro instance type:

Don’t include a key pair and select the following Security groups to attach to the instances launched from this Launch Template:
WP Web Servers to allow connections from the Application Load Balancer
WP Database Clients to allow instances to connect to the Aurora MySQL DB
WP EFS Clients to allow instances to mount the NFS export of the EFS file system

Expand Advanced details and use the script below to populate the User Data field as text.

Update the variables in the Bash script below with the values from your environment.

EFS_FS_ID
This should be set to the file system ID of the Elastic Filesystem deployed in the previous lab. To obtain the file system ID visit the EFS console .
DB_NAME
This is the name of the database which Wordpress should use to store its data. If you entered the default values in Lab 2 this should be a value of wordpress. To confirm visit the details page for your RDS database and look for DB name under Configuraiton.
DB_HOST
This is the hostname of your database Writer instance. To obtain this visit the details page for your RDS database and look under Connectivity & Security. Use the Writer type instance hostname, a value such as wordpress-workshop.cluster-ctdnyvvewl6s.eu-west-1.rds.amazonaws.com.
DB_USERNAME
This will be the database username you specified in Lab 2. It can be found as Master username under Configuration on the details page for your RDS instance.
DB_PASSWORD
This is the password for the database user created in Lab 2.

    #!/bin/bash

    DB_NAME="wordpress"
    DB_USERNAME="wpadmin"
    DB_PASSWORD=""
    DB_HOST="wordpress-workshop.cluster-xxxxxxxxxx.eu-west-1.rds.amazonaws.com"
    EFS_FS_ID="fs-xxxxxxxxx"

    dnf update -y

    #install wget, apache server, php and efs utils
    dnf install -y httpd wget php-fpm php-mysqli php-json php amazon-efs-utils

    #create wp-content mountpoint
    mkdir -p /var/www/html/wp-content
    mount -t efs $EFS_FS_ID:/ /var/www/html/wp-content

    #install wordpress
    cd /var/www
    wget https://wordpress.org/latest.tar.gz
    tar -xzf latest.tar.gz
    cp wordpress/wp-config-sample.php wordpress/wp-config.php
    rm -f latest.tar.gz

    #change wp-config with DB details
    cp -rn wordpress/* /var/www/html/
    sed -i "s/database_name_here/$DB_NAME/g" /var/www/html/wp-config.php
    sed -i "s/username_here/$DB_USERNAME/g" /var/www/html/wp-config.php
    sed -i "s/password_here/$DB_PASSWORD/g" /var/www/html/wp-config.php
    sed -i "s/localhost/$DB_HOST/g" /var/www/html/wp-config.php

#change httpd.conf file to allowoverride
# enable .htaccess files in Apache config using sed command
sed -i '//,/<\/Directory>/ s/AllowOverride None/AllowOverride All/' /etc/httpd/conf/httpd.conf

# create phpinfo file
echo "<?php phpinfo(); ?>" > /var/www/html/phpinfo.php

# Recursively change OWNER of directory /var/www and all its contents
chown -R apache:apache /var/www

systemctl restart httpd
systemctl enable httpd

Review the final configuration under Summary and click Create launch template. You can disregard warnings about being able to SSH into the server and can also choose Proceed without keypair as you will not need to remotely access these servers.

Create the app server

In this lab you will use the load balancer and launch configuration from the previous 2 labs to create an auto scaling fleet of Wordpress application servers.

Create the ASG for the back-end web servers

Once you have created the launch configuration you can proceed to creating the Autoscaling group for the Wordpress web servers.
To do that select Auto Scaling Groups in the EC2 console, click on Create an Auto Scaling Group specify a name and select the previously created launch template:

In the next screen make sure that the wordpress-workshop VPC is selected, together with the Application Subnet A and Application Subnet B subnets for the web servers:

In the next screen Configure advanced options choose the option Attach to an existing load balancer and choose the target group you created earlier from the Existing load balancer target groups list.

In the Health checks section, make sure to Turn on Elastic Load Balancing health checks.

Click Next to configure group size and scaling policies with the following values:

In the Group size section, enter 2 in the Desired capacity field.
In the Scaling section, enter 2 as Min desired capacity and 4 as Max desired capacity
Select Target tracking scaling policy and enter 80 as Target value for the Average CPU utilization

Click through and accept the remaining defaults to complete the creation of Auto Scaling Group by clicking on Create Auto Scaling group.

The autoscaling group will now begin creating the desired number of EC2 instances based on the launch template you created. As the systems come online, the target group is updated with the instance details for your EC2 instances and the load balancer will begin distributing traffic across the instances. As instances are added or removed, the autoscaling group and load balancer will work in concert with one another to ensure that only healthy instances receive traffic.

When your targets are deemed healthy in your target group you can open the DNS name for your Application Load Balancer in your web browser to view your newly created Wordpress installation.

Next steps

You have now created a highly-available auto-scaling deployment of Wordpress that will scale in and out in response to client traffic hitting the website.

Clean Up

If you used your own account to follow this workshop, please perfom the actions below to remove all resources you created and stop incurring charges for them.

Delete the Auto Scaling Group

Go to the Auto Scaling Groups section of the EC2 Console, select the Autoscaling Group created in Lab 6, open the Actions **menu and select **Delete. To confirm deletion, type delete in the text field of the dialog that will open and click Delete.

Delete the Load Balancer

On the EC2 Console, go the Load Balancers section, select the Application Load Balancer created in Lab 4, open the Actions **menu and select **Delete load balancer. To confirm deletion, type confirm in the text field of the dialog that will open and click Delete.

Delete the Target Group

Go to the Target Groups section, select the Target Group created in Lab 4, open the Actions **menu and select **Delete. Click on Yes, delete on the dialog that will open.

Delete the Launch Template

Move to the Launch Templates section, select the Launch Template created in Lab 5, open the Actions menu **and select **Delete template.
To confirm deletion, type Delete in the field and click Delete

Verify instances

Once you delete the Auto Scaling Group, the instances will begin shutting down and eventually they will be terminated. Verify that all instances launched by the Auto Scaling Group have been terminated correctly.
You can use the aws:autoscaling:groupName attribute to filter instances launched by the Auto Scaling Group created in Lab 6.

Delete the Aurora cluster

Go to RDS console , select the wordpress-workshop ***Regional cluster *and click **Modify.
Scroll to the bottom of the page, unckeck Enable deletion protection and click on Continue

Select the option Apply immediately and click Modify cluster

On the RDS console , select the Reader instance, go to the Actions **menu and select **Delete. To confirm deletion, type delete me into the field and click Delete.

Now select the Writer instance, go to the **Actions* menu and select Delete.
To confirm deletion, type delete me into the field and click Delete.
You can delete the Regional cluster now. Select it, go to the Actions menu and select Delete.
Make sure to

uncheck Create final snapshot
check the acknowledgement

To confirm deletion, type delete me into the field and click on Delete DB cluster

Verify RDS Snapshots

Once the RDS Aurora Cluster has been completely deleted, move to the RDS Snapshots page, select the System tab and make sure no automated snapshots are present for the Aurora cluster created during the workshop.

Delete EFS filesystem

Go to the EFS Console , select the file system created in Lab 3 and click Delete.
Confirm the deletion by entering the file system’s ID in the dialog that will appear and click Confirm.

Delete VPC

Delete NAT Gateways

Go to the NAT gateways page of the VPC Console, select one NAT Gateway at a time, go to the Actions **menu and select **Delete NAT gateway.
To confirm deletion, type delete in the field and click Delete

Delete VPC

Select the wordpress-workshop VPC from Your VPCs page in the VPC Console. Go to the Actions **menu and select **Delete VPC.

You will be able to delete the VPC only if no network interfaces are still present in any of the subnets of the VPC. The dialog that will appear when you click on Delete VPC will show any remaining ENIs.

Click on the network interfaces link to check which services are still using the VPC and take appropriate actions.

Once all ENIs have been removed, you will be able to delete the VPC.
The dialog that will appear shows which resources will be deleted once you delete the VPC.
To confirm deletion, type delete in the field and click Delete

Release Elastic IPs

Go to the Elastic IPs page of the VPC Console, select all unassociated Elastic IPs (Association ID *value is -), open the **Actions **menu and select **Release Elastic IP addresses*.

Remove the IAM User

If you created the workshop-user IAM User to follow the workshop labs, make sure to delete it from the IAM Console logging in with a IAM User or IAM Role with the appropriate permissions.

Challenges Faced and Solutions

Database Connection Issues: When setting up RDS, there were some configuration issues with access control.
Solution: Adjusted VPC security groups and ensured that EC2 instances had the correct permissions to access RDS.
WordPress File Management: Managing WordPress media files on multiple instances was initially challenging.
Solution: Implemented Amazon EFS as a shared file system, enabling seamless media management across instances.

Conclusion

This AWS project demonstrates how to create a highly available, scalable WordPress environment, ideal for production-grade sites with high traffic. It showcases AWS’s capabilities in managing infrastructure to minimize manual intervention while maximizing uptime.

Explore my GitHub repository.

Let’s connect: Linkdin, Twitter, Github

Building a High-Availability Multi-Tier Web App on AWS with Amazon VPC, EC2, and Aurora RDS

Shubham Murti — Tue, 12 Nov 2024 09:15:38 +0000

Introduction

In this project, I developed a highly available, multi-tier, and fault-tolerant web application on AWS, focusing on uptime, scalability, and security - making it suitable for production use. This experience allowed me to work hands-on with essential AWS services, like Amazon VPC, Amazon EC2, Amazon Aurora, and Amazon S3, to build an architecture that provides high performance, resilience, and cost efficiency.

Tech Stack

Amazon VPC: Provides isolated networking environments for secure data flow.
Amazon EC2: Hosts scalable web and application server instances.
Amazon Aurora: A managed, high-performance relational database with automated failover.
Amazon S3: Stores and serves static content with durability and low latency.

Prerequisites

AWS Account: Required to access and configure all necessary AWS services.
AWS CLI: For managing resources, configurations, and deployment tasks.
Basic Networking Knowledge: Familiarity with networking concepts like subnets, load balancing, and security groups.
AWS Console Proficiency: Experience using the AWS Console for deploying and configuring services.

Problem Statement or Use Case

Problem: Traditional on-premises infrastructure often fails to meet the needs of applications requiring high availability and fault tolerance, especially under varying loads.

Solution: This project implements a multi-tier architecture with high availability and fault tolerance using AWS. By setting up a web application across multiple tiers (frontend, application logic, and database), the solution ensures seamless user experience, even in case of server failure or maintenance.

Real-World Relevance: The solution suits production environments where uptime is crucial, such as e-commerce platforms, content-driven websites, and customer-facing applications. The architecture can dynamically adjust resources to accommodate fluctuating traffic, making it scalable and cost-effective.

Architecture Diagram

Below is a high-level overview of the architecture used:

Component Breakdown

Amazon VPC: Provides network isolation, enabling private and public subnets to securely route traffic between the internet and internal services.
Amazon EC2: Hosts web and application server instances, with auto-scaling groups to dynamically adjust resources as traffic demands change.
Load Balancer: Manages incoming requests and distributes them to healthy EC2 instances across different availability zones, ensuring high availability.
Amazon Aurora: A managed relational database that automatically replicates data and performs failovers, providing a resilient storage solution.
Amazon S3: Stores and delivers static content like images, CSS, and JavaScript files, reducing load on EC2 instances and improving performance.

Step-by-Step Implementation

Network — Amazon VPC

Create VPC

Amazon Virtual Private Cloud (Amazon VPC) allows you to start AWS resources with a user-defined virtual network. This virtual network, along with the benefits of using AWS’s scalable infrastructure, is very similar to the existing network operating in the customer’s own data center.

Move on to VPC service

After logging in to the AWS console, select VPC from the service menu.

If the screenshot below is different from the screen that you’re viewing, enable the New VPC Experience toggle to active.

Create VPC through VPC Wizard

Select VPC Dashboard and click Create VPC to create your own VPC.

To create a space to provision AWS resources used in this lab, we will create a VPC and Subnets. Select VPC and more in Resource to create tab and change name tag to VPC-Lab. Leave the default setting for IPv4 CIDR block.

It is a best practice to deploy resources across multiple Availability Zones for high availability and fault tolerance.

To design high availability architecture, we create 2 subnet space and select 2a and 2c for Customize AZs. And set the CIDR value of the public subnet that can communicate directly with the Internet as shown in the screen below. Set the CIDR value of the private subnet as shown in the screen.

You can use a NAT gateway so that instances in your private subnets can connect to services outside your VPC, but external services cannot initiate direct connections to these instances. In this lab, we will create a NAT gateway in only one Availability Zone to save cost. Also, for DNS options, enable both DNS hostnames and DNS resolution. After confirming the setting value, click the Create VPC button.

As the VPC is created, you can see the process of creating network-related resources as shown in the screen below. For NAT Gateway, provisioning may take longer compared to other resources.

You can check the information of the created VPC. Check related information such as CIDR value, route table, network ACL, etc. Check that the values you just set are correct.

Architecture Configured So Far

If VPC is completed through the VPC Wizard, the environment configured so far is as follows.

Challenges Faced and Solutions

Cross-AZ Latency: Replicating data across availability zones resulted in some latency in data access.
Solution: Used Aurora’s automated cross-region replication to reduce latency while maintaining data consistency.
Auto Scaling Configuration: Initially faced challenges with EC2 instances not scaling back down after load reduction.
Solution: Adjusted Auto Scaling policies to ensure smoother scaling transitions, keeping resource usage cost-effective.

Create VPC Endpoint

In this section, you create an endpoint for S3 to learn a VPC endpoint. Skip to do this step will not affect your progress to the next lab.

VPC Endpoint

In VPC Dashboard, select Endpoints. Click Create endpoint button.

Type s3 endpoint for name and select AWS services in Service category tab. In the search bar below, type s3 and select the list at the top.

For S3 VPC endpoints, there are gateway types and interface types. For this lab, select the gateway type. And for the deployment location, select the VPC-Lab-vpc created in this lab.

Choose a route table to reflect the endpoint. Select the two private subnets as shown below. Additional routing information for using the endpoint is automatically added to the selected route table.

You can also configure policies to control access to endpoints as shown below.

You can use VPC endpoint policies to allow full access to AWS services or create custom policies. Check Use VPC endpoint policies

Confirm that the route to access Amazon S3 through the gateway endpoint has been automatically added to the private route table specified earlier.

VPC endpoints are communications within the AWS network and have the security and compliance advantage of being able to control traffic through the endpoints. You can also optimize the data processing cost if you transfer your data through a VPC endpoint rather than a NAT gateway.

In this section, you created a S3 gateway endpoint to allow private S3 access from within the VPC without needing an internet gateway. This keeps S3 traffic private within the AWS network.

Compute — Amazon EC2

Launch a web server instance

This chapter starts with the default Amazon Linux instance and lets you automatically configure the Apache/PHP Web server during initial step.

Launch instance and connect to web service

In the AWS console search bar, type EC2 and select it. Then click EC2 Dashboard **at the top of the left menu. Press the **Launch instance **button and select **Launch instance from the menu.

In Name, put the value Web server for custom AMI. And check the default setting in Amazon Machine Image below.

Select t2.micro in Instance Type.

For Key pair, choose Proceed without a key pair.

Click the Edit button in Network settings to set the space where EC2 will be located.

And choose the VPC-Lab-vpc created in the previous lab, and for the subnet, choose public subnet. Auto-assign public IP is set to Enable.

Right below it, create Security groups to act as a network firewall. Security groups will specify the protocols and addresses you want to allow in your firewall policy. For the security group you are currently creating, this is the rule that applies to the EC2 that will be created. After entering Immersion Day - Web Server in Security group name and Description, select Add Security group rule and set Type to HTTP. Also allow TCP/80 for Web Service by specifying it. Select My IP in the source.

It is a best practice to configure security groups following the principle of least privilege, allowing only the minimum required traffic.

All other values accept the default values, expand by clicking on the Advanced Details tab at the bottom of the screen.

Click the Meta Data version dropdown and select V2 only (token required)

Enter the following values in the User data field and select Launch instance.

#!/bin/sh

#Install a LAMP stack
dnf install -y httpd wget php-fpm php-mysqli php-json php php-devel
dnf install -y mariadb105-server
dnf install -y httpd php-mbstring

#Start the web server
chkconfig httpd on
systemctl start httpd

#Install the web pages for our lab
if [ ! -f /var/www/html/immersion-day-app-php7.zip ]; then
   cd /var/www/html
   wget -O 'immersion-day-app-php7.zip' 'https://static.us-east-1.prod.workshops.aws/public/2e449d3a-fc13-44c9-8c99-35a37735e7f5/assets/immersion-day-app-php7.zip'
   unzip immersion-day-app-php7.zip
fi

#Install the AWS SDK for PHP
if [ ! -f /var/www/html/aws.zip ]; then
   cd /var/www/html
   mkdir vendor
   cd vendor
   wget https://docs.aws.amazon.com/aws-sdk-php/v3/download/aws.zip
   unzip aws.zip
fi

# Update existing packages
dnf update -y

User Data is a user-defined initialization script that is executed when the first instance is created.

Information indicating that the instance creation is in progress is displayed on the screen. You can view the list of EC2 instances by selecting View Instances in the lower right corner.
After the instance configuration is complete, you can check the Availability Zone in which the instance is running, and externally accessible IP and DNS information.

Wait for the instance’s Instance state result to be Running. Open a new web browser tab and enter the Public DNS or IPv4 Public IP of your EC2 instance in the URL address field. If the page is displayed as shown below, the web server instance is configured normally.
If you are using the Chrome web browser, when you attach the Public IPv4 DNS value to the web browser, if it does not run, https may be automatically added in front of the DNS value, so it may not run. Therefore, it is recommended to enter http://.

Access the web service

Go to the EC2 instance console. Select the instance you want to connect to and click the Connect button in the center.

In the Connect your instance window, select the EC2 Instance Connect tab, then click the Connect button in the lower right corner.

After a while, you can use the browser-based SSH console as shown below. Just close the window after the CLI test.

Connect to the Linux instance using Session Manager

You must click the Access your Linux instance using Session Manager link below to proceed with the exercise.

In the database lab to be followed, we connect to RDS database using the IAM role granted to the web server. Therefore, refer to Accessing Linux instance using Session Manager to assign IAM role to EC2 instance and connect to your Linux instance using Session Manager

Create a custom AMI

In the AWS EC2 console, you can create an Custom AMI to meet your needs. This can then be used for future EC2 instance creation. In this page, let’s create an AMI using the web server instance that we built earlier.

In the EC2 console, select the instance that we made earlier in this lab, and click Actions > Image and templates > Create Image.

In the Create Image console, type as shown below and press Create image to create the custom image.

Verify in the console that the image creation request in completed.
In the left navigation panel, Click the AMIs button located under IMAGES. You can see that the Status of the AMI that you just created. It will show either Pending or Available.

Terminate the instance

Custom AMI (Golden Image) creation has been completed for the auto scaling by using the EC2 instance you just created. Therefore, the EC2 instance currently running is no longer needed, so let’s try to terminate it. ( In Deploy auto scaling web service, we will use custom AMI to create a new web server.)

Do not terminate the “Web server for custom AMI” Instance until the AMI creation process is fully completed. Ensure the AMI status shows as Available before proceeding.

In the left navigation panel of the EC2 dashboard, select Instances. Then select the instance that should be deleted. From there, click Instance state -> Terminate instance.

When the alert message appears, click Terminate to delete.

The instance status changes to Shutting down. After that, the instance status turned to terminated. The instance deletion is complete. You may see the instance for a short period of time for deletion logging.

Architecture Configured So Far

If you mark the resources that have been configured so far in conceptual terms, it is same with the picture below.

Congratulations! You have successfully created a Custom AMI (Golden Image) using the EC2 web server, which can be utilized for deploying an auto-scaling web service in the next section.

Deploy auto scaling web service

Using the network infrastructure created in the Network- AMazon VPC lab, we will deploy a web service that can automatically scale out/in under load and ensure high availability. We use the web server AMI created in the previous chapter and the network infrastructure named VPC-Lab.

Configure Application Load Balancer

AWS Elastic Load Balancer supports three types of load balancers: Application Load Balancer, Network Load Balancer, and Gateway Load Balancer. In this lab, you will configure and set up the Application Load Balancer to handle load balancing HTTP requests.

From the EC2 Management Console in the left navigation panel, click Load Balancers under Load Balancing. Then click Create Load Balancer. In the Select load balancer type, click the Create button under Application Load Balancer.

Name the load balancer. In this case, name Name as Web-ALB. Leave the other settings at their default values.

It is a best practice to deploy resources across multiple Availability Zones for fault tolerance and high availability.

Scrolling down a little bit, there is a section for selecting availability zones. First, Select the VPC-Lab-vpc created previously. For Availability Zones select the 2 public subnets that were created previously. This should be Public Subnet for ap-northeast-2a and Public Subnet C for ap-northeast-2c.

In the Security groups section, click the Create new security group hyperlink. Enter web-ALB-SG as the security group name and check the VPC information. Scroll down to modify the Inbound rules. Click the Add rule button and select HTTP as the Type and Anywhere-IPv4 as the Source. And create a security group.

Return to the load balancer page again, click the refresh button, and select the web-ALB-SG you just created. Remove the default security group.

In Listeners and routing column, click Create target group. Put Web-TG for Target group name and check all settings same with the screen below. After that click Next button.

This is where we would register our instances. However, as we mentioned earlier, there are not instances to register at this moment. Click Create target group.

Again, move into the Load balancers page, click refresh button and select Web-TG. And then Click Create load balancer.

Configure launch template

Now that ALB has been created, it’s time to place the instances behind the load balancer. To configure an Amazon EC2 instance to start with Auto Scaling Group, you can use Launch Template, Launch Configuration, or EC2 Instance. In this workshop, we will use the Launch Template to create an Auto Scaling group.

The launch template configures all parameters within a resource at once, reducing the number of steps required to create an instance. Launch templates make it easier to implement best practices with support for Auto Scaling and spot fleets, as well as spot and on-demand instances. This helps you manage costs more conveniently, improve security, and minimize the risk of deployment errors.

The launch template contains information that Amazon EC2 needs to start an instance, such as AMI and instance type. The Auto Scaling group refers to this and adds new instances when a scaling out event occurs. If you need to change the configuration of the EC2 instance to start in the Auto Scaling group, you can create a new version of the launch template and assign it to the Auto Scaling group. You can also select a specific version of the launch template that you use to start an EC2 instance in the Auto Scaling group, if necessary. You can change this setting at any time.

Create security group

Before creating a launch template, let’s create a security group for the instances created through the launch template to use.

From the left navigation panel of the EC2 console, select Security Groups under the Network & Security heading and click Create Security Group in the upper right corner.

Scroll down to modify the Inbound rules. First, select the Add rule button to add the Inbound rules, and select HTTP in the Type. For Source, type ALB in the search bar to search for the security group created earlier Web-ALB-SG. This will configure the security group to only receive HTTP traffic coming from ALB.

Leave outbound rules’ default settings and click Create Security Group to create a new security group. This creates a security group that allows traffic only for HTTP connections (TCP 80) that enter the instance via ALB from the Internet.

Create launch template

In the EC2 console, select Launch Templates from the left navigation panel. Then click Create Launch Template.

Let’s proceed with setting up the launch template step by step. First, set Launch template name and Template version description as shown below, and select Checkbox for Provide guidance in Auto Scaling guidance. Select this checkbox to enable the template you create to be utilized by Amazon EC2 Auto Scaling.

Scroll down to set the launch template contents. In Amazon Machine Image(AMI), set the AMI to Web Server v1, which was created in the previous EC2 lab. You can find it by typing Web Server v1 in the search section, or you can scroll down to find it in the My AMI section. Next, select t2.micro for the instance type. We are not going to configure SSH access because this is only for Web service server. Therefore, we do not use key pairs.

Leave the other parts as default. Let’s take a look at the Network Settings section. In security group dropdown, find and apply ASG-Web-Inst-SG created before.

Follow the Storage’s default values without any additional change. Go down and define the Instance tags. Click Add tag and Name for Key and Web Instance for Value. Select Resource types as Instances and Volumes.

Finally, in the Advanced details tab, set the IAM instance profile to SSMInstanceProfile. If IAM role was not created earlier, refer Create an IAM instance profile for Systems Manager to create the SSMInstanceProfile IAM role.

Leave all other settings as default, and click the Create launch template button at the bottom right to create a launch template.

After checking the values set in Summary on the right, click Create launch template to create a template.

Set Auto Scaling Group

Now, let’s create the Auto Scaling Group.

Enter the EC2 console and select Auto Scaling Groups at the bottom of the left navigation panel. Then click the Create Auto Scaling group button to create an Auto Scaling Group.

In [Step 1: Choose launch template or configuration], specify the name of the Auto Scaling group. In this workshop, we will designate it as Web-ASG. Then select the launch template that you just created named Web. The default settings for the launch template will be displayed. Confirm and click the lower right Next button.

Next, proceed to set up load balancing. First, select Attach to an existing load balancer. Then in Choose a target group for your load balancer, select Web-TG created during in ALB creation. At the Monitoring, select Check box for Enable group metrics collection within CloudWatch. This allows CloudWatch to see the group metrics that can determine the status of Auto Scaling groups. Click the Next button at the bottom right.

In the step of Configure group size and scaling policies, set scaling policy for Auto Scaling Group. In the Group size column, specify Desired capacity and Minimum capacity as 2 and Maximum capacity as 4. Keep the number of the instances to 2 as usual, and allow scaling of at least 2 and up to 4 depending on the policy.

In the Scaling policies section, select Target tracking scaling policy and type 30 in Target value. This is a scaling policy for adjusting the number of instances based on the CPU average utilization remaining at 30% overall. Leave all other settings as default and click the Next button in the lower right corner.

We will not Add notifications. Clcik the Next button to move to the next step. In the Add tags step, we will simply assign name tag. Click Add tag, type Name in Key, ASG-Web-Instance in Value, and then click Next.

Now we are in the final stage of review. After checking the all settings, click the Create Auto Scaling Group button at the bottom right.
Auto Scaling group has been created. You can see the Auto Scaling group created in the Auto Scaling group console as shown below.

Instances created through the Auto Scaling group can also be viewed from the EC2 Instance menu.

Architecture Configured So Far

Now, we’ve built a web service that is high available and automatically scales under load! The configuration of the services we have created so far is as follows.

Congratulations! You successfully deployed a scalable and highly available web service using an Application Load Balancer, security groups, launch template, and Auto Scaling group.

Check web service and test

Now, let’s test the service you have configured for successful operation. First, let’s check whether you can access the website normally and whether the load balancer works, and then load the web server to see if Auto Scaling works.

Check web service and load balancer

To access through the Application Load Balancer configured for the web service, click the Load Balancers menu in the EC2 console and select the Web-ALB you created earlier. Copy DNS name from the basic configuration.

Open a new tab in your web browser and paste the copied DNS name. You can see that web service is working as shown below. For the figure below, you can see that the web instance placed in ap-northeast-2a is running this web page.

If you click the refresh button here, you can see that the host serving the web page has been replaced with an instance of another availability zone area (ap-northeast-2c) as shown below. This is because routing algorithms in ALB target groups behave Round Robin by default.

Currently, in the the Auto Scaling group, scaling policy’s baseline has been set to 30% CPU utilization for each instance.

If the average CPU utilization of an instance is less than 30%, Reduce the number of instances.
If the average CPU utilization of an instance is over 30%, Additional instances will be deployed, load will be distributed, and adjusted to ensure that the average CPU utilization of the instances is 30%.

Now, let’s test load to see whether Auto Scaling works well. On the web page above, click the LOAD TEST menu. The web page changes and the applied load is visible. Click on the logo at the top left of the page to see that each instance is under load.

Before load:

The principle that causes CPU load is that when the CPU Idle value is over 50, the PHP code operates every five seconds to create, compress, and decompress arbitrary files. Traffic is distributed and operated by the ALB, so the load is applied to other instances continuously.

Enter Auto Scaling Groups from the left side menu of the EC2 console and click the Monitoring tab. Under Enabled metrics, click EC2 and set the right time frame to 1 hour. If you wait for a few seconds, you’ll see the CPU Utilization (Percent) graph changes.

Wait for about 5 minutes (300 seconds) and click the Activity tab to see the additional EC2 instances deployed according to the scaling policy.
When you click on the Instance management tab, you can see that two additional instances have sprung up and a total of four are up and running.
If you use the ALB DNS that you copied earlier to access and refresh the web page, you can see that it is hosting the web page in two instances that were not there before. The current CPU load is 0% because it is a new instance. It can also be seen that each of them was created in a different availability zone. If it’s not 0%, it can look more than 100% because it’s a constant load situation.

So far, we’ve checked that Auto Scaling group is working through a load test on the web service. If the page that causes the CPU load is working, close the page to prevent additional load.

Database — Amazon Aurora

Create VPC security group

The RDS service uses the same security model as EC2. The most common usage format is to provide data as a database server to an EC2 instance operating as an applicatiojn server within the same VPC, or to configure it to be accessible to the DB Application client outside of the VPC. The VPC Security Group must be applied for proper access control.

In the previous Compute — Amazon EC2 lab, we created web server EC2 instances using Launch Template and Auto Scaling Group. These instances use Launch Template to apply the security group ASG-Web-Inst-SG . Using this information, we will create a security group so that only web server instances within the Auto Scaling Group can access RDS instances.

On the left side of the VPC dashboard, select Security Groups and then select Create Security Group.
Enter Security group name and Description as shown below. Choose the VPC that was created in the first lab. It should be named VPC-Lab.

Following the principle of least privilege, it is a best practice to allow inbound traffic to your database only from trusted sources, such as your application servers.

Scroll down to the Inbound rules column. Click Add rule to create a security group policy that allows access to RDS from the EC2 Web servers that you previously created through the Auto Scaling Group. Under Type, select MySQL/Aurora The port range should default to 3306. The protocol and port ranges are automatically specified. The Source type entry can specify the IP band (CIDR) that you want to allow acces to, or other security groups that the EC2 instances to access are already using. Select the security group(named ASG-Web-Inst-SG ) that is applied to the web instances of the Auto Scaling group in the Compute — Amazon EC2

When settings are completed, click Create Security Group at the bottom of the list to create this security group.

Create RDS instance

Since the security group that RDS will use has been created, let’s create an instance of RDS Aurora (MySQL compatible).

In the AWS Management console, go to the RDS (Relational Database Service) .

Select Create Database in dashboard to start creating a RDS instance.

You want to select the RDS instances’ database engine. In Amazon RDS, you can select the database engine based on open source or commercial database engine. In this lab, we will use Amazon Aurora with MySQL-compliant database engine. Select Standard Create in the choose a database creation method section. Set Engine type to Aurora (MySQL Compatible), Set Version to Aurora (MySQL 5.7) 2.11.4.

Select Production in Template. Under Settings, we want to specify administrator information for identifying the RDS instances. Enter the information as it appears below.

For production workloads, it is a best practice to enable high availability and fault tolerance by creating read replicas in different Availability Zones.

Under DB instance size select Memory Optimized class. Under Availability & durability select Create an Aurora Replica or reader node in a different AZ. Select db.r5.large for instance type.

It is a best practice to deploy databases within a private subnet of a VPC for better security and network isolation

Set up network and security on the Connectivity page. Select the VPC-Lab that you created earlier in the Virtual private cloud (VPC) and specify the subnet that the RDS instance will be placed in, public access, and security groups. Enter the information as it appears below.

Scroll down and click Additional configuration. Set database options as shown below. Be aware of the uppercase and lowercase letters of Initial database name.

Subsequent items such as Backup, Entry, Backtrack, Monitoring, and Log exports all accept the default values, and press Create database to create a database.
A new RDS instance is now creating. This may take more than 5 minutes. You can use an RDS instance when the DB instance’s status changed to Available.

Architecture Configured So Far

The configuration of the services we have created so far is as follows.

Connect RDS with Web App server

The Web Server instance that you created in the previous computer lab contains code that generates a simple address book to RDS. The Endpoint URL of the RDS must be verified first in order to use the RDS on the EC2 Web Server.

Storing RDS Credentials in AWS Secrets Manager

The web server we built includes sample code for our address book. In this lab, you specify which database to use in the sample code and how to connect it. We will store that information in AWS Secrets Manager.

In this chapter, we will create a secret containing data connection information. Later, we will give the web server the appropriate permission to retrieve the secret.

In the console window, open AWS Secrets Manager (https://console.aws.amazon.com/secretsmanager/ ) and click the Store a new secret button.

It is a best practice to store database credentials and other sensitive information securely using AWS Secrets Manager, instead of hard-coding them in application code.

Under Secret Type, choose Credentials for Amazon RDS database. Write down the user name and password you entered when creating the database. And under Database select the database you just created. Then click the Next button.

Name your secret, mysecret. The sample code is written to ask for the secret by this specific name. Click Next.

Leave Secret rotation at default values. Click Next.

Review your choices. Click Store.

You can check the list of secret values with the name mysecret as shown below.

Click mysecret hyperlink and find Secret value tab. And click Retrieve secret value button.

Click Edit button, and check whether there is dbname and immersionday in key/value section. If they were not, click Add button, fill out the value and click save button.

Access RDS from EC2

Now that you have created a secret, you must give your web server permission to use it. To do this, we will create a Policy that allows the web server to read a secret. We will add this policy to the Role you previously assigned to the web server.

Allow the web server to access the secret

To follow the principle of least privilege, it is a best practice to grant the minimum required permissions to resources. In this case, you will grant permissions for the web server instances to access the specific secret containing the database credentials.

Sign in to the AWS Management Console and open the IAM console . In the navigation pane, choose Policies, and then choose Create Policy.

Click Choose a service.

Type Secrets Manager into the search box. Click Secrets Manager.

Under Access level, click on the carat next to Read and then check the box by GetSecretValue.

Click on the carat next to Resources. For this lab, select All resources. Click Next: Tags.

For the lab, we’re allowing EC2 to access all secrets. With a real workload, you should consider allowing access to specific secrets.

Click Next: Review.

On the Review Policy screen, give your new policy the name ReadSecrets. Click Create policy.

In the navigation pane, choose Roles and type SSMInstanceProfile into the search box. This is the role you created previously in Connect to your Linux instance using Session Manager. Click SSMInstanceProfile.

Under Permissions policies, click Attach policies.

Search for the policy you created called ReadSecrets. Check the box and click Attach policy.

Under Permissions policies, verify that AmazonSSMManagedInstanceCore and ReadSecrets are both listed.

Try the Address Book

Access the EC2 Console window and click load balancer. After copying the DNS name of the load balancer created in the compute lab, open a new tab in your browser and paste it.

After connecting to the web server, go to the RDS tab.

Now you can check the data in the database you created.

This is a very basic exercise in interacting with a MySQL database managed by AWS. RDS can support much more complex relational database scenarios, but hopefully this simple example will make the point clear. You are free to add/edit/delete content from the RDS database using the Add Contact, Edit and Remove links in the address book.

Architecture Configured So Far

Now, with the work done so far, you have built a web service with guaranteed high availability. The infrastructure architecture we have constructed so far is as follows.

RDS Management Features

In multiple AZ deployments, Amazon RDS automatically provisions and maintains synchronous spare replicas in different availability zone. The default DB instance is synchronized from the availability zone to the spare replica to provide data redundancy.

RDS Failover Tests

When multiple AZs are enabled, Amazon RDS automatically switches to a spare replica in another availability zone if the DB instance has a planned or unplanned outage. The amount of time that failover takes to complete depends on the database activity and other conditions when the default DB instance becomes unavailable. The time required for failover is typically 60–120 seconds. However, if the transaction is large or the recovery process is complex, the time required for failover can be increased. When failover is complete, the RDS console UI takes additional time to reflect in the new availability zone.

From the RDS management console, select Databases, select the instance that you want to proceed with the failover, and click Failover in the task menu.

A message asking whether you’re going to failover the rdscluster. Press the Failover button.

The refresh button changes the status of rdscluster in the DB identifier to Failing-over. In a few minutes, press the Refresh button to see Reader and Writer roles changed. The failover is complete.

Create RDS Snapshot

Let’s take a snapshot of the RDS in production. Snapshot can be created at any frequency for backup to database instances, and the database can be restored at any time based on the snapshots created.

From the RDS management console, select Databases, and Select the instance on which you want to perform the snapshot operation. Select Actions > Take snapshot in the upper right corner.

Type the name you want to use for the snapshot as immersionday-snapshot. Press the Take Snapshot button to complete the creation.

From the left RDS menu, select Snapshots and check the creation status of the snapshot. The state of the snapshot is the first creating state, and you can use that snapshot to restore the database when state become available. To restore, select the snapshot and select Actions to see what you can do with that snapshot. Restore Snapshot allows you to create RDS instances with the same data based on snapshots taken. This lab will not perform a restore.

Change RDS Instance Type

Scale-Up/Scale-Down of RDS instances can be done very simply through the RDS Management Console.

Let’s change the specification of the RDS instance by selecting the instance you want to change and clicking Modify.
You can select the specification of the instance that you want to change by selecting the list box of instance classes. Let’s choose db.r6g.large here.
Scroll to the bottom and select Continue to go to the page where you check the instance’s current value and new value and select when to apply.
Select Apply immediately. In this case, RDS changes its instance immediately after perform a back up task. Then click Modify DB Instance. Depending on the type of instance and the amount of data to back up, it can take several minutes. Therefore, you should expect a certain amount of downtime for RDS services(Redundant configuration minimizes downtime).

In case of selecting Apply during the next scheduled maintenance window, make the change in the user’s Maintenance Window, which is specified on a weekly basis.

You can see that the status of the instance has changed to Modifying.
When you click refresh button again, you can see that the Writer instance has changed. This is because the instance you selected earlier for the size change was the Writer instance. RDS minimizes downtime through failover before resizing. If you wait a moment, you will see that the change to Available status has been completed as shown below.

RDS can change the size of the instance at any time. However, the size of the database does not support shrink after scaling up.

Connect RDS Aurora

Let’s try to make an RDS connection through MySQL CLI, which is used for general database management/operation.

To do this,

Create an EC2 instance with the AMI created in Public Subnet within the VPC-Lab. The networking option should allow Public IP.
Changes the security group settings for RDS Aurora. Configure the newly created EC2 instance to accept security groups as sources.
Log in to the EC2 instance you just created with SSH, and connect to RDS Aurora through the MySQL Client. The EC2 web server already has MySQL client installed during EC2 deployment.

Organizing the above items will be a challenge. Once the setup is successful, you can connect to the CLI environment and perform mysql commands as shown below.

$ ssh -i AWS-ImmersionDay-Lab.pem ec2-user@”EC2 Host FQDN or IP”
Last login: Sun Feb 18 14:41:59 2018 from 112.148.83.236

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2017.09-release-notes/


$ mysql -u awsuser -pawspassword -h awsdb.ccjlcjlrtga1.ap-northeast-2.rds.amazonaws.com

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 34
Server version: 5.6.10 MySQL Community Server (GPL)


Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| immersionday       |
| mysql              |
| performance_schema |
+--------------------+
4 rows in set (0.01 sec)

mysql> use immersionday;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+------------------------+
| Tables_in_immersionday |
+------------------------+
| address                |
+------------------------+
1 row in set (0.01 sec)

mysql> select * from address;
+----+-------+--------------+---------------------+
| id | name  | phone        | email               |
+----+-------+--------------+---------------------+
|  1 | Bob   | 630-555-1254 | bob@fakeaddress.com |
|  2 | Alice | 571-555-4875 | alice@address2.us   |
+----+-------+--------------+---------------------+
2 rows in set (0.00 sec)

mysql>

Storage — Amazon S3

Create Bucket on S3

All objects in Amazon S3 are stored within a bucket. You must create a Bucket before storing data on Amazon S3.

Create Bucket

From the AWS Management Console, connect to S3 . Press Create bucket to create a bucket.

It is a best practice to use S3 Bucket Policy and Access Control Lists (ACLs) to control access to your S3 buckets and objects, following the principle of least privilege.

Enter a unique bucket name in the Bucket name field. For this lab, type immersion-day-user_name, substituiting user-name with your name. All bucket names in Amazon S3 have to be unique and cannot be duplicated. In the Region drop-down box, specify the region to create the bucket. In this lab, select the region closest to you. The images will show the Asia Pacific (Seoul) region. Object Ownership change to ACLs enabled. Bucket settings for Block Public Access use default values, and select Create bucket in the lower right corner.

Bucket names must comply with these rules:

Can contain lowercase letters, numbers, dots (.), and dashes (-).
Must start with a number or letter.
Can be specified from a minimum of 3 to a maximum of 255 characters in length.
Cannot be specified in the format like the IP address (e.g., 265.255.5.4).

There may be additional restrictions depending on the region in which the bucket is created. The name of the bucket cannot be changed once it is created and is included in the URL to specify objects stored within the bucket. Please make sure that the bucket you want to create is named appropriately.

A bucket has been created on Amazon S3.

There are no costs incurred for creating bucket. You pay for storing objects in your S3 buckets. The rate you’re charged depends on the region you are using, your objects’ size, how long you stored the objects during the month, and the storage class. There are also per-request fees. Click for more information

Adding objects to buckets

If the bucket has been created successfully, you are ready to add the object. Objects can be any kind of file, including text files, image files, and video files. When you add a file to Amazon S3, you can include information about the permissions and access settings for that file in the metadata.

Adding objects for static Web hosting

This lab hosts static websites through S3. The static website serves as a redirect to an instance created by the VPC Lab when you click on a particular image. Therefore, prepare one image file, one HTML file, and an ALB DNS name.

Download the image file aws.png and save it as aws.png.
1. Write index.html using the source code below.

<html>
        <head>
            <meta charset="utf-8">
            <title> AWS General Immersion Day S3 HoL </title>
        </head>
        <body>
            <center>
            <br>
            <h2> Click image to be redirected to the EC2 instance that you created </h2>
            <img src="{{Replace with your S3 URL Address}}" onclick="window.location='DNS Name'"/>
            </center>
        </body>
    </html>

Upload the aws.png file to S3. Click S3 Bucket that you just created.

Click the Upload button. Then click the Add files button. Select the pre-downloaded aws.png file through File Explorer. Alternatively, place the file in Drag and Drop to the screen.

Check the file information named aws.png to upload, then click the Upload button at the bottom.

Check the URL information to fill in the image URL in index.html file. Select the uploaded aws.png file and copy the Object URL information from the details on the right.

Paste Object URL into the image URL part of the index.html. Then specify the ALB DNS Name of the load balancer created by Deploy auto scaling web service to redirect to ALB when you click on the image.

Upload the index.html file to S3 following the same instructions as you did to upload the image.

If you check the objects in your S3 bucket, you should see 2 files.

Congratulations! You have successfully created an S3 bucket and uploaded objects into it.

View objects

Now that you’ve added an object to your bucket, let’s check it out in your web browser.

View Objects

In the Amazon S3 Console, please click the object you want to see. You can see detailed information about the object as shown below.

By default, all objects in the S3 bucket are owner-only(Private). To determine the object through a URL of the same format as https://{Bucket}.s3.{region}.amazonaws.com/{Object}, you must grant Read permission for external users to read it. Alternatively, you can create a signature-based Signed URL that contains credentials for that object, allowing unauthorized users to access it temporarily.

Return to the previous page and select the Permissions tab in the bucket. To modify the application of Block public access (bucket settings), press the right Edit button.

Uncheck box and press the Save changes button.

Enter confirm in the bucket's Edit Block public access pop up window and press the Confirm button.

Click the Objects tab, select the uploaded files, click the Action drop-down button, and press the Make public button to set them to public.

When the confirmation window pops up, press the Make public button again to confirm.

It is a best practice to periodically review and audit the permissions and access settings for your S3 buckets and objects to ensure they align with your security requirements and the principle of least privilege.

Return to the bucket page, select index.html, and click the Object URL link in the Show Details entry.

When you access the HTML object file object URL, the following screen is printed.

When you click on an image, it is redirected to the instance’s web page you created.

Enable Static Web Site Hosting

You can use Amazon S3 to host static websites.

Static Web Site Settings

A static website refers to a website that contains static content (HTML, image, video) or client-side scripts (Javascript) on a web page. In contrast, dynamic websites require server-side processing, including server-side scripts such as PHP, JSP, or ASP.NET. Server-side scripting is not supported on Amazon S3. If you want to host a dynamic website, you can use other services such as EC2 on AWS.

In the S3 console, select the bucket you just created, and click the Properties tab. Scroll down and click the Edit button on Static website hosting.

Activate the static website hosting function and select the hosting type and enter the index.html value in the Index document value, then click the save changes button.

Click Bucket website endpoint created in the Static website hosting entry to access the static website.

This allows you to host static websites using Amazon S3.

Move objects

You have seen the ability to add objects to buckets and verify them so far. Now, let’s see how we can move objects to different buckets or folders.

Move Objects

Create a temporary bucket for moving objects between buckets (Bucket name: immersion-day-myname-target). Substitute myname with your name. Rememeber the naming rules for the bucket. Block all public access Uncheckbox for quick configuration.

Check the notification window below and select Create bucket.

In the Amazon S3 Console, select the bucket that contains the object (the first bucket you created) and click the checkbox for the object you want to move. Select the Actions menu at the top to see the various functions you can perform on that object. Select Move from the listed features.

Select the destination as bucket, then click the Browse S3 button to find the new bucket you just created.

Click the bucket name in the pop-up window, then select the destination (arrival) bucket. Click the Choose destination button.

Check that the object has moved to the target bucket.

Even though you move an object, its existing permissions remain intact.

Enable Bucket versioning

You can use Bucket Versioning if you want to update existing files to the latest version within the same bucket, but still want to keep the existing version.

It is a best practice to enable versioning on your S3 buckets to protect against accidental deletion or overwrites of objects, and to maintain a history of changes to your data.

Enable versioning

In the Amazon S3 Console, select the first S3 bucket we created. Select the Properties menu. Click the Edit button in Bucket Versioning.

Click the enable radio button on Bucket Versioning, then click Save changes.

In this lab, the index.html file will be modified and re-uploaded with the same name. Make some changes to the index.html file. Then upload the modified file to the same S3 bucket.
When the changed file is completely uploaded, click the object in the S3 Console. You can view current version information by clicking the Versions tab on the page that contains object details.

Congratulations on your progress! You’ve successfully learned how to add and verify objects in Amazon S3 buckets, move objects between buckets or folders, and utilize bucket versioning to update files while preserving existing versions.

Deleting objects and buckets

You can delete unnecessary objects and buckets to avoid unnecessary costs.

In the Amazon S3 Console, select the Bucket that you want to delete. Then click Delete. A dialog box appears for deletion.

There is a warning that buckets cannot be deleted because they are not empty. Select empty bucket configuration to empty buckets.

Empty bucket performs a one-time deletion of all objects in the bucket. Confirm by typing permanently delete in the box. Then click the Empty button.

Now the bucket is empty. Perform task 1 again. Enter a bucket name and press the Delete bucket button.

Congratulations!! You have completed all the workshop. Thank you for your efforts.

Clean up resource

If you participated in an AWS event using an AWS-provisioned account, no cleanup is necessary. However, if you completed this workshop with your own account, we strongly recommend following this guide to delete the resources and avoid incurring costs

Delete the resources you created for the lab in reverse order.

Database

Delete an Amazon RDS Cluster

After accessing to the Amazon RDS console, select DB Instances.

By default, an Amazon RDS cluster has delete protection enabled to prevent accidental deletions. To disable it, select the Cluster and click the Modify button.

Uncheck the Enable deletion protection button and click the Continue button.
For immediate deletion, select Apply immediately and click the Modify cluster button.
In order to delete a DB Cluster, you must first delete the DB instances included in the cluster. They can be deleted in any order, but we will delete the Writer instance first. Select the Writer instance, and click the Delete button on the Actions menu.

Type delete me in the blank and click the Delete button.
This time, we will delete the Reader instance. Select the Reader instance and click the Delete button on the Actions menu.
Type delete me in the blank and click the Delete button.
Lastly, we will delete the DB Cluster. Click the Delete button on the Actions menu.
Uncheck the Take a final snapshot button, check the I acknowledge that automatic backups, including system snapshots and point-in-time recovery, are no longer available when I delete an instance button, and type delete me in the blank. Click Delete DB Cluster and the DB cluster will be deleted.

Delete a Amazon RDS Snapshot

To delete the snapshot of the DB Cluster created during the lab, select immersionday-snapshot and click the Delete snapshot button on the Actions menu.
Click the Delete button.

Delete a secret in AWS Secrets Manager

We’re going to delete the secret that stored a RDS credential during the lab. Type Secrets Manager in the AWS console search bar and then select it.
Select mysecret.
Click Delete secret on the Actions menu.
To prevent accidental deletion of secrets, AWS Secrets Manager has a deletion wait time of minimum 7 days and maximum 30 days. Enter the minimum time of 7 days and press the Schedule deletion button.

Compute

Delete an Auto Scaling Group

We’re going to delete the Auto Scaling Group that we used during the lab. Type EC2 in the AWS Console search bar and select it. Select Auto Scaling Groups from the left menu. Select the Web-ASG that we created in the lab and click the Delete button on the Actions menu.
Type delete in the blank and click the Delete button.

Delete an Application Load Balancer

Next, we’re going to delete the Application Load Balancers. Select Load Balancers from the left menu. Then select the Web-ALB that we created in the lab and click the Delete load balancer button in the Actions menu.
Type confirm in the blank and click the Delete button.

Delete a Target Group

We’re going to delete the Target Group we created when we created the Application Load Balancer. Select Target Groups from the left menu. Select the Target Group we created in the lab, web-TG, and click the Delete button on the Actions menu.
Click the Yes, delete button.

Delete EC2 AMIs

Select AMIs from the left menu. Select the AMI named Web Server v1 that you created in the lab. Click the Deregister AMI button on the Actions menu.
Click the Deregister AMI button.

Delete EC2 Snapshots

You’ve just deleted an AMI, but this action doesn’t automatically remove the associated snapshot. So you need to remove it manually. From the left menu, choose Snapshots. Be sure to note the snapshot’s creation date. Then, select the snapshot you created in the lab, and click the Delete snapshot button on the Actions menu.
Click the Delete button.
Select Launch Templates from the left menu. Select the template named Web that you created in the lab. Click the Delete template button on the Actions menu.
Type Delete in the blank and click the Delete button.

(Optional) Delete an EC2 instance

If you went through the (Optional) Connect RDS Aurora section during the database lab, you need to delete the EC2 instance you created in the lab. Select Instances **from the left menu. Select the EC2 instance you created during the lab, and click the **Terminate instance **button on the **Instance state menu.
Click the Terminate button.

Network

Delete VPC endpoints

You’re almost there. Type VPC in the AWS Console search bar and select it. Select Endpoints from the left menu. Select S3 endpoint, the endpoint you created in the lab, and click the Delete VPC endpoints button on the Actions menu.
Type delete in the blank, and click the Delete button.

Delete a NAT gateway

Select NAT gateways from the left menu and select VPC-Lab-nat-public you created during the lab. Click the Delete NAT gateway button on the Actions menu.
Type delete in the blank and click the Delete button.

Delete an Elastic IP

You’ve just deleted the NAT gateway, but this action doesn’t automatically delete the Elastic IP that the NAT gateway used, so you need to remove it manually. Select Elastic IPs from the left menu, and select VPC-Lab-eip-ap-northeast-2a. (The name after VPC-Lab-eip may vary depending on your region.) Click the Release Elastic IP addresses button on the Actions menu. If it says it is still associated with the NAT gateway and cannot be deleted, refresh the webpage and try again.
Click the Release button.

Delete a Security Group

We’re going to delete the Security Group you created during the lab. Select Security Groups from the left menu. Select Immersion Day — Web Server and DB-SG first, and then click the Delete security groups button on the Actions menu. The reason for not deleting all security groups at once is that some security groups reference other security groups in their inbound rules. A security group that is being referenced cannot be deleted until the security group that is referencing it is deleted. Therefore, delete the security groups in the following order: Immersion Day — Web Server, DB-SG -> ASG-Web-Inst-SG -> web-ALB-SG.
Type delete in the blank and click the Delete button.
Select ASG-Web-Inst-SG and click the Delete security groups button on the Actions menu.
Click the Delete button.
Select web-ALB-SG and click the Delete security groups button on the Actions menu.
Click the Delete button.

Delete a VPC

Finally, select Your VPCs from the left menu, and select the VPC-Lab-vpc that you created during the lab. Click the Delete VPC button in the Actions menu.
Type delete in the blank and click the Delete button.

We strongly recommend that you double-check to make sure you haven’t missed anything, as some resources that weren’t cleared may incur costs.

Conclusion

This project demonstrates the power of AWS for building scalable, resilient applications with best practices in networking, compute, database, and storage services. From implementing VPC security to auto-scaling EC2 instances and configuring a fault-tolerant Aurora database, this architecture is well-suited for real-world applications that demand reliability and flexibility.

Explore my GitHub repository.

Let’s connect: Linkdin, Twitter, Github

Building with Generative AI on AWS : PartyRock, Amazon Bedrock, and Retrieval-Augmented Generation (RAG)

Shubham Murti — Mon, 11 Nov 2024 12:24:16 +0000

Generative AI on AWS: PartyRock, Amazon Bedrock, and Amazon Titan

Introduction

This project explores three distinct applications of Generative AI on AWS utilizing PartyRock, Amazon Bedrock, and Amazon Titan. Through practical experiments, we delve into no-code app development, advanced AI model integration, and retrieval-augmented generation (RAG) workflows. These projects demonstrate scalable AI solutions for various needs, from a book recommendation chatbot to context-aware response systems, all powered by AWS services.

Tech Stack

PartyRock: A no-code AI app builder with pre-configured widgets.
Amazon Bedrock: Access to cutting-edge AI models, including:
- Claude 3 Sonnet: For chat functionalities.
- Amazon Titan: For text generation.
- Titan Image Generator: Creates images based on prompts.
FAISS: Enables similarity searches and vector storage for RAG.
Amazon Titan Text Embeddings: Converts text to vectors for document-based AI models.

Prerequisites

AWS Account: Access to Bedrock, PartyRock, and Titan.
AWS CLI: Used for configuration management.
Basic AWS Knowledge: Familiarity with IAM roles, Bedrock, and AI model concepts.
No-Code Access: PartyRock allows non-developers to build apps.

Use Case Overview

This project highlights the potential of Generative AI in creating real-time content, context-aware responses, and AI-driven images. Here are three main applications:

No-code Book Recommendation Chatbot: Uses PartyRock to deliver personalized recommendations.
Foundation Model Integration: Powered by Amazon Bedrock, supporting real-time text, chat, and image generation.
Retrieval-Augmented Generation (RAG): Combines Amazon Titan, FAISS, and Claude 3 Sonnet to provide accurate responses based on stored knowledge.

Industries Benefiting from AI-driven Solutions

Industries such as customer service, e-commerce, and education can benefit significantly from these scalable and AI-driven applications.

Step-by-Step Implementation

Project 1: Build Generative AI Applications with PartyRock

In this section, we’ll learn how to use PartyRock to generate AI apps without any code.

What is PartyRock?

PartyRock is a shareable Generative AI app-building playground that allows you to experiment with prompt engineering in a hands-on and fun way. In just a few clicks, you can build, share, and remix apps, getting inspired while playing with Generative AI. Some examples of what you can do with PartyRock include:

Build an app to generate dad jokes on any topic of your choice.
Create and play a virtual trivia game with friends around the world.
Build an AI storyteller to guide your next fantasy roleplaying campaign.

By building and playing with PartyRock apps, you will learn the fundamental techniques and capabilities needed to get started with Generative AI. This includes understanding how a foundational model responds to prompts, experimenting with different text-based inputs, and chaining prompts together to create more dynamic outputs.

Anyone can experiment with PartyRock by creating a profile using a social login from Amazon.com, Apple, or Google. PartyRock is separate from the AWS console and does not require an AWS account to get started.

Exercise 1: Building a PartyRock Application

To highlight the power of PartyRock, we’re going to build an application that can provide book recommendations based on your mood.

Head over to the PartyRock website.
Log in with a social account (Amazon, Apple, or Google).
Click on Build your own app and enter the following prompt: "Provide book recommendations based on your mood and a chatbot to talk about the books."
Click Generate app.

Using the App

PartyRock will create the interface needed to take in user input, provide recommendations, and create a chatbot—just from your prompt! Try the following:

Enter a mood, like "Happy."
Ask the chatbot for more information about the book recommendations by typing: "Can you tell me more about one of the books that was listed?"

You can also share your app by clicking the Make public and Share buttons.

Updating Your App

In PartyRock, each UI element is a widget, which displays content, takes input, connects to other widgets, and generates output. Widgets that take input allow users to interact with the app, while widgets that create output use prompts and references to generate something like an image or text.

Types of Widgets

There are three types of AI-powered widgets:

Image Generation
Chatbot
Text Generation

You can edit these widgets to connect them to others and change their outputs.

Additionally, there are three other widgets:

User Input: Allows users to change the output by connecting it to AI-powered widgets.
Static Text: Provides a space for text descriptions.
Document Upload: Lets users upload documents that can be processed by the app.

For more details, check out the PartyRock Guide.

Exercise 2: Playtime with PartyRock

Now that you have a basic app, it’s time to explore! Try the following:

Update the prompts in your app.
Play with the settings.
Chain outputs together to create new workflows.

Get creative and explore what PartyRock can do. For example, try adding a widget that can draw an image of the book based on its description.

Remixing an Application

With PartyRock, you can Remix applications, which lets you make a copy of an app and edit it to your liking. You can remix your own apps, or remix public apps from friends or from the PartyRock Discover page.

Try remixing one of the apps from the Discover page to create new variations!

Creating a Snapshot

Did you get a funny or interesting response from an app you’re using? You can share a snapshot with others! Here's how:

Make sure the app is in public mode.
Click Snapshot in the top right corner of the app page.
The URL that includes the current input and output of your app will be copied to your clipboard, so you can share it with others.

Wrap Up

With PartyRock, you can demo and propose ideas that leverage Generative AI in a fun, easy-to-use environment. When you're ready to build apps for production, you can implement those ideas using Amazon Bedrock.

Project 2: Use Foundation Models in Amazon Bedrock

Amazon Bedrock is a fully managed service that offers access to high-performing foundation models (FMs) from leading AI companies like Stability AI, Anthropic, and Meta, all via a single API. With Amazon Bedrock, you can securely integrate and deploy Generative AI capabilities into your applications, using the AWS services you’re already familiar with—without the need to manage infrastructure.

In this module, we'll explore how to use Amazon Bedrock through the console and the API to generate both text and images.

Model Access

Before you can start building with Bedrock, you will need to grant model access to your AWS account.

Head to the Model Access page.
Select the Enable specific models button.
Check the models you wish to activate. If you're running this from your own account, there’s no cost to activate the models—you only pay for what you use during the labs.

Here’s a list of supported models:

Amazon (select to automatically activate all Amazon Titan models)
Anthropic: Claude 3.5 Sonnet, Claude 3 Sonnet, Claude 3 Haiku
Meta: Llama 3.1 405B Instruct, Llama 3.1 70B Instruct, Llama 3.1 8B Instruct
Mistral AI
Stability AI: SDXL 1.0

After selecting the models, click Request model access to activate them in your account.

Monitor Model Access Status

It may take a few minutes for the models to transition from "In Progress" to "Access granted" status. You can use the Refresh button to periodically check for updates.

Once the status shows Access granted, you're ready to begin.

Using the Amazon Bedrock Playground

The Amazon Bedrock Playground is a great way to experiment with different foundation models directly inside the AWS Console. You can compare model outputs, load example prompts, and even export API requests.

The playground currently supports three modes:

Chat: Experiment with a wide range of language processing tasks in a turn-by-turn interface.
Text: Test fast iterations on a variety of language processing tasks.
Image: Generate compelling images by providing text prompts to pre-trained models.

You can access the playground via the links above or from the Amazon Bedrock Console under the Playgrounds side menu. Take a few minutes to explore the examples.

Playground Examples

Here are some examples you can try in each playground mode:

Chat Mode

Click the Select model button to open the Model Selection popup.
Choose Anthropic Claude 3 Sonnet.

Click Load examples and select Advanced Q&A with Citations.
Once the example is loaded, click Run to start the chat.

You can adjust the model’s configuration in the sidebar. Try changing the Temperature to 1 to make the model more creative in its responses.

Text Mode

In this example, we selected Amazon Titan Text G1 - Express as the model and loaded the JSON creation example.

Try changing the model by selecting Change and choosing Mistral → Mistral Large 2.
Clear the output and run the prompt again.

Notice how the output differs. It's important to experiment with different foundation models to find the one that best fits your use case.

Image Mode

Select Titan Image Generator G1 as the model and load the Generate images from a text prompt example.

Try changing the Prompt Strength to 10 and experiment with different prompts, such as:
- "Unicorns in a magical forest. Lots of trees and animals around. The mood is bright, and there is lots of natural lighting."
- "Downtown City, with lots of skyscrapers. At night time, lots of lights in the buildings."

Play around with different prompts to see how the model responds to variations in input.

Wrap Up: Using Amazon Bedrock in Applications via API

Now that you’ve explored the Bedrock Playground, let’s see how we can bring the power of Amazon Bedrock to applications using the API.

The playground is a great starting point, but integrating these models into your applications will give you the flexibility to create production-level solutions. Whether you’re generating text, images, or handling interactive chat, Amazon Bedrock offers a serverless, scalable way to leverage powerful foundation models.

Project 3: Chat with your Documents

This module teaches how to use Retrieval Augmented Generation (RAG), a powerful approach that combines document retrieval with generative AI models to answer questions based on the content of documents. We'll learn how to set up a RAG system using Amazon Bedrock and work with various types of documents, including PDFs and knowledge bases, to generate accurate responses based on the context of the document.

Architecture Overview

In this module, the Retrieval Augmented Generation (RAG) system consists of several key components:

Embeddings: Text is converted into vector representations (embeddings) using models like Amazon Titan or Anthropic Claude. These embeddings capture the meaning and context of the text.
Vector Database: Once converted into embeddings, the text data is stored in a vector database, which enables quick and relevant retrieval of documents using similarity searches.
Bedrock for RAG: With embeddings and a vector store, we use Amazon Bedrock to retrieve the most relevant documents based on the user query and then pass those documents to a language model (like Claude or Titan) to generate a relevant answer.
LangChain: This Python framework simplifies the development of applications with LLMs and helps with managing the RAG process.

Exercise 1: Getting Started with RAG

The base_rag.py script demonstrates how RAG works by using a small set of documents (sentences) and querying them to generate answers based on context.

Document Setup: Define a set of sentences representing different topics, such as pets, cities, and colors.

sentences = [
    "Your dog is so cute.",
    "How cute your dog is!",
    "You have such a cute dog!",
    "New York City is the place where I work.",
    "I work in New York City.",
    "What color do you like the most?",
    "What is your favorite color?",
]

Embedding with Amazon Bedrock: Use the Amazon Titan Text Embeddings model to convert these sentences into embeddings.

embeddings = BedrockEmbeddings(
    client=bedrock_runtime,
    model_id="amazon.titan-embed-text-v1",
)

Vector Store with FAISS: Store the embeddings in a FAISS vector store, which allows for efficient similarity searches.

local_vector_store = FAISS.from_texts(sentences, embeddings)

RAG Workflow: Given a user query, convert the query into an embedding, retrieve the relevant documents, and combine the context of the documents to generate a coherent answer using a model like Claude.

context = ""
for doc in docs:
    context += doc.page_content

prompt = f"""Use the following pieces of context to answer the question at the end.

{context}

Question: {query}
Answer:"""

Run the Code: Test the RAG system by running the script in the terminal.

python3 rag_examples/base_rag.py

You can experiment by modifying the query in the code. For instance, try asking, "What city do I work in?" and see how the system responds based on the context.

Exercise 2: Chat with a PDF

Now let's work with a PDF document. In the file chat_with_pdf.py, the function chunk_doc_to_text will read the PDF and chunk its content into pieces, each of 1000 characters, before storing them in the vector database.

PDF Chunking: The process of chunking the document allows you to process large files and query sections of the document efficiently.
Querying the PDF: After the PDF has been chunked, you can use a query like "What are some good use cases for non-SQL databases?" to test the retrieval and response capabilities.

Run the code as follows:

python3 rag_examples/chat_with_pdf.py

Play around with the queries and see how the model answers questions based on the PDF context.

Creating a Knowledge Base

A Knowledge Base (KB) in Amazon Bedrock helps store and query documents in a structured manner. By uploading a dataset (e.g., AWS Well-Architected Framework) to Amazon S3, you can automatically create a KB, which will handle the embeddings and vector database setup.

Create Knowledge Base: Navigate to the Knowledge Base Console and create a new Knowledge Base by uploading documents to an S3 bucket and selecting the embeddings model.
Sync Data: Once the Knowledge Base is created, click the Sync button to load the data. You can then query the KB for information like "What is a VPC?" and retrieve relevant responses.
Query via Console: Use the console to enter a query and see how the system responds. For example:

Can you explain what a VPC is?

Exercise 3: Using the Knowledge Base API

You can also query your Knowledge Base programmatically via the API using the retrieve or retrieve_and_generate methods. These methods can be invoked to fetch documents or generate answers using the RAG process.

Open the file kb_rag.py and update the KB_ID with your created Knowledge Base ID.
Run the script using:

python3 rag_examples/kb_rag.py

You can modify the QUERY variable to test different questions and see how the KB API performs.

Building Agents for Amazon Bedrock

In this section, we'll build an Agent using Amazon Bedrock. An agent is a task-specific AI model that can interact with different services, such as querying Knowledge Bases or invoking AWS Lambda functions.

Create an Agent: In the Agent Console, create a new agent named Agent-AWS, select the model (Claude 3 Sonnet), and provide a role description (e.g., AWS Certified Solutions Architect).

Define Action Groups: Action groups are predefined tasks that the agent can perform. For instance, you can define an action to process data by reading records from a database.

Knowledge Base Integration: You can add the Knowledge Base you created earlier to the agent. This allows the agent to query the KB and use it to answer AWS-related questions.
Test the Agent: Once the agent is created, use the console to test it. Ask questions like "What can you tell me about S3 buckets?" and inspect the agent’s responses. You can also simulate errors to test the agent's capabilities to troubleshoot issues.

Debugging Lambda Functions with Amazon Q

Use the following test event JSON to mimic the agent calling the fucntion

{
  "agent": {
    "alias": "TSTALIASID",
    "name": "Agent-AWS",
    "version": "DRAFT",
    "id": "ADI6ICMMZZ"
  },
  "sessionId": "975786472213626",
  "httpMethod": "GET",
  "sessionAttributes": {},
  "inputText": "Can you get the number of records in the databse",
  "promptSessionAttributes": {},
  "apiPath": "/get_num_records",
  "messageVersion": "1.0",
  "actionGroup": "agent_action_group"
}

If you encounter errors in Lambda functions triggered by the agent, you can use Amazon Q to debug and resolve issues.

Invoke Lambda Function: In the Lambda console, invoke your function and intentionally trigger an error.

Troubleshoot with Amazon Q: Use Amazon Q to debug the error and resolve issues by following the troubleshooting steps.

Clean-up

Once you've finished, make sure to clean up your resources by deleting:

S3 Objects: Delete any objects in the S3 buckets (e.g., awsdocsbucket, openapiBucket).
IAM Roles: Delete IAM roles associated with Bedrock and other services.
Knowledge Base: Delete the Knowledge Base and any related data.
Agent: Delete the agent.
OpenSearch Collection: Delete the vector store.
CloudFormation Stack: Delete the CloudFormation stack to remove all resources.

Conclusion

This project demonstrates the flexibility of Generative AI on AWS to build dynamic applications. Whether it’s no-code development with PartyRock, scalable AI integrations with Bedrock, or context-driven responses through RAG, AWS offers the tools to create efficient and powerful AI solutions. These applications highlight the potential for AI-driven solutions in customer support, knowledge management, and interactive content creation.

Explore my GitHub repository.

Let’s connect: Linkdin, Twitter, Github

Building a Serverless Recipe Generator : AWS Project

Shubham Murti — Mon, 11 Nov 2024 09:14:04 +0000

Introduction

This project walks through creating a serverless recipe generator application using AWS Amplify and Amazon Bedrock. Users can input ingredients and receive AI-generated recipes via a simple web interface. By combining Generative AI with a serverless architecture, this application offers a scalable and interactive experience.

Tech Stack

AWS Amplify: Provides full-stack hosting and deployment services for the application.
Amazon Bedrock: Utilized as an AI model to generate recipes based on user input.
AWS AppSync: Manages real-time API connections between the frontend and backend.
AWS Cognito: Provides secure user authentication.
AWS Lambda: Handles serverless backend functionality for processing requests.
Node.js, npm, Git, GitHub: Essential tools for frontend dependencies and version control.

Prerequisites

AWS Account: Permissions for Amplify, Cognito, Bedrock, and Lambda.
AWS CLI and Amplify CLI: For resource management.
Node.js & npm: For running React app.
GitHub: Enables Amplify’s continuous deployment.

Use Case

Problem: Searching for suitable recipes based on available ingredients can be time-consuming, and manually sifting through results often yields non-personalized options.

Solution: This serverless recipe generator offers a real-time AI-powered solution, allowing users to input ingredients and receive tailored recipes instantly. The use of Amazon Bedrock makes it possible to generate diverse, personalized recipes on demand, saving users time and enhancing their cooking experience.

Real-World Relevance: This type of application has potential in e-commerce, cooking platforms, and smart home solutions. By integrating Generative AI and serverless architecture, we demonstrate a scalable, real-time solution for creating personalized recipes or content.

Architecture Diagram

To follow along with this tutorial, you will need:

AWS Account: Requires permissions for Amplify, Cognito, Bedrock, and Lambda.
AWS CLI: Needed for resource management.
Node.js & npm: For running the React app.
Amplify CLI: For initializing and configuring the Amplify project.
Basic AWS Knowledge: Familiarity with Amplify, Cognito, AppSync, and Lambda services.
GitHub Repository: To enable Amplify’s continuous deployment integration.

Component Breakdown

Frontend (React Application):

User-facing interface where users input ingredients and receive generated recipes. Hosted on AWS Amplify with continuous deployment.
AWS Cognito:

Manages secure authentication for users, ensuring only authorized access to backend resources.
AWS AppSync:

Provides the GraphQL API to handle communication between the frontend and backend services.
AWS Lambda:

Serverless function that takes user input and forwards it to Amazon Bedrock for AI-powered recipe generation.
Amazon Bedrock (Claude 3 Sonnet):

Utilizes generative AI to create unique recipes based on input ingredients.

Step-by-Step Implementation

Task 1: Host a Static Website

In this task, you will create a React application and deploy it to the Cloud using AWS Amplify.

Step 1: Create a New React Application

In a new terminal or command line window, run the following command to use Vite to create a React application:

   npm create vite@latest ai-recipe-generator -- --template react-ts -y

Navigate to the project directory:

   cd ai-recipe-generator

Install the necessary dependencies:

   npm install

Start the development server:

   npm run dev

This will start your React application locally. You should see the Vite + React app running.

In the terminal window, open the local link to view the application.

Step 2: Initialize a GitHub Repository

In this step, you will create a GitHub repository and commit your code to it.

Sign in to GitHub at GitHub.
Create a new repository:
- For Repository name, enter ai-recipe-generator.
- Choose the Public radio button.
- Select Create a new repository.
Open a new terminal window, navigate to your project’s root folder (ai-recipe-generator), and run the following commands to initialize Git and push the application to your GitHub repository:

   git init
   git add .
   git commit -m "first commit"
   git remote add origin git@github.com:<your-username>/ai-recipe-generator.git
   git branch -M main
   git push -u origin main

Replace the git@github.com:<your-username>/ai-recipe-generator.git with your own GitHub SSH URL.

Step 3: Install the Amplify Packages

Open a new terminal window, navigate to your app's root folder (ai-recipe-generator), and run the following command:

   npm create amplify@latest -y

This will scaffold a lightweight Amplify project in your app’s directory.

After installation, push the changes to GitHub:

   git add .
   git commit -m 'installing amplify'
   git push origin main

Step 4: Deploy Your App with AWS Amplify

Sign in to the AWS Management Console and open the AWS Amplify console at AWS Amplify.
Click Create new app.
On the Start building with Amplify page, select GitHub for Deploy your app and click Next.
Authenticate with GitHub. You will be automatically redirected back to the Amplify console. Choose the repository and main branch you created earlier, then select Next.
Leave the default build settings and click Next.
Review your inputs and click Save and deploy.

AWS Amplify will now build your source code and deploy your app at https://...amplifyapp.com. Your app will automatically update with every Git push. It may take up to 5 minutes to deploy.

Once the build is complete, click Visit deployed URL to see your web app live.

Task 2: Manage Users

In this task, you will configure Amplify Auth for user authentication and enable Amazon Bedrock foundation model access.

Step 1: Set up Amplify Auth

The app uses email as the default login mechanism. When users sign up, they will receive a verification email. Customize the verification email by updating the following code in the ai-generated-recipes/amplify/auth/resource.ts file:

   import { defineAuth } from "@aws-amplify/backend";

   export const auth = defineAuth({
     loginWith: {
       email: {
         verificationEmailStyle: "CODE",
         verificationEmailSubject: "Welcome to the AI-Powered Recipe Generator!",
         verificationEmailBody: (createCode) =>
           `Use this code to confirm your account: ${createCode()}`,
       },
     },
   });

Save the file.

The image on the right shows an example of the customized verification email.

Step 2: Set up Amazon Bedrock Model Access

Verify that you are in the N. Virginia (us-east-1) region, and select Get started.

In the Foundation models section, choose the Claude model.
Scroll down to the Claude models section, select the Claude 3 Sonnet tab, and click Request model access.

If you already have access to some models, the button will show Manage model access instead.

In the Base models section, for Claude 3 Sonnet, choose Available to request, then select Request model access.
On the Edit model access page, click Next.
On the Review and Submit page, click Submit.

Now your application should be up and running, with user authentication and AI-powered recipe generation using Amazon Bedrock's Claude 3 Sonnet.

Here's the reformatted version of Task 3: Build a Serverless Backend and Task 4: Deploy the Backend API, with improved structure and readability for a Medium blog post:

Task 3: Build a Serverless Backend

In this task, you will use AWS Amplify and AWS Lambda to build a serverless function.

Step 1: Create a Lambda Function for Handling Requests

On your local machine, navigate to the ai-recipe-generator/amplify/data folder, and create a new file named bedrock.js.
Update the file with the following code:

The code defines a request function that constructs the HTTP request to invoke the Claude 3 Sonnet foundation model in Amazon Bedrock. The response function parses the response and returns the generated recipe.

   export function request(ctx) {
     const { ingredients = [] } = ctx.args;

     // Construct the prompt with the provided ingredients
     const prompt = `Suggest a recipe idea using these ingredients: ${ingredients.join(", ")}.`;

     // Return the request configuration
     return {
       resourcePath: `/model/anthropic.claude-3-sonnet-20240229-v1:0/invoke`,
       method: "POST",
       params: {
         headers: {
           "Content-Type": "application/json",
         },
         body: JSON.stringify({
           anthropic_version: "bedrock-2023-05-31",
           max_tokens: 1000,
           messages: [
             {
               role: "user",
               content: [
                 {
                   type: "text",
                   text: `\n\nHuman: ${prompt}\n\nAssistant:`,
                 },
               ],
             },
           ],
         }),
       },
     };
   }

   export function response(ctx) {
     // Parse the response body
     const parsedBody = JSON.parse(ctx.result.body);

     // Extract the text content from the response
     const res = {
       body: parsedBody.content[0].text,
     };

     // Return the response
     return res;
   }

The request function constructs the AI prompt and sends it to Amazon Bedrock. The response function parses the generated recipe from the returned JSON.

Step 2: Add Amazon Bedrock as a Data Source

Open the amplify/backend.ts file and add the following code. Save the file.

The code adds an HTTP data source for Amazon Bedrock to your API and grants it the necessary permissions to invoke the Claude model.

   import { defineBackend } from "@aws-amplify/backend";
   import { data } from "./data/resource";
   import { PolicyStatement } from "aws-cdk-lib/aws-iam";
   import { auth } from "./auth/resource";

   const backend = defineBackend({
     auth,
     data,
   });

   const bedrockDataSource = backend.data.resources.graphqlApi.addHttpDataSource(
     "bedrockDS",
     "https://bedrock-runtime.us-east-1.amazonaws.com",
     {
       authorizationConfig: {
         signingRegion: "us-east-1",
         signingServiceName: "bedrock",
       },
     }
   );

   bedrockDataSource.grantPrincipal.addToPrincipalPolicy(
     new PolicyStatement({
       resources: [
         "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-sonnet-20240229-v1:0",
       ],
       actions: ["bedrock:InvokeModel"],
     })
   );

This code integrates Amazon Bedrock into your backend, enabling the serverless app to invoke the Claude 3 Sonnet model.

Task 4: Deploy the Backend API

Step 1: Set Up Amplify Data

In the amplify/backend.ts file, define the schema and backend resources as follows:

   import { type ClientSchema, a, defineData } from "@aws-amplify/backend";

   const schema = a.schema({
     BedrockResponse: a.customType({
       body: a.string(),
       error: a.string(),
     }),

     askBedrock: a
       .query()
       .arguments({ ingredients: a.string().array() })
       .returns(a.ref("BedrockResponse"))
       .authorization((allow) => [allow.authenticated()])
       .handler(
         a.handler.custom({ entry: "./bedrock.js", dataSource: "bedrockDS" })
       ),
   });

   export type Schema = ClientSchema<typeof schema>;

   export const data = defineData({
     schema,
     authorizationModes: {
       defaultAuthorizationMode: "apiKey",
       apiKeyAuthorizationMode: {
         expiresInDays: 30,
       },
     },
   });

In this step, you define a GraphQL schema and link the askBedrock query to the custom Lambda function you created earlier (bedrock.js). The schema allows authenticated users to invoke the function and retrieve a recipe suggestion.

Step 2: Deploy Cloud Resources

Open a new terminal window, navigate to your app's project folder (ai-recipe-generator), and run the following command to deploy cloud resources into an isolated development space:

   npx ampx sandbox

This command sets up a sandbox environment where you can quickly iterate on your changes.

After the sandbox has been fully deployed, you will see a confirmation message in the terminal, and an amplify_outputs.json file will be generated and added to your project.

The deployment process is now complete, and you can begin interacting with your serverless backend.

The serverless backend is now set up and ready to handle recipe generation requests using Amazon Bedrock. You've created a Lambda function to handle interactions with the Claude 3 Sonnet model, integrated it with AWS Amplify, and deployed it to the cloud.

Here's how you can present Task 5: Build the Frontend and the challenges faced in the process for your blog:

Task 5: Build the Frontend

Now that the serverless backend is set up, it's time to create the frontend. This step will guide you through building the user interface (UI) using AWS Amplify libraries, styled components, and React. We’ll also implement authentication to ensure that only authorized users can access the app.

Step 1: Install the Amplify Libraries

To get started with integrating AWS Amplify into your frontend, you'll need to install the core Amplify libraries. The aws-amplify library provides all the necessary APIs to interact with your backend, while @aws-amplify/ui-react contains pre-built UI components that help you scaffold the authentication flow and other UI elements.

Open a new terminal window, navigate to your project’s root folder (ai-recipe-generator), and run the following command to install the libraries:

   npm install aws-amplify @aws-amplify/ui-react

Step 2: Style the App UI

Next, we’ll style the frontend to create a clean, modern interface. We'll focus on centering the layout and styling the form where users will input their ingredients.

Open ai-recipe-generator/src/index.css, and update it with the following code to set global styles and center the UI:

   :root {
     font-family: Inter, system-ui, Avenir, Helvetica, Arial, sans-serif;
     line-height: 1.5;
     font-weight: 400;
     color: rgba(255, 255, 255, 0.87);
     max-width: 1280px;
     margin: 0 auto;
     padding: 2rem;
   }

   .card {
     padding: 2em;
   }

   .box:nth-child(3n + 1) {
     grid-column: 1;
   }
   .box:nth-child(3n + 2) {
     grid-column: 2;
   }
   .box:nth-child(3n + 3) {
     grid-column: 3;
   }

This CSS will help ensure that the app's layout is centered, the font is legible, and the UI components have a clean, consistent style.

Next, open ai-recipe-generator/src/App.css and update it with the following code to style the ingredient input form:

   .app-container {
     margin: 0 auto;
     padding: 20px;
     text-align: center;
   }

   .header-container {
     padding-bottom: 2.5rem;
     text-align: center;
   }

   .main-header {
     font-size: 2.25rem;
     font-weight: bold;
     color: #1a202c;
   }

   .main-header .highlight {
     color: #2563eb;
   }

   .description {
     font-weight: 500;
     font-size: 1.125rem;
     max-width: 65ch;
     color: #1a202c;
   }

   .form-container {
     margin-bottom: 20px;
   }

   .search-container {
     display: flex;
     flex-direction: column;
     gap: 10px;
     align-items: center;
   }

   .wide-input {
     width: 100%;
     padding: 10px;
     font-size: 16px;
     border: 1px solid #ccc;
     border-radius: 4px;
   }

   .search-button {
     width: 100%;
     max-width: 300px;
     padding: 10px;
     font-size: 16px;
     background-color: #007bff;
     color: white;
     border: none;
     border-radius: 4px;
     cursor: pointer;
   }

   .search-button:hover {
     background-color: #0056b3;
   }

   .result-container {
     margin-top: 20px;
     transition: height 0.3s ease-out;
     overflow: hidden;
   }

   .loader-container {
     display: flex;
     flex-direction: column;
     align-items: center;
     gap: 10px;
   }

   .result {
     background-color: #f8f9fa;
     border: 1px solid #e9ecef;
     border-radius: 4px;
     padding: 15px;
     white-space: pre-wrap;
     word-wrap: break-word;
     color: black;
     font-weight: bold;
     text-align: left;
   }

This will ensure that the ingredients form and result display are properly styled.

Step 3: Implement the UI

Now, let’s build the main React component for the app. We’ll integrate the AWS Amplify Authentication components for user sign-up, sign-in, and password recovery.

Open ai-recipe-generator/src/main.tsx and update it with the following code. This will use the Amplify Authenticator component to wrap your app and provide a complete authentication flow:

   import React from "react";
   import ReactDOM from "react-dom/client";
   import App from "./App.jsx";
   import "./index.css";
   import { Authenticator } from "@aws-amplify/ui-react";

   ReactDOM.createRoot(document.getElementById("root")!).render(
     <React.StrictMode>
       <Authenticator>
         <App />
       </Authenticator>
     </React.StrictMode>
   );

The Authenticator component will handle user authentication, including sign-up, sign-in, and MFA (Multi-Factor Authentication).

Next, open ai-recipe-generator/src/App.tsx and update it with the following code to implement the form for ingredient submission and the logic for querying the askBedrock function.

   import { FormEvent, useState } from "react";
   import { Loader, Placeholder } from "@aws-amplify/ui-react";
   import "./App.css";
   import { Amplify } from "aws-amplify";
   import { Schema } from "../amplify/data/resource";
   import { generateClient } from "aws-amplify/data";
   import outputs from "../amplify_outputs.json";

   import "@aws-amplify/ui-react/styles.css";

   Amplify.configure(outputs);

   const amplifyClient = generateClient<Schema>({
     authMode: "userPool",
   });

   function App() {
     const [result, setResult] = useState<string>("");
     const [loading, setLoading] = useState(false);

     const onSubmit = async (event: FormEvent<HTMLFormElement>) => {
       event.preventDefault();
       setLoading(true);

       try {
         const formData = new FormData(event.currentTarget);

         const { data, errors } = await amplifyClient.queries.askBedrock({
           ingredients: [formData.get("ingredients")?.toString() || ""],
         });

         if (!errors) {
           setResult(data?.body || "No data returned");
         } else {
           console.log(errors);
         }
       } catch (e) {
         alert(`An error occurred: ${e}`);
       } finally {
         setLoading(false);
       }
     };

     return (
       <div className="app-container">
         <div className="header-container">
           <h1 className="main-header">
             Meet Your Personal
             <br />
             <span className="highlight">Recipe AI</span>
           </h1>
           <p className="description">
             Simply type a few ingredients using the format ingredient1,
             ingredient2, etc., and Recipe AI will generate an all-new recipe on
             demand...
           </p>
         </div>
         <form onSubmit={onSubmit} className="form-container">
           <div className="search-container">
             <input
               type="text"
               className="wide-input"
               id="ingredients"
               name="ingredients"
               placeholder="Ingredient1, Ingredient2, Ingredient3,...etc"
             />
             <button type="submit" className="search-button">
               Generate
             </button>
           </div>
         </form>
         <div className="result-container">
           {loading ? (
             <div className="loader-container">
               <p>Loading...</p>
               <Loader size="large" />
               <Placeholder size="large" />
               <Placeholder size="large" />
               <Placeholder size="large" />
             </div>
           ) : (
             result && <p className="result">{result}</p>
           )}
         </div>
       </div>
     );
   }

   export default App;

The app allows users to input a list of ingredients, submits the request to the backend via Amplify and Amazon Bedrock, and then displays the generated recipe.

Step 4: Run and Test the App

Open a new terminal window and navigate to your project’s root directory (ai-recipe-generator), then run:

   npm run dev

Visit the local host URL to open the app in your browser and test it.

Step 5: Deploy the App

Once you’ve confirmed that the app is working as expected locally, it’s time to deploy it to AWS Amplify.

In the terminal, commit your changes:

   git add .
   git commit -m 'connect to bedrock'
   git push origin main

Go to the AWS Amplify console, and your app should automatically build and deploy.

After deployment, you can access your live app at the provided Amplify URL.

Challenges and Solutions

Handling Bedrock Latency: Managed response delays with retry logic.
Real-Time Data Sync: AppSync ensures consistent updates across all instances.

Conclusion

This project combines AWS Amplify, Amazon Bedrock, and AWS Cognito to build a scalable, AI-powered recipe generator. It highlights the potential of serverless applications with Generative AI, making it ideal for real-time, interactive user experiences.

Explore my GitHub repository.

Let’s connect: Linkdin, Twitter, Github

Full-Stack Web Application with AWS Amplify: AWS Project

Shubham Murti — Sun, 10 Nov 2024 16:57:39 +0000

Introduction

This project demonstrates the creation of a full-stack web application using AWS Amplify, featuring a React frontend with user authentication, a serverless function for user sign-ups, and DynamoDB for data storage. AWS Amplify’s managed services make it easy to build a scalable and secure web application with seamless backend integration.

Why AWS Amplify?

AWS Amplify provides backend services like hosting, authentication, and data storage, allowing developers to focus on application functionality without managing infrastructure.

Learning Outcomes

Hosting: Deploy a React app on AWS’s global CDN.
Authentication: Enable user sign-in and sign-out.
Database Integration: Use a real-time API and DynamoDB.
Function Execution: Trigger Lambda functions on user sign-up.

Tech Stack

AWS Amplify: Hosting and backend services.
AWS AppSync: Real-time API management.
AWS Lambda: Serverless function for user data.
Amazon DynamoDB: NoSQL database for storing user emails.
React: Frontend framework.
Node.js & npm: For dependencies.
Git & GitHub: Version control.

Prerequisites

Familiarity with AWS services like Amplify, Lambda, and DynamoDB.
AWS CLI configured with appropriate IAM permissions.
Node.js, npm, and Git installed locally.

Problem Statement

Building scalable web applications often requires authentication, data storage, and API integration. AWS Amplify simplifies these needs with a managed backend that integrates seamlessly with the frontend.

Architecture Diagram

Implementation Steps

Tasks Overview

This project is structured into six main tasks:

Create Web App: Deploy a React app using AWS Amplify Console.
Build Serverless Function: Create a Lambda function.
Create Data Table: Set up DynamoDB for data persistence.
Link Function to Web App: Deploy function with API Gateway.
Add Web App Interactivity: Update the frontend to call the API.
Clean Up Resources: Delete resources to avoid charges.

Step-by-Step Implementation

Task 1: Host a Static Website

In this task, you will create a React application and deploy it to the Cloud using AWS Amplify.

Step 1: Create a New React Application

In a new terminal or command line window, run the following command to use Vite to create a React application:

   npm create vite@latest ai-recipe-generator -- --template react-ts -y

Navigate to the project directory:

   cd ai-recipe-generator

Install the necessary dependencies:

   npm install

Start the development server:

   npm run dev

This will start your React application locally. You should see the Vite + React app running.

In the terminal window, open the local link to view the application.

Step 2: Initialize a GitHub Repository

In this step, you will create a GitHub repository and commit your code to it.

Sign in to GitHub at GitHub.
Create a new repository:
- For Repository name, enter ai-recipe-generator.
- Choose the Public radio button.
- Select Create a new repository.
Open a new terminal window, navigate to your project’s root folder (ai-recipe-generator), and run the following commands to initialize Git and push the application to your GitHub repository:

   git init
   git add .
   git commit -m "first commit"
   git remote add origin git@github.com:<your-username>/ai-recipe-generator.git
   git branch -M main
   git push -u origin main

Replace the git@github.com:<your-username>/ai-recipe-generator.git with your own GitHub SSH URL.

Step 3: Install the Amplify Packages

Open a new terminal window, navigate to your app's root folder (ai-recipe-generator), and run the following command:

   npm create amplify@latest -y

This will scaffold a lightweight Amplify project in your app’s directory.

After installation, push the changes to GitHub:

   git add .
   git commit -m 'installing amplify'
   git push origin main

Step 4: Deploy Your App with AWS Amplify

Sign in to the AWS Management Console and open the AWS Amplify console at AWS Amplify.
Click Create new app.
On the Start building with Amplify page, select GitHub for Deploy your app and click Next.
Authenticate with GitHub. You will be automatically redirected back to the Amplify console. Choose the repository and main branch you created earlier, then select Next.
Leave the default build settings and click Next.
Review your inputs and click Save and deploy.

AWS Amplify will now build your source code and deploy your app at https://...amplifyapp.com. Your app will automatically update with every Git push. It may take up to 5 minutes to deploy.

Once the build is complete, click Visit deployed URL to see your web app live.

Task 2: Manage Users

In this task, you will configure Amplify Auth for user authentication and enable Amazon Bedrock foundation model access.

Step 1: Set up Amplify Auth

The app uses email as the default login mechanism. When users sign up, they will receive a verification email. Customize the verification email by updating the following code in the ai-generated-recipes/amplify/auth/resource.ts file:

   import { defineAuth } from "@aws-amplify/backend";

   export const auth = defineAuth({
     loginWith: {
       email: {
         verificationEmailStyle: "CODE",
         verificationEmailSubject: "Welcome to the AI-Powered Recipe Generator!",
         verificationEmailBody: (createCode) =>
           `Use this code to confirm your account: ${createCode()}`,
       },
     },
   });

Save the file.

The image on the right shows an example of the customized verification email.

Step 2: Set up Amazon Bedrock Model Access

Verify that you are in the N. Virginia (us-east-1) region, and select Get started.

In the Foundation models section, choose the Claude model.
Scroll down to the Claude models section, select the Claude 3 Sonnet tab, and click Request model access.

If you already have access to some models, the button will show Manage model access instead.

In the Base models section, for Claude 3 Sonnet, choose Available to request, then select Request model access.
On the Edit model access page, click Next.
On the Review and Submit page, click Submit.

Now your application should be up and running, with user authentication and AI-powered recipe generation using Amazon Bedrock's Claude 3 Sonnet.

Here's the reformatted version of Task 3: Build a Serverless Backend and Task 4: Deploy the Backend API, with improved structure and readability for a Medium blog post:

Task 3: Build a Serverless Backend

In this task, you will use AWS Amplify and AWS Lambda to build a serverless function.

Step 1: Create a Lambda Function for Handling Requests

On your local machine, navigate to the ai-recipe-generator/amplify/data folder, and create a new file named bedrock.js.
Update the file with the following code:

   export function request(ctx) {
     const { ingredients = [] } = ctx.args;

     // Construct the prompt with the provided ingredients
     const prompt = `Suggest a recipe idea using these ingredients: ${ingredients.join(", ")}.`;

     // Return the request configuration
     return {
       resourcePath: `/model/anthropic.claude-3-sonnet-20240229-v1:0/invoke`,
       method: "POST",
       params: {
         headers: {
           "Content-Type": "application/json",
         },
         body: JSON.stringify({
           anthropic_version: "bedrock-2023-05-31",
           max_tokens: 1000,
           messages: [
             {
               role: "user",
               content: [
                 {
                   type: "text",
                   text: `\n\nHuman: ${prompt}\n\nAssistant:`,
                 },
               ],
             },
           ],
         }),
       },
     };
   }

   export function response(ctx) {
     // Parse the response body
     const parsedBody = JSON.parse(ctx.result.body);

     // Extract the text content from the response
     const res = {
       body: parsedBody.content[0].text,
     };

     // Return the response
     return res;
   }

The request function constructs the AI prompt and sends it to Amazon Bedrock. The response function parses the generated recipe from the returned JSON.

Step 2: Add Amazon Bedrock as a Data Source

Open the amplify/backend.ts file and add the following code. Save the file.

The code adds an HTTP data source for Amazon Bedrock to your API and grants it the necessary permissions to invoke the Claude model.

   import { defineBackend } from "@aws-amplify/backend";
   import { data } from "./data/resource";
   import { PolicyStatement } from "aws-cdk-lib/aws-iam";
   import { auth } from "./auth/resource";

   const backend = defineBackend({
     auth,
     data,
   });

   const bedrockDataSource = backend.data.resources.graphqlApi.addHttpDataSource(
     "bedrockDS",
     "https://bedrock-runtime.us-east-1.amazonaws.com",
     {
       authorizationConfig: {
         signingRegion: "us-east-1",
         signingServiceName: "bedrock",
       },
     }
   );

   bedrockDataSource.grantPrincipal.addToPrincipalPolicy(
     new PolicyStatement({
       resources: [
         "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-sonnet-20240229-v1:0",
       ],
       actions: ["bedrock:InvokeModel"],
     })
   );

This code integrates Amazon Bedrock into your backend, enabling the serverless app to invoke the Claude 3 Sonnet model.

Task 4: Deploy the Backend API

Step 1: Set Up Amplify Data

In the amplify/backend.ts file, define the schema and backend resources as follows:

   import { type ClientSchema, a, defineData } from "@aws-amplify/backend";

   const schema = a.schema({
     BedrockResponse: a.customType({
       body: a.string(),
       error: a.string(),
     }),

     askBedrock: a
       .query()
       .arguments({ ingredients: a.string().array() })
       .returns(a.ref("BedrockResponse"))
       .authorization((allow) => [allow.authenticated()])
       .handler(
         a.handler.custom({ entry: "./bedrock.js", dataSource: "bedrockDS" })
       ),
   });

   export type Schema = ClientSchema<typeof schema>;

   export const data = defineData({
     schema,
     authorizationModes: {
       defaultAuthorizationMode: "apiKey",
       apiKeyAuthorizationMode: {
         expiresInDays: 30,
       },
     },
   });

Step 2: Deploy Cloud Resources

Open a new terminal window, navigate to your app's project folder (ai-recipe-generator), and run the following command to deploy cloud resources into an isolated development space:

   npx ampx sandbox

This command sets up a sandbox environment where you can quickly iterate on your changes.

After the sandbox has been fully deployed, you will see a confirmation message in the terminal, and an amplify_outputs.json file will be generated and added to your project.

The deployment process is now complete, and you can begin interacting with your serverless backend.

Here's how you can present Task 5: Build the Frontend and the challenges faced in the process for your blog:

Task 5: Build the Frontend

Step 1: Install the Amplify Libraries

Open a new terminal window, navigate to your project’s root folder (ai-recipe-generator), and run the following command to install the libraries:

   npm install aws-amplify @aws-amplify/ui-react

Step 2: Style the App UI

Next, we’ll style the frontend to create a clean, modern interface. We'll focus on centering the layout and styling the form where users will input their ingredients.

Open ai-recipe-generator/src/index.css, and update it with the following code to set global styles and center the UI:

   :root {
     font-family: Inter, system-ui, Avenir, Helvetica, Arial, sans-serif;
     line-height: 1.5;
     font-weight: 400;
     color: rgba(255, 255, 255, 0.87);
     max-width: 1280px;
     margin: 0 auto;
     padding: 2rem;
   }

   .card {
     padding: 2em;
   }

   .box:nth-child(3n + 1) {
     grid-column: 1;
   }
   .box:nth-child(3n + 2) {
     grid-column: 2;
   }
   .box:nth-child(3n + 3) {
     grid-column: 3;
   }

This CSS will help ensure that the app's layout is centered, the font is legible, and the UI components have a clean, consistent style.

Next, open ai-recipe-generator/src/App.css and update it with the following code to style the ingredient input form:

   .app-container {
     margin: 0 auto;
     padding: 20px;
     text-align: center;
   }

   .header-container {
     padding-bottom: 2.5rem;
     text-align: center;
   }

   .main-header {
     font-size: 2.25rem;
     font-weight: bold;
     color: #1a202c;
   }

   .main-header .highlight {
     color: #2563eb;
   }

   .description {
     font-weight: 500;
     font-size: 1.125rem;
     max-width: 65ch;
     color: #1a202c;
   }

   .form-container {
     margin-bottom: 20px;
   }

   .search-container {
     display: flex;
     flex-direction: column;
     gap: 10px;
     align-items: center;
   }

   .wide-input {
     width: 100%;
     padding: 10px;
     font-size: 16px;
     border: 1px solid #ccc;
     border-radius: 4px;
   }

   .search-button {
     width: 100%;
     max-width: 300px;
     padding: 10px;
     font-size: 16px;
     background-color: #007bff;
     color: white;
     border: none;
     border-radius: 4px;
     cursor: pointer;
   }

   .search-button:hover {
     background-color: #0056b3;
   }

   .result-container {
     margin-top: 20px;
     transition: height 0.3s ease-out;
     overflow: hidden;
   }

   .loader-container {
     display: flex;
     flex-direction: column;
     align-items: center;
     gap: 10px;
   }

   .result {
     background-color: #f8f9fa;
     border: 1px solid #e9ecef;
     border-radius: 4px;
     padding: 15px;
     white-space: pre-wrap;
     word-wrap: break-word;
     color: black;
     font-weight: bold;
     text-align: left;
   }

This will ensure that the ingredients form and result display are properly styled.

Step 3: Implement the UI

Now, let’s build the main React component for the app. We’ll integrate the AWS Amplify Authentication components for user sign-up, sign-in, and password recovery.

Open ai-recipe-generator/src/main.tsx and update it with the following code. This will use the Amplify Authenticator component to wrap your app and provide a complete authentication flow:

   import React from "react";
   import ReactDOM from "react-dom/client";
   import App from "./App.jsx";
   import "./index.css";
   import { Authenticator } from "@aws-amplify/ui-react";

   ReactDOM.createRoot(document.getElementById("root")!).render(
     <React.StrictMode>
       <Authenticator>
         <App />
       </Authenticator>
     </React.StrictMode>
   );

The Authenticator component will handle user authentication, including sign-up, sign-in, and MFA (Multi-Factor Authentication).

Next, open ai-recipe-generator/src/App.tsx and update it with the following code to implement the form for ingredient submission and the logic for querying the askBedrock function.

   import { FormEvent, useState } from "react";
   import { Loader, Placeholder } from "@aws-amplify/ui-react";
   import "./App.css";
   import { Amplify } from "aws-amplify";
   import { Schema } from "../amplify/data/resource";
   import { generateClient } from "aws-amplify/data";
   import outputs from "../amplify_outputs.json";

   import "@aws-amplify/ui-react/styles.css";

   Amplify.configure(outputs);

   const amplifyClient = generateClient<Schema>({
     authMode: "userPool",
   });

   function App() {
     const [result, setResult] = useState<string>("");
     const [loading, setLoading] = useState(false);

     const onSubmit = async (event: FormEvent<HTMLFormElement>) => {
       event.preventDefault();
       setLoading(true);

       try {
         const formData = new FormData(event.currentTarget);

         const { data, errors } = await amplifyClient.queries.askBedrock({
           ingredients: [formData.get("ingredients")?.toString() || ""],
         });

         if (!errors) {
           setResult(data?.body || "No data returned");
         } else {
           console.log(errors);
         }
       } catch (e) {
         alert(`An error occurred: ${e}`);
       } finally {
         setLoading(false);
       }
     };

     return (
       <div className="app-container">
         <div className="header-container">
           <h1 className="main-header">
             Meet Your Personal
             <br />
             <span className="highlight">Recipe AI</span>
           </h1>
           <p className="description">
             Simply type a few ingredients using the format ingredient1,
             ingredient2, etc., and Recipe AI will generate an all-new recipe on
             demand...
           </p>
         </div>
         <form onSubmit={onSubmit} className="form-container">
           <div className="search-container">
             <input
               type="text"
               className="wide-input"
               id="ingredients"
               name="ingredients"
               placeholder="Ingredient1, Ingredient2, Ingredient3,...etc"
             />
             <button type="submit" className="search-button">
               Generate
             </button>
           </div>
         </form>
         <div className="result-container">
           {loading ? (
             <div className="loader-container">
               <p>Loading...</p>
               <Loader size="large" />
               <Placeholder size="large" />
               <Placeholder size="large" />
               <Placeholder size="large" />
             </div>
           ) : (
             result && <p className="result">{result}</p>
           )}
         </div>
       </div>
     );
   }

   export default App;

The app allows users to input a list of ingredients, submits the request to the backend via Amplify and Amazon Bedrock, and then displays the generated recipe.

Step 4: Run and Test the App

Open a new terminal window and navigate to your project’s root directory (ai-recipe-generator), then run:

   npm run dev

Visit the local host URL to open the app in your browser and test it.

Step 5: Deploy the App

Once you’ve confirmed that the app is working as expected locally, it’s time to deploy it to AWS Amplify.

In the terminal, commit your changes:

   git add .
   git commit -m 'connect to bedrock'
   git push origin main

Go to the AWS Amplify console, and your app should automatically build and deploy.

After deployment, you can access your live app at the provided Amplify URL.

Challenges Faced and Solutions

Integrating Bedrock API with Lambda: The response latency from Amazon Bedrock sometimes caused delays.
- Solution: Error handling and retry logic were implemented to handle API delays and ensure smoother user experiences.
Frontend and Amplify Configuration: Understanding the integration of Amplify DataStore with AppSync for real-time data updates was tricky.
- Solution: Amplify’s documentation and UI components helped speed up the integration process and simplified backend connectivity.

Conclusion

This project provided valuable experience in creating a web application with user authentication and data storage using AWS Amplify. This serverless application setup is ideal for rapid development and scalability, making it suitable for various applications, from prototypes to production-level solutions. AWS Amplify enables efficient web application management and backend integration, covering a range of use cases.

Shubham Murti — Aspiring Cloud Security Engineer | Weekly Cloud Learning !!

Let’s connect: Linkdin, Twitter, Github