Forem: Nandini Rajaram

Deploy Snake and Ladder Game on Amazon ECS

Nandini Rajaram — Mon, 05 Jan 2026 07:26:02 +0000

Now that our Snake and Ladder Docker image is available in Amazon ECR, the next step is to deploy it using Amazon ECS Express.
ECS Express is the fastest way to run containerized applications on AWS without manually creating clusters, task definitions, or services.

What is ECS Express?

ECS Express is a simplified deployment option in Amazon ECS that:

Automatically creates the ECS cluster
Creates the task definition
Sets up the service
Deploys your container using AWS Fargate

Deploying a Snake & Ladder Game on Amazon ECS using AWS Fargate

Managing servers can quickly become overwhelming—even for simple applications. In this blog, we’ll deploy a Snake & Ladder game on Amazon ECS using AWS Fargate, where AWS handles the infrastructure while we focus on the application.

Create an ECS Cluster

Open Amazon ECS Console
Click Clusters → Create cluster

Choose:

Cluster name: snake-ladder-cluster

Infrastructure: AWS Fargate

Click Create

Fargate removes the need to manage EC2 instances—AWS takes care of scaling and capacity.

Create a Task Definition

A task definition is a blueprint for running containers in ECS.

Task basics

Family: snake-ladder-task

Launch type: AWS Fargate

Compute configuration

CPU: 0.25 vCPU

Memory: 0.5 GB or 1 GB

These values are perfect for a lightweight game app.

Do We Need a Task Role?

When creating the ECS task definition, you might notice that we did not select a Task Role.

This is intentional.

The Task Role is required only when the application running inside the container needs to access AWS services such as:

Amazon S3, DynamoDB, SQS, Secrets Manager, Parameter Store

Our Snake & Ladder game is a standalone web application and does not interact with any AWS services. Therefore, assigning a task role is unnecessary.

However, we did assign a Task Execution Role , which allows ECS to:

Pull the Docker image from Amazon ECR
Push container logs to Amazon CloudWatch

Selecting “Create default role” lets ECS automatically create the required Task Execution Role for you.

When enabled, AWS:

Creates the ecsTaskExecutionRole

Attaches AmazonECSTaskExecutionRolePolicy

Configures the trust relationship for ECS

👉 No manual IAM setup required.

Container configuration

Container name: snakeAndladder

Image URI:

.dkr.ecr..amazonaws.com/snake-ladder-game:latest

Port mapping:

Container port: 80(or your app’s port)

For Logging, Keep the default values

Click Create button to create the Task Definition

Create ECS Service (Fargate)

Now we connect everything together.

Open ECS → Clusters → snake-ladder-cluster

Click Create service

Service settings

Task definition: Choose the Task definition created in the above step

Service name: snake-ladder-service

Desired tasks: 1
Leave the rest to default values

You can use a Load Balancer if needed, Here I am proceeding without a Load Balancer as I have only 1 desired task

Click Create service 🚀

Wait for task deployment to complete, Navigate to Task Tab, You should be able to see the task

Once the Task is opened, Go to networking tab and click on the public IP to open the application

Access the Snake & Ladder Game 🎉

🎮 Your Snake & Ladder game is now live on AWS!

Pushing Snake and Ladder Docker Image to Container Registry

Nandini Rajaram — Sat, 03 Jan 2026 18:40:22 +0000

So far, we have successfully Dockerized the Snake and Ladder game and verified that it runs perfectly on our local machine. That’s a great first milestone.

But running a Docker image only on one machine isn’t enough.

In real-world applications, we need a place where our Docker images can be stored, shared, and reused—so they can be pulled by servers, cloud services, or container platforms like ECS and Kubernetes.

In this next step, we will push our Docker image to a container registry, making it accessible beyond our local system. We’ll also understand the two most commonly used Docker repositories:

Docker Hub – for public and learning use cases

Amazon ECR (Elastic Container Registry) – for secure, production-ready AWS deployments

What Is Docker Hub?

Docker Hub is a cloud-based repository where you can store, share, and download Docker images.

Think of Docker Hub as GitHub for Docker images.

What Docker Hub Does?

Stores Docker images online
Lets you push your images
Lets others pull and use them
Hosts thousands of pre-built images like:
- nginx, mysql, node, python

🔹 Key Features

Public and private repositories
Easy to use for beginners
Free public image hosting
Widely used for learning and open-source projects

Creating a Repository in Docker Hub

First, We need to create a Repository to DockerHub. If you already have an account in Dockerhub. You can simply Sign In else do a Sign Up

I am Signing In, and Next step is to create a repository

Click Create Repository button, Provide a name and a short description and

Click Create button. Remember, we have chosen a public repository

We have successfully created a repository in Docker Hub

Push the Snake and Ladder Docker Image to Repository in Docker Hub

Lets come back to ec2 console,

Come back to your EC2 instance and connect to it using EC2 Instance Connect.
Type docker images command to see the available images.
Now Authenticate Docker with Docker Hub using the EC2 command line.
Run the following command:

docker login
When prompted, enter your Docker Hub username and password.

After a successful login, you will see a confirmation message indicating that Docker is now authenticated with Docker Hub.

Now, let’s tag the Snake and Ladder Docker image so it can be pushed to Docker Hub.

Docker Hub requires images to follow a specific naming format:
docker-username/repository-name:tag

In our case, we will tag the image as:

nandinivijayr/snake-ladder-game:latest

Run the following command:

docker image tag snake-ladder-game:latest nandinivijayr/snake-ladder-game:latest

By tagging the image in this format, Docker knows which Docker Hub repository the image belongs to, making it ready to be pushed to Docker Hub in the next step.
Now lets verify if the image is tagged properly by using Docker images command

Now lets push the image to Docker hub using the below command
docker push nandinivijayr/snake-ladder-game

Push the Docker Image to Amazon ECR

Amazon ECR (Elastic Container Registry) is a fully managed container image registry service provided by AWS. It is used to store, manage, and secure Docker container images so they can be easily deployed on AWS services like ECS, EKS, and Fargate.

In simple terms:
👉 ECR is AWS’s private Docker image repository.
Why Do We Need ECR?
When applications are containerized using Docker, the images need a secure place to live so cloud services can pull and run them. Amazon ECR provides that storage inside AWS, with built-in security and scalability.

Key Features of Amazon ECR

Secure by Default
Repositories are private by default
Access is controlled using IAM roles and policies
No usernames or passwords required
Deep AWS Integration
Works seamlessly with: Amazon ECS, Amazon EKS, AWS Fargate
Images can be pulled directly by AWS services
Fully Managed
No servers to manage
Automatically scales
Highly available and reliable
Image Management
Version images using tags
Scan images for vulnerabilities
Store multiple versions safely

Now, Lets see how we can push image to ECR

Create an ECR Repository

Go to home page of aws console and Navigate to the ECR service

Create a private ECR repository using AWS Console by clicking Create Repository button

Provide a repository name and scroll down

Go with the default values and click create button

Wait for a couple of minutes, You can see the new repository entry under Private Repositories

Come back to your EC2 instance and connect to it using EC2 Instance Connect.

Now Lets Authenticate Docker to ECR using the below command

aws ecr get-login-password --region <region> | \ docker login --username AWS --password-stdin <account-id>.dkr.ecr.<region>.amazonaws.com

Authenticate Docker with Amazon ECR from EC2

Before pushing a Docker image to Amazon ECR, Docker must be authenticated with AWS.
The recommended and secure way is to use an IAM role attached to the EC2 instance.

Create an IAM Role for EC2

Go to IAM → Roles → Create role

Choose AWS service → EC2

Attach the policy:

AmazonEC2ContainerRegistryFullAccess

Create the role by providing a role name and clicking Create Role button

Attach the Role to EC2

Go to EC2 → Instances

Select your instance

Actions → Security → Modify IAM role

Here Select the I A M role that was created in the previous step and click Update IAM Role button

Wait 1–2 minutes for changes to apply.

Verify the Role

On the EC2 instance, run:

aws sts get-caller-identity

Sample output:

{
"Account": "123456789",
"Arn": "arn:aws:sts::123456789:assumed-role//i-0fea76487ea9663b1"
}

Docker uses a temporary token generated by AWS.

aws ecr get-login-password --region ap-south-1 | \ docker login --username AWS --password-stdin \ 123456789.dkr.ecr.ap-south-1.amazonaws.com

Expected Output
Login Succeeded

Note: Replace 123456789 with your own AWS Account ID.
Do not change the username or password values — the username must be AWS, and the password is automatically passed using --password-stdin.

Now push the image to ecr respository that was created

Push Docker Image to Amazon ECR Repository

Now that Docker authentication is successful, the next step is to push the Docker image to the Amazon ECR repository created earlier. This makes the container image available for deployment using AWS services such as Amazon ECS, App Runner, or EKS.

Let’s tag the Snake and Ladder Docker image with the ECR repository URI so it’s ready to be pushed.

Use the following command to tag
docker tag <IMAGE NAME> <ECR Repository URI>

IMAGE NAME is snake-ladder-game which we built earlier. You can get it using docker images command

Now let us push the tagged image using docker push command
docker push <ECR Repository URI>

Docker will upload the image layers to Amazon Elastic Container Registry (ECR). Depending on the image size, this process may take a few moments.

Verify the Image in Amazon ECR

After the push completes:

Open the AWS Management Console

Navigate to Amazon ECR → Repositories

Select your repository

Confirm that the image with the latest tag is listed

Dockerize a Snake and Ladder Game Application in EC2

Nandini Rajaram — Fri, 02 Jan 2026 19:29:46 +0000

Prerequisite

Refer https://dev.to/aws-builders/docker-fundamentals-on-amazon-ec2-a-practical-introduction-2l60

The html code which is used can be accessed here https://github.com/nandinicloudpractioner/snakeandladdergamewebapplication/blob/main/snake-and-ladder.html

Lets Dockerize the snake and ladder game application

I have the html file snake and ladder.html file which I have created using Amazon Q cli

Now Lets Dockerize that

Lets begin with creating the directories and storing the code

Create a folder in ec2 instance and store the code files

mkdir snake-ladder-game

Move into the folder
cd snake-ladder-game

Now we need to create two files and store it inside our folder
snake-ladder-game/ │── snake-and-ladder.html │── Dockerfile
To create a file in ec2, Lets use a nano editor
Lets first create snake-and-ladder.html
Type the command to open nano editor
nano snake-and-ladder.html
Copy the code which is available in https://github.com/nandinicloudpractioner/snakeandladdergamewebapplication/blob/main/snake-and-ladder.html and press
Ctrl + O
Nano will ask:

File Name to Write: snake-and-ladder.html

👉 Just press Enter to confirm.
Exit nano by pressing

_Ctrl + X_

Now, Create a Dockerfile for the HTML Game
Create a Dockerfile:

nano Dockerfile
Add the below content

# Use lightweight Nginx image
FROM nginx:alpine

# Remove default nginx website
RUN rm -rf /usr/share/nginx/html/*

# Copy our game HTML file to nginx web directory
COPY snake-and-ladder.html /usr/share/nginx/html/index.html

# Expose port 80
EXPOSE 80

Copy the code and press
Ctrl + O
Nano will ask:

File Name to Write: Dockerfile

👉 Just press Enter to confirm.

Exit nano by pressing

_Ctrl + X_

Explanation of the Dockerfile

nginx:alpine – lightweight web server image
COPY – moves your HTML file into the container
EXPOSE 80 – opens the web server port

This Dockerfile turns your HTML file into a runnable web application.

Build the Docker Image

Run the following command from the project directory:

docker build -t snake-ladder-game .

This creates a Docker image named snake-ladder-game.

Lets check if the image is ready by using the command
docker images

Yes, snake-ladder-game image is ready

Run the Docker Container

Start the container using:

docker run -d -p 80:80 snake-ladder-game

80 is the local port
80 is the container port

Lets check the run the command

docker ps
to see active containers

Open your browser using:

http://<EC2-PUBLIC-IP>

Make sure your EC2 security group allows inbound traffic on port 80.

Docker Fundamentals on Amazon EC2: A Practical Introduction

Nandini Rajaram — Thu, 01 Jan 2026 13:51:16 +0000

Lets Start with a simple story
Imagine this.

A developer builds an application on their laptop. It runs perfectly. Feeling confident, they send it to the testing team.

But then comes the message:

“The app is not working on our server.”

The developer checks everything and replies:

“But it works on my machine!”

If this sounds familiar, you’ve just met one of the biggest problems in software development.

What Actually Went Wrong?

The application depended on:

A specific version of a programming language

Certain libraries and tools

Configuration files set up on the developer’s laptop

The server had a slightly different setup—and that was enough to break the app.
🐳 Enter Docker

Docker steps in like a smart packaging system.

Instead of sending just the application code, Docker packs the app together with everything it needs to run—the runtime, libraries, and configurations—into a single unit called a container.

Now when the app moves:

From laptop → test server

From test server → production

From local system → cloud

…it behaves exactly the same everywhere.

Think of Docker Like This

Imagine you’re sending a homemade dish to a friend.
Instead of sending just the recipe and hoping they have the right ingredients and stove, you send the entire ready-to-eat meal in a sealed box.

That’s Docker

Install Docker on ec2

Launch an Ec2 Instance with Amazon AMI and do an instance connect to get connected to terminal
Execute the following commands

sudo yum update -y
sudo yum install docker -y
sudo systemctl start docker
sudo systemctl enable docker
sudo usermod -aG docker ec2-user  # Allow running Docker without sudo

Verify Docker Installation

docker --version Exit and reconnect to ec2 Instance connect and Run the below command docker ps # Should return an empty list

Running a Container

The Docker run command creates and starts a container from a specified image. For example, to start an redis container, simply execute:

docker run redis

If the redis image already exists on your host, Docker immediately starts a new container. However, if the image is not present, Docker will pull it from Docker Hub. Once the image is downloaded, any subsequent run command will use the cached image.

To count the total number of containers running on the host

Run the command
docker ps
You can view running containers using the docker ps command. This command provides an overview, including container IDs, image names, statuses, and container names.

Run the command
docker ps -a to list all containers, including those that have stopped or exited

To stop the container

Run the command

docker stop <container_name>
or

docker stop <container_id> (first few letters of container id that can uniquely identify the container would be good)

To remove the container

docker rm <container_id>(first few letters of container id that can uniquely identify the container would be good)
Note: Containers should be stopped before removing them

To count the number of images available on the host

Run the command
docker images

To pull an image

docker pull <image_name>:<tag_name>
Note: if tag not mentioned, image with latest tag will be pulled

To delete an image

To delete an image no longer needed, ensure no container is using it.
Stop and remove all the containers created from that image and then
Run the command.
docker rmi <image_name>
or
docker rmi <image_id>

To run the container in detached mode

To run the web application in detached (background) mode, add the -d option

docker run -d <image_name>
docker run -d --name=webapp1 nginx:1.14-alpine

To host a local registry

`docker run -d --name my-registry -p 80:5000 --restart always registry:2'

To pull ngnix image

docker pull nginx:latest

To tag the image with a new name

docker image tag nginx:latest localhost:80/nginx:latest

To push the image

docker push localhost:80/nginx:latest

To verify image pushed

curl -X GET localhost/v2/_catalog

Port mapping the containers for Webapplications

To expose the application externally, you must map a free host port to the container’s port using the -p parameter.

docker run -d --name=flask -p 80:80 nandinivijayr/myflaskapplication
The above command will map the container port 80 to host port 80 and -d indicates that we are running the container in a detached mode

Inspecting the containers and images

For detailed information about a particular container or an Image the docker inspect command provides comprehensive configuration details in JSON format.
docker inspect container_id/ container_name will give details about the container
docker inspect image_id / image_name will give details about the image

To retrive container logs

docker logs <container_name/ container_id >

Question time?

Run a container with mysql image and name it

From Data to Magic: Generative BI with Amazon Quick Suite Integrated with Amazon Q

Nandini Rajaram — Mon, 08 Dec 2025 18:54:06 +0000

Amazon Quick Suite

Amazon QuickSuite is a powerful cloud-based analytics tool that makes data access and visualisation easy and accessible for everyone.

Preview of Quicksuite once Signed In

Creation of Data set
In Amazon QuickSuite, a dataset is the collection of data you import or connect to—such as files, databases, or applications—that you prepare and use to create analyses and visuals.

Let’s get hands-on with AWS Quick Suite by starting with the creation of our first dataset.

You can upload a csv file by clicking upload file

You can create a dataset by uploading a file, or you can set up a data source, which connects Quick Suite to data from external applications or databases.

I am uploading a simple csv file which contains data regarding Mental Health and Social Media

In order to do some alterations on the data like renaming the columns or adding new column, we need to click Edit/ preview data

Here you can add calculated fields based on formulas

For example, I am creating a new column user type based on age, if age greater than 30, User type will be adult and less than 30, User type will be Youth
Formula is ifelse({Age} < 30, "Youth", "Adult")

Here, You can see a new column user type getting generated on preview
based on our formula

You can also perform several other operations on the dataset like renaming the column, changing the data type and so on.

Now Lets save and publish the data set

Creation of Analysis
In Amazon QuickSuite, an analysis is the workspace where you explore your data—adding datasets, creating visuals (like charts and tables), applying filters, and building insights before publishing them as dashboards.

Now, we have the dataset ready, its time to create analysis

Click on the Quick Suite logo and come to the base page, select Analysis from the left pane

Click Create Analysis button and Select the Dataset that we recently created and Click Select button

Here, You will be asked to create a new sheet for analysis.
Click Create button and proceed.

Now, We should be able to build visuals using Prompts

Click Build Visual and give a prompt-(This is powered by Amazon Q)
I am going to get a pie chart for the user type values indicating the percentage of Adults and Youths

If satisfied with the response, You can add it to Analysis by clicking ADD TO ANALYSIS button and It will be added to the sheet

You can also build required charts manually by picking the charts and configuring them from the visuals pane

Now, Let me add some more visuals

You can also modify the visual using prompts.
For example, if you want to change a vertical bar chart to a horizontal bar chart, you can simply ask through a prompt.

Here You go, The sheet with some more visuals added

Manage Q & A

Amazon Quick suite offers a useful option called “Manage Q&A,” which lets you control whether users can ask natural-language questions in your analysis and choose which data source—either a dataset or a linked topic—Q&A should use to generate answers.

Click on Manage Q & A button and select Use datasets for Build visual and Click Apply changes button

By selecting this option, we are instructing the analysis to use the datasets directly attached to it for both building visuals and answering Q&A queries.

Publishing the dashboard
Now, Its time to publish the dashboard

Click the Publish Button on the top right

Make necessary configurations and click publish button

Now, Click on the Amazon Quick Suite Logo to navigate to the base page.
Select Dashboards from Left pane

You should be able to see the recently published dashboard

Topics in Amazon Quick Suite

A topic is a curated data layer in Quick Suite that makes natural-language querying (Q) more accurate, secure, and easier for business users. It help users get relevant answers through natural language queries.

Note: If you want to define a Topic, A subscription to Quick Suite Author Pro is required

Lets start by defining a topic for our current dataset to analyse further.
On the Left pane, click Topics and then click create topic button.
Specify a topic name and description and Check Enable Q & A and Click Continue

Now Select the dataset and click Create button

Please wait for couple of minutes while our topic is getting created

Here You go, Our topic is ready.

Refresh Schedule in Topic
A Topic is built on a dataset. The refresh schedule decides how frequently that dataset is refreshed so your Topic always answers questions using the latest data.

Now let us learn about how to setup refresh schedules on Topic

Navigate to the corresponding topic

Go to Data tab and click on Datasets

Now You can create a topic refresh schedule from here

You can also refresh the data manually

Data Fields in Topic

Data fields are the columns from your dataset that you add to a Topic so Quick Suite Q can understand them.
These fields help Q answer natural-language questions correctly.

Now, Lets work on the Data fields
Here You can select the field you want to include and add some synonyms

Synonyms are alternative words or phrases that you assign to a data field, named entity, or metric so that Quick Suite Q can understand different ways users might ask the same thing.

They improve natural-language understanding.

NAMED ENTITY
Named Entities are custom business terms you define inside a Topic so Quick Suite Q can understand natural language questions more accurately.

They represent real-world concepts that may not exist as a single column in your dataset.

For example I am creating a Survey Participant with the following basic fields

Now its Question Time

Click on Ask about this topic and Ask some questions

The good thing is that You will get some questions as suggestions

Post some questions and verify the answer and Click the button Mark as verified. You can also change the interpretation by clicking the pencil icon

Now In the suggested Questions tab, Inside Verified Tab, You will be able to view the Questions which you have posted and marked as verified

You can also view the Questions came as suggestions on AI generated section

Quick Come back to Analysis

Now Coming back to Analysis, We had a Manage Q & A button and on clicking that we get one more option Use a linked topic for Build visual and Q&A. When user selects this option, Q&A uses Topic instead of your dataset ensuring better accuracy, cleaner data, and consistent business logic.

Scenarios in Quick Suite

In Amazon Quick Suite, Scenarios are guided, AI-powered question prompts that help users explore data quickly using natural language.

They act like predefined, business-friendly question templates that users can click on instead of typing their own queries.

A Scenario is basically a:

Pre-built conversational workflow
Designed around a specific business use case
Powered by Q (natural language) + your dataset

That guides the user through steps such as:
“Analyze trends → explore causes → compare segments → generate insights → recommend actions”

So instead of users typing questions freely, a scenario provides a structured set of prompts tailored to your business problem.

Lets create a Scenario and Move further

Click the Quick sight logo and go the base page. Click on Scenarios from the left pane and Click Create Scenario button

Provide a description of the business problem and add data
Data can be added by uploading files or Selecting from Quick Sight Dashboards

Now I am going to select the data from Dashboards, Let me click on the Find Data button and Select the Dashboard we published and select required visuals. Click Add button to proceed further

Now, You can Preview the data and Select the fields you want from the dataset and click Add Data button to proceed further

Lets wait for a couple of minutes and click Start Analysis button

Provide a prompt and click submit button to start the thread

I gave a prompt

Identify factors affecting happiness index.
Lets See the result

It gave me a lot of information

You can continue the thread by clicking the + button below the generated result and describe what you want to analyze next.

You can also start a new thread from the top to run a fresh analysis.

The good thing is that you will be also getting prompt suggestions and you can use them if you want

Let me also Rename to Scenario from New Scenario to Social Media Overview

You can Share the Scenario created to team members/ Users by clicking Share button on the top right

Now, Its time for Story Telling

Stories

A Story in Quick Suite is a curated, step-by-step presentation created from visuals in your analysis.
It lets you:

Highlight key insights
Walk users through findings in order
Add explanations or commentary
Turn your analysis into a narrative experience.

Lets create a story about our Mental Health and Social Media Analysis

Click on the Quick Suite Logo and Navigate to Stories from Left Pane

Click Create Story button to proceed further

Here Describe Your story as well as add some visuals

Here You should be able to add Visuals from the Dashboard you have published

Click Build button to proceed further

Wait for a couple of minutes to get our story served

Here you go, We have a beautiful story served

You should be able to edit the story according to your taste and Preview it as well before getting published

Lets go back to the Story editor and do some edittings
You can customize the story theme and adjust the animations for how each block appears.

You can add your own block by clicking the + button add content to it

You can also move the block up / down
duplicate it or delete it

Another fantastic feature is You can alter the lines of each block to longer or shorter using edit the narrative option given by Amazon Q

and Finally You should be able to publish and share the Story by clicking the Share button and selecting the users, Click Publish and Share button

Quickly Set Up a GraphQL API for DynamoDB CRUD operations with Just a Few Clicks Using AWS AppSync

Nandini Rajaram — Sun, 10 Nov 2024 18:23:39 +0000

What is GraphQL?

GraphQL is a query language for APIs and a runtime for executing those queries. It was developed by Facebook in 2012 and released publicly in 2015

Why GraphQL?

Instead of an API where you hit a URL and accept whatever data comes back, GraphQL allows you to ask for specific data, giving clients more control over what information is sent.

A Practical Example

For example, you may say, "Hey REST API, give me the titles of the books available on Cloud Tuner Online Library," in order to retrieve data from the Rest API. The way this works is that you contact a specific endpoint, or URL, in our instance the Cloud Tuner Online Library, and that URL controls the data that returns. When you retrieve a URL using the REST API, it usually returns something in the form of a Javascript object or JSON with data on it. This leads to either repeated journeys (requests) to satisfy different queries, or undesired data that we have to sift through to retrieve our needed data. But GraphQL isn't like that.
Rather than using an API where you click a URL and consent

Queries and Mutations

Query
In GraphQL, a query is used to fetch specific data from the server. You define the fields you want to retrieve, and the server returns only that data. Let's look at a typical GraphQL query example and break it down.

Example Query
Imagine you want to get a list of books, their titles, authors, and publication years from the "Cloud Tuner Online Library." Here's how you would write that query in GraphQL:

 query {
  books {
    title
    author {
      name
    }
    publicationYear
  }
}

query: This is the keyword that tells GraphQL you’re making a read operation to fetch data.
books: This is the field you're querying. In this case, it refers to the list of books available in the library.
title, author, publicationYear: These are the specific fields you want to retrieve for each book.
author { name }: This is an example of requesting a nested field. Here, you're asking for the name field inside the author object associated with each book.

Mutation
A mutation in GraphQL is used to modify or create data on the server. While queries are used to fetch data, mutations are used for actions like adding, updating, or deleting records.

Example Mutation
Let’s say you want to add a new book to the "Cloud Tuner Online Library." Here’s how you would write a GraphQL mutation to do that:

mutation {
  addBook(title: "New Book Title", author: "New Author", publicationYear: 2024) {
    id
    title
    author
    publicationYear
  }
}

mutation: This keyword indicates that you're making a write operation, meaning you're modifying data on the server.
addBook: This is the mutation operation name. It corresponds to a mutation defined on the server that handles the logic for adding a book to the library.
Arguments: Inside the parentheses, you provide the necessary inputs for the mutation, such as the title, author, and publicationYear of the book.
Fields: After the mutation is successful, you can request specific fields to be returned. Here, we're asking for the id, title, author, and publicationYear of the newly added book.

CRUD APIs with AWS AppSync and DynamoDB

AWS AppSync is a fully managed, pay-per-use GraphQL service that allows developers to create and test web and mobile applications utilizing real-time data. AppSync allows developers to construct an API that is available via a single endpoint and uses GraphQL as a unified query language. AppSync manages data requests and responses, as well as connections to one or more data sources, such as AWS Lambda, AWS DynamoDB, and HTTP APIs.

Amazon DynamoDB is a cloud-based NoSQL database that is ideal for anyone looking for a dependable and completely managed NoSQL solution. DynamoDB is intended to allow automated storage scaling with low latency. It is especially beneficial when your application must read and store large volumes of data while maintaining speed and dependability (Amazon uses database replicas in three separate Availability Zones). Amazon DynamoDB is completely managed. Simply select an AWS region and define the necessary indexes for each table you will create, and Amazon will handle the rest.

Here We will work with the AWS AppSync interface via the AWS console to build a GraphQL API backed by DynamoDB datasource

Prerequisites

Books Table in DynamoDB with Items added.

Please refer the below link for creating the DynamoDB table
Cook a recipe with AWS: Dynamo DB Table

AWS AppSync has the capability to instantly generate a GraphQL schema and link resolvers to an already-existing data source, like an AWS DynamoDB database. Developers can first concentrate on database architecture rather than the GraphQL schema thanks to this functionality. For users who are unfamiliar with GraphQL, the automatically produced schema and resolvers can also serve as a foundation that can be readily built upon.

Navigate to Appsync Console

Click Create API button

Under Select API type, select GraphQL APIs, under GraphQL API Data Source, select Start with a DynamoDB table and Click Next button

Enter An API name (I have given BooksAPI for the API name)

In the Import DynamoDB Table form, configure the following, then click Next button
- Region: Select US-WEST-2 from the drop-down menu
- Table name: Select BooksTable
- Create or use an existing role: You can create an new role A role with following permission will be created for you

Role grants AppSync the permissions it needs to perform create, read, update, and delete (CRUD) operations on data stored in DynamoDB.

Click next button after filling the details

Now On the Model information step, configure the following:
- Model Name - Iam giving the name as BooksTable
- The Fields section can be populated with the Table Fields, including details such as the Field Type, whether the Field is an array, and if the Field is mandatory. These details should align with the DynamoDB Books Table. The Primary Key Field is autopopulated.

Configure the remaining Fields aligning with Books Table of DynamoDB

Under Additional settings, select Velocity Template Language (VTL), then click Next, review the information and then click Create API:

Now You can see the magic

Using this Book model, AppSync will create a GraphQL schema that includes queries, mutations, and subscriptions to handle the basic CRUD operations for your API( Creation of items into DynamoDB, Reading items from DynamoDB,Updating Item in DynamoDB and Deletion of items from DynamoDB)
AppSync will also generate the necessary resolvers and mapping templates to manage the data flow between the client and the BooksTable data source.

Here You can see the BooksAPI created

Click Schema and You will be navigated to the view the Schema of BooksAPI

Click Data sources to view the DynamoDB Books table Data source

Click Queries in the left navigation pane and you'll be redirected to the Queries page for BooksAPI:

Here You will be able to see the Query editor which can be used for writing, validating, and testing GraphQL operations, including queries, mutations, and subscriptions.

Fetching The Items from BooksTable of DynamoDB using GraphQL Query

Click Run Button and Select listBooksTables

This will execute the listBooksTables query which will fetch all items from BooksTable of DynamoDB as Data Objects in JSON Format

A data object is returned that includes an items array. This array consists of all three books stored in BooksTable. Each item has all available fields returned in the output, which is declared in the listBooks query, within items.

Here You can see the Query result as data objects with all attributes as selected by the user on the left pane

Specifying only the title and author fields within items will return only those two fields:

Click Run Button and Select listBooksTables

Adding an Item to the Books Table of DynamoDB Using GraphQL Mutatation

To add a new Book item to the table, replace the JSON object defined in the Query Variables section with the following:

{
  "createbookstableinput": {
    "title": "ChatGPT for Dummies",
    "id": "fBGILBdSSSAw2aDMQo8GaS4w1ozyChos",
    "author": "ChatGPT",
    "releaseDate": "01-01-2023",
    "genre": "Education",
    "pageCount": 500,
    "price": 25
  }
}

The createbookstableinputis an input type defined in the API schema. During the creation process, AppSync automatically generates these input definitions for you. More examples of input types will be available as you explore the generated schema.

The createBooksTable mutation takes a createbookstableinputargument and uses the DynamoDB PutItem action to store the new book in the BooksTable. After the operation completes, it returns the newly created object.

Click Run Button and Select createBooksTable

You can see the output on right pane

You can also see the item added on DynamoDB Console

Its Postman Time

Now Lets Connect to Postman Tool

To connect to postman tool we need some details from aws appsync console

Click Settings

Copy the GraphQL endpoint and paste to a notepad
Scroll down and Copy API Key and paste to a notepad

Now, Open Postman tool and Click Newto create a request

We need to create an http request

Select GraphQLand Proceed

Paste the GraphQL endpoint from notepad here

Click Authorization

Select API Key from Dropdown
Provide the valid credentials
- Type x-api-key to the Key Field
- Paste the apikey value copied from notepad

Navigate to Query Tab
Copy the below Query Mutation and paste in Query Section and Click Query Button


mutation MyMutation {
  deleteBooksTable(input: {title: "The Mannequin in the Mirror"}) {
    id
  }
}

You should be able to see the item deleted from DynamoDB Console as well

Taming the Testing With AWS

Nandini Rajaram — Thu, 15 Aug 2024 06:57:21 +0000

As a software developer in test, one of the most challenging tasks I faced was ensuring the seamless integration of multiple microservices in a cloud-based environment. The project involved a complex web application built on microservices architecture, deployed on AWS. These microservices communicated with each other through RESTful APIs. Testing these APIs manually was not only time-consuming but also prone to errors, leading to significant delays in the release cycle. To tackle this challenge, I had to automate the testing process effectively, combining the power of AWS, API testing tools, and Selenium for end-to-end testing.

Smoke Testing APIs with AWS Canaries: A Smooth Approach

In a recent project, I faced the challenge of ensuring that a set of APIs powering a microservices-based application was always functional and reliable. Given the complexity and the critical role these APIs played, a failure could have far-reaching consequences. To catch potential issues early, I decided to implement smoke testing using AWS Canaries.

The Challenge

The primary challenge was maintaining confidence that the APIs were operational and performing as expected after each deployment. With frequent updates and deployments, relying solely on manual testing or scheduled test suites wasn’t feasible. We needed a way to continuously monitor the APIs, ensuring that any critical issues were detected as soon as they occurred.

The Solution

AWS Canaries, part of the Amazon CloudWatch Synthetics, emerged as the ideal solution for this problem. Canaries are scripts that run on a schedule, simulating the actions of a user or service. They are perfect for performing smoke tests, which are quick, high-level tests to check the basic functionality of the APIs.

Here’s how I implemented it:

Creating the Canaries:

I started by writing Canary scripts in Python using AWS CloudWatch Synthetics. These scripts made HTTP requests to the API endpoints, verifying that the responses were as expected. The scripts were designed to check the most critical API functions, like retrieving data, performing searches, and handling user authentication.

Setting Up Alerts:

After deploying the Canaries, I configured CloudWatch Alarms to trigger notifications if any of the Canary tests failed. This setup ensured that the team was immediately alerted to potential issues, allowing for rapid investigation and resolution.

Scheduling and Continuous Monitoring:

I scheduled the Canaries to run at regular intervals, such as every 5 or 15 minutes, depending on the criticality of the API. This continuous monitoring provided real-time insights into the health of the APIs and allowed us to detect issues before they could impact users.

Additional Links

A step by step tutorial can be found below

The following DevTo post provides a step by step tutorial to set up canaries to do smoke testing
Smoke Testing using AWS Canaries

Automating Testing with AWS, Selenium, and Java: Overcoming the Reporting Challenge

In one of my projects, I was tasked with testing a complex web application deployed on AWS. The application had multiple components, each interacting with different APIs and databases, making it essential to ensure that every piece functioned correctly before release. Given the scale and complexity, manual testing was quickly becoming unmanageable, and automation was the only way forward. However, the real challenge lay not just in automating the tests but also in generating comprehensive and clear test reports that the development team could easily understand and act upon.

The Challenge

The project was built using microservices, each hosted on AWS, with a mix of RESTful APIs and a dynamic user interface. Our team had chosen Selenium for automating the UI tests and Java for writing the test scripts, but as the application grew, so did the number of test cases. Running these tests manually was no longer feasible, and the lack of clear, detailed reporting made it difficult to track test results, identify issues, and communicate them effectively to the development team.

The existing test reports were basic, often limited to console outputs or text files that were hard to decipher without deep technical knowledge. As a result, developers spent too much time trying to understand test failures, which slowed down the bug-fixing process and hampered the overall efficiency of the project.

The Solution

To overcome these challenges, I decided to revamp our entire test automation framework, focusing not just on running tests but also on generating meaningful, detailed reports.

Building the Test Automation Framework:

I started by setting up a robust test automation framework using Selenium WebDriver with Java. TestNG was chosen as the testing framework due to its ability to run tests in parallel, generate HTML reports, and integrate easily with other tools. The framework was designed to be modular, with reusable components that could handle the different parts of the application.

Integrating with AWS

Given that our application was deployed on AWS, I integrated the test automation with AWS services for better scalability and management. We used AWS CodeBuild to run the tests in a CI/CD pipeline, ensuring that tests were automatically triggered on every code commit. This integration allowed us to run tests in parallel across different environments, significantly reducing the time required for test execution

Enhancing Test Reporting and Storing them in S3

The real game-changer was the implementation of an enhanced reporting system. I integrated the ExtentReports library with our TestNG framework. ExtentReports provided detailed HTML reports with visual representations of the test results, including screenshots for failed tests, detailed logs, and an intuitive interface that made it easy for developers to navigate through the results.

Additionally, I set up Allure Reports, another reporting tool that offered even more detailed insights. Allure Reports were particularly useful for their interactive dashboards and the ability to track test case histories over time, which helped in identifying patterns in recurring issues.

The reports were stored in S3 with unique identifiers (based on timestamp and test suite names) to ensure easy retrieval and organization.

The S3 bucket was configured with appropriate permissions to ensure that only authorized team members could access the reports. Additionally, S3 versioning was enabled to maintain a history of all test reports, which was useful for tracking the evolution of test results over time.

Sending Notifications with SNS:

Once the test reports were successfully uploaded to S3, the next step was to notify the team. For this, I used AWS SNS to send email notifications.

I set up an SNS topic and subscribed all relevant team members' email addresses to this topic. Then, using the AWS SDK, I wrote a script that triggered an SNS notification as soon as the reports were uploaded to S3.

The notification included a brief summary of the test results (e.g., the number of tests passed, failed, and skipped) and a direct link to the S3 bucket where the full reports could be accessed. This allowed the team to quickly review the results and take action if needed.

Integrating with CI/CD Pipeline:

With the new reporting system in place, I also integrated these reports into our CI/CD pipeline. AWS CodePipeline was configured to trigger the test suite, store the reports in S3, and send out the SNS notifications upon completion. This automation ensured that the entire process was hands-free and reliable, providing continuous feedback to the team. Developers could now quickly see which tests had failed, why they failed, and even view screenshots of the failure, all within minutes of the test run.

The Outcome

Implementing this solution significantly improved our workflow. The test reports were now stored in a centralized location, easily accessible to all team members at any time. The automated email notifications ensured that everyone was immediately aware of the test results, enabling faster response times to issues.

Let's celebrate PartyRock, an Amazon Bedrock Playground

Nandini Rajaram — Thu, 16 Nov 2023 19:47:04 +0000

Here comes a smart friend that helps you build Generative AI apps

PartyRock

An Amazon Bedrock Playground is a generative AI app development playground that enables interactive experimentation with prompt engineering in an accessible and enjoyable manner.

Good Eats Restaurant Recommendations

My First Generative AI app created using PartyRock

Good Eats Restaurant Recommendations is an app that is created using PartyRock playground that will provide restaurant recommendations based on your preferences on location, cuisine, and meal

Here is the app link:

https://partyrock.aws/u/Nandini/VKvXSeZHH/Good-Eats-Restaurant-Recommendations/snapshot/GoQwp-oSg

Features observed while building the first app

A code-free playground where An app can be created with just a few clicks

Yes, Its that simple. With a few clicks, we can create our desired app. No lengthy programs

Featured apps built with PartyRock to start with

There are some simple AI-powered app ideas which can be used to build the apps

Access to powerful foundation models from Amazon Bedrock

Amazon Bedrock is a fully managed solution that makes foundation models (FMs) from Amazon and prominent AI firms available on AWS. It is the simplest way to construct and scale generative AI applications using foundation models.

PartyRock Integrates with Amazon Bedrock and can provide widgets that can be used to generate AI-powered images, text, and chatbots.

Remix existing apps to create new ones with a unique flavor

You can Remix the existing apps to create your own copy of them and customize them to bring a different flavor

Sharing the app - made simple

With PartyRock, you can easily share apps and content you develop. You can even use the snapshot option to preserve your inputs and outputs while sharing the app link

Build your favorite AI-powered apps using PartyRock Play Ground

visit https://partyrock.aws/

The Code Whisper Magic with Amazon Code Whisperer

Nandini Rajaram — Thu, 26 Oct 2023 22:15:32 +0000

Amazon CodeWhisperer

A coding companion that is intended to improve developer efficiency and productivity with AI Power

Features of Code Whisperer

Supports Multiple programming languages and IDEs
CodeWhisperer supports 15 programming languages and can be used in several IDEs and code editors

Languages Supported

Java

Python

Java Script

Type Script

C#

Script

Ruby

Go

PHP

C++

C

Shell

Scala

Rust

Kotlin

SQL

IDE Supported

Amazon SageMaker Studio

JupyterLab

Visual Studio Code

IDEs produced by JetBrains

AWS Cloud9

AWS Lambda console

Boost the speed of any coding process

CodeWhisperer can understand comments given in natural language and generate multiple code ideas in real time. This tool has the potential to dramatically increase developer productivity by offering relevant code recommendations rapidly.

CodeWhisperer accelerates the implementation of unit tests

It effectively incorporates logic for the majority of the explicit unit tests required and also makes suggestions for edge cases that may have gone unnoticed, increasing test coverage and ensuring comprehensive testing beyond what was initially planned.

How CodeWhisperer generate code?

Prompts are the comments that you add to your code.
CodeWhisperer uses your prompts to make suggestions.
CodeWhisperer can create an entire function based on a comment that you add to your code in supported languages, and it adds the function to your code line by line, where you can review each line before accepting it.

Keyboard shortcuts for Amazon Code Whisperer

Demo : AWS CodeWhisperer integration with Cloud9 IDE

Prerequisite

An EC2 Environment created in Cloud9

To activate CodeWhisperer in the AWS Cloud9 console code editor, complete these steps.

Navigate to AWS Cloud9 Console

Click Open link beside your Cloud9 Ec2 environment

Go to Preferences

Ensure that the two options related to Code Whisperer is enabled on Project Settings --> AWS Toolkit

From inside your existing AWS Cloud9 environment, choose the AWS logo on the left edge of the window.

A panel will expand rightward.

In the lower part of the panel, under Developer tools, open the CodeWhisperer dropdown and choose Enable CodeWhisperer

The following YouTube video illustrates the Creation of java program in Amazon Cloud9 IDE using Amazon Code Whisperer

Note: Delete the cloud9 ec2 environment after use

Game of Templates with AWS Cloud Formation: Simple and Templates with Intrinsic Functions, including ForEach and FindInMap

Nandini Rajaram — Thu, 24 Aug 2023 22:35:21 +0000

AWS CloudFormation

AWS CloudFormation is a service provided by Amazon Web Services (AWS) that allows you to define and manage your infrastructure as code.

Key Terms

CloudFormation templates

A CloudFormation template is used to define your AWS resources in a structured text format, either YAML or JSON.

CloudFormation stack

A "stack" is a collection of AWS resources that may be managed as a single unit in AWS CloudFormation.

StackSet

A named collection of stacks that have the same template but are applied to distinct accounts and regions.

Change Set

A Change Set is a list of proposed changes that will be applied to a stack when you run an update. It serves as a preview method, allowing you to review and validate potential modifications before they are implemented in your infrastructure.

You are only charged for the resources you create and the API calls that CloudFormation performs on your behalf.

The Template Study

Let's dive into the structure of an AWS CloudFormation template

The following example shows an AWS CloudFormation YAML template structure and its top-level objects:



AWSTemplateFormatVersion: 'version date' (optional) # version of the CloudFormation template. Only accepted value is '2010-09-09'

Description: 'String' (optional) # a text description of the Cloudformation template

Metadata: 'template metadata' (optional) # objects that provide additional information about the template

Parameters: 'set of parameters' (optional) # a set of inputs used to customize the template

Rules: 'set of rules' (optional) # a set of rules to validate the parameters provided at deployment/update

Mappings: 'set of mappings' (optional) # a mapping of keys and associated values

Conditions: 'set of conditions' (optional) # conditions that control whether certain resources are created

Transform: 'set of transforms' (optional) # for serverless applications

Resources: 'set of resources' (required) # a components of your infrastructure

Hooks: 'set of hooks' (optional) # Used for ECS Blue/Green Deployments

Outputs: 'set of outputs' (optional) # values that are returned whenever you view your stack's properties

Now, Its time for building some CloudFormation Templates and explore various features.

Lets begin the game now

Building Cloud Formation Templates

Simple Templates

Template 1

A Simple CloudFormation Template that creates an S3 Bucket



AWSTemplateFormatVersion: '2010-09-09'
Resources:
  MyS3Bucket:
    Type: 'AWS::S3::Bucket'
    Properties:
      BucketName: templateonebucket

Replace templateonebucket with a unique name for your S3 bucket. S3 bucket names are globally unique across all of AWS, so you may need to come up with a creative and unique name.

Save this template in a .yaml file and then use the AWS CloudFormation service to deploy it.

Lets Understand the above snippet

Certainly! Let me explain each part of the CloudFormation template:

AWSTemplateFormatVersion: '2010-09-09':
This specifies the version of the CloudFormation template format being used. In this case, it's set to the earliest version '2010-09-09'. This version indicates the overall structure and syntax of the template.
Resources:
The Resources section is where you define the AWS resources you want to create or manage using this CloudFormation template. Resources can include things like EC2 instances, S3 buckets, DynamoDB tables and more.
MyS3Bucket:
This is the logical name of the resource being defined. In this case, it's an S3 bucket, and the logical name is MyS3Bucket. You'll use this logical name later to refer to this resource in other parts of the template or when interacting with AWS services.
Type: 'AWS::S3::Bucket':
This line specifies the type of AWS resource you want to create. In this case, you're creating an S3 bucket, so the Type is set to AWS::S3::Bucket. This tells CloudFormation that you want to create an S3 bucket resource.
Properties:
The Properties section is where you provide the specific configuration for the resource you're creating. For an S3 bucket, there are various properties you can set, such as BucketName, AccessControl, VersioningConfiguration etc
BucketName: templateonebucket:
This is where you set the properties for the S3 bucket. In this example, you're setting the BucketName property to templateonebucket.
Replace templateonebucket with a unique name for your S3 bucket. As mentioned before, S3 bucket names must be globally unique across AWS, so make sure to choose a unique name.

Once you have this CloudFormation template ready, you can use the AWS CloudFormation service to deploy it. This will create an S3 bucket with the bucket name specified

Template 2

Create another s3 bucket with versioning enabled



AWSTemplateFormatVersion: '2010-09-09'
Resources:
  MyS3Bucket:
    Type: 'AWS::S3::Bucket'
    Properties:
     BucketName: mybucketname
     VersioningConfiguration:
        Status: Enabled

Replace mybucketname with a unique name for your S3 bucket. S3 bucket names are globally unique across all of AWS, so you may need to come up with a creative and unique name.

Save this template in a .yaml file and then use the AWS CloudFormation service to deploy it.

VersioningConfiguration is a property that sets the versioning configuration for the bucket.
Statusis set to Enabled to enable versioning for the bucket.

Intrinsic functions

Intrinsic functions are AWS CloudFormation built-in functions that you may use within CloudFormation templates to dynamically produce values, apply conditionals, and control resource characteristics.

You can use intrinsic functions in resource properties, outputs, metadata attributes, and update policy attributes.

Lets dive into some commonly used intrinsic functions in CloudFormation

Ref

This function is used to reference a resource declared within the same CloudFormation stack. It returns the value of the specified resource's logical name.

Template 3

A simple S3 bucket with the name of the bucket getting displayed in output tab using Ref



AWSTemplateFormatVersion: '2010-09-09'
Resources:
  MyS3Bucket:
    Type: 'AWS::S3::Bucket'
    Properties:
      BucketName: my-unique-bucket-name

Outputs:
  BucketNameOutput:
    Description: "Name of the S3 bucket"
    Value: !Ref MyS3Bucket

Replace my-unique-bucket-name with a unique name for your S3 bucket. S3 bucket names are globally unique across all of AWS, so you may need to come up with a creative and unique name.

Save this template in a .yaml file and then use the AWS CloudFormation service to deploy it.

We define an output named BucketNameOutput to display the name of the created S3 bucket.
We use the !Ref function to reference the MyS3Bucket resource, which returns the logical name of the resource, in this case, MyS3Bucket

Outputs in CloudFormation provide a way to make important information available after your infrastructure has been deployed. This can include resource identifiers, endpoints, URLs, and more. By defining outputs, you can make your CloudFormation stacks more flexible and interconnected, enabling you to build complex and dynamic infrastructures.

In Summary, When you deploy this CloudFormation template, it will create an S3 bucket with the specified name, and the output BucketNameOutput will display the logical name of the bucket resource (MyS3Bucket).

You can View the output value on the AWS CloudFormation console , in the Outputs tab.

Fn::Join

This function joins values together using a delimiter to create a single string. It's often used to construct resource names or ARNs

Every Fn::Join object requires two parameters,

a string delimiter
a list of strings to be joined or a function that returns a list of strings to be joined.

Template 4

A simple S3 bucket with bucket name getting generated using Fn:Join



AWSTemplateFormatVersion: '2010-09-09'
Resources:
  MyS3Bucket:
    Type: 'AWS::S3::Bucket'
    Properties:
      BucketName:
        Fn::Join:
          - '-'
          - - 'my-bucket-prefix'
            -  Ref: 'AWS::Region'
            - 'unique-id'

Save this template in a .yaml file and then use the AWS CloudFormation service to deploy it.

Under the Resources section, we define an S3 bucket resource named MyS3Bucket.
We use the Fn::Join intrinsic function to dynamically generate the bucket name.
As the Fn::Join function takes two arguments:

The delimiter to use to join the elements (in this case, a hyphen -)
A list of elements to join together

Inside the list of elements, we include:

A static string 'my-bucket-prefix'.
The Ref intrinsic function referencing the AWS region. This will be replaced with the actual AWS region where the stack is being created.
Another static string 'unique-id'.
- The resulting bucket name will be something like: my-bucket-prefix-us-east-2-unique-id, where us-east-2 being the actual AWS region I use the CloudFormation service

Fn::Sub

The Fn::Sub intrinsic function in AWS CloudFormation is used for variable substitution in strings. It allows you to create dynamic strings by substituting variables with their corresponding values from resources, parameters, or intrinsic functions

Syntax:



Fn::Sub:
  - "string to substitute ${variable}"
  - { "variable": value }

Template 5

A simple S3 bucket with a bucket name derived by the value of the Environment variable using Fn::Sub



AWSTemplateFormatVersion: '2010-09-09'
Resources:
  MyBucket:
    Type: 'AWS::S3::Bucket'
    Properties:
      BucketName:
        Fn::Sub:
          - "my-awesome-bucket-${Environment}"
          - { "Environment": "production" }

Save this template in a .yaml file and then use the AWS CloudFormation service to deploy it.

Under the Resources section, we define an S3 bucket resource named MyBucket.
The BucketName property of the S3 bucket resource uses the Fn::Sub intrinsic function to create a dynamic bucket name.
The first argument is the string template with a variable ${Environment} that needs to be replaced.
The second argument to Fn::Sub is a JSON object that specifies the values of the variables used in the string template.
In this case, {"Environment": "production"} means that ${Environment} will be replaced with the value "production".
After deployment, the bucket name will be my-awesome-bucket-production.

Fn::GetAtt

The Fn::GetAtt intrinsic function returns the value of an attribute from a resource in the template.

Syntax

!GetAtt logicalNameOfResource.attributeName

Template 6

Create an S3 bucket resource, along with a few outputs being generated using Fn::GetAtt



AWSTemplateFormatVersion: '2010-09-09'
Resources:
  MyS3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: my-unique-bucket-name

Outputs:
  BucketName:
    Value: !Ref MyS3Bucket
  BucketArn:
    Value: !GetAtt MyS3Bucket.Arn
  BucketDomainName:
    Value: !GetAtt MyS3Bucket.DomainName

Replace my-unique-bucket-name with a unique name for your S3 bucket. S3 bucket names are globally unique across all of AWS, so you may need to come up with a creative and unique name.

Save this template in a .yaml file and then use the AWS CloudFormation service to deploy it.

MyS3Bucket is defined as an S3 bucket resource with a specified unique bucket name.
The BucketName output uses !Ref to reference the bucket resource, effectively providing the name of the bucket.
The BucketArn output uses !GetAtt to retrieve the Amazon Resource Name (ARN) of the bucket resource.
The BucketDomainName output uses !GetAtt to retrieve the domain name of the bucket.

You can View the output values on the AWS CloudFormation console , in the Outputs tab.

Fn::Select

The Fn::Select intrinsic function in AWS CloudFormation is used to select an element from a list based on its index. It's commonly used when you have a list of items and you want to retrieve a specific item from that list.

Syntax:



Fn::Select: [ index, listOfItems ]

index: The zero-based index of the item you want to select from the list.
listOfItems: The list of items from which you want to select an item.

The !Select function allows you to dynamically select values based on their indices, enabling you to parameterize and customize your resource properties within the template.

Template 7

Creates an AWS S3 bucket resource named MyBucket with a specific bucket name generated using Fn::Select function



AWSTemplateFormatVersion: '2010-09-09'
Resources:
  MyBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Select [1, ["my-bucket-1", "my-bucket-2", "my-bucket-3"]]

Replace my-bucket-1,my-bucket-2,my-bucket-3 with a unique name for your S3 bucket. S3 bucket names are globally unique across all of AWS, so you may need to come up with a creative and unique name.

Save this template in a .yaml file and then use the AWS CloudFormation service to deploy it.

AWSTemplateFormatVersion: '2010-09-09': This indicates the version of the CloudFormation template format being used. In this case, the template is using the 2010-09-09 version.
Resources: This section defines the AWS resources that you want to create or manage using the CloudFormation template.
MyBucket: This is the logical ID of the AWS S3 bucket resource being created.
Type: AWS::S3::Bucket:This specifies the resource type as an S3 bucket. It indicates that CloudFormation should create an S3 bucket resource.

Properties: This section contains the properties specific to the resource being created. In this case, it's the properties of the S3 bucket resource.

BucketName: This property specifies the name of the S3 bucket. The value is retrieved using the !Select intrinsic function.

-!Select [1, ["my-bucket-1", "my-bucket-2", "my-bucket-3"]]: This uses the !Select function to select an element from a list.

The arguments are as follows:
- 1: This is the index of the element to be selected. Since indices are zero-based, 1 corresponds to the second element in the list.
- ["my-bucket-1", "my-bucket-2", "my-bucket-3"]: This is the list of bucket names. The second element in this list is "my-bucket-2".

When this CloudFormation template is executed, it will create an S3 bucket resource named MyBucket with the bucket name "my-bucket-2".

Fn::Split

The Fn::Split intrinsic function in AWS CloudFormation is used to split a string into a list of substrings based on a specified delimiter.

Syntax:



Fn::Split: [ delimiter, sourceString ]

Template 8

Add a tag for the S3 bucket resource MyBucket



AWSTemplateFormatVersion: '2010-09-09'
Resources:
  MyS3Bucket:
    Type: 'AWS::S3::Bucket'
    Properties:
      BucketName: my-bucket-name
      Tags:
        - Key: Name
          Value:
            'Fn::Select':
              - 1
              - 'Fn::Split':
                  - '-'
                  - my-bucket-name

Replace my-bucket-name with a unique name for your S3 bucket. S3 bucket names are globally unique across all of AWS, so you may need to come up with a creative and unique name.

Save this template in a .yaml file and then use the AWS CloudFormation service to deploy it.

AWSTemplateFormatVersion: '2010-09-09': This line specifies the version of the CloudFormation template format being used. In this case, it's the 2010-09-09 version.
Resources: This section defines the AWS resources that you want to create or manage using the CloudFormation template.
MyBucket: This is the logical ID of the AWS S3 bucket resource being created.
Type: AWS::S3::Bucket: This line indicates that the resource type being created is an S3 bucket. The CloudFormation template will create an S3 bucket resource.
Properties: This section contains the properties specific to the S3 bucket resource.
BucketName: my-bucket-name: This sets the name of the S3 bucket to "my-bucket-name".
Tags: This property allows you to assign tags to the S3 bucket.

Tags are key-value pairs used to label and categorize resources for better organization and management.

-Key: Name: This defines a tag with the key "Name" The key represents the label or category of the tag.

Value: The value of the tag is determined using the Fn::Select and Fn::Split intrinsic functions.
Fn::Split: This function is used to split the string "my-bucket-name" into a list of substrings using the "-" delimiter. The result is a list of substrings: ["my", "bucket", "name"].
Fn::Select: This function is used to select an element from the list of substrings. In this case, the second element ("bucket") is selected using an index of 1.

As a result, the S3 bucket resource MyBucket will have a tag with the key "Name" and the value "bucket."

You can view the tags on the properties tab of your bucket in S3 Console

The dynamic tagging is achieved by using Fn::Split to manipulate the string "my-bucket-name" and Fn::Select to select the desired substring for the tag value.

Fn::ForEach

The Fn::ForEach intrinsic function takes a collection and a fragment, and applies the items in the collection to the identifier in the provided fragment

Syntax:



Fn::ForEach::UniqueLoopName:
    - Identifier
    - - Value1 # Collection
      - Value2
    - OutputKey:
        OutputValue

UniqueLoopName A name for this loop.
Identifier The identifier you want to replace in the OutputKey and OutputValue parameters that represent the template fragment that is replicated
Collection
The collection of values to iterate over. This can be an array in this parameter, or it can be a Ref to a CommaDelimitedList.
OutputKey
The key in the transformed template. ${Identifier} must be included within the OutputKey parameter.
OutputValue
The value that is replicated in the transformed template for each item in the Collection parameter

You must use the AWS::LanguageExtensions transform to use the Fn::ForEach intrinsic function.

Template 9

Using the Fn::ForEach intrinsic function, create S3 buckets with versioning enabled, by looping through a list of bucket names and building S3 buckets with those names



AWSTemplateFormatVersion: 2010-09-09
Transform: 'AWS::LanguageExtensions'
Resources:
  Fn::ForEach::Buckets:
    - BucketName
    - - my-unique-bucket-name-one
      - my-unique-bucket-name-two
    - ${BucketName}:
        Type: AWS::S3::Bucket
        Properties:
          BucketName: !Ref BucketName
          VersioningConfiguration:
             Status: Enabled

Replace my-unique-bucket-name-one and my-unique-bucket-name-two with a unique names for your S3 buckets. S3 bucket names are globally unique across all of AWS, so you may need to come up with a creative and unique name.

Save this template in a .yaml file and then use the AWS CloudFormation service to deploy it.

Transform: AWS::LanguageExtensions: This indicates that you're using a custom CloudFormation transform named AWS::LanguageExtensions. This transform extends the template syntax with additional capabilities, and in this case, it's used to enable the Fn::ForEach functionality.
Resources: This is where you define your CloudFormation resources.
Fn::ForEach::Buckets: This is the resource defined with the Fn::ForEach intrinsic function. It iterates over the list of bucket names and creates S3 bucket resources for each name.
- BucketName: This is the logical ID for the iteration. This is the parameter for the iteration, representing the list of bucket names that you want to iterate over.

my-unique-bucket-name-one and my-unique-bucket-name-two : These are the actual bucket names specified in the list. Each name represents an iteration.

${BucketName}: This is the dynamic reference to the iteration variable. It's used to create the logical IDs for the S3 buckets.

Type: AWS::S3::Bucket: This specifies that the resource type is an S3 bucket.

BucketName: !Ref BucketName: This sets the name of the S3 bucket based on the value of the BucketName parameter.

VersioningConfiguration: This defines the bucket's versioning configuration.

Status: Enabled: This enables versioning for the S3 bucket.

With Fn::ForEach, you can replicate parts of your templates with minimal lines of code.
You can use Fn::ForEach to simplify your template layout and make it easier and faster for you and your peers to review your code.
Fn::ForEach helps reduce human errors such as updating wrong properties or missing out on updating multiple target properties in your template.

Fn::FindInMap

The Fn::FindInMap intrinsic function is used in AWS CloudFormation templates to retrieve a value from a predefined mapping. This function is particularly useful when you want to parameterize or customize resources based on certain conditions, such as regions or environments.

A Mappings section is a top level section of a CloudFormation template. It is used to define maps, their keys and values which can be then referenced in your template.

When you include the AWS::LanguageExtensions transform in an AWS CloudFormation template, you can define the fields of Fn::FindInMap using intrinsic functions. so that, If no mapping is detected, you can utilize a new optional field to return a default value.

Template 10

Create S3 buckets with names based on AWS regions, while also providing a default bucket name in case a specific region mapping is not defined. using !FindInMap



AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::LanguageExtensions'
Mappings:
  S3BucketNames:
    us-east-1:
      BucketName: my-bucket-us-east-1
    us-west-1:
      BucketName: my-bucket-us-west-1

Resources:
  MyS3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName:
        Fn::FindInMap:
          - S3BucketNames
          - !Ref AWS::Region
          - BucketName
          - DefaultValue: my-default-bucket

Replace my-default-bucket, my-bucket-us-east-1 and my-bucket-us-west-1 with a unique names for your S3 buckets.

S3 bucket names are globally unique across all of AWS
so you may need to come up with a creative and unique name.

Save this template in a .yaml file and then use the AWS CloudFormation service to deploy it.

Try deploying the template in us-east-1, us-west-1 and us-west-2 regions.

The above CloudFormation template showcases the usage of the AWS::LanguageExtensions transform along with the Fn::FindInMap function and a default value extension.

Transform: 'AWS::LanguageExtensions': This line indicates that the CloudFormation template is using the AWS::LanguageExtensions transform, which allows the use of language extensions like default values for intrinsic functions.
Mappings: Defines a mapping named S3BucketNames, which associates different S3 bucket names with different AWS regions.
S3BucketNames: This key under the Mappings section contains the mapping for S3 bucket names based on regions.
us-east-1 and us-west-1: These region keys under the S3BucketNames mapping correspond to different AWS regions. For each region, there's a sub-key BucketName that holds the corresponding bucket name.
Resources: Specifies the AWS resources created as part of this CloudFormation stack.
MyS3Bucket: Defines an S3 bucket resource.
Type: AWS::S3::Bucket: Indicates that this resource represents an S3 bucket.
Properties: Sets the properties for the MyS3Bucket resource.

BucketName: This property designates the bucket name of the S3 bucket.

The Fn::FindInMap function is used to dynamically retrieve the bucket name based on the AWS region of the stack ( !Ref AWS::Region).

There is a twist here,

DefaultValue: my-default-bucket: This extension, facilitated by the AWS::LanguageExtensions transform, allows specification of a fallback value if the specific mapping key isn't found.

In this scenario, if the stack is deployed in a region not included in the S3BucketNames mapping, such as a region other than us-east-1 or us-west-1, the bucket name will default to
my-default-bucket.

If you try to deploy the CloudFormation Template in regions say
US West (Oregon), us-west-2 (which is not mentioned in mapping-S3BucketNames) The name of the bucket would be my-default-bucket

If you try to deploy the cloudformation Template in regions say
US East (N. Virginia) us-east-1 The name of the bucket would be my-bucket-us-east-1

If you try to deploy the cloudformation Template in regions say
US West (N. California) us-west-1 The name of the bucket would be my-bucket-us-west-1

Create CloudFormation stacks using the AWS Management Console

The following YouTube video illustrate how you can create a CloudFormation stack using the console.

Deleting a CloudFormation stack will remove all resources created by that stack. So once you have finished learning, You can delete the stacks

Build a Quick Game using Amazon GameLift service

Nandini Rajaram — Sun, 28 May 2023 22:36:34 +0000

Game software is created using a game engine, such as Unity, Unreal, or Lumberyard, and is then run on the player's gaming device, such as a game console, computer, phone, tablet, TV set top box, or web browser.

Game Server

A game server is a piece of software that manages multi-player activities and offers accurate game state. it also communicates and provides services to the Game Software during gameplay.

The Game Server software runs on an EC2 instance on AWS.

Game Hosting Service

The system that provisions game server instance and manages the loading and running of the Game Server is called the Hosting Service. The minimum work that a Game Server Hosting Service performs is infrastructure Management (provisioning compute resources on which to host Game Servers)

AWS Game Server hosting solutions go far beyond that.

Session Management involves staying connected to the Game Server
Reporting information like the available slots in a multiplayer game.
Recognizing when a game session has ended, or when a new game session is started and whether it is currently ready to receive players or not.
Finding an available Game Server to host a game that is within latency tolerances and located near players (Game Session Placement)
Shifting placement to different regions as availability of instances and location of player demand changes.
Matchmaking where multiple players can request to play a game together on the same Game Server

Amazon GameLift

Amazon GameLift is a managed service that provides Game Server hosting abstractions on top of Amazon EC2. GameLift is designed for session-based games. It has features to build multiplayer games and battle royale games with 100 or more players. And it scales to multiple Regions and globally.

Fleet

Instances are grouped into fleets and are composed of identical instances.
Each instance is loaded with Game Server software, which is used to start one or more Game Sessions.
A fleet can be manually or automatically scaled

GameLift supports for Spot Instances. This can provide up to 70 percent cost savings compared to existing on-premises deployments.

GameLift includes a feature called FleetIQ

FleetIQ

FleetIQ manages availability by backing it up with an On Demand fleet and designating the Spot fleet as unavailable to maintain the quality of the gaming experience. This minimizes Spot disruption of game sessions.

Queue

Fleets are placed in a queue. When a request comes in to play a game, the system uses the queue to find an acceptable fleet and identify an available Game Session.

The game software measures latency to regional endpoints and provides this information in the game request. The lowest-latency fleet is selected. If it is not available or has no open Game Sessions, the next one is chosen. The system is also aware of fleet costs, and it will choose an available lower-cost Spot fleet before an equivalent On-Demand fleet when other factors are equal. You can set up multiple queues for global Game Server hosting.

FlexMatch

Amazon GameLift’s FlexMatch capabilities let you match players together based on rules you define.

For example, you can have a rule for the game that the match be accepted by all the players before the Game Session is placed.

Lets Build a game now

Navigate to Amazon GameLift console

On the left pane, click Try a Sample game link

You will be navigated to Try a sample game page where create and deploy a sample game server using Amazon GameLift.

Click Add build button which will Create an example game build.

Once build is added, Click Create fleet button to run the sample game

Fleet creation might take some time to complete

Once the fleet is created, Click Start game session button

Now, Click Generate player token button and connect up to 8 players to the game

You will be navigated to Connect Players screen where you need to download and open game client by clicking Download game client button

For every game client session, a player token need to be generated and this need to be pasted it to the client. Click Generate player token button to generate Player token and copy the token by clicking copy button

Extract the GameLiftClientPackage and open the Bin64vc141.Release\MultiplayerSampleLauncher.exe to launch the game

Click the player session token text area, and paste the player token copied

Now, Click Connect button and start playing the game

Now that we have successfully built a game and the take away is

Amazon GameLift is an ideal solution for hosting multiplayer games with the most focus on developing the Game Server and the least work required on development operations.

Quit the game and Delete the resources once you have finished playing the game

An ETL Job using AWS Glue Studio to inner join DynamoDB tables, Apply Queries and Store the result in S3

Nandini Rajaram — Thu, 23 Mar 2023 01:27:26 +0000

ETL Jobs

ETL stands for extract, transform, and load. And this is the most common paradigm for combining data from various systems into a single database, data store, or warehouse for legacy storage or analytics.

Extraction
Extraction is the process of retrieving data from one or more sources. Following data retrieval, ETL is to compute work that loads the data into a staging area and prepares it for the next phase.
Transformation
Transformation is the process of mapping, reformatting and other operations to prepare data for consumption.
Loading
Loading involves successfully inserting the transformed data into the target database, data store or data warehouse.

All of this work is processed in what the business intelligent developers call an ETL job.

AWS Glue

AWS Glue is the key service that integrates data over a data lake, data warehouse, and purpose-built data stores which simplifies data movements from multiple sources

AWS Glue Studio

AWS Glue Studio is an easy-to-use graphical interface for creating, running, and monitoring AWS Glue extract, transform, and load (ETL) jobs.

Olive wants to purchase a finger ring with her birth stone.

The details of Birth stones and ring prices are stored in two DynamoDB tables

Now, We are going to create an ETL job using AWS Glue Studio which inner join two DynamoDB tables Birthstone and RingsTable on condition RingsTable.RingStoneName = Birthstone.Stone Name

After that, apply a Query on the result table to find the ring stone and its price matching to Olive's Birth Month and then save the result to the s3 bucket named "gemstonejobresult"

We use git for the source code management so finally we are going to create a remote git repository on AWS Code Commit and push the job to the git repository

Prerequisites

Two tables created using AWS DynamoDB Service: Birthstone, RingsTable

(Recipe: https://dev.to/aws-builders/cook-a-recipe-with-aws-dynamo-db-table-1kn1)

S3 Bucket to store the result - gemstonejobresult
A git repository on AWS Code commit - gemstone_job_repository
An IAM role named AWSGlueServiceRoledemo- with permission to your Amazon S3 sources, targets, temporary directory, scripts, and any libraries used by the job.

Lets set up the Prerequisites first

1 : DynamoDB Tables

Here, I have created two Tables named "Birthstone" and "Birthstone" and some items to them
Make sure to enable PITR on both the tables

The Table schema details provided below

Table Data

2: S3 Bucket for Storing the Query Result

Navigate to S3 console

Click Create Bucket button
You will be navigated to bucket creation page
Specify a unique name to the bucket, Leave the rest of the fields to default values

Scroll down and click create bucket button

You will be navigated back to bucket list page on S3 Console and the newly created bucket will be listed here

3: A git repository on AWS Code commit

AWS Code commit

AWS CodeCommit is a fully-managed source control service that makes it easy for companies to host secure and highly scalable private Git repositories.
CodeCommit eliminates the need to operate your own source control system or worry about scaling its infrastructure.

Repository

In CodeCommit, the fundamental version control object is a repository.

It's where you keep your project's code and files safe. It also keeps track of your project's history, from the first commit to the most recent changes.

You can collaborate on a project by sharing your repository with other users.

Before you can push changes to a CodeCommit repository, you must configure an IAM user in your Amazon Web Services account, or set up access for federated access or temporary credentials.
DO NOT USE ROOT USER

Navigate to code commit console

Click Create Repository button to create a git repository for storing our code

Specify the repository name, I am naming it as "gemstone_job_repository" and click create repository button

Lets Create a Readme.md file and commit it to the repository, Click Create file button
Add some data to the file and save the file as ReadMe.md
The file will be saved to the main branch
Specify the Author name,Email address and commit message (Though optional, its a good practice to provide commit message)
Click the button commit changes

Branches in git
Branches can be used to develop new features, save a specific version of your project from a specific commit, and much more.

When you create your first commit in aws code commit console , a default branch main is created for you

Now, we have created a repository on aws console , made a first commit to it by saving a ReadMe file to the main branch

4: IAM role

Navigate to IAM Console,

Click on Roles link on Left Pane

Click Create Role button to create a role

Select AWS Service as Trusted entity type
Select glue option from Use cases for other services dropdown

Add the below permissions to the role and click Next button

Specify a role name "AWSGlueServiceRoledemo" and Click Create Role button

Create an ETL job using AWS Glue Studio to inner join DynamoDB Tables

Navigate to AWS Glue Studio from aws console

Click Jobs from Left pane

You will be navigated to Jobs page, where you can create ETL Jobs
By default, Visual with source and target option would be selected
Select Source as DynamoDB and Target as Amazon S3 in Create job
Click Create button

An untitled job with the source as DynamoDB and target as Amazon S3 will be created for you. And You will be navigated to the page of untitled job where you will be able to edit the configuratons

Visual Tab

First You will be on the Visual tab

Customers can create data integration jobs using the Amazon Glue API from a JSON object that displays a visual step-by-step workflow thanks to the API offered by AWS Glue.
Clients can then work with these jobs using the visual editor in Amazon Glue Studio.

Now we are going to create a job which reads data from two dynamodb tables BirthStone table and Ring table.
We are going to inner join these two tables based on name attribute
and apply a query to find birtstone suitable for Olive and its price
First we will add the required Source blocks, Action blocks and Target Blocks
Store the Query result in S3 bucket
- Nodes Required:
  - Two DataSource DynamoDB (RingsTable, Birthstone)
  - Transfrom Join (For applying inner join)
  - SQL Query ( For querying the result table)
  - Data target - S3 Bucket
Click on the Source and add one more DynamoDB table
Remove the ApplyMapping Block, if its already there, since we don't need it for this job

(Click on the ApplyMapping Block and Click Remove)
From the Action, Select Join and Add it
From the Action, Select SQL Query and Add it
S3 bucket will be there on the flow as target , leave it as it is

Now, Lets Configure the flow

Source Nodes

Click on One of the Data Source DynamoDB nodes,
On the Right pane, configure the node properties as shown in the picture below

Select the option Choose the Dynamodb Table directly radio button for DynamoDB source
Select the Birthstone table from the Dropdown, Make sure PITR is enabled for the Birth stone table , if not the table will not be populated on the dropdown, (If you do not see the table in the dropdown, Go to the Dynamodb service, enable PITR for the Birthstone table and Ring Table. Then Refresh this page to get the tables on this dropdown)
Click on the second DynamoTable Data source node,
On the Right pane, configure the node properties as shown in the picture below

Now, We have configured the Source Nodes successfully

Action Nodes

Click on the Transform join node
On the Right pane, Click on the Node Properties Tab
Select the two DynamoDB tables as Node Parents

Click on the Transform Tab, Make sure Inner join is selected as Join Type, Click Add Condition button

Specify the inner join condition as shown in the image below

Click on Transform SQL Query node
On the Right pane, Click on the Node Properties Tab
Select Transform Check box from the Node Parents dropdown list, You can see Join node getting selected

Click on the Transform Tab to specify the query on the Inner joined Table
As You see here, Inner Joined table is alised as myDataSource
Now Lets Write the SQL Query to generate the Birth Stone name and Price of the Birst stone matching to olive's Birth Month

Olive's Birth Month is December since she is born on a christmas day , Lets Create a query to get the Birth Stone gem and its price matching to the month

select RingStoneName, Price from myDataSource where BirthMonth = 'December' limit 1

Yes, Now we have configured Transform nodes successfully

Target Node

We need to store the query result in the s3 bucket

Click on the Data target - S3 bucket node
On the Right pane, Click on the Node Properties Tab and Tick SQL Query checkbox

Click on Data Target Properies - S3 Tab
Configure it as shown in the below image

As you see, Format should be Parquet
The bucket location should be selected from s3 so that a Parquet file with the result data will be saved on the S3 bucket once the job execution is completed successfully
After all the configurations, The Visual Tab will be seen as below image

Script Tab

Based on the Job Configuration, Python script will be generated on the Script tab

Job details

Now, Navigate to Job details tab to specify the job details

By default, Job name would be Untitled, Specify a job name in the Name Field.
Specify a description as well

Select the AWSGlueServiceRole-demo which we already created as a part of prerequisites

Leave the remaining to default settings

Click Save button to save the Job

Version Control Tab

Navigate to Version Control Tab

Here, You need to do the git configurations
Select AWS Code commit option for git service

In the Repository Configuration section, choose the git repository that we created in aws code commit (Refer Prerequisites)

Choose the branch as main

Click Save button on the top right

Click the Actions Button, Choose Push to repository option

The job will be pushed to the repository and a JSON file will be saved on to the code repository.

Navigate to aws code commit and verify if files are saved on to the repository

Executing the ETL Job and Verifying the result in S3 bucket

Once you have saved the job, its time to Run the job

Click the Run button on the Top Right pane

You will be getting the following message

Navigate to Runs Tab to see the job status

Once the Run status becomes succeeded, Navigate to S3 Console and click on the gemstonejobresult bucket
You will be able to see the result saved as a parquet file

Download the Parquet file and save the file locally by clicking the Download button

Open the website https://www.parquet-viewer.com/ to read results from parquet file

Now You will be able to view the result

Reading the Parquet File object using S3 select

Alternatively, You can Query the Parquet file with S3 select

Navigate to S3 Console and click on the gemstonejobresult bucket
Click on the Parquet file name link where the result is saved

Now, Click the Object Actions button and from the list, select the Query With S3 Select and give a click
You will be navigated to Query with S3 Select Page, Now by default Apache Parquet will be chosen in input settings, Keep it as it is
You can choose the Format in Output settings.

S3 Select allows you to specify how the query results are serialized in the output. You can choose formats like JSON or CSV. This determines how the data will be structured in the output files.

Lets choose JSON here so that the result of the query will be displayed in JSON Format

We are going to read the Parquet File using the below Select Query which is displayed in the query editor by default.

SELECT * FROM s3object s LIMIT 5

Now Click Run SQL query button to execute the query

Once the Query is executed successfully, You will be able to view the result in JSON format (The format chosen in Output settings)

You can download the result by clicking the Download results button

Yaay , We found it , Olive needs to purchase a Turquoise ring and its price is 612$

And that was Olive’s story.

Challenge Task

Set up an ETL job by following the above instructions
Find out the month corresponding to the Birth stone which is priced highest
Post a screenshot of the answer in comment section