Forem: Vishal Raju

Zomato Clone: Secure Deployment with DevSecOps CI/CD

Vishal Raju — Fri, 10 Jan 2025 17:52:43 +0000

Table of contents
Steps:-
Step 1:Launch an Ubuntu(22.04) T2 Large Instance
Step 2 — Install Jenkins, Docker and Trivy
2A — To Install Jenkins
2B — Install Docker
2C — Install Trivy
Step 3 — Install Plugins like JDK, Sonarqube Scanner, NodeJs, OWASP Dependency Check
3A — Install Plugin
3B — Configure Java and Nodejs in Global Tool Configuration
3C — Create a Job
Step 4 — Configure Sonar Server in Manage Jenkins
Step 5 — Install OWASP Dependency Check Plugins
Step 6 — Docker Image Build and Push
Step 8: Terminate instances.
Complete Pipeline

Show less
Hey there! Get ready for an exciting journey as we embark on deploying a React JS Zomato clone. Our trusty companion on this adventure is Jenkins, serving as our CI/CD tool, while the magic happens inside a Docker container. Dive into the details in this comprehensive blog — your go-to guide for the entire process.

Steps:-
Step 1 — Launch an Ubuntu(22.04) T2 Large Instance

Step 2 — Install Jenkins, Docker and Trivy. Create a Sonarqube Container using Docker.

Step 3 — Install Plugins like JDK, Sonarqube Scanner, Nodejs, and OWASP Dependency Check.

Step 4 — Create a Pipeline Project in Jenkins using a Declarative Pipeline

Step 5 — Install OWASP Dependency Check Plugins

Step 6 — Docker Image Build and Push

Step 7 — Deploy the image using Docker

Step 8 — Terminate the AWS EC2 Instances.

Now, let’s get started and dig deeper into each of these steps:-

Step 1:Launch an Ubuntu(22.04) T2 Large Instance
Launch an AWS T2 Large Instance. Use the image as Ubuntu. You can create a new key pair or use an existing one. Enable HTTP and HTTPS settings in the Security Group and open all ports (not best case to open all ports but just for learning purposes it’s okay).

Step 2 — Install Jenkins, Docker and Trivy
2A — To Install Jenkins
Connect to your console, and enter these commands to Install Jenkins

Copy

Copy
vim jenkins.sh
COPY

Copy

!/bin/bash

sudo apt update -y

sudo apt upgrade -y

wget -O - https://packages.adoptium.net/artifactory/api/gpg/key/public | tee /etc/apt/keyrings/adoptium.asc
echo "deb [signed-by=/etc/apt/keyrings/adoptium.asc] https://packages.adoptium.net/artifactory/deb $(awk -F= '/^VERSION_CODENAME/{print$2}' /etc/os-release) main" | tee /etc/apt/sources.list.d/adoptium.list
sudo apt update -y
sudo apt install temurin-17-jdk -y
/usr/bin/java --version
curl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key | sudo tee \
/usr/share/keyrings/jenkins-keyring.asc > /dev/null
echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
https://pkg.jenkins.io/debian-stable binary/ | sudo tee \
/etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt-get update -y
sudo apt-get install jenkins -y
sudo systemctl start jenkins
sudo systemctl status jenkins

Copy

Copy
sudo chmod 777 jenkins.sh
./jenkins.sh # this will installl jenkins
Once Jenkins is installed, you will need to go to your AWS EC2 Security Group and open Inbound Port 8080, since Jenkins works on Port 8080.

Now, grab your Public IP Address

Copy

Copy

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Unlock Jenkins using an administrative password and install the suggested plugins.

Jenkins will now get installed and install all the libraries.

Create a user click on save and continue.

Jenkins Getting Started Screen.

2B — Install Docker

Copy

Copy
sudo apt-get update
sudo apt-get install docker.io -y
sudo usermod -aG docker $USER #my case is ubuntu
newgrp docker
sudo chmod 777 /var/run/docker.sock
After the docker installation, we create a sonarqube container (Remember to add 9000 ports in the security group).

Copy

Copy
docker run -d --name sonar -p 9000:9000 sonarqube:lts-community

Now our sonarqube is up and running

Enter username and password, click on login and change password

Copy

Copy
username admin
password admin

Update New password, This is Sonar Dashboard.

2C — Install Trivy

Copy

Copy
vim trivy.sh

Copy

Copy
sudo apt-get install wget apt-transport-https gnupg lsb-release -y
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | gpg --dearmor | sudo tee /usr/share/keyrings/trivy.gpg > /dev/null
echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main" | sudo tee -a /etc/apt/sources.list.d/trivy.list
sudo apt-get update
sudo apt-get install trivy -y
Next, we will log in to Jenkins and start to configure our Pipeline in Jenkins

Step 3 — Install Plugins like JDK, Sonarqube Scanner, NodeJs, OWASP Dependency Check
3A — Install Plugin
Goto Manage Jenkins →Plugins → Available Plugins →

Install below plugins

1 → Eclipse Temurin Installer (Install without restart)

2 → SonarQube Scanner (Install without restart)

3 → NodeJs Plugin (Install Without restart)

3B — Configure Java and Nodejs in Global Tool Configuration
Goto Manage Jenkins → Tools → Install JDK(17) and NodeJs(16)→ Click on Apply and Save

3C — Create a Job
Create a job as Zomato Name, select pipeline and click on ok.

Step 4 — Configure Sonar Server in Manage Jenkins
Grab the Public IP Address of your EC2 Instance, Sonarqube works on Port 9000, so :9000. Goto your Sonarqube Server. Click on Administration → Security → Users → Click on Tokens and Update Token → Give it a name → and click on Generate Token

click on update Token

Create a token with a name and generate

copy Token

Goto Jenkins Dashboard → Manage Jenkins → Credentials → Add Secret Text. It should look like this

You will this page once you click on create

Now, go to Dashboard → Manage Jenkins → System and Add like the below image.

Click on Apply and Save

The Configure System option is used in Jenkins to configure different server

Global Tool Configuration is used to configure different tools that we install using Plugins

We will install a sonar scanner in the tools.

In the Sonarqube Dashboard add a quality gate also

Administration → Configuration →Webhooks

Click on Create

Add details

Copy

in url section of quality gate

http://jenkins-public-ip:8080/sonarqube-webhook/

Let’s go to our Pipeline and add the script in our Pipeline Script.

Copy

To see the report, you can go to Sonarqube Server and go to Projects.

You can see the report has been generated and the status shows as passed. You can see that there are 1.3k lines. To see a detailed report, you can go to issues.

Step 5 — Install OWASP Dependency Check Plugins
GotoDashboard → Manage Jenkins → Plugins → OWASP Dependency-Check. Click on it and install it without restart.

First, we configured the Plugin and next, we had to configure the Tool

Goto Dashboard → Manage Jenkins → Tools →

Click on Apply and Save here.

Now go configure → Pipeline and add this stage to your pipeline and build.

Copy

    stage('OWASP FS SCAN') {
        steps {
            dependencyCheck additionalArguments: '--scan ./ --disableYarnAudit --disableNodeAudit', odcInstallation: 'DP-Check'
            dependencyCheckPublisher pattern: '**/dependency-check-report.xml'
        }
    }
    stage('TRIVY FS SCAN') {
        steps {
            sh "trivy fs . > trivyfs.txt"
        }
    }

The stage view would look like this,

You will see that in status, a graph will also be generated and Vulnerabilities.

Step 6 — Docker Image Build and Push
We need to install the Docker tool in our system, Goto Dashboard → Manage Plugins → Available plugins → Search for Docker and install these plugins

Docker

Docker Commons

Docker Pipeline

Docker API

docker-build-step

and click on install without restart

Now, goto Dashboard → Manage Jenkins → Tools →

Add DockerHub Username and Password under Global Credentials

Add this stage to Pipeline Script

Copy

    stage("Docker Build & Push"){
        steps{
            script{
               withDockerRegistry(credentialsId: 'docker', toolName: 'docker'){   
                   sh "docker build -t zomato ."
                   sh "docker tag zomato mudit097/zomato:latest "
                   sh "docker push mudit097/zomato:latest "
                }
            }
        }
    }
    stage("TRIVY"){
        steps{
            sh "trivy image mudit097/zomato:latest > trivy.txt" 
        }
    }

You will see the output below, with a dependency trend.

When you log in to Dockerhub, you will see a new image is created

Now Run the container to see if the app coming up or not by adding the below stage

Copy

Copy
stage('Deploy to container'){
steps{
sh 'docker run -d --name zomato -p 3000:3000 mudit097/zomato:latest'
}
}
stage view

Jenkins-public-ip:3000

You will get this output

Step 8: Terminate instances.
Efficiently manage resources by terminating the AWS EC2 Instances to ensure cost-effectiveness and environmental responsibility, completing the deployment lifecycle. Utilize AWS management tools or commands to gracefully shut down and terminate the Ubuntu(22.04) T2 Large Instance, concluding the deployment process while maintaining operational efficiency.

Complete Pipeline

Copy

Copy
pipeline{
agent any
tools{
jdk 'jdk17'
nodejs 'node16'
}
environment {
SCANNER_HOME=tool 'sonar-scanner'
}
stages {
stage('clean workspace'){
steps{
cleanWs()
}
}
stage('Checkout from Git'){
steps{
git branch: 'main', url: 'https://github.com/mudit097/Zomato-Clone.git'
}
}
stage("Sonarqube Analysis "){
steps{
withSonarQubeEnv('sonar-server') {
sh ''' $SCANNER_HOME/bin/sonar-scanner -Dsonar.projectName=zomato \
-Dsonar.projectKey=zomato '''
}
}
}
stage("quality gate"){
steps {
script {
waitForQualityGate abortPipeline: false, credentialsId: 'Sonar-token'
}
}
}
stage('Install Dependencies') {
steps {
sh "npm install"
}
}
stage('OWASP FS SCAN') {
steps {
dependencyCheck additionalArguments: '--scan ./ --disableYarnAudit --disableNodeAudit', odcInstallation: 'DP-Check'
dependencyCheckPublisher pattern: '**/dependency-check-report.xml'
}
}
stage('TRIVY FS SCAN') {
steps {
sh "trivy fs . > trivyfs.txt"
}
}
stage("Docker Build & Push"){
steps{
script{
withDockerRegistry(credentialsId: 'docker', toolName: 'docker'){

sh "docker build -t zomato ."
sh "docker tag zomato mudit097/zomato:latest "
sh "docker push mudit097/zomato:latest "
}
}
}
}
stage("TRIVY"){
steps{
sh "trivy image mudit097/zomato:latest > trivy.txt"
}
}
stage('Deploy to container'){
steps{
sh 'docker run -d --name zomato -p 3000:3000 mudit097/zomato:latest'
}
}
}
}
In conclusion, this comprehensive guide has walked you through the essential steps to set up a robust and efficient CI/CD pipeline on an Ubuntu 22.04 T2 Large Instance using Jenkins, Docker, and Trivy. From the initial launch of the AWS EC2 instance to the installation of critical tools such as Jenkins, Docker, and Trivy, as well as the creation of a Sonarqube container, each step has been meticulously covered.

The integration of essential plugins like JDK, Sonarqube Scanner, Nodejs, and OWASP Dependency Check further enhances the pipeline’s capabilities. The creation of a Declarative Pipeline in Jenkins streamlines the development process, and the incorporation of OWASP Dependency Check Plugins fortifies security measures.

The guide concludes with the crucial steps of Docker image build, push, deployment, and, ultimately, the termination of AWS EC2 instances, ensuring a seamless and controlled workflow. By following these steps, you’ve not only established a powerful CI/CD pipeline but also prioritized security and efficiency throughout the entire development lifecycle.

Playlists created and managed seamlessly with Terraform!

Vishal Raju — Tue, 13 Aug 2024 18:40:36 +0000

Project Overview
This project focuses on using Terraform to create and manage multiple Spotify playlists for various occasions such as morning routines, evening relaxation, and party nights. Terraform automates the entire process, making it efficient and scalable.

Prerequisites

Terraform Installed: Ensure Terraform is installed on your machine.
Docker Installed: Make sure Docker is installed and running.
Spotify Account: A Spotify account is required (premium access not necessary).
Spotify Developer Account: Register and create an application to obtain the Client ID and Client Secret.
Spotify Provider for Terraform: Install and configure the Spotify provider for Terraform.
VS Code Editor: Recommended for editing Terraform files.
Steps to Complete the Project

Set Up Terraform Project

Create a new directory for your Terraform project and navigate to it in your terminal.
Create a file named spotifyterra.tf to define your configuration.
Define the Spotify Provider
terraform {
required_providers {
spotify = {
source = "conradludgate/spotify"
version = "0.2.7"
}
}
}

provider "spotify" {
api_key = var.api_key
}

In the spotifyterra.tf file, define the Spotify provider. Refer to the official HashiCorp documentation for the correct provider block.
Obtain API Credentials

To interact with Spotify's API, you'll need a Client ID and Client Secret.
Create a Spotify App

Visit the Spotify Developer Dashboard.
Log in with your Spotify account.
Click "Create an App" and complete the required settings.
Copy the Client ID and Client Secret keys and paste them into a .env file.
Run the Authorization Proxy

Run the following command to start the authorization process:

docker run --rm -it -p 27228:27228 --env-file .env ghcr.io/conradludgate/spotify-auth-proxy

After running this command, you should see a "Authorization Successful" message.
Add Playlists

Use Terraform to add a playlist by defining it in your configuration file.
Add Multiple Playlists

Utilize the data source block in Terraform to access Spotify's platform directly and create multiple playlists automatically.
Verify Playlists Creation

Check your Spotify account to see the newly created playlists.
By following these steps, your playlists will be created and managed seamlessly with Terraform!

Automating Email Notifications for S3 Object Uploads Using AWS SNS

Vishal Raju — Fri, 02 Aug 2024 04:45:27 +0000

In the ever-evolving landscape of cloud computing, automation is no longer a luxury—it's a necessity. The ability to automate processes not only enhances operational efficiency but also strengthens monitoring and security, allowing teams to respond swiftly to changes and potential issues. Amazon Web Services (AWS) provides an extensive suite of tools that make automation both powerful and accessible. In this blog post, I’ll guide you through a practical example: setting up an automated email notification system for when objects are uploaded to an S3 bucket using AWS Simple Notification Service (SNS).

Why Automate Notifications?
Before diving into the implementation, let's consider why automating notifications is important:

Real-Time Monitoring: Automated notifications provide immediate awareness of events, allowing you to stay informed without manually checking logs or dashboards.
Enhanced Security: Being promptly notified of any changes to your storage can be crucial for detecting unauthorized uploads or other suspicious activity.
Workflow Efficiency: Automating notifications can streamline your workflow, ensuring that relevant stakeholders are informed instantly without any manual intervention.
Overview of the Process
In this tutorial, we’ll walk through the following steps:

Creating an S3 Bucket: The storage space where your objects will be uploaded.
Setting Up an SNS Topic: The communication channel through which notifications will be sent.
Subscribing via Email: Configuring the SNS topic to send notifications to an email address.
Updating the SNS Topic Access Policy: Adjusting permissions to allow S3 to publish messages to the SNS topic.
Configuring S3 Notifications: Setting up the S3 bucket to send notifications to the SNS topic upon object uploads.
Step 1: Creating an S3 Bucket
Amazon S3 (Simple Storage Service) is a highly scalable, durable, and secure object storage service. To get started:

Log in to your AWS Management Console.
Navigate to the S3 service.
Click on the "Create Bucket" button.
Give your bucket a unique name, select the appropriate AWS region, and configure the settings according to your requirements.
Click "Create" to finalize the bucket creation.
Your S3 bucket is now ready to store objects. As a best practice, consider enabling versioning and server-side encryption to enhance the security and durability of your data.

Step 2: Setting Up an SNS Topic
Amazon SNS (Simple Notification Service) is a fully managed messaging service that allows you to send messages or notifications from one application to another or to end users. To create an SNS topic:

In the AWS Management Console, navigate to the SNS service.
Click on "Create topic."
Choose the type of topic (Standard or FIFO). For this example, we’ll use a Standard topic.
Enter a name for your topic and configure any additional settings as needed.
Click "Create topic" to finalize.
Your SNS topic is now set up and ready to handle notifications. This topic will act as the communication hub, forwarding messages to subscribers whenever an event occurs.

Step 3: Subscribing via Email
Next, we need to subscribe an email address to the SNS topic so that notifications can be sent to it:

After creating your topic, click on "Create subscription."
Select the protocol as "Email."
Enter the email address where you want to receive the notifications.
Click "Create subscription."
You will receive a confirmation email at the provided address. Click the confirmation link in the email to activate the subscription. Once confirmed, your email address will be successfully subscribed to the topic.

Step 4: Updating the SNS Topic Access Policy
For the SNS topic to receive notifications from S3, you need to modify the topic’s access policy:

Go to the SNS topic you created and click on "Edit" under the "Access policy" section.

In the access policy JSON editor, add a policy that allows S3 to publish to the SNS topic. The policy should look like this:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "s3.amazonaws.com"
},
"Action": "SNS:Publish",
"Resource": "arn:aws:sns:your-region:your-account-id:your-topic-name",
"Condition": {
"ArnLike": {
"aws:SourceArn": "arn:aws:s3:::your-bucket-name"
}
}
}
]
}
Replace your-region, your-account-id, your-topic-name, and your-bucket-name with your specific details.

Save the changes to update the policy.

This policy grants Amazon S3 permission to publish messages to the SNS topic when an event occurs.

Step 5: Configuring S3 Notifications
Finally, you need to configure your S3 bucket to send notifications to the SNS topic whenever an object is uploaded:

Navigate to the S3 service in the AWS Management Console.
Click on the bucket you created earlier.
Go to the "Properties" tab and scroll down to "Event notifications."
Click "Create event notification."
Name your event and select the event type (e.g., "All object create events").
In the "Send to" section, choose "SNS topic" and select the topic you created earlier.
Save the event notification configuration.
Your S3 bucket is now configured to send notifications to the SNS topic whenever a new object is uploaded. These notifications will then be sent to the subscribed email address.

Testing the Setup
To test the setup, upload a file to your S3 bucket. Within a few moments, you should receive an email notification indicating that a new object has been uploaded. The email will contain details such as the name of the bucket, the key of the uploaded object, and the event type.

Conclusion
By integrating Amazon S3 and SNS, you can automate email notifications for object uploads, ensuring that you are always in the loop about changes in your cloud storage. This setup not only improves your monitoring capabilities but also enhances your security posture by alerting you to any unexpected uploads. As cloud environments grow more complex, automations like this become increasingly valuable for maintaining operational efficiency and security.

I hope this guide helps you in setting up your own notification system. Feel free to dive into the AWS documentation for more advanced configurations, such as filtering notifications based on object key prefixes or suffixes.

Excited to hear your thoughts and discuss more about cloud deployment strategies and automation. Feel free to leave comments or reach out if you have any questions!

This blog post provides a comprehensive guide to setting up automated email notifications for S3 object uploads using AWS SNS, with detailed steps and explanations to help readers understand and implement the process.

Deploying a Static Website on Amazon S3 with Terraform!

Vishal Raju — Fri, 26 Jul 2024 06:47:17 +0000

Introduction
Static websites are a simple and effective way to present content without the need for server-side processing. Amazon S3 provides a robust platform for hosting these websites, ensuring high availability and scalability. Terraform, an Infrastructure as Code (IaC) tool, can automate the creation and management of your AWS resources, making the deployment process even more streamlined.

In this guide, we will walk through the process of hosting a static website on Amazon S3 using Terraform, leveraging a modular file structure for clarity and ease of management. By the end of this tutorial, you will have a fully functional static website hosted on Amazon S3, managed entirely through Terraform.

Prerequisites
Ensure you have the following prerequisites before starting:

AWS Account: Sign up for an AWS account if you don’t have one.
Terraform: Download and install Terraform from the official website.
AWS CLI: Install the AWS CLI by following the instructions here.
AWS Credentials: Configure your AWS CLI with your credentials by running aws configure.

Step 1: Create the Project Directory
Begin by creating a directory for your Terraform project and navigating into it.

mkdir my-static-website
cd my-static-website

Step 2: Define the Terraform Configuration
Create a file named terraform.tf and define your provider configuration to set up Terraform with the AWS provider.

terraform.tf

terraform {
required_version = "1.8.5"
required_providers {
aws = {
source = "hashicorp/aws"
version = "5.40.0"
}
}
}

provider "aws" {
profile = "default"
region = "us-east-1"
}
Step 3: Create the S3 Bucket
Create a file named bucket.tf to define your S3 bucket and upload an index.html file to it.

bucket.tf

resource "aws_s3_bucket" "terraform_demo_bucket" {
bucket = "terraform-demo-1808"
}

resource "aws_s3_object" "index_file" {
bucket = aws_s3_bucket.terraform_demo_bucket.id
key = "index.html"
source = "index.html"
content_type = "text/html"
etag = filemd5("index.html")
}

resource "aws_s3_bucket_website_configuration" "website_config" {
bucket = aws_s3_bucket.terraform_demo_bucket.id

index_document {
suffix = "index.html"
}
}
Step 4: Set Up Bucket Policies
Create a file named policy.tf to define your S3 bucket policies to allow public access.

policy.tf

resource "aws_s3_bucket_public_access_block" "public_access_block" {
bucket = aws_s3_bucket.terraform_demo_bucket.id
block_public_acls = false
block_public_policy = false
}

resource "aws_s3_bucket_policy" "bucket_policy" {
bucket = aws_s3_bucket.terraform_demo_bucket.id

policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Sid = "PublicReadGetObject"
Effect = "Allow"
Principal = ""
Action = ["s3:GetObject"]
Resource = "${aws_s3_bucket.terraform_demo_bucket.arn}/"
},
]
})
depends_on = [aws_s3_bucket_public_access_block.public_access_block]
}
Step 5: Configure the Output
Create a file named output.tf to define the output variable for your website’s URL.

output.tf

output "website_url" {
value = "http://${aws_s3_bucket.terraform_demo_bucket.bucket}.s3-website.${aws_s3_bucket.terraform_demo_bucket.region}.amazonaws.com"
}
Step 6: Initialize Terraform
Initialize Terraform to prepare the working directory for managing infrastructure. This command downloads and installs the required provider plugins.

terraform init

Step 7: Validate the Configuration
Validate the Terraform configuration files to ensure the syntax is correct.

terraform validate

Step 8: Plan the Deployment
Generate and review the execution plan to understand the changes Terraform will apply.

terraform plan

Step 9: Apply the Configuration
Apply the Terraform configuration to create the infrastructure.

terraform apply

Step 10: Access Your Website
After the apply process completes, Terraform will output your website's URL. Visit the URL to see your static website live.

Conclusion
Congratulations! You have successfully hosted a static website on Amazon S3 using Terraform. This approach ensures that your infrastructure is version-controlled and easily reproducible. By following this guide, you can quickly deploy static websites for various purposes, such as personal blogs, portfolios, or documentation sites. Explore the power of Infrastructure as Code with Terraform and elevate your web hosting experience!

My project-Sentiment analysis

Vishal Raju — Wed, 24 Jul 2024 14:55:25 +0000

IDEA

TECH STACK/ARCHITECTURE
DESCRIPTION
• The workflow begins with the creation of text files containing the data to be analyzed.
• reads the text files, processes them using Amazon Comprehend APIs, and generates insights such as language, sentiment, and key phrases.
• The results of the analysis can be stored in a database or sent to other AWS services for further processing.

My Project: Implementing Amazon Macie with EventBridge and SNS

Vishal Raju — Wed, 24 Jul 2024 14:29:02 +0000

Project Introduction

In this project we are implementing the functionality of Amazon Macie. Amazon Macie is a machine learning service which can automatically evaluate any data stored in s3, identify and take action based on sensitive data.
• We will be creating a discovery job to identify findings within Macie.
• Findings will be identified using Managed data identifiers and/or Custom data identifier.
• A SNS topic is created that will have the configuration of valid publishers and subscribers to the topic.
• A subscription is created for the above SNS topic with email endpoint.
• A ‘pattern form event’ is created in EventBridge with event source as AWS Macie and Target as SNS topic.
Steps & Workflow

Uploading sensitive data to s3
Create a discovery job in Macie using Managed Data Identifier
Create SNS Topic for notifications
Create an EventBridge rule whenever Macie has any findings
Create Custom Data Identifier Demo
Uploading sensitive data to s3 Let’s begin by creating a new bucket in s3 and upload the data

Upload the text files which contains sensitive data

accesscredentials.txt
AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
AWS_SESSION_TOKEN=AQoDYXdzEPT//////////wEXAMPLEtc764bNrC9SAPBSM22wDOk4x4HIZ8j4FZTwdQWLWsKWHGBuFqwAeMicRXmxfpSPfIeoIYRqTflfKD8YUuwthAx7mSEI/qkPpKPi/kMcGdQrmGdeehM4IC1NtBmUpp2wUE8phUZampKsburEDy0KPkyQDYwT7WZ0wq5VSXDvp75YU9HFvlRd8Tx6q6fE8YQcHNVXAkiY9q6d+xo0rKwT38xVqr7ZD0u0iPPkUL64lIZbqBAz+scqKmlzm8FDrypNC9Yjc8fPOLn9FX9KSYvKTr4rvx3iSIlTJabIQwj2ICCR/oLxBA==
github_key: c8a2f31d8daeb219f623f484f1d0fa73ae6b4b5a
github_api_key: c8a2f31d8daeb219f623f484f1d0fa73ae6b4b5a
github_secret: c8a2f31d8daeb219f623f484f1d0fa73ae6b4b5a
creditcards.txt
American Express
5135725008183484 09/26
CVE: 550

American Express
347965534580275 05/24
CCV: 4758

Mastercard
5105105105105100
Exp: 01/27
Security code: 912
customdata.txt (Australian license plates)

Victoria

1BE8BE
ABC123
DEF-456

New South Wales

AO31BE
AO-15-EB
BU-60-UB

Queensland

123ABC
000ZZZ
987-YXW
employeedata.txt
74323 Julie Field
Lake Joshuamouth, OR 30055-3905
1-196-191-4438x974
53001 Paul Union
New John, HI 94740
Amanda Wells

354-70-6172
242 George Plaza
East Lawrencefurt, VA 37287-7620
GB73WAUS0628038988364
587 Silva Village
Pearsonburgh, NM 11616-7231
LDNM1948227117807
Brett Garza

Create a discovery job in Macie using Managed Data Identifier We are now creating a job to analyze the data in s3 bucket.

Select Managed data identifier as ‘Recommended’ i.e. All managed data identifiers that AWS recommends.

At this stage we are not adding any custom data identifiers. We will be doing that later in the project

The discovery job is now created

Click on the job → Show results → Show findings.

We will be able to see the type of sensitive data that has been identified by Macie. In our case, it is personal, credentials and financials.
Notice that license plates information is not flagged. We need to create custom data identifier for Macie to flag that content. We will be doing that in step 5. In the next step we are setting up a notification so that we get notified whenever Macie has identified a data as sensitive.

Create SNS Topic for notifications Navigate to SNS console and create a SNS topic

Next, we need to create a subscription for this SNS topic.

The subscription is now created:

Create an EventBridge rule whenever Macie has any findings

Select the target for the event. In our case it is SNS topic. Whenever Macie identifies any sensitive data, this event bridge rule will be triggered and it will send that event through SNS and we will be notified (since we subscribed to the SNS topic).

The rule is now enabled

Create Custom Data Identifier Now lets create a custom data identifier to identify the below data. If we notice the below data is not flagged earlier by Macie. customdata.txt (Australian license plates) # Victoria 1BE8BE ABC123 DEF-456 # New South Wales AO31BE AO-15-EB BU-60-UB # Queensland 123ABC 000ZZZ 987-YXW Navigate to Macie → Custom Data Identifier

In the regular expression type the below expression. This is the expression that will identify Australian license plates
([0-9][a-zA-Z][a-zA-Z]-?[0-9][a-zA-Z][a-zA-Z])|([a-zA-Z][a-zA-Z][a-zA-Z]-?[0-9][0-9][0-9])|([a-zA-Z][a-zA-Z]-?[0-9][0-9]-?[a-zA-Z][a-zA-Z])|([0-9][0-9][0-9]-?[a-zA-Z][a-zA-Z][a-zA-Z])|([0-9][0-9][0-9]-?[0-9][a-zA-Z][a-zA-Z])
Now, lets configure a new job and select the created custom data identifier for Macie to identify Australian License plates

The job is now created

If we navigate to show findings we can notice that the license plates data is now flagged by Macie

We also received an email notification via EventBridge

Thanks for reading the article.

My Project: Implementing Amazon Macie with EventBridge and SNS

Vishal Raju — Wed, 24 Jul 2024 14:29:02 +0000

Project Introduction

Uploading sensitive data to s3
Create a discovery job in Macie using Managed Data Identifier
Create SNS Topic for notifications
Create an EventBridge rule whenever Macie has any findings
Create Custom Data Identifier Demo
Uploading sensitive data to s3 Let’s begin by creating a new bucket in s3 and upload the data

Upload the text files which contains sensitive data

American Express
347965534580275 05/24
CCV: 4758

Mastercard
5105105105105100
Exp: 01/27
Security code: 912
customdata.txt (Australian license plates)

Victoria

1BE8BE
ABC123
DEF-456

New South Wales

AO31BE
AO-15-EB
BU-60-UB

Queensland

123ABC
000ZZZ
987-YXW
employeedata.txt
74323 Julie Field
Lake Joshuamouth, OR 30055-3905
1-196-191-4438x974
53001 Paul Union
New John, HI 94740
Amanda Wells

354-70-6172
242 George Plaza
East Lawrencefurt, VA 37287-7620
GB73WAUS0628038988364
587 Silva Village
Pearsonburgh, NM 11616-7231
LDNM1948227117807
Brett Garza

Create a discovery job in Macie using Managed Data Identifier We are now creating a job to analyze the data in s3 bucket.

Select Managed data identifier as ‘Recommended’ i.e. All managed data identifiers that AWS recommends.

At this stage we are not adding any custom data identifiers. We will be doing that later in the project

The discovery job is now created

Click on the job → Show results → Show findings.

Create SNS Topic for notifications Navigate to SNS console and create a SNS topic

Next, we need to create a subscription for this SNS topic.

The subscription is now created:

Create an EventBridge rule whenever Macie has any findings

The rule is now enabled

Create Custom Data Identifier Now lets create a custom data identifier to identify the below data. If we notice the below data is not flagged earlier by Macie. customdata.txt (Australian license plates) # Victoria 1BE8BE ABC123 DEF-456 # New South Wales AO31BE AO-15-EB BU-60-UB # Queensland 123ABC 000ZZZ 987-YXW Navigate to Macie → Custom Data Identifier

The job is now created

If we navigate to show findings we can notice that the license plates data is now flagged by Macie

We also received an email notification via EventBridge

Thanks for reading the article.

"Connecting to an EC2 Instance Using SSH: Easy Steps to Access Your Instance"

Vishal Raju — Thu, 04 Jul 2024 06:30:01 +0000

Gather EC2 Instance Details
Public DNS (or Public IP): Identify the Public DNS (or IP address) of your EC2 instance. You can find this in the AWS Management Console under the Instances section.
Key Pair File (.pem): Ensure you have the private key (.pem file) that was used when launching the instance. If you don’t have it, you might need to create a new key pair and associate it with your instance.
Set Permissions for Your Key Pair File
Ensure that the permissions on your key pair file are set correctly to maintain security. In a terminal or command prompt, use the following command:
bash
Copy code
chmod 400 /path/to/your-key-pair.pem
Replace /path/to/your-key-pair.pem with the actual path to your key pair file.
Open Your Terminal or Command Prompt
Open a terminal window (Linux or macOS) or a command prompt (Windows).
Connect Using SSH
Use the ssh command to connect to your EC2 instance. The syntax is:

bash
Copy code
ssh -i /path/to/your-key-pair.pem ec2-user@your-instance-public-dns
Replace:
/path/to/your-key-pair.pem: Path to your .pem file.
ec2-user: Username for your instance (can vary by operating system; for example, ubuntu for Ubuntu instances).
your-instance-public-dns: Public DNS (or IP address) of your EC2 instance.
For example:

bash
Copy code
ssh -i ~/Downloads/your-key-pair.pem ec2-user@ec2-11-22-33-44.compute-1.amazonaws.com

Authenticate and Connect
If it’s your first time connecting to this instance, you may see a message about the authenticity of the host. Type yes to continue connecting.
You should now be connected to your EC2 instance via SSH.
Post-Connection Tasks
Once connected, you can execute commands on your EC2 instance terminal just like you would on a local terminal.
Troubleshooting Tips:
Security Group Settings: Ensure that your EC2 instance’s security group allows SSH access (port 22) from your current IP address or IP range.
Instance State: Verify that your EC2 instance is running and reachable over the network.

This detailed guide should help you connect to your EC2 instance securely using SSH.

A DEEP DIVE INTO TERRAFORM

Vishal Raju — Thu, 27 Jun 2024 17:13:03 +0000

What is Infrastructure as Code with Terraform?
Getting Started with Terraform on AWS
Infrastructure as Code (IaC) lets you manage infrastructure with configuration files. Terraform, HashiCorp's IaC tool, offers several
advantages as follows:
• Multi-Cloud Management: Manage resources across AWS, Azure, GCP, etc.
• Declarative Language: Write and maintain infrastructure code easily.
• State Management: Track resource changes with Terraform's state file.
• Version Control: Safely collaborate using version control systems.

Terraform Workflow

Scope: Identify infrastructure needs.
Author: Write configuration files.
Initialize: Install necessary plugins.
Plan: Preview changes.
Apply: Implement the changes.

Collaboration and Tracking
• State File: Acts as the source of truth for your infrastructure.
• HCP Terraform: Share state securely, prevent race conditions, and integrate with VCS like GitHub.

1)Install Terraform
Creating Your First AWS EC2 Instance with Terraform
To get started with Terraform and AWS, follow these steps:

Prerequisites: o Install Terraform CLI (1.2.0+) and AWS CLI. o Have an AWS account with credentials ready.
Set AWS Credentials:
bash
Copy code
$ export AWS_ACCESS_KEY_ID=
$ export AWS_SECRET_ACCESS_KEY=
Write Configuration:
o Create a directory and main.tf file:
bash
Copy code
$ mkdir learn-terraform-aws-instance
$ cd learn-terraform-aws-instance
$ touch main.tf
o Paste the configuration into main.tf:
hcl
Copy code
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4.16"
}
}

required_version = ">= 1.2.0"
}

provider "aws" {
region = "us-west-2"
}

resource "aws_instance" "app_server" {
ami = "ami-830c94e3"
instance_type = "t2.micro"

tags = {
Name = "ExampleAppServerInstance"
}
}

Initialize and Apply Configuration: bash Copy code $ terraform init $ terraform apply
Inspect State: bash Copy code $ terraform show That's it! You've now created your first AWS EC2 instance using Terraform. Explore further by modifying configurations and diving deeper into Terraform's capabilities. Happy provisioning!

2)Change infrastructure
Prerequisites
Ensure you have:
• Terraform CLI (1.2.0+) installed.
• AWS CLI configured with a default profile.
Setting Up Your Project

Create Directory and Configuration File: Start by creating a directory and main.tf file: bash Copy code $ mkdir learn-terraform-aws-instance $ cd learn-terraform-aws-instance $ touch main.tf
Configure main.tf: Add AWS instance configuration to main.tf: hcl Copy code terraform { required_providers { aws = { source = "hashicorp/aws" version = "~> 4.16" } }

required_version = ">= 1.2.0"
}

provider "aws" {
region = "us-west-2"
}

resource "aws_instance" "app_server" {
ami = "ami-830c94e3"
instance_type = "t2.micro"

tags = {
Name = "ExampleAppServerInstance"
}
}

Initialize and Apply Configuration: Initialize and apply your configuration: bash Copy code $ terraform init $ terraform apply

3)Updating Infrastructure
To update instance configuration (e.g., change AMI):

Modify main.tf: Update ami under aws_instance.app_server: diff Copy code resource "aws_instance" "app_server" {
ami = "ami-830c94e3"
ami = "ami-08d70e59c07c61a3a" // New AMI ID instance_type = "t2.micro"
Apply Changes: Apply changes to update the instance: bash Copy code $ terraform apply Execution Plan • Terraform's plan (terraform apply) shows actions like creating new resources or updating existing ones. • Changing the AMI forces recreation (-/+ destroy and then create replacement) due to AWS constraints. Conclusion Terraform simplifies AWS resource management with automation and consistency.

4)Destroy infrastructure
Managing Infrastructure Lifecycle with Terraform
In this tutorial, you've learned how to create and update an EC2 instance on AWS using Terraform. Now, let's explore how to destroy resources when they are no longer needed.
Why Destroy?
• Cost Reduction: Stop paying for unused resources.
• Security: Minimize exposure by removing unnecessary components.
Destroying Resources
To destroy managed resources:
bash
Copy code
$ terraform destroy
Execution Plan
Terraform outlines what will be destroyed:
text
Copy code

destroy

Terraform will perform the following actions:

# aws_instance.app_server will be destroyed

resource "aws_instance" "app_server" {
- ami = "ami-08d70e59c07c61a3a" -> null
- arn = "arn:aws:ec2:us-west-2:561656980159:instance/i-0fd4a35969bd21710" -> null ##...

Plan: 0 to add, 0 to change, 1 to destroy.
Confirm and Execute
Terraform requires confirmation before proceeding:
text
Copy code
Do you really want to destroy all resources?
Terraform will destroy all your managed infrastructure, as shown above.
There is no undo. Only 'yes' will be accepted to confirm.

Enter a value:
Finalization
Once confirmed, Terraform begins destroying the resources:
text
Copy code
aws_instance.app_server: Destroying... [id=i-0fd4a35969bd21710]
aws_instance.app_server: Destruction complete after 31s

Destroy complete! Resources: 1 destroyed.
Conclusion
By following these steps, you've seen how Terraform efficiently manages the lifecycle of your cloud infrastructure, ensuring cost-effectiveness and security.

5)Define input variables
Streamlining Infrastructure Management with Terraform Variables
In this tutorial, you'll optimize your Terraform setup by introducing variables for more flexible infrastructure configuration.
Prerequisites
Ensure:
• Terraform CLI (1.2.0+) is installed.
• AWS CLI is configured with a default profile.
• Directory learn-terraform-aws-instance exists with main.tf configured as specified.
Configuring Variables

Create variables.tf: Define a instance_name variable to customize the EC2 instance's Name tag: hcl Copy code variable "instance_name" { description = "Name tag for the EC2 instance" type = string default = "ExampleAppServerInstance" }
Update main.tf: Modify the aws_instance resource to utilize the instance_name variable: hcl Copy code resource "aws_instance" "app_server" { ami = "ami-08d70e59c07c61a3a" instance_type = "t2.micro"

tags = {
Name = var.instance_name
}
}
Applying Configuration

Initialize and Apply: Initialize Terraform and apply the configuration: bash Copy code $ terraform init $ terraform apply
Customize Instance Name: Override the default instance name using -var flag during apply: bash Copy code $ terraform apply -var "instance_name=YetAnotherName" Verification • Terraform presents an execution plan before applying changes for clarity and safety. • Confirm changes when prompted, observing Terraform's efficient handling of resource updates. Conclusion By using Terraform variables, you've enhanced your infrastructure's adaptability and reduced configuration repetition.

6)Query data with outputs
Streamlining Terraform with Output Values
In this guide, we'll maximize Terraform's capabilities by utilizing output values to extract essential information about our AWS infrastructure.
Prerequisites
Ensure:
• Terraform CLI (1.2.0+) is installed.
• AWS CLI is configured with a default profile.
• You have a directory named learn-terraform-aws-instance with configured main.tf and variables.tf.
Initial Setup
Recap your current configuration in main.tf and variables.tf:
hcl
Copy code

main.tf

terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4.16"
}
}

required_version = ">= 1.2.0"
}

provider "aws" {
region = "us-west-2"
}

resource "aws_instance" "app_server" {
ami = "ami-08d70e59c07c61a3a"
instance_type = "t2.micro"

tags = {
Name = var.instance_name
}
}

variables.tf

variable "instance_name" {
description = "Name tag for the EC2 instance"
type = string
default = "ExampleAppServerInstance"
}
Defining Outputs
Create outputs.tf to specify outputs for the instance's ID and public IP:
hcl
Copy code

outputs.tf

output "instance_id" {
description = "ID of the EC2 instance"
value = aws_instance.app_server.id
}

output "instance_public_ip" {
description = "Public IP address of the EC2 instance"
value = aws_instance.app_server.public_ip
}
Applying Configuration

Initialize and Apply Configuration: bash Copy code $ terraform init $ terraform apply
Inspect Output Values: Upon applying, Terraform displays outputs such as instance_id and instance_public_ip, crucial for managing and automating your infrastructure. bash Copy code Outputs:

instance_id = "i-0bf954919ed765de1"
instance_public_ip = "54.186.202.254"
Conclusion
Utilizing Terraform outputs enhances operational visibility and automation by providing essential resource details. These outputs are seamlessly integrable with other infrastructure components or subsequent Terraform projects.
Cleanup (Optional)
If not continuing to further tutorials, clean up your infrastructure:
bash
Copy code
$ terraform destroy
Confirm destruction to optimize cost and security by removing unused resources.

7)Store remote state
Getting Started with Terraform and HCP Terraform
Overview
Terraform simplifies infrastructure management by treating it as code. This guide helps you set up Terraform to provision AWS resources and integrate with HashiCorp Cloud Platform (HCP) Terraform for centralized state management.
Prerequisites

Configuration Setup: Create a directory named learn-terraform-aws-instance and save the following in main.tf: hcl Copy code terraform { required_providers { aws = { source = "hashicorp/aws" version = "~> 4.16" } } required_version = ">= 1.2.0" }

provider "aws" {
region = "us-west-2"
}

resource "aws_instance" "app_server" {
ami = "ami-08d70e59c07c61a3a"
instance_type = "t2.micro"
}

Initialize and Apply: Initialize Terraform and apply your configuration: bash Copy code $ terraform init $ terraform apply Setting up HCP Terraform
Log in to HCP Terraform: Use the Terraform CLI to log in and authenticate with HCP Terraform: bash Copy code $ terraform login
Configure for HCP Terraform: Modify main.tf to integrate with HCP Terraform: hcl Copy code terraform { cloud { organization = "organization-name" workspaces { name = "learn-terraform-aws" } }

required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4.16"
}
}
}

Initialize and Migrate State: Re-initialize Terraform to migrate state to HCP Terraform: bash Copy code $ terraform init Confirm migration and delete the local state file.

Applying Configuration and Managing Workspace

Set Workspace Variables: Configure AWS credentials in HCP Terraform's workspace variables.
Apply Configuration: Apply your Terraform configuration to ensure infrastructure consistency: bash Copy code $ terraform apply Destroying Infrastructure Clean up resources using: bash Copy code $ terraform destroy

Conclusion
You've completed the essentials for Terraform and HCP Terraform integration

Frequently Asked Questions:-
1.Do I need prior programming or infrastructure experience to follow the guide?
No, prior programming or infrastructure experience is not necessary to follow the guide. It is designed to cater to beginners and assumes no prior knowledge of Terraform. The guide provides step-by-step explanations and examples to help newcomers understand and apply the concepts effectively.

2.Are there any prerequisites for using Terraform?
The guide may mention a few prerequisites, such as having a basic understanding of cloud computing concepts and having an account with a cloud provider (if you plan to provision resources in the cloud). Additionally, it may recommend installing Terraform and a text editor suitable for writing code.

3.Does the guide provide hands-on examples and exercises?
Yes, the Terraform Beginner's Guide typically includes hands-on examples and exercises throughout the content. These examples help solidify the concepts and allow readers to practice writing Terraform configurations, executing commands, and managing infrastructure resources.

4.How does Infrastructure as Code handle infrastructure updates and changes?
Infrastructure as Code tools typically handle updates and changes by comparing the desired state defined in the code with the current state of the infrastructure. When changes are made to the code, the tools generate an execution plan that outlines the modifications required to achieve the desired state. This plan can be reviewed and then applied to update or modify the infrastructure accordingly.

5.Can I use Infrastructure as Code for existing infrastructure?
Yes, Infrastructure as Code can be used for existing infrastructure. By defining the existing infrastructure in code, you can capture its current state and make modifications to it using code-based configuration files. This approach allows you to manage existing infrastructure in a consistent and automated manner.