Forem: Ashwathy Nair

Deploy Scalable VPC Architecture on AWS Cloud

Ashwathy Nair — Sun, 30 Mar 2025 06:08:52 +0000

Deploy a Modular and Scalable Virtual Network Architecture with Amazon VPC.

Pre-Requisites

1] AWS Account: Ensure you have an AWS account with necessary permissions (IAM Role or User with privileges).
2] Source Code: Prepare the web application repository hosted on GitHub/ BitBucket.
Source Code :

ashunair / VPC-Architecture

Deploy the scale the VPC architecture using AWS

Deploy Scalable VPC Architecture on AWS Cloud

Goal

Deploy a Modular and Scalable Virtual Network Architecture with Amazon VPC.

Pre-Requisites

You must be having an AWS account to create infrastructure resources on AWS cloud.
Source Code

Pre-Deployment

Customize the application dependencies mentioned below on AWS EC2 instance and create the Golden AMI.

AWS CLI
Install Apache Web Server
Install Git
Cloudwatch Agent
Push custom memory metrics to Cloudwatch.
AWS SSM Agent

VPC Deployment

Build VPC network ( 192.168.0.0/16 ) for Bastion Host deployment as per the architecture shown above.
Build VPC network ( 172.32.0.0/16 ) for deploying Highly Available and Auto Scalable application servers as per the architecture shown above.
Create NAT Gateway in Public Subnet and update Private Subnet associated Route Table accordingly to route the default traffic to NAT for outbound internet connection.
Create Transit Gateway and associate both…

View on GitHub

Pre-Deployment

3] Golden AMI:

Launch an EC2 instance with a public subnet. (Ubuntu 24.0)

4] Install dependencies:

Update sudo and install Apache Web server, Git, CloudWatch Agent, and AWS SSM Agent.
Install Apache Web Server:
sudo yum install httpd sudo enable systemctl httpd sudo start systemctl httpd

Install Git
sudo yum install git -y git --version
Install CloudWatch Agent

□ Download the CloudWatch Agent- sudo yum install amazon-cloudwatch-agent

□ Configure and start the agent: - sudo systemctl enable amazon-cloudwatch-agent

□ Save the File: Ensure the file is saved at this path: /opt/aws/amazon-cloudwatch-agent/bin/memory_metrics.json.

□ Example command to copy the file: sudo cp /home/ubuntu/VPC-Architecture/VPC Architecture/memory_metrics.json /opt/aws/amazon-cloudwatch-agent/bin/memory_metrics.json

□ Apply the Configuration File: If you prefer to keep the file in the original location, run:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/home/ubuntu/VPC-Architecture/VPC\Architecture/memory_metrics.json -s

Install AWS SSM Agent

□ Install the AWS SSM Agent - sudo yum install amazon-ssm-agent -y

□ Start and check status of cloud watch service: -
sudo systemctl start amazon-cloudwatch-agent
sudo systemctl status amazon-cloudwatch-agent

Navigate to the CloudWatch Dashboard in AWS and go to Metrics. You should see the custom metrics created.

Check the CloudWatch logs captured in amazon-cloudwatch-agent.log

5] Create AMI - Amazon Master Image

Navigate to the AWS EC2 Dashboard.
Select the Stopped EC2 Instance.
From the Actions Menu, choose:
□ Image and templates > Create Image.
Provide the following details:
□ Image Name: Give your AMI a descriptive name, e.g., GoldenAMI-v1.
□ Description: Mention what this AMI includes.
□ No Reboot: Leave unchecked (default) to ensure a clean AMI.
Click Create Image and wait for the process to complete.

6] Test the Golden AMI by launching new EC2 instance with created AMI

Launch an EC2 Instance from the Golden AMI
Log in to your AWS Management Console.
Go to the EC2 Dashboard.
Click on Launch Instance and choose My AMIs.
Select your Golden AMI and configure the instance settings:

□ Select an instance type (e.g., t2.micro for testing).
□ Attach the appropriate security group.
□ Assign a public IP for SSH testing.
□ Add necessary storage if required.
□ Assign an IAM role that matches your AMI's requirements (e.g., SSM access).
□ update the user data with script in the image

7] Verify Connectivity

Connect to the instance using SSH :ssh -i "key_name.pem" ubuntu@ec2-18-222-53-96.us-east-2.compute.amazonaws.com
Create a new IAM role and assign to EC2

Step 1: Access the IAM Console

Go to the IAM Console.
Click on the "Create role" button.

Step 2: Specify Trusted Entity

Select trusted entity type: Choose AWS service.
Use case for other AWS service: Select EC2.
Click Next.

Step 3: Attach Policies

Search for and select the following policies: AmazonSSMManagedInstanceCore (required for SSM functionality) CloudWatchAgentServerPolicy (if you're using CloudWatch Agent for monitoring).
Click Next.

Step 4: Add Tags (Optional)

Add any tags if required for identification or cost tracking.
Click Next.

Step 5: Review and Create

Provide a Role name: Example- EC2_SSM_Access_Role.
Review the permissions and confirm that AmazonSSMManagedInstanceCore is included.
Click Create role.

Step 6: Attach Role to the EC2 Instance

Go to the EC2 Console.
Select the instance you want to associate the role with.
Click Actions > Security > Modify IAM Role.
Select the newly created IAM role (EC2_SSM_Access_Role).
Click Update IAM role.

Setup Permission for S3 bucket

□ Navigate to IAM, in the dashboard select Roles.
□ Click Create Role.
□ Select permission policies, AmazonS3ReadOnlyAccess and other required policies

VPC Deployment

Create VPC network ( 192.168.0.0/16 ) for Bastion Host deployment

Create a VPC network ( 172.32.0.0/16 ) to deploy scalable application servers

Access the VPC dashboard to display the created VPC

Create Internet gateways to attach the gateways to Bastion app (ed-vpc-01)

Create a public subnet for bastion app (ed-vpc-01) and configure the subnet

Create a two public subnet and two private subnet for application server VPC (ed-vpc-02) with IP- 172.32.0.0/16 .

Public Subnet 1 (ed-pub-sub-01): 172.32.1.0/24
Public Subnet 2 (ed-pub-sub-02): 172.32.2.0/24
Private Subnet 1 (ed-priv-sub-01): 172.32.3.0/24
Private Subnet 2 (ed-priv-sub-02): 172.32.4.0/24

Create NAT Gateway(ed-nat-gw-01) in Public Subnet (ed-pub-sub-02) and update Private Subnet associated Route Table accordingly to route the default traffic to NAT for outbound internet connection.

Create public route table (ed-rt-vpc-01-pub) for VPC (ed-vpc-01)
Create public route table (ed-rt-vpc-02-pub) for VPC (ed-vpc-02)
Create private route table (ed-vpc-02-priv-01) for VPC (ed-vpc-02)
Subnet associated with route table (ed-rt-vpc-02-pub)
Subnet associated with route table (ed-rt-vpc-01-pub)
Subnet associated with route table (ed-rt-vpc-02-pub)

Create Internet Gateway for each VPC and Public Subnet associated Route Table accordingly to route the default traffic to IGW for inbound/outbound internet connection.
Create Transit Gateway and associate both VPCs to the Transit Gateway for private communication.

Create Cloudwatch Log Group with two Log Streams to store the VPC Flow Logs of both VPCs and enable Flow Logs for both VPCs and push the Flow Logs to Cloudwatch Log Groups and store the logs in the respective Log Stream for each VPC.

Create Security Group for bastion host allowing port 22 from public.

Create Target Group (tg-01) and associate it with ASG.
Create Network Load balancer in Public Subnet and add Target Group as target.

Create Auto Scaling Group with Min: 2 Max: 4 with two Private Subnets associated to 1a and 1b zones.

Update route53 hosted zone with CNAME record routing the traffic to NLB. For that initially create record in Route53 > Hosted Zone

Validation

As DevOps Engineer login to Private Instances via Bastion Host.

Automated CI/CD pipeline to deploy app on Google cloud using GKE

Ashwathy Nair — Fri, 21 Feb 2025 21:33:16 +0000

In collaboration with Disha Patel

Introduction:

This document outlines the steps required to create a Cloud CICD pipeline to deploy a containerized application on Google Kubernetes Engine (GKE) by defining two environments — staging and production. Follow these steps to set up and manage your application in a GKE environment.

Code reference:

ashunair / CICD-Pipeline

Google Cloud Platform based CI-CD pipeline

Cloud CI/CD Pipeline Deployment on GKE

Overview

This project provides a step-by-step guide to building a Cloud CI/CD pipeline for deploying a containerized application on Google Kubernetes Engine (GKE). It covers the setup of staging and production environments, ensuring smooth and automated deployments using Google Cloud services.

Key Highlights:

GKE Cluster Setup: Creating Kubernetes clusters for staging and production.

Artifact Registry Configuration: Storing and managing Docker images.

Cloud Build Integration: Automating build and deployment pipelines.

Cloud Deploy Setup: Managing continuous deployment to GKE.

Skaffold for Kubernetes Automation: Simplifying deployment processes.

GitHub to Cloud Build Triggers: Enabling automated deployments on code commits.

This project is available on Devpost Blog: Documentation Link with detailed explanations and images to guide you through each step. For the complete implementation, refer to this code repository: GitHub Repo.

View on GitHub

Setup Google Cloud SDK(required if you are pushing your code from local/on-prem)

Ensure you have Google Cloud SDK installed on your machine. This includes gcloud and kubectl command-line tools. Note that us-central1 is used across all the services in this project.
Configure gcloud when using Cloud Shell
Authenticate and set the project you want to work with using the commands: gcloud auth login and gcloud config set project [YOUR_PROJECT_ID].

1. Create a GKE Cluster

Create two Kubernetes cluster in GKE using the command with different naming conventions to identify the environments:

gcloud container clusters create [CLUSTER_NAME] — zone [ZONE] — num-nodes [NUM_NODES] — machine-type e2-medium — disk-size 10GB

gcloud container clusters create stg-pipeline --zone us-central1-a --num-nodes 2 --machine-type e2-small --disk-size 10GB

1.1 — Creating cluster using Cloud Shell

1.2 — Kubernetes Clusters for Staging and Prod

2. Setup Artifact Registry

Create a repository to store the docker images. Choose Docker in format, region should be us-central1 and keep rest as default.

2.1 — Docker Artifact Repository

2.2 — Docker Artifact Repository Metadata

3. File setup

Create git repo for your website and follow below mentioned file structure:

.
└── CICD-Pipeline/
├── Demo/
│ └── Index.html
├── DockerFile
├── cloudbuild. yaml
├── clouddeploy.yaml
├── kubernetes.yaml
└── skaffold.yaml

3.1 Create a Dockerfile:
Here is a step by step guild about how to define your Dockerfile. However, below is all we need for this project.

FROM nginx:alpine
COPY ./demo /usr/share/nginx/html 
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

FROM nginx:alpine
- This sets the base image as the official Nginx Alpine image.
- Alpine is a minimal Linux distribution, making the image smaller and more efficient.
COPY ./demo /usr/share/nginx/html
- This copies the local ./demo directory (which contains static files like HTML, CSS, JavaScript) into the Nginx default document root at /usr/share/nginx/html.
EXPOSE 80
- This declares that the container will listen on port 80 (default HTTP port).
- However, this does not actually publish the port — it just informs Docker that this port is intended to be exposed.
CMD ["nginx", "-g", "daemon off;"]
- This is the command that runs when the container starts.
- nginx is started with the -g "daemon off;" flag to keep it running in the foreground.
- This prevents the container from exiting immediately, ensuring that Nginx stays active.

3.2 Create cloudbuild.yaml

steps:
# 1. Docker Build
- name: 'gcr.io/cloud-builders/docker'
  args: ['build', '-t', 'us-central1-docker.pkg.dev/innate-valor-451418-i0/app-repo/cicd-app:$SHORT_SHA', '.']
# 2. Docker Push
- name: 'gcr.io/cloud-builders/docker'
  args: ["push", "us-central1-docker.pkg.dev/innate-valor-451418-i0/app-repo/cicd-app:$SHORT_SHA"]

# 3. create cloud deploy pipeline 
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
  entrypoint: 'bash'
  args:
    - '-c'
    - |
      if ! gcloud deploy delivery-pipelines describe demopipeline --region=us-central1 &> /dev/null; then
        echo "Creating Cloud Deploy pipeline..."
        gcloud deploy apply --file=clouddeploy.yaml --region=us-central1 --project=$PROJECT_ID
      else
        echo "Cloud Deploy pipeline already exists."
      fi
# 4. cloud deploy pipeline release 
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
  entrypoint: 'bash'
  args:
  - '-c'
  - >
    gcloud deploy releases create release-$BUILD_ID
    --delivery-pipeline=cicd-app
    --region=us-central1
    --source=./
    --images=sample-app=us-central1-docker.pkg.dev/innate-valor-451418-i0/app-repo/cicd-app:$SHORT_SHA

# 5. Logging Configuration
options:
  logging: CLOUD_LOGGING_ONLY

Docker Build
- This step builds a Docker image from the current directory (.).
- The image is tagged with a unique identifier $SHORT_SHA (a short commit hash) to track different builds.
- The image is stored in Artifact Registry under: us-central1-docker.pkg.dev/chrome-sum-415007/demopipeline/demopipeline:$SHORT_SHA.
Docker Push
- This step pushes the built Docker image to google artifact registry, making it available for deployment.
Create Cloud Deploy Pipeline
- Checks if the Google Cloud Deploy pipeline named demopipeline exists.
- If it does not exist, it creates the pipeline using clouddeploy.yaml.
- If it already exists, it skips creation.
Create a New Release for Deployment
- Creates a new release in Google Cloud Deploy named release-$BUILD_ID.
- Uses the demopipeline to deploy the newly built image.
- References the container image from artifact registry.
Logging Configuration
- Ensures logs are stored in google cloud logging only.

3.3 Create a clouddeploy.yaml:

# 1. Delivery Pipeline Definition
apiVersion: deploy.cloud.google.com/v1beta1
kind: DeliveryPipeline
metadata:
 name: cicd-app
description: cicd-app application 
serialPipeline:
 stages:
 - targetId: staging
 - targetId: prod
---

# 2. Staging Environment Target

apiVersion: deploy.cloud.google.com/v1beta1
kind: Target
metadata:
 name: staging
description: "staging cluster"
gke:
 cluster: projects/innate-valor-451418-i0/locations/us-central1-a/clusters/stg-pipeline
---

# 3. Production Environment Target

apiVersion: deploy.cloud.google.com/v1beta1
kind: Target
metadata:
 name: prod
description: prod cluster
requireApproval: true
gke:
 cluster: projects/innate-valor-451418-i0/locations/us-central1-a/clusters/prod-pipeline

Delivery Pipeline Definition
- Defines the cloud deploy pipeline named demopipeline.
- Uses a serial pipeline (deploys sequentially).
- Has two stages: > staging (first deployment stage) > prod (final production stage)
Staging Environment Target
- Defines a staging environment in a GKE cluster.
- Cluster location: us-central1.
- This is the first step in the pipeline.
Production Environment Target
- Defines the production environment.
- Uses a separate GKE cluster in us-central1-a.
- requireApproval: true → Requires manual approval before deploying to production.

3.4 Create Kubernetes Deployment and Service yaml file

# 1. Deployment Configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web-app
  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
        - name: web-app
          image: sample-app
          ports:
            - containerPort: 80
---
# 2. Service Configuration

apiVersion: v1
kind: Service
metadata:
  name: web-app
spec:
  type: LoadBalancer
  selector:
    app: web-app
  ports:
    - port: 80
      targetPort: 80

Deployment Configuration
- Creates a Deployment named demopipeline.
- Runs 2 replicas for high availability.
- Uses label selectors (app: demopipeline) to identify Pods.
- Deploys a container with the name demopipeline, using the image sample-app.
- Exposes port 80 inside the container for web traffic.
Service Configuration
- Creates a Service named demopipeline.
- Exposes the application externally using a LoadBalancer.
- Routes traffic to Pods matching app: demopipeline.
- Listens on port 80 and forwards requests to port 80 inside the Pods.

3.5 Create Skaffold manifest file to automate Kubernetes deployments

# 1. Skaffold Configuration Overview

apiVersion: skaffold/v2beta16
kind: Config
metadata:
  name: web-app

# 2. Build Configuration

build:
  artifacts:
  - image: sample-app
    context: .
    docker:
      dockerfile: Dockerfile
  tagPolicy:
    gitCommit: {}
  local:
    useBuildkit: false

# 3. Deployment Configuration

deploy:
  kubectl:
    manifests:
    - kubernetes.yaml

# 4. Profile Configuration (For Google Cloud Build)

profiles:
- name: gcb

Skaffold Configuration Overview
- Defines Skaffold configuration (v2beta16 API version).
- The project name is demopipeline.
Build Configuration
- build: Defines how to build the container image for the application.
- artifacts: Uses the Dockerfile in the current directory and builds the Docker image as sample-app.
- tagPolicy: Uses the Git commit hash (gitCommit: {}) for tagging images, ensuring unique versions.
- local: Local Docker build.
Deployment Configuration
- Uses kubectl to deploy the application to Kubernetes.
- Deploys the kubernetes.yaml file, which likely contains a Deployment and Service definition.
Profile Configuration (For Google Cloud Build)
- Defines a Skaffold profile named gcb (Google Cloud Build).
- Profiles allow different environments (e.g., local vs. cloud) by specifying different build and deployment settings.

Now that we have a Skaffold-based Kubernetes deployment pipeline, the next step is to automate the build and deployment process using GitHub and cloud build. When code is pushed and committed to GitHub, it should automatically trigger Cloud Build, which will:

✅ Build the Docker image using the cloudbuild.yaml file.
✅ Push the image to Google Artifact Registry.
✅ Deploy the application to Kubernetes using Google Cloud Deploy.

4. Set Up a Cloud Build Trigger in Google Cloud

4.1 Setup connection from GitHub to Cloud Build

4.1.1 — Connecting a GitHub repository to Google Cloud Build triggers.

Go to Google Cloud Console → Navigate to Cloud Build.
Click Triggers → Create Trigger.
Select “Connect Repository” and link your GitHub repository.

4.2 Configure the Trigger

4.2.1 — Configuring a Cloud Build trigger in Google Cloud

4.2.2 — Cloud Build Trigger

Trigger Type: Push to a branch (e.g., main or dev).
Branch Filter: Set it to main or any specific branch.
Build Configuration: Choose “cloudbuild.yaml” (it should already exist in the repo).
Service Account: Use a service account with Cloud Build, Artifact Registry, and Cloud Deploy permissions.
Save the trigger and test it with a sample push.

5. Final Execution

The setup is complete, and when new code is pushed to the selected branch (e.g., staging), it will trigger Cloud Build, which interacts with the cloudbuild.yaml file and executes the steps sequentially.

5.1 — Cloud Build waiting for approval

Here, as I have set up approval to execute the build, each time new code is committed, it will require approval before proceeding. Once approved, it will run the commands mentioned in cloudbuild.yaml. For detailed information, refer to section 3.2 above.

5.2— Cloud Build Succeeded

You will see the release in Staging here in Cloud Deploy:

5.3 — Delivery Pipeline created

5.4 — Release in Staging

And with that, workload is created in GKE Cluster:

5.5 — Workload created in GKE

Inside the workload, you can see the endpoint created. Click on the IP to see your webpage(index.html in this project).

5.6 — Endpoint IP highlighted
Now, navigate to Cloud Deploy and you can promote it to Production. It will require an approval.

5.7 — Getting ready for Prod Release

5.8 — Approval required for Prod Release

Hence after the prod environment is approved, again you can see the release in GKE Workload for Prod.

5.9 — Prod release in GKE

Thus, it will create another service endpoint for Prod.

6 Additional Steps:

Since vulnerabilities can be introduced into the artifact during the architecture phase, we enable the Container Scanning API to scan Docker images and detect any anomalies.

Additionally, we set up monitoring alerts for the approval process when transitioning from staging (STG) to production (PROD). This includes configuring an email approval system and integrating Slack notifications for approvals.

Steps to add notification channel in Google Cloud using email:

Open Cloud Monitoring → Go to alerting.
Edit Notification Channels → Click Add New under Email.
Add Email Address → Enter approval email & click Save.
From the cloud console, go to the delivery pipeline and select the pipeline.
To configure alerts, select recommended alerts and set appropriate required policies.
Select the email channel.
Create Policy → Alerts will now trigger email notifications.

6.1 Creating email notification channel

Steps to add notification channel in Google Cloud by integrating Slack:

Open Cloud Monitoring → Go to alerting.
Edit Notification Channels → Click Add New under Slack.
This will pop-up a new window for permission to Slack app → Allow.
Add Slack channel → Save.
From the cloud console, go to the delivery pipeline and select the pipeline.
To configure alerts, select recommended alerts and set appropriate required policies.
Select the Slack channel.
Create Policy → Alerts will now trigger Slack notifications.

6.2 Permission access to Slack app via Google Cloud

6.3 Setting Slack notification channel in GCP

6.4 Creating policy to set up email and Slack alerts

Whenever a new commit is made to the code in the delivery pipeline, the alert notification will be sent to the email and Slack channel.

6.5 CI/CD pipeline approval pending to PRD

6.6 CI/CD pipeline approval alert notification in email

6.7 CI/CD pipeline approval alert notification in slack channel

🎉Phewwww 😮‍💨... the hard work paid off 🎉

Finally, you have successfully automated your CI/CD pipeline! 🚀

Now sit back, grab a coffee ☕, and watch the magic happen—because from now on, your code will deploy itself 😎

Got any questions or need help troubleshooting? Drop them below! ⬇️💬

References:

Forem: Ashwathy Nair

Deploy Scalable VPC Architecture on AWS Cloud

Deploy a Modular and Scalable Virtual Network Architecture with Amazon VPC.

ashunair / VPC-Architecture

Deploy the scale the VPC architecture using AWS

Deploy Scalable VPC Architecture on AWS Cloud

TABLE OF CONTENTS

Goal

Pre-Requisites

Pre-Deployment

VPC Deployment

Pre-Deployment

VPC Deployment

Validation

Automated CI/CD pipeline to deploy app on Google cloud using GKE

Introduction:

ashunair / CICD-Pipeline

Google Cloud Platform based CI-CD pipeline

Cloud CI/CD Pipeline Deployment on GKE

Overview

Key Highlights:

Setup Google Cloud SDK(required if you are pushing your code from local/on-prem)

1. Create a GKE Cluster

2. Setup Artifact Registry

3. File setup

4. Set Up a Cloud Build Trigger in Google Cloud

5. Final Execution

6 Additional Steps: