Forem: Kunal Shah

Secure AWS VPC using Public and Private Subnets

Kunal Shah — Wed, 11 Dec 2024 08:37:05 +0000

Secure AWS VPC using Public and Private Subnets

AWS Cloud Hands-on Lab Practice Series

Project Overview —

In this lab, you will gain hands-on experience creating and configuring a Virtual Private Cloud (VPC) using the AWS Management Console. The project will culminate in the deployment of a common two-tiered cloud architecture, including front-end and back-end components.

Solutions Architecture —

Prerequisite —

AWS Account with Admin Access.
Knowledge about basic networking concepts (such as IP Addressing, CIDR notation, and routing), an understanding with basic cloud operations.
Familiarity with navigating the AWS Management Console.
AWS LEVEL — BEGINNER — AWS 100

AWS Services Usage —

AWS VPC, EC2, SSM, IGW, NGW, Route Table, SG, NACL and IAM

STEP BY STEP GUIDE -

STEP 1 : Create VPC

Login to AWS Console with an IAM user having Admin privileges
Select us-west-2 region.
In the AWS Management Console search bar, enter VPC, and click the VPC result under Services.
Click Your VPCs in the left navigation pane.
Click Create VPC to begin creating a new VPC.
Specify the following VPC details:
Resources to create: Select VPC only
Name tag: Enter acloudguy-vpc-demo
CIDR block: Enter 10.0.0.0/16 (This is a CIDR block from the private (non-publicly routable) IP address ranges as specified in RFC 1918.)
Tenancy: Select Default (Dedicated tenancy ensures your instances run on single-tenant hardware. For the purposes of this Lab, the default is fine though.)

Notice the VPC and more. option. Choosing this option launches a wizard that makes setting up and configuring a new VPC very simple.
For learning the core concepts we are taking VPC only approach.
Scroll to the bottom of the page and click Create VPC.

Amazon creates the requested VPC and the following linked services:
DHCP options set: Enables DNS for instances that need to communicate over the VPC’s Internet gateway
Main route table: Table that contains a set of rules, called routes, that are used to determine where network traffic is directed
Network ACL: List of rules to determine whether traffic is allowed in or out of any subnet associated with the network ACL

STEP 2 : Create IGW

An Internet Gateway is a horizontally scaled, redundant, and highly available VPC component that allows communication between instances in your VPC and the Internet.
It imposes no availability risks or bandwidth constraints on your network traffic.
From the VPC Dashboard, click Internet Gateways in the left navigation pane.
Click Create internet gateway to begin creating a new gateway with the following -
Name tag — acloudguy-igw
Click Create Internet Gateway

The State of your Internet Gateway will be detached to start.
Now you need to attach the new gateway to the VPC you created earlier.
Click Actions then Attach to VPC
Click Attach internet gateway

*IMP *— An Internet Gateway can only be attached to one VPC.

STEP 3 : Create Public Subnet

In the VPC Dashboard, click Subnets, Click Create subnet.
Configure the following Public subnet details:
VPC ID: Select acloudguy-vpc-demo
Subnet name: Enter Public-A
Availability Zone: Select us-west-2a from the drop-down menu
IPv4 subnet CIDR block: Enter 10.0.20.0/24
Click Create subnet.
In the left navigation pane, click Route Tables, Click Create route table.
Configure the following route table settings:
Name: Enter PublicRouteTable
VPC: Select the acloudguy-vpc-demo VPC from the drop-down menu.

Scroll to the bottom of the page and click Create route table.
On the route details page, switch to the Routes tab and click Edit routes.
Click Add route and Configure the following route settings.
Destination: Enter 0.0.0.0/0.
Target: Select Internet Gateway, then acloudguy-igw.
Click Save changes.

Select the Public-A subnet and click the Route table tab.
Click the Edit route table association button
Select PublicRouteTable from the Route table ID drop-down menu and confirm the following routes
Click Save

This Public subnet will require a route to the internet, so the associated route table has now been configured to use PublicRouteTable to determine traffic rules.

STEP 3 : Create NAT Gateway

In the VPC Dashboard, click NAT Gateways.
Click Create NAT gateway.
Name: Enter NAT-GW
Subnet: Select Public-A
Connectivity type: Ensure Public is selected
The Public connectivity type will allow this NAT Gateway the ability to access the public internet.
Click Allocate Elastic IP next to the Elastic IP allocation ID
Click Create NAT gateway.

STEP 4 : Create Private Subnet

In the VPC Dashboard, click Subnets, **Click **Create subnet.
Configure the following Private subnet details:
VPC ID: Select acloudguy-vpc-demo
Subnet name: Enter Private-A
Availability Zone: Select us-west-2a from the drop-down menu
IPv4 subnet CIDR block: Enter 10.0.10.0/24
Click Create subnet.
In the left navigation pane, click Route Tables, Click Create route table.
Configure the following route table settings:
Name: Enter PrivateRouteTable
VPC: Select the acloudguy-vpc-demo VPC from the drop-down menu.
Click Create route table.
In the PrivateRouteTable details page, in the Routes tab, click Edit routes
Click Add route and configure the following route settings:
Destination: Enter 0.0.0.0/0
Target: Select IGW for temporary testing.
This is to make you understand why Instance is not able to reach Internet even if IGW is attached.
Click Save changes.
Click Subnets from the left navigation pane, then select the Private-A subnet.
In the Route Table tab, and click Edit route table association
Select PrivateRouteTable from the Route table ID drop-down menu.
Click Save.

STEP 5 : Create a Network ACL for a Private Subnet

A **Network Access Control List (NACL) **is an optional layer of security that acts as a firewall for controlling traffic in and out of a subnet.
Click Network ACLs **under **Security.
Click Create Network ACL
Configure the following Network ACL settings:
Name: Enter Private-NACL
VPC: Select **acloudguy-vpc-demo **from the drop-down menu
Click Create network ACL
Select Private-NACL from the Network ACLs list and click the Subnet associations tab
Click Edit subnet associations: Select the check box for the Private-A subnet to associate it with the network ACL.
Click Save changes

STEP 6 : Add rules to a Private Network ACL.

Select Private-NACL from the list of Network ACLs
Click the Inbound rules tab below the table and click Edit inbound rules
Click Add new rule and configure the following:

Rule number: Enter 100
Type: Select SSH
Source: Enter 10.0.20.0/24
Allow / Deny: Select Allow from the drop-down menu
For the second rule, click Add new rule and configure the following:

Rule number: Enter 200
Type: Select Custom TCP Rule
Port Range: Enter 1024–65535
Source: Enter 0.0.0.0/0
Allow / Deny: Select Allow from the drop-down menu
This will allow return traffic for the outbound rules you will add shortly (the range is specified as 1024–65535 because these are the available ports and not reserved). This enables resources inside the subnet to receive responses to their outbound traffic.
Click Save changes.
Ensure the Private-NACL is still selected then click the Inbound rules tab below the table to verify your inbound rules match the following.

With the Private-NACL still selected, switch to the Outbound rules tab and click Edit outbound rules.

Click Add new rule and configure the following:
Rule number: Enter 100
Type: Select HTTP from the drop-down menu
Destination: Enter 0.0.0.0/0
Allow / Deny: Select Allow from the drop-down menu
For the second outbound rule, click Add new rule and configure the following:

Rule number: Enter 200
Type: Select HTTPS from the drop-down menu
Destination: Enter 0.0.0.0/0
Allow / Deny: Select Allow from the drop-down menu
For the third outbound rule, click Add new rule and configure the following:

Rule number: Enter 300
Type: Select Custom TCP from the drop-down menu
Port Range: Enter*32768–61000*
Destination: Enter 10.0.20.0/24 *(The CIDR block of your public subnet)
**Allow / Deny: Select **Allow* from the drop-down menu
Click Save changes.

**IMP : **When you add or remove rules from a network ACL, the changes are automatically applied to the subnets it is associated with. NACLs may take longer to propagate, as opposed to security groups, which take effect almost immediately.

STEP 7 : Launching EC2 Instance on a Private Subnet.

In the AWS Management Console search bar, enter EC2, and click the EC2 result under Services.
Create a Key pair from EC2 left pane.

Click Launch instances.
In the Name and tags section, enter private under Name.

In the Instance Type section, you should not change any options. Simply make sure the default **t2.micro **is selected.
Select the Key Pair created earlier from drop down.
In the Network settings section, click Edit, and configure the following instance details:
VPC: Select the acloudguy-vpc-demo VPC
Subnet: Select the **Private-A **subnet
Auto-assign Public IP: Make sure this is disabled
Firewall: Select Create security group
Security group name: Enter SG-Private
Description: Enter Security group for private subnet instances. Accept SSH inbound requests from Bastion host only.
Type: SSH
Protocol: TCP
Port: 22
Source type: Custom
Source: SG-bastion
Tip: *If you don’t recall the name of your bastion host’s security group, leave the **Source* as Custom, and start typing “bastion”. It will find the security group for you. (Example: SG-bastion)
Click Add security group rule
Type: HTTPS
Protocol: TCP
Port: 443
Source type: Custom
Source: 10.0.20.0/24 (Public VPC CIDR)
Note: If you also needed Windows access, you would add another rule: Type RDP; Protocol TCP; Port 3389; Source SG-bastion

Review the Summary section and click Launch instance

STEP 8 : Test Internet access from EC2 Instance on a Private Subnet.

Connect to EC2 instance & hit sudo yum update

Although the private instance security group is configured correctly, and you should have outbound access to the internet, it still timed out.
The time out is caused by the private NACL denying inbound HTTP traffic.
You will need Network Address Translation (NAT) to allow your private instance *outgoing *connectivity to the Internet.

STEP 9 : Change Routes for Private Subnet from IGW to NGW.

In the PrivateRouteTable details page, in the Routes tab, click Edit routes
Click Add route and configure the following route settings:
Destination: Enter 0.0.0.0/0
Target: Remove IGW & ADD NGW created in STEP 3*.*
This route will eventually send traffic originating from your private subnet and bound for the public internet, to a NAT device.
Click Save changes.
Click Subnets from the left navigation pane, then select the Private-A subnet.
In the Route Table tab, and click Edit route table association
Select PrivateRouteTable from the **Route table ID **drop-down menu.
Click Save.

STEP 10 : Final Test Internet access from EC2 Instance on a Private Subnet.

Important!
There are two important configurations worth mentioning again as to why this command should work in your lab environment:
The private NACL has an Outbound Rule permitting HTTP (port 80) or HTTPS (port 443) access to anywhere on the internet (0.0.0.0/0)
The security group for the NAT device allows HTTP/S access from any instance in the private subnet (that uses the private instance security group, which permits any destination as well)
Connect to EC2 instance and Run sudo yum update -y
SUCCESS !! It worked!

STEP 11 : Decommission:

Go to EC2 -> Terminate the Instance
Go to Nat Gateways -> Delete the Nat Gateway
Go to Elastic IP address -> release IP address
Go to VPC -> Delete VPC

LEARNINGS :

The VPC has been configured with two subnets, a public subnet, and a private subnet. If a subnet’s traffic is routed to an Internet gateway, the subnet is known as a public subnet.
If a subnet doesn’t have a route to the Internet gateway, the subnet is known as a private subnet. Instances launched in a private subnet do not have publicly routable internet addresses either.
Both subnets have a route table associated with them. Instances on the public subnet route internet traffic through the internet gateway. The private subnet routes internet traffic through the NAT device (gateway or instance).
Each instance launched in either subnet has its own security group with inbound and outbound rules, to guarantee access is locked down to specific ports and protocols.
For example, private instances on the private subnet allow any outbound traffic but only allow SSH access from the bastion host.
As another example, although the NAT device is in the public subnet, it cannot be reached from the internet. It has an inbound rule that only grants instances from the private security group (private instances) access.
Note that you might allow SSH access from your personal IP address or specific administrator’s as well, or perhaps grant ICMP (ping) access during setup and troubleshooting efforts.
In addition to security groups, the private subnet also has a network access control list (NACL) as an added measure of security.
NACL’s allow for inbound and outbound rules, specified in priority order. They are set up as implicit allow rules.
If none of them are matched, all other traffic is denied.
This was proven to work in the Lab by performing operating system updates once the NAT device was in place. The private route table sends the traffic from the instances in the private subnet to the NAT device in the public subnet.
The NAT device sends the traffic to the Internet gateway for the VPC. The traffic is attributed to the Elastic IP address of the NAT device.

I am **Kunal Shah, AWS Community Builder, AWS Certified Professional Solutions Architect, helping clients to achieve optimal solutions on the Cloud. Cloud Enabler by choice, DevOps Practitioner having 9+ Years of overall experience in the IT industry.

I love to talk about Cloud Technology, DevOps, Digital Transformation, Analytics, Infrastructure, Cloud Native, Generative AI, Dev Tools, Operational efficiency, Serverless, Cost Optimization, Cloud Networking & Security.

aws #community #builders #VPC #network #security #troubleshoot #hybrid #network #peering #segmentation #validatations #isolated #solution #war #reliability #operations #Excellence #infrastructure #deployment #private #secure #design #acloudguy

*You can reach out to me @ *acloudguy.in

AWS Network Access Analyzer Overview

Kunal Shah — Tue, 27 Aug 2024 11:26:00 +0000

AWS Cloud Hands-on Lab Practice Series

Project Overview —

The AWS Network Access Analyzer project aims to provide a comprehensive understanding and practical demonstration of the Network Access Analyzer (NAA) feature within Amazon VPC. This lab will equip participants with the knowledge and skills to effectively assess, verify, and improve their VPC network security posture.

Solutions Architecture —

First Let’s understand the real world use case :

Network Access Analyzer uses automated reasoning algorithms to analyze the network paths that a packet can take between resources in an AWS network.

The key concepts of the Network Access Analyzer are:

Network Access Scope
Findings

A Network Access Scope determines the types of findings that the analysis produces. You add entries to MatchPaths to specify the types of network paths to identify. You add entries to ExcludePaths to specify the types of network paths to exclude.

Findings are potential paths in your network that match any of the MatchPaths entries in your Network Access Scope, but do not match any of the ExcludePaths entries in your Network Access Scope.

Security Audits and Compliance Verification:

Use Case: A financial services company needs to ensure that their network configurations comply with industry regulations such as PCI-DSS or HIPAA.
Solution: The Network Access Analyzer can be used to scan the VPC configurations and identify any non-compliant access patterns, helping the company verify that their network setup adheres to the required security standards.

2. Proactive Threat Detection:

Use Case: A healthcare provider wants to minimize the risk of unauthorized access to sensitive patient data stored in their AWS environment.
Solution: The analyzer helps identify potential security gaps or misconfigurations, such as overly permissive security groups, that could be exploited by malicious actors, allowing the provider to tighten security controls before an incident occurs.

3. Network Segmentation Validation:

Use Case: An e-commerce platform separates its production, development, and testing environments within the same VPC to maintain strict isolation between them.
Solution: The Network Access Analyzer can verify that the segmentation is correctly configured, ensuring that there is no unintended communication between environments that could lead to data leakage or cross-environment attacks.

4. Incident Response and Forensics:

Use Case: A tech company experiences a potential security breach and needs to quickly assess if the VPC’s network configuration contributed to the incident.
Solution: The Network Access Analyzer can be used to analyze the network access paths that existed at the time of the incident, helping the incident response team identify misconfigurations or unauthorized access that may have facilitated the breach.

Prerequisite —

AWS Account with Admin Access.
Knowledge about basic networking concepts (such as IP Addressing, CIDR notation, and routing), an understanding with basic cloud operations.
Familiarity with navigating the AWS Management Console.

AWS Services Usage —

AWS VPC, EC2, SSM, S3, Endpoints, Network Analyzer, CloudFormation and IAM

STEP BY STEP GUIDE -

STEP 1 : Clone the GitHub Repo

Navigate to following GitHub Repository **VPC-Network-Access-Analyzer**
Clone the repo to download the CloudFormation Template used for this project lab.
CloudFormation template name — custom-vpc-cfn-template.yaml

STEP 2 : Creating AWS resources through CloudFormation service.

Login to AWS account, Navigate to AWS CloudFormation Service.
Head over & change the region of the aws console where you want to deploy the Primary region resources.
NOTE : CODE IS TESTED IN Middle East (Bahrain) me-south-1 region.
If you want to use other region then you will have to modify the CloudFormation template accordingly.
Click on Create Stack & upload the template downloaded in the step 1.

This stack will create Three VPC and 3 EC2 Instances as depicted on the architecture diagram with public and private subnet, an internet gateway, Nat Gateway, Security Groups required for this lab exercise.
You can verify the deployments by exploring VPC, EC2 sections on AWS management Console.

STEP 3 : Explore and Analyze

TASK 1 : Use a Network Access Scope template to analyze ingress traffic

In this scenario, we will use a pre-built template to analyze available traffic paths from an internet gateway.
In the left navigation pane, scroll down to the bottom of the page, and choose Network Manager.

Now In the left navigation pane, choose Network Access Analyzer.
Choose Get started .

We are going to use templates to create new Network Access Scopes.
Choose Create Network Access Scope .

Choose the template named Identify access from Internet Gateways.
Choose Next , then configure:

Name: naa-ingress-routes
Leave the default configuration in the rest of the fields.
Choose Next .
Review the template and conditions within the Network Access Scope.
Choose Create Network Access Scope .

Select the naa-ingress-routes and click on Analyze.
The analysis takes a few minutes or less to complete.
After completion, the result is Findings detected.

This page also presents a path display. The display presents the network elements within the ingress path from Source to Destination.
The ingress path analysis starts from the internet gateway of VPC3 all the way up to the network interface of VPC3-public-ec2.

Review the network architect diagram provided at the top Both VPC2 and VPC3 have an internet gateway. But Findings are only of VPC3.
The reason is — the Network Access Scope definition for this analysis has a source of internet gateway and a destination of network interface, in this case VPC2-private-ec2 is in a private subnet, and internet gateways do not have a direct path to network interfaces in a private subnet.
This analysis helps verify ingress traffic paths, and even demonstrate compliance in certain use cases.

TASK 2 : Verify the use of a NAT gateway for internet traffic

A very common use case for having a NAT gateway in a VPC is to enable internet access for a private subnet.
There are some use cases where you have a private subnet that does not require access to the internet.
In our architecture Both VPC1 and VPC2 have private subnets.
However, Only VPC2 contains NAT gateway which means only Private subnet in VPC2 requires access to Internet, while private subnet in VPC1 does not require internet access.
Lets use Network Access Analyzer to validate this.
Select Network Access Analyzer.
Choose Create Network Access Scope.
Select Empty template.

Choose Next , then configure:
Name: NAT-Gateway-usecase

Choose Add match condition.
No conditions configured under Source section.
Under Destination section:
Resource selection: choose ‘Resource types’
Resource types: choose ‘Internet Gateways’
Choose Next and Review the Network Access Scope definition.
Choose Create Network Access Scope .

Select the NAT-Gateway-usecase and click on Analyze.
The analysis takes a few minutes or less to complete.
After completion, the result is Findings detected.

As expected, there are two Findings:

A private EC2 Instance in VPC2 accessing the internet gateway.
A public EC2 Instance in VPC3 accessing the internet gateway.
The findings do not include the private EC2 instance in VPC1.
Toggle between the two findings and review the display path for each finding.
You see a NAT gateway within the path for VPC2 instance.
This validates the architecture diagram, showing internet access is enabled for the private subnet in VPC2 through the NAT gateway; while the private subnet in VPC1 does not have access to the internet.

TASK 3 : Duplicate and modify a Network Access Scope

Network Access Scopes can be duplicated, modified, then used to run a new analysis.
This can help save time from creating a new Network Access Scope.
In this task, duplicate the NAT-Gateway-usecase Network Access Scope created in the previous task, and use the new scope to check any internet access path that doesn’t include a NAT gateway.
Select NAT-Gateway-usecase.
From the Actions button, choose ‘Duplicate and modify’.

In the Duplicate Network Access Scope page, configure:
Name: NO-NAT-Gateway-usecase.
In the Exclusion conditions section, choose Add exclusion condition.
Under Through section, choose Resource types.
Resource Types: NAT Gateways
Choose Duplicate and analyze Network Access Scope.

After a few moments the ‘Analysis status’ for the newly created scope transitions to Complete.
There is one Finding -
EC2 Instance (VPC3-public-ec2) in VPC3 accessing the internet gateway.
This is because instance is in a public subnet.
This confirms what we observed in the architecture diagram.
VPC3 can access the internet directly, without the need of a NAT gateway.

TASK 4 : Demonstrate a network configuration meets compliance requirements

In this final task, We configure a network change and demonstrate a compliant configuration.
Here the private instance in VPC2 is required to access a specific IP address and port number.
Allow egress traffic to destination IP address — 172.217.164.110 (this is an google.com IP address).
Destination port number — 443.
Lets review the current internet bound configuration path for VPC2-private-ec2.
Choose Network Access Analyzer.
Choose Create Network Access Scope.
Choose Empty template.
Choose Next , then configure:
Name: VPC2-private-outbound-path
Choose Add match condition.
Under Source section:
Resource selection: choose ‘Resource IDs’
Resource types: choose EC2 Instances
Resource IDs: choose VPC2-private-ec2
Under Destination section:
Resource selection: choose ‘Resource types’
Resource types: choose Internet Gateways

Choose Next .
Review the Network Access Scope definition.
Choose Create Network Access Scope.
select VPC2-private-outbound-path.
Hit Analyze .

When the analysis is complete, the result is: Findings detected.
Review the path analysis and notice the second element in the path is the Security Group
The Destination address is 0.0.0.0/0 ; and the outbound port is 80
Now lets Update the security group to match the requirement.
Select the security group label.
Choose the security group Resource ID.
Click the Outbound rules tab.
Choose Edit outbound rules .
In the HTTP **drop down list, choose **HTTPS.
Replace the 0.0.0.0/0 destination address with 172.217.164.110/32
Choose Save rules .

Now select VPC2-private-outbound-path.
Hit Analyze.
Review the path analysis and notice the second element in the path is the Security Group
The Destination address is 172.217.164.110/32 ; and the outbound port is 443

This analysis validates the current configuration is compliant with the requirement.
Understood, verified, and improved a network security posture.
Demonstrated a network configuration meets compliance requirements.

STEP 4 : Decommission:

Go to CloudFormation Stack.
Delete the CloudFormation Stack created for this lab.

Congrats ! We have successfully completed lab to Improve and Analyze AWS VPC Security Posture with AWS Network Access Analyzer.

I am Kunal Shah, AWS Community Builder, AWS Certified Professional Solutions Architect, helping clients to achieve optimal solutions on the Cloud. Cloud Enabler by choice, DevOps Practitioner having 9+ Years of overall experience in the IT industry.

aws #community #builders #VPC #network #access #analyzer #cloudformation #compliance #troubleshoot #network #security #hybrid #network #peering #segmentation #validatations #isolated #solution #war #reliability #operations #Excellence #infrastructure #deployment #private #secure #design #acloudguy

You can reach out to me @ acloudguy.in

Deploy Rancher on Azure AKS using Azure Cli & Helm Charts

Kunal Shah — Wed, 01 May 2024 08:20:10 +0000

Azure Cloud Hands on Lab Practice Series

Project Overview —

This project revolves around Azure AKS where we deploy Rancher (platform for Kubernetes management). Rancher is a Kubernetes management tool to deploy and run clusters anywhere and on any provider. Azure AKS & Rancher is very vital combination when it comes to managing multi-cluster Kubernetes workloads from Single Dashboard.

SOLUTIONS ARCHITECTURE OVERVIEW -

First Let’s understand the real world use case :

Hybrid Cloud Management for a Global E-Commerce Platform: Imagine a large e-commerce platform that serves customers globally. The platform’s infrastructure spans multiple regions to ensure low-latency access and high availability. The architecture includes microservices running on Kubernetes clusters managed by AWS EKS. Additionally, the company has on-premises data centers hosting legacy applications. Rancher provides a unified dashboard for managing EKS clusters spread across various regions.
Finance: Multi-Tiered Application Deployment: A financial institution is migrating its legacy monolithic applications to microservices architecture on AWS EKS. The institution operates in multiple regions and requires secure and efficient deployment of multi-tiered applications. Rancher simplifies the deployment of microservices across EKS clusters in different regions.
Retail: Seasonal Application Scaling: A retail chain experiences significant fluctuations in website traffic during holiday seasons. They need a solution to dynamically scale their applications on EKS to handle increased demand. Rancher enables automated scaling of applications based on predefined policies. Provides visibility into application performance, helping to optimize resource allocation during peak times.
Manufacturing: Edge Computing for IoT Devices: A manufacturing company utilizes IoT devices across its facilities to monitor and optimize production processes. They need a solution to manage Kubernetes clusters at the edge for real-time data processing. Rancher supports the deployment of Kubernetes clusters at the edge, close to IoT devices. It also enables centralized management of edge clusters for easier monitoring and updates.
Media and Entertainment: Content Delivery Optimization: A media streaming service operates globally and needs to optimize content delivery for users. They want to deploy and manage Kubernetes clusters efficiently to ensure low-latency streaming. It Integrates with AWS services like Amazon CloudFront for efficient content caching. Allows for easy scaling and updating of streaming applications across clusters. Facilitates the deployment of edge-native applications to process data locally.

These diverse use cases demonstrate the versatility of Rancher on AZURE AKS in addressing industry-specific challenges and enhancing the management of Kubernetes clusters in various contexts. Rancher simplifies operations, enhances security, and provides a unified platform for managing the hybrid cloud environment.

Prerequisite —

AZURE ACCOUNT with valid Subscription ID.
Resource Group for creating Azure resources.
AZURE CLI (on local machine)
HELM
KUBECTL

Azure Services Usage —

Azure Resources Group
Azure AKS
Azure Vnet
Azure Loadbalancer
Azure Managed Identity
Virtual Machine Scale Set

STEP BY STEP GUIDE -

STEP 1 : Setting environment variables

Start the new terminal on local machine (linux/ubuntu/wsl)
Set following env variables :
AZURE_LOCATION=Azure Region (example uaenorth)
CERT_MANAGER_VERSION=v1.12.5 (Cert Manager Version)
KUBERNETES_VERSION=v1.26.6 (K8s Version)
NODE_COUNT=2 (Number of Nodes required)
RANCHER_VERSION=2.7.6 (Rancher Manager Version)
RESOURCE_PREFIX=aks-demo (Name of Resource Group in Azure)
SUBSCRIPTION_ID=XXXXXXX-XXXXXX (Subscription ID of Azure)
VM_SIZE=Standard_D2s_v3 (Azure VM Instance Type)
EMAIL_ADDRESS=youremail@domain.com (Azure account email)

STEP 2 : Add Helm Repositories

helm repo add devpro https://devpro.github.io/helm-charts
This repository contains Helm charts to build clusters with all components running in containers.
helm repo add jetstack https://charts.jetstack.io
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
helm repo update
helm repo list

STEP 2 : Login to Azure through Azure CLI for Authentication

Run az login on local terminal (It will open a window in your browser automatically.)
If not then you will get a link in CLI output, copy that generated link & paste in your browser to authenticate.
Set Subscription if you have multiple subscription. (optional if you have only one subscription)
az account set —subscription $SUBSCRIPTION_ID
Create the resource group. (optional if you have already created)
az group create —name $RESOURCE_PREFIX —location ${AZURE_LOCATION}

STEP 3 : Creating Kubernetes Cluster (AKS) in NEW VNET

CASE 1 : AKS in new VNET
az aks create —resource-group $RESOURCE_PREFIX —name $RESOURCE_PREFIX —kubernetes-version $KUBERNETES_VERSION -node-count $NODE_COUNT —node-vm-size $VM_SIZE
CASE 2 : AKS in existing VNET
az aks create —resource-group $RESOURCE_PREFIX —name $RESOURCE_PREFIX —kubernetes-version $KUBERNETES_VERSION -node-count $NODE_COUNT —node-vm-size $VM_SIZE —network-plugin azure -vnet-subnet-id/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_PREFIX/providers/Microsoft.Network/virtualNetworks/AZURE_VNET_NAME/subnets/AZURE_SUBNET_NAME
We are demonstrating CASE 1 — NEW VNET
After running the command from CASE 1 — This will take some time to create AKS & store ssh keys at your /home/username/.ssh
These keys are used to ssh into the worker nodes (VMs)
Once the AKS is ready it will give you cluster information in the CLI output.

STEP 4 : Access the AKS Cluster through kubectl

let’s add cluster credentials to local config of kubectl.
az aks get-credentials — resource-group $RESOURCE_PREFIX -name $RESOURCE_PREFIX
Check the nodes & all resources using kubectl.
kubectl get nodes
kubectl get all -A
TIP — If you face any permission issue then change permission for ~/.kube/config
Vanilla Cluster on AKS is ready with requested worker nodes.

STEP 5 : Ngnix Ingress Installation (exposing to Internet)

Run below mentioned commands from local machine.
helm upgrade —install ingress-nginx ingress-nginx/ingress-nginx —namespace ingress-nginx —create-namespace —set controller.service.annotations.”service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path”=/healthz
Check the services: kubectl get services -n ingress-nginx

Let’s store value of Public IP which needs to be used at the time of Rancher Installation.
PUBLIC_IP_NGINX= kubectl get service -n ingress-nginx ingress-nginx-controller —output jsonpath=’{.status.loadBalancer.ingress[0].ip}’
echo $PUBLIC_IP_NGINX (It should give Public IP)

STEP 6 : Install Certificates manager.

Run below mentioned commands from local machine.
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/$CERT_MANAGER_VERSION/cert-manager.crds.yaml
helm upgrade —install cert-manager jetstack/cert-manager —namespace cert-manager —create-namespace —version $CERT_MANAGER_VERSION
kubectl get pods — namespace cert-manager ( 3 pods should be in Running status)

helm upgrade —install letsencrypt devpro/letsencrypt —set registration.emailAddress=$EMAIL_ADDRESS —namespace cert-manager
kubectl get clusterissuer -n cert-manager ( 2 Cluster issuers should be True)

STEP 7 : Rancher Installation.

Run below mentioned commands from local machine.
kubectl create namespace cattle-system
helm upgrade —install rancher rancher-latest/rancher —namespace cattle-system —set hostname=rancher.$PUBLIC_IP_NGINX.sslip.io -set ‘ingress.extraAnnotations.cert-manager\.io/cluster-issuer=letsencrypt-prod’ —set ingress.ingressClassName=nginx —set ingress.tls.source=secret —set ingress.tls.secretName=tls-rancher —set replicas=2 —version $RANCHER_VERSION
Check the status of installation & wait for it to complete.
kubectl -n cattle-system rollout status deploy/rancher

Fetch the generated password & copy it
kubectl get secret —namespace cattle-system bootstrap-secret -o go-template=’ .data.bootstrapPassword|base64decode“\n”’

STEP 8 : Accessing the Rancher UI :

echo https://rancher.$PUBLIC_IP_NGINX.sslip.io/
Copy URL & Paste on browser for Initial setup of Rancher.

Check the logs → kubectl logs -n ingress-nginx -l app.kubernetes.io/component=controller
You will be asked for password ( copied earlier )
Hit Log in with Local User.
Define the admin password, check the box and click on “Continue”. (store it)
Finally Rancher is running and you can explore “local” cluster.

STEP 10 : Decommission

Run below command to destroy all resources.
kubectl delete ns cert-manager
kubectl delete ns ingress-nginx
Decommission the resources under the Azure Resources group.
Delete the Resources Group at the end.

Congrats ! We have successfully completed lab for Deploying Rancher on Azure AKS using Azure CLI & Helm Charts.

I am Kunal Shah, AWS Certified Solutions Architect, helping clients to achieve optimal solutions on the Cloud. Cloud Enabler by choice, DevOps Practitioner having 9+ Years of overall experience in the IT industry.

I love to talk about Cloud Technology, DevOps, Digital Transformation, Analytics, Infrastructure, Dev Tools, Operational efficiency, Serverless, Cost Optimization, Cloud Networking & Security.

aws #community #builders #devops #azure #aks #managed #kubernetes #solution #rancher #solution #management #centralize #dashboard #easy #management #scalability #operational #efficiency #robust #infrastructure #highly #available #reliable #controlled #design #acloudguy

You can reach out to me @ acloudguy.in

Failover Mechanism in Amazon Route 53 Private Hosted Zones

Kunal Shah — Wed, 20 Mar 2024 11:11:50 +0000

Failover Mechanism in Amazon Route 53 Private Hosted Zones

AWS Cloud Hands-on Lab Practice Series

Project Overview —

The AWS Route 53 Failover project aims to architect a secure, cost-efficient, Fault tolerant and Highly available cloud environment. The project’s primary goal is to provide how to configure DNS entries on Amazon Route 53 to do dynamic routing between resources in different AWS regions while adhering to AWS best practices and compliance requirements.

SOLUTIONS ARCHITECTURE OVERVIEW -

First Let’s understand the real world use case :

1.Disaster Recovery: In a disaster recovery setup, where you have primary and secondary data centers or regions, failover policies can automatically redirect traffic from the primary to the secondary site in case of a failure.

Example: let’s say you have a primary data center in US-East (Virginia) and a secondary data center in US-West (Oregon). If the primary data center experiences an outage, Route 53 failover policies can redirect traffic to the secondary data center.

2.Multi-Region Redundancy: For global applications that require high availability, you may deploy identical application stacks in multiple AWS regions. Failover policies can route traffic to the closest healthy region or distribute traffic evenly across regions based on health checks.

Example: If your application is deployed in US-East (Virginia) and EU-West (Ireland), Route 53 can route traffic to the region with the lowest latency or the region that passes health checks.

3.Blue/Green Deployments: During software updates or deployments, you can use failover policies to perform blue/green deployments.

Example : For instance, suppose you have a production environment (blue) and a staging environment (green). You can update the staging environment, run health checks, and then switch traffic from the production environment to the staging environment using failover policies once the update is successful.

4.Highly Available Database Replication: Failover policies can be used to manage database failover scenarios where you have active-passive or active-active database replication setups.

Example: For instance, if you have a primary database in one availability zone and a standby database in another availability zone, Route 53 can automatically redirect traffic to the standby database if the primary database becomes unavailable.

5.Content Delivery Networks (CDNs): In CDN setups, failover policies can be employed to route traffic to alternate CDN endpoints in case of CDN node failures or performance degradation.

Example: If you have CDN endpoints in different regions or with different providers, Route 53 can direct traffic to the next best endpoint if the primary endpoint experiences issues.

Also when considering the use of failover policies in Amazon Route 53 Private Hosted Zones within the context of the Well-Architected Framework (AWS WAR), it primarily aligns with the “Reliability” and “Operational Excellence” pillars.

Prerequisite —

AWS Account with Admin Access.
AWS CLI user with Access key & Secret Key.

AWS Services Usage —

AWS Route53, VPC, EC2, SSM, S3, CloudFormation and IAM

STEP BY STEP GUIDE -

STEP 1 : Clone the GitHub Repo

Navigate to following GitHub Repository AWS-Route53-Failover
Clone the repo to download the CloudFormation Template & code used for this lab.
CloudFormation template name — route53-vpc-cfn-template.yaml

STEP 2 : Creating AWS resources through CloudFormation service.

Login to AWS account, Navigate to AWS CloudFormation Service.
Head over & change the region of the aws console where you want to deploy the Primary region resources. (SITE-A)
Click on Create Stack & upload the template downloaded in the step 1.
VPC CIDR (SITE-A) : 10.0.0.0/16
PublicSubnet : 10.0.1.0/24

This stack will create one VPC and create a public subnet with two EC2 instances, an internet gateway, with a default route on the public subnet in regions you deployed.

NOW Head over & change the region of the aws console where you want to deploy the Secondary region resources. (SITE-B)
Click on Create Stack & upload the template downloaded in the step 1.
VPC CIDR (SITE-B) : 10.1.0.0/16
PublicSubnet : 10.1.1.0/24

This stack will create one VPC and create a public subnet with two EC2 instances, an internet gateway, with a default route on the public subnet in regions you deployed.

STEP 3 : Creating a VPC Peering to connect both regions

Option 1: Use AWS CloudShell to run the commands.
Option 2: From your local machine where you have aws cli working with appropriate access key & secret key.
Make sure to edit the regions before running the commands
Run the commands given in VPC env variable script.txt file.

Get the VPC ID of the respective VPC’s created.
Now Run the commands given VPC Peering Script.txt file.
This will create VPC peering between two VPCs from different AWS Regions (SITE A & SITE B).

STEP 4 : Creating Private DNS entries:

Create a Route 53 — Private Hosted Zone for the acloudguy.internal DNS entries by associating the Site A.
You can change the hosted zone name according to your wish.
Run the commands from private DNS entries.txt file.

STEP 5 : Creating Route 53 Health Check:

Get the public IP of both Web Servers. This is because Route 53 health checkers are public and they can only monitor hosts with IP addresses that are publicly routable on the internet.
Run the command from Public-IP-For-Route53-HealthChecks.txt file.

You need to create a health check policy file: health check policy.txt

Now, Let’s create the health check for our primary endpoint that is in SITE A region.
Run the command from Primary-HealthCheck-Record.txt file.
The health check will be active in 30–60 seconds.
Go to Route53 Console & check the health check section.

STEP 6 : Creating Route 53 Failover Policy:

Get the private IP of both Web Servers.
Run the command from Private-IP-For-Route53-HealthChecks.txt file.

Run the Failover routing policy from Failover policy.txt file.
TIP : It’s being used Private IP to keep the communication through VPC Peering.

Now Associate the traffic policy to Route53 Private Hosted Zone.
Run the command from Associate Failover Policy.txt file.

We just created a policy similar to the image below: (your public IP & Private IP will be different)

STEP 7 : Test the failover policy:

Connect to EC2 Instance through EC2 Connect from SITE A.
Try to access the website using “service.acloudguy.internal”
dig +short service.acloudguy.internal
curl service.acloudguy.internal
You will below responses from it.

Now to test the failover, remove Security Group Inbound rules of Primary Site A EC2 Instance from AWS Console.
Once there is no Inbound rule to EC2 Instance.
After 2 minutes, the Health Check will mark the PRIMARY EC2 Instance as UNHEALTHY and will trigger the failover mechanism.

Health Check will fail & route the traffic to Secondary Site B EC2 Instance.
We can run the commands again
dig +short service.acloudguy.internal
curl service.acloudguy.internal

STEP 8 : Decommission:

Delete all the resources created during the lab.
aws ec2 delete-vpc-peering-connection — vpc-peering-connection-id $PEERING_ID
Delete the CloudFormation Stacks from SITE A & SITE B Regions.
Delete Route53 Health Checks from AWS Console.
Delete Route53 Traffic Policy and Private Hosted Zone from Console
Delete files health-check-config.json & failover-policy.json

Congrats ! We have successfully completed lab for How to Implement Failover Policies in Amazon Route 53 Private Hosted Zones.

I am Kunal Shah, AWS Certified Solutions Architect, helping clients to achieve optimal solutions on the Cloud. Cloud Enabler by choice, DevOps Practitioner having 8+ Years of overall experience in the IT industry.

I love to talk about Cloud Technology, DevOps, Digital Transformation, Analytics, Infrastructure, Dev Tools, Operational efficiency, Serverless, Cost Optimization, Cloud Networking & Security.

aws #community #builders #VPC #route53 #failover #mechanism #cloudformation #disaster #recovery #network #security #hybrid #network #peering #isolated #solution #war #reliability #operations #Excellence #infrastructure #scalable #highly #available #blue #green #deployment #private #secure #design #acloudguy

You can reach out to me @ acloudguy.in

Shielding Your Data: Safeguarding AWS S3 via VPC Endpoints.

Kunal Shah — Mon, 19 Feb 2024 10:24:31 +0000

Shielding Your Data: Safeguarding AWS S3 via VPC Endpoints.

Security & Cost Optimization at the same time.

Project Overview —

The AWS VPC Endpoint project aims to architect a secure, cost-efficient, and scalable cloud environment. The project’s primary goal is to provide a standardized framework for setting up AWS VPC endpoint for accessing AWS services privately while adhering to AWS best practices and compliance requirements.

SOLUTIONS ARCHITECTURE OVERVIEW -

First Let’s understand the real world use case -

1.Securing Access to AWS Services: VPC endpoints will let you get admission to AWS services inclusive of S3, DynamoDB, or Amazon RDS from within your VPC with out exposing them to the general public net. This enables enhance security by means of reducing the attack floor and gets rid of the need to configure public-dealing with security controls.

Example: A enterprise’s application jogging in an AWS VPC desires to get right of entry to records saved in Amazon S3 buckets securely. By the use of VPC endpoints for S3, the employer guarantees that data switch among the application and S3 stays within the AWS community, decreasing exposure to external threats.

2.Cost Optimization: VPC endpoints can help optimize expenses through decreasing information transfer prices incurred when getting access to AWS services over the public internet. Since information transfer between your VPC and the endpoint service stays in the AWS network, you can avoid information transfer costs associated with net traffic.

Example: An employer regularly transfers large volumes of facts between its AWS VPC and Amazon S3 buckets for backup and storage functions. By the usage of VPC endpoints for S3, the organization can extensively reduce facts switch costs compared to having access to S3 over the internet.

3.Compliance Requirements: VPC endpoints can assist meet regulatory and compliance necessities with the aid of ensuring that statistics transfer between your VPC and AWS offerings stays private and stable. This is in particular crucial for industries with strict information privateness and compliance requirements.

Example: A healthcare business enterprise desires to ensure that patient data saved in Amazon DynamoDB stays blanketed and compliant with HIPAA regulations. By using VPC endpoints for DynamoDB, the agency ensures that data get entry to is confined to authorized sources inside the VPC, supporting meet compliance necessities.

4.Improved Network Performance: VPC endpoints can enhance community performance by using lowering latency and enhancing throughput for accessing AWS offerings. Since facts transfer takes place within the AWS community, it may bring about quicker and greater reliable conversation between your VPC and endpoint services.

Example: A gaming organization's multiplayer online game calls for actual-time get entry to to Amazon DynamoDB for storing player profiles and sport nation. By the usage of VPC endpoints for DynamoDB, the business enterprise guarantees low-latency get right of entry to to the database, providing a continuing gaming revel in for players.

Overall, AWS VPC endpoints provide a steady, cost-powerful, and green way to get entry to AWS services from inside your VPC, making them an vital thing of many AWS architectures. By leveraging VPC endpoints, organizations can increase protection, reduce charges, and enhance performance for your AWS cloud workloads.

Prerequisite —

AWS Account with Admin Access.

AWS Services Usage —

AWS VPC, Endpoints, EC2, SSM, S3, CloudFormation and IAM

STEP BY STEP GUIDE -

STEP 1 : Clone the GitHub Repo

Navigate to following GitHub Repository **s3-vpc-endpoint-lab**
Clone the repo to download the CloudFormation Template for this lab.
CloudFormation template name — endpoint-lab-cft.yml

STEP 2 : Creating AWS resources through CloudFormation service.

Login to AWS account, Navigate to AWS CloudFormation Service.
Head over & change the region of the aws console where you want to deploy the resources. (default is ap-south-1)
If you want to deploy in any other region you will have modify prefix list in CloudFormation template.
Click on Create Stack & upload the template downloaded in the step 1.
Keep rest of the settings as default & hit create.
This stack will create a VPC, EC2, VPC endpoints, Instance profile, Security Group, subnets, route tables.

STEP 3 : Verify the CloudFormation deployment.

Check all the resources created/deployed through CloudFormation.
Verify Security group of EC2 to check there is no inbound rules.
Verify & validate Security group of endpoints where only VPC CIDR is allowed in inbound rules.
all endpoints — ssm, ec2, s3 are deployed.

STEP 4 : Creating two AWS s3 buckets.

Navigate to AWS S3 on aws console.
Create a bucket in the same region where CloudFormation is deployed.
This is to have a bucket in same region as of s3 gateway endpoint.
Create another bucket in different region for testing use case.

STEP 5 : Connect to AWS EC2 Instance through SSM

As we have deployed AWS ssm endpoint, we should be able to connect the private ec2 instance through ssm connect.
This is entirely secure & traffic remains in isolated VPC.
Once connected to EC2, list S3 buckets through usual aws cli command

aws s3 ls
You should get no response as EC2 is private & no traffic is intended to Internet. Hence it is not able to query s3.ap-south-1.amazonaws.com

STEP 6 : Accessing AWS S3 via AWS VPC Gateway endpoint

Now we will use AWS VPC Gateway endpoint to access our regional AWS s3 buckets.
From EC2 console hit the command —
aws s3 ls — region (same region)

aws s3 ls --region ap-south-1

List contents from a bucket via s3 gateway endpoint. (same region)

aws s3 ls s3://mybucket --region ap-south-1

List contents from a bucket via s3 gateway endpoint. (different region)
This will give no response as there is no regional Gateway for it.

aws s3 ls s3://mybucket --region us-east-1

STEP 7 : Accessing AWS S3 via PrivateLink

Copy the S3InterfaceEndpointDnsEntry from CloudFormation Output or find the entry from the VPC endpoint console where the s3 PrivateLink is created.
List all buckets via S3 PrivateLink using endpoint dns

aws s3 ls --endpoint-url https://bucket.vpce-111111111111-aaaaaaaa.s3.ap-south-1.vpce.amazonaws.com

List contents from a S3 bucket created in Step 4.

aws s3 ls s3://newbucket --endpoint-url https://bucket.vpce-111111111111-aaaaaaaa.s3.ap-south-1.vpce.amazonaws.com

You can run the same command from another VPC or On-Premises that has connectivity with current VPC and be able to access the bucket via PrivateLink.

STEP 8 : Decommission

Delete the CloudFormation Stack to delete all the deployed resources.
Delete the s3 buckets created in step 4.

STEP 9 : More to read -

PrivateLink : https://docs.aws.amazon.com/AmazonS3/latest/userguide/privatelink-interface-endpoints.html
Gateway Endpoints: https://docs.aws.amazon.com/vpc/latest/privatelink/gateway-endpoints.html

Congrats ! We have successfully completed lab for Shielding Your Data: Safeguarding AWS S3 via VPC Endpoints.

I love to talk about Cloud Technology, DevOps, Digital Transformation, Analytics, Infrastructure, Dev Tools, Operational efficiency, Serverless, Cost Optimization, Cloud Networking & Security.

aws #community #builders #VPC #endpoints #privatelink #connectivity #cloudformation #cost #optimization #network #security #hybrid #network #prefixlist #isolated #solution #centralize #secure #access #performance #edge #locations #operations #infrastructure #scalable #reliable #highly #available #private #secure #design #acloudguy

You can reach out to me @ acloudguy.in

Multi-Cloud Harmony: Connecting AWS and Azure with Site-to-Site VPN

Kunal Shah — Tue, 19 Dec 2023 10:22:16 +0000

Multi-Cloud Harmony: Connecting AWS and Azure with Site-to-Site VPN

For Seamless Secure Private Cross-Cloud Integration

AWS Cloud Hands-on Lab Practice Series

Project Overview —

In the evolving landscape of cloud computing, the adoption of multicloud architectures has become a common strategy for businesses aiming to address diverse operational needs. This approach involves distributing data and resources across multiple cloud providers, with AWS, Azure and GCP standing out as the most prevalent choices. As a solution architect, mastering the management of the entire multicloud perimeter is crucial. One fundamental aspect is establishing connections between cloud environments to enable seamless communication across resources.

This blog focuses on bridging AWS and Azure through the implementation of a Virtual Private Network (VPN). By doing so, you can create a unified cloud environment, promoting collaboration and resource sharing. To enhance resilience, security and ensure private connectivity, the setup will involve the creation of a VPN tunnel.

SOLUTIONS ARCHITECTURE OVERVIEW -

First Let’s understand the real world use case :

1. Hybrid Cloud Deployments:

Use Case: Organizations may have existing infrastructure on both AWS and Azure due to historical reasons, mergers, or specific service preferences.
Benefits: Linking the two environments allows for a cohesive hybrid cloud strategy, enabling seamless data and workload movement between the on-premises, Azure, and AWS environments.

2. Multi-Cloud Disaster Recovery:

Use Case: Ensuring business continuity is crucial, and some enterprises choose to leverage both Azure and AWS for disaster recovery purposes.
Benefits: Establishing VPN connections enables the replication of critical data and applications across clouds, providing redundancy and quick recovery in case of a failure in one cloud provider.

3. Vendor Diversification and Risk Mitigation:

Use Case: Companies seeking to mitigate vendor lock-in risks or wanting to take advantage of specific services from each provider.
Benefits: By linking AWS and Azure, businesses can maintain flexibility in choosing the best services from each provider while avoiding dependency on a single vendor. It also provides a fallback option in case of service outages or disruptions.

4. Compliance and Data Residency:

Use Case: Adherence to specific data residency or compliance requirements may necessitate the use of different cloud providers for different regions.
Benefits: Establishing VPN connections ensures that data can be securely transferred and accessed across compliant regions, helping businesses meet regulatory requirements without compromising on operational efficiency.

Prerequisite —

AWS Account with IAM Admin privileges.
Azure Account with Admin privileges.

AWS & AZURE Services Usage —

AWS VPC, IAM, EC2, SG, CLOUDFORMATION, VPG, CGW, VPN
AZURE VNET, Resource group, VPN Gateway, Local Gateway, S2S Connection, Virtual Machines.

STEP BY STEP GUIDE -

STEP 1 : Clone the GitHub Repo.

Navigate to following GitHub Repository AWS-AZURE-S2S-VPN
Clone the repo to download the CloudFormation Template for VPC.
CloudFormation template name — VPC_With_Managed_NAT_And_Private_Subnet.yaml

STEP 2 : Creating AWS VPC through CloudFormation service.

Login to AWS account, Navigate to AWS CloudFormation Service.
Head over & change the region of the aws console where you want to deploy the resources & VPN.
Click on Create Stack & upload the template downloaded in the step 1.
Keep rest of the settings as default & hit create.
It will create a New VPC along with IGW, NAT Gateway, Public & Private subnets, Route Tables, IAM roles.
AS NAT Gateway is involved we are going to get charged for NAT Gateway as per the AWS VPC Pricing.

STEP 3 : Creatin Azure Vnet through Azure console.

Login to Azure account, Navigate to Resources groups.
Create a resource group on Azure to deploy the resources.
Select the subscription you want to use for this activity.
Resource Group Name: your choice
Region: your choice
Review + Create = Deployment complete.

Now head over to Create Virtual Network
Select the Resource Group Name created above.
Select the same Region as above.
VNet Name: your choice
VNet IPv4 Address Space: 192.168.0.0/16
Subnet Name: your choice
Subnet IPv4 Address Space: 192.168.1.0/24
Review + Create = Deployment complete.
IMP : Go ahead to subnets section & create a new GatewaySubnet.
This subnet will be used by Azure VPN Gateway as depicted in architecture diagram.

STEP 4 : Creating Azure VPN Gateway through Azure console.

Navigate to Azure VPN Gateway.
VPN Gateway Name: your choice
Region: your choice
Gateway Type: VPN
SKU: VpnGw1
Generation: Generation 1
Virtual Network: vnet-azure
Public IP Address Name: pip-vpn-azure-aws
Public IP Address Type: Standard
Assignment: Assignment
Enable active-active mode: Disabled
Configure BGP: Disabled
Review + Create = Deployment complete.
It will take around 45–60 minutes for Azure VPN deployment to get completed & come active.

STEP 5 : Create a customer gateway pointing to the Public IP Address of Azure VPN Gateway

Navigate to AWS VPC -> Customer Gateway.
NAME : Enter the name for AWS Customer Gateway.
BGP ASN : Keep it as it is
IP address : Enter the Public IP address of Azure VPN Gateway
Hit Create & let it become active.

STEP 6 : Create the AWS Virtual Private Gateway then attach to the VPC

Navigate to AWS VPC -> Virtual Private Gateway.
Name: your choice
Hit Create & after its status shows detached.
Go to Actions on right hand of screen & select Attach VPC.
Attach the VPC created in Step 1.
Wait for its status to change to attached.

STEP 7: Create a site-to-site VPN Connection in AWS Console.

Navigate to AWS VPC -> Site-to-Site VPN connections.
Create a New VPN Connection.
Name: your choice
Target gateway type: Virtual private gateway (Select your Virtual private gateway created in step 6)
Customer gateway: Existing (Select your Customer gateway created in step 5)
Routing options: Static
Static IP prefixes: 192.168.1.0/24 (Azure Subnet created in step 3)
Leave rest of them as default
Hit Create VPN connection.

STEP 8: Download the AWS VPN configuration file for Azure Tunnel Configuration

Navigate to AWS VPC -> Site-to-Site VPN connections.
Once the VPN connection is active in status.
Click on Download Configuration & select dropdown.
Vendor: Generic, Platform: Generic, Software: Vendor Agnostic
In this configuration file you will note that there are the Shared Keys and the Public Ip Address for each of one of the two IPSec tunnels created by AWS. ( Highly Confidential Information )

STEP 9: Creating Azure Local Gateway.

Navigate to Azure Console -> Search Local network gateways
Create a new Local network gateways
Name: your choice
Resource Group Name: created in step 3
Region: same as previous choice
IP address: Get the Outside IP address from the configuration file downloaded in step 8. (You can also get this info from AWS VPN Connection Section -> Tunnel Status.
Address Space(s): 10.10.0.0/16 (AWS VPC CIDR Created in Step 2)
Leave rest of them as default.
Review + Create = Deployment complete.

STEP 10: Create the connection on the Virtual Network Gateway in Azure

Navigate to Azure Console -> Search Virtual Network Gateway
Go to connections -> Add
Name: your choice
Connection Type: Site-to-Site
Local Network Gateway: Select the Local Network Gateway which you created in step 9.
Shared Key: Get the Shared Key from the configuration file downloaded in step 8.
Wait till the Connection Status changes to — Connected
In the same way, check in AWS Console for the status of 1st tunnel in VPN Connection.
Once both side shows connected, It means your phase 1 tunnel is UP.
If not then it means you made a mistake along the way.

VERY IMPORTANT STEP:

STEP 11: Edit the route table associated with our AWS VPC

Add the route for Azure subnet through the AWS Virtual Private Gateway
Goto Public Route Table ( It will have IGW attached in routes )
Add Destination: 192.168.1.0/24
Target: Virtual Private Gateway that we created in step 6.

STEP 12: Create Virtual Machines in both AWS & Azure cloud.

Create a AWS Linux/Windows VM in the same AWS VPC Network, Public Subnet with Public IP address Assignment.
Make sure Security Group allows traffic for ICMP, SSH.
Create a Azure Virtual Machine (Ubuntu/Windows) in the same Azure Vnet & Subnet allowing traffic for ICMP.
Once the VM are active try ping AWS to Azure & Azure to AWS.
This successful private IP ping shows that we have created VPN Tunnel between AWS & Azure.
AWS TO AZURE PING RESULTS :
AZURE TO AWS PING RESULTS :

STEP 13: Decommission

For AWS : Delete VPN Connection, Customer Gateway, Virtual Private Gateway & Detach it. Delete AWS EC2 VM, SG & then delete the CloudFormation Stack.
For Azure : Delete Connections, Local Gateway, VPN Gateway, Virtual Machines, VNET, Subnets
Cross check in AWS NAT Gateways are deleted & EIP are not available.

Congrats ! We have successfully completed lab for Connecting AWS and Azure with Site-to-Site VPN.

I love to talk about Cloud Technology, DevOps, Digital Transformation, Analytics, Infrastructure, Dev Tools, Operational efficiency, Serverless, Cost Optimization, Cloud Networking & Security.

aws #community #builders #VPN #cloudformation #s2s #azure #connectivity #hybrid #network #solution #multicloud #management #centralize #secure #speed #performance #edge #locations #operations #infrastructure #scalable #reliable #highly #available #private #secure #design #acloudguy

You can reach out to me @ acloudguy.in

AWS CloudFront for High Availability

Kunal Shah — Tue, 28 Nov 2023 11:07:11 +0000

AWS CloudFront for High Availability

AWS Cloud Hands on Lab Practice Series

Project Overview —

This project revolves around AWS CloudFront where we configure an distribution, origin groups for a website to be highly available, accessible globally to securely deliver content with low latency and high transfer speeds. By leveraging AWS S3 static website hosting we configure our bucket as a static website, along with it we deploy one more website on AWS EC2 which can act as failover/redundant website target.

SOLUTIONS ARCHITECTURE OVERVIEW -

First Let’s understand the real world use case :

E-commerce Product Images and Static Assets: An e-commerce platform wants to optimize the delivery of product images, style sheets, and other static assets to enhance the online shopping experience for users.
Use Case: AWS CloudFront can be utilized to cache and deliver these static assets from edge locations, ensuring faster page loads and a more responsive user interface.
Accelerated Gaming Content Delivery: Online gaming platforms want to deliver game updates, patches, and in-game assets with low latency for a seamless gaming experience.
Use Case: AWS CloudFront accelerates the delivery of gaming content by caching frequently accessed assets at edge locations. This minimizes download times for users and enhances the overall gaming experience.
High-Traffic Event Websites: Event organizers are expecting a high volume of traffic for a specific event website, such as ticket sales or live streaming of an event.
Use Case: AWS CloudFront helps handle the surge in traffic by distributing content across multiple edge locations, ensuring that users experience minimal latency and preventing the origin server from being overwhelmed.

Prerequisite —

AWS Account with Admin privileges

AWS Services Usage —

AWS CloudFront
AWS S3
AWS EC2
AWS VPC
AWS IAM

STEP BY STEP GUIDE -

STEP 1 : Creating AWS S3 bucket through console.

Login to AWS account, Navigate to AWS S3 Service.
Click on create bucket
Provide Unique Bucket Name (It will be website name)
Select the AWS Region according to nearest user location.
Uncheck the Block all public access.
Tick Mark — I acknowledge that the current settings might result in this bucket and the objects within becoming public.
Keep rest of the settings as is & hit Create bucket.

STEP 2 : Creating AWS S3 bucket policy through console.

Navigate to permissions tab of newly created s3 bucket.
Copy, edit & paste it in the policy section
Edit the bucket ARN & validate the json formatting & save it.
This policy allow anyone to make GET request from Internet.

{
“Version”: “2012–10–17”,
“Statement”: [
{
“Effect”: “Allow”,
“Principal”: “”,
“Action”: “s3:GetObject”,
“Resource”: “/”
}
]
}

STEP 3 : Enable Static website hosting

Navigate to properties tab on your bucket.
Scroll till down to enable static website hosting.
select Host a static website
index document → index.html -> save
You will get a FQDN which is accessible over internet to serve the content.

STEP 4 : Create index.html & upload it.

Create a file named index.html in notepad, Copy the content from GitHub.
You can update the content of index.html as per your choice.
Upload the index.html on s3 bucket.
Your website is up & ready to be shared !! Congrats…

STEP 5 : Launch an EC2 Instance with User Script:

Navigate to EC2 & hit the Launch Instance in the region of your choice.
Enter Name -> Select OS -> Select EC2 Type (t2.micro) -> no key pair.
For Network select default VPC -> Enable Auto Assign Public IP.
Create New Security Group with Inbound access of HTTP & HTTPS.
In ADVANCE SETTING *scroll down till last & copy/upload the **userscript_for_ec2 **from this *GitHub Repo.
Hit the Launch Instance & wait for its status to running.

Enter the PUBLIC IPv4 / EC2 PUBLIC DNS in the browser.
http://
http:///index.html
http:///index2.html
You can notice we have deployed 2 index pages & are serving as expected from AWS EC2 Instance.

STEP 6 : Create CloudFront Distribution:

Navigate to CloudFront Service on AWS Console.
Select the Origin -> S3 Bucket Endpoint (Bucket created in step 1)
Protocol -> HTTP Only
Keep rest of the settings as default & hit the create distribution.
Once the CloudFront distribution is deployed.
Visit the CloudFront distribution Domain Name from Browser.
You will be redirected to S3 static website.

STEP 7 : Add EC2 as Origin in CDN Distribution:

Navigate to Origin Tab of CloudFront Service on AWS Console.
Create Origin -> Enter EC2 PUBLIC DNS as Origin Domain.
NOTE : ONLY EC2 PUBLIC DNS WILL WORK NOT PUBLIC IP.
Protocol -> HTTP Only
Keep rest of the settings as default & hit the create Origin.

STEP 8 : Create Origin Groups of S3 & EC2:

Navigate to Origin Tab of CloudFront Service on AWS Console.
Create Origin Group
Choose Origins -> 1st s3 Origin & 2nd EC2 Origin.
Enter Name for **Origin Group **of your choice.
Select all Failover Criteria.
Finally, Create Origin Group.
*IMP: **This is for origin failover for scenarios that require high availability. We created an *origin group with two origins, a primary(S3) and a secondary(EC2). If the primary origin is unavailable, or returns specific HTTP response status codes that indicate a failure, CloudFront automatically switches to the secondary origin.

STEP 9 : Change Behavior of CDN :

Navigate to Behavior's Tab of CloudFront Service on AWS Console.
Select the one listed & hit the edit button.
Now change the Origin to Origin Group (created in step 8)
Save it & lets play around with CloudFront.

STEP 10 : Lets play with CloudFront :

Now hit the below URLs to understand the working of distribution.
http:// (Pointing towards s3 index.html)
http:///index.html (Pointing towards s3 index.html)
http://index2.html (Pointing towards EC2 index.html)
Lets Do a FAILOVER →
Remove index.html from s3 bucket.
After a while it will point it towards EC2 index.html
This concludes our lab, which shows how CloudFront Distribution works along with origins & origin groups to achieve High Availability, speed & secure content delivery.

STEP 10 : Decommission :

Terminate the EC2 Instance.
Disable & Delete the CloudFront Distribution.
Permanently delete the S3 object & S3 buckets.

Congrats ! We have successfully completed lab for AWS CloudFront for High Availability.

I love to talk about Cloud Technology, DevOps, Digital Transformation, Analytics, Infrastructure, Dev Tools, Operational efficiency, Serverless, Cost Optimization, Cloud Networking & Security.

aws #community #builders #devops #cloudfront #cdn #s3 #ec2 #content #delivery #network #solution #management #centralize #quick #secure #speed #performance #edge #locations #operations #infrastructure #scalable #reliable #highly #available #design #acloudguy

You can reach out to me @ acloudguy.in

Deploy Rancher on AWS EKS using Terraform & Helm Charts

Kunal Shah — Tue, 14 Nov 2023 12:39:42 +0000

Step-by-step guide to deploy Rancher on AWS EKS using Terraform and Helm Charts.

AWS Cloud Hands-on Lab Practice Series

Project Overview —

This project revolves around AWS EKS where we deploy Rancher (platform for Kubernetes management). Rancher is a Kubernetes management tool to deploy and run clusters anywhere and on any provider. Amazon EKS & Rancher is a vital combination when it comes to managing multi-cluster Kubernetes workloads from a Single Dashboard.

SOLUTIONS ARCHITECTURE OVERVIEW -

First, let’s understand the real-world use case:

Hybrid Cloud Management for a Global E-Commerce Platform:
Imagine a large e-commerce platform that serves customers globally. The platform’s infrastructure spans multiple regions to ensure low-latency access and high availability. Rancher provides a unified dashboard for managing EKS clusters spread across various regions.
Finance: Multi-Tiered Application Deployment:
A financial institution is migrating its legacy monolithic applications to a microservices architecture on AWS EKS. Rancher simplifies the deployment of microservices across EKS clusters in different regions.
Retail: Seasonal Application Scaling:
A retail chain experiences significant fluctuations in website traffic during holiday seasons. Rancher enables automated scaling of applications based on predefined policies.
Manufacturing: Edge Computing for IoT Devices:
A manufacturing company utilizes IoT devices across its facilities to monitor and optimize production processes. Rancher supports the deployment of Kubernetes clusters at the edge, close to IoT devices.
Media and Entertainment: Content Delivery Optimization:
A media streaming service operates globally and needs to optimize content delivery for users. Rancher integrates with AWS services like Amazon CloudFront for efficient content caching.

These diverse use cases demonstrate the versatility of Rancher on AWS EKS in addressing industry-specific challenges and enhancing the management of Kubernetes clusters in various contexts. Rancher simplifies operations, enhances security, and provides a unified platform for managing the hybrid cloud environment.

Prerequisite —

AWS ACCOUNT with admin privileges.
AWS EC2 Instance (Bastion Host)
Terraform Installation Guide
HELM Installation Guide
KUBECTL Installation Guide

AWS Services Usage —

AWS EKS
AWS IAM
AWS EC2
AWS ELB
AWS VPC

Step-by-Step Guide -

Step 1: Clone the repo & check the versions of installed tools

Login to AWS EC2 instance (Bastion Host).
Install tools — Terraform, helm, kubectl, aws cli.
Check versions: aws version, kubectl version, helm version.
Now clone the Git Repo: Terraform Repo Link
Give Star & Follow me on GitHub
Repo has 3 files — main.tf, variables.tf, & terraform.tf

Step 2: Deploying AWS resources through Terraform.

Now go to the folder location & run below commands.



Terraform init
Terraform validate
Terraform plan
Terraform apply

This will deploy infrastructure resources in AWS.
It will take around 15–20 mins to get it deployed.
It will have 1 EKS master cluster & 2 worker nodes (ec2 spot instances) attached to the AWS EKS master cluster.

STEP 3 : Add Helm Repositories

helm repo add devpro https://devpro.github.io/helm-charts
This repository contains Helm charts to build clusters with all components running in containers.



helm repo add jetstack https://charts.jetstack.io

helm repo add ingress-nginx  https://kubernetes.github.io/ingress-nginx

helm repo add rancher-latest https://releases.rancher.com/server-charts/latest

helm repo update

helm repo list

STEP 4 : Access the EKS Cluster through kubectl

Run below mentioned commands from EC2 Bastion:



aws eks - region me-south-1 update-kubeconfig - name <eks_cluster_name>

kubectl get nodes

kubectl get all -A

TIP — If you face any permission issue then change permission for ~/.kube/config
This confirms that Cluster on EKS is ready with requested worker nodes.

STEP 5 : Ngnix Ingress Installation (exposing to Internet)

Run below mentioned command :



helm upgrade --install \
      ingress-nginx ingress-nginx/ingress-nginx \
      --namespace ingress-nginx \
      --set controller.service.type=LoadBalancer \
      --version 4.8.3 \
      --create-namespace

Check the services: kubectl get services -n ingress-nginx
Copy & save Loadbalancer DNS ( It will used in step 7)

STEP 6 : Install Certificates manager.

Set env variables
CERT_MANAGER_VERSION=v1.12.4
RANCHER_VERSION=2.7.6
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/${CERT_MANAGER_VERSION}/cert-manager.crds.yaml



helm upgrade --install cert-manager jetstack/cert-manager \
      --namespace cert-manager \
      --create-namespace \
     --version $CERT_MANAGER_VERSION

Check the pods : kubectl get pods -namespace cert-manager ( 3 pods should be in Running status)

Lets have new certificate from letsencrypt



helm upgrade --install letsencrypt devpro/letsencrypt \
      --set registration.emailAddress=$EMAIL_ADDRESS \
      --namespace cert-manager

Now run this command : kubectl get clusterissuer -n cert-manager ( 2 Cluster issuers should be True)

STEP 7 : Install Rancher



- kubectl create namespace cattle-system

helm upgrade --install rancher rancher-latest/rancher \
--namespace cattle-system \
--set hostname=<LOAD_BALANCER_DNS> \
--set 'ingress.extraAnnotations.cert-manager\.io/cluster-issuer=letsencrypt-prod' \
--set ingress.ingressClassName=nginx \
--set ingress.tls.source=secret \
--set ingress.tls.secretName=rancher-tls \
--set replicas=2 \
--version $RANCHER_VERSION**

Check the status of installation & wait for it to complete.



kubectl -n cattle-system rollout status deploy/rancher

kubectl get secret - namespace cattle-system bootstrap-secret -o go-template='{{ .data.bootstrapPassword|base64decode}}{{ "\n" }}'

STEP 8 : Accessing the Rancher UI

Copy Load balancer DNS URL & Paste on browser for Initial setup of Rancher.
You will be asked for password ( copied earlier )
Hit Log in with Local User.
Define the admin password, check the box and click on “Continue”. (store it)
Finally, Rancher is running and you can explore “local” cluster.

STEP 9 : Check AWS EKS Console

STEP 10 : Decommission

Run below command to destroy all resources.



kubectl delete ns cert-manager

kubectl delete ns ingress-nginx

terraform destroy --auto-approve

Congrats ! We have successfully completed lab for Deploying Rancher on AWS EKS using Terraform & Helm Charts.

I love to talk about Cloud Technology, DevOps, Digital Transformation, Analytics, Infrastructure, Dev Tools, Operational efficiency, Serverless, Cost Optimization, Cloud Networking & Security.

aws #community #builders #devops #eks #managed #kubernetes #solution #rancher #solution #management #centralize #dashboard #easy #management #scalability #operational #efficiency #robust #infrastructure #highly #available #reliable #controlled #design #acloudguy

You can reach out to me @ acloudguy.in

Create a free website to share large files stored on AWS S3

Kunal Shah — Wed, 01 Nov 2023 14:48:04 +0000

Create a free website to share large files stored on AWS S3

AWS Cloud Hands on Lab Practice Series

Project Overview —

This project revolves around AWS S3 where we configure an temporary storage or file sharing website to share resources quickly. By leveraging AWS S3 static website hosting we configure our bucket as a static website, the website is available at the AWS Region-specific website endpoint of the bucket.

SOLUTIONS ARCHITECTURE OVERVIEW -

First Let’s understand the real world use case :

File Sharing and Distribution — You can use presigned URLs to allow users to download large files, such as video files, software installers, datasets, etc., directly from your S3 bucket. This can be especially useful for distributing content to a large audience without overwhelming your server infrastructure.
Secure File Sharing: Presigned URLs have an expiration time, making them ideal for secure file sharing. For instance, you can generate a presigned URL for a file that expires after a certain period, thus controlling access to the file. Also share private content securely, restricting access only to intended recipients by providing time-limited links.
Temporary Access to Files: When your application needs temporary access to specific files in S3 (e.g., for generating dynamic content), presigned URLs can be used for access without compromising security.
Media Distribution: For media or streaming applications, you can create presigned URLs to enable temporary access to specific video or audio content. This allows users to access content for a limited time without directly exposing the S3 bucket.
Backup and Restore Operations: Generate presigned URLs for specific backup files or data. This can aid in sharing and restoring backups securely, restricting access after a set time frame.
In essence, using S3 with presigned URLs for sharing content allows for flexible, secure, and controlled access to specific files or objects in an S3 bucket. This method is ideal for scenarios where quick and easy transfer of files or objects is necessary.
This is not for production grade solution. It compromises security risk of making s3 bucket & objects public.

Prerequisite —

AWS Account with Admin privileges
Access to AWS S3 & IAM

AWS Services Usage —

AWS S3
AWS IAM

STEP BY STEP GUIDE -

STEP 1 : Creating AWS S3 bucket through console.

Login to AWS account, Navigate to AWS S3 Service.
Click on create bucket
Provide Unique Bucket Name (It will be website name)
Select the AWS Region according to nearest user location.
Uncheck the Block all public access.
Tick Mark — I acknowledge that the current settings might result in this bucket and the objects within becoming public.
Keep rest of the settings as is & hit Create bucket.

PRO TIP: Alternatively you can create through AWS CLI if cli is configured
aws s3 mb s3://bucket_name — region region-code

STEP 2 : Creating AWS S3 bucket policy through console.

Navigate to permissions tab of newly created s3 bucket.
Copy, edit & paste it in the policy section

{
“Version”: “2012–10–17”,
“Statement”: [
{
“Effect”: “Allow”,
“Principal”: “”,
“Action”: “s3:GetObject”,
“Resource”: “/”
}
]
}

Edit the bucket ARN & validate the json formatting & save it.
This policy allow anyone to make GET request from Internet.

STEP 3 : Enable Static website hosting

Navigate to properties tab on your bucket.
Scroll till down to enable static website hosting.
select Host a static website
index document → index.html -> save
You will get a FQDN which is accessible over internet to serve the content.

STEP 4: Upload contents on s3 & create presigned URLs

Now upload files on your s3 bucket through console.
After successful uploads
Go to specific object -> Actions -> Share with a presigned URL
Give the timeline according to how long you want the presigned URL to be active ( maximum is 12 hours )
Copy the URL & save it for future use.

STEP 5: Create index.html & upload it.

Create a file named index.html in notepad & edit it with below content.
format it & validate the html syntax.
You can update the content of index.html as per your choice.
Upload the index.html on s3 bucket.
Your website is up & ready to be shared !! Congrats…

>  <!DOCTYPE html>
<html>
<head>
 <title>Download Links</title>
</head>
<body>
 <h1>Download Links — Below links will get expired in next 8 hours</h1>
 <h2>Click on each file names to download & create folders according to its Title</h1>
 <ul>
DOWNLOAD LINKS
 <li><a href=”PRESIGNED URL THAT YOU COPIED & SAVED IN ABOVE STEP”>FILE NAME</a></li>
 <! — Add more links as needed →
 </ul>
</body>
</html>

STEP 6: Decommission

Delete s3 buckets & its content.

NOTE — This is not for production grade solution. It compromises security risk of making s3 bucket & objects public.

Congrats ! We have successfully completed lab for creating a website acting like temporary Google Drive on AWS S3.

I love to talk about Cloud Technology, DevOps, Digital Transformation, Analytics, Infrastructure, Dev Tools, Operational efficiency, Serverless, Cost Optimization, Cloud Networking & Security.

aws #community #builders #devops #s3 #google #drive #storage #solution #management #centralize #quick #easy #large #file #sharing #operations #infrastructure #scalable #reliable #controlled #design #acloudguy

You can reach out to me @ acloudguy.in

AWS Security Monitoring using CloudWatch Agent

Kunal Shah — Fri, 22 Sep 2023 13:15:52 +0000

AWS Security Monitoring using CloudWatch Agent

AWS Cloud Hands on Lab Practice Series

Elevating Security with AWS CloudWatch Monitoring and Alerts

Project Overview —

This project revolves around AWS SysOps best practices where we configure an Amazon Linux instances to send log files to Amazon CloudWatch and then create Amazon CloudWatch alarms and notifications to alert for a specified number of login failures on our EC2 instances. Finally, we create a CloudWatch alarm and notification to monitor outgoing traffic through a NAT gateway. By leveraging AWS CloudWatch capabilities we stay proactive & cover observability of our AWS Infrastructure.

SOLUTIONS ARCHITECTURE OVERVIEW -

First Let’s understand the real world use case -

Web Application Security Monitoring — Real-time alerts will help us identify and respond to unauthorized access attempts. Monitoring failed logins across our instances helps us spot potential brute-force attacks. We can take immediate action, such as blocking IP addresses, when CloudWatch alarms trigger.
Compliance and Auditing for Critical Services — Detailed logs and alerts provide the necessary evidence for compliance audits. We can set up CloudWatch alarms to detect deviations from security and compliance policies in real-time. Notifications can be used to inform security teams or compliance officers when an alarm is triggered, enabling swift remediation.
Network Traffic Analysis for Cost Optimization- Real-time alerts will help us identify spikes in outgoing traffic that could lead to unexpected AWS data transfer costs. Monitoring traffic patterns allows us to make informed decisions about scaling resources up or down. We can automate scaling actions or adjust your application’s behavior based on traffic trends.
Application Performance Monitoring- Set up custom CloudWatch dashboards to visualize critical metrics and detect performance bottlenecks. Create CloudWatch alarms to trigger notifications when metrics exceed predefined thresholds, indicating potential performance issues. We can integrate CloudWatch with AWS X-Ray for end-to-end tracing and monitoring of application requests.
Serverless Application Monitoring- Set up custom CloudWatch Metrics to track specific business-related events or performance metrics within your serverless application. Use AWS CloudWatch Insights to perform advanced log analytics and gain deeper insights into function behavior. Implement AWS Step Functions to orchestrate serverless workflows and monitor their execution using CloudWatch Metrics and Alarms.

These use cases demonstrate the versatility of Amazon CloudWatch in monitoring and managing various AWS resources and services, from traditional EC2 instances to serverless applications. By leveraging CloudWatch, we can enhance security, optimize costs, and maintain compliance across various scenarios in real-world AWS environments. By effectively utilizing CloudWatch features, we can proactively manage and secure your infrastructure while meeting specific business needs and objectives.

Prerequisite —

AWS Account with Admin privileges
Access to CloudFormation, EC2, CloudWatch, IAM

AWS Services Usage —

AWS VPC, IAM, EC2, CloudWatch

STEP BY STEP GUIDE -

STEP 1 : Deploying the proposed architecture Infrastructure.

Download the **CloudFormation template from the Repo.**
Create a new stacks for both the templates.
This CloudFormation stacks will deploy resources required for this hands on Lab (2 EC2 Instance, NLB, VPC, Nat Gateways, Lambda Functions, IAM roles & policies)
It will take around 5–10 mins for entire Infrastructure to become active.

STEP 2 : Deploying the CloudWatch Agent on DB Instance

Go to EC2 dashboard, select Database Server Instance.
Connect to Instance through AWS session manager.
Now, Install the CloudWatch agent package using below command
sudo yum install -y amazon-cloudwatch-agent
The CloudWatch agent installer creates a new user named cwagent.
Now provide read and run access to the /var/log/secure file for the cwagent user.
sudo setfacl -m u:cwagent:rx /var/log/secure
We are only sending security logs to CloudWatch & hence we gave access to /var/log/secure only.

STEP 3 : Start the CloudWatch Agent on DB Instance

Run the below command to start the CloudWatch agent configuration wizard
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
We will see the wizard opens with the following options:
Enter 1 for Linux
For Are you using EC2 or On-Premises hosts, enter 1 for EC2.
For Do you want to turn on StatsD daemon, enter 2 for no.
For Do you want to monitor metrics from CollectD. enter 2 for no.
For Do you want to monitor any host metrics, enter 2 for no.
Do you have any existing CloudWatch Log Agent configuration file to import for migration, enter 2 for no.
For Do you want to monitor any log files, enter 1 for yes.
For Log file path, enter /var/log/secure
For Log group name, enter database_server_security_logs
For Log stream name, press ENTER to keep the default value of Instance ID
For Log Group Retention in days, enter 9 for 90
For Do you want to specify any additional log files to monitor, enter 2 for no.
For Do you want to store the config in SSM parameter store, enter 2 for no.
Now lets start the CloudWatch agent service:
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-ahgent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json
We will see the information about the configuration file that is loaded by the agent.
Check CloudWatch agent service status- sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-ahgent-ctl -m ec2 -a status

STEP 4 : Generate Auth failures in the security logs

Run the following command to log in as *dbdev *:
su dbdev
when prompted for a password, try to give wrong creds (to create auth failures)
Repeat this process for 4–5 times to generate logs.

STEP 5 : Check log files in CloudWatch Logs

Navigate to CloudWatch -> logs -> log groups
On the log groups, select the group by the name database_server_security_logs log group.
On the **database_server_security_logs, **choose Log streams tab.
We will see one log stream from DB server Instance.
This will have security logs from DB server Instance.

STEP 6 : Create a Metric Filter

On the database_server_security_logs page, choose the Metric filters tab.
Now create metric filter → Define pattern
For Filter pattern, enter “authentication failure”
In the Test pattern, select the EC2 Instance ID of DB server.
Choose Test pattern -> Next
For Assign metric, For Filtername give db server auth failures
For Metric namespace, enter authentication failures & create new is on.
For Metric name, enter db server auth failures
Metric value 1 and Default value 0
Now review & create -> choose Create metric filter.

STEP 7 : Create a CloudWatch alarm & SNS notification.

On the database_server_security_logs page, choose the Metric filters tab, we will see **db server auth failures **filter.
Choose Create alarm -> Specify metrics & conditions.
For Metric name -> db server auth failures
For Statistic -> Sum and Period -> 5 mins. (your choice)
In the Conditions -> Threshold type select Static -> Greater -> 2
Expand the additional configuration section & for Missing data treatment, select Treat missing data as good (not breaching threshold)
Select Next -> Configure actions -> Notification
For Alarm state trigger, select In alarm
For Select an SNS topic, select Create New Topic.
For Create a new Topic, enter DB_Cloudwatch_alarm_notifications.
For Email endpoints -> enter your email address.
Choose Create Topic -> Next -> Add name & description.
For Alarm name -> database server authentication failures alarm
For Alarm description -> Alarms and notifies for more than 2 auth failures over a span of 5 minutes ( you can edit according to your use case)
Choose Next -> Preview and Create -> Choose Create alarm
Check email & we must confirm email address & then SNS can send notification messages.

8 : Test the alarm & SNS notification.

Login to DB server again & hit su dbdev
when prompted for a password, try to give wrong creds (to create auth failures)
Repeat this process for 4–5 times to generate logs.
Now we should get the notification on email as soon as alarm get in In alarm state.

STEP 9 : Lets monitor NAT Gateway

Create New alarm → select metric -> Metrics -> AWS namespaces Choose NATGateway.
Choose Nat Gateway Metrics
Search for **BytesOutToDestination **in metric search bar.
Select the NatGatewayId that matches the NatGatewayId value listed.
Choose Select metric -> Metric Name -> BytesOutToDestination
For NatGatewayId, verify it matches the NatGatewayId value
Statistic -> Sum & Period -> 15 mins
In the conditions -> Threshold type -> Static -> Greater -> 2000000
Choose Next -> Configure actions -> Notification -> Alarm state trigger -> In alarm
Select SNS -> Existing topic ->**DB_Cloudwatch_alarm_notifications (created earlier) -> **Next
Add Name & description according to your use case.
Preview and Create, Choose Create alarm.

STEP 10 : Generate traffic & activate the alarm

Upload any file which is higher than 5 MB to S3 bucket.
NAT Gateway outgoing traffic alarm will go In alarm state.
Check your email for notification email.

STEP 11 : Decommission

Delete all the created alarm & SNS topics
Delete the CloudFormation stacks from CloudFormation.

Congrats ! We have successfully completed lab for AWS Security Monitoring using CloudWatch Agent

I am Kunal Shah, AWS Certified Solutions Architect, helping clients to achieve optimal solutions on the Cloud. Cloud Enabler by choice, DevOps Practitioner having 7+ Years of overall experience in the IT industry.

I love to talk about Cloud Technology, DevOps, Digital Transformation, Analytics, Infrastructure, Dev Tools, Operational efficiency, Serverless, Cost Optimization, Cloud Networking & Security.

aws #community #builders #devops #cloudwatch #natgateway #observability #proactive #incident #management #centralize #network #security #logging #monitoring #operations #controlled #infrastructure #authentication #control #acloudguy

You can reach out to me @ acloudguy.in

Enabling Seamless AWS Multi-Account Connectivity with AWS Transit Gateway

Kunal Shah — Fri, 01 Sep 2023 09:48:00 +0000

Enabling Seamless AWS Multi-Account Connectivity with AWS Transit Gateway

AWS Cloud Hands on Lab Practice Series

Connecting our AWS World through AWS Transit Gateway.

Project Overview —

The AWS Transit Gateway Setup project is anticipated to result in a more streamlined, secure, and manageable network infrastructure for organizations operating across multiple AWS accounts. By embracing AWS Transit Gateway’s capabilities, empower businesses to focus on their core activities while enjoying enhanced network connectivity without the complexity typically associated with multi-account networking & peering.

SOLUTIONS ARCHITECTURE OVERVIEW -

First Let’s understand the real world use case -

Simplified Network Architecture: The enterprise no longer needs to establish complex, point-to-point connections between each VPC which puts an end to complex peering relationships. The transit gateway acts as a central hub, reducing the need for managing numerous connections.
Isolation and Security: Each VPC remains isolated, ensuring that sensitive data and resources are protected. Traffic between VPCs passes through controlled routes, enhancing security. It can connect Amazon VPCs, AWS accounts, and on-premises networks to a single gateway.
Efficient Data Transfer: By centralizing traffic flow, the transit gateway optimizes data transfer between VPCs, resulting in improved network efficiency. AWS Transit Gateway provides statistics and logs that are then used by services such as Amazon CloudWatch and Amazon VPC Flow Logs.
Scalability: As the enterprise grows and introduces new departments or business units, scaling the network becomes easier, as new VPCs can be integrated with the transit gateway seamlessly. Just like modern day Landing zone where multiple accounts are deployed & managed. Transit gateway plays a vital role for network connectivity across AWS accounts.
Operational Simplicity: With Transit Gateway, the project illustrates how network administrators can manage, monitor, and troubleshoot connectivity from a single interface, enhancing operational efficiency.

Overall, AWS Transit Gateway offers secure edge connectivity, network management, single interface monitor and scalable approach for managing network connectivity across complex multi-account AWS environments & on-premise networks. It enables organizations to efficiently manage their cloud networking, operational efficiency, and streamline practices.

Prerequisite —

3 AWS Account with Admin Access.
This we have created in my AWS Landing Zone setup through AWS Control Tower Blog.
VPC in each AWS account with 2 Private subnets, 2 Public subnet, 3 Route Table, 1 Nat gateway & IGW.

AWS Services Usage —

AWS VPC, IAM, Transit Gateway, EC2, SG

STEP BY STEP GUIDE -

STEP 1 : Create VPC in all 3 of the AWS accounts.

1st AWS Account for Management VPC.

In the VPC Dashboard, choose Create VPC.
Under VPC settings, choose VPC and more.
Complete these fields as follows:
Keep Auto-generated selected under Name tag auto-generation.
Change project to .
The IPv4 CIDR block should be 10.100.0.0/16.
Keep No IPv6 CIDR block option selected.
The Tenancy should remain Default.
Select 2 for the Number of Availability Zones (AZs).
Select 2 for the Number of public subnets and private subnets.
Select 1 Nat Gateway & VPC endpoints should be None.
DNS options — check mark both.
Choose Create VPC. It takes several minutes for the VPC to be created.

2nd AWS Account for Audit VPC.

Follow the same steps as followed above in 2nd AWS Account.
Change project to .
The IPv4 CIDR block should be 10.110.0.0/16.

3rd AWS Account for Logs VPC.

Follow the same steps as followed above in 3rd AWS Account.
Change project to .
The IPv4 CIDR block should be 10.120.0.0/16.

Now All 3 AWS Accounts ( Management, Audit & Logs ) have custom VPC.

STEP 2 : Create Transit Gateway

Log in to the AWS Management Console of the 1st AWS Account (Management account)
Navigate to VPC and select Transit Gateways
Add Name tag and Description for AWS Transit Gateway.
Enter the private ASN for your transit gateway. The range is from 64512 to 65534 for 16-bit ASNs. Read more on ASN
For now enter 64555 as Amazon Side Autonomous System Number (ASN). It must be unique and cannot be the same one used for your Direct Connect or VPN.
Select checkbox for DNS Support, VPN ECMP Support, Default Route Table association and Default Route Table Propagation.

Now head over to VPC > Subnets & copy Private subnet CIDR.
Add copied private subnet CIDR in Transit gateway CIDR blocks.
Give tags & hit Create transit gateway.
It will take about 30 seconds for the Transit Gateway to be in the available state.

STEP 3 : Share Resources across Organization AWS accounts.

AWS RAM (Resource Access Manager) helps us securely share our resources across AWS accounts, within our organization or organizational units (OUs), and with IAM roles and users for supported resource types.
Open the AWS RAM console
Choose Create resource share from the Shared by me
For Name, enter a any name for the resource share.
Under Resources, from dropdown select Transit Gateway.
Select the Transit gateway which we created above in step 2 & hit next.
For Associate managed permissions — optional — hit next.
For Grant access to principals
Select Allow sharing only within your organization & select Principal ID
Review & Create resource share.

STEP 4 : Transit Gateway Attachments

Open the Amazon VPC console in 1st AWS Account (Management account)
On the navigation pane, choose Transit Gateway Attachments.
Choose Create transit gateway attachment.
For Name tag, optionally enter a name for the transit gateway attachment.
For Transit gateway ID, choose the transit gateway for the attachment. We can choose a transit gateway that we created.
For Attachment type, choose VPC.
Choose whether to enable DNS Support and IPv6 Support.
For VPC ID, choose the VPC to attach to the transit gateway.
For Subnet IDs, select one private subnet for each Availability Zone to be used by the transit gateway to route traffic.
Choose Create transit gateway attachment.

STEP 5 : Transit Gateway Attachments for other AWS Accounts

Follow the same steps in 2nd AWS Account (Audit) & 3rd AWS Account (Logs)
We already shared the Transit Gateway in this accounts through AWS RAM. Hence we don't need to create a new Transit Gateway.
We will use the same Transit Gateway & create new Transit Gateway attachments.
After completing the Transit Gateway attachments we will see the snapshot as below

STEP 6 : Creating Transit Gateway Route Tables, Associations, and Propagations

Transit Gateway Route Tables are the elements that allows traffic to pass through the Transit Gateway between each attachment.
They only exist in the account that owns the Transit Gateway. In our case its 1st AWS Account ( Management account )
Transit Gateway Attachments are associated to a Transit Gateway Route Table.
An attachment can only be associated to one route table. However, you can have multiple attachments associated to a single route table.
Navigate to VPC -> Transit Gateways route tables
Click on Routes tab.
Our routing table should be populated with “Management VPC”, “Audit VPC”, “Logs VPC” routes

STEP 7 : Update Route Tables of VPCs

Navigate to Route Tables, select Management VPC Private Route Table 1.
Click on Routes tab.
Edit routes & add routes for Audit & Logs VPC CIDR.

Navigate to 2nd AWS Audit Account -> Route Tables, select Audit VPC Private Route Table 1.
Click on Routes tab.
Edit routes & add routes for Management & Logs VPC CIDR.

Navigate to 3rd AWS Logs Account -> Route Tables, select Logs VPC Private Route Table 1.
Click on Routes tab.
Edit routes & add routes for Management & Audit VPC CIDR.

STEP 8 : Create EC2 Instances for connectivity test

We will create Total 3 EC2 instances, 1 each in each AWS account.
CloudFormation Template for Creating EC2 Instance, Security Group, IAM role & Instance Profile.
ec2-tgw.yml
Verify Instance status checks after creating in Management, Audit & Logs AWS account.

STEP 8 : Ping Test across AWS accounts

Now go ahead & login to Management account EC2 Instance to start pinging to Audit & Log EC2 Instances.
ping audit_ec2_ip_address
ping log_ec2_ip_address
Ping result needs to be successfully transmitted.

Now login to Audit account EC2 Instance to start pinging to Management & Log EC2 Instances.
ping management_ec2_ip_address
ping log_ec2_ip_address

Now login to Logs account EC2 Instance to start pinging to Management & Audit EC2 Instances.
ping management_ec2_ip_address
ping audit_ec2_ip_address

This results prove that connectivity is established across AWS accounts & respective VPCs.

STEP 9 : Decommission

Delete EC2 CloudFormation template across accounts to terminate EC2 Instances.
Remove route table entries of transit gateway from respective private route table.
Terminate Transit Gateway & VPC if not required.

NOTE : Don’t forget to check out the Transit Gateway Pricing page as well as the Transit Gateway FAQ and the Transit Gateway documentation pages to get complete details on how to configure a Transit Gateway.

The AWS Transit Gateway is a network model that can impactfully simplify a network architecture in AWS. Before Transit Gateway, connectivity to multi-VPC environments was accomplished through Direct Connect Gateway or VPC Peering & Transit VPC solutions which proved to be complex in nature. However, the use of AW Transit Gateway allows use cases as simple as inter-VPC connectivity to more complex scenarios involving connectivity to on-site networks.

Congrats ! We have successfully completed lab for Enabling Seamless AWS Multi-Account Connectivity with AWS Transit Gateway

I love to talk about Cloud Technology, DevOps, Digital Transformation, Analytics, Infrastructure, Dev Tools, Operational efficiency, Serverless, Cost Optimization, Cloud Networking & Security.

aws #community #builders #devops #transit #gateway #landing #zone #organization #management #centralize #network #security #logging #audit #operations #scaling #infrastructure #connectivity #account #compliance #control #acloudguy

You can reach out to me @ acloudguy.in

AWS Landing Zone setup through AWS Control Tower

Kunal Shah — Sat, 05 Aug 2023 10:18:24 +0000

AWS Landing Zone setup through AWS Control Tower

AWS Cloud Hands on Lab Practice Series

Streamline Cloud Governance : Secure, Scale, Succeed

Project Overview —

The AWS Landing Zone Setup project aims to architect a secure, well-governed, and scalable cloud environment using AWS Control Tower. The project’s primary goal is to provide a standardized framework for setting up multiple AWS accounts while adhering to AWS best practices and compliance requirements.

SOLUTIONS ARCHITECTURE OVERVIEW -

First Let’s understand the real world use case -

Multi-account Management: Today many organizations use an AWS Landing Zone to establish a multi-account strategy, where each account serves a specific purpose, such as security, logging, audit, operations, development, testing, production, or for individual teams or projects. This segmentation allows better isolation, resource allocation, and account level management. It gives holistic view of all accounts associated to it.
Security and Compliance: AWS Landing Zones help enforce consistent security and compliance standards across all AWS accounts within an organization. By implementing predefined security controls, access policies, and configurations, AWS Landing Zones reduce the risk of security breaches and ensure regulatory compliance. It acts as a single point for security measures.
Cost Optimization: With an AWS Landing Zone, organizations can effectively manage AWS costs by implementing cost-tracking mechanisms, usage policies, and access controls. This allows better visibility into resource usage and cost allocation across various accounts.
Operational and Governance Automation: A well-designed AWS Landing Zone enables automation of repetitive operational tasks, such as user provisioning, account setup, resource deployment, centralized audit & logging and policy enforcement. This streamlines operations and reduces the chances of manual errors.
Network Connectivity and Architecture: AWS Landing Zones facilitate the creation of consistent networking and architecture patterns across AWS accounts, allowing organizations to maintain a standardized and well-organized cloud infrastructure.

Overall, AWS Landing Zone offers a standardized, secure, and scalable approach for managing AWS environments in complex, multi-account scenarios. They enable organizations to efficiently manage their cloud resources, enhance security, and streamline governance and compliance practices.

Prerequisite —

AWS Account with Admin Access.
3 unique email addresses.

AWS Services Usage —

AWS Control Tower, IAM, CloudFormation

Terminologies —

AWS Organization: An AWS Organization is a group of AWS accounts created to simplify the management and billing of multiple AWS accounts. It serves as the root of your account hierarchy.
Organizational Units (OUs): OUs are logical groupings of AWS accounts within an AWS Organization. They help you organize and manage accounts with common requirements or purposes.
Guardrails: Guardrails are a set of predefined policies and best practices that AWS Control Tower enforces to ensure security, compliance, and governance across all accounts within the organization.
Service Control Policies (SCPs): SCPs are policies that you attach to OUs or individual AWS accounts to manage permissions and access control across the organization.
IAM Identity Center (SSO): AWS IAM Identity Center is a service that enables centralized management of user access and permissions across AWS accounts and business applications.

STEP BY STEP GUIDE -

Note — AWS Control Tower sets up paid services, such as AWS CloudTrail, AWS Config, Amazon CloudWatch, Amazon S3, and Amazon VPC. When used, these services may incur costs, as shown on the pricing page.

STEP 1 : Review pricing & select regions

Log in to the AWS Management Console of the AWS Account where you plan to deploy AWS Control Tower. This account will be referred to as the Management account.
Select the service Control Tower under Management & Governance.
Make sure you are in one of the supported regions. Keep in mind that the region selected here is the HOME REGION and cannot be changed once AWS Control Tower is installed.
On AWS Control Tower home page, select Set up landing zone button.
select the region deny setting to not enabled & click next.

STEP 2 : Configure organizational units (OUs)

Now give names to Organizational units (OUs) & click next

STEP 3 : Configure shared accounts

Proceed by giving email addresses for Management account, Log archive account & Audit account. Click next

STEP 4 : Additional Configurations

Select AWS Control Tower sets up AWS account access with IAM Identity Center.
This helps in managing users, roles, policies under one umbrella.
Next, Enable CloudTrail configuration.
Set up frequency of Logs retention.
Keep KMS encryption as not selected as its optional & click next

STEP 5 : Review and set up landing zone

Review all details thoroughly.
Select the check box “I understand..
Click Set up landing zone

STEP 6 : Accept Invitation to join IAM Identity center.

Now, process of deploying AWS resources is started.
All 3 email addresses provided will receive “ACCEPT INVITATION” along with AWS portal link.
Click on ACCEPT INVITATION to join IAM Identity center.

STEP 7 : Track progress & complete the setup

Keep the track of progress as it will take around 30–45 mins to complete the entire setup.
Check CloudFormation stacks to check aws resources provisioned.
AWS Landing Zone is Finally Active & ready to manage.

IMP NOTE — This DEMO/POC might incur some charges if kept active for long time. So please make sure to clean up the environment once done.

Congrats ! We have successfully completed lab for AWS Landing Zone setup through AWS Control Tower.

-------------------------------------*******----------------------------------------

I love to talk about Cloud Technology, DevOps, Digital Transformation, Analytics, Infrastructure, Dev Tools, Operational efficiency, Serverless, Cost Optimization, Cloud Networking & Security.

aws #community #builders #devops #control #tower #landing #zone #organization #iam #centralize #security #logging #audit #operations #scaling #infrastructure #account #vending #machine #acloudguy

You can reach out to me @ acloudguy.in