Forem: Venkatesh K

I Gave an AI Agent Access to My AWS Account - Here's What It Found

Venkatesh K — Tue, 03 Mar 2026 14:33:48 +0000

I Gave an AI Agent Access to My AWS Account - Here's What It Found

The Moment It Clicked

I asked my AI agent what OS it was running on.

It answered correctly, without me telling it anything.

That's when I stopped thinking of it as a chatbot and started thinking of it as something else entirely: an autonomous operator inside my cloud environment.

The agent is called Om. I built it using an open-source framework called OpenClaw, deployed on an EC2 instance in AWS. Instead of logging into consoles, running CLI commands, or digging through dashboards, I just talk to it. Through Telegram.

This post is a walkthrough of what I tested, what Om did, and more importantly, what it made me think about.

What I Built and How It Works

Before the demo, a quick note on the architecture, because it matters for the security conversation later.

OpenClaw AI Agent Architecture

Om runs on an EC2 instance in AWS. It can connect to messaging platforms like Telegram, WhatsApp, MS Teams, Slack, so you can interact with it from whichever platform your team already uses.

Because it runs on the EC2, it has direct access to local system state: OS details, disk usage, running processes. No external API call needed for that.

For AWS access, it authenticates via an IAM Role attached to the EC2 instance, openclaw-iam-role. No hardcoded credentials. No secrets in config files. Just IAM, the way it was designed to work.

Through that role, Om can reach: Cost Explorer, EC2, S3, IAM, VPC, CloudTrail and other AWS Resources.

That's the whole architecture. Simple, clean, and IAM-governed end to end.

Now, here's what happened when I put it to work.

Test 1: System Awareness

What I asked: "What OS am I running?" / "What's my disk utilization?" / "Which directories are consuming the most space?"

Om didn't guess. It inspected the host directly.

It came back with:

OS: Ubuntu 24.04.4 LTS (Noble Numbat), Kernel 6.17.0, running on AWS
Disk: 57% utilized - 4.9 GB used of 8.7 GB total
Top offenders: npm global packages (1.4 GB), npm cache (755 MB), a CUDA library at 432 MB, the Node executable at 118 MB

Then, unprompted, it recommended running npm cache clean --force to reclaim the 755 MB cache.

That last part matters. It didn't just report. It reasoned about the data and made a recommendation. That's the difference between a monitoring tool and an agent.

Test 2: AWS Cost Intelligence and a Deliberate Failure

What I asked: "Check my AWS costs this month."

It failed. ❌

Not because it couldn't, but because the IAM role attached to the instance lacked ce:GetCostAndUsage. And instead of hallucinating a cost figure, Om came back with exactly that explanation: here's what's missing, here's the role name, here's how to fix it.

That moment of honest failure is one of the most important things I saw in this entire test. An agent that knows what it doesn't know is far more useful, and far less dangerous, than one that guesses.

I attached ReadOnlyAccess to the role and asked again.

This time it returned:

February bill to date: effectively $0.00
Forecasted end-of-month: ~$0.06
Account status: comfortably within AWS Free Tier
Service-level cost breakdown across EC2, S3, and others

For a team running production workloads, this same query returns real spend, broken down by service and region, without opening the AWS console once.

Test 3: Full Infrastructure Inventory

What I asked: "Give me a list of all user-created AWS resources, exclude anything AWS creates by default."

This is the kind of query that normally requires a tool like Steampipe, AWS Config, or a manual sweep across every service and region. Om did it in one message.

It returned:

IAM: 7 custom users and 9 custom roles, filtered from 20+ AWS service-linked roles it correctly excluded.
Compute: 1 EC2 instance in ap-south-1, with its root volume, security group, and full VPC topology, subnet, internet gateway, route table.
Storage: 1 S3 bucket.

But here's the flag that jumped out: Lambda-specific IAM roles existed in the account with no corresponding Lambda functions in the region.

Om caught that and called it out. Orphaned permissions. A cleanup task. Exactly what a security review would surface, and it appeared without me asking for it.

Test 4: Security Audit - The Real Test

What I asked: "Who created this EC2 instance? When? From where? Treat this as a request from the security team."

Om went to CloudTrail and assembled a complete forensic report:

Created by: venkatesh
Method: Terraform 1.14.5
Timestamp: February 20, 2026, 15:45:26 UTC
Source IP: 106.51.xx.xx
Identity: SSO-assumed AdministratorAccess role via AWS IAM Identity Center
48-hour activity: Instance stopped Feb 20 (same IP), restarted Feb 22 from a different IP, 106.51.xx.xx indicating the user was on a different network
IAM role assumed on boot: Confirmed correct
IMDSv2 status: Required ✅ - SSRF protection in place

No dashboards. No CloudTrail filter setup. No JSON log parsing.

One sentence. Full chain of custody.

A security analyst using Wiz or AWS Security Hub would get most of this, but they'd need to know which dashboards to open, which filters to apply, which time ranges to set. Om assembled it from plain English.

What Surprised Me Most: The Agent Respected IAM Boundaries

It failed when permissions were missing. It worked when properly authorized.

That's exactly how it should behave. ✅

But that observation opened a bigger question.

The Real Conversation: IAM as an AI Control Plane

We've spent years designing AWS environments for humans, console users, CLI users, Terraform pipelines.

Now we're introducing autonomous agents that can call APIs, read audit logs, inspect infrastructure, and correlate activity across services.

Most AWS accounts today were not designed with that in mind.

If an AI agent assumed the same IAM permissions many engineers casually run with today, it wouldn't just "assist." It could map your entire account in minutes.

That's not a capability problem. That's an architecture question.

Think about what that means in practice:

An over-permissioned agent with AdministratorAccess isn't just a misconfigured tool. It's an autonomous actor with full account visibility.
An agent scoped to read-only, service-specific permissions behaves predictably and safely.
The IAM role an agent assumes is its blast radius.

When AI agents operate inside your cloud environment, IAM design becomes architecture, not just access control.

If permissions are sloppy, agents become risky. ⚠️ If permissions are precise, agents become powerful. 🚀

Your IAM design is now your AI control plane.

What's Next?

This is early. Om today handles system inspection, cost queries, resource inventory, and CloudTrail audits. The architecture is designed to expand, more AWS service integrations, smarter anomaly detection, deeper security correlation.

But the more interesting question isn't what Om can do next.

It's, are your cloud environments ready for agents like this?

I Want to Test Your Hardest Problems

If you had direct access to Om inside an AWS account, what would you throw at it?

Cost anomaly detection? 💰
Security misconfiguration audit? 🔐
Unused resource cleanup? 🧹
Terraform drift detection? 🏗️
Over-permissioned IAM role analysis? 🛑
Public exposure scan? 🌍
Something more aggressive?

Drop your toughest CloudOps / FinOps / Security challenge in the comments.

I'll run it and share the results publicly.

Let's see how ready our architectures really are. 👀

Om is built on OpenClaw, an open-source AI agent framework. Follow me for more on cloud-native AI agents, AWS architecture, and infrastructure engineering.

Connect: X@venkatesh111 | YouTube@LetUsCloud | LinkedIn@venkatesh111

AWS Solutions Architect Associate Cheat Sheet

Venkatesh K — Sun, 31 Aug 2025 13:18:37 +0000

AWS Solutions Architect Associate Cheat Sheet

About This Cheat Sheet

This cheat sheet was prepared based on:

Udemy Practice tests by Jon Bonso, Neal Davis, Stephane Maarek
AWS Documentation
AWS FAQ
AWS Whitepapers

Note: This sheet is for last-minute or quick reference only. These were my notes for a final day glance before the actual SAA exam.

Credits

All credits to excellent SAA-C03 course and practice tests by:

Adrian Cantrill
Chad Smith
Jon Bonso
Neal Davis
Ranga Karnam
Stephane Maarek

You can enroll in any ONE of the courses listed below + Practice Exams to gain knowledge and clear the certification.

Course Links

Adrian Cantrill Course

AWS Certified Solutions Architect Associate SAA-C02

Chad Smith O'Reilly Live Classes

O'Reilly Learning (search by "author: Chad Smith")

Jon Bonso Practice Tests

Neal Davis Course & Practice Tests

Ranga Karnam Course and Exam Review

Stephane Maarek Course & Practice Tests

In-Detail Cheat Sheets by Neal and Jon

AWS Official Study Guides and Trainings

SAA-C03 Update Information

The new exam SAA-C03 started from Aug-2022. Only the weight distribution for domains changed; no major changes from SAA-C02. See What's New with the SAA-C03 by Jon Bonso.

My Tips on Practice Exams

Practice exams are not to determine pass/fail, but to test your understanding of AWS services and choose the most appropriate service for a scenario.
Read the entire question and every option, think like a Solution Architect, and eliminate wrong answers.
Google for help during practice exams (not answers), to learn and make notes.
After completion, check explanations for every answer and make notes.
Actual SAA-C03 is easier than practice exams (practice tests help you learn and prepare).

Connect with the Author

Are you preparing for SAA-C03? Have doubts or want to collaborate for AWS/DevOps certifications? Connect:

Twitter: venkatesh111
LinkedIn: venkatesh111

If planning for AWS Certified Developer Associate, see: AWS DVA-C01 Cheat Sheet

Key Words - to watch out for in questions !

Region

Physical or Geographical location
Made up of two or more Availability Zones
Each region is isolated from others
Multiple Availability Zones per region
Data replication across regions is possible
Communication between regions via public internet

AWS Region Examples

Code	Name
us-east-1	US East (N. Virginia)
ap-south-1	Asia Pacific (Mumbai)
eu-west-2	Europe (London)
me-south-1	Middle East (Bahrain)

Availability Zone

Group of one or more discrete data centres
Redundant power, networking, connectivity
Low latency, high throughput, highly redundant network
Highly available, fault tolerant, scalable infrastructure

Durability

Likelihood of data loss
Example: Multiple copies of data in different locations increases durability
AWS S3 offers 99.999999999% durability

Availability

How readily a service is available
Example: More ATM machines = higher availability
Deploying EC2/RDS in multiple AZs increases availability

Resilient

Recover from failure induced by load, attacks, failures
Partial system failure doesn't take down the whole system

Fault Tolerant

System remains operational even if some components fail

AWS Services

IAM

Explicit deny policy always overrides explicit allow

IAM Roles

Using IAM Role for EC2, ASG is more secure than providing access via IAM user
ECS tasks can also be assigned with IAM ROLES just like IAM Role or EC2 instances
Want EC2 instance to access other AWS services (Example S3) use IAM ROLE
Sharing CloudTrail logs between AWS accounts then use IAM Roles

Cross Account Access

Your developers/Ops want to access particular resources in 2 or more different (PROD, TEST) AWS accounts
Temporary access to resources in a second account (use with STS)

Custom Identity Broker

If your On-Prem LDAP is not compatible with SAML, and you want users to use LDAP to authenticate to AWS use custom identity brokers

Note: You cannot attach IAM Role to On-Prem Instances, use IAM credentials

External ID

To give a third-party access to your AWS resources (delegate access)
Monitor your AWS account and help optimize costs
Perform some analytics etc

IAM Best Practices

Lock away your AWS account root user access keys
Create individual IAM users
Enable MFA
Use user groups
Grant least privilege
Use roles for applications that run on Amazon EC2 instances
Use roles to delegate permissions

AWS IAM Best Practices Documentation

IAM permissions boundaries helps you to restrict AWS IAM admin access and prevent privilege escalation, or allowing them to bypass any other security rules.

S3

Storage Classes Overview

Storage Class	Use Case	Retrieval Time	Cost
S3 Intelligent-Tiering	Unpredictable/changing access patterns	Milliseconds	Moderate
S3 Standard	Frequently accessed data (>1/month)	Milliseconds	Standard
S3 Standard-IA	Infrequently accessed, retained ≥1 month	Milliseconds	Lower
S3 One Zone-IA	Reproducible, lower resiliency requirement	Milliseconds	Lowest IA
S3 Glacier Instant Retrieval	Rarely accessed, immediate retrieval needed	Milliseconds	Archive
S3 Glacier Flexible Retrieval	Rarely accessed, minutes–hours retrieval	1–12 hours	Archive
S3 Glacier Deep Archive	Lowest cost, hours retrieval	12–48 hours	Lowest

Glacier Retrieval Options

Storage Class	Expedited	Standard	Bulk
S3 Glacier Instant Retrieval	N/A	N/A	N/A
S3 Glacier Flexible Retrieval	1–5 min	3–5 hours	5–12 hours
S3 Glacier Deep Archive	N/A	≤12 hours	≤48 hours

Replication at S3

SRR – Same Region Replication

Aggregate logs into a single bucket – If you store logs in multiple buckets or across multiple accounts, you can easily replicate logs into a single, in-Region bucket. Doing so allows for simpler processing of logs in a single location.
Configure live replication between production and test accounts – If you or your customers have production and test accounts that use the same data, you can replicate objects between those multiple accounts, while maintaining object metadata.
Abide by data sovereignty laws – You might be required to store multiple copies of your data in separate AWS accounts within a certain Region. Same-Region Replication can help you automatically replicate critical data when compliance regulations don't allow the data to leave your country.

Encryption at S3

Server-Side Encryption
Request Amazon S3 to encrypt your object before saving it on disks in its data centers and then decrypt it when you download the objects.

Client-Side Encryption
Encrypt data client-side and upload the encrypted data to Amazon S3. In this case, you manage the encryption process, the encryption keys, and related tools.

AWS KMS (SSE-KMS)

Encryption at rest
Automatic key rotation every 1 year
Operational efficiency (least manual efforts)

Server-side Encryption (SSE):

Customer Provided Keys (SSE-C)
S3 Managed Keys (SSE-S3)
KMS Managed Keys (SSE-KMS)

Client-side Encryption (CSE):

Customer managed master encryption keys (CSE-C)
KMS managed master encryption keys (CSE-KMS)

In order to ensure all objects uploaded to S3 are encrypted, create an S3 bucket policy that denies any S3 Put request that does not include the x-amz-server-side-encryption header.

s3:x-amz-server-side-encryption: AES256 → use S3-managed keys
s3:x-amz-server-side-encryption: aws:kms → use AWS KMS managed keys

https://aws.amazon.com/blogs/security/how-to-prevent-uploads-of-unencrypted-objects-to-amazon-s3/

For cost effective analysis on data stored on S3 use Amazon Athena to run SQL queries.

S3 Object Lock

prevent deleting or modifying object for fixed amount of time.
Object Lock can help prevent objects from being deleted or overwritten for a fixed amount of time or indefinitely
Object Lock to help meet regulatory requirements that require WORM (write-once-read-many) storage
Adds layer of protection against object changes and deletion
Object locks must be enabled at the time of creation of buckets (new bucket)
Bucket versioning is automatically enabled (cant be disabled) for Object lock enabled buckets

S3 Static website hosting

Only for static content, can also contain client-side scripts.
Does NOT support Server-side processing/scripting like PHP, JSP, or ASP.NET.

Accessing S3 from EC2 or ECS

IAM Role or Instance profile attached to EC2 to access S3
Data transfer between S3 and EC2 in same region is FREE

S3 Transfer Acceleration

fast, easy, and secure transfers of files over long distances between your client and an S3 bucket
S3 Transfer acceleration uses globally distributed edge locations in Amazon CloudFront
Additional data transfer charges might apply.
Use for large scale (more than 20GB) download and upload of data into S3 from various edge location

Use cases

Your customers upload to a centralized bucket from all over the world.
You transfer gigabytes to terabytes of data on a regular basis across continents.
You can't use all of your available bandwidth over the internet when uploading to Amazon S3.

S3 Cost

Enabling versioning will have additional cost (each versioned objects are charged)
Incomplete S3 multipart uploads are charged
Data transfer cost between S3 buckets in same region is free

VPC endpoint and S3

VPC endpoint for Amazon S3 to upload files/images from EC2 instance in private subnet
VPC endpoint for Amazon S3 reduces Direct connect costs

Feature	Gateway Endpoint for S3	Interface Endpoint for S3
Network Traffic	Remains on AWS network	Remains on AWS network
IP Addresses Used	Amazon S3 public IP addresses	Private IP addresses from your VPC
Access from On-Premises	No access	Allows access
Access from Another AWS Region	No access	Allows access via VPC peering or Transit Gateway
Cost	Free	Not free

EFS

EFS provides hierarchical directory structure
Can be used with both AWS and on-premises resources
Access the files concurrently by multiple EC2 instances
Multiple compute instances, including EC2, ECS (both Fargate and EC2 nodes), and Lambda, can access an EFS file system at the same time

EFS Storage Classes

Reduce storage cost by moving to different EFS storage classes
EFS will automatically and transparently move your files to the lower cost regional EFS Standard-IA or EFS One Zone-IA based on the last time they were accessed
Amazon EFS Intelligent-Tiering: moves the files between storage class

EFS Storage classes

EFS Standard-IA
EFS One Zone-IA

Bursting Throughput mode

It is the default mode, the amount of throughput scales as your file system grows, the more you store, the more throughput is available to you.
Does not incur any additional charges and you have a baseline rate of 50 KB/s per GB of throughput that comes included with the price you pay for your EFS standard storage.

Provisioned Throughput mode

Allows you to burst above your allocated allowance, which is based upon your file system size, so if your file system was relatively small but the use case for your file system required a high throughput rate, then the default bursting throughput options may not be able to process your request quickly enough. In this instance, you would need to use provisioned throughput.
This option does incur additional charges where you will need to pay for any bursting above the default capacity allowed from the standard bursting throughput.

Amazon FSx

Amazon FSx for Windows File Server

Distributed File System Replication (DFSR) ↔ Amazon FSx
Accessible over Server Message Block (SMB) protocol
Amazon FSx is accessible from Windows, Linux, and MacOS
You can use Active Directory domain for authentication

Amazon FSx for Lustre

Enable high performance computing (HPC)
Mounting FSx for Lustre on an AWS Fargate launch type isn't supported. (Use EFS)
Can be mounted on EC2 worker nodes

AWS Snowball Edge

Offline data transfer from remote areas (or on-prem) to AWS
Unstable internet connection
It has on-board storage and compute power, Provides you with storage and processing capacity
Support local data processing and collection in disconnected environments such as ships, windmills, and remote factories

- Used by disaster response team in case of natural disasters like hurricane, storm

AWS DataSync

Online data transfer from on-prem to AWS
Unstable internet connection
Can be used even with loss of Internet access for brief time
- If a task is interrupted, for instance, if the network connection goes down or the AWS DataSync agent is restarted, the next run of the task will transfer missing files, and the data will be complete and consistent at the end of this run.
AWS DataSync is a secure, online service that automates and accelerates moving data between on premises and AWS storage services
DataSync can copy data between:
- NFS, SMB, HDFS (Hadoop), EFS, FSx
- Self-managed object storage, AWS Snowcone, AWS S3
Migrate SMB/NFS from On-Prem to AWS than choice is AWS DataSync
Use cases:
- Migrate your data to AWS, Move data between on-premises and AWS
- Reduce on-premises storage costs by moving data directly to S3 Glacier
- Replicate your data into AWS S3

AWS Storage Gateway

File Gateway

NFS/SMB, over file protocol
Supports local (on-prem) caching

Amazon S3 File Gateway

Useful if you have application making use of EFS to gather and store the content and they may be processed by numerous Amazon EC2 Linux instances
Data lakes, backups, and ML workflows

Amazon FSx File Gateway

Volume Gateway

iSCSI block storage
Hybrid cloud block storage
Supports local (on-prem) caching
Volume Gateway stores and manages on-premises data in Amazon S3 on your behalf
You can take point-in-time copies of your volumes (cached or stored) using AWS Backup

Cached Volume Gateway

Primary data is stored in Amazon S3
Frequently accessed data is retained locally (On-Prem) in the cache for low latency access

Stored Volume Gateway

Primary data is stored locally
Entire dataset is available for low latency access on premises while also asynchronously getting backed up to Amazon S3

Tape Gateway

iSCSI VTL
Replacement for your on-prem physical tapes, without changing existing backup workflows
Supports local (on-prem) caching, Caches virtual tapes on premises for low-latency data access
Encrypts data between the gateway and AWS for secure data transfer
Transitions virtual tapes between Amazon S3 and Amazon S3 Glacier Flexible Retrieval, or Amazon S3 Glacier Deep Archive, to minimize storage costs

CloudFront

Use geographic restrictions (geo blocking) to prevent/block users in specific geographic locations (nations) from accessing content that you're distributing through a CloudFront distribution
You can use CloudFront for on demand (VOD) or live streaming (real time) video
To reduce latency of the images/files hosted on S3 bucket use CloudFront
Cache media, can serve secret/private content
CloudFront is used for only Delivery (CDN) max download size over CloudFront is 20GB

CloudFront Origin Access Identity (OAI)

Restrict access to Amazon S3 bucket so that objects can be accessed only through my Amazon CloudFront distribution

High Availability with CloudFront

You create an origin group with two origins: a primary and a secondary
If the primary origin is unavailable CloudFront automatically switches to the secondary origin

Example

You have s3 bucket in us-west-1 and data is being replicated to ap-southeast-1 then:

Create an additional CloudFront origin pointing to the ap-southeast-1 bucket
Set up a CloudFront origin group with the us-west-1 bucket as the primary and the ap-southeast-1 bucket as the secondary

Field-Level Encryption

Adds an additional layer of security that lets you protect specific data throughout system processing so that only certain applications can see it
Enable your users to securely upload sensitive information to your web servers

Lambda@Edge

Customized material (from website) depending on the device (mobile, desktop, tablet) from which they view the website
Improve search engine optimization (SEO) for your website
Route requests to different origins based on different viewer characteristics
Route requests to origins within a home region, based on a viewer's location

AWS Lambda@Edge Documentation

AWS Global Accelerator

With Global Accelerator, you are provided two global static public IPs that act as a fixed entry point to your application, improving availability
AWS Global Accelerator Reduce Internet latency
Corporate proxies (On-Prem) can also whitelist your application's static IP addresses in their firewalls
Provides static IP which we can bind in the on-prem firewall
Add or remove your AWS application endpoints, such as ALB, NLB, EC2 Instances, and Elastic IPs without making user-facing changes
Real Time Messaging Protocol (RTMP), deliver content over TCP from across the globe
AWS Global Accelerator also performs health checks automatically and route traffic to healthy endpoints

CloudFront Vs Global Accelerator

CloudFront

HTTP, Cacheable TO users over CDN

Global Accelerator

HTTP, and non-HTTP such as TCP, UDP (gaming), RTMP (real time high video and audio), MQTT (IoT), VoIP
From user location

Amazon EC2

Hibernation

You are not charged for Hibernated Instance usage. You pay only for the EBS volumes and Elastic IP Addresses attached to it. There are no other hourly charges (just like any other stopped instance)
To preserve contents of the instance's memory whenever the instance is unavailable
The EBS root volume is restored to its previous state
The RAM contents are reloaded

Spot Instances

Cost-effective choice if your applications can be interrupted
Examples: data analysis, batch jobs, background processing

Placement Groups

There is no charge for creating a placement group

Cluster

Packs instances close together inside an Availability Zone
Low-latency and high network performance necessary for tightly-coupled node-to-node communication
High Performance Computing HPC applications

Partition

Groups of instances in one partition do not share the underlying hardware with groups of instances in different partitions
Used by large distributed and replicated workloads, such as Hadoop, Cassandra, and Kafka

Spread

Strictly places a small group of instances across distinct underlying hardware to reduce correlated failures

AWS Elastic Beanstalk

Ideal for simple web application, NOT ideal for micro services
Supports Time Based scaling
You can scale AWS Elastic Beanstalk environments on a defined schedule. Useful when you know the details around when issue is occurring
Elastic Beanstalk automatically handles the deployment, from capacity provisioning, load balancing, auto-scaling to application health monitoring

ECS - Elastic Container Service

Auto Scaling can be triggered on ECS service based on ECS services CPU utilization
For Amazon ECS cluster, using the Fargate ECS task launch type, use AWS Application Auto Scaling with target tracking policies to scale
By default, Fargate tasks are spread across Availability Zones

Elastic Load Balancers

NLB (Network Load Balancer)

Operates at Layer 4, TCP/UDP
Ultra-low latency
Can handle tens of millions of requests per second while maintaining high throughput at ultra-low latency

ALB (Application Load Balancer)

Operates at Layer 7, HTTP, HTTPS
You can create a listener rule on the ALB to redirect HTTP traffic to HTTPS
ALB supports path-based routing (route the traffic to different target group based on the url/path)

Session Management

Sticky Sessions or Session Affinity (local)

Route a site user to the particular web server that is managing that individual user's session
It's cost effective, generally fast because it eliminates network latency

Drawbacks:

In the event of node failure, session data is lost
If using ASG, traffic may be unevenly distributed

Distributed Session

ElastiCache for Redis, and ElastiCache for Memcached
Provide a shared data storage for sessions that can be accessible from any individual web server
There is additional cost and network latency
These are extremely fast and provide sub-millisecond latency
Can cache any data, not just HTTP sessions

AWS Session Management Documentation

Auto Scaling Groups

You can perform EC2 auto scaling based on SQS queue also

Scheduled Scaling

Schedule according to predictable load changes
Known holidays, known history
Example: Every week the traffic to your web application starts to increase on Wednesday, remains high on Thursday, and starts to decrease on Friday, you can configure a schedule for Amazon EC2 Auto Scaling to increase capacity on Wednesday and decrease capacity on Friday

Dynamic Scaling

Reactive in nature

Target Tracking Scaling Policy

Unpredictable workloads and traffic spikes
To keep the average aggregate CPU utilization of your Auto Scaling group at X percent

Predictive Scaling

Regular patterns of traffic increases (business hours) and applications that take a long time to initialize
Potentially save you money on your EC2 bill by helping you avoid the need to overprovision capacity
Cyclical traffic, such as high use of resources during regular business hours and low use of resources during evenings and weekends
Recurring on-and-off workload patterns, such as batch processing, testing, or periodic data analysis
Applications that take a long time to initialize, causing a noticeable latency impact on application performance during scale-out events

Suspend-Resume Feature

Temporarily pause scaling activities
Useful when you are making a change or investigating a configuration issue

RDS

OLTP is RDS
RDS Storage Auto Scaling automatically scales storage capacity in response to growing database workloads, with zero downtime
Using AWS DMS, you can migrate Oracle relational database running in an on-premises data center to Amazon RDS without modifying the application's code

RDS Read Replica

Read replica for read operation, helps improve RDS overall performance (as reads are redirected to read replica)
Read Replica support multi-region

When you create a read replica

Amazon RDS takes a DB snapshot of your source DB instance and begins replication
If you create multiple read replicas only one snapshot is created at the start of the first create action
You experience a brief I/O suspension on your source DB instance while the DB snapshot occurs

Points to consider for creating read replica

You must enable automatic backups on the source DB instance by setting the backup retention period to a value other than 0
Long-running transaction can slow the process of creating the read replica
AWS recommend that you wait for long-running transactions to complete before creating a read replica

When to use Read Replica?

For performance improvement of RDS (note Multi-AZ is for DR)
For internal systems request data from the RDS DB instance
Scaling beyond the compute or I/O capacity of a single DB instance for read-heavy database workloads
Serving read traffic while the source DB instance is unavailable (data on the read replica may be "stale")
Business reporting or data warehousing scenarios: You may want business reporting queries to run against a read replica rather than your primary, production DB Instance

Additional RDS Notes

Standby Instance are AZ specific, read replicas can be over multiple regions
Use AWS Secret Manager to protect your RDS database with password and automatic key rotation
ElastiCache for Redis to improve RDS DB instance speed/performance
Gaming leaderboard, Top 10 players, real-time score update

To create an encrypted RDS from unencrypted RDS

Take a Snapshot of the RDS instance
Create an encrypted copy of the snapshot
Restore the RDS instance from the encrypted snapshot

Multi AZ RDS Deployment

RPO less than 1 sec
Multi AZ RDS deployment is limited to same region, Cross-Region Multi-AZ isn't supported

RDS HA and DR Metrics

Feature	RPO (approx)	RTO (approx)
Amazon RDS Multi-AZ	0	1-2 Minutes
Read replica promotion (in-Region)	Minutes	< 5 Minutes
PITR (in-Region) using automated backups	5 Minutes	Minutes-Hours
PITR (cross-Region) using automated backups	6-20 Minutes	Minutes-Hours
Snapshot restore	Hours	Minutes-Hours

AWS RDS Read Replicas Documentation
AWS RDS FAQ

AWS Aurora

Multi-Region DB
Multi-AZ DB, and read performance issue from secondary than Aurora is choice, read replication latency of less than one second
Aurora Auto Scaling for the read replica, helps with read replica latency issue if any

Aurora Global Database (DR Purpose of aurora)

Provides disaster recovery from region-wide outages, use for DR purpose between 2 different aws region
Allows a single Amazon Aurora database to span multiple AWS regions
Replicates your data with no impact on database performance, enables fast local reads with low latency in each region
RPO is 1 second and RTO is less than 1 minute

Aurora Serverless

Automatically starts up, shuts down, and scales capacity up or down based on your application's needs
Run your database in the cloud without managing any database capacity
You can create a database endpoint without specifying the DB instance class size
Useful for infrequently accessed Database (For example, your database usage might be heavy for a short period of time, followed by long periods of light activity or no activity at all.)

AWS Aurora Serverless Documentation

Aurora Read Replicas

Useful to improve performance of primary DB of Amazon Aurora
Offload read workloads from the primary DB instance
Supports only read operations
Each Aurora DB cluster can have up to 15 Aurora Replicas
Maintain high availability by locating Aurora Replicas in separate Availability Zones
Aurora automatically fails over to an Aurora Replica in case the primary DB instance becomes unavailable

AWS RedShift

Data warehousing
Both structured and unstructured
Complex or complicated analytical queries and joins

Amazon EMR

Perform Big Data analytics
Big Data processing, examples: Apache Spark, Hive, Presto

Lambda

Max runtime 15min
Minimal operational overhead expenditures → Lambda (not EKS or EC2 or ECS)

DynamoDB

Can handle several million queries per second at its peak and respond in milliseconds
User data in the form of JSON documents, then it is DynamoDB (not RDS)
Key-value store → DynamoDB
The maximum item size in DynamoDB is 400 KB

On-Demand

Your application traffic is difficult to predict (unpredictable) and control
Flash sale
Your workload has large spikes of short duration, or if your average table utilization is well below the peak
New applications, or applications whose database workload is complex to forecast
Developers working on serverless stacks with pay-per-use pricing
SaaS provider and independent software vendors (ISVs) who want the simplicity and resource isolation of deploying a table per subscriber

VPC Endpoints for DynamoDB

Helps you to connect to DynamoDB within AWS network (VPC)
You need route table entry created for the endpoint

DynamoDB Time to Live (TTL)

TTL is useful if you store items that lose relevance (delete items) after a specific time

Use cases:

Remove user or sensor data after one year of inactivity in an application
Archive expired items to an Amazon S3 data lake via Amazon DynamoDB Streams and AWS Lambda
Retain sensitive data for a certain amount of time according to contractual or regulatory obligations
Orders placed after one month will no longer be monitored

Secondary Index

Search item using more than one key: value
Tracking ID, or customer ID, or order ID

Minimal operational overhead expenditures → DynamoDB (not RDS)

DynamoDB Accelerator (DAX)

Fully managed, highly available In-memory caching system used in front of DynamoDB
Performance improvement from milliseconds to microseconds for DynamoDB

API Gateway

RESTful services, REST APIs
Minimal operational overhead expenditures → Lambda (not EKS or EC2 or ECS)

Serverless Microservices

Minimal operational overhead expenditures design serverless:

Frontend/Web Layer

S3 with CloudFront static website hosting

Application Layer

API Gateway and AWS Lambda functions
API Gateway, NLB and AWS Fargate
ALB and AWS ECS

DB Layer

DynamoDB → DB layer to store user data
Aurora → DB layer to store user data
ElastiCache → DB layer to store/cache user data

CloudWatch

To get Memory and disk related metric install cloudwatch agent
- Example: SwapUtilization
You can configure an Amazon CloudWatch alarm that triggers the recovery of the EC2 instance if it becomes impaired (Instance check fails only not system check fails)

CloudTrail

By default, only Management events are logged and not data events
Additional charges apply for data or Insights events

Protecting CloudTrail Logs

Log to a dedicated and centralized Amazon S3 bucket
Enable CloudTrail log file integrity
Encrypting CloudTrail log files with AWS KMS-managed keys (SSE-KMS)
Amazon S3 bucket policy for CloudTrail
Sharing CloudTrail log files between AWS accounts
By default, the log files delivered by CloudTrail to your bucket are encrypted by SSE-S3
To provide a security layer that is directly manageable, you can instead use server-side encryption with AWS KMS-managed keys (SSE-KMS) for your CloudTrail log files

CloudTrail Security Best Practices

To share CloudTrail log files between multiple AWS accounts

Create an IAM role for each account that you want to share log files with
For each of these IAM roles, create an access policy that grants read-only access to the account you want to share the log files with
Have an IAM user in each account programmatically assume the appropriate role and retrieve the log files

CloudTrail Log Sharing Documentation

Route 53

A Record

A record value is always an IP address
A record maps your website like example.com to IP address (ex: Elastic IP)

CNAME

CNAME can never be an IP address
CNAME record maps a name to another name
CNAME are for actual DNS servers, you can't create a CNAME record for example.com

Example:

An A record for example.com points to the server IP address
A CNAME record for www.example.com points to example.com

Alias

Alias record is an Amazon Route 53-specific virtual record
It works only with Amazon Route 53 (AWS specific resources)

AWS specific resources:

ELB
CloudFront Distribution
Elastic Beanstalk
S3 static websites
From one record in a hosted zone to another record

AAAA Record

AAAA record is similar to an A record but it is for IPv6 addresses

MX Record

MX records (Mail Exchange records) is used for setting up Email servers
MX records must be mapped correctly to deliver email to your address

Important: A CNAME can't be used for naked/root domain names. Root domain names must be mapped with either an A record or an Alias record (in Route 53).

Route 53 Routing Policies

Simple Routing Policy

Route traffic to single resource

Failover Routing Policy

Active-passive failover

Latency Routing Policy

You have resources in multiple AWS Regions and you want to route traffic to the Region that provides the best latency with less round-trip time

Geolocation Routing Policy

Route traffic based on the location of your users

Geoproximity Routing Policy

Route traffic based on the location of your resource
Shift traffic from resources in one location to resources in another

Multivalue Answer Routing Policy

You want Route 53 to respond to DNS queries with up to eight healthy records selected at random
Return multiple values for a DNS query and route traffic to multiple IP addresses
Associating a Route 53 health check with records

Weighted Routing Policy

Route traffic to multiple resources in proportions (based on weight 30%, 60% etc) that you specify

DNS Records Reference

SQS

Decouple your architecture

Standard Queues

At-least-once message delivery
Duplicate messages can be delivered
Best effort ordering
Better throughput than FIFO

FIFO Queues

Exactly-once processing
No Duplicate messages
FIFO Order
Low throughput

SQS Temporary Queue Client

Request-Response method, short-lived, lightweight messaging destinations
No intention to use SQS for long term
Leverages virtual queues instead of creating/deleting SQS queues

Additional SQS Notes

Use SQS FIFO for Asynchronously updates to database, avoid dropping writes to the database

Deduplication of messages can be enabled by:

Enable content-based deduplication
Explicitly provide the message deduplication ID

Priority: Use separate queues (both can be standard Q) to provide prioritization of work and EC2 to perform prioritization

Amazon Kinesis

Collect, process, and analyze real-time, streaming data
Real-time data processing
Clickstream data processing
It is fully managed, highly scalable
Default retention is 24hrs, but can be extended to 7days (useful when the destination (S3) is not getting all the data from Kinesis)

Kinesis Video Streams

Securely stream video from connected devices to AWS for analytics, machine learning (ML), and other processing

Kinesis Data Streams

Real-time data streaming service that can continuously capture gigabytes of data per second from hundreds of thousands of sources

Kinesis Data Firehose

Capture, transform, and load data streams into AWS data stores for near real-time analytics

Amazon Kinesis Data Analytics

Process data streams in real time with SQL or Apache Flink without having to learn change existing code/application

SNS

Fanout Scenario

Message published (S3 event notification) to an SNS topic is replicated and pushed to multiple endpoints, such as Kinesis Data Firehose delivery streams, Amazon SQS queues, HTTP(S) endpoints, and Lambda functions. This allows for parallel asynchronous processing

Example: Event-based strategy to run the multiple programs in parallel

SES

Send mail from within any application
Send email securely, globally, and at scale

Use Cases

Transactional emails (purchase confirmations or password resets)
Marketing emails (promotions, special offers and newsletters)
Mass email communications (notifications and announcements)

AWS Secrets Manager

Automatic Key Rotation possible
Helps RDS database with password protection and automatic key rotation
Application on EC2, or if Lambda function needs credentials to be retrieved than best choice is AWS Secrets Manager

AWS Inspector

Automated vulnerability management service that continually scans EC2 and container workloads for software vulnerabilities and unintended network exposure
AWS Inspector is specific to EC2 and Container workloads
Provides Automated Security Assessments for EC2 instances
Requires agent installation on EC2 for Host (vulnerability assessment/best practices) OR can do Network Assessment for EC2 without installing agent

AWS GuardDuty

Threat detection service that continuously monitors your AWS accounts and workloads for malicious activity
It uses Machine Learning, anomaly detection
Can protect against Crypto Currency attacks

Aim is to analyze logs:

CloudTrail Logs: unusual API calls, unauthorized deployments
VPC Flow Logs: unusual internal traffic, unusual IP address
DNS Logs: compromised EC2 instances sending encoded data within DNS queries

AWS Macie

Macie helps identify and alert you to sensitive data, such as personally identifiable information (PII)
AWS Macie is specific to S3

AWS Shield

Avoid DDoS Attacks

AWS WAF

AWS Web Application Firewall (WAF) protect web applications and APIs from attacks
AWS WAF is your first line of defense against web exploits
Use AWS WAF to protect your API Gateway APIs
Protect from SQL injection, Cross-site scripting (XSS)
Protect against HTTP flooding attacks
Use AWS WAF to access or restrict from embargoed nation

Important Notes

AWS WAF rules are evaluated before other access control features, such as resource policies, IAM policies, Lambda authorizers, and Amazon Cognito authorizers
WAF can be integrated with Application Load Balancer (ALB) (and NOT NLB)

You can deploy AWS WAF on:

CloudFront
Application Load Balancer
API Gateway
AWS AppSync

AWS WAF with API Gateway Documentation

VPC

Expanding the VPC's IP Address Capacity

It's NOT possible to change/modify the IP address range of an existing VPC or subnet

You can do one of the following:

Add an additional IPv4 CIDR block as a secondary CIDR to your VPC
Create a new VPC with your preferred CIDR block and then migrate the resources from your old VPC to the new VPC (if applicable)

Additional Notes:

You cannot disable IPv4 support for your VPC and subnet
You can have both IPv4 and IPv6, but not just IPv6 in your VPC

VPC Sharing

Allows multiple AWS accounts (within Same AWS Organization) to create their application resources, such as EC2, RDS, Redshift clusters, and Lambda functions, into shared, centrally-managed virtual private clouds (VPCs)

Use case:

EC2 from "Test Account" want to access Redshift cluster in "Prod Account"

VPC Flow Logs

Capture information about the IP traffic going to and from network interfaces in your VPC
VPC Flow log data can be published to:
- Amazon CloudWatch Logs
- Amazon S3

Flow logs can be used for:

Monitoring the traffic that is reaching your instance
Diagnosing overly restrictive security group rules
Determining the direction of the traffic to and from the network interfaces

NAT Gateway

NAT Gateway is resilient within a single-AZ (loss of AZ is loss of NAT Gateway)
Must create multiple NAT Gateway in multiple AZ for fault-tolerance
Launched in Public subnet, can be used by private instance to connect (routes need to be added) to internet

Site-to-Site VPN Connections

Site to site VPN connection can be established immediately
Site to site VPN connection is cheaper (compared to AWS Direct Connect)
A single VPN tunnel still has a maximum throughput of 1.25 Gbps
Use AWS Transit Gateway to scale an AWS Site-to-Site VPN throughput beyond a single IPsec tunnel's maximum limit of 1.25 Gbps limit
To resolve slower VPN connection, use a transit gateway with equal cost multipath routing and add additional VPN tunnels
Transit Gateway enables you to scale the IPsec VPN throughput with equal cost multi-path (ECMP) routing support over multiple VPN tunnels

Scaling VPN Throughput with Transit Gateway

AWS Direct Connect

Data transfer pricing over Direct Connect is lower than data transfer pricing over the internet

Maximum Resiliency (resiliency as much as possible) for critical or crucial Workloads

Separate connections terminating on separate devices in more than one location

Resilience to:

Device failure
Connectivity failure
Complete location failure

High Resiliency for Critical Workloads

One connection at multiple locations

Resilience to:

Device failure
Connectivity failure due to a fiber cut
Complete location failure

AWS Direct Connect Resiliency Recommendations
AWS Direct Connect Disaster Recovery

Multiple VPC Connection

To connect multiple VPC (Prod, Dev, Test) with On-prem and avoid resource sharing among the connected devices create an AWS Direct Connect connection and a VPN connection for each VPC to connect back to the data center. You cannot use Transit gateway here because you need to avoid resource sharing between VPC

Transit Gateway VPC Connection Documentation

AWS Organizations

You can use aws:PrincipalOrgID condition key in your resource-based policies (S3 bucket policies) to more easily restrict access to IAM principals from accounts in your AWS organization.

[Boost]

Venkatesh K — Fri, 03 Jan 2025 06:15:02 +0000

AWS Quick Guide - Amazon S3

Venkatesh K ・ Jan 1

#aws #s3 #letuscloud

AWS Quick Guide - Amazon S3

Venkatesh K — Wed, 01 Jan 2025 14:20:15 +0000

Amazon S3

Key Concepts

Amazon S3 (Simple Storage Service) is an object storage service designed for:
- Scalability
- High Availability
- Durability
- Security
- Performance
Buckets: Containers for storing objects. Each bucket has a globally unique name.
Objects: Files stored in S3, consisting of data, metadata, and a unique key within the bucket.
High Availability & Durability: S3 Objects are automatically stored across multiple devices spanning a minimum of three Availability Zones (S3 Standard, S3 Standard-IA, and S3 Glacier).
Data Protection:
- Encryption at rest:
  - Server-side encryption with Amazon S3 managed keys (SSE-S3).
  - Server-side encryption with AWS Key Management Service keys (SSE-KMS).
  - Dual-layer (two separate layers of encryption) server-side encryption with AWS Key Management Service keys (DSSE-KMS).
- Encryption in transit (HTTPS).
Storage Classes:
- S3 Standard: Frequently accessed data.
- S3 Standard-IA: Infrequently accessed data.
- S3 Intelligent-Tiering: Automatically optimizes cost for infrequent data.
- S3 One Zone-IA: Infrequently accessed data, stored in single AZ.
- S3 Glacier Instant Retrieval: Long-term archival data with instant retrieval.
- S3 Glacier Flexible Retrieval: Long-term archival data with retrieval times in minutes.
- S3 Glacier Deep Archive: Lowest-cost storage with retrieval times in hours.
Versioning: Maintains multiple versions of objects to protect against accidental overwrites or deletions.
Access Control: Managed through Bucket Policies, ACLs, and IAM policies.
Lifecycle Management: Automates moving objects between storage classes or deletion.
Cross-Region Replication (CRR): Automatically replicates data across AWS regions for disaster recovery.
Event Notifications: Triggers actions when specific events occur, like object creation.
Object Lock: Write-once-read-many (WORM) compliance to prevent object deletion.
Multipart Upload: Uploads large files in parts for efficiency and reliability.

Amazon S3 Durability and Availability

Durability

Durability: Sustainability/loss of Data
- S3 offers high durability of 99.999999999 (or 11 9’s) for objects across multiple Availability Zones.
- If you store 10,000,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000 years.
- 11 9’s durability across all storage classes.

Availability

Availability: Accessibility to the Data
- How readily the service is available?
- Amazon S3 Standard offers 99.99% availability.
- In a given year, S3 Standard may not be available for:
  - Yearly: 52m 35s
  - Monthly: 4m 22s
  - Weekly: 1m 0s
  - Daily: 8s
- Availability varies from one storage class to another.

Amazon S3 Storage Classes:

Amazon Storage Life Cycle Policies:

A lifecycle policy is a set of rules that define how objects in an S3 bucket should be:
- Transitioned between different storage classes.
- Expired or deleted after a certain period.

1. Transitions

Transitions allow you to automatically move objects between different S3 storage classes as they age.
This helps optimize storage costs by moving less frequently accessed data to cheaper storage tiers.

2. Expirations

Expiration rules define when objects should be permanently deleted from the bucket.
This helps manage data retention and prevents unnecessary storage of outdated files.

Common Use Cases

1. Archiving Old Data

Move log files or backup data older than 30 days to cheaper storage classes.
Automatically delete project archives after 7 years.

2. Cost Optimization

Transition infrequently accessed data to cheaper storage tiers.
Remove temporary files or older versions of objects.

3. Compliance and Retention

Automatically delete sensitive documents after a specific retention period.
Ensure compliance with data retention policies.
S3 supports waterfall model for transitioning objects between storage classes.

Amazon S3 Security

Identity-Based Policies: Define what actions a user/group/role can perform.

{
        "Version": "2012-10-17",
        "Statement": [
                {
                        "Effect": "Allow",
                        "Action": [
                                "s3:GetObject",
                                "s3:PutObject"
                        ],
                        "Resource": "arn:aws:s3:::venkatesh-bucket/*"
                }
        ]
}

Resource-Based Policies: Define who (Principal) can perform actions such as allow or deny.

{
    "Version": "2012-10-17",
    "Statement": [
            {
                    "Sid": "PublicRead",
                    "Effect": "Allow",
                    "Principal": "*",
                    "Action": "s3:GetObject",
                    "Resource": "arn:aws:s3:::venkatesh-bucket/public/*"
            }
    ]
}

Access Control List (ACL): Fine-grained control over individual objects.
Block Public Access:
- Block Public Access is applicable to only Public/Anonymous access.
- Block public access settings can override ACLs and bucket policies public access.
- You can apply block public access settings to:
  - Individual buckets or to
  - All buckets in your account.
Presigned URLs:
- Allows you to share objects or allow users to upload objects to buckets without AWS security credentials or permissions.
- Shared URLs use logged-in user credentials (but in a secure way) to access objects.

Identity-Based Policies Vs Resource-Based Policies Vs ACLs

Description	Identity-Based Policies	Resource-Based Policies	Access Control List (ACL)
Attachment	Attached to Users/Groups/Roles	Attached to buckets and applicable at the bucket level	Attached to both buckets and individual objects
Purpose	Define what actions a user/group/role can perform	Define who can perform actions such as allow or deny	Fine-grained control over individual objects
Scope	Identity-level permissions	Bucket-level permissions	Bucket and object-level permissions
Limitations	N/A	You can't have a bucket policy for individual objects	ACLs are legacy, and AWS suggests using IAM policies
Best Use Case	Control user, group, or role actions within the AWS environment	Enable cross-account access or restrict access to specific IP ranges, services, or accounts	Provide quick and specific permissions for individual objects (if legacy usage or fine-grained permissions needed)

Amazon S3 Replication

Automatic, asynchronous copying of objects across Amazon S3 buckets.
S3 Objects can be configured to replicate across multiple destination buckets.
You can change the storage class of replicated S3 objects when performing replication.
Replication options:
- Across AWS Regions (CRR).
- Within the Same AWS Region (SRR).
- Across AWS accounts.
Replication configurations are configured at SOURCE bucket.
Versioning Must be enabled on BOTH source and destination buckets.
Specify Destination Bucket.
IAM Role that the S3 bucket should assume to replicate objects.
You can optionally change destination storage class.
Replication only supports copying new S3 objects after it is enabled.

Amazon S3 Encryption

Encryption at Rest: Server-side encryption with Amazon S3 managed keys (SSE-S3).
Encryption in Transit: Server-side encryption with AWS Key Management Service keys (SSE-KMS).
Dual-layer Encryption: Dual-layer (two separate layers of encryption) server-side encryption with AWS Key Management Service keys (DSSE-KMS).
Encryption by Default: Server-side encryption is enabled by default for new buckets.

Amazon S3 Event Notifications

You can configure S3 to send notifications when certain events occur, such as object creation, deletion, or modification.
Object Created: Triggered when an object is created in a bucket.
Object Deleted: Triggered when an object is deleted from a bucket.
Object Modified: Triggered when an object is modified in a bucket.
Destination Includes:
- SNS: Fan-out messages to systems for parallel processing or directly to people.
- SQS: Send notifications to an SQS queue to be read by a server.
- Lambda: Run a Lambda function script based on S3 events.

Amazon S3 Access Points

AWS S3 Access Points solve the complexity of managing bucket policies for multiple teams and applications by creating separate access points with individualized policies for each team or use case.
Traditional S3 bucket policies become difficult to manage as more teams and applications are added, leading to complex, hard-to-audit policies with limited size and flexibility.
Create a separate S3 access point for every user or application that needs access to S3 bucket.
You can use access points to control access to buckets, objects (folders), or both.
With S3 Access Points, organizations can create multiple access points for a single bucket, each with its own unique properties:
- Unique S3 URI
- Specific Access Point ARN
- Customizable access policy
- Network origin control (internet or VPC-specific)
Access Points allow granular folder-level access control, enabling different teams to access specific folders within the same S3 bucket while maintaining strict access boundaries.
Network origin features let you restrict access points to specific VPCs or make them publicly accessible, providing enhanced security and network segmentation.
To use Access Points effectively, both the Access Point policy and the underlying bucket policy must permit the requested action, with a recommended bucket policy that delegates access control to the Access Points.
Deny policies can be implemented at both the Access Point and bucket levels to provide additional security, such as preventing object deletion in specific folders or restricting direct bucket access.

Amazon S3 Access Points Best Practices:

Granting minimal necessary permissions.
Using Access Point ARNs in IAM policies.
Delegating access control to Access Points through bucket policies.
Leveraging Access Points for scalable and manageable S3 access control.

Amazon S3 Multi-Region Access Points

Multi-region Access Points enable you to create a single access point that spans multiple AWS Regions, providing a unified view of data across all Regions.
Multi-region Access Points are useful for cross-Region replication and cross-Region data access.
With Multi-region Access Points, you can access data across multiple AWS Regions using a single URL, ensuring data consistency and reducing the risk of data loss or corruption.

References

Amazon EC2 Auto Scaling

Venkatesh K — Tue, 01 Oct 2024 06:37:25 +0000

Amazon EC2 Auto Scaling

Auto Scaling Groups (ASG)

An Auto Scaling Group is a collection of EC2 instances that are treated as a logical grouping for the purposes of automatic scaling and management
ASG helps you automatically manage the scaling(in/out) of EC2 instances to meet the dynamic workload of your applications
ASG are horizontal (increase or decrease in EC2 instance Count) Scaling
There are no additional fees with Amazon EC2 Auto Scaling, You only pay for the AWS resources (for example, EC2 instances, EBS volumes, and CloudWatch alarms) that you use.

Advantages of ASG

Fault tolerance
- Amazon EC2 Auto Scaling can detect when an instance is unhealthy, terminate it, and launch an instance to replace it
- You can also configure Amazon EC2 Auto Scaling to use multiple Availability Zones. If one Availability Zone becomes unavailable, Amazon EC2 Auto Scaling can launch instances in another one to compensate.
High Availability
- ASG helps ensure that your application always has the right amount of capacity to handle the current traffic demand
Better Cost Management
- Amazon EC2 Auto Scaling can dynamically increase and decrease capacity as needed.
- You pay for the EC2 instances you use, you save money by launching instances when they are needed and terminating them when they aren't.

Pricing for Amazon EC2 Auto Scaling

There are no additional fees with Amazon EC2 Auto Scaling.
You only pay for the AWS resources (for example, EC2 instances, EBS volumes, and CloudWatch alarms) that you use.

Understanding the Need for Amazon EC2 Auto Scaling

Variable demand

Consider a basic web application running on AWS.
Purpose: Allows employees to search for conference rooms for meetings.
Usage pattern:
- Minimal usage at the beginning and end of the week.
- Increased usage in the middle of the week as more employees schedule meetings.
Graph below Displays the application’s capacity usage over the course of a week.
Traditional Capacity Planning Options:
- Option 1: Add Enough Servers to Always Meet Demand
- Ensures sufficient capacity at all times.
- Downside: Extra capacity remains unused on low-demand days, increasing costs.
- Example: Inefficiency of buying more capacity than needed.
- Option 2: Handle Average Demand
- Less expensive as it avoids purchasing rarely used equipment.
- Downside: Risk of poor customer experience when demand exceeds capacity.
- Example: Poor customer experience due to insufficient capacity.
Amazon EC2 Auto Scaling:
- Option 3: Dynamic Scaling
- Adds new instances only when necessary.
- Terminates instances when no longer needed.
- Cost-effective: Pay only for the instances used.
- Provides the best customer experience while minimizing expenses.
- Example: Adjusting capacity as needed with Amazon EC2 Auto Scaling.

Balancing capacity across Availability Zones

The following image shows an overview of multi-tier architecture deployed across three Availability Zones.

Instance Distribution:
- Amazon EC2 Auto Scaling aims to maintain equivalent numbers of instances in each enabled Availability Zone.
- It attempts to launch new instances in the Availability Zone with the fewest instances.
- If multiple subnets are chosen for the Availability Zone, a subnet is selected at random.
- If the launch attempt fails, it tries to launch instances in another Availability Zone until successful.
- In cases where an Availability Zone becomes unhealthy or unavailable:
- Instance distribution may become uneven across Availability Zones.
- Upon recovery, Amazon EC2 Auto Scaling re-balances the Auto Scaling group.
- It launches instances in the enabled Availability Zones with the fewest instances and terminates instances elsewhere.

Amazon EC2 Auto Scaling instance lifecycle

Start: The beginning of the instance lifecycle.
Pending: The instance is preparing to enter service.
- Pending:Wait: A lifecycle hook where custom actions can be performed before the instance becomes active.
- Pending:Proceed: The instance proceeds to the next state after the custom actions are completed.
InService: The instance is now serving traffic.
- User puts instance into Standby: The instance is manually moved to Standby state.
Standby: The instance is not serving traffic but is still part of the Auto Scaling group.
- User returns instance to service: The instance is manually moved back to InService state.
Terminating: The instance is in the process of being shut down.
- Terminating:Wait: A lifecycle hook where custom actions can be performed before the instance is terminated.
- Terminating:Proceed: The instance proceeds to termination after the custom actions are completed.
Terminated: The instance has been shut down and removed from service.
Detaching: The instance is being detached from the Auto Scaling group.
Detached: The instance has been successfully detached from the Auto Scaling group but not terminated.
End: The lifecycle of the instance has concluded.

Scale out, `InService`, Scale in

Scale out

Scale-Out Events: Direct Auto Scaling group to launch and attach EC2 instances.
- Manual Increase: Manually increase the size of the group.
- Scaling Policy: Automatically increase the size based on demand.
- Scheduled Scaling: Increase the size at a specific time.
Scale-Out Process:
- Auto Scaling group launches required EC2 instances using the launch template.
- Instances start in Pending state.
- Lifecycle Hook: Perform custom actions if added.
- Instances fully configured and pass Amazon EC2 health checks.
- Instances attach to Auto Scaling group and enter InService state.
- Counted against desired capacity of the Auto Scaling group.
Load Balancer Integration:
- If configured to receive traffic from an Elastic Load Balancing load balancer:
- Auto Scaling automatically registers the instance with the load balancer.
- Instance marked as InService after registration.

Instances in Service, `InService`

Instances remain in the InService state until one of the following occurs:
1. Scale-In Event: A scale-in event occurs, and Amazon EC2 Auto Scaling chooses to terminate this instance to reduce the size of the Auto Scaling group.
2. Standby State: You put the instance into a Standby state.
3. Detach Instance: You detach the instance from the Auto Scaling group.
4. Health Check Failure: The instance fails a required number of health checks, so it is removed from the Auto Scaling group, terminated, and replaced.

Scale in

Scale-In Events
- Direct Auto Scaling Group: Detach and terminate EC2 instances.
- Manual Decrease: Manually reduce the group size.
- Scaling Policy: Automatically reduce the group size based on demand.
- Scheduled Scaling: Reduce the group size at a specific time.
Scale-In Process
- Instance Termination: Auto Scaling group terminates instances using its termination policy.
- Terminating State: Instances enter the Terminating state and can't be put back into service.
- Lifecycle Hook: Perform custom actions on terminating instances if added.
- Complete Termination: Instances are fully terminated and enter the Terminated state.
Load Balancer Integration
- De-registration: If using an Elastic Load Balancing load balancer, Auto Scaling automatically de-registers terminating instances.
- Request Redirection: New requests are redirected to other instances, while existing connections continue until the de-registration delay expires.

Launch Templates

Launch Templates are instance configuration template that an Auto Scaling group uses to launch EC2 instances
Launch templates are similar to launch configurations
Key components of launch templates:
- AMI ID
- Instance type
- Key pair
- Security groups
- Other EC2 instance parameters
Advantages over launch configurations:
- Support for multiple versions
- Ability to create subsets of parameters
- Reusability across versions
Versioning benefits:
- Can create a base configuration without AMI or user data
- Add specific AMI and user data in new versions
- Maintain general configuration parameters separately
- Delete testing versions when no longer needed
Recommended over launch configurations for:
- Access to latest features and improvements
- Support for advanced features like:
- Mixed Spot and On-Demand Instances
- Multiple instance types in one Auto Scaling group
Compatible with newer EC2 features:
- Systems Manager parameters (AMI ID)
- Current generation EBS Provisioned IOPS volumes (io2)
- EBS volume tagging
- T2 Unlimited instances
- Capacity Reservations
- Capacity Blocks
- Dedicated Hosts
Template creation:
- All parameters are optional
- Without an AMI specified, you can't add one when creating the Auto Scaling group
- If AMI is specified but no instance type, you can add instance types when creating the group

Launch Template Versioning

The diagram below shows a single launch template with three versions:
Each version can add or modify parameters
Default version (Version 2 in this case) is used unless another version is specified
Allows for flexible configurations within a single launch template

Version 1

Includes:
- t2.micro instance type
- ami-1a2b
- subnet-1111
- key-pair-1
Basic configuration without security group

Version 2 (Default)

Builds on Version 1
Adds:
- sg-2222 (security group)
Set as the default version
Will be used if no specific version is requested when launching an instance

Version 3

Changes some parameters from previous versions
Uses:
- t2.medium instance type
- ami-3c4d
Keeps:
- subnet-1111
- key-pair-1 from Version 1
Adds:
- sg-3333 (different security group from Version 2)

Launch Templates Vs Launch Configuration

Feature	Launch Template	Launch Configuration
State	Newer and more flexible	Older and less flexible
Modification	Can be modified after creation	Cannot be modified once created
Versioning	Supports versioning	Does not support versioning
Use Cases	EC2 instances, Spot Fleets, and ASGs	Only ASGs
Configuration Changes	Allows partial changes	Requires full configuration for changes
Instance Types	Supports multiple instance types	Limited to a single instance type
AMI and Instance Type	Can be specified at launch time	Must be specified when creating
AWS Recommendation	Recommended for new deployments	Legacy option
Flexibility	More flexible and versatile	Less flexible
Updates	Can be updated	Cannot be updated, must create new

ASG LifeCycle hooks

Lifecycle hooks are like a pause button in Amazon EC2 Auto Scaling activity
Lifecycle hooks puts instances into a wait state to perform custom actions during launch or before termination.
Instances remain in a wait state until the lifecycle action is completed or the timeout period ends.
default Timeout: 1 hour
Use cases:
- Launch (Scale-Out) Hook: Installing software, run scripts, or configuring the instance before an instance goes into service.
  - Newly launched instance enters a wait state after startup
  - Run scripts to download and install necessary software
  - Ensure instance is fully ready before receiving traffic
  - Use complete-lifecycle-action command to continue
- Terminate (Scale-In) Hook: Downloading logs, backup data, or perform any clean-up tasks before an instance is terminated.
  - Instance pauses before termination
  - Send notification via Amazon EventBridge
  - Allows actions like invoking AWS Lambda functions or connecting to the instance
  - Opportunity to download logs or other data before full termination
Examples:
- Controlling instance registration with Elastic Load Balancing
- Ensuring bootstrap scripts complete successfully
- Verifying applications are ready to accept traffic
- Registering instances to the load balancer after lifecycle hook completion

Scale-Out Event:
- Instances launch and start in Pending state.
- With autoscaling:EC2_INSTANCE_LAUNCHING hook:
- Move to Pending:Wait state.
- Complete lifecycle action.
- Move to Pending:Proceed state.
- Fully configured instances attach to Auto Scaling group and enter InService state.
Scale-In Event:
- Instances terminate and detach from Auto Scaling group, entering Terminating state.
- With autoscaling:EC2_INSTANCE_TERMINATING hook:
- Move to Terminating:Wait state.
- Complete lifecycle action.
- Move to Terminating:Proceed state.
- Fully terminated instances enter Terminated state.

ASG Warm Pool

warm pool helps to decreases latency for applications with long boot times.
warm pool ensures instances are ready to quickly start serving application traffic during a scale-out event.
Instances in the warm pool count toward the desired capacity when they leave the pool (known as a warm start).
Instance States: Instances in the warm pool can be in one of three states: Stopped, Running, or Hibernated.
- Stopped: Minimizes costs, pay only for volumes and Elastic IP addresses.
- Hibernated: Saves RAM contents to Amazon EBS root volume, pay for EBS volumes and Elastic IP addresses.
- Running: Discouraged to avoid unnecessary charges.
Warm Pool Size:
- Default size: maximum capacity - desired capacity = Default warm pool size
- Example: if maximum capacity = 10 and Desired capacity = 6 than Warm pool size = 10-6 = 4.
- Custom size: Use MaxGroupPreparedCapacity option to set a custom value.
- Example : Maximum capacity = 20, Desired capacity = 6, custom capacity = 8 than Warm pool size = 2.
Lifecycle Hooks:
- Put instances into a wait state for custom actions during launch or termination.
- Delay instances from being stopped or hibernated until they finish initializing.
Instance Reuse Policy:
- Default: Terminates instances when scaling in and launches new instances into the warm pool.
- Reuse Policy: Return instances to the warm pool instead of terminating them, ensuring the pool is not over-provisioned.

ASG Scaling

Scaling in AWS Auto Scaling Group (ASG) refers to the automatic adjustment of the number of EC2 instances in response to changes in demand for your application.
AWS ASG ensures that your application has the right amount of compute capacity at any given time by scaling out (increasing instances) or scaling in (decreasing instances) based on predefined conditions or policies
Scaling Out: Adding more instances to handle an increase in traffic or workload.
Scaling In : Reducing the number of instances when the demand decreases, saving costs.

Types of Scaling

1. Manual Scaling

Manually adjust the number of instances in the ASG.
Useful for predictable workloads or during testing.

2. Automatic Scaling

Scheduled Scaling:
- How it Works: Scale based on a schedule.Scales the number of instances up or down at predetermined times.
- Example:
- Maintain 4 desired instance 6 max and 2 min at specific time of the day.
- Add 5 instances at 8 AM every weekday, remove 5 instances at 6 PM.
- Use Case:
- Ideal for predictable load changes, such as daily or weekly traffic patterns.
Predictive Scaling:
- Uses machine learning to analyze historical load patterns.
- Proactively scales capacity up or down based on predictions.
Dynamic Scaling:
- Automatically scale based on real-time metrics.
- Uses CloudWatch alarms to trigger scaling actions.
- 3 Types of Dynamic Scaling Policies
  1. Simple Scaling
  2. Step Scaling
  3. Target Tracking Scaling

1. Simple Scaling Policy

How it Works: Adds or removes a fixed number of instances when a specific metric breaches a threshold.
Example: Add 1 instance when CPU utilization exceeds 75%, remove 1 instance when CPU falls below 30%.
Use Case: Suitable for basic scaling needs.

2. Step Scaling Policy

How it Works: Scales in steps based on how much the monitored metric deviates from the threshold.
Example:
- CPU usage > 75%, add 2 instances.
- CPU usage > 80%, add 4 instances.
- CPU usage < 30%, remove 1 instance.
Use Case: Ideal for handling variable demand with predefined increments.

3. Target Tracking Scaling Policy

How it Works: Adjusts the instance count to maintain a target value for a specific CloudWatch metric (e.g., CPU utilization).
Example: Set a target CPU utilization to 50%. ASG will scale instances to maintain this target.
Use Case: Best for maintaining consistent performance.

Comparison of Scaling Policies

Scaling Policy	Trigger	Scaling Behavior	Use Case
Scheduled Scaling	Predefined time intervals	Scaling occurs at specified times	Predictable workload patterns
Simple Scaling	Metric breaches a threshold	Fixed increase or decrease in instances	Basic threshold-based scaling
Step Scaling	Metric exceeds/falls below thresholds	Scales in increments based on metric deviations	Variable traffic with sharp demand changes
Target Tracking	CloudWatch metric reaching a target	Scales to maintain a target metric	Continuous, steady-state applications

References

Terraform Basics

Venkatesh K — Mon, 27 Nov 2023 07:26:28 +0000

What is Terraform?

Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. It is used to create/build an environment.
Terraform can manage existing and popular service providers as well as custom in-house solutions.
Terraform can be used for multiple cloud providers, like AWS, Azure, or GCP.
Terraform is developed in the GO language.
Terraform templates are written in a custom Domain Specific Language (DSL) called HashiCorp Configuration Language or HCL.
Terraform templates/files end with a .tf extension.

Terraform Basics

Terraform Definition: Terraform is an open-source Infrastructure as Code (IaC) tool developed by HashiCorp. It allows you to define and provision infrastructure in a declarative configuration language. Terraform supports multiple cloud providers, including AWS, Azure, Google Cloud, and more.

Key Concepts:

Configuration files describe to Terraform the components needed to run a single application or your entire data center.
Execution plan: Terraform generates an execution plan describing what it will do to reach the desired state, and then executes it to build the described infrastructure.
Versioning: As the configuration changes, Terraform is able to determine what changed and create incremental execution plans which can be applied.
The infrastructure Terraform can manage includes low-level components such as compute instances, storage, and networking , as well as high-level components such as DNS entries, SaaS features, etc.
Declarative Configuration: Terraform uses declarative syntax, where you define the desired state of your infrastructure. It then determines the necessary actions to bring the actual infrastructure to that state.
Infrastructure as Code (IaC): Terraform enables you to manage and version your infrastructure as code, providing the benefits of version control, collaboration, and repeatability.
Providers: Terraform uses providers to interact with various infrastructure platforms. For AWS, you would use the AWS provider to manage resources within the AWS environment.

How Terraform Helps with AWS:

Resource Provisioning: Terraform allows you to define AWS resources such as EC2 instances, S3 buckets, and RDS databases in code. It then provisions and manages these resources based on your configurations.
Infrastructure Versioning: Since Terraform configurations are code, you can version control them using tools like Git. This facilitates collaboration, rollback to previous states, and tracking changes over time.
State Management: Terraform maintains a state file that keeps track of the current state of your infrastructure. This state is crucial for understanding what Terraform has created and managing changes to the infrastructure.

How does Terraform work?

Terraform creates and manages resources on cloud platforms and other services through their application programming interfaces (APIs).
Providers enable Terraform to work with virtually any platform or service with an accessible API.

Terraform workflow

1. Write Infrastructure Code (HCL):

You start by writing Terraform configuration files using HashiCorp Configuration Language (HCL).
These files define the infrastructure components you want to create, such as virtual machines, networks, or storage.

2. Initialize:

Run the terraform init command in the directory where your configuration files are located.
This command downloads the necessary providers and sets up the backend to store the Terraform state.

3. Plan:

Execute terraform plan to preview the changes Terraform will make to your infrastructure. It doesn't make any changes yet; it just shows you what will happen.
This is a safety check before making any modifications.

4. Apply:

If the plan looks good, you can apply the changes using terraform apply.
Terraform will prompt you to confirm that you want to make the proposed changes. If you're satisfied, type "yes," and Terraform will create or update your infrastructure according to the configuration.

5. Review and Iterate:

After applying the changes, you can review the outputs and make any necessary adjustments to your configuration.
If you need to make changes, go back to step 3 (plan) and then apply again.

6. Destroy (Optional):

If you want to tear down the infrastructure created by Terraform, you can use terraform destroy.
This command will prompt you to confirm the destruction of resources defined in your configuration.

7. Version Control:

It's a good practice to use version control (e.g., Git) to manage your Terraform configurations.
This helps track changes, collaborate with others, and roll back to previous versions if needed.

Let's connect and explore Terraform and AWS.

References :

What is Terraform

Terraform Resource Block

Venkatesh K — Sun, 29 Oct 2023 08:46:43 +0000

In this article, we are going to understand

What is Terraform resource block
Understand Terraform basic commands
Understand terraform resource behavior on executing terraform basic commands

Terraform Resource Block

A Terraform Resource is a fundamental unit used to model and manage infrastructure components.
Each resource block describes one or more infrastructure objects that you want to create, modify, or manage.
resource Syntax
Example of a Resource Block:

Terraform Resource Behaviors

Terraform resource behaviors refer to,

How Terraform manages and interacts with resources in your infrastructure.
These behaviors determine how resources are

Create :

Terraform attempts to create resources in your target infrastructure based on your configuration.
Terraform creates resources that exist in the configuration but are not associated(present) with a real infrastructure object in the state

Destroy :

Destroys resources that exist in the state/infra but no longer exist in the configuration.
Removing a resource from your Terraform configuration leads to the planned destruction of that resource in the infrastructure.

Update in-place :

update the resources whose arguments have changed
Terraform detects differences between the desired state in your configuration and the current state in the infrastructure. It plans and applies changes to update resources accordingly.

Destroy and re-create :

Terraform will destroy and re-create resources whose arguments have changed but which cannot be updated in-place due to remote API limitations
Example: Changing the Availability zone of an AWS EC2 instance

Dependency Management :

Terraform ensures dependent resources are created or updated before resources that rely on them to avoid issues.

Concurrency Control :

Terraform manages resource operation concurrency to prevent conflicts and ensure consistency.

State Management :
Terraform maintains a state file that records the current state of the infrastructure, which is used to plan and apply updates.

Understanding Terraform Resource Behavior with Example

Let's Create an AWS EC2 instance and understand Terraform Resource(EC2) Behavior
Let's Execute Terraform commands to understand resource behavior

terraform destroy: destroy or delete Resources

terraform destroy is like the " off" switch for your Terraform-managed infrastructure.
It tells Terraform to tear down and delete all the resources in your infrastructure that were created or managed by Terraform.
Let's understand terraform destroy in more detail:

Execution: Terraform analyzes your configuration and the current state of your infrastructure, just like terraform apply.
Resource Destruction: However, instead of creating or updating resources, terraform destroy focuses on destroying and deleting them.
User Confirmation: Similar to terraform apply, it shows you a summary of what it's about to destroy. It can be overridden with auto-approve.
User Approval: You must confirm by typing " yes" when prompted, ensuring you're aware of the resources that are going to be deleted.
Execution: Once you confirm, Terraform executes the destruction, and you can see the progress in real-time.
Completion: After the resources are destroyed, Terraform provides a summary of what was deleted.

Example of terraform destroy

After you type yes to terraform destroy prompt, terraform will start destroying resources mentioned in the plan

You should also be able to check on your AWS Console resource (EC2) being shutting down and getting ready for termination

Once terraform completes the execution you should be able to check on your AWS Console resource (EC2) is successfully terminated.

terraform apply -auto-approve and terraform destroy -auto-approve

The -auto-approve flag is an option that can be added to the terraform apply command to skip the confirmation step.
When you use terraform apply -auto-approve or terraform destroy -auto-approve, Terraform will not ask for your confirmation and will immediately apply the changes described in the execution plan.
This can be useful for automation, scripting, or CI/CD pipelines where manual confirmation is not possible
However, be cautious when using -auto-approve in production environments, as it can lead to unintended changes if the execution plan is not thoroughly reviewed.
Always review the execution plan carefully before using -auto-approve command to ensure that the changes are as expected and won't cause any issues in your infrastructure.

Let's connect and explore Terraform and AWS.

References :

AWS CloudWatch Metric - Monitor EC2 instance reachability to EBS volumes

Venkatesh K — Fri, 13 Oct 2023 05:27:29 +0000

AWS announced on Oct 11, 2023, the new CloudWatch Metric StatusCheckFailed_AttachedEBS
You can now use the StatusCheckFailed_AttachedEBS CloudWatch Metric to monitor EBS Volume reachability from EC2 and also EBS I/O performance
EBS Status Checks Overview:
- Monitor Amazon EBS volumes attached to instances.
- Check if volumes are reachable and can complete I/O operations.
CloudWatch Key Metric: StatusCheckFailed_AttachedEBS :
- Metric StatusCheckFailed_AttachedEBS is s binary value indicating impairment if attached EBS volumes fail I/O operations.
- Detects issues with compute or EBS infrastructure.
Causes of Failure:
- Hardware or software issues on the storage subsystems underlying the EBS volumes
- Hardware issues on the physical host that impact the reachability of the EBS volumes
- Connectivity issues between instance and EBS volumes.
Actions for Failed Checks:
- Wait for AWS resolution or take proactive measures.
- Options include replacing affected volumes or stopping/restarting the instance.
Improving Workload Resilience:
- Use the StatusCheckFailed_AttachedEBS metric to create CloudWatch alarms.
- Trigger actions like failing over to a secondary instance or Availability Zone.
Monitoring I/O Performance:
- Utilize EBS CloudWatch metrics to monitor and replace impaired volumes.
- Address physical host issues impacting EBS volume reachability.
Note:
- Attached EBS status check metric available only for Nitro instances.
- The most used t2.micro instance by beginners is not Nitro based instance, which means you cannot implement the StatusCheckFailed_AttachedEBS metric for t2.micro instance
- The following virtualized instances are built on the Nitro System:
  - General purpose: A1, M5, M5a, M5ad, M5d, M5dn, M5n, M5zn, M6a, M6g, M6gd, M6i, M6id, M6idn, M6in, M7a, M7g, M7gd, M7i, M7i-flex, T3, T3a, and T4g
  - Compute optimized: C5, C5a, C5ad, C5d, C5n, C6a, C6g, C6gd, C6gn, C6i, C6id, C6in, C7a, C7g, C7gd, C7gn, C7i, Hpc6a, Hpc7g, and Hpc7a
  - Memory optimized: Hpc6id, R5, R5a, R5ad, R5b, R5d, R5dn, R5n, R6a, R6g, R6gd, R6i, R6idn, R6in, R6id, R7a, R7g, R7gd, R7iz, U-3tb1, U-6tb1, U-9tb1, U-12tb1, U-18tb1, U-24tb1, X2gd, X2idn, X2iedn, X2iezn, and z1d
  - Storage optimized: D3, D3en, I3en, I4g, I4i, Im4gn, and Is4gen
  - Accelerated computing: DL1, G4ad, G4dn, G5, G5g, Inf1, Inf2, P3dn, P4d, P4de, P5, Trn1, Trn1n, and VT1
- Not viewable using the describe-instance-status AWS CLI command.
Optimize Your Workload:
- Proactive monitoring and actions for enhanced AWS performance.
- Leverage metrics to ensure reliability and address issues promptly.
For More details please check the below links
Let's connect and explore Cloud and AWS.

#AWS #EBS #CloudWatch #EC2 #Cloud

Cloud Deployment Models

Venkatesh K — Tue, 08 Aug 2023 06:25:45 +0000

What is a Cloud Deployment Model?

Cloud deployment refers to:

How you distribute your cloud computing resources such as "compute", "storage", "database" etc in terms of ownership, accessibility, and management.
It also defines, how cloud services or infrastructure are deployed and made available to users or *organizations*

Three very popular cloud deployment models

Public Cloud (Cloud)
Private Cloud (On-Premise)
Hybrid Cloud (Public+Private)

Public Cloud

Definition:

A cloud-based application is fully deployed in the cloud and all parts of the application run in the cloud.
Applications are either created in the cloud or migrated from existing infrastructure to leverage cloud computing benefits.

Ownership:

All hardware, software, and supporting infrastructure are owned and managed by the cloud provider.
You share the same hardware, storage and network devices with other organizations or cloud “tenants”.

Accessibility:

Publicly accessible by multiple users and organizations over the Internet.

Scalability:

Highly scalable, resources can be easily scaled to meet your demanding needs.
Cloud resources such as compute (EC2) can be dynamically increased or decreased based on demand.

Cost:

Pay-as-you-go model: You pay for what you consume
Cost-efficient especially for start-ups and small businesses

Advantages:

Lower costs: no need to purchase hardware or software and you pay only for the service you use.
No maintenance: The cloud service provider provides the maintenance.
Near-unlimited scalability: on-demand resources are available to meet your business needs.
High reliability: a vast network of servers ensures against failure.

Examples:

Amazon Web Services (AWS), Microsoft Azure, Google Cloud

Private Cloud

Definition:

Cloud Offering that is created and owned by the same organization
Cloud offering that is dedicated to single/same organization.
The infrastructure is provisioned on the organization's premises but may be hosted in a third-party data center. In most cases, a private cloud infrastructure is implemented and hosted in an on-premise data center using a virtualization layer

Ownership:

Private Cloud is managed and owned by the organization itself

Accessibility:

Restricted access, limited to a specific organization

Scalability:

Scalability depends on the organization's infrastructure

Cost:

Higher upfront costs and maintenance expenses
Private Clouds are more expensive than Public Cloud due to the capital expenditure involved in building and maintaining the infrastructure.

Advantages:

Private cloud address the security and privacy concerns raised in public cloud offerings.

Examples:

Open Stack, Azure Stack

Hybrid Cloud

Definition:

Hybrid Clouds combine on-premises infrastructure, or Private Cloud, with Public Cloud
Its combination of public and private cloud infrastructure

Ownership:

Both Public and Private Clouds are used, Private Cloud ownership lies with organizations while Cloud Providers maintain the ownership of Public Cloud

Accessibility:

Accessible to multiple users and organizations

Scalability:

Scalability can be achieved by leveraging both environments

Cost:

Costs vary depending on the mix of public and private usage

Advantages:

Make the best of Public Cloud scalability and reliability features while maintaining security controls within the organization

Examples:

AWS Outposts, Microsoft Azure Hybrid Cloud, Google Anthos

Common Use Cases:

you might have legacy applications that are better-maintained on-premises, or government regulations require your business to keep certain records on premises
Data Center Extension: Extending an organization's infrastructure into the cloud while connecting cloud resources to the internal system.
VMware Cloud on AWS: is an integrated cloud offering delivering a scalable, secure and innovative service that allows organizations to seamlessly migrate and extend their on-premises VMware vSphere-based environments to the AWS Cloud running on next-generation Amazon Elastic Compute Cloud (Amazon EC2) bare metal infrastructure.

Let's Summarize

Attribute	Public Cloud	Private Cloud	Hybrid Cloud
Definition	A cloud-based application fully deployed in the cloud with shared resources among multiple users and organizations.	Cloud offering dedicated to a single organization, hosted on-premise or in a third-party data center using a virtualization layer.	Combination of public and private cloud infrastructure, offering both on-premises and cloud resources.
Ownership	All hardware, software, and supporting infrastructure are owned and managed by the cloud provider.	Managed and owned by the organization itself.	Public Cloud owned by providers, Private Cloud owned by organizations.
Accessibility	Publicly accessible by multiple users and organizations over the Internet.	Restricted access, limited to a specific organization.	Accessible to multiple users and organizations.
Scalability	Highly scalable, resources can be easily scaled based on demand.	Scalability depends on the organization's infrastructure.	Scalability achieved by leveraging both environments.
Cost	Pay-as-you-go model, cost-efficient for start-ups and small businesses.	Higher upfront costs and maintenance expenses.	Costs vary depending on the mix of public and private usage.
Advantages	Lower costs, no maintenance, near-unlimited scalability, high reliability.	Addresses security and privacy concerns, more control over resources.	Utilizes best of public cloud scalability while maintaining security controls.
Examples	Amazon Web Services (AWS), Microsoft Azure, Google Cloud.	Open Stack, Azure Stack.	AWS Outposts, Microsoft Azure Hybrid Cloud, Google Anthos.
Common Use Cases	Applications fully in the cloud, cost-efficient scalability.	Legacy applications and regulatory compliance.	Data center extension, VMware Cloud on AWS.

Let's connect and discuss Cloud

LinkedIn : https://www.linkedin.com/in/venkatesh111/

Twitter : https://twitter.com/venkatesh111

YouTube : https://www.youtube.com/@letuscloud

AWS Region – Hyderabad, Now Open for Business

Venkatesh K — Tue, 22 Nov 2022 09:24:51 +0000

Announcement

On Nov 21st 2022, AWS opened its 30th AWS Region, Asia Pacific (Hyderabad) Region, with three Availability Zones

Location
- Telangana, India
Region API name :
- ap-south-2
Availability Zones :
1. ap-south-2a (aps2-az1)
2. ap-south-2b (aps2-az2)
3. ap-south-2c (aps2-az3)

Quick Definition

Region

Is a physical or a geographical location (Mumbai, Ohio, London, Paris and now Hyderabad) made up of two or more Availability Zones
Each region is completely isolated from the other
AWS Regions consists of multiple Availability Zones which are physically separated and isolated
You can enable and control data replication across regions.
Communication between regions happens via public internet.

Availability Zones

Group of one or more discrete data centers
Each AZ is provided with redundant power, networking, and connectivity , housed in separate facilities
Availability Zones are connected with low latency, high throughput, and highly redundant network Availability Zones helps your production environments with highly available, fault tolerant, and scalable infrastructure than would be possible from a single data center

Enabling Hyderabad Region

Please note by default you may not see the new region enabled, you will have to enable the region from your account settings, Details below

Click on your Account id on top right corner
Select " Account"
Scroll Down to section " AWS Regions"
Look for Asia Pacific (Hyderabad) region and select Enable

Select Enable region on next screen

Once enabled it will take few sec to get activated...

You can click on refresh icon to see the changes

after successfully enabling the region you should be able to see the Hyderabad under your region selection

Availability Zones:

New AWS region has 3 Availability Zones

Surely, I wanted to try few services and I launched an EC2 instance and created S3 bucket.

EC2 instance in Hyderabad Region

S3 bucket in Hyderabad Region

If its Hyderabad than how can one not bring up biryani :)

AWS Services in Hyderabad Region:

In the new Hyderabad Region, youcan useC5,C5d, C6g,M5,M5d,M6gd,R5,R5d, R6g, I3,I3en, T3, and T4g instances, anduse a long list of AWS services including: Amazon API Gateway, AWS AppConfig, AWS Application Auto Scaling, Amazon Aurora, Amazon EC2 Auto Scaling, AWS Config, AWS Certificate Manager, AWS Cloud Control API, AWS CloudFormation, AWS CloudTrail, Amazon CloudWatch, Amazon CloudWatch Events, Amazon CloudWatch Logs, AWS CodeDeploy, AWS Database Migration Service, AWS Direct Connect, Amazon DynamoDB, Amazon Elastic Block Store (Amazon EBS), Amazon Elastic Compute Cloud (Amazon EC2), Amazon Elastic Container Registry (Amazon ECR), Amazon Elastic Container Service (Amazon ECS), Amazon ElastiCache, Amazon EMR, Elastic Load Balancing, Elastic Load Balancing Network (NLB), Amazon EventBridge, AWS Fargate, AWS Health Dashboard, AWS Identity and Access Management (IAM), Amazon Kinesis Data Streams, AWS Key Management Service (AWS KMS), AWS Lambda, AWS Marketplace, Amazon OpenSearch Service, Amazon Relational Database Service (Amazon RDS), Amazon Redshift, Amazon Route 53, AWS Secrets Manager, Amazon Simple Storage Service (Amazon S3), Amazon S3 Glacier, Amazon Simple Notification Service (Amazon SNS), Amazon Simple Queue Service (Amazon SQS), AWS Step Functions, AWS Support API, Amazon Simple Workflow Service (Amazon SWF), AWS Systems Manager, AWS Trusted Advisor, VM Import/Export, Amazon Virtual Private Cloud (Amazon VPC), AWS VPN, and AWS X-Ray.

With this launch, AWS now spans 96 Availability Zones within 30 geographic Regions around the world, with three new Regions launched in 2022, including the AWS Middle East (UAE) Region, the AWS Europe (Zurich) Region, and the AWS Europe (Spain) Region. AWS also announced plans for 15 more Availability Zones and five more AWS Regions in Australia, Canada, Israel, New Zealand, and Thailand.

Source :

AWS News Blog

Amazon Prime Day 2022

Venkatesh K — Tue, 26 Jul 2022 13:39:27 +0000

A look at how different AWS services were utilized during Amazon Prime Day 2022 (US) (10th July - 16th July 2022)

Preparations :

A multitude of two-pizza teams worked together to make sure that every part of AWS infrastructure was scaled , tested , and ready to serve end users who were eager to shop during US Amazon Prime day sale (10th July - 16th July)

for more details please check below AWS News Bloghttps://aws.amazon.com/blogs/aws/amazon-prime-day-2022-aws-for-the-win

Follow me on LinkedIn and Twitter for more such articles in cloud tech

Amazon EC2 - Protect instances from unintentional shutdown

Venkatesh K — Fri, 22 Jul 2022 08:45:05 +0000

You can now enable accidental stop protection for EC2 Instance just like accidental termination protection

here is how you can do in 6 easy steps

Select your EC2 Instance
Click on Actions
Click Instance Settings
Click Change stop protection ( this is the new feature added)
Select Enable under Stop protection
Click Save

To disable the feature just uncheck the "Enable" option at Step 5

Special Considerations:

Not applicable for Scheduled Event that stops the instance
No change to Auto scaling behavior ( if instance is unhealthy instance will be terminated per ASG rules)
Not applicable for instance store-backed instances
Not applicable for Spot Instances.

for more details check AWS announcement page and AWS documentation

https://aws.amazon.com/about-aws/whats-new/2022/05/amazon-ec2-enables-protect-instances-unintentional-stop-actions

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Stop_Start.html#Using_StopProtection