Forem: Yashvi Kothari

AWS Storage Gateway(Hybrid Storage Solution)

Yashvi Kothari — Sun, 04 Jan 2026 04:59:22 +0000

AWS Storage Gateway and its architecture

Which type of storage gateway to use for your use case, and

How to create a storage gateway using the EC2 platform

Using AWS Storage Gateway(Hybrid Storage Solution)

AWS Storage Gateway is a powerful service that enables hybrid cloud storage by seamlessly connecting on-premises environments with AWS cloud storage. In this blog, we’ll explore AWS Storage Gateway and its architecture, helping you understand how it works and how it fits into modern cloud and hybrid infrastructures.

We’ll also break down the different types of AWS Storage Gateway—File Gateway, Volume Gateway, and Tape Gateway—and discuss which gateway type to choose based on your specific use case.

To make things practical, the blog will walk you through creating and deploying an AWS Storage Gateway using the Amazon EC2 platform, demonstrating how to set up and configure the gateway step by step.

Prerequisites

To get the most out of this guide, you should have a solid understanding of AWS storage services, including:

Amazon S3

Amazon FSx

Amazon EBS

Amazon Glacier

Basic familiarity with Amazon EC2 is also recommended, as it will help you better understand the deployment and configuration steps covered in the hands-on sections.

The AWS Storage Gateway: The Missing Bridge Between On‑Prem and Cloud Fluency

Every hybrid cloud story begins at the same line: “We still have data on‑prem.”

But the real question AWS architects ask isn’t why — it’s how cleanly can we bridge that world to S3, FSx, or Glacier?

There’s a reason the AWS Well‑Architected Framework flags data movement and hybrid storage patterns in almost every cost and operations review. When on‑prem systems start talking to AWS, you either build a bridge — or a bottleneck.

What the Storage Gateway Really Is

Storage Gateway isn’t just a connector. It’s a translator layer that turns your local protocols — NFS, SMB, iSCSI — into native AWS storage semantics.

At its core, it’s a lightweight virtual machine or a hardware appliance that establishes a secure, encrypted data path to AWS. You can deploy it in three ways:

As a VM on VMware ESXi, Microsoft Hyper‑V, or Linux KVM — ideal for existing datacenter setups.
As a hardware appliance, pre‑configured and shipped by AWS for plug‑and‑play deployment.
As an EC2 instance inside your VPC or within VMware Cloud on AWS — perfect for hybrid use cases with cloud‑resident workloads.

Once deployed, the gateway allocates ~150 MB of local disk space for caching. This cache doesn’t just stage uploads — it works as a low‑latency read buffer, shaving milliseconds off retrieval time for your most frequently accessed objects.

The Four Personalities of Storage Gateway

Each gateway type reflects a different file access pattern, not a marketing SKU.

S3 File Gateway

Talk SMB or NFS. Store natively in S3. Perfect for backup workloads, shared datasets, or applications that don’t speak API‑native S3.
FSx File Gateway

Serve SMB workloads that demand Windows‑native semantics — think ACLs, DFS namespaces, and domain integration — backed by FSx for Windows File Server.
Tape Gateway

The quiet killer of legacy tape libraries. It abstracts your physical tape system into iSCSI‑based virtual tapes, with automatic tiering into Amazon S3 Glacier Flexible Retrieval or Deep Archive.
Volume Gateway

The bridge for block‑based applications. It exports iSCSI block volumes backed by EBS snapshots or S3. Great for backup targets or lift‑and‑shift migrations.

Ask yourself: which protocol do your on‑prem workloads truly speak — files, blocks, or tapes?

That one question reveals your gateway type.

The Economics That Actually Matter

AWS pricing psychology: you’re not paying for the box. You’re paying for where your bits live and how they move.

Three line items drive your AWS Storage Gateway cost model:

Storage pricing — Mirrors the backing service (S3, FSx, etc.).
Request pricing — Matches the standard S3 or FSx operation costs.
Data transfer pricing — Free inbound to AWS, but outbound traffic to on‑prem is billed per terabyte transferred.

Pro tip: moving data out of AWS through the Storage Gateway has the same economics as any AWS data egress — so architect for data gravity, not convenience.

When to Use It — and When to Rethink It

If your workload can’t or won’t move fully to the cloud (yet), the Storage Gateway gives you hybrid leverage without rearchitecting everything overnight.

But if every request ends up bouncing between S3 and on‑prem, you may need to rethink placement, not performance.

The best use cases we see in the wild:

Archival storage for compliance-heavy workloads.
Hybrid file shares across geographies.
Tape backup elimination projects.
Gradual cloud adoption: data first, compute later.

Thought

Every cloud architect eventually realizes this truth: hybrid isn’t transitional — it’s strategic.

The AWS Storage Gateway exists not to extend your datacenter, but to teach it cloud fluency — one NFS mount, one SMB share, one iSCSI target at a time.

Choosing the Right AWS Storage Gateway — A Principal Engineer’s Take

Most hybrid storage decisions don’t fail in architecture. They fail in clarity.
The AWS console gives you four gateway types — S3 File, FSx File, Tape, and Volume. But behind each type lies a philosophy of data motion: how your workloads talk to AWS, how latency collapses, and how cost behaves over time.

Why Storage Gateway Exists

AWS built Storage Gateway for one quiet but urgent reason: most enterprises aren't cloud‑only — they're latency‑bound.
They need to keep data gravity close to compute without giving up cloud durability or cost efficiency.

Storage Gateway bridges that gap. It speaks on‑premises languages (NFS, SMB, iSCSI) while translating natively to AWS storage APIs (S3, FSx, EBS, Glacier).
Think of it as a bilingual edge device — fluent in both legacy infrastructure and cloud semantics.

Scenario 1: Backups and Archives That Refuse to Die On‑Prem

You’ve got Hadoop clusters and SQL databases sitting in your datacenter.
You want backups in the cloud. You don’t want to rebuild your entire pipeline.

Here’s your moment of clarity:

Use S3 File Gateway when you want object control. Use FSx File Gateway when you want Windows consistency. Use Volume Gateway when you want disaster recovery agility.

The trade‑offs are architectural, not cosmetic.

S3 File Gateway:

Stores data natively in S3, with direct access to S3 Object Lock, versioning, and lifecycle management.
Ideal for large files, logs, and database backups that can benefit from cost tiering into Glacier.
Local cache (up to 64 TB) keeps active data hot — ideal for low‑latency reads in analytics or backup workloads.
FSx File Gateway:

Joins your Microsoft Active Directory and feels like any other SMB file system.
Optimized for multi‑user, mixed file workloads — project shares, home directories, or Exchange backups.
Plays well when your org runs Windows clients or third‑party tools expecting full NTFS parity (shadow copies, permissions, DFS).
Volume Gateway:

Perfect for structured data in block storage form.
Lets you back up volumes as EBS snapshots, which you can later restore to EC2 for cloud migration or disaster recovery.
Comes in two flavors:
Cached volumes: Primary data in AWS, frequent reads cached locally.
Stored volumes: Primary data on‑prem, async backups to AWS. (Think “DR‑first architecture.”)
You can decide which to deploy based on one question:
Do you need object features, file semantics, or block storage integration?

Scenario 2: The Tape Archive Nobody Wants to Touch

Here’s the truth every sysadmin quietly knows: tapes die faster than budgets get renewed.

Tape Gateway solves this elegantly. Not with buzzwords — with compatibility.

It creates a virtual tape library (VTL) that plugs directly into your existing backup software (CommVault, NetBackup, IBM, etc.) over iSCSI. For the backup tool, nothing changes. But instead of shipping tapes to dusty warehouses, you’re sending them to Amazon S3 and auto‑archiving to S3 Glacier.

This is hybrid done right: zero cultural resistance, immediate operational gain.

When to Choose Each

The Hidden “Why Now”

Once your backup data lives in AWS, you’ve quietly unlocked optionality.
Today, it’s backup. Tomorrow, it’s pipeline input for analytics or ML.
That’s the real value — AWS turns what used to be retained data into active data.

If your hybrid architecture still treats storage as passive, you’re leaving compounding value on the table.

Final Thought

Each gateway type reflects a philosophy of transition — from legacy to leverage.
The best AWS builders know that hybrid isn't a stopgap; it’s a design state.
Storage Gateway doesn’t replace your infrastructure. It teaches it to speak cloud fluently.

To Do: Hands-On

Configure an AWS S3 File Storage Gateway to test end to end.

Connect an EC2-based NFS file system to an Amazon S3 bucket and verify that files created on EC2 are automatically stored in S3.

Storage Gateway Configuration

Opened AWS Storage Gateway from the AWS Console

Created a new gateway named sgw-demo

Selected S3 File Gateway

Chose Amazon EC2 as the hosting platform

Selected the appropriate time zone

Gateway Deployment on EC2

Launched an EC2 instance for the gateway:

Instance name: sgw-demo-instance

Instance type: m5.xlarge (as recommended)

Public IP enabled

Configured a security group with required ports:

22 (SSH)

80 (Gateway activation)

2049, 111, 20048 (NFS)

Added an extra EBS volume (150 GiB+) for cache storage

Gateway Activation

Used the public IP address of the EC2 instance to activate the gateway

Selected public internet connectivity

Disabled CloudWatch logs and alarms to reduce cost

Allocated cache storage successfully

NFS File Share Creation

Created a new NFS file share

Linked it to an existing S3 bucket (sgw-bucket-99)

Selected S3 Standard as the storage class

Left cache and permission settings as default

Mounting the File Share

Connected to a separate EC2 instance (application server)

Created a mount directory: /web-files

Mounted the NFS file share using the provided Linux mount command

Verified the mount using df -h

Testing the Setup

Created a file index.html inside /web-files

Added sample content to the file

Checked the S3 bucket and confirmed the file appeared there

Downloaded the file from S3 to verify the content

✅ Final Outcome

The NFS file share and S3 bucket were successfully linked

Any file created on the EC2-mounted file system was automatically stored in Amazon S3

This confirms a 1:1 mapping between the NFS file system and the S3 bucket

In simple terms:
Files written on EC2 using NFS were transparently backed up to Amazon S3 using AWS Storage Gateway.

Flow:

On‑prem data (files, volumes, or backups) flows to the Storage Gateway.

The gateway caches frequently accessed data locally for low‑latency access.

Data is securely transferred to AWS storage services (S3, FSx, Glacier, or EBS).

AWS maintains scalable, durable storage and offers lifecycle management, versioning, or snapshotting depending on gateway type.

Administrators monitor and control the entire setup through the AWS Management Console or APIs.

Image Credits Alana Layton

enable FIPS

connect ssh and copy that example command

Secure Identity and Access Management with AWS IAM

Yashvi Kothari — Thu, 25 Dec 2025 14:11:14 +0000

Understand what is meant identity and access management and the difference between authentication, authorization, and access control

Learn the components of IAM as well as its reporting features

Intended Audience
AWS Administrators
Security Engineers
Security Architects
Anyone who is looking to increase their knowledge of the IAM service in preparation for an AWS certification

The First Rule of Infrastructure: Identity is the Perimeter

Most teams treat AWS Identity and Access Management (IAM) as a configuration hurdle. They see it as a series of checkboxes to get a service running.

They are wrong.

In a cloud-native world, the network perimeter is a ghost. Your real firewall isn't an IP range; it’s your IAM policy. If you haven't mastered the distinction between who a user is and what they can do, you aren't managing an environment—you're managing a liability.

The Two Pillars: Authentication vs. Authorization

Confusion here is the root of most "Access Denied" loops. Let’s strip the jargon.

1. Identity Management (The "Who")

Authentication is the process of proving you exist.

The Claim: You provide a unique identifier (a username). IAM ensures this is a singleton within your account; there is no room for ambiguity.
The Proof: You verify that claim. Usually, this is a password. In high-trust environments, it’s a password plus MFA. If you aren't using MFA, you don't have a security policy; you have a suggestion.

2. Access Management (The "What")

Once we know who you are, authorization determines your reach.

The Permission: This is the logic that lives in your JSON policies.
The Scope: Does this identity need FullAccess to EC2, or just ReadOnly to a specific RDS instance?

Status Signal: Principal Engineers don’t grant permissions to "make it work." They grant permissions to make it secure.

The Mechanics of Control

Access control is the delivery vehicle for your security logic. It’s how the "Who" meets the "What." You have three primary levers:

Standard Credentials: The basic handshake of username and password.
Multi-Factor Authentication (MFA): The second layer that turns a stolen password from a catastrophe into a non-event.
Federation: The hallmark of a mature enterprise. Why manage 500 IAM users when you can delegate trust to an external Identity Provider (IdP)? You don't want more identities; you want better ones.

Why This Matters for the Long Game

IAM is the nervous system of your AWS account. It governs, manages, and audits every heartbeat of your infrastructure.

What if your scaling issues aren’t about capacity, but about throttled IAM calls or misconfigured roles? What if your biggest security risk isn't an external hacker, but an internal developer with AdministratorAccess and a leaked API key?

There’s a reason the AWS Well-Architected Framework starts with Identity. If you get the foundation wrong, the rest of the house is just waiting for a reason to fall.

Build for the identity you need, not the convenience you want.

The Architect’s Burden: Why IAM is Your Only Real Firewall

In the cloud, distance is an illusion. Your resources aren't "somewhere else"—they are behind an API.

If you aren't managing your Identity and Access Management (IAM) with surgical precision, you aren't running an infrastructure; you’re running a risk. Without IAM, your AWS account is an open room in a crowded city. With it, it’s a vault. But here is the truth: AWS provides the vault; you provide the combination.

There’s a reason the AWS Well-Architected Framework treats Security as its second pillar. The responsibility for "tight" security is yours alone. It’s the difference between a system that scales and a system that leaks.

1. The Dashboard: Your Command Center

When you step into the IAM Console, you aren't just looking at settings; you're looking at your security posture in a single pane of glass.

The Sign-In URL: This is your front door. Customizing it isn't just about branding—it’s about operational clarity. In a multi-account environment, knowing exactly which door you’re standing at prevents the kind of "oops" that ends up on the morning news.
The Resource Summary: A lean, high-level heartbeat. How many users? How many roles? If these numbers are climbing without a known project, you have an IAM sprawl problem.

2. The Identity Stack: Users, Groups, and Roles

Most engineers confuse these three. Principals do not.

Users (The Permanent): Objects representing a person or an application. They have long-term credentials. Use them sparingly.
Groups (The Logical): You don't give permissions to people; you give permissions to functions. Put the user in the "SRE" group, and let the group hold the policy. It’s cleaner, it’s faster, it’s more of simplicity.
Roles (The Temporal): This is the gold standard. Roles use STS (Security Token Service) to grant temporary, rotating credentials. Why manage a password that can be stolen when you can use a token that expires in an hour?

3. Policies: The Logic of Permission

IAM policies are written in JSON. They are the "code" in "Infrastructure as Code."

Managed Policies: Your library. Use AWS Managed Policies for speed, but pivot to Customer Managed Policies for the Principle of Least Privilege.
Inline Policies: The "strictly local" option. Harder to manage, harder to audit. Ask yourself: if this policy is so special it only applies to one user, is your architecture too complex?

4. Advanced Guardrails: SCPs and Access Analysis

As your organization grows, manual reviews become impossible. You need automated governance.

Service Control Policies (SCPs): These are the "Maximum Permissions." Think of them as a fence. If the SCP says "No S3," it doesn't matter if an IAM user has AdministratorAccess—they are blocked. SCPs don't grant; they restrict.
Access Analyzer: Your silent auditor. It flags resources—like S3 buckets or IAM roles—that are accessible from outside your "Zone of Trust."
The Credential Report: A simple CSV that tells a story. When was the last time Dev-Account-1 was used? If it’s been 90 days, why does it still exist?

The Mastery Shift

What if your security bottlenecks aren't caused by "too much security," but by an outdated understanding of how IAM flows?

Once those IAM roles are assumed, misconfigured trust relationships become silent attack vectors. It’s not about having more rules; it’s about having the right ones.

Manage Message Queues Using Amazon SQS

Yashvi Kothari — Wed, 24 Dec 2025 17:05:46 +0000

Queues are where your architecture tells the truth.
Not in the slide deck, not in the Jira ticket. In the SQS queue quietly filling up while everyone assumes “the system is fine.

If you work in support or ops, you have already met this queue.
A spike in errors, a delayed order, a Lambda that “randomly” retries for hours, a batch job that never finishes. Someone opens CloudWatch, someone else opens SQS, and suddenly there are ten thousand messages waiting for a consumer that is already at one hundred percent CPU.

There is a reason AWS Well Architected keeps repeating the same pattern decouple, buffer, retry, isolate failures. SQS is the boring primitive that quietly makes that possible.

This guide is not another “click here, then here” tour of the console.
You will build one queue, send one message, receive it, and delete it. But the real goal is different you will understand what is actually happening to that message in between those click.

By the end you will know

Why SQS exists when you already have HTTP, events, and direct database writes.

When to reach for a Standard queue and when a FIFO queue is the only safe option.

How the message lifecycle really works send, store, visibility timeout, retry, delete and where bugs usually hide.

What the console lab is teaching you under the hood so you can later switch to CLI or SDK without feeling lost.

If you are used to Windows, GUIs, and “next, next, finish” wizards, think of this as your first real conversation with a cloud native queue.
The console is just the training wheels. The mental model is what you keep

Goal

Create Amazon SQS queues

Send messages to an SQS queue

Retrieve and delete messages using the AWS Management Console

Pre-requisite

following background knowledge is helpful but not required:

Basic familiarity with the AWS Management Console
OR

Basic understanding of the AWS CLI

Introduction to Amazon Simple Queue Service (SQS)

Amazon Simple Queue Service (Amazon SQS) is a fully managed message queuing service that enables you to build fast, reliable, and scalable distributed applications. It allows different components of an application to communicate with each other asynchronously by sending, storing, and receiving messages—without requiring those components to be available at the same time.

At its core, an SQS queue acts as a temporary message repository. Messages remain in the queue until they are successfully processed and deleted, ensuring that no data is lost even if a component fails or is temporarily unavailable. This makes Amazon SQS a powerful tool for decoupling application components and improving fault tolerance and scalability.

Creating SQS Queue and Publishing Messages

Amazon Simple Queue Service (Amazon SQS) is a fully managed messaging service that provides fast, reliable, and scalable queues for storing messages. It enables seamless communication between distributed components of an application, allowing each component to perform different tasks independently without losing messages or requiring constant availability.

An SQS queue acts as a temporary repository for messages that are waiting to be processed. It serves as a buffer between the component that produces and sends data and the component that receives and processes it. This buffering capability helps resolve common challenges, such as when a producer generates data faster than a consumer can handle, or when either component is intermittently connected to the network.

Amazon SQS guarantees that each message is delivered at least once and supports multiple producers and consumers accessing the same queue simultaneously. A single queue can be safely shared by many distributed application components without requiring them to coordinate with one another, making it an ideal solution for building loosely coupled and highly scalable systems.

Guide: Creating and Sending Messages to an Amazon SQS Queue

Follow the steps below to create an Amazon SQS queue and send your first message using the AWS Management Console.

Step 1: Open Amazon SQS

In the search bar at the top of the AWS Management Console, type SQS.
Under Services, select Simple Queue Service.

Step 2: Create a New Queue

On the SQS dashboard, click Create queue.

Step 3: Configure Queue Settings

Enter q-labs as the Queue name

Leave all other settings at their default values

You will notice two available queue types:

Standard

FIFO (First-In, First-Out)

For now, keep Standard selected. Although Standard queues provide weaker guarantees for message order and delivery compared to FIFO queues, they are more cost-effective and support the highest throughput.

Step 4: Create the Queue

Scroll to the bottom of the page and click Create queue.

After a short moment, a page will load displaying the details of your newly created queue.

Step 5: Return to the Queue List

At the top of the page, click Queues in the breadcrumb navigation to return to the main queue list.

Step 6: Send a Message

Select the q-labs queue and click Send and receive messages.

An SQS message consists of:

A message body

Optional message attributes

The message body can be plain text or structured data such as JSON.

Step 7: Enter the Message Body

In the Message body field, enter:

This is my first message!

Step 8: Add Message Attributes

Expand the Message attributes section and add the following:

Name: WorkerId

Type: Number

Value: 123456

Step 9: Send the Message

In the top-right corner of the page, click Send message.

A confirmation notification will appear indicating the message was successfully sent.

Step 10: View Message Details

Click View details to see:

The SQS message identifier

MD5 checksums for the message body and attributes

These MD5 hashes allow publishers to verify message integrity. When messages are sent programmatically, publishers can compare their locally generated hashes with those returned by Amazon SQS to detect data corruption or tampering—an important feature in regulated or high-compliance environments.

Step 11: Close Message Details

Click Done to close the message details window.

Now, you successfully created an Amazon SQS queue using the AWS Management Console and sent your first message with custom attributes.
This demonstrates how SQS enables reliable message-based communication between distributed application components.

Polling for SQS Messages and Deleting Messages

Use the AWS Management Console to poll for messages from an Amazon Simple Queue Service (SQS) queue. You will review message details and then delete the message after processing it.

Instructions: Polling and Deleting SQS Messages
Step 1: Poll for Messages

To retrieve messages from the queue, click Poll for messages.

If a message is available, it will appear in the messages list.

Note: When requesting messages from an SQS queue, you cannot specify which message to retrieve. Instead, you specify the maximum number of messages to receive (up to 10), and Amazon SQS returns up to that number. Because Amazon SQS is a distributed system, the response may sometimes be empty—especially when the queue contains only a small number of messages.

Step 2: Open Message Details

Click the Message ID located on the far left of the message entry in the queue.

Step 3: Review Message Properties

In the Message Details modal, review the available information by clicking through the different tabs.

You will see details similar to those observed when the message was sent. Notice that the Details section includes a Sender account ID. Amazon SQS queues are often used across AWS accounts, allowing message publishers and consumers to operate in different accounts.

Step 4: Close the Message Details Window

Click Done to close the message details modal.

Step 5: Delete the Message

Select the message in the Messages table and click Delete.

Step 6: Confirm Deletion

In the Delete Messages confirmation dialog box, click Delete.

You will be returned to the Send and receive messages page, where a notification confirms that the message has been successfully deleted.

Now, you successfully polled an Amazon SQS queue for messages, reviewed detailed message properties, and deleted the message after processing. This demonstrates the full lifecycle of receiving and managing messages in an SQS queue using the AWS Management Console.

The Message Lifecycle Nobody Explains

Every message you just sent follows an invisible lifecycle that breaks most first timers.
Send it. SQS stores it with a visibility timeout (default 30 seconds). Poll it. Process it. Delete it explicitly, or it reappears for the next consumer.

Miss the delete, and you get duplicates. Set visibility too low, and retries overlap into chaos. AWS Well Architected calls this out because one forgotten delete turns a queue into a memory leak.

What if your consumer crashes mid process? The message reverts to the queue automatically. No data loss. That's SQS quietly earning trust while your Lambda or ECS task restarts.

Dead Letter Queues: Your Debug Lifeline

Support tickets spike when queues fill with "poison messages" that no consumer can process.
Enter Dead Letter Queues (DLQ). Set max receives to 3. Fourth failure moves it to DLQ automatically. Your main queue stays clean.

In the console lab above, add a DLQ now. Edit queue. Dead letter queue section. Target another queue. Set receive limit to 2. Send a malformed JSON message. Watch it migrate after two polls.

Reality check: 80% of production SQS issues trace back to unmonitored DLQs. Check them daily. They surface the malformed payloads, bad IAM, and deserialization bugs before customers notice.

Standard vs FIFO: When Order Breaks You

Your q-labs queue used Standard. At scale, order means nothing. Messages might arrive shuffled. Throughput hits millions per second. Cost: fractions of a penny.

FIFO queues guarantee order and exactly once delivery. Use them for payment confirmations or inventory decrements. Limit: 300 TPS unless you pay for high throughput mode.

Question for you:

Does shuffled order break your app, or is eventual consistency fine? Most support workloads pick Standard and save the complexity. AWS flags FIFO in Well Architected only when sequencing is non negotiable.

Real Integrations Beyond the Console

SQS rarely runs solo. Lambda polls it natively. ECS tasks batch receive 10 at a time. Connect to EventBridge for fanout. All without custom polling loops.

Common trap:

Forgetting idempotency. Same message ID might hit your handler twice. Check message deduplication ID on receive. Delete only after your business logic commits.

Next time CloudWatch alarms on queue depth, you know the fix: Scale consumers, not producers. Add DLQ. Switch to long polling (20 seconds). Costs drop 40%. Reliability climbs.

Lego City: How to Build Anything Without It Falling Over

Yashvi Kothari — Sat, 20 Dec 2025 12:38:04 +0000

Modern software is too complex to be managed by human memory alone. If you want to build a system that can book tickets to Mars, you don’t need more developers—you need a better factory.

DevOps is the art of taking an idea from your brain to a user’s hand without human error getting in the way. If you want to explain this to a child (or a CEO), don't talk about "kernels" and "runtimes." Talk about Legos.

Magic Lego Factory
Imagine you have a great idea for a Lego castle.

The Workshop (Development): You build it on your bedroom floor. It looks great, but if you shut down your laptop (turn off the lights), no one can see it.

Museum (Production): You want to show it to the world, so you move it to a special display case in the city square that stays lit 24/7.

Instructions (Git): Your friends want to help. If you all grab the same bricks at once, you’ll fight. So, you use Git. It helps everyone work on the same castle at the same time and resolve conflicts efficiently.

Robot Builder (CI/CD): Moving the castle piece-by-piece to the museum is slow and you might drop a brick. You build a Robot (Jenkins/GitHub Actions). Every time you finish a new tower, the robot automatically builds it, tests it, and puts it in the museum for you.

Magic Boxes (Docker): Sometimes a brick fits at home but falls off at the museum because the table is different. You put your set inside a Clear Plastic Box (Container). If it works inside the box at home, it will work inside the box anywhere.

City Architect (Kubernetes): Now you have 1,000 boxes. You need a Super-Manager to stack them, replace broken ones, and add more tables when more people come to watch.

DevOps Lifecycle

In the real world, we aren't moving plastic bricks; we are moving code.

Version Control: The Source of Truth

Tools: Git, GitHub, GitLab.

The Reality: Developers write code in text format (VS Code/PyCharm) and push it to a central hub. Git enables versioning and collaboration so teams don't step on each other's toes.

CI/CD: The Automation Engine

Tools: Jenkins, GitHub Actions, GitLab CI/CD.

The Reality: "Building" converts text code into an executable or binary. CI/CD pipelines automate the manual tasks of pulling code, building it, testing it for bugs, and deploying it to production. This allows you to ship features faster with less manual effort.

Containerization: The Shipping Container

Tools: Docker.

The Reality: Applications need specific libraries and runtimes (Python, Java) to run. Docker packages the app and these dependencies into an "image". This ensures the software behaves the same on a developer's laptop as it does on a production server.

Orchestration: The Fleet Manager

Tools: Kubernetes.

The Reality: When users increase, you need more "containers". Kubernetes manages these instances, ensures they are healthy, and auto-scales the underlying infrastructure based on need.

Infrastructure as Code (IaC): The Blueprint

Tools: Terraform, Ansible.

The Reality: Setting up servers manually via a cloud GUI leads to human error. Terraform allows you to define virtual machines and storage in a "manifest file"—treating your infrastructure exactly like code. Ansible then handles post-configuration, like installing specific software on those servers.

Observability: The Vital Signs

Tools: Prometheus, Grafana.

The Reality: You must monitor CPU utilization and memory consumption to take preventive measures. Prometheus collects these metrics, and Grafana visualizes them into charts and graphs so you can make sense of the data.

They Laughed When I Sat Down At My Laptop;But Then My Code Deployed Itself." The secret to high-quality software isn't more hours. It's better systems.

Don't tell your users you have a 'highly-available microservices architecture.' Tell them you have a system so reliable, they can book a ticket to Mars while you're asleep.

KalaKraft Workplace Office in CSS Art Form

Yashvi Kothari — Sun, 27 Jul 2025 12:24:11 +0000

This is a submission for Frontend Challenge: Office Edition, CSS Art: Office Culture.

Inspiration

You don’t remember offices by the shape of their chairs.You remember because the stories whispered at the desk, the streak of sunlight at 4PM, and the hum of teamwork in the background.
Why I Code my dream Office?

Once I started with backend engineering,I started seeing CSS as a basic tool—a way to make buttons blue and borders round.

The Idea

Where most see “code vs. art,” office culture is both:- a living, breathing web of personalities behind every semi-colon.

Later I thought to experiment with AI prompts and then I realised for Frontend human creativity and optimization at depth still matters a lot.

We spend half our waking hours in offices, virtual and real. Why not make them art?
So tried what if I could turn it into a paintbrush, capturing the messy magic of real work and real relationships with AI prompting?

Inside every line of code, there’s a story waiting to be drawn.

Any Office is den of personalities:

1.keyboard warrior type

2.Team/Group at the coffee machine for gossip after their daily scrum.

3.Desk Decor type with motivation or sticky notes with work item list.

<html> </html>

I tried to make imagination live today with software engineers, a gifting segment, poets, and a content team, the environment and dynamics would bring a vibrant mix of technology, creativity.

listed itemwise ideas to pick:-

Phasewise

Demo

With AI prompting,there were headaches—pixels misaligned, z-index mishaps, creative blocks. But that’s office life too, right? Every little bug became a new “war story” to share.

Idea:-

Live Link:
https://yashvikothari.github.io/kalakraft_officeview_css

Source Code:

https://github.com/yashvikothari/yashvikothari.github.io/tree/master/kalakraft_officeview_css

Journey

Art isn’t just what you see. It’s what you feel when you know you’re part of something bigger—a team, a story, a culture.

In a world moving remote and asynchronous, code can be more than code.

It can be a snapshot of our office tribes, a tribute to our rituals, a call to turn every day at the desk into something worth remembering.

What detail of office life would YOU code?

Feel free to comment below, remix my experiment, or share your own story—because in the end, office culture is always a work in progress.

Amazon Q in Action ! Pacman Inspired game deployed

Yashvi Kothari — Fri, 27 Jun 2025 18:19:02 +0000

Building a Pac-Man Inspired Game with Amazon Q CLI

By Yashvi Kothari

Published: June 27, 2024

Introduction

amazon q

It turns out a simple, 200-line Pac-Man demo into a full-fledged, 2000+ line game using Amazon Q CLI. Along the way, I learned how AI can speed up development, sharpen prompting skills, and shape the future of coding.

Why Pac-Man?

Inspiration: https://community.aws/content/2y6egGcPAGQs8EwtQUM9KAONojz/build-games-challenge-build-classics-with-amazon-q-developer-cli?trk=b3ed9c83-eb20-4f68-b5b4-ffdc878e85c6&sc_channel=em&bb=237784

https://community.aws/content/2vlITBGRfv8slpeU1UlTrpT4bBI/vibe-coding-in-practice-building-a-super-mario-game-with-amazon-q-developer-cli

Complexity in layers—from basic movement to ghost AI.
Real-time needs: 60 FPS, collision checks, responsive controls.
Familiar rules make it easy to judge quality.
Clear specs help shape prompts.

Core Challenges

Rendering with HTML5 Canvas
Collision detection (walls, ghosts, dots)
Ghost pathfinding and behaviors
State management (levels, lives, scores)
Audio via Web Audio API
Performance tuning

Mastering AI Prompting

https://youtube.com/shorts/ALgU6YR8Kkc

From Broad to Precise

Too vague:

"Create a Pac-Man game"

→ Minimal output
More detail:

   "Make an 800×440 Canvas game with:
   - Grid movement
   - Wall collisions
   - Dot collection
   - Score tracking"

→ Working prototype

Contextual ask:

   "Add four ghost AIs:
   - Blinky (chases Pac-Man)
   - Pinky (ambushes ahead)
   - Inky (patrols)
   - Clyde (random)
   Ensure proper pathfinding."

→ Distinct ghost behaviors

Prompting Tips

Iterate: Build feature by feature.
Keep context: Refer to earlier code.
Break problems down: Solve movement → input buffering → corner turns.
Be specific: Provide dimensions, rates, algorithms.

Tackling Key Problems

Youtube Video | Hindi-ENglish (Hinglish)

https://www.youtube.com/watch?v=iMuAm4wQ2vE

Working as per AI but not even ready for local/test setup

try to first verify what commands fall under root permissions
verify port is up & running

Collision Detection

AI generated both:

function checkCollision(a, b) { … }    // Circle vs. circle
function checkWallCollision(e, dir) { … }  // Grid-based

Ghost AI Behaviors

State machine per personality:

switch (ghost.type) {
  case 'aggressive': target = pacman.pos; break;
  case 'ambush':    target = ambushPos(); break;
  …
}
ghost.pathfind();

Performance

Object pooling for particles
Dirty-rectangle rendering
Frame-limited loop with requestAnimationFrame

Automating Development

Tests

AI wrote unit and integration tests:

test('collect dot', () => { … });
test('ghost collision', () => { … });

Docs & Assets

Generated API docs and CSS animations:

@keyframes pacman-chomp { … }

When AI Surprises

Audio System

Full Web Audio API class without prompt:

class AudioSystem {
  generateTone(freq, dur) { … }
  play('dot');
}

Input Buffering

Handles rapid inputs and old entries:

inputBuffer.push({dir, time});
processBuffer();

Testing & Bug Fixes

Manual: Unit → integration → UX tests.
Automated: AI-generated test suites.
Bug fixes:
- Hidden loading screen
- Pause button state issues

Final Product

Code: 2000+ lines, 15+ files
Features: Ghost AI, particle effects, achievements, mobile support
Performance: Steady 60 FPS on major browsers
Deployment: Live on GitHub Pages

Live URLs (After Deployment):**
• Main Game: https://yashvikothari.github.io/pacman
• Original Version: https://yashvikothari.github.io/pacman/versions/v0.1/
• Working Version: https://yashvikothari.github.io/pacman/versions/v2.0.1-working/
• Flexible Version: https://yashvikothari.github.io/pacman/versions/v2.0.1-flexible/

Audio & other version is not deployed here !

Lessons Learned

AI multiplies, not replaces, your skills.
Prompting is an art—detail and context matter.
AI handles boilerplate, tests, docs well.
Human oversight is key for architecture, UX, and performance.

What I’d Do Differently next time

Better upfront architecture planning
Test-driven prompts
Smaller, frequent commits with feature branches
Living documentation alongside code
integrate with git first

Recommendation

Yes, for prototyping, learning, and solo projects—if you:

Practice prompting
Understand generated code
Review performance
Pair AI with your own creativity

Conclusion

This Pac-Man project shows how AI can help you go from a rough demo to a polished game in hours. Treat AI as a coding partner, iterate often, and combine your creativity with AI efficiency to build amazing things faster than ever.

Using Pulumi IaC to deploy NextJs static website with AWS S3 and EC2

Yashvi Kothari — Sat, 05 Apr 2025 19:28:35 +0000

This is a submission for the Pulumi Deploy and Document Challenge: Fast Static Website Deployment

Reference: https://www.pulumi.com/templates/static-website/aws/

What I Built

Static Website Deployment Next JS Framework publish to s3 and ec2using Pulumi IaC

Live Demo Link

NA

Before

After

Project Repo

NA

My Journey

I had face issue related to my hardening envt and restriction based sandbox due to security reasons normal production replica:-
1.aws linux 2 flavor with no option to migrate to latest(legacy servers)
2.not using latest nextjs,ts,npm,node version(legacy codebase)
3.unable to create new ec2
4.having s3 public access blocked
5.IAM limited scope and putpolicy for s3 not given

Using Pulumi

install pulumi

verify pulumi

verify pulumi specific version

configure Pulumi to use a local backend for state

before launching pulumi check node and npm version

Amazon Linux EC2 /RHEL Based

curl -fsSL https://rpm.nodesource.com/setup_16.x | sudo bash -
sudo yum install -y nodejs

now check node -version (i came across this while troubleshooting)

create new directory

only in empty directory pulumi project can be created
pulumi new aws-typescript
now enter values one by one

project name: webserver
project description: Yashvi Deploying on AWS using Pulumi IaC
stack name: Press enter to accept the default dev
passphrase to protect config/secrets: Enter Secret123 (you will be prompted to enter this two times)
aws:region: us-west-2

Pulumi installed dependencies and Now project ready

Now Pulumi to use the IAM role attached to your EC2 instance for AWS authentication

pulumi config set aws:skipMetadataApiCheck false

You can Refer to the [AWS Classic Pulumi package documentation ]

Now to avoid repeated passphrase prompts in Pulumi, set a Bash environment variable:

Now to retrieve the public IP address of EC2 instance, execute

curl https://checkip.amazonaws.com

Now, Open a new browser tab and paste the following URL into the address bar, replacing with the IP address 54.218.250.5

http://54.218.250.5:3000

Trying to add website content and IaC for creating s3

Now by default

As we intent Static Website Deployment as per our project submission so i can have nextjs demo app live

Replace this part in index.ts

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";

// Create an AWS resource (S3 Bucket)
const bucket = new aws.s3.BucketV2("my-bucket");

// Export the name of the bucket
export const bucketName = bucket.id;

with

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as fs from "fs";
import * as path from "path";

// 1. Build Next.js app (generates 'out' folder)
const nextjsBuildDir = path.join(process.cwd(), "../nextjs-app");
if (!fs.existsSync(nextjsBuildDir)) {
    throw new Error("Next.js app not found at ../nextjs-app");
}
// Simulate build process (run this manually first)
// pulumi.log.info("Run 'npm run build' in your Next.js app first!");

// 2. Create S3 bucket with website hosting
const siteBucket = new aws.s3.Bucket("nextjs-site", {
    website: {
        indexDocument: "index.html",
        errorDocument: "404.html",
    },
    acl: "public-read",
});

// 3. Upload static export files from Next.js
const outDir = path.join(nextjsBuildDir, "out");
if (fs.existsSync(outDir)) {
    const uploadFile = (filePath: string) => {
        const relativePath = path.relative(outDir, filePath);
        new aws.s3.BucketObject(relativePath, {
            bucket: siteBucket.id,
            source: new pulumi.asset.FileAsset(filePath),
            contentType: getContentType(filePath),
            acl: "public-read",
        });
    };

    // Recursively upload files
    const walkDir = (dir: string) => {
        fs.readdirSync(dir).forEach(file => {
            const fullPath = path.join(dir, file);
            if (fs.statSync(fullPath).isDirectory()) {
                walkDir(fullPath);
            } else {
                uploadFile(fullPath);
            }
        });
    };
    walkDir(outDir);
} else {
    // Fallback demo content if Next.js app isn't built
    new aws.s3.BucketObject("index.html", {
        bucket: siteBucket.id,
        content: "<h1>Next.js Static Site (Demo)</h1><p>Run 'npm run build' in your Next.js app!</p>",
        contentType: "text/html",
        acl: "public-read",
    });
}

// Helper to detect MIME types
function getContentType(filePath: string): string {
    const ext = path.extname(filePath);
    switch (ext) {
        case ".html": return "text/html";
        case ".css": return "text/css";
        case ".js": return "application/javascript";
        case ".json": return "application/json";
        case ".png": return "image/png";
        case ".jpg": case ".jpeg": return "image/jpeg";
        default: return "application/octet-stream";
    }
}

// 4. Export URLs
export const websiteUrl = siteBucket.websiteEndpoint;

Structure (making nexjs app folder sibling one)

Pulumi code in webserver/index.ts should correctly reference the sibling ../nextjs-app folder is must

I tried for Pulumi IaC for S3 and for EC2 (legacy system and troubleshooting :) )

pulumi up

keep doing yes by y

Now bucket website URL is final output of website deployment

WeCoded Landing Page Celebrate in Code

Yashvi Kothari — Mon, 31 Mar 2025 13:56:03 +0000

This is a submission for the WeCoded Challenge: Celebrate in Code

My WeCoded Landing Page

Demo

https://wecodedyashvi.netlify.app/

How I Built It

Used DEV Article API, fetched tag to achieve WeCoded stories and SheCoded Stories and created sorting and filter functionality.

Source Code

https://github.com/yashvikothari/wecoded_page

EC2 Key Pair Generate

Yashvi Kothari — Mon, 17 Mar 2025 17:56:50 +0000

Activity: Create Key Pair

ABC Company is in the process of migrating part of their infrastructure to AWS.
1.Break down migration into smaller tasks,
2.Align as per Company ABC to maintain control and optimize resources effectively.
Manager be like make sure Yashvi ABC have smooth transition with minimal disruptions in phase rollout wise manner.

KEY_PAIR Basics

Imagine Alice and Bob are two friends who want to send secret messages to each other, but they don’t trust the postal system to keep their messages private. So, they come up with a lock-and-key system:

Public Key = Open Lock

(Lock A or Lock B)

Alice and Bob each have their own unique lock (public key) that they share with everyone.
These locks are special: once locked, they can only be unlocked with the matching private key.

Private Key = Secret Key

(Lock B or Lock A’s Key)

Alice and Bob each keep a private key that can open only their own lock.
This private key must never be shared.

How Messages are Sent Securely

Bob wants to send Alice a secret message.
He takes Alice’s open lock (public key) and locks a box containing the message.
Once locked, only Alice’s private key can unlock and read the message.
Similarly, Alice can use Bob’s public lock to send him a secure message.

How Digital Signatures Work

Imagine Alice wants to prove she sent a message.
Instead of locking the message, she locks a signature using her private key.
Bob can verify the authenticity of the signature by using Alice’s public lock (public key).
If the lock opens, he knows the message really came from Alice!

Requirements:

Key Pair Name: given as per naming convention in ABC Company
Key Pair Type: rsa
Region: us-east-1

Yashvi will collect AWS Console credentials and 12 digit account number/login link

Method1: UI Console steps after using credentials to login:

1.Navigate to EC2 Dashboard:

2.Ensure you are in the us-east-1 region(as per architecture requirement).

3.Create Key Pair:

Select Key Pairs from the left-hand menu under Network & Security.
Click on Create Key Pair.
Enter the Key Pair Name as givenname-kp.
Have rsa as the Key Pair Type.
Click on Create Key Pair.

4.Download Key Pair:
-The key pair givenname-kp.pem will be automatically downloaded to your local machine.
-Store this key pair securely as it is required for SSH access to your instances.

You don’t understand anything until you learn it more than one way.

– Marvin Minsky

** Method 2: Have it simple na Yashvi with CLI ? not fan of UI/Click Ops
**

Here's the catch -prerequistes:

IAM user (yashvi-kothari-1234) should have permission around create EC2 key pairs.

Sample policy is attached to user yashvi-kothari-1234

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:CreateKeyPair", "ec2:DescribeKeyPairs" ], "Resource": "*" } ] }
2.Install AWS CLI on machine

macOS/Linux:
download package using curl command & run/install

Windows:
https://awscli.amazonaws.com/AWSCLIV2.msi

Verify Installation:
aws --version

4.Now before configuring AWS CLI,generate creds for AWS CLI

1.Create a new user with programmatic access, or use your own or request existing user to generate access keys.
2.Navigate to Users, select the user (here yashvi exists) or create a new one for cli, and then go to the Security credentials tab.
3.Under Access keys, click on Create access key and note down the Access Key ID and Secret Access Key.

5.Configure AWS CLI
Open your terminal and configure the AWS CLI using the generated credentials.
aws configure

Now enter in prompt

6.Key-Pair using command:

aws ec2 create-key-pair --key-name givenname-kp --key-type rsa --region us-east-1 --query 'KeyMaterial' --output text > givenname-kp.pem

Short Summary

AWS S3 System Design Concepts

Yashvi Kothari — Sun, 12 Jan 2025 19:10:23 +0000

AWS S3 (Simple Storage Service) is a cornerstone of cloud storage, offering a vast, scalable, and highly durable object storage service. This deep dive will explore the system design considerations, key components, and trade-offs involved in building a system like S3.

Object Store

High-Level Design (HLD)

Stores data as **objects (key-value pairs)** where the key is the object's unique identifier (e.g., "image.jpg") and the value is the actual data.
Provides a **flat namespace** within a bucket.
Supports **metadata** associated with each object.
Highly scalable and designed for **large datasets**.

Low-Level Design (LLD)

**Metadata Storage:**
- **Consistent Hashing** (e.g., Consistent Hashing) to distribute metadata across multiple servers for high availability and scalability.
- **Replicate metadata** across multiple availability zones for fault tolerance.
- Use a distributed database (like **Cassandra** or **DynamoDB**) for efficient metadata storage and retrieval.
**Object Storage:**
- Store object data in **chunks** across multiple servers within an availability zone.
- Utilize **erasure coding techniques** (like Reed-Solomon) to provide data redundancy and fault tolerance.
- Implement efficient **data placement algorithms** to optimize read/write performance and minimize data transfer.

File Store

High-Level Design (HLD)

Stores data in a **hierarchical structure** (directories and files) similar to a traditional file system.
Supports operations like create, read, write, delete, and move files and directories.
Provides a more familiar interface for users accustomed to file systems.

Low-Level Design (LLD)

**Metadata Storage:**
- Utilize a distributed file system (like **HDFS**) to store metadata (file names, directories, permissions).
- Implement a **metadata server** to handle metadata operations and maintain data consistency.
**Data Storage:**
- Store data in chunks across multiple servers.
- Implement **data replication** and **fault tolerance mechanisms**.

Block Store

High-Level Design (HLD)

Stores data as a collection of **blocks** (fixed-size units of data).
Provides low-level storage abstraction for building higher-level storage services (e.g., file systems, databases).
Offers high performance for random read/write operations.

Low-Level Design (LLD)

**Data Storage:**
- Divide the storage into logical units (e.g., 4KB blocks).
- Assign each block to a specific storage device (e.g., **SSD**, **HDD**) based on performance and cost requirements.
- Implement **data striping** and **replication** across multiple devices for fault tolerance and performance.

AWS S3: A Deeper Dive

**Bucket:** A fundamental unit of storage in S3. Each bucket has a globally unique name.
**Object:** A data unit within a bucket. Objects can be any type of data (images, videos, documents, etc.).
**URI:** A unique identifier for an object within S3 (e.g., `s3://bucket-name/object-key`).
**Durability:** S3 offers industry-leading durability (99.999999999%) with data replicated across multiple availability zones.
**Availability:** S3 provides high availability with multiple availability zones and redundant infrastructure.

AWS Ecosystem

S3 seamlessly integrates with other AWS services, such as:

**EC2:** For running applications that interact with S3.
**Lambda:** For serverless functions that process data stored in S3.
**Glacier:** For archiving infrequently accessed data.
**EBS:** For persistent storage for EC2 instances.

Monitor Your Static App memory usage EC2 Instances with Prometheus and Grafana

Yashvi Kothari — Sun, 12 Jan 2025 18:57:30 +0000

This blog post will guide you through setting up Prometheus and Grafana on a launched Ubuntu EC2 instance to monitor CPU and memory utilization metrics, and then visualize them in Grafana dashboards for effective resource allocation / optimization.

Prerequisites:

1.An AWS account with an EC2 instance launched and running Ubuntu.
2.SSH access to your EC2 instance with a PEM key file.

🔍 Objective:

1.Configure Prometheus to scrape metrics from EC2 instances

2.Set up Grafana to visualize CPU and memory utilization metrics

3.Create dashboards in Grafana for monitoring and analysis

4.Optimize resource allocation based on monitoring data

Step 1: Launch an Ubuntu EC2 Instance

Log in to your AWS Management Console and navigate to the EC2 service.
Click on "Launch Instance".
Choose an Ubuntu AMI (Amazon Machine Image).
Select an appropriate instance type based on your requirements.
Configure instance details like storage, security group, and networking.
Launch the instance.

Step 2: Install and Configure Prometheus

Connect to your EC2 instance using SSH with your PEM key file.

ssh -i "yashvikotharivm1_pem_key.pem" ubuntu@<your_public_ip_address.awsamazon.com>
`

Update the package lists:

sudo apt update

Install Prometheus:
sudo apt install prometheus

Edit the Prometheus configuration file (/etc/prometheus/prometheus.yml):

sudo nano /etc/prometheus/prometheus.yml
Add the following configuration snippet to scrape metrics from NodeExporter (a Prometheus exporter that collects system metrics):

YAML

scrape_configs:
  - job_name: node_exporter
    static_configs:
      - targets: ["localhost:9101"]

Note: The default port for Prometheus scraping targets is 9090, but NodeExporter uses a different port (9101) to avoid conflicts.

Save the changes and restart Prometheus:
sudo systemctl restart prometheus
Step 3: Install and Configure Grafana`

Install Grafana:

sudo apt install grafana
Configure security groups to allow inbound traffic on port 3000 (Grafana's default web interface port).

Restart Grafana:
sudo systemctl restart grafanaAccess the Grafana web interface:
http://:3000

Step 4: Add Prometheus as a Data Source in Grafana

Log in to Grafana using the default credentials (username: admin, password: admin).
Navigate to "Configuration" > "Data Sources".
Click "Add data source".
Select "Prometheus" as the type.
Enter the URL of your Prometheus instance (http://localhost:9090) in the "URL" field.
Click "Save & Test" to establish the connection.
Step 5: Create a Dashboard in Grafana

Click on "Dashboards" and then "New dashboard".
Give your dashboard a name (e.g., "EC2 Instance Monitoring").
Click "Add panel" and select "Graph" as the panel type.
In the "Title" field, enter "CPU Usage".
In the "Metrics" tab, select "Prometheus" as the data source.
In the "Query" box, paste the following query to visualize CPU usage:
rate(node_exporter_cpu_usage{job="node_exporter"}[1m])
Click "Save" to save the panel.

Repeat steps 5-7 to create additional panels for memory utilization or other metrics you want to monitor.

You can customize the panels with different visualizations (e.g., line graphs, heatmaps) and adjust the time range to view

More details available as below in github
(Complete Live project notes will be provided soon):-

IAM Deep Dive: AWS Security SCS-C02 Exam Prep 1

Yashvi Kothari — Sun, 29 Sep 2024 20:10:10 +0000

1. What is Identity and Access Management?

**
A service/system that protects your valuable AWS resources.

Definition IAM

who can access your AWS account
what they can do.

IAM itself

Identity Management:

who can access your AWS account.
IAM uses unique usernames to identify individuals within your account, preventing duplicate user accounts.

Authentication:

identified user is who they claim to be.
involves providing a username and password. OR also use Multi-Factor Authentication (MFA) for extra security.

Access Management

what resources an authenticated user can access.
grant granular permissions, such as "Full Access" to EC2 or "Read Only" to RDS.

Access Control

method used to grant access:

username/password
Traditional method
Basic security
While effective for simple setups, it's vulnerable to password breaches.
MFA

MFA significantly reduces the risk of unauthorized access, even if credentials are compromised.

Time-based One-Time Password (TOTP): Generates a unique code that expires after a short time.
Push notifications: Sends a notification to a registered device, requiring user confirmation.
Hardware tokens: Physical devices that generate unique codes.

federated access

simplifies user management and provides seamless login experience.
use security measures of the external identity provider.
External identity provider: Allows users outside your AWS account to access resources using credentials from a trusted external identity provider.

Single sign-on (SSO): Enables users to log in to multiple applications with a single set of credentials.
Social login: Allows users to sign in using their existing accounts from social media platforms (e.g., Google, Facebook).
Enterprise identity providers: Integrates with your organization's existing identity management systems (e.g., Active Directory)

Why IAM ?

Minimize risks: Restrict access to resources, preventing unauthorized individuals from causing damage.
Enhance compliance: Meet industry standards and regulations by adhering to secure access protocols.
Improve manageability: Simplify user access and resource management within your AWS account.

2.AWS IAM Features

Access Management:

Users:
individual identities (people or applications) needing access to AWS resources.
Each user has a unique ARN (Amazon Resource Name).
User have Multi-Factor Authentication (MFA) for enhanced security.
IAM User Groups:

add IAM users.
Attached Policies to grant or deny access to resources.

Roles:

Temporary credentials used by users, other AWS services, or applications to access resources.
Roles don't have passwords but can be assumed by authorized identities.

Policies (JSON documents): Define what resources can be accessed (or denied) and by whom. Policies can be attached to users, groups, or roles.

Account Settings: Enforce password policies with minimum security requirements.
Security Token Service (STS):

Provides temporary, limited-privilege credentials for IAM and federated users.
Regional endpoints are available for lower latency.

Access Reports:

Access Analyzer:
Identifies policies granting access to resources from outside your trusted zone (e.g., cross-account access).
Helps identify potential security risks.

Credential Report:

Generates a CSV file listing all IAM users with details like last used date, password change history, and MFA status.

Organization Activity (for AWS Organizations users):

Shows service activity for the past year within an account or organizational unit (OU).
Identifies active users and services accessed.

Service Control Policies (SCPs):
Set boundaries for permissions across AWS accounts within Organizations. SCPs can override identity-based policies for stricter control.

Eg:
If a user has full access to S3, RDS, and EC2 through an IAM policy, but the Service Control Policy (SCP) denies access to S3, the user will only be able to access RDS and EC2. The SCP takes precedence and limits the maximum permissions allowed.