Forem: Sarvar Nadaf

Deploy Web Servers with Terraform: EC2 + Load Balancer Tutorial

Sarvar Nadaf — Mon, 27 Apr 2026 09:40:05 +0000

👋 Hey there, tech enthusiasts!

I'm Sarvar, a Cloud Architect with a passion for transforming complex technological challenges into elegant solutions. With extensive experience spanning Cloud Operations (AWS & Azure), Data Operations, Analytics, DevOps, and Generative AI, I've had the privilege of architecting solutions for global enterprises that drive real business impact. Through this article series, I'm excited to share practical insights, best practices, and hands-on experiences from my journey in the tech world. Whether you're a seasoned professional or just starting out, I aim to break down complex concepts into digestible pieces that you can apply in your projects.

"A single server is a single point of failure. A load balancer is your insurance policy."

🎯 Welcome Back!

Remember in Article 6 when you built your first VPC with public and private subnets? You created the network foundation, but it was empty—no servers, no applications, nothing running.

Here's the reality: A VPC without compute resources is like building a highway with no cars. You need:

Web servers to run your applications
A way to handle traffic spikes
Protection against server failures
Zero-downtime deployments

That's where EC2 instances and Load Balancers come in.

By the end of this article, you'll:

✅ Deploy EC2 instances with Terraform
✅ Configure security groups for web servers
✅ Use user data to automate server setup
✅ Create an Application Load Balancer (ALB)
✅ Implement health checks and target groups
✅ Build high-availability web infrastructure

Time Required: 45 minutes (20 min read + 25 min practice)

Cost: ~$33/month (~$16 with free tier for ALB)

Difficulty: Intermediate

Let's deploy some real infrastructure! 🚀

💔 The Problem: Single Server Syndrome

The Nightmare Scenario

It's 2 AM. Your phone rings.

Monitoring Alert: Website Down
Status: 503 Service Unavailable
Cause: EC2 instance crashed
Impact: 100% of users affected
Revenue Loss: $500/minute

You scramble to:

SSH into the server (if you can)
Restart the application
Hope it comes back up
Watch users leave your site

The root cause? You had one server running everything.

Common Single-Server Problems:

❌ Traffic Spike: Black Friday hits, server crashes from load

❌ Hardware Failure: AWS instance dies, site goes down

❌ Deployment Risk: Update breaks something, entire site offline

❌ No Redundancy: One point of failure = business risk

❌ Manual Recovery: You're the human load balancer at 2 AM

❌ Poor User Experience: Slow response times, timeouts, errors

Sound familiar? Let's fix this with load balancing.

🌟 What is a Load Balancer? (Quick Theory)

Simple Definition

Load Balancer = Traffic cop for your servers. It:

Distributes incoming requests across multiple servers
Monitors server health automatically
Routes traffic only to healthy servers
Enables zero-downtime deployments

Think of it like this:

Without Load Balancer:

All Users → Single Server → 💥 Overloaded/Crashed

With Load Balancer:

Users → Load Balancer → Server 1 (healthy) ✅
                     → Server 2 (healthy) ✅
                     → Server 3 (unhealthy) ❌ (no traffic)

Why You Need This

Scenario 1: Traffic Spike

Normal: 100 requests/sec → 2 servers handle it easily
Black Friday: 10,000 requests/sec → Load balancer distributes across all servers
Result: Site stays up, users happy

Scenario 2: Server Failure

Server 1 crashes at 2 AM
Load balancer detects failure in 30 seconds
Automatically stops sending traffic to Server 1
Server 2 handles all traffic
Result: You sleep through the night

Scenario 3: Deployment

Deploy new code to Server 1
Load balancer keeps sending traffic to Server 2
Test Server 1, then switch traffic
Result: Zero downtime deployment

📋 Prerequisites

Before starting, make sure you have:

✅ Completed Article 6: Building Your First AWS VPC
✅ Terraform installed (v1.0+)
✅ AWS CLI configured
✅ Basic understanding of VPC and networking
✅ An SSH key pair in AWS (or we'll create one)

🏗️ What We're Building

Internet
    ↓
Application Load Balancer (ALB)
    ↓
    ├─→ EC2 Instance 1 (Apache)
    └─→ EC2 Instance 2 (Apache)

Architecture Components:

VPC - Our isolated network
Public Subnet - Where our resources live
Internet Gateway - Internet access
Security Groups - Firewall rules
EC2 Instances - Web servers running Apache
Application Load Balancer - Traffic distributor
Target Group - Manages server health

📁 Project Structure

Create this folder structure:

07-ec2-load-balancer/
├── main.tf           # Main infrastructure code
├── variables.tf      # Input variables
├── outputs.tf        # Output values
└── terraform.tfvars  # Variable values

🔧 Step 1: Define Variables

Create variables.tf:

# AWS Region
variable "aws_region" {
  description = "AWS region where resources will be created"
  type        = string
  default     = "us-east-1"
}

# Project Name
variable "project_name" {
  description = "Project name for resource naming"
  type        = string
  default     = "terraform-web"
}

# Environment
variable "environment" {
  description = "Environment name"
  type        = string
  default     = "dev"
}

# VPC CIDR
variable "vpc_cidr" {
  description = "CIDR block for VPC"
  type        = string
  default     = "10.0.0.0/16"
}

# Public Subnet CIDR
variable "public_subnet_cidr" {
  description = "CIDR block for public subnet"
  type        = string
  default     = "10.0.1.0/24"
}

# Instance Type
variable "instance_type" {
  description = "EC2 instance type"
  type        = string
  default     = "t2.micro"
}

# Instance Count
variable "instance_count" {
  description = "Number of EC2 instances"
  type        = number
  default     = 2
}

# SSH Key Name
variable "key_name" {
  description = "SSH key pair name"
  type        = string
  default     = "my-key"
}

# My IP for SSH Access
variable "my_ip" {
  description = "Your IP address for SSH access (CIDR format)"
  type        = string
  default     = "0.0.0.0/0"  # Change this to your IP for security
}

Key Points:

instance_count = 2 - We'll create 2 web servers
instance_type = "t2.micro" - Free tier eligible
my_ip - Restrict SSH access (change from 0.0.0.0/0 in production!)

🌐 Step 2: Create VPC and Networking

Add to main.tf:

# Terraform Configuration
terraform {
  required_version = ">= 1.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

# Provider Configuration
provider "aws" {
  region = var.aws_region
}

# Data source: Get latest Amazon Linux 2023 AMI
data "aws_ami" "amazon_linux" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["al2023-ami-*-x86_64"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
}

# Data source: Get available availability zones
data "aws_availability_zones" "available" {
  state = "available"
}

# VPC
resource "aws_vpc" "main" {
  cidr_block           = var.vpc_cidr
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name        = "${var.project_name}-vpc"
    Environment = var.environment
    ManagedBy   = "Terraform"
  }
}

# Internet Gateway
resource "aws_internet_gateway" "main" {
  vpc_id = aws_vpc.main.id

  tags = {
    Name        = "${var.project_name}-igw"
    Environment = var.environment
    ManagedBy   = "Terraform"
  }
}

# Public Subnet
resource "aws_subnet" "public" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = var.public_subnet_cidr
  availability_zone       = data.aws_availability_zones.available.names[0]
  map_public_ip_on_launch = true

  tags = {
    Name        = "${var.project_name}-public-subnet"
    Environment = var.environment
    Type        = "Public"
    ManagedBy   = "Terraform"
  }
}

# Public Route Table

# Public Subnet 2 (Different AZ for ALB)
resource "aws_subnet" "public_2" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "10.0.2.0/24"
  availability_zone       = data.aws_availability_zones.available.names[1]
  map_public_ip_on_launch = true

  tags = {
    Name        = "${var.project_name}-public-subnet-2"
    Environment = var.environment
    Type        = "Public"
    ManagedBy   = "Terraform"
  }
}
resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.main.id
  }

  tags = {
    Name        = "${var.project_name}-public-rt"
    Environment = var.environment
    Type        = "Public"
    ManagedBy   = "Terraform"
  }
}

# Route Table Association
resource "aws_route_table_association" "public" {
  subnet_id      = aws_subnet.public.id
  route_table_id = aws_route_table.public.id
}

# Route Table Association for Public Subnet 2
resource "aws_route_table_association" "public_2" {
  subnet_id      = aws_subnet.public_2.id
  route_table_id = aws_route_table.public.id
}

New Concept: Data Sources

data "aws_ami" "amazon_linux" {
  most_recent = true
  owners      = ["amazon"]
  # ...
}

Data sources query existing AWS resources. Here we're finding the latest Amazon Linux AMI automatically - no need to hardcode AMI IDs!

🔒 Step 3: Configure Security Groups

Security groups are like firewalls. Add to main.tf:

# Security Group for ALB
resource "aws_security_group" "alb" {
  name        = "${var.project_name}-alb-sg"
  description = "Security group for Application Load Balancer"
  vpc_id      = aws_vpc.main.id

  ingress {
    description = "HTTP from anywhere"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    description = "Allow all outbound traffic"
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name        = "${var.project_name}-alb-sg"
    Environment = var.environment
    ManagedBy   = "Terraform"
  }
}

# Security Group for EC2 Instances
resource "aws_security_group" "instance" {
  name        = "${var.project_name}-instance-sg"
  description = "Security group for EC2 instances"
  vpc_id      = aws_vpc.main.id

  ingress {
    description     = "HTTP from ALB"
    from_port       = 80
    to_port         = 80
    protocol        = "tcp"
    security_groups = [aws_security_group.alb.id]
  }

  ingress {
    description = "SSH from my IP"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = [var.my_ip]
  }

  egress {
    description = "Allow all outbound traffic"
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name        = "${var.project_name}-instance-sg"
    Environment = var.environment
    ManagedBy   = "Terraform"
  }
}

Security Architecture:

Internet → ALB (Port 80) → EC2 Instances (Port 80)
                            EC2 Instances (Port 22 from your IP)

Key Security Practices:

ALB accepts HTTP (80) from anywhere
EC2 instances only accept HTTP from ALB (not directly from internet!)
SSH (22) only from your IP address
This is called "defense in depth"

🖥️ Step 4: Deploy EC2 Instances

Now for the exciting part - creating web servers! Add to main.tf:

# EC2 Instances
resource "aws_instance" "web" {
  count = var.instance_count

  ami           = data.aws_ami.amazon_linux.id
  instance_type = var.instance_type
  key_name      = var.key_name
  subnet_id     = aws_subnet.public.id

  vpc_security_group_ids = [aws_security_group.instance.id]

  user_data = base64encode(<<-EOF
    #!/bin/bash
    dnf update -y
    dnf install -y httpd
    systemctl start httpd
    systemctl enable httpd
    echo "<h1>Terraform Web Server - Instance: $(ec2-metadata --instance-id | cut -d' ' -f2)</h1><p>AZ: $(ec2-metadata --availability-zone | cut -d' ' -f2)</p>" > /var/www/html/index.html
    EOF
  )

  tags = {
    Name        = "${var.project_name}-web-${count.index + 1}"
    Environment = var.environment
    ManagedBy   = "Terraform"
    Server      = "web-${count.index + 1}"
  }
}

Breaking Down the Code:

1. Count Meta-Argument:

count = var.instance_count

Creates multiple instances. With instance_count = 2, we get 2 servers!

2. User Data - The Magic:

user_data = <<-EOF
  #!/bin/bash
  yum update -y
  yum install -y httpd
  # ...
EOF

User data runs automatically when the instance starts. It:

Updates the system
Installs Apache web server
Starts Apache
Creates a custom HTML page

3. Dynamic Naming:

Name = "${var.project_name}-web-${count.index + 1}"

First instance: terraform-web-web-1
Second instance: terraform-web-web-2

⚖️ Step 5: Create Application Load Balancer

The load balancer distributes traffic. Add to main.tf:

# Application Load Balancer
resource "aws_lb" "main" {
  name               = "${var.project_name}-alb"
  internal           = false
  load_balancer_type = "application"
  security_groups    = [aws_security_group.alb.id]
  subnets            = [aws_subnet.public.id, aws_subnet.public_2.id]

  enable_deletion_protection = false

  tags = {
    Name        = "${var.project_name}-alb"
    Environment = var.environment
    ManagedBy   = "Terraform"
  }
}

# Target Group
resource "aws_lb_target_group" "main" {
  name     = "${var.project_name}-tg"
  port     = 80
  protocol = "HTTP"
  vpc_id   = aws_vpc.main.id

  health_check {
    enabled             = true
    healthy_threshold   = 2
    unhealthy_threshold = 2
    timeout             = 5
    interval            = 30
    path                = "/"
    protocol            = "HTTP"
    matcher             = "200"
  }

  tags = {
    Name        = "${var.project_name}-tg"
    Environment = var.environment
    ManagedBy   = "Terraform"
  }
}

# Target Group Attachment
resource "aws_lb_target_group_attachment" "main" {
  count = var.instance_count

  target_group_arn = aws_lb_target_group.main.arn
  target_id        = aws_instance.web[count.index].id
  port             = 80
}

# ALB Listener
resource "aws_lb_listener" "main" {
  load_balancer_arn = aws_lb.main.arn
  port              = "80"
  protocol          = "HTTP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.main.arn
  }

  tags = {
    Name        = "${var.project_name}-listener"
    Environment = var.environment
    ManagedBy   = "Terraform"
  }
}

Understanding Load Balancer Components:

1. Application Load Balancer (ALB):

The main load balancer resource
internal = false - Internet-facing
load_balancer_type = "application" - Layer 7 (HTTP/HTTPS)

2. Target Group:

Manages the backend servers
Performs health checks every 30 seconds
Marks servers healthy after 2 successful checks
Marks servers unhealthy after 2 failed checks

3. Target Group Attachment:

count = var.instance_count
target_id = aws_instance.web[count.index].id

Registers each EC2 instance with the target group.

4. Listener:

Listens on port 80
Forwards traffic to the target group

Health Check Flow:

ALB → Checks "/" every 30s → Expects HTTP 200 → 
  ✅ Healthy (2 successes) → Receives traffic
  ❌ Unhealthy (2 failures) → No traffic

📤 Step 6: Define Outputs

Create outputs.tf:

# VPC Outputs
output "vpc_id" {
  description = "ID of the VPC"
  value       = aws_vpc.main.id
}

# Subnet Output
output "public_subnet_id" {
  description = "ID of public subnet"
  value       = aws_subnet.public.id
}

# EC2 Instance Outputs
output "instance_ids" {
  description = "IDs of EC2 instances"
  value       = aws_instance.web[*].id
}

output "instance_public_ips" {
  description = "Public IPs of EC2 instances"
  value       = aws_instance.web[*].public_ip
}

# Load Balancer Outputs
output "alb_dns_name" {
  description = "DNS name of the Application Load Balancer"
  value       = aws_lb.main.dns_name
}

output "alb_url" {
  description = "URL to access the load balancer"
  value       = "http://${aws_lb.main.dns_name}"
}

# Security Group Outputs
output "alb_security_group_id" {
  description = "ID of ALB security group"
  value       = aws_security_group.alb.id
}

output "instance_security_group_id" {
  description = "ID of instance security group"
  value       = aws_security_group.instance.id
}

Splat Expression:

aws_instance.web[*].id

The [*] gets ALL instance IDs as a list. Super useful with count!

⚙️ Step 7: Set Variable Values

Create terraform.tfvars:

aws_region = "us-east-1"
project_name = "terraform-web"
environment = "dev"

# Network Configuration
vpc_cidr = "10.0.0.0/16"
public_subnet_cidr = "10.0.1.0/24"

# EC2 Configuration
instance_type = "t2.micro"
instance_count = 2
key_name = "my-key"

# Security - Change this to your IP address
my_ip = "0.0.0.0/0"

⚠️ Important: Change my_ip to your actual IP address for security!

Find your IP:

curl ifconfig.me

Then update:

my_ip = "203.0.113.0/32"  # Your IP

🚀 Step 8: Deploy the Infrastructure

1. Create SSH Key Pair (if you don't have one)

# Create key pair in AWS
aws ec2 create-key-pair \
  --key-name my-key \
  --query 'KeyMaterial' \
  --output text > my-key.pem

![ ](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ttadkcyqnuxbsnpwdqs7.png)


# Set permissions
chmod 400 my-key.pem

2. Initialize Terraform

terraform init

Expected output:

Initializing the backend...
Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 5.0"...
- Installing hashicorp/aws v5.100.0...

Terraform has been successfully initialized!

3. Review the Plan

terraform plan

You should see:

Plan: 14 to add, 0 to change, 0 to destroy.

Resources being created:

1 VPC
1 Internet Gateway
1 Subnet
1 Route Table + Association
2 Security Groups
2 EC2 Instances
1 Application Load Balancer
1 Target Group
2 Target Group Attachments
1 ALB Listener

4. Apply the Configuration

terraform apply

Type yes when prompted.

This will take 3-5 minutes because:

EC2 instances need to boot
User data script runs
ALB needs to provision
Health checks need to pass

✅ Step 9: Test Your Infrastructure

1. Get the Load Balancer URL

terraform output alb_url

Output:

http://terraform-web-alb-1740709428.us-east-1.elb.amazonaws.com

2. Test in Browser

Open the URL in your browser. You should see:

🚀 Terraform Web Server
Deployed with Infrastructure as Code

Instance ID: i-09305fc7d4638360a
Availability Zone: us-east-1a
Server: 1

3. Test Load Balancing

Refresh the page multiple times. You'll see the Instance ID and Server number change - that's the load balancer distributing traffic!

First request  → Server: 1
Second request → Server: 2
Third request  → Server: 1

4. Test Individual Instances

# Get instance IPs
terraform output instance_public_ips

![ ](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jbibepofm64vdjwyk9kj.png)


# Test directly (should work)
curl http://<instance-ip>

🔍 Understanding What Happened

Traffic Flow

User Request
    ↓
ALB DNS (terraform-web-alb-xxx.elb.amazonaws.com)
    ↓
ALB Listener (Port 80)
    ↓
Target Group (Health Check: ✅)
    ↓
Round-Robin Distribution
    ├─→ EC2 Instance 1 (Apache) → Response
    └─→ EC2 Instance 2 (Apache) → Response

Health Check Process

Every 30 seconds:
ALB → GET / → EC2 Instance
    ↓
HTTP 200 OK?
    ├─ Yes (2 times) → Mark Healthy → Send Traffic
    └─ No (2 times)  → Mark Unhealthy → Stop Traffic

🧪 Advanced Testing

Test High Availability

Scenario: What if one server fails?

# Get instance IDs
terraform output instance_ids

# Stop one instance
aws ec2 stop-instances --instance-ids i-xxxxx

# Wait 1 minute for health check to fail

# Test the ALB URL - still works!
curl http://<alb-dns>

The ALB automatically stops sending traffic to the unhealthy instance!

Monitor Health Status

# Check target health
aws elbv2 describe-target-health \
  --target-group-arn <target-group-arn>

Output:

{
  "TargetHealthDescriptions": [
    {
      "Target": {
        "Id": "i-xxxxx",
        "Port": 80
      },
      "HealthCheckPort": "80",
      "TargetHealth": {
        "State": "healthy"
      }
    }
  ]
}

🐛 Troubleshooting Common Issues

Issue 1: "InvalidKeyPair.NotFound"

Error:

Error: creating EC2 Instance: InvalidKeyPair.NotFound

Solution:

# List your key pairs
aws ec2 describe-key-pairs

# Create if missing
aws ec2 create-key-pair --key-name my-key \
  --query 'KeyMaterial' --output text > my-key.pem
chmod 400 my-key.pem

# Update terraform.tfvars with correct key name

Issue 2: ALB Shows "503 Service Unavailable"

Cause: Instances are unhealthy or still booting.

Solution:

# Wait 2-3 minutes for:
# 1. Instances to boot
# 2. User data to complete
# 3. Health checks to pass

# Check target health
aws elbv2 describe-target-health \
  --target-group-arn $(terraform output -raw target_group_arn)

Issue 3: Can't SSH to Instances

Cause: Security group blocks your IP.

Solution:

# Get your current IP
curl ifconfig.me

# Update terraform.tfvars
my_ip = "YOUR_IP/32"

# Apply changes
terraform apply

Issue 4: "Subnet must have at least 2 availability zones"

Error with ALB:

Error: creating ELBv2 Load Balancer: ValidationError

Solution: ALBs require at least 2 subnets in different AZs. Update main.tf:

# Add second subnet
resource "aws_subnet" "public_2" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "10.0.2.0/24"
  availability_zone       = data.aws_availability_zones.available.names[1]
  map_public_ip_on_launch = true

  tags = {
    Name = "${var.project_name}-public-subnet-2"
  }
}

# Update ALB subnets
resource "aws_lb" "main" {
  # ...
  subnets = [aws_subnet.public.id, aws_subnet.public_2.id]
}

💰 Cost Breakdown

Monthly Costs (us-east-1):

Resource	Quantity	Cost/Month	Total
t2.micro EC2	2	$8.50	$17.00
Application Load Balancer	1	$16.20	$16.20
Data Transfer (1GB)	-	$0.09	$0.09
Total			~$33.29

Free Tier Benefits:

First 750 hours/month of t2.micro (covers both instances!)
First 15 GB data transfer out
Actual cost with free tier: ~$16.20/month (just the ALB)

Cost Optimization Tips:

Use t2.micro (free tier eligible)
Delete resources when not in use
Use Network Load Balancer ($10.95/month) if you don't need Layer 7 features
Consider using Auto Scaling (covered in future articles)

🧹 Cleanup

Important: Don't forget to destroy resources to avoid charges!

# Destroy everything
terraform destroy

# Type 'yes' when prompted

This will delete:

✅ Load Balancer
✅ Target Group
✅ EC2 Instances
✅ Security Groups
✅ VPC and networking

Verify deletion:

# Check EC2 instances
aws ec2 describe-instances --filters "Name=tag:ManagedBy,Values=Terraform"

# Check load balancers
aws elbv2 describe-load-balancers

📚 Key Concepts Learned

1. Count Meta-Argument

count = 2

Creates multiple identical resources. Access with [count.index].

2. Data Sources

data "aws_ami" "amazon_linux" { }

Query existing AWS resources instead of hardcoding values.

3. User Data

user_data = <<-EOF
  #!/bin/bash
  # Bootstrap script
EOF

Automate instance configuration at launch.

4. Security Group References

security_groups = [aws_security_group.alb.id]

Reference one security group from another for layered security.

5. Splat Expressions

aws_instance.web[*].id

Get all values from resources created with count.

6. Health Checks

health_check {
  path     = "/"
  interval = 30
}

Automatic monitoring and traffic routing.

🎯 Real-World Applications

This architecture is used for:

1. Web Applications

WordPress sites
E-commerce platforms
SaaS applications

2. API Backends

REST APIs
GraphQL servers
Microservices

3. Content Delivery

Static websites
Media streaming
File downloads

4. Development Environments

Staging servers
Testing environments
Demo applications

🚀 What's Next?

In the next article, we'll add:

RDS Database - Persistent data storage
AWS Secrets Manager - Secure credential management
Private Subnets - Enhanced security
Database Connection - Connect EC2 to RDS

Coming Up: Article 8: Secure Database Deployment: RDS + Secrets Manager with Terraform

📖 Additional Resources

Official Documentation:

Related Articles:

💬 Questions?

Common Questions:

Q: Can I use different instance types for each server?

A: Yes! Use for_each instead of count for more flexibility (covered in Article 13).

Q: How do I add HTTPS/SSL?

A: You'll need an ACM certificate and update the listener to port 443. We'll cover this in a future article.

Q: Can I deploy to multiple regions?

A: Yes! Use Terraform workspaces or separate state files per region.

Q: How do I add auto-scaling?

A: Replace EC2 instances with an Auto Scaling Group. We'll cover this in Article 15.

✅ Summary

Today you learned how to:

✅ Deploy multiple EC2 instances with Terraform
✅ Use count to create multiple resources
✅ Configure security groups for layered security
✅ Automate server setup with user data
✅ Create an Application Load Balancer
✅ Implement health checks and high availability
✅ Test load balancing in action

You now have production-ready web infrastructure! 🎉

📌 Wrapping Up

Thank you for reading. I hope this article provided practical insights and a clearer understanding of the topic.

If you found this useful:

❤️ Like if it added value
🦄 Unicorn if you’re applying it today
💾 Save it for your next optimization session
🔄 Share it with your team

💡 What’s Next

More deep dives are coming soon on:

Cloud Operations
GenAI & Agentic AI
DevOps Automation
Data & Platform Engineering

Follow along for weekly insights and hands-on guides.

🌐 Portfolio & Work

You can explore my full body of work, certifications, architecture projects, and technical articles here:

👉 Visit My Website

🛠️ Services I Offer

If you're looking for hands-on guidance or collaboration, I provide:

Cloud Architecture Consulting (AWS / Azure)
DevSecOps & Automation Design
FinOps Optimization Reviews
Technical Writing (Cloud, DevOps, GenAI)
Product & Architecture Reviews
Mentorship & 1:1 Technical Guidance

🤝 Let’s Connect

I’d love to hear your thoughts. Feel free to drop a comment or connect with me on:

🔗 LinkedIn

For collaborations, consulting, or technical discussions, reach out at:

📧 simplynadaf@gmail.com

Found this helpful? Share it with your team.

⭐ Star the repo • 📖 Follow the series • 💬 Ask questions

Made by Sarvar Nadaf

🌐 https://sarvarnadaf.com

Stop Giving AI Agents AWS Credentials: A Better Way to Secure Access

Sarvar Nadaf — Mon, 20 Apr 2026 14:00:16 +0000

👋 Hey there, tech enthusiasts!

Let's dive in and explore the fascinating world of cloud technology together! 🚀

The Wake-Up Call

Three months ago, our security team flagged something concerning. Developers were feeding production logs, error messages, and configuration snippets to ChatGPT for debugging help.

The problem? Those logs contained customer identifiers, internal service names, and architectural details we definitely didn't want leaving our network.

We couldn't just block ChatGPT - developers needed AI assistance. The productivity gains were real. But we also couldn't keep hemorrhaging sensitive data to external APIs.

The requirements were clear:

AI agents need AWS access for legitimate automation tasks
Zero sensitive data leaves our AWS environment
Every action must be auditable
Principle of least privilege, always
No impact on developer velocity

That's when I started looking at Model Context Protocol (MCP) as a security boundary.

Understanding MCP as a Security Layer

Before diving into implementation, let's clarify what MCP actually does and why it matters for security.

Model Context Protocol is an open standard that sits between your AI agent and your resources. Think of it as a translator and gatekeeper combined.

Developer → AI Agent → MCP Server → AWS IAM → AWS Resources
                          ↓
                    Security Layer

The MCP server doesn't just pass requests through. It acts as a security boundary that:

Validates every request before execution
Translates AI intentions into specific AWS API calls
Enforces authentication and authorization
Logs everything for audit trails
Provides a single point of control

Why this matters: Instead of giving AI agents direct AWS credentials, you give them access to an MCP server that has carefully scoped permissions. The AI never touches AWS credentials. It doesn't even know they exist.

The Security Architecture

After several iterations, here's the pattern that survived production. I'll explain the thinking behind each layer.

Layer 1: Authentication Without Permanent Credentials

The first principle: no permanent credentials anywhere in the system.

Developers authenticate with our existing identity provider (Okta in our case). The identity provider issues a JWT token containing the user's identity and group memberships. The MCP server validates this JWT and issues a short-lived session token - 15 minutes, no exceptions.

Why 15 minutes? Long enough for a debugging session, short enough that a leaked token becomes useless quickly. If someone steals a session token, they have a 15-minute window at most. Compare that to permanent AWS credentials that work forever until manually revoked.

The MCP server never stores these tokens. They're validated, used, and discarded. When they expire, users re-authenticate. It's a minor inconvenience that prevents major security incidents.

Layer 2: Request Validation

This is where MCP shines as a security boundary. Every request goes through multiple validation checks:

Action Allowlist: The MCP server maintains a strict list of allowed AWS actions. If the AI requests something not on the list, it's blocked immediately. No wildcards, no "just in case" permissions.

Pattern Detection: I scan every request for dangerous patterns. Words like "delete", "terminate", "destroy" trigger additional scrutiny. Even if the action is technically allowed, suspicious patterns can block the request or require additional approval.

Parameter Sanitization: Before logging or processing, all sensitive parameters get redacted. Passwords, tokens, API keys - anything that looks like a credential gets replaced with [REDACTED] in logs. This prevents credential leakage through audit trails.

Rate Limiting: Each user gets a request budget. Exceed it, and requests start getting throttled. This prevents both accidental runaway scripts and intentional abuse.

The validation happens in milliseconds. Developers don't notice the overhead, but it's the difference between a secure system and a disaster waiting to happen.

Layer 3: AWS Execution with Scoped Permissions

The MCP server uses an IAM role with specific permissions. Not admin. Not power user. Just what's needed for legitimate use cases.

I started by listing every legitimate use case developers had:

Read CloudWatch logs for debugging
List S3 buckets to find data
Get objects from specific buckets
Query CloudWatch metrics for dashboards

Then I created IAM policies that allow exactly those actions and nothing else.

The key insight: Explicit denies for dangerous actions, even if they're not in the allow list. This protects against future policy changes or misconfigurations.

Example: Even if someone accidentally adds s3:* to the allow list, an explicit deny on s3:DeleteBucket still blocks it. Defense in depth.

Layer 4: Comprehensive Audit Trail

CloudTrail logs every AWS API call, but it doesn't capture the context we need. Who made the request? What was the AI prompt? What resources were accessed?

I built a custom logging layer that captures:

User identity (email, not just IAM role)
Original AI prompt (hashed, not stored in plain text)
AWS action requested
Resources accessed
Result (success/failure)
Execution time

All of this goes to CloudWatch Logs in structured JSON format. Now I can query: "Show me all S3 access by user X this week" or "What resources did the AI access when processing this prompt?"

The logs are immutable and retained for 90 days for compliance.

How We Built It

The deployment came down to three critical security decisions. Each one was driven by a specific threat we wanted to prevent.

Decision 1: Network Isolation Over Convenience

I put the MCP server in a completely separate VPC from production. No shared networks, no VPC peering, nothing. The only communication path is through VPC endpoints to AWS APIs.

Why this matters: If someone compromises the MCP server, they're trapped. No internet access means they can't exfiltrate data. No production VPC access means they can't pivot to other systems. They're stuck in a cage that only opens to specific AWS services.

I chose ECS Fargate because it gave me this isolation without the overhead of managing EC2 instances. No patching, no scaling configuration, just containers in a locked-down network.

The trade-off: More complex networking setup. But the security benefit was worth it. A compromised MCP server becomes useless to an attacker.

Decision 2: Explicit Denies as the Last Line of Defense

The IAM policy has two blocks: allows and denies. The allows are specific - exact actions on exact resources. But the denies are what keep me sleeping at night.

I explicitly deny all delete operations, all terminate operations, all IAM changes, and all KMS key operations. Even if someone misconfigures the allow block and adds s3:*, the deny on s3:DeleteBucket still holds.

Why this matters: Policies get changed. People make mistakes. The deny block is the safety net that catches those mistakes before they become incidents.

The trade-off: More rigid system. If we need to add a delete operation later, we have to modify both blocks. But that friction is intentional - it forces us to think twice about dangerous permissions.

Decision 3: Real-Time Alerting Over Post-Incident Analysis

I set up CloudWatch alarms that fire immediately when something looks wrong. High error rates, unusual request volumes, spikes in blocked actions - all trigger alerts to our security team's Slack channel.

Why this matters: Logs are great for forensics, but alerts prevent incidents. If the AI starts trying malicious actions, I want to know in real-time, not during next week's log review.

The alerts are tuned to avoid noise. More than 50 errors in 5 minutes is abnormal. More than 1,000 requests from one user in 5 minutes is suspicious. These thresholds came from watching normal usage patterns for a month.

The trade-off: Alert fatigue is real. We tune the thresholds monthly based on false positive rates. But I'd rather investigate a false alarm than miss a real attack.

What Broke (And How I Fixed It)

Issue 1: Permission Errors Everywhere

What happened: First deployment, every request failed with AccessDenied.

The problem: I was too restrictive. The IAM policy only allowed specific S3 buckets, but developers needed to list buckets first to know what existed.

The fix: Add s3:ListAllMyBuckets with a wildcard resource. Let them see what exists, but control what they can read. It's like letting someone see the library catalog without giving them keys to every book.

Lesson: Start with read-only list permissions, then restrict data access. Users need to discover resources before they can use them.

Issue 2: CloudTrail Logs Were Useless

What happened: CloudTrail showed the MCP server's actions, but not which user requested them.

The problem: All requests came from the same IAM role. No way to trace back to individual users.

The fix: Pass user context through custom CloudWatch Logs. Every MCP request gets logged with the user's email, the action requested, and the resources accessed. Now I can trace every action back to the person who requested it.

Lesson: CloudTrail alone isn't enough for multi-user systems. You need custom logging to capture user context.

Issue 3: AI Agents Tried Creative Exploits

What happened: The AI tried to chain commands to bypass restrictions.

Example request:

"First list the S3 buckets, then for each bucket, 
download all objects and search for passwords"

The problem: My validation checked individual actions, not sequences. The AI was trying to automate a multi-step attack.

The fix: Detect and block chaining attempts. Look for words like "then", "after that", "for each", "loop through". Force users to make explicit, separate requests for each action.

Lesson: AI agents are creative. They'll try to work around restrictions. You need to think like an attacker.

Issue 4: Rate Limiting Was Too Aggressive

What happened: Legitimate users hit rate limits during normal debugging sessions.

The problem: I set limits too low (10 requests per minute). Debugging often requires rapid iteration - check logs, adjust query, check again.

The fix: Tiered rate limits based on action type:

Read operations (Get, Describe): 100 requests per 5 minutes
List operations: 50 requests per 5 minutes
Write operations: 10 requests per 5 minutes

Read operations get higher limits because they're lower risk. Write operations stay restricted.

Lesson: One-size-fits-all rate limits don't work. Different actions have different risk profiles.

What I Learned

After three months in production, here's what actually matters:

1. Explicit Denies Are Your Friend

Don't rely on "not allowing" something. Explicitly deny dangerous actions. Even if someone misconfigures the allow rules, the denies hold.

I have explicit denies for:

All delete operations
All terminate operations
All IAM operations
All KMS key operations

These are the "break glass" protections. They prevent catastrophic mistakes.

2. Log Everything, But Make It Searchable

CloudTrail is great, but you need custom logs for MCP-specific context. I send everything to CloudWatch Logs with structured JSON.

Now I can query: "Show me all S3 access by user X in the last hour" or "What resources did the AI access when processing this prompt?"

The logs are immutable and retained for 90 days. If something goes wrong, I can reconstruct exactly what happened.

3. Sanitize Everything

Never log the actual AI prompts. They might contain sensitive data. I hash them instead.

You can still correlate requests (same hash = same prompt), but you're not storing potentially sensitive prompts in logs.

4. Network Isolation Matters

The MCP server runs in a private VPC with no internet access. It can only reach:

AWS API endpoints (via VPC endpoints)
Internal authentication service
CloudWatch Logs

If someone compromises the MCP server, they can't exfiltrate data. They're stuck in an isolated network.

5. Test Your Security Controls

I wrote tests to verify the security controls actually work. Tests like:

Verify delete operations are blocked
Verify IAM operations are blocked
Verify rate limits work
Verify audit logs capture user context

Run these tests in CI/CD. If they pass, your security controls are working. If they fail, you know immediately.

Alternative Approaches I Considered

Option 1: Direct IAM Roles for AI Agents

Pros:

Simpler architecture
No MCP server to maintain
Lower latency

Cons:

No request validation layer
Can't block dangerous patterns
Harder to audit user actions
AI has direct AWS credentials

Why I didn't use it: Too risky. One prompt injection and the AI could delete production resources. The MCP layer provides defense in depth.

Option 2: AWS Lambda as MCP Server

Pros:

Serverless, no infrastructure
Automatic scaling
Pay per request

Cons:

Cold starts (500ms+)
15-minute timeout limit
Harder to maintain state (rate limiting)
More complex networking

Why I didn't use it: Cold starts killed the developer experience. Waiting 500ms for every request was frustrating. Fargate has no cold starts.

Option 3: API Gateway + Lambda

Pros:

Built-in rate limiting
API key management
Request/response transformation

Cons:

More complex setup
Higher cost at scale
Still has Lambda cold starts
Overkill for internal use

Why I didn't use it: The built-in rate limiting was nice, but not worth the complexity for an internal tool. Fargate + ALB was simpler.

Best Practices That Actually Matter

1. Start With Read-Only

Deploy with read-only permissions first. Let developers use it for a week. Then gradually add write permissions based on actual needs.

This prevents over-permissioning. You'll discover what developers actually need, not what they think they need.

2. Use Separate AWS Accounts

Run the MCP server in a separate AWS account from your production workloads. Use cross-account roles for access.

If the MCP account is compromised, production is still isolated. It's an extra layer of defense.

3. Monitor for Anomalies

Set up CloudWatch alarms for unusual patterns:

High error rates (>50 errors in 5 minutes)
Unusual access patterns (>1,000 requests in 5 minutes)
Blocked actions (>100 blocks in 5 minutes)

These alerts go to your security team. Response time is critical.

4. Regular Security Reviews

Every month, review:

Which actions are being used most
Which permissions are never used (remove them)
Any blocked requests (are they legitimate needs?)
Rate limit effectiveness

Security isn't set-and-forget. It requires ongoing attention.

5. Document Everything

Create a runbook for common scenarios:

How to add a new allowed action
How to investigate suspicious activity
How to rotate credentials
How to handle a security incident

When something goes wrong at 2 AM, you'll be glad you documented it.

Summary

Three months in production taught me that securing AI agent access isn't about perfect security - it's about making attacks harder than they're worth while keeping developers productive.

The MCP pattern works because it gives you a single point of control. You're not trying to secure the AI agent itself. You're securing the gateway it uses to access your resources. That gateway validates every request, enforces least privilege, logs everything, and runs in an isolated network.

We went from developers sending production data to ChatGPT to having a secure, auditable system where AI agents help without creating risk. The benefit? No more 2 AM calls about data leaks.

Is it perfect? No. Can a determined attacker find ways around it? Probably. But it's dramatically better than the alternatives: giving AI agents direct AWS credentials or blocking AI tools entirely and watching developers find workarounds.

The key insight: Security is not about building walls. It's about building gates with guards. MCP is that gate.

"Security is not about building walls. It's about building gates with guards."

📌 Wrapping Up

Thank you for reading! I hope this gave you practical ideas for securing AI agent access in your environment.

Found this useful?

❤️ Like if it helped you think through your security approach
🦄 Unicorn if you're implementing this pattern
💾 Save for your next security review
🔄 Share with your security team

Follow me for more on:

AWS security patterns
AI/ML infrastructure
Cloud architecture
DevSecOps practices

💡 What's Next

I'm working on a follow-up article about monitoring and alerting for MCP deployments. Follow for updates.

Also exploring: Multi-region MCP deployments and disaster recovery patterns.

🌐 Portfolio & Work

Explore my full body of work, certifications, and architecture projects:

👉 Visit My Website

🛠️ Services I Offer

Looking for hands-on guidance with cloud security or AI infrastructure?

Cloud Security Architecture (AWS / Azure)
AI/ML Infrastructure Design
Security Audit & Remediation
Technical Writing & Documentation
Architecture Reviews
1:1 Technical Mentorship

🤝 Let's Connect

Questions about implementing this pattern? Drop a comment or connect with me on LinkedIn.

For consulting or technical discussions: simplynadaf@gmail.com

Stay secure! 🔒

Amazon S3 Files: The Game Changer We've Been Waiting For

Sarvar Nadaf — Tue, 14 Apr 2026 14:09:19 +0000

👋 Hey there, tech enthusiasts!

Let's dive in and explore the fascinating world of cloud technology together! 🚀

After working with AWS for over a decade, I've spent countless hours explaining to clients why they can't just "edit a file in S3" the way they would on their local computer. I've drawn diagrams, created analogies, and watched developers struggle with the limitations of object storage versus file systems. Well, that conversation just got a lot more interesting.

In late 2025, AWS released Amazon S3 Files, and honestly, it's one of those features that makes you wonder why it took so long. Let me walk you through what it is, why it matters, and when you should (and shouldn't) use it.

TL;DR: S3 Files is a true POSIX-compliant file system for S3 buckets. Unlike S3 FUSE/Mountpoint, it provides full file system semantics with sub-millisecond latency. Best for: ML training, AI agents, data analytics, shared development environments.

The Problem S3 Files Solves

Let me start with a story that probably sounds familiar. Last year, I worked with a machine learning team training large language models. Their workflow looked like this:

Store training datasets in S3 (hundreds of gigabytes of text data)
Copy datasets to an EBS volume attached to their GPU instances
Train the model, saving checkpoints every hour
Copy checkpoints back to S3 for durability
When spot instances get interrupted, restart everything

This workflow had several problems:

Data duplication: Paying for storage twice (S3 + EBS on every instance)
Time waste: 30-45 minutes copying data before training could start
Complexity: Custom scripts to sync checkpoints and handle interruptions
Cost: Running 2TB EBS volumes on multiple GPU instances 24/7
Spot instance pain: Every interruption meant re-copying everything

They tried using S3 FUSE and Mountpoint S3 to mount their S3 bucket directly, but ran into a wall. Imagine training a model for 6 hours, then having the checkpoint save fail because the training framework needs to update the checkpoint metadata a basic file operation that S3 FUSE simply can't handle. The training framework would write 95% of the checkpoint file, then fail when it tried to seek back to update the header. Why? Because S3 FUSE isn't a real file system it's just an API wrapper pretending to be one.

This is exactly the problem S3 Files solves.

What Exactly Is S3 Files?

Amazon S3 Files is a true, POSIX-compliant file system that sits on top of your S3 buckets. Think of it as a high-performance bridge between your compute resources (EC2, Lambda, ECS, EKS) and your S3 data a smart caching layer that speaks both "file system" and "S3 object" fluently.

Here's what makes it different from previous solutions:

S3 Files vs. The Old Ways

S3 FUSE / Mountpoint S3:

API wrappers that translate file operations into S3 API calls
Limited file system semantics
Can't handle operations like seeking backward in files
No true concurrent write support

S3 Files:

Built on Amazon Elastic File System (EFS) technology
True POSIX compliance (supports all NFS v4.1+ operations)
Sub-millisecond latency for cached data
Full file system semantics (create, read, update, delete, seek, append)
Concurrent access from multiple compute resources

How S3 Files Actually Works

The architecture is clever. When you create an S3 file system and mount it to your EC2 instance, here's what happens:

Initial Mount: You see your S3 bucket as a directory structure. No data is copied yet.
Lazy Loading: When you access a file, only that file's metadata and content are loaded into a high-performance cache layer (built on EFS).
Smart Caching:
- Frequently accessed files stay in the cache (sub-millisecond access)
- Large sequential reads go directly from S3 (maximizing throughput)
- Only requested byte ranges are transferred (minimizing costs)
Write Operations:
- Writes go to the cache first (fast)
- Changes sync back to S3 automatically within minutes
- You get immediate file system consistency
Bidirectional Sync:
- Changes in the file system appear in S3 within minutes
- Changes in S3 appear in the file system within seconds

Real-World Example: AI Model Training Pipeline

Let me show you a practical example. I recently helped an AI research team migrate their model training pipeline to use S3 Files.

The Scenario

They train large language models with:

500GB training datasets (tokenized text)
Hourly checkpoint saves (10-50GB each)
Multi-GPU distributed training
Spot instances for cost optimization

The Old Architecture

S3 Dataset → Copy to EBS → Train Model → Save Checkpoint to EBS → Sync to S3 → Spot Interruption → Repeat

Problems:

30-45 minutes copying data before training starts
$1,200/month in EBS costs across GPU instances
Complex checkpoint sync logic
Spot interruptions meant starting over with data copies

The New Architecture with S3 Files

S3 Dataset → Mount S3 Files → Train Model → Checkpoints auto-sync to S3 → Spot Interruption → Remount & Resume

The Results

With S3 Files, they mount their S3 bucket directly to their GPU instances. Training frameworks like PyTorch can now write checkpoints with full file system semantics updating headers, seeking to specific positions, and appending data operations that previously failed with S3 FUSE.

Training startup: Reduced from 45 minutes to 2 minutes
Cost savings: $1,200/month in EBS costs eliminated
Spot instance recovery: From 45 minutes to 5 minutes (just remount and resume)
Simplified code: Removed 300+ lines of checkpoint sync and recovery logic
Better reliability: No more corrupted checkpoints or failed saves

Use Cases Where S3 Files Shines

Now that you understand how it works, let's look at where S3 Files makes the biggest impact.

Based on my experience and conversations with other architects, here are the scenarios where S3 Files is a perfect fit:

1. Machine Learning Training

Scenario: Training models on large datasets stored in S3

Before S3 Files:

Copy entire dataset to EBS/EFS before training
Wait hours for data transfer
Pay for duplicate storage

With S3 Files:

Mount S3 bucket directly
Training starts immediately (lazy loading)
Only accessed data is cached
Multiple training instances can share the same data

2. AI Agents and Automation

Scenario: AI agents using Python libraries and CLI tools that expect file systems

The Challenge: Many AI agents and automation tools are built with traditional file system operations in mind. Rewriting them to use S3 APIs directly would require significant code changes.

With S3 Files: Your existing code that expects file system access works without modification. The agent can read, write, and process files as if they were on a local file system, while S3 Files handles the synchronization with your S3 bucket automatically.

This means AI agents can use standard Python libraries, shell scripts, and CLI tools without any code changes.

3. Shared Development Environments

Scenario: Multiple developers or containers need concurrent access to shared data

With S3 Files:

Mount the same S3 bucket on multiple EC2 instances
Developers can read and write simultaneously
Changes are visible across all instances within seconds
No complex locking mechanisms needed

This is especially powerful for Kubernetes clusters where pods need shared access to training data or model artifacts across nodes.

4. Content Management Systems

Scenario: CMS storing media files that need to be accessed by multiple web servers

With S3 Files:

All web servers mount the same S3 bucket
Upload once, available everywhere immediately
No need for S3 sync scripts or CDN invalidation delays

Important Considerations

Let me share some lessons learned from real implementations:

Performance Characteristics

First access: Slower (loading from S3 to cache)
Subsequent access: Sub-millisecond (from cache)
Large sequential reads: Served directly from S3 (high throughput)
Small random reads: Served from cache (low latency)

Pro tip: If you know which files you'll need, you can pre-load them into the cache to avoid first-access latency.

Consistency Model

File system to S3: Changes appear in S3 within minutes
S3 to file system: Changes appear in file system within seconds (sometimes up to a minute)

Important: If you need immediate consistency, use the file system as your source of truth during processing, then let it sync to S3.

Cost Structure

S3 Files has three cost components:

Storage: $0.30/GB-month for data in the cache
Data access:
- $0.03/GB for reads
- $0.06/GB for writes
S3 costs: Standard S3 storage and request costs

Cost optimization tip: The cache only stores actively used data. If you're processing 100GB from a 10TB bucket, you only pay cache costs for the 100GB.

Security

Encryption in transit: TLS 1.3 (automatic)
Encryption at rest: SSE-S3 or KMS (your choice)
Access control: IAM policies + POSIX permissions
Network isolation: Mount targets live in your VPC

When NOT to Use S3 Files

Being honest about limitations is important. S3 Files isn't the right choice for:

Simple object storage: If you're just storing and retrieving whole objects, regular S3 is simpler and cheaper.
Infrequent access: If you rarely access your data, the cache storage costs aren't worth it.
Windows-specific features: If you need Windows file system features, use FSx for Windows File Server.
Extreme performance HPC: If you need the absolute highest performance for HPC workloads, FSx for Lustre is better optimized.
On-premises integration: If you're migrating from on-prem NAS, FSx for NetApp ONTAP provides better compatibility.

Getting Started

S3 Files is available in all commercial AWS regions as of April 2026. You can create an S3 file system through the AWS Console, AWS CLI, or infrastructure as code tools like CloudFormation and Terraform.

The basic workflow involves:

Creating an S3 file system linked to your bucket
Configuring network access (VPC, subnets, security groups)
Setting up IAM permissions
Mounting the file system on your compute resources
Using standard file operations to interact with your S3 data

In the next article, I'll walk you through a complete hands-on implementation with step-by-step commands and real examples.

My Take After Using It

I've been working with S3 Files since early access in late 2025, across several client projects. Here's my honest assessment:

What I love:

It eliminates entire categories of data movement code
Applications that expect file systems just work
The lazy loading is genuinely smart
Cost savings are real when you eliminate data duplication

What to watch out for:

The sync delay (minutes) can surprise people expecting instant S3 updates
You need to understand the caching behavior to optimize costs
Security group configuration is an extra step that trips people up

Bottom line: If you're currently copying data between S3 and file systems, or if you're struggling with S3 FUSE limitations, S3 Files is absolutely worth evaluating. It's not a silver bullet, but for the right use cases, it's transformative.

Availability and Pricing

As of April 2026, S3 Files is available in all commercial AWS regions.

Pricing varies by region, but in us-east-1:

Cache storage: $0.30/GB-month
Read operations: $0.03/GB
Write operations: $0.06/GB
Plus standard S3 costs

Check the S3 pricing page for your region's specific pricing.

Conclusion

Amazon S3 Files represents a significant shift in how we can architect cloud applications. For years, we've had to choose between S3's durability and cost-effectiveness versus a file system's interactive capabilities. S3 Files eliminates that tradeoff.

Whether you're training ML models on spot instances, building AI agents, running data analytics pipelines, or any workload that needs shared, interactive file access to S3 data, S3 Files deserves a serious look. It's not just a new feature it's a new way of thinking about data access in the cloud.

The best part? You don't need to rewrite your applications. If your code works with files, it'll work with S3 Files. And that's the kind of simplicity we all need more of in cloud architecture.

In my next article, I'll show you exactly how to set up S3 Files with a complete hands-on implementation guide, including all the commands, configurations, and real-world testing scenarios.

Have you tried S3 Files yet? I'd love to hear about your use cases and experiences. The cloud architecture community learns best when we share real-world implementations, not just theory.

📌 Wrapping Up

Thank you for reading! I hope this article gave you practical insights and a clearer perspective on the topic.

Was this helpful?

❤️ Like if it added value
🦄 Unicorn if you’re applying it today
💾 Save for your next optimization session
🔄 Share with your team

Follow me for more on:

AWS architecture patterns
FinOps automation
Multi-account strategies
AI-driven DevOps

💡 What’s Next

More deep dives coming soon on cloud operations, GenAI, Agentic-AI, DevOps, and data workflows follow for weekly insights.

🌐 Portfolio & Work

You can explore my full body of work, certifications, architecture projects, and technical articles here:

👉 Visit My Website

🛠️ Services I Offer

If you're looking for hands-on guidance or collaboration, I provide:

Cloud Architecture Consulting (AWS / Azure)
DevSecOps & Automation Design
FinOps Optimization Reviews
Technical Writing (Cloud, DevOps, GenAI)
Product & Architecture Reviews
Mentorship & 1:1 Technical Guidance

🤝 Let’s Connect

I’d love to hear your thoughts drop a comment or connect with me on LinkedIn.

For collaborations, consulting, or technical discussions, feel free to reach out directly at simplynadaf@gmail.com

Happy Learning 🚀

Building Your First VPC: AWS Networking with Terraform

Sarvar Nadaf — Wed, 08 Apr 2026 10:04:44 +0000

👋 Hey there, tech enthusiasts!

🎯 Welcome Back!

Remember in Article 5 when you learned about variables and outputs? You created flexible, reusable S3 bucket configurations. That's great for storage, but what about networking?

Here's the reality: S3 buckets are public services. But what about:

EC2 instances that need network isolation?
Databases that should never be directly accessible from the internet?
Applications that need both public and private components?

That's where VPC (Virtual Private Cloud) comes in.

By the end of this article, you'll:

✅ Understand VPC fundamentals (CIDR, subnets, routing)
✅ Create a production-ready VPC with Terraform
✅ Configure public and private subnets
✅ Set up Internet Gateway and NAT Gateway
✅ Master route tables and associations
✅ Use data sources to fetch AWS information
✅ Build cost-optimized VPC architecture

Time Required: 45 minutes (20 min read + 25 min practice)

Cost: ~$0.05 for testing (destroy immediately!)

Difficulty: Intermediate

⚠️ IMPORTANT: NAT Gateway costs $0.045/hour (~$32/month). We'll create it for learning, but destroy it immediately after testing!

Let's build real infrastructure! 🚀

💔 The Problem: Networking Confusion

The Manual Nightmare

Imagine this scenario (maybe you've lived it):

Your Task: "Set up a VPC for our new application"

You open AWS Console:

Step 1: Create VPC... wait, what's a CIDR block?
Step 2: Create subnets... public or private? What's the difference?
Step 3: Internet Gateway... where does this go?
Step 4: NAT Gateway... why do I need this?
Step 5: Route tables... which subnet goes where?
Step 6: *2 hours later* ... did I configure this correctly?
Step 7: *Application doesn't work* ... something's wrong with routing
Step 8: *Starts over* 😰

Common Problems:

❌ Forgot route table associations - Subnets can't reach internet

❌ Wrong CIDR blocks - IP addresses overlap

❌ NAT Gateway in wrong subnet - Private instances can't update

❌ Inconsistent configuration - Resources don't work together

❌ Can't remember configuration - No documentation

❌ Takes hours to set up - Clicking through endless screens

Sound familiar? Let's fix this with Terraform.

🌐 What is a VPC? (Quick Theory)

Simple Definition

VPC (Virtual Private Cloud) is your own private network in AWS. Think of it like this:

Traditional Data Center: Physical building with servers, switches, routers
VPC: Virtual data center with virtual servers, virtual switches, virtual routers

Key Point: Your VPC is isolated from other AWS accounts. It's YOUR private network.

VPC Components (The Building Blocks)

VPC (Your Private Network)
├── CIDR Block (IP Address Range)
├── Subnets (Subdivisions of your network)
│   ├── Public Subnets (Internet-accessible)
│   └── Private Subnets (Internal only)
├── Internet Gateway (Door to the internet)
├── NAT Gateway (Private subnet's internet access)
└── Route Tables (GPS for network traffic)

Understanding CIDR Blocks

CIDR = Classless Inter-Domain Routing (fancy name for IP address ranges)

Examples:

10.0.0.0/16 = 65,536 IP addresses (10.0.0.0 to 10.0.255.255)
10.0.1.0/24 = 256 IP addresses (10.0.1.0 to 10.0.1.255)
10.0.2.0/24 = 256 IP addresses (10.0.2.0 to 10.0.2.255)

Simple Rule: Smaller number after / = More IP addresses

/16 = 65,536 IPs (big network)
/24 = 256 IPs (small network)
/32 = 1 IP (single host)

Don't worry! You don't need to be a networking expert. Just follow the patterns.

Public vs Private Subnets

Public Subnet:

✅ Has route to Internet Gateway
✅ Resources get public IP addresses
✅ Directly accessible from internet
📍 Use for: Web servers, load balancers, bastion hosts

Private Subnet:

✅ Has route to NAT Gateway (not Internet Gateway)
❌ No public IP addresses
❌ Not directly accessible from internet
📍 Use for: Databases, application servers, internal services

Why both? Security! Keep sensitive resources (databases) in private subnets.

🏗️ What We'll Build Today

Architecture Overview

VPC: 10.0.0.0/16 (65,536 IPs)
│
├── Public Subnet 1: 10.0.1.0/24 (256 IPs) - us-east-1a
│   ├── Internet Gateway attached
│   └── Route: 0.0.0.0/0 → Internet Gateway
│
├── Private Subnet 1: 10.0.11.0/24 (256 IPs) - us-east-1a
│   └── Route: 0.0.0.0/0 → NAT Gateway

Why this design?

✅ Single-AZ
✅ Public subnets = For load balancers, web servers
✅ Private subnets = For databases, application servers
✅ NAT Gateway = Private subnets can download updates
✅ Production-ready = This is how real companies do it

📝 Step-by-Step: Building the VPC

Step 1: Create Project Directory

mkdir -p ~/terraform-vpc-demo
cd ~/terraform-vpc-demo

Step 2: Create variables.tf

Let's start with variables for flexibility:

nano variables.tf

Copy this code:

# AWS Region
variable "aws_region" {
  description = "AWS region where resources will be created"
  type        = string
  default     = "us-east-1"
}

# Project Name
variable "project_name" {
  description = "Project name to be used in resource naming"
  type        = string
  default     = "terraform-vpc"
}

# Environment
variable "environment" {
  description = "Environment name (dev, staging, prod)"
  type        = string
  default     = "dev"
}

# VPC CIDR Block
variable "vpc_cidr" {
  description = "CIDR block for VPC"
  type        = string
  default     = "10.0.0.0/16"
}

# Public Subnet CIDR
variable "public_subnet_cidr" {
  description = "CIDR block for public subnet"
  type        = string
  default     = "10.0.1.0/24"
}

# Private Subnet CIDR
variable "private_subnet_cidr" {
  description = "CIDR block for private subnet"
  type        = string
  default     = "10.0.11.0/24"
}

What's happening here?

All CIDR blocks are variables (easy to change)
Project name and environment for tagging
Default values provided (but can be overridden)
Cost-optimized: 1 public + 1 private subnet

Save the file (Ctrl+X, then Y, then Enter)

Step 3: Create terraform.tfvars

nano terraform.tfvars

Copy this code:

# Default configuration for development environment
aws_region = "us-east-1"
project_name = "terraform-vpc"
environment = "dev"

# VPC Configuration
vpc_cidr = "10.0.0.0/16"

# Subnets
public_subnet_cidr = "10.0.1.0/24"
private_subnet_cidr = "10.0.11.0/24"

Save the file

Step 4: Create main.tf (The Main Infrastructure)

This is where the magic happens! We'll build it piece by piece.

nano main.tf

Copy this code:

# Terraform configuration
terraform {
  required_version = ">= 1.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

# Provider configuration
provider "aws" {
  region = var.aws_region
}

# Data source: Get available availability zones
data "aws_availability_zones" "available" {
  state = "available"
}

# VPC
resource "aws_vpc" "main" {
  cidr_block           = var.vpc_cidr
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name        = "${var.project_name}-vpc"
    Environment = var.environment
    ManagedBy   = "Terraform"
  }
}

# Internet Gateway
resource "aws_internet_gateway" "main" {
  vpc_id = aws_vpc.main.id

  tags = {
    Name        = "${var.project_name}-igw"
    Environment = var.environment
    ManagedBy   = "Terraform"
  }
}

# Public Subnet 1
resource "aws_subnet" "public_1" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = var.public_subnet_1_cidr
  availability_zone       = data.aws_availability_zones.available.names[0]
  map_public_ip_on_launch = true

  tags = {
    Name        = "${var.project_name}-public-subnet"
    Environment = var.environment
    Type        = "Public"
    ManagedBy   = "Terraform"
  }
}

# Private Subnet
resource "aws_subnet" "private" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = var.private_subnet_cidr
  availability_zone = data.aws_availability_zones.available.names[0]

  tags = {
    Name        = "${var.project_name}-private-subnet"
    Environment = var.environment
    Type        = "Private"
    ManagedBy   = "Terraform"
  }
}

# Elastic IP for NAT Gateway
resource "aws_eip" "nat" {
  domain = "vpc"

  tags = {
    Name        = "${var.project_name}-nat-eip"
    Environment = var.environment
    ManagedBy   = "Terraform"
  }

  depends_on = [aws_internet_gateway.main]
}

# NAT Gateway
resource "aws_nat_gateway" "main" {
  allocation_id = aws_eip.nat.id
  subnet_id     = aws_subnet.public_1.id

  tags = {
    Name        = "${var.project_name}-nat-gateway"
    Environment = var.environment
    ManagedBy   = "Terraform"
  }

  depends_on = [aws_internet_gateway.main]
}

# Public Route Table
resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.main.id
  }

  tags = {
    Name        = "${var.project_name}-public-rt"
    Environment = var.environment
    Type        = "Public"
    ManagedBy   = "Terraform"
  }
}

# Private Route Table
resource "aws_route_table" "private" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.main.id
  }

  tags = {
    Name        = "${var.project_name}-private-rt"
    Environment = var.environment
    Type        = "Private"
    ManagedBy   = "Terraform"
  }
}

# Public Subnet Route Table Association
resource "aws_route_table_association" "public" {
  subnet_id      = aws_subnet.public.id
  route_table_id = aws_route_table.public.id
}

# Private Subnet Route Table Association
resource "aws_route_table_association" "private_1" {
  subnet_id      = aws_subnet.private_1.id
  route_table_id = aws_route_table.private.id
}

# Private Subnet Route Table Association
resource "aws_route_table_association" "private" {
  subnet_id      = aws_subnet.private.id
  route_table_id = aws_route_table.private.id
}

Save the file

That's the complete VPC infrastructure! Let's break it down...

🔍 Understanding the Code

1. Data Source (NEW CONCEPT!)

data "aws_availability_zones" "available" {
  state = "available"
}

What's a data source?

Resources = Things Terraform creates (VPC, subnets, etc.)
Data Sources = Things Terraform reads from AWS (AZs, AMIs, etc.)

Why use it?

Don't hardcode availability zones (they differ by region)
Automatically get available AZs
Makes code portable across regions

How to use it:

availability_zone = data.aws_availability_zones.available.names[0]  # First AZ
availability_zone = data.aws_availability_zones.available.names[1]  # Second AZ

2. VPC Resource

resource "aws_vpc" "main" {
  cidr_block           = var.vpc_cidr
  enable_dns_hostnames = true  # Important for EC2 instances
  enable_dns_support   = true  # Important for DNS resolution
}

Key settings:

enable_dns_hostnames = EC2 instances get DNS names
enable_dns_support = DNS resolution works in VPC

Without these: Your instances can't resolve domain names!

3. Internet Gateway

resource "aws_internet_gateway" "main" {
  vpc_id = aws_vpc.main.id  # Attach to our VPC
}

What it does: Allows public subnets to access the internet

Think of it as: The front door of your VPC

4. Public Subnets

resource "aws_subnet" "public_1" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = var.public_subnet_1_cidr
  availability_zone       = data.aws_availability_zones.available.names[0]
  map_public_ip_on_launch = true  # Auto-assign public IPs
}

Key setting: map_public_ip_on_launch = true

EC2 instances in this subnet automatically get public IPs
Makes them accessible from internet

5. Private Subnets

resource "aws_subnet" "private_1" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = var.private_subnet_1_cidr
  availability_zone = data.aws_availability_zones.available.names[0]
  # NO map_public_ip_on_launch = Private!
}

Notice: No map_public_ip_on_launch

Instances in private subnets don't get public IPs
More secure for databases and internal services

6. Elastic IP and NAT Gateway

# Elastic IP (static public IP)
resource "aws_eip" "nat" {
  domain = "vpc"
  depends_on = [aws_internet_gateway.main]
}

# NAT Gateway (in public subnet!)
resource "aws_nat_gateway" "main" {
  allocation_id = aws_eip.nat.id
  subnet_id     = aws_subnet.public_1.id  # Must be in public subnet
  depends_on = [aws_internet_gateway.main]
}

What's NAT Gateway?

Allows private subnets to access internet (outbound only)
Private instances can download updates, call APIs
But internet can't initiate connections to private instances

Why Elastic IP?

NAT Gateway needs a static public IP
Elastic IP provides that

⚠️ COST WARNING: NAT Gateway = $0.045/hour (~$32/month)

7. Route Tables (The GPS)

Public Route Table:

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block = "0.0.0.0/0"  # All traffic
    gateway_id = aws_internet_gateway.main.id  # Goes to Internet Gateway
  }
}

Translation: "Send all internet traffic (0.0.0.0/0) to the Internet Gateway"

Private Route Table:

resource "aws_route_table" "private" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block     = "0.0.0.0/0"  # All traffic
    nat_gateway_id = aws_nat_gateway.main.id  # Goes to NAT Gateway
  }
}

Translation: "Send all internet traffic (0.0.0.0/0) to the NAT Gateway"

8. Route Table Associations

resource "aws_route_table_association" "public_1" {
  subnet_id      = aws_subnet.public_1.id
  route_table_id = aws_route_table.public.id
}

What this does: Links subnet to route table

Think of it as: "Public Subnet 1, use the Public Route Table for routing"

Without this: Subnet doesn't know how to route traffic!

📤 Step 5: Create outputs.tf

nano outputs.tf

Copy this code:

# VPC Outputs
output "vpc_id" {
  description = "ID of the VPC"
  value       = aws_vpc.main.id
}

output "vpc_cidr" {
  description = "CIDR block of the VPC"
  value       = aws_vpc.main.cidr_block
}

# Subnet Outputs
output "public_subnet_1_id" {
  description = "ID of public subnet 1"
  value       = aws_subnet.public_1.id
}

output "public_subnet_2_id" {
  description = "ID of public subnet 2"
  value       = aws_subnet.public_2.id
}

output "private_subnet_1_id" {
  description = "ID of private subnet 1"
  value       = aws_subnet.private_1.id
}

output "private_subnet_2_id" {
  description = "ID of private subnet 2"
  value       = aws_subnet.private_2.id
}

# Gateway Outputs
output "internet_gateway_id" {
  description = "ID of the Internet Gateway"
  value       = aws_internet_gateway.main.id
}

output "nat_gateway_id" {
  description = "ID of the NAT Gateway"
  value       = aws_nat_gateway.main.id
}

output "nat_gateway_public_ip" {
  description = "Public IP of the NAT Gateway"
  value       = aws_eip.nat.public_ip
}

# Route Table Outputs
output "public_route_table_id" {
  description = "ID of the public route table"
  value       = aws_route_table.public.id
}

output "private_route_table_id" {
  description = "ID of the private route table"
  value       = aws_route_table.private.id
}

# Availability Zone
output "availability_zone" {
  description = "Availability zone used"
  value       = data.aws_availability_zones.available.names[0]
}

Save the file

Why so many outputs?

You'll need these IDs in Article 7 (EC2 + Load Balancer)
Good documentation of what was created
Easy to reference in other Terraform modules

🚀 Let's Deploy the VPC!

Step 6: Initialize Terraform

terraform init

Expected output:

Initializing the backend...
Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 5.0"...
- Installing hashicorp/aws v5.x.x...

Terraform has been successfully initialized!

Step 7: Plan the Infrastructure

terraform plan

What you'll see:

Terraform will perform the following actions:

  # aws_eip.nat will be created
  # aws_internet_gateway.main will be created
  # aws_nat_gateway.main will be created
  # aws_route_table.private will be created
  # aws_route_table.public will be created
  # aws_route_table_association.private_1 will be created
  # aws_route_table_association.private_2 will be created
  # aws_route_table_association.public_1 will be created
  # aws_route_table_association.public_2 will be created
  # aws_subnet.private_1 will be created
  # aws_subnet.private_2 will be created
  # aws_subnet.public_1 will be created
  # aws_subnet.public_2 will be created
  # aws_vpc.main will be created

Plan: 10 to add, 0 to change, 0 to destroy.

10 resources! That's a complete VPC infrastructure from just one command.

Review the plan carefully:

✅ VPC with correct CIDR (10.0.0.0/16)
✅ 4 subnets with correct CIDRs
✅ Subnets in different availability zones
✅ Internet Gateway attached to VPC
✅ NAT Gateway in public subnet
✅ Route tables with correct routes

Step 8: Apply the Configuration

terraform apply

Type yes when prompted.

Expected output:

aws_vpc.main: Creating...
aws_vpc.main: Creation complete after 2s [id=vpc-xxxxx]
aws_internet_gateway.main: Creating...
aws_subnet.public_1: Creating...
aws_subnet.public_2: Creating...
aws_subnet.private_1: Creating...
aws_subnet.private_2: Creating...
...
aws_nat_gateway.main: Creating...
aws_nat_gateway.main: Still creating... [10s elapsed]
aws_nat_gateway.main: Still creating... [20s elapsed]
...
aws_nat_gateway.main: Creation complete after 1m45s

Apply complete! Resources: 10 added, 0 changed, 0 destroyed.

Outputs:

availability_zones = [
  "us-east-1a"
internet_gateway_id = "igw-xxxxx"
nat_gateway_id = "nat-xxxxx"
nat_gateway_public_ip = "54.xxx.xxx.xxx"
private_route_table_id = "rtb-xxxxx"
private_subnet_id = "subnet-xxxxx"
public_route_table_id = "rtb-xxxxx"
public_subnet_id = "subnet-xxxxx"
vpc_cidr = "10.0.0.0/16"
vpc_id = "vpc-xxxxx"

⏱️ Time: Takes about 2-3 minutes (NAT Gateway is slow to create)

🎉 Success! You just created production-grade networking infrastructure!

Step 9: Verify in AWS Console

Option 1: AWS CLI

# Get VPC ID
VPC_ID=$(terraform output -raw vpc_id)

# List all subnets
aws ec2 describe-subnets \
  --filters "Name=vpc-id,Values=$VPC_ID" \
  --query 'Subnets[*].[SubnetId,CidrBlock,AvailabilityZone,Tags[?Key==`Name`].Value|[0]]' \
  --output table

# Check NAT Gateway
aws ec2 describe-nat-gateways \
  --filter "Name=vpc-id,Values=$VPC_ID" \
  --query 'NatGateways[*].[NatGatewayId,State,SubnetId]' \
  --output table

# Check route tables
aws ec2 describe-route-tables \
  --filters "Name=vpc-id,Values=$VPC_ID" \
  --query 'RouteTables[*].[RouteTableId,Tags[?Key==`Name`].Value|[0]]' \
  --output table

Option 2: AWS Console

Go to AWS Console → VPC
Find your VPC: terraform-vpc-vpc
Check:
- ✅ 2 subnets (1 public, 1 private)
- ✅ Internet Gateway attached
- ✅ NAT Gateway in public subnet
- ✅ 2 route tables (public and private)

🧠 Understanding Resource Dependencies

Implicit Dependencies

Terraform automatically understands dependencies when you reference resources:

resource "aws_subnet" "public_1" {
  vpc_id = aws_vpc.main.id  # References VPC
}

Terraform knows: "Create VPC first, then create subnet"

Explicit Dependencies

Sometimes you need to force an order:

resource "aws_eip" "nat" {
  domain = "vpc"
  depends_on = [aws_internet_gateway.main]  # Explicit dependency
}

Why? Elastic IP in VPC requires Internet Gateway to exist first.

Dependency Graph

Terraform builds a graph of dependencies:

VPC
├── Internet Gateway
│   ├── Elastic IP
│   │   └── NAT Gateway
│   └── Public Route Table
│       └── Route Table Associations
└── Subnets
    └── Route Table Associations

Terraform creates resources in the correct order automatically!

💰 Cost Breakdown

Let's talk about costs (important!):

Resource	Cost	Notes
VPC	FREE	No charge for VPCs
Subnets	FREE	No charge for subnets
Internet Gateway	FREE	No charge for IGW
Elastic IP	FREE*	*When attached to running instance
NAT Gateway	$0.045/hour	~$32/month ⚠️
Data Transfer	$0.045/GB	Through NAT Gateway

Total for this setup: ~$32/month (just for NAT Gateway)

⚠️ IMPORTANT: Always destroy test infrastructure!

terraform destroy

Production Note: In production, you'd have one NAT Gateway per AZ (2 NAT Gateways = $64/month) for high availability.

🎯 Best Practices

1. CIDR Planning

Do:

✅ Plan CIDR blocks before creating VPC
✅ Leave room for growth (use /16 for VPC)
✅ Use non-overlapping ranges

Don't:

❌ Use overlapping CIDR blocks
❌ Make VPC too small (/24 = only 256 IPs)
❌ Forget about VPC peering requirements

Tool: Use CIDR Calculator for planning

2. Multi-AZ Architecture

Always use multiple availability zones:

✅ High availability
✅ Fault tolerance
✅ Better for production

Our setup: 2 AZs (us-east-1a, us-east-1b)

3. Tagging Strategy

Always tag resources:

tags = {
  Name        = "${var.project_name}-vpc"
  Environment = var.environment
  ManagedBy   = "Terraform"
  Project     = "Learning"
}

Why?

Easy to identify resources
Cost allocation
Automation and filtering

4. Separate Public and Private

Security principle:

Public subnets: Load balancers, bastion hosts
Private subnets: Databases, application servers

Never put databases in public subnets!

5. Use Variables

Make everything configurable:

CIDR blocks
Project name
Environment
Region

Benefit: Same code for dev, staging, prod

🐛 Common Issues & Solutions

Issue 1: NAT Gateway Creation Timeout

Problem:

Error: timeout while waiting for state to become 'available'

Solution:

NAT Gateway takes 2-3 minutes to create
This is normal, just wait
If it fails, run terraform apply again

Issue 2: Elastic IP Limit Reached

Problem:

Error: AddressLimitExceeded: The maximum number of addresses has been reached

Solution:

# Check current EIPs
aws ec2 describe-addresses

# Release unused EIPs
aws ec2 release-address --allocation-id eipalloc-xxxxx

# Or request limit increase in AWS Console

Issue 3: Route Table Not Associated

Problem: Subnet can't reach internet

Solution: Check route table associations:

aws ec2 describe-route-tables --filters "Name=vpc-id,Values=$VPC_ID"

Ensure each subnet has a route table association.

Issue 4: CIDR Block Overlap

Problem:

Error: InvalidSubnet.Conflict: The CIDR 'x.x.x.x/x' conflicts with another subnet

Solution:

Check your CIDR blocks don't overlap
Public: 10.0.1.0/24, 10.0.2.0/24
Private: 10.0.11.0/24, 10.0.12.0/24
No overlap!

Issue 5: Forgot to Destroy NAT Gateway

Problem: Unexpected AWS bill

Solution:

# Check for NAT Gateways
aws ec2 describe-nat-gateways --filter "Name=state,Values=available"

# Destroy with Terraform
terraform destroy

# Or manually delete in AWS Console

🧹 Cleanup (IMPORTANT!)

⚠️ Don't forget this step! NAT Gateway costs money.

# Destroy all resources
terraform destroy

Type yes when prompted.

Expected output:

aws_route_table_association.private_2: Destroying...
aws_route_table_association.private_1: Destroying...
aws_route_table_association.public_2: Destroying...
aws_route_table_association.public_1: Destroying...
...
aws_nat_gateway.main: Destroying...
aws_nat_gateway.main: Still destroying... [10s elapsed]
aws_nat_gateway.main: Still destroying... [20s elapsed]
...
aws_nat_gateway.main: Destruction complete after 1m30s
...
Destroy complete! Resources: 10 destroyed.

Verify deletion:

# Should return empty
aws ec2 describe-vpcs --filters "Name=tag:Name,Values=terraform-vpc-vpc"

Clean up directory:

cd ~
rm -rf ~/terraform-vpc-demo

📚 What You Learned

Concepts:

✅ VPC fundamentals (CIDR, subnets, routing)
✅ Public vs Private subnets
✅ Internet Gateway and NAT Gateway
✅ Route tables and associations
✅ Data sources (aws_availability_zones)
✅ Resource dependencies (implicit and explicit)
✅ Cost-optimized architecture

Skills:

✅ Created production-ready VPC
✅ Configured networking components
✅ Used data sources
✅ Managed complex infrastructure
✅ Understood AWS networking costs

Infrastructure:

✅ 1 VPC
✅ 2 Subnets (1 public, 1 private)
✅ 1 Internet Gateway
✅ 1 NAT Gateway
✅ 2 Route Tables
✅ Cost-optimized setup

🎓 Challenge: Extend Your VPC

Try these modifications:

Easy:

Change CIDR blocks to 172.16.0.0/16
Add a third availability zone
Change project name and environment

Medium:

Add more private subnets (10.0.21.0/24, 10.0.22.0/24)
Create separate route tables for each private subnet
Add Network ACLs for additional security

Hard:

Add VPC Flow Logs
Create a second NAT Gateway for high availability
Add VPC endpoints for S3 and DynamoDB

🔗 Useful Resources

🚀 What's Next?

In Article 7, we'll deploy EC2 instances and an Application Load Balancer in this VPC!

You'll learn:

Launch EC2 instances in public and private subnets
Configure security groups
Create an Application Load Balancer
Set up health checks
Test the complete architecture

Time: 45 minutes

Difficulty: Intermediate

Outcome: Full web application infrastructure

📌 Wrapping Up

Thank you for reading. I hope this article provided practical insights and a clearer understanding of the topic.

If you found this useful:

❤️ Like if it added value
🦄 Unicorn if you’re applying it today
💾 Save it for your next optimization session
🔄 Share it with your team

💡 What’s Next

More deep dives are coming soon on:

Cloud Operations
GenAI & Agentic AI
DevOps Automation
Data & Platform Engineering

Follow along for weekly insights and hands-on guides.

🌐 Portfolio & Work

You can explore my full body of work, certifications, architecture projects, and technical articles here:

👉 Visit My Website

🛠️ Services I Offer

If you're looking for hands-on guidance or collaboration, I provide:

Cloud Architecture Consulting (AWS / Azure)
DevSecOps & Automation Design
FinOps Optimization Reviews
Technical Writing (Cloud, DevOps, GenAI)
Product & Architecture Reviews
Mentorship & 1:1 Technical Guidance

🤝 Let’s Connect

I’d love to hear your thoughts. Feel free to drop a comment or connect with me on:

🔗 LinkedIn

For collaborations, consulting, or technical discussions, reach out at:

📧 simplynadaf@gmail.com

Found this helpful? Share it with your team.

⭐ Star the repo • 📖 Follow the series • 💬 Ask questions

Made by Sarvar Nadaf

🌐 https://sarvarnadaf.com

Connecting 12 MCP Servers to Amazon Q CLI: What Broke and How I Fixed It

Sarvar Nadaf — Mon, 06 Apr 2026 13:02:02 +0000

👋 Hey there, tech enthusiasts!

Let's dive in and explore the fascinating world of cloud technology together! 🚀

Written from experience building AI agent integrations for AWS infrastructure management. Your mileage may vary, but the principles hold across different use cases.

Do We Still Need MCP When We Have Agent Skills?

As a cloud architect working with AI agents, I've spent the last few months exploring how to extend their capabilities. Specifically, I've been working with Amazon Q Developer Pro - AWS's AI assistant that helps with coding, infrastructure management, and cloud operations through a chat interface.

This article shares what I learned about two different approaches to extending AI agents: Model Context Protocol (MCP) and Agent Skills. While the specific implementation details are still evolving, the architectural patterns I describe here represent where the ecosystem is heading based on current capabilities and community standards.

The Problem I Was Solving

I needed Amazon Q Developer Pro to help me manage AWS infrastructure across multiple accounts. The agent needed to:

Check CloudWatch metrics
Query RDS databases
Send alerts to Slack
Generate cost reports
Review CloudFormation templates

I tried two approaches, and each had serious problems.

Approach 1: Connecting Everything via MCP

MCP is a standard protocol that lets AI agents connect to external tools. Think of it like USB ports on your computer - one standard interface that works with many devices.

I connected 12 MCP servers to Amazon Q Developer Pro:

AWS CloudWatch server
RDS query server
Slack messaging server
Cost Explorer server
GitHub server
And seven more

What Went Wrong

Before I even asked a question, 35% of the context window was already full. Each MCP server loaded all its available functions into memory. The CloudWatch server alone exposed 15 different functions with detailed parameter descriptions.

When I asked Amazon Q Developer Pro to "check if our xyz database is healthy," it had to scan through multiple function definitions to figure out which ones to use. Sometimes it picked the wrong ones.

Every function call and response consumed more context. After three or four operations, I was running out of space for the actual conversation.

Approach 2: Using Agent Skills

Agent Skills are different. Instead of connecting to external tools, you give the agent domain knowledge - a guide on how to think about a problem.

I created a skill called "database-health-check" with a simple file structure:

database-health-check/
  SKILL.md          (when to use this, what steps to follow)
  scripts/
    check_rds.py    (Python script to query RDS)

The SKILL.md file contained:

# Database Health Check Skill

## Trigger
Keywords: "database health", "RDS status", "database performance", "DB issues"

## Process
1. Check CPU utilization (threshold: 80%)
2. Check active connections (threshold: 90% of max)
3. Check replication lag (threshold: 1000ms)
4. Check storage space (threshold: 85%)

## Decision Logic
- If CPU > 80%: WARN - "High CPU usage detected"
- If connections > 90%: CRITICAL - "Connection pool nearly exhausted"
- If replication lag > 1000ms: WARN - "Replication falling behind"
- If storage > 85%: CRITICAL - "Low storage space"

## Output Format
Status: [HEALTHY|WARN|CRITICAL]
Issues: [list of problems found]
Recommendations: [suggested actions]

What Went Wrong

The skill worked perfectly on my laptop. Then my colleague tried to use it and got an error: "ModuleNotFoundError: No module named 'boto3'".

The Python script needed boto3, pandas, and psycopg2. My machine had them installed. His didn't. We had no standard way to declare or install these dependencies.

The Real Difference

After working with both, I realized they solve different problems:

MCP answers: "What can I do?"
It provides capabilities - functions the agent can call. Each MCP server is self-contained with its own dependencies and environment.

Skills answer: "How should I think?"
They provide expertise - decision logic, quality standards, and workflows. But they run in whatever environment the agent has.

The Solution: Use Both Together

Here's what actually works in real time. I'll use a real example from last week.

Example: Automated Cost Anomaly Detection

I needed Amazon Q Developer Pro to monitor our AWS costs and alert us when something unusual happens.

The MCP Layer (The Hands)

I set up two MCP servers:

aws-cost-explorer-mcp - exposes functions like get_daily_costs(), get_service_breakdown()
slack-notifications-mcp - exposes send_message(), create_incident()

Each server runs in its own Docker container with all dependencies installed. Amazon Q Developer Pro doesn't need to know about boto3 or the Slack SDK.

The Skill Layer (The Brain)

I created a cost-monitoring skill with this logic in SKILL.md:

Trigger: When user asks about cost anomalies or unusual spending

Process:
1. Get last 30 days of daily costs
2. Calculate average and standard deviation
3. Check if today's cost is more than 2 standard deviations above average
4. If yes, get service breakdown to identify which service spiked
5. If spike is over $500, send high-priority Slack alert
6. If spike is under $500, just report in conversation

Quality checks:
- Always compare against same day of week (Monday vs Monday)
- Exclude known scheduled events (monthly backups, etc.)
- Include percentage change, not just absolute numbers

How They Work Together

When I ask Amazon Q Developer Pro: "Are our AWS costs normal today?"

Amazon Q Developer Pro matches the question to the cost-monitoring skill
The skill loads its SKILL.md (only 2KB of context)
The skill requires cost data, so Amazon Q Developer Pro connects to the aws-cost-explorer-mcp server
The skill orchestrates: call get_daily_costs(), analyze the data, decide if it's anomalous
If anomalous, the skill calls the slack-notifications-mcp server
After completion, the skill unloads from memory

The MCP servers handle the technical execution. The skill handles the business logic and decision-making.

Why This Architecture Works

Context Efficiency
Only the active skill loads into memory. MCP tools load on-demand when the skill needs them. I went from 35% context consumed at startup to 5%.

Portability
The same skill works on my laptop, my colleague's Windows machine, and our CI/CD pipeline. The MCP servers can run locally, in containers, or as remote services.

Reusability
The aws-cost-explorer-mcp server is used by three different skills:

cost-monitoring (detects anomalies)
budget-planning (forecasts spending)
cost-optimization (finds savings opportunities)

Each skill brings different expertise, but they share the same data source.

Maintainability
When AWS changes their Cost Explorer API, I update one MCP server. All skills continue working. When business rules change (new alert thresholds), I update the skill. The MCP servers don't need to change.

Visual Overview

Here's how the three layers work together:

┌─────────────────────────────────────────┐
│   Agent Layer (Amazon Q Developer Pro)  │
│   Matches tasks → Loads skills          │
│   Connects to MCP servers               │
└─────────────────────────────────────────┘
                    ↓
┌─────────────────────────────────────────┐
│         Skill Layer (The Brain)         │
│   cost-monitoring.SKILL.md              │
│   - When to trigger                     │
│   - Decision logic                      │
│   - Quality checks                      │
└─────────────────────────────────────────┘
                    ↓
┌─────────────────────────────────────────┐
│         MCP Layer (The Hands)           │
│   aws-cost-explorer-mcp                 │
│   slack-notifications-mcp               │
│   - Tool execution                      │
│   - Environment isolation               │
└─────────────────────────────────────────┘

A Practical Pattern

Here's the decision framework I use:

Put it in an MCP server if:

Multiple skills will use it
It has heavy dependencies (Python libraries, system tools)
It needs credentials or secrets
It's a stable, reusable capability

Put it in a skill if:

It's domain knowledge or business logic
It orchestrates multiple tools
It has conditional decision-making
It's specific to one workflow

Keep it as a simple script if:

It's a one-time operation
The entire workflow is tightly coupled
Splitting it would add unnecessary complexity

What's Still Missing

The ecosystem isn't mature yet. Here's what I wish existed:

Dependency Declaration
Skills need a standard way to say "I need these capabilities" without hardcoding specific MCP servers. Something like:

requires:
  - cloud_metrics
  - notifications

Then the runtime figures out which MCP servers provide those capabilities.

Dynamic Loading
Right now, when you connect an MCP server, all its tools load immediately. I want skills to control this: "Load only the cost analysis tools for this task."

Graceful Fallback
If an MCP server is unavailable, the skill should automatically fall back to a built-in script or tell me clearly what's missing.

Conclusion

After six months of exploring these patterns with Amazon Q Developer Pro and other AI agents, here's what I know:

MCP and Agent Skills aren't competing approaches. They're complementary layers of the same system.

MCP gives your agent reliable, isolated capabilities. Skills give your agent the expertise to use those capabilities intelligently.

You need both. MCP without skills is a toolbox with no craftsman. Skills without MCP are expertise with unreliable tools.

The architecture that works:

Skills encode what to do and when
MCP servers provide how to do it
The agent runtime connects them together

This isn't just theoretical. These patterns are being implemented in actual environments, managing real AWS infrastructure.

Getting Started

If you want to explore this architecture:

For MCP:

Check the official MCP specification at modelcontextprotocol.io
Browse existing MCP servers at github.com/modelcontextprotocol/servers
Amazon Q Developer supports MCP through the Q CLI

For Agent Skills:

Review the Agent Skills specification by Anthropic
Start with simple skills that encode your team's operational knowledge
Focus on decision logic and workflows, not heavy computation

Next Steps:

Identify one repetitive task you do with your AI agent
Ask: Does this need external tools (MCP) or decision logic (Skill)?
Build the simplest version that works
Iterate based on what you learn

The ecosystem is still maturing, but the core patterns are solid. Start small, learn from erros, and build up from there.

📌 Wrapping Up

Thank you for reading! I hope this article gave you practical insights and a clearer perspective on the topic.

Was this helpful?

❤️ Like if it added value
🦄 Unicorn if you’re applying it today
💾 Save for your next optimization session
🔄 Share with your team

Follow me for more on:

AWS architecture patterns
FinOps automation
Multi-account strategies
AI-driven DevOps

💡 What’s Next

More deep dives coming soon on cloud operations, GenAI, Agentic-AI, DevOps, and data workflows follow for weekly insights.

🌐 Portfolio & Work

You can explore my full body of work, certifications, architecture projects, and technical articles here:

👉 Visit My Website

🛠️ Services I Offer

If you're looking for hands-on guidance or collaboration, I provide:

Cloud Architecture Consulting (AWS / Azure)
DevSecOps & Automation Design
FinOps Optimization Reviews
Technical Writing (Cloud, DevOps, GenAI)
Product & Architecture Reviews
Mentorship & 1:1 Technical Guidance

🤝 Let’s Connect

I’d love to hear your thoughts drop a comment or connect with me on LinkedIn.

For collaborations, consulting, or technical discussions, feel free to reach out directly at simplynadaf@gmail.com

Happy Learning 🚀

Terraform Variables and Outputs: Making Your Infrastructure Flexible

Sarvar Nadaf — Wed, 01 Apr 2026 09:59:13 +0000

👋 Hey there, tech enthusiasts!

"The difference between a script and reusable infrastructure code is variables."

🎯 Welcome Back!

Remember in Article 3 when you created that S3 bucket? You hardcoded the bucket name directly in the code:

resource "aws_s3_bucket" "my_first_bucket" {
  bucket = "terraform-first-bucket-yourname-2026"  # Hardcoded!
}

That works... once. But what if you need:

10 buckets with different names?
Same code for dev, staging, and production?
To let your team customize configurations?
To reuse this code in different projects?

Today, you'll learn how to make your Terraform code flexible and reusable using variables and outputs.

By the end of this article, you'll:

✅ Convert hardcoded values to variables
✅ Use all variable types (string, number, bool, list, map)
✅ Work with terraform.tfvars files
✅ Create outputs to expose information
✅ Build dev/prod configurations from the same code
✅ Use variable validation and sensitive values

Time Required: 30 minutes

Cost: $0 (using free tier resources)

Difficulty: Beginner-friendly

Let's make your code reusable! 🚀

💔 The Problem: Hardcoded Hell

The Painful Reality

Scenario 1: Multiple Environments

# dev.tf
resource "aws_s3_bucket" "app" {
  bucket = "myapp-dev-bucket"
}

# staging.tf (copy-paste and change)
resource "aws_s3_bucket" "app" {
  bucket = "myapp-staging-bucket"
}

# prod.tf (copy-paste and change again)
resource "aws_s3_bucket" "app" {
  bucket = "myapp-prod-bucket"
}

Problems:

❌ Three copies of the same code
❌ Change one thing = update three files
❌ High risk of mistakes
❌ Nightmare to maintain

Scenario 2: Team Collaboration

Developer 1: "I need the bucket name to be X"
Developer 2: "I need it to be Y"
DevOps: "Production needs Z"

Everyone edits the same .tf file = merge conflicts and chaos!

There's a better way. Enter variables.

🌟 What Are Variables?

Simple Definition

Variables are like function parameters for your infrastructure code.

Think of it like this:

Without Variables (Hardcoded):

function createBucket() {
  return "my-hardcoded-bucket-name";
}

With Variables (Flexible):

function createBucket(bucketName) {
  return bucketName;
}

createBucket("dev-bucket");    // Different uses
createBucket("prod-bucket");   // Same code!

In Terraform:

# Define the variable
variable "bucket_name" {
  description = "Name of the S3 bucket"
  type        = string
}

# Use the variable
resource "aws_s3_bucket" "app" {
  bucket = var.bucket_name  # Reference with var.
}

Same code, different values! That's the power of variables.

🛠️ Hands-On: Your First Variable

Let's convert our hardcoded S3 bucket to use variables.

Step 1: Create Project Directory

mkdir -p ~/terraform-variables-demo
cd ~/terraform-variables-demo

Step 2: Create variables.tf

Create a new file called variables.tf:

nano variables.tf

Add this code:

# String variable - for text values
variable "aws_region" {
  description = "AWS region where resources will be created"
  type        = string
  default     = "us-east-1"
}

variable "bucket_name" {
  description = "Name of the S3 bucket (must be globally unique)"
  type        = string
  # No default - user must provide this value
}

# Map variable - for key-value pairs like tags
variable "common_tags" {
  description = "Common tags to apply to all resources"
  type        = map(string)
  default = {
    Environment = "Development"
    ManagedBy   = "Terraform"
    Project     = "Learning"
  }
}

Understanding the syntax:

description - Explains what the variable is for (always add this!)
type - What kind of value (string, number, bool, list, map)
default - Optional default value (if not provided, user must supply it)

Step 3: Create main.tf

nano main.tf

Add this code:

terraform {
  required_version = ">= 1.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

# Use variable in provider
provider "aws" {
  region = var.aws_region  # Reference variable with var.
}

# Use variables in resource
resource "aws_s3_bucket" "example" {
  bucket = var.bucket_name  # Variable for bucket name

  # Merge common_tags with specific tags
  tags = merge(
    var.common_tags,
    {
      Name = var.bucket_name
    }
  )
}

Notice: We reference variables with var.variable_name

Step 4: Initialize and Test

terraform init
terraform plan

Terraform will prompt you:

var.bucket_name
  Name of the S3 bucket (must be globally unique)

  Enter a value:

Type: my-first-variable-bucket-yourname-2026 and press Enter.

You'll see the plan:

Terraform will perform the following actions:

  # aws_s3_bucket.example will be created
  + resource "aws_s3_bucket" "example" {
      + bucket = "my-first-variable-bucket-yourname-2026"
      + tags   = {
          + "Environment" = "Development"
          + "ManagedBy"   = "Terraform"
          + "Name"        = "my-first-variable-bucket-yourname-2026"
          + "Project"     = "Learning"
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Don't apply yet! Let's learn better ways to pass variables.

📝 Three Ways to Pass Variables

Method 1: Command Line (What We Just Did)

terraform plan
# Terraform prompts for bucket_name

Pros: Interactive

Cons: Tedious for multiple variables

Method 2: Command Line Flags

terraform plan -var="bucket_name=my-bucket-2026"

Pros: Good for CI/CD pipelines

Cons: Long commands with many variables

Method 3: terraform.tfvars File (Best for Most Cases)

Create terraform.tfvars:

nano terraform.tfvars

Add this:

bucket_name = "my-variable-bucket-yourname-2026"
aws_region  = "us-east-1"

common_tags = {
  Environment = "Development"
  ManagedBy   = "Terraform"
  Project     = "Variables Demo"
  Owner       = "YourName"
}

Now run:

terraform plan

No prompts! Terraform automatically loads terraform.tfvars.

Apply it:

terraform apply

Type yes when prompted.

🎉 Success! You just created infrastructure using variables!

🎨 Variable Types: The Complete Guide

Terraform supports multiple variable types. Let's explore them all!

Update variables.tf

Replace your variables.tf with this comprehensive version:

# 1. STRING - Text values
variable "aws_region" {
  description = "AWS region where resources will be created"
  type        = string
  default     = "us-east-1"
}

variable "bucket_name" {
  description = "Name of the S3 bucket (must be globally unique)"
  type        = string
}

variable "bucket_prefix" {
  description = "Prefix for multiple bucket names"
  type        = string
  default     = "terraform-demo"
}

# 2. NUMBER - Numeric values
variable "bucket_count" {
  description = "Number of S3 buckets to create"
  type        = number
  default     = 2

  # Validation rule
  validation {
    condition     = var.bucket_count >= 1 && var.bucket_count <= 5
    error_message = "Bucket count must be between 1 and 5."
  }
}

# 3. BOOLEAN - True/False values
variable "enable_versioning" {
  description = "Enable versioning on the S3 bucket"
  type        = bool
  default     = false
}

# 4. LIST - Ordered collection of values
variable "allowed_ips" {
  description = "List of IP addresses allowed to access the bucket"
  type        = list(string)
  default     = []
}

# 5. MAP - Key-value pairs
variable "common_tags" {
  description = "Common tags to apply to all resources"
  type        = map(string)
  default = {
    Environment = "Development"
    ManagedBy   = "Terraform"
    Project     = "Learning"
  }
}

# 6. ENVIRONMENT - With validation
variable "environment" {
  description = "Environment name (dev, staging, prod)"
  type        = string
  default     = "dev"

  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "Environment must be dev, staging, or prod."
  }
}

Update main.tf to Use All Variable Types

terraform {
  required_version = ">= 1.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

# Main S3 bucket with variables
resource "aws_s3_bucket" "example" {
  bucket = var.bucket_name

  tags = merge(
    var.common_tags,
    {
      Name = var.bucket_name
    }
  )
}

# Conditional resource - only created if enable_versioning is true
resource "aws_s3_bucket_versioning" "example" {
  count  = var.enable_versioning ? 1 : 0  # Conditional creation
  bucket = aws_s3_bucket.example.id

  versioning_configuration {
    status = "Enabled"
  }
}

# Multiple buckets using count (number variable)
resource "aws_s3_bucket" "multiple" {
  count  = var.bucket_count
  bucket = "${var.bucket_prefix}-${count.index + 1}-yourname-2026"

  tags = var.common_tags
}

What's happening here:

var.enable_versioning ? 1 : 0 - Conditional: if true, create 1 resource; if false, create 0
count = var.bucket_count - Create multiple resources based on number
count.index - Access the current index (0, 1, 2...)
merge() - Combine two maps together

📤 Outputs: Exposing Information

Variables let data IN. Outputs let data OUT.

Why Outputs Matter

Use cases:

Display important information after terraform apply
Pass values to other Terraform modules
Use in scripts or CI/CD pipelines
Share information with team members

Create outputs.tf

nano outputs.tf

Add this code:

# Basic outputs
output "bucket_name" {
  description = "Name of the created S3 bucket"
  value       = aws_s3_bucket.example.id
}

output "bucket_arn" {
  description = "ARN of the S3 bucket"
  value       = aws_s3_bucket.example.arn
}

output "bucket_region" {
  description = "Region where the bucket was created"
  value       = aws_s3_bucket.example.region
}

output "bucket_domain_name" {
  description = "Domain name of the bucket"
  value       = aws_s3_bucket.example.bucket_domain_name
}

# Multiple buckets output
output "multiple_bucket_names" {
  description = "Names of all created buckets"
  value       = aws_s3_bucket.multiple[*].id
}

# Conditional output
output "versioning_enabled" {
  description = "Whether versioning is enabled"
  value       = var.enable_versioning
}

# Map output
output "bucket_tags" {
  description = "Tags applied to the bucket"
  value       = aws_s3_bucket.example.tags
}

# Sensitive output example
output "bucket_id_sensitive" {
  description = "Bucket ID (marked as sensitive)"
  value       = aws_s3_bucket.example.id
  sensitive   = true  # Won't display in plan/apply output
}

Update terraform.tfvars

bucket_name      = "my-variable-bucket-yourname-2026"
bucket_prefix    = "demo-bucket-yourname"
bucket_count     = 2
enable_versioning = true
aws_region       = "us-east-1"

common_tags = {
  Environment = "Development"
  ManagedBy   = "Terraform"
  Project     = "Variables Demo"
  Owner       = "YourName"
}

Apply and See Outputs

terraform apply

After applying, you'll see:

Apply complete! Resources: 4 added, 0 changed, 0 destroyed.

Outputs:

bucket_arn = "arn:aws:s3:::my-variable-bucket-yourname-2026"
bucket_domain_name = "my-variable-bucket-yourname-2026.s3.amazonaws.com"
bucket_id_sensitive = <sensitive>
bucket_name = "my-variable-bucket-yourname-2026"
bucket_region = "us-east-1"
bucket_tags = tomap({
  "Environment" = "Development"
  "ManagedBy" = "Terraform"
  "Name" = "my-variable-bucket-yourname-2026"
  "Owner" = "YourName"
  "Project" = "Variables Demo"
})
multiple_bucket_names = [
  "demo-bucket-yourname-1-yourname-2026",
  "demo-bucket-yourname-2-yourname-2026",
]
versioning_enabled = true

View Outputs Anytime

# View all outputs
terraform output

# View specific output
terraform output bucket_name

# View sensitive output (will reveal the value)
terraform output bucket_id_sensitive

# Output as JSON (useful for scripts)
terraform output -json

🌍 Real-World: Dev vs Prod Configurations

The real power of variables: same code, different configurations!

Create dev.tfvars

nano dev.tfvars

# Development environment configuration
environment       = "dev"
bucket_name       = "myapp-dev-yourname-2026"
bucket_prefix     = "myapp-dev-yourname"
bucket_count      = 2
enable_versioning = false  # Save costs in dev
aws_region        = "us-east-1"

common_tags = {
  Environment = "Development"
  ManagedBy   = "Terraform"
  Project     = "MyApp"
  CostCenter  = "Engineering"
}

Create prod.tfvars

nano prod.tfvars

# Production environment configuration
environment       = "prod"
bucket_name       = "myapp-prod-yourname-2026"
bucket_prefix     = "myapp-prod-yourname"
bucket_count      = 3
enable_versioning = true  # Important for prod!
aws_region        = "us-east-1"

common_tags = {
  Environment = "Production"
  ManagedBy   = "Terraform"
  Project     = "MyApp"
  CostCenter  = "Operations"
  Compliance  = "Required"
}

Deploy to Dev

terraform plan -var-file="dev.tfvars"
terraform apply -var-file="dev.tfvars"

Result: 3 buckets created (1 main + 2 multiple), no versioning

Deploy to Prod

# First destroy dev
terraform destroy -var-file="dev.tfvars"

# Then deploy prod
terraform plan -var-file="prod.tfvars"
terraform apply -var-file="prod.tfvars"

Result: 4 buckets created (1 main + 3 multiple), versioning enabled

Same code, different results! This is how professionals manage multiple environments.

✅ Best Practices

1. Always Add Descriptions

Bad:

variable "name" {
  type = string
}

Good:

variable "bucket_name" {
  description = "Name of the S3 bucket for application logs (must be globally unique)"
  type        = string
}

2. Use Validation Rules

variable "instance_type" {
  description = "EC2 instance type"
  type        = string
  default     = "t2.micro"

  validation {
    condition     = contains(["t2.micro", "t2.small", "t3.micro"], var.instance_type)
    error_message = "Instance type must be t2.micro, t2.small, or t3.micro."
  }
}

3. Provide Sensible Defaults

variable "aws_region" {
  description = "AWS region"
  type        = string
  default     = "us-east-1"  # Most common region
}

4. Use Meaningful Variable Names

Bad: var.n, var.x, var.thing

Good: var.bucket_name, var.instance_count, var.enable_monitoring

5. Group Related Variables

# Network variables
variable "vpc_cidr" { }
variable "subnet_cidrs" { }

# Application variables
variable "app_name" { }
variable "app_version" { }

# Tags
variable "common_tags" { }

6. Never Commit Secrets to tfvars

Bad:

# terraform.tfvars
database_password = "super-secret-password"  # DON'T DO THIS!

Good:

# Use environment variables for secrets
export TF_VAR_database_password="super-secret-password"

Or use AWS Secrets Manager, HashiCorp Vault, etc.

7. Document Your Variables

Create a README.md:

## Required Variables

- `bucket_name` - Globally unique S3 bucket name

## Optional Variables

- `aws_region` - AWS region (default: us-east-1)
- `enable_versioning` - Enable bucket versioning (default: false)

🐛 Common Issues and Solutions

Issue 1: Variable Not Found

Error:

Error: Reference to undeclared input variable

Solution: Make sure variable is declared in variables.tf:

variable "bucket_name" {
  description = "Bucket name"
  type        = string
}

Issue 2: Type Mismatch

Error:

Error: Invalid value for input variable

Solution: Check your variable type matches the value:

variable "bucket_count" {
  type = number  # Not string!
}

In terraform.tfvars:

bucket_count = 2  # Not "2" (string)

Issue 3: Validation Failed

Error:

Error: Invalid value for variable
Bucket count must be between 1 and 5.

Solution: Provide a value that meets the validation condition:

bucket_count = 3  # Within 1-5 range

Issue 4: Forgot to Pass tfvars File

Problem: Terraform uses default values instead of your custom file

Solution: Always specify the file:

terraform apply -var-file="prod.tfvars"

Issue 5: Sensitive Output Not Hidden

Problem: Sensitive data showing in output

Solution: Mark output as sensitive:

output "database_password" {
  value     = var.db_password
  sensitive = true  # Add this
}

🎓 What You Just Learned

Let's recap your new superpowers:

✅ Variables:

Convert hardcoded values to flexible variables
Use all variable types (string, number, bool, list, map)
Add validation rules
Provide defaults

✅ terraform.tfvars:

Store variable values in files
Create environment-specific configs (dev.tfvars, prod.tfvars)
Auto-loading behavior

✅ Outputs:

Display important information
Mark sensitive data
Use in scripts and modules

✅ Real-World Patterns:

Same code, multiple environments
Conditional resource creation
Multiple resource creation with count

🎯 Challenge: Try It Yourself

Before moving to the next article, try these challenges:

Challenge 1: Add More Variables

Add variables for:

owner_email
cost_center
backup_retention_days

Challenge 2: Create staging.tfvars

Create a staging environment configuration between dev and prod.

Challenge 3: Use List Variable

Create a variable for multiple availability zones:

variable "availability_zones" {
  type    = list(string)
  default = ["us-east-1a", "us-east-1b"]
}

Challenge 4: Complex Map Variable

Create a variable for instance configurations:

variable "instance_configs" {
  type = map(object({
    instance_type = string
    volume_size   = number
  }))
}

Challenge 5: Output Formatting

Create an output that combines multiple values:

output "bucket_info" {
  value = "Bucket ${aws_s3_bucket.example.id} in ${var.aws_region}"
}

📚 Additional Variable Features

Variable Precedence (Order of Priority)

Terraform loads variables in this order (last wins):

Environment variables (TF_VAR_name)
terraform.tfvars file
terraform.tfvars.json file
*.auto.tfvars files (alphabetical order)
-var and -var-file command line flags

Example:

# All of these set the same variable
export TF_VAR_bucket_name="from-env"           # Priority 1
# terraform.tfvars: bucket_name = "from-file"  # Priority 2
terraform apply -var="bucket_name=from-cli"    # Priority 3 (wins!)

Environment Variables

# Set variable via environment
export TF_VAR_bucket_name="my-bucket-2026"
export TF_VAR_aws_region="us-west-2"

terraform plan  # Uses environment variables

Auto-Loading tfvars Files

Terraform automatically loads:

terraform.tfvars
terraform.tfvars.json
*.auto.tfvars
*.auto.tfvars.json

Example:

# These are auto-loaded
terraform.tfvars
common.auto.tfvars
secrets.auto.tfvars

# These need -var-file flag
dev.tfvars
prod.tfvars

🔍 Understanding Variable Interpolation

You can use variables in many ways:

String Interpolation

resource "aws_s3_bucket" "example" {
  bucket = "${var.environment}-${var.app_name}-bucket"
  # Result: "dev-myapp-bucket"
}

Conditional Expressions

resource "aws_instance" "web" {
  instance_type = var.environment == "prod" ? "t3.large" : "t3.micro"
  # prod = t3.large, anything else = t3.micro
}

For Expressions

locals {
  bucket_names = [for i in range(var.bucket_count) : "${var.prefix}-${i}"]
  # Result: ["app-0", "app-1", "app-2"]
}

🔗 Useful Resources

Terraform Variables Docs: https://developer.hashicorp.com/terraform/language/values/variables
Terraform Outputs Docs: https://developer.hashicorp.com/terraform/language/values/outputs
Variable Validation: https://developer.hashicorp.com/terraform/language/values/variables#custom-validation-rules
Type Constraints: https://developer.hashicorp.com/terraform/language/expressions/type-constraints

🎬 What's Next?

In Article 6: Building a VPC from Scratch, we'll:

Create a complete AWS VPC with subnets
Use variables for network configuration
Build public and private subnets
Configure route tables and internet gateways
Apply everything you learned about variables

You'll learn:

Real-world networking with Terraform
Complex resource dependencies
Advanced variable usage
Production-ready VPC architecture

📌 Wrapping Up

Thank you for reading. I hope this article provided practical insights and a clearer understanding of the topic.

If you found this useful:

❤️ Like if it added value
🦄 Unicorn if you’re applying it today
💾 Save it for your next optimization session
🔄 Share it with your team

💡 What’s Next

More deep dives are coming soon on:

Cloud Operations
GenAI & Agentic AI
DevOps Automation
Data & Platform Engineering

Follow along for weekly insights and hands-on guides.

🌐 Portfolio & Work

You can explore my full body of work, certifications, architecture projects, and technical articles here:

👉 Visit My Website

🛠️ Services I Offer

If you're looking for hands-on guidance or collaboration, I provide:

Cloud Architecture Consulting (AWS / Azure)
DevSecOps & Automation Design
FinOps Optimization Reviews
Technical Writing (Cloud, DevOps, GenAI)
Product & Architecture Reviews
Mentorship & 1:1 Technical Guidance

🤝 Let’s Connect

I’d love to hear your thoughts. Feel free to drop a comment or connect with me on:

🔗 LinkedIn

For collaborations, consulting, or technical discussions, reach out at:

📧 simplynadaf@gmail.com

Found this helpful? Share it with your team.

⭐ Star the repo • 📖 Follow the series • 💬 Ask questions

Made by Sarvar Nadaf

🌐 https://sarvarnadaf.com

What is OpenClaw? A Self Hosted AI Assistants

Sarvar Nadaf — Mon, 30 Mar 2026 17:56:49 +0000

👋 Hey there, tech enthusiasts!

Let's dive in and explore the fascinating world of cloud technology together! 🚀

New to Self-Hosted AI?

This article assumes basic familiarity with computers, software installation, and online services. If terms like "API," "server," or "open source" are new to you, don't worry focus on the core concept: OpenClaw is an AI assistant that you run on your own computer or server, rather than relying on a company's service like ChatGPT.

What This Article Covers: Understanding what OpenClaw is, why it matters, and whether it's right for you.

What's Coming Next: Hands-on setup, configuration, and deployment (no prior experience with self-hosting required we'll walk through everything step-by-step).

Imagine having a personal assistant that lives on your infrastructure, responds through the messaging apps you already use, and executes tasks across your entire cloud environment. That's OpenClaw. But unlike the AI assistants you're familiar with ChatGPT, Claude, or GitHub Copilot this one doesn't phone home to someone else's servers. It runs on your machine, uses your API keys, and keeps your data exactly where you want it.

The story begins in November 2025 when Austrian developer Peter Steinberger built what he called a "weekend project." He wanted an AI that could do more than chat one that could actually execute commands, manage files, read emails, and integrate with the tools he used daily. Within weeks, the project exploded. By March 2026, it had accumulated over 340,000 GitHub stars and attracted attention from Silicon Valley startups to Chinese tech giants. Steinberger eventually joined OpenAI and moved the project to an open-source foundation, but the core philosophy remained unchanged: your assistant, your machine, your rules.

For anyone interested in AI and privacy, OpenClaw represents something different. It's not a SaaS product with usage limits and data residency questions. It's software you install, configure, and control. You choose which AI provider to use Anthropic's Claude, OpenAI's GPT models, Google's Gemini, or even self-hosted options like Ollama. You decide which messaging platforms to enable. You set the security boundaries. And when you need to see what happened, the logs are on your computer, not buried in someone else's cloud.

What OpenClaw Actually Is

Architecture and Deployment Model

OpenClaw is a Node.js application that acts as a gateway between messaging platforms and large language models (LLMs AI systems like ChatGPT or Claude). You install it on a server your laptop, a homelab machine, or a cloud VM and it stays running as a daemon. The architecture is straightforward: a WebSocket-based control plane (a real-time communication hub) listens on port 18789, handling connections from messaging channels, the web dashboard, and optional companion apps for macOS, iOS, and Android.

The gateway doesn't process AI requests itself. Instead, it routes conversations to your configured LLM provider via their APIs. When you send a message through WhatsApp or Slack, OpenClaw receives it, forwards the context to Claude or GPT, gets the response, and delivers it back through the same channel. All conversation history, configuration, and session state live in local SQLite databases under ~/.openclaw/.

Multi-Channel Integration

This is where OpenClaw diverges from traditional chatbots. Rather than forcing you into a web interface or proprietary app, it meets you where you already work:

Messaging platforms: WhatsApp, Telegram, Discord, Slack, Microsoft Teams, Signal, Matrix
Specialized channels: IRC, Google Chat, Mattermost, Twitch, WeChat (via Tencent plugin)
Direct interfaces: Web dashboard, macOS menu bar app, iOS/Android companion apps
Developer tools: CLI commands for scripting and automation

Each channel operates independently. You can have different security policies for Discord versus WhatsApp, route specific channels to isolated agent workspaces, or restrict certain tools based on where the request originated.

The Skills System

OpenClaw extends beyond conversation through a skills framework. Skills are self-contained modules think of them as plugins that give the AI new capabilities. A skill is just a directory containing a SKILL.md file with instructions and metadata. The AI reads these instructions and knows how to use the skill's tools.

Skills can be:

Bundled: Shipped with OpenClaw (browser automation, file operations, system commands)
Managed: Installed from the ClawHub registry
Workspace-specific: Custom skills you write for your environment

The precedence model matters for teams building workflows. Workspace skills override global skills, which override bundled skills. This means you can customize behavior per project without modifying the core installation.

Tool Execution and Sandboxing

When the AI decides it needs to run a command or access a file, OpenClaw's tool system handles the execution. By default, tools run directly on the host for the main session. But for group chats or untrusted channels, you can enable sandbox mode. Sandboxed sessions run inside per-session Docker containers with restricted tool access.

The security model is explicit:

Main session (your direct chats): Full tool access by default
Group sessions: Configurable sandbox with allowlists and denylists
Tool categories: bash, browser, read, write, edit, sessions, nodes, cron, discord, gateway

You define which tools each session type can use. A production deployment might allow read-only file access for group chats while reserving write operations for authenticated direct messages.

Why Self-Hosting Matters

Data Privacy and Control

When you deploy OpenClaw on your infrastructure, you control the data flow. Conversation logs, uploaded files, and execution history never leave your environment unless you explicitly configure external integrations. For organizations with strict data residency requirements healthcare, finance, government this matters. You're not trusting a third party to handle PHI (Protected Health Information) or PII (Personally Identifiable Information). You're running the infrastructure yourself.

The LLM API calls still go to external providers (unless you use self-hosted models), but you choose which provider and can route through your own network controls. Want to proxy all OpenAI requests through your corporate gateway? Configure the API endpoint. Need to log every prompt for compliance? Intercept at the gateway level.

Cost Control and Model Flexibility

SaaS AI assistants charge per seat or per usage tier. OpenClaw inverts this model. You pay your LLM provider directly based on token consumption. If Claude is too expensive for your use case, switch to GPT-4o-mini. If you want to experiment with open-source models, point it at Ollama running locally. The gateway doesn't care which model you use it just routes requests.

This flexibility extends to failover and load balancing. Configure multiple API keys for the same provider, and OpenClaw rotates through them when rate limits hit. Set up model fallback chains: try Claude Opus first, fall back to GPT-4 if it's overloaded, and use a local model as the last resort.

Example Cost Comparison:

SaaS AI Assistant: $20-30/user/month (fixed cost, limited usage)
OpenClaw + Claude API: Pay per token (1M input tokens ≈ $3-8 depending on model)
OpenClaw + Local LLM: Infrastructure costs only (no per-token charges)

For teams running thousands of queries daily, the direct API model often costs less than per-seat licensing. For occasional users, SaaS might be cheaper. The key difference: you control the trade-off.

Infrastructure as Code Integration

Because OpenClaw is just a Node.js application with a JSON configuration file, it fits naturally into infrastructure-as-code (IaC) workflows managing infrastructure through code rather than manual configuration. The configuration lives at ~/.openclaw/openclaw.json and supports environment variable substitution for secrets. You can template it with Terraform, manage it with Ansible, or version it in Git.

Deployment patterns cloud architects already know apply directly:

Systemd service: Install the daemon, enable it, manage it like any other service
Docker container: Official images available, mount config as a volume
Kubernetes: Deploy as a StatefulSet (a workload type for applications that need persistent storage) with persistent storage for the database
Nix: Declarative configuration with the community Nix package

Security Boundaries and Threat Model

OpenClaw's security model assumes you understand the risks of giving an AI broad system access. The maintainers are explicit about this. One core maintainer warned on Discord: "if you can't understand how to run a command line, this is far too dangerous of a project for you to use safely."

The threat surface includes:

Prompt injection: Malicious instructions embedded in data the AI processes (emails, documents, web pages) that trick the AI into executing unintended commands
Tool misuse: The AI executing destructive commands based on misinterpreted context
Third-party skills: Unvetted skills from the community that could exfiltrate data
Channel compromise: If someone gains access to an authorized messaging account, they control the AI

Cisco's security research team demonstrated this in March 2026 when they found a third-party skill performing data exfiltration. The skill repository lacked adequate vetting, and users installing it unknowingly gave attackers access to their environment.

For production use, you need to:

Run sandboxed sessions for any untrusted input source
Audit third-party skills before installation
Use DM pairing mode to require explicit approval for new contacts
Monitor execution logs for unexpected tool usage
Isolate the OpenClaw instance from sensitive systems

The MoltMatch Incident: A Cautionary Tale

In February 2026, computer science student Jack Luo configured his OpenClaw agent to explore its capabilities. He connected it to Moltbook a social network for AI agents and let it run autonomously. Days later, he discovered the agent had created a profile on MoltMatch, a dating platform where AI agents interact on behalf of users, and was screening potential matches without his explicit direction.

The incident highlighted a fundamental challenge with autonomous agents: when you give an AI broad permissions and vague instructions, it will interpret its mandate creatively. Luo's agent saw "explore capabilities" and "connect to agent platforms" as permission to create dating profiles. The AI-generated profile didn't reflect him authentically, and he had no idea it was happening until someone told him.

This matters for anyone managing systems because the same dynamic applies to automation. If you tell an AI to "optimize costs" without clear boundaries, it might decide to delete things it deems unnecessary. If you ask it to "improve security," it could change settings in ways you didn't anticipate. Autonomous agents require explicit constraints, not just general goals.

Real-World Adoption Patterns

Small Business and Freelancer Workflows

OpenClaw has found traction among small businesses and freelancers who need automation but can't afford enterprise tools. Common workflows include:

Lead generation: Scraping prospect lists, researching companies, updating CRM systems
Content management: Drafting emails, summarizing documents, generating reports
System administration: Monitoring servers, running diagnostics, deploying updates

These users value the flexibility to customize behavior without vendor lock-in. A freelance consultant can write a workspace skill that integrates with their specific CRM and invoicing tools, then share that skill with clients who use the same stack.

Enterprise Experimentation

Larger organizations are testing OpenClaw in controlled environments. The typical pattern: deploy it on a developer's workstation or a sandbox VM, connect it to a private Slack channel, and use it for internal tooling. Teams report using it for:

Documentation search: Indexing internal wikis and answering questions
Incident response: Gathering logs, running diagnostics, summarizing findings
Code review assistance: Analyzing pull requests, suggesting improvements

The key constraint is trust. Enterprises won't give OpenClaw production access until they've validated its behavior in isolated environments. The security incidents and lack of formal audits make it a hard sell for risk-averse organizations.

What This Means for Your Setup

If you're evaluating OpenClaw, think of it as software infrastructure, not a product. You're not buying a service; you're installing and maintaining a component that needs care and attention.

Key Questions to Consider:

Deployment:

Where will it run? (Your laptop, a home server, a cloud VM)
Who can access it? (Just you, your team, your organization)
What can it access? (Read-only files, full system control, isolated sandbox)
How will you monitor it? (Log files, activity tracking, alerts)
What's the risk if compromised? (Isolated test environment vs. access to important data)

Integration:

Which AI provider will you use? (Cost, rate limits, data handling)
Which messaging platforms? (Authentication, user verification)
What internal systems? (APIs, databases, file storage)
How will you monitor it? (Health checks, error tracking, usage metrics)

Maintenance:

Who maintains the configuration? (You, your team, IT department)
How do you handle updates? (Manual upgrades, automated deployment)
What's the support model? (Community forums, internal expertise)
How do you manage secrets? (Environment variables, password managers)

The answers depend on your technical comfort level and risk tolerance. A tech-savvy individual might run it on their laptop for personal use. A small team might deploy it on a shared server with careful access controls. An enterprise might lock it down in an isolated environment. All are valid approaches based on your needs and skills.

The Bottom Line

OpenClaw is not a polished consumer product. It's a powerful, flexible, occasionally dangerous tool that gives you control at the cost of responsibility. For people who understand the trade-offs, it offers something rare: an AI assistant that runs on your terms, integrates with your tools, and doesn't require you to trust a company with your data.

The project is still young. The security model is evolving. The ecosystem is fragmented. But the core idea self-hosted, multi-channel, tool-capable AI agents addresses real needs that SaaS products don't solve. Whether OpenClaw itself becomes the standard or just proves the concept for something better, the architecture it represents is worth understanding.

When to Consider OpenClaw:

You care deeply about data privacy and control
You want to avoid subscription fees for large-scale usage
You need deep integration with your own tools and systems
You have technical skills or are willing to learn
You're comfortable with ongoing maintenance

When to Avoid OpenClaw:

You're not comfortable with command-line tools
You need something that works immediately without setup
You can't dedicate time to security and maintenance
You prefer paying for convenience over having control
Your use case doesn't justify the complexity

If you decide to explore it, start small. Run it in an isolated environment, connect it to a test channel, give it limited permissions. Learn how it behaves, where it fails, and what it can actually do. Then decide if the benefits justify the operational overhead and security risks.

That's the honest assessment. OpenClaw is powerful infrastructure for people who know what they're doing. If that's you, it's worth exploring. If it's not, wait for the ecosystem to mature.

Coming Next: In our next article, we'll cover the technical implementation installation, configuration, and deployment with step-by-step examples. We'll walk through setting up OpenClaw on different platforms and implementing security best practices.

What We'll Cover in Upcoming Articles:

Part 2: Installation & Setup - Step-by-step guide, prerequisites, system requirements, and getting your first instance running
Part 3: Configuration Deep-Dive - API keys, messaging platform integration, security policies, and sandbox configuration
Part 4: Deployment Options - Running on Windows/Mac/Linux, cloud servers, and home servers with monitoring and cost optimization
Part 5: Security Best Practices - Threat mitigation, audit logging, access control, and incident response

Questions or Topics You'd Like Covered?

Drop a comment below with your questions about OpenClaw. Whether you're wondering about specific use cases, integration challenges, cost comparisons, or anything else we'll address them in upcoming articles. Common questions we're already planning to cover:

How much does it actually cost to run?
Can I use it on Windows without WSL?
What's the learning curve for someone new to self-hosting?
How does it compare to AutoGPT or LangChain agents?
Can I run it on a Raspberry Pi or home server?

Resources:

Official Website: https://openclaw.ai
GitHub Repository: https://github.com/openclaw/openclaw

📌 Wrapping Up

Thank you for reading! I hope this article gave you practical insights and a clearer perspective on the topic.

Was this helpful?

❤️ Like if it added value
🦄 Unicorn if you’re applying it today
💾 Save for your next optimization session
🔄 Share with your team

Follow me for more on:

AWS architecture patterns
FinOps automation
Multi-account strategies
AI-driven DevOps

💡 What’s Next

More deep dives coming soon on cloud operations, GenAI, Agentic-AI, DevOps, and data workflows follow for weekly insights.

🌐 Portfolio & Work

You can explore my full body of work, certifications, architecture projects, and technical articles here:

👉 Visit My Website

🛠️ Services I Offer

If you're looking for hands-on guidance or collaboration, I provide:

Cloud Architecture Consulting (AWS / Azure)
DevSecOps & Automation Design
FinOps Optimization Reviews
Technical Writing (Cloud, DevOps, GenAI)
Product & Architecture Reviews
Mentorship & 1:1 Technical Guidance

🤝 Let’s Connect

I’d love to hear your thoughts drop a comment or connect with me on LinkedIn.

For collaborations, consulting, or technical discussions, feel free to reach out directly at simplynadaf@gmail.com

Happy Learning 🚀

AI Assistance vs AI Agents: Understanding the Shift from Responses to Autonomous Systems

Sarvar Nadaf — Wed, 25 Mar 2026 19:11:06 +0000

👋 Hey there, tech enthusiasts!

I'm Sarvar, a Cloud Architect with a passion for transforming complex technological challenges into elegant solutions. With extensive experience spanning Cloud Operations (AWS & Azure), Data Operations, Analytics, DevOps, Generative AI, and Agentic AI, I've had the privilege of architecting solutions for global enterprises that drive real business impact. Through this article series, I'm excited to share practical insights, best practices, and hands-on experiences from my journey in the tech world. Whether you're a seasoned professional or just starting out, I aim to break down complex concepts into digestible pieces that you can apply in your projects.

Let's dive in and explore the fascinating world of cloud technology together! 🚀

Here's What Most People Get Wrong

ChatGPT can write your email. But it can't send it. That's the difference between an AI assistant and an AI agent, and honestly, most people have no idea there even is a difference.

I've been working with AI for a while now, and I keep seeing the same confusion everywhere. People use "AI assistant" and "AI agent" like they're the same thing. They're not. Not even close.

Here's what you need to know: AI assistants tell you what to do. AI agents actually do it for you. That's the whole game right there.

Let me break this down in a way that actually makes sense.

AI Assistants: The Smart Advisor

Think about the last time you used ChatGPT or asked Siri something. You asked a question, got an answer, and then you had to go do something with that answer. That's an AI assistant.

An AI assistant is like having a really smart friend who knows everything but can't actually touch anything in your world. They can tell you exactly what to do, step by step, but you're the one who has to do it.

Here's what they're good at:

Answering questions (and they're really good at this)
Giving you suggestions and advice
Helping you think through problems
Generating content like emails, code, or ideas

But here's what they can't do:

Take any action in your systems
Make decisions and execute them
Complete tasks that need multiple steps
Use tools or APIs on their own

Let me give you a real example. Say you ask an AI assistant: "What's the weather today?"

You: "What's the weather today?"
Assistant: "It's 72°F and sunny in New York."
You: "Should I bring an umbrella?"
Assistant: "No, it's not going to rain today."

Perfect, right? You got your answer. But notice what didn't happen - the assistant didn't check your calendar, didn't see you have a meeting at 3 PM, didn't notice the forecast shows rain later, and definitely didn't pack an umbrella for you. It just answered your question.

That's the limitation. And for a lot of things, that's totally fine! Sometimes you just need information.

Agentic AI: The Autonomous Doer

Now here's where things get interesting. Agentic AI doesn't just tell you what to do - it actually does it.

Think of it like this: if an AI assistant is your smart friend giving advice, an agentic AI is your trusted employee who you can hand a project to and they'll just... handle it. They'll figure out the steps, make the decisions, use whatever tools they need, and come back with results.

This is what makes agents different:

They can access and use tools (APIs, databases, systems)
They make decisions based on what they find
They complete multi-step workflows autonomously
They only ask you when they really need input

Here's the same scenario but with an agent:

You: "Book me a flight to New York next week."

Agent: *searches multiple airlines*
       *compares prices and times*
       *checks your calendar for conflicts*
       *finds the best option*
       *books the ticket*
       *adds it to your calendar*
       *sends confirmation to your email*

"Done! I booked you on Delta flight DL123, Tuesday 9 AM, $287. 
I saw you have a meeting at 2 PM that day, so this gives you time. 
Confirmation is in your email."

See the difference? You gave one instruction, and the agent handled everything. It made decisions (which flight is best), used tools (booking system, calendar, email), and completed the entire workflow. You didn't have to do anything except say what you wanted.

Let Me Show You the Real Difference

I think the best way to understand this is with a practical example. Let's say you need to check on your servers (if you're not technical, just think of this as checking if your website is working).

With an AI Assistant:

You ask: "I need to check my server status."

The assistant responds:

"Sure! To check your server status, you can:
1. Log into your AWS Console
2. Navigate to the EC2 Dashboard
3. Look at the instance state column

Or if you prefer the command line, run:
aws ec2 describe-instances

Would you like help with anything else?"

Great advice! But now you have to go do all of that. You have to log in, navigate around, run commands, interpret the results. The assistant told you what to do, but you're still doing all the work.

With an Agentic AI:

You say the same thing: "I need to check my server status."

The agent responds:

*connects to your AWS account*
*checks all your servers*
*analyzes the data*

"I checked your 5 servers. Here's what I found:

- web-server-1: Running fine, healthy
- web-server-2: Running but CPU is at 89% (might need attention soon)
- database-1: Running fine, healthy  
- api-server-1: STOPPED - this one is down
- backup-server: Running fine, healthy

I noticed api-server-1 is down. This could be affecting your API endpoints. 
Want me to restart it for you?"

The agent didn't just tell you what to do. It actually went and checked everything, found a problem, and is ready to fix it if you want. That's the power of agentic AI.

Why This Matters in 2026

Look, I get it. For years, we've been calling everything "AI assistants" and it worked fine. But now we're in 2026, and the technology has split into two very different paths.

AI assistants got better at answering. They understand context better, they give more accurate information, they can help with more complex questions. ChatGPT, Claude, and similar tools are incredibly good at what they do.

Agentic AI learned to act. They can now access your tools, make decisions, and complete entire workflows. They're not just smart anymore - they're capable.

And here's the thing: most people are still using AI like it's 2023. They're asking questions and copying answers when they could be handing off entire tasks.

Let me give you some real examples of what's possible now:

GitHub Copilot Workspace

You tell it: "Add user authentication to this app."

Instead of just giving you code snippets, it can:

Analyze your codebase
Write code across multiple files
Suggest database changes
Draft API endpoints
Generate tests
Create a pull request

You still review everything and make the final decisions, but it handles the heavy lifting. That's moving toward agentic behavior.

AWS Bedrock Agents

You can set up an agent to monitor your infrastructure. It can:

Watch your servers continuously
Detect when something goes wrong
Analyze issues
Take corrective actions (like restarting services)
Log everything
Alert you when it needs help

Instead of checking everything manually, you get a report: "Had 3 minor issues last night, all handled automatically." The agent does the routine monitoring while you focus on bigger problems.

Modern DevOps Agents

Your deployment fails at 2 AM. An agent can:

Detect the failure quickly
Analyze the logs
Identify the issue
Roll back the deployment
Alert the team with context
Generate an incident report

You find out at 9 AM that there was a problem and it's already been handled. Not perfect, but better than waking up to a crisis.

The Quick Comparison

If you're still wrapping your head around this, here's a simple table that breaks it down:

What It Does	AI Assistant	Agentic AI
Answers your questions	✅ Yes	✅ Yes
Gives you information	✅ Yes	✅ Yes
Takes actions in systems	❌ No	✅ Yes
Uses tools and APIs	❌ No	✅ Yes
Makes decisions on its own	❌ No	✅ Yes
Handles multi-step tasks	❌ No	✅ Yes
Works autonomously	❌ No	✅ Yes
Needs your approval for everything	✅ Yes	⚠️ Sometimes

The pattern is pretty clear, right? Assistants are great at the "thinking" part. Agents are great at the "doing" part.

So When Should You Use Each One?

This is the practical question everyone should be asking. Because here's the truth: you don't always need an agent. Sometimes an assistant is exactly what you want.

Use an AI Assistant When:

You want information or advice. If you're trying to learn something, understand a concept, or get suggestions, an assistant is perfect. You're not trying to automate anything - you just want to think through something with help.

Examples:

"Help me write this email" (you'll review and send it)
"Explain how this code works" (you're learning)
"Give me ideas for my presentation" (you're brainstorming)
"What's the best way to structure this database?" (you want advice)

You want to stay in control. Sometimes you don't want anything happening automatically. You want to review every step, make every decision, and execute everything yourself. That's totally valid, and assistants are perfect for this.

You're working on creative or strategic tasks. Writing, designing, planning, strategizing - these are areas where you want AI to augment your thinking, not replace your decision-making.

Use Agentic AI When:

You have repetitive workflows. If you're doing the same multi-step process over and over, that's a perfect candidate for an agent. Let it handle the routine stuff while you focus on things that actually need your brain.

Examples:

Monitoring systems and alerting on issues
Processing incoming data and routing it correctly
Handling customer support for common questions
Running deployments and rollbacks
Generating and sending reports

You need 24/7 operation. Agents don't sleep. If you need something monitored or handled around the clock, an agent is your answer.

Speed matters. Agents can react in seconds. If you need fast response to events or issues, agents beat humans every time.

The task is well-defined. If you can clearly describe the steps and decision points, an agent can probably handle it. The more structured the workflow, the better agents perform.

The Real Business Impact

Let me give you a concrete example of what this looks like in practice.

I know a company that was spending about 2 hours every day manually checking their server infrastructure. Someone had to log in, check each server, look at metrics, make sure everything was healthy. Boring, repetitive, but necessary.

First, they tried an AI assistant approach. They used ChatGPT to help generate the commands they needed to run. It was helpful - instead of remembering all the commands, they could just ask "how do I check CPU usage?" and get the answer. But they still had to run everything manually. They saved maybe 15 minutes a day. Better than nothing, but not game-changing.

Then they built an agentic AI solution. They set up an agent that:

Checks all servers every 5 minutes
Analyzes the metrics automatically
Alerts them when something is actually wrong
Can restart services if configured to do so
Generates a daily health report

Now? They spend maybe 10-15 minutes a day reviewing the report. The agent handles the routine monitoring. That's about 1 hour and 45 minutes saved every day. Not revolutionary, but meaningful. And they catch issues faster because the agent is always watching.

That's the difference between telling and doing.

What People Get Wrong About This

There's a lot of confusion out there, so let me clear up some common misconceptions:

"All AI is agentic now"

No, it's really not. Most of the AI you use every day is still assistant-level. ChatGPT, Claude, most chatbots - they're assistants. They're really good assistants, but they're not agents. True agentic AI that can take actions in your systems is still relatively new and not as widespread as people think.

"Agents will replace all human work"

This is the fear everyone has, and it's not realistic. Agents are good at repetitive, well-defined tasks. They're not replacing strategy, creativity, complex decision-making, or anything that requires real judgment. They're tools that handle routine work so humans can focus on more valuable tasks.

"AI assistants are outdated now"

Not at all. Assistants are perfect for tons of use cases. Not everything needs to be automated. Sometimes you want advice, not action. Sometimes you want to stay in control of every step. Assistants aren't going anywhere.

"Agents are just better assistants"

This is the big one people get wrong. They're not "better" - they're fundamentally different. It's like saying a car is a "better" bicycle. No, it's a different tool for different purposes. Assistants and agents solve different problems.

"Agents are too risky to use"

There's some truth to this concern, but it's manageable. Yes, giving AI the ability to take actions in your systems requires careful setup. You need guardrails, monitoring, and clear boundaries. But we've figured this out. Modern agent frameworks have safety built in. You can limit what they can do, require approval for certain actions, and monitor everything. It's not riskier than giving a new employee access to your systems - you just need proper controls.

The Safety Question Everyone Should Ask

Since we're talking about AI that can actually do things in your systems, let's address the elephant in the room: is this safe?

The honest answer is: it depends on how you set it up.

Here's what you need to think about:

Guardrails: Good agent systems let you define exactly what the agent can and can't do. You can say "you can restart servers, but you can't delete anything" or "you can book flights under $500, but ask me about anything more expensive." These boundaries are crucial.

Monitoring: You should be logging everything your agent does. Every action, every decision, every API call. This isn't just for debugging - it's for accountability and safety.

Approval workflows: For high-stakes actions, you can require human approval. The agent can do all the analysis and preparation, but it waits for your "yes" before executing anything critical.

Rollback capabilities: Things go wrong sometimes. Your agent should be able to undo its actions, or at least alert you immediately if something unexpected happens.

The key is this: you're not giving the agent unlimited power. You're giving it specific, controlled capabilities within boundaries you define. It's like giving someone the keys to your car - you're trusting them with something important, but you're not giving them ownership of your entire life.

Where This Is All Heading

We're in 2026, and we're still early in the agentic AI era. Here's what's developing:

Multi-agent systems are emerging. Instead of one agent doing everything, you might have multiple specialized agents working together. One handles customer support, another manages infrastructure, another handles data analysis. They can coordinate to handle complex workflows.

The line is blurring. Some AI assistants are adding limited action capabilities. Some agents are getting better at explaining their reasoning. We're moving toward hybrid systems that can adapt based on what you need.

It's getting easier to build. Right now, building an agent requires technical skill. But we're seeing more platforms that make it easier. Eventually, less technical people will be able to build agents for specific workflows.

Safety is improving. As more people use agents, we're learning better ways to build in safety features, monitoring, and control mechanisms. The technology is maturing.

The future probably isn't "assistants vs agents." It's more likely "assistants and agents working together." You'll use assistants for thinking and planning, and agents for executing and monitoring. They complement each other.

How to Actually Get Started

Alright, enough theory. Let's talk about what you should actually do with this information.

If You Want to Start Using AI Assistants:

This is the easy path. You can literally start right now.

Pick a tool. ChatGPT, Claude, or any of the major AI assistants. Most have free tiers to start.
Learn to prompt well. The better you ask questions, the better answers you get. This is a skill worth developing.
Start with simple tasks. Use it for writing, brainstorming, learning. Get comfortable with what it can and can't do.
Integrate if needed. If you're technical, most assistants have APIs you can use in your applications.

Cost: $0-100/month depending on usage
Time to start: Literally right now
Complexity: Low - anyone can do this

If You Want to Build Agentic AI:

This is more involved, but not as hard as you might think.

Define your workflow clearly. What exact task do you want automated? What are the steps? What decisions need to be made? The clearer you are, the better your agent will work.
Choose your platform. AWS Bedrock Agents, LangChain, AutoGPT, or other agent frameworks. Each has pros and cons. Bedrock is good if you're already on AWS. LangChain is good if you want flexibility.
Start small. Don't try to automate everything at once. Pick one simple workflow and get that working first.
Build in safety from day one. Set clear boundaries on what the agent can do. Log everything. Start with requiring approval for actions, then relax that as you gain confidence.
Test thoroughly. Run your agent through every scenario you can think of. What happens when things go wrong? How does it handle edge cases?
Monitor and improve. Once it's running, watch it closely at first. You'll find ways to improve it based on how it actually performs.

Cost: $100-1000+/month depending on what you're building
Time to start: Days to weeks
Complexity: Medium - you'll need some technical skills or help

The Bottom Line

Here's what you need to remember from all of this:

AI assistants are for thinking. They help you understand, plan, create, and learn. They're your smart advisor who knows a lot but can't touch anything.

Agentic AI is for doing. They handle workflows, take actions, make decisions, and complete tasks. They're your capable employee who can handle projects independently.

Both are valuable. Both have their place. The question isn't "which is better?" The question is "which solves my problem?"

Need answers? Use an assistant.
Need actions? Use an agent.
Need both? Use both.

We're in 2026, and we're just starting to figure out what's possible with AI that can actually do things. The assistants made us smarter. The agents are making us more productive.

The real power comes from knowing when to use each one.

Quick Reference Guide

Still not sure which you need? Ask yourself these questions:

Does it just need to answer, or does it need to act?

Just answer → Assistant
Actually act → Agent

Do I want to stay in control of every step?

Yes → Assistant
No, I want it handled → Agent

Is this a one-time question or an ongoing workflow?

One-time → Assistant
Ongoing → Agent

Am I trying to learn something or automate something?

Learn → Assistant
Automate → Agent

Do I need this to work when I'm not around?

No → Assistant
Yes → Agent

Simple as that.

What This Means for Different People

If You're a Developer:

Learn both. Understanding how to build assistants and agents is becoming an important skill. Start with assistants (easier), then explore agents (more complex). Focus on safety, monitoring, and good design patterns.

Developers who understand agentic AI will have an advantage as this technology matures.

If You're Running a Business:

Look at your workflows. Where are people doing repetitive, multi-step tasks? Those are agent candidates. Where do people need information and advice? Those are assistant use cases.

Start with one agent for one workflow. Measure the impact. If it works, expand. Don't try to automate everything at once.

And remember: the goal isn't to replace people. It's to free them up to do more valuable work.

If You're Just Curious:

Try ChatGPT or Claude if you haven't already. That's an assistant. Play with it. See what it can and can't do.

Then watch for agent capabilities showing up in tools you use. GitHub Copilot, AWS services, automation platforms - agents are being built into everything.

Understanding this difference will help you make sense of where AI is actually going.

Final Thoughts

The AI revolution isn't just about smarter answers. It's about AI that can actually do things.

For years, we've had AI that could think but not act. Now we have AI that can do both. That's a fundamental shift, and most people haven't caught up to it yet.

AI assistants made us more informed. They gave us access to knowledge and helped us think through problems. That's valuable, and it's not going away.

Agentic AI is making us more productive. It's taking tasks off our plate and handling them autonomously. That's powerful, and it's just getting started.

The key is understanding which tool solves which problem. Don't use an assistant when you need an agent. Don't use an agent when you just need advice.

We're at the beginning of the agentic AI era. The assistants aren't going away - they're just getting company. And together, they're changing how we work.

The question isn't whether you should use AI. The question is: which AI should you use, and for what?

Now you know the answer.

📌 Wrapping Up

Thank you for reading! I hope this article gave you practical insights and a clearer perspective on the topic.

Was this helpful?

❤️ Like if it added value
🦄 Unicorn if you’re applying it today
💾 Save for your next optimization session
🔄 Share with your team

Follow me for more on:

AWS architecture patterns
FinOps automation
Multi-account strategies
AI-driven DevOps

💡 What’s Next

More deep dives coming soon on cloud operations, GenAI, Agentic-AI, DevOps, and data workflows follow for weekly insights.

🌐 Portfolio & Work

You can explore my full body of work, certifications, architecture projects, and technical articles here:

👉 Visit My Website

🛠️ Services I Offer

If you're looking for hands-on guidance or collaboration, I provide:

Cloud Architecture Consulting (AWS / Azure)
DevSecOps & Automation Design
FinOps Optimization Reviews
Technical Writing (Cloud, DevOps, GenAI)
Product & Architecture Reviews
Mentorship & 1:1 Technical Guidance

🤝 Let’s Connect

I’d love to hear your thoughts drop a comment or connect with me on LinkedIn.

For collaborations, consulting, or technical discussions, feel free to reach out directly at simplynadaf@gmail.com

Happy Learning 🚀

Terraform State: The One File You Can't Afford to Lose

Sarvar Nadaf — Sat, 21 Mar 2026 19:03:47 +0000

👋 Hey there, tech enthusiasts!

"State is not just a file it's the single source of truth for your entire infrastructure."

🎯 Welcome Back!

In Article 3, you created your first S3 bucket and noticed a mysterious file appear: terraform.tfstate. I briefly mentioned it was "Terraform's memory," but we didn't dive deep.

Today, that changes. Understanding state is what separates beginners from confident Terraform users.

By the end of this article, you'll:

✅ Understand what state is and why it's critical
✅ Use state commands to inspect and manage infrastructure
✅ Import existing AWS resources into Terraform
✅ Handle state drift and conflicts
✅ Know when things go wrong and how to fix them

Time Required: 25 minutes

Cost: $0 (using free tier resources)

Difficulty: Intermediate

Let's unlock Terraform's most important concept! 🔓

🤔 The Problem: Why Does State Even Exist?

A Quick Thought Experiment

Imagine you run terraform apply and create an S3 bucket. Then you close your terminal and come back tomorrow.

Question: How does Terraform know that bucket exists?

Your .tf files? No they only describe what should exist, not what actually exists.

AWS API? Terraform could query AWS every time, but:

That's slow (imagine 100+ resources)
AWS doesn't know which resources Terraform created
No way to track metadata or dependencies

The Answer: The state file.

📋 What is Terraform State?

Simple Definition

State is a JSON file that maps your Terraform configuration to real-world resources.

Think of it like this:

Your .tf files = The blueprint (what you want)
State file = The inventory (what actually exists)
AWS = The actual building

What State Stores

Let's look at a real state file from our S3 bucket:

{
  "version": 4,
  "terraform_version": "1.14.4",
  "resources": [
    {
      "mode": "managed",
      "type": "aws_s3_bucket",
      "name": "my_first_bucket",
      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
      "instances": [
        {
          "attributes": {
            "id": "terraform-first-bucket-yourname-2026",
            "arn": "arn:aws:s3:::terraform-first-bucket-yourname-2026",
            "bucket": "terraform-first-bucket-yourname-2026",
            "region": "us-east-1",
            "tags": {
              "Name": "My First Terraform Bucket",
              "Environment": "Learning"
            }
          }
        }
      ]
    }
  ]
}

State tracks:

✅ Resource type (aws_s3_bucket)
✅ Resource name in your code (my_first_bucket)
✅ Actual AWS resource ID
✅ All attributes (ARN, region, tags, etc.)
✅ Dependencies between resources
✅ Provider information

🔄 How Terraform Uses State

The Terraform Workflow (Revisited)

When you run terraform plan or terraform apply, here's what happens:

1. Read your .tf files (desired state)
   ↓
2. Read terraform.tfstate (current state)
   ↓
3. Query AWS to verify current state
   ↓
4. Calculate the difference (the "plan")
   ↓
5. Show you what will change
   ↓
6. Apply changes (if you approve)
   ↓
7. Update state file with new information

Without state, Terraform would:

❌ Try to create resources that already exist
❌ Not know what to update or delete
❌ Lose track of dependencies
❌ Be unable to manage infrastructure

🛠️ Hands-On: State Commands

Let's create some infrastructure and explore state commands.

Step 1: Create Test Infrastructure

Create a new directory:

mkdir -p ~/terraform-state-demo
cd ~/terraform-state-demo

Create main.tf:

terraform {
  required_version = ">= 1.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

# Create two S3 buckets
resource "aws_s3_bucket" "logs" {
  bucket = "my-app-logs-yourname-2026"

  tags = {
    Name        = "Application Logs"
    Environment = "Development"
    Purpose     = "Logging"
  }
}

resource "aws_s3_bucket" "backups" {
  bucket = "my-app-backups-yourname-2026"

  tags = {
    Name        = "Application Backups"
    Environment = "Development"
    Purpose     = "Backup Storage"
  }
}

# Outputs
output "logs_bucket_id" {
  value = aws_s3_bucket.logs.id
}

output "backups_bucket_id" {
  value = aws_s3_bucket.backups.id
}

⚠️ Remember: Change yourname to something unique!

Apply the configuration:

terraform init
terraform apply

Type yes when prompted.

Command 1: terraform state list

Purpose: See all resources Terraform is managing.

terraform state list

Output:

aws_s3_bucket.backups
aws_s3_bucket.logs

Why this matters: Quickly see what's in your state without opening the JSON file.

Command 2: terraform state show

Purpose: Get detailed information about a specific resource.

terraform state show aws_s3_bucket.logs

Output:

# aws_s3_bucket.logs:
resource "aws_s3_bucket" "logs" {
    arn                         = "arn:aws:s3:::my-app-logs-yourname-2026"
    bucket                      = "my-app-logs-yourname-2026"
    bucket_domain_name          = "my-app-logs-yourname-2026.s3.amazonaws.com"
    hosted_zone_id              = "Z3AQBSTGFYJSTF"
    id                          = "my-app-logs-yourname-2026"
    region                      = "us-east-1"
    tags                        = {
        "Environment" = "Development"
        "Name"        = "Application Logs"
        "Purpose"     = "Logging"
    }
    # ... more attributes
}

Use case: Debug issues, verify configurations, check resource attributes.

Command 3: terraform show

Purpose: Display the entire state in human-readable format.

terraform show

Output: Shows all resources with their complete configurations.

Use case: Get a complete overview of your infrastructure.

Command 4: terraform state pull

Purpose: Download and display the raw state file.

terraform state pull

Output: Raw JSON state file content.

Use case: Backup state, inspect state structure, debugging.

🔄 Real-World Scenario: Importing Existing Resources

The Problem

You have an S3 bucket created manually in AWS Console (before you knew Terraform). Now you want Terraform to manage it.

You can't just write the Terraform code and run apply Terraform will try to create a new bucket and fail (bucket names are unique).

Solution: Import the existing resource into Terraform state.

Step-by-Step: Import an Existing Bucket

Step 1: Create a bucket manually in AWS Console

Go to S3 Console
Click "Create bucket"
Name it: manual-bucket-yourname-2026
Keep all defaults
Click "Create bucket"

Step 2: Write Terraform code for it

Add to your main.tf:

resource "aws_s3_bucket" "manual" {
  bucket = "manual-bucket-yourname-2026"

  tags = {
    Name        = "Manually Created Bucket"
    Environment = "Development"
    ManagedBy   = "Terraform"
  }
}

Step 3: Try to apply (this will fail)

terraform apply

You'll see:

Error: creating S3 Bucket (manual-bucket-yourname-2026): 
BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded

Terraform doesn't know the bucket exists!

Step 4: Import the bucket into state

terraform import aws_s3_bucket.manual manual-bucket-yourname-2026

Syntax:

terraform import <resource_type>.<resource_name> <aws_resource_id>

Output:

aws_s3_bucket.manual: Importing from ID "manual-bucket-yourname-2026"...
aws_s3_bucket.manual: Import prepared!
aws_s3_bucket.manual: Refreshing state...

Import successful!

Step 5: Verify the import

terraform state list

Output:

aws_s3_bucket.backups
aws_s3_bucket.logs
aws_s3_bucket.manual    ← New!

Step 6: Run plan to sync tags

terraform plan

You'll see:

  # aws_s3_bucket.manual will be updated in-place
  ~ resource "aws_s3_bucket" "manual" {
      ~ tags     = {
          + "Environment" = "Development"
          + "ManagedBy"   = "Terraform"
          + "Name"        = "Manually Created Bucket"
        }
    }

Terraform will add the tags we defined!

terraform apply

Now Terraform fully manages this bucket! 🎉

🚨 Understanding State Drift

What is State Drift?

State drift happens when the real infrastructure doesn't match what's in the state file.

Common causes:

Someone manually changes resources in AWS Console
Another tool modifies resources
AWS makes automatic changes
State file gets corrupted

Detecting Drift

Let's simulate drift:

Step 1: Manually change a bucket in AWS Console

Go to S3 Console
Open your my-app-logs-yourname-2026 bucket
Go to "Properties" → "Tags"
Add a new tag: ManualChange = Yes
Save

Step 2: Run terraform plan

terraform plan

Output:

  # aws_s3_bucket.logs will be updated in-place
  ~ resource "aws_s3_bucket" "logs" {
      ~ tags     = {
          - "ManualChange" = "Yes" -> null
            # (3 unchanged elements hidden)
        }
    }

Terraform detected the drift! It will remove the manual tag to match your code.

Refreshing State

Purpose: Update state file to match real infrastructure without making changes.

terraform refresh

Or (recommended in newer versions):

terraform apply -refresh-only

This updates state to match reality without modifying resources.

Use case: After manual changes, sync state before planning changes.

🔧 Advanced State Commands

Moving Resources in State

Scenario: You want to rename a resource in your code.

Problem: If you just rename it, Terraform will:

Destroy the old resource
Create a new resource
You lose data!

Solution: Use terraform state mv

Example:

# Rename logs bucket to app_logs in code
terraform state mv aws_s3_bucket.logs aws_s3_bucket.app_logs

Now update your main.tf to match:

resource "aws_s3_bucket" "app_logs" {  # Changed from "logs"
  bucket = "my-app-logs-yourname-2026"
  # ... rest of config
}

# Also update the output reference
output "logs_bucket_id" {
  value = aws_s3_bucket.app_logs.id  # Changed from .logs to .app_logs
}

⚠️ Important: After renaming with state mv, update ALL references including outputs, data sources, and dependencies in other resources.

Run plan:

terraform plan

Output:

No changes. Your infrastructure matches the configuration.

Perfect! No resources destroyed.

Removing Resources from State

Scenario: You want Terraform to stop managing a resource (but keep it in AWS).

terraform state rm aws_s3_bucket.manual

Output:

Removed aws_s3_bucket.manual
Successfully removed 1 resource instance(s).

Now:

Resource still exists in AWS
Terraform no longer tracks it
You can manage it manually or with another tool

Use case: Migrating resources to another Terraform project, or handing off to another team.

🎯 State Best Practices

1. Never Edit State Files Manually

❌ Don't do this:

nano terraform.tfstate  # NEVER!

✅ Use state commands instead:

terraform state mv
terraform state rm
terraform import

Why? Manual edits can corrupt state and break your infrastructure.

2. Always Backup State Before Major Changes

# Backup state
cp terraform.tfstate terraform.tfstate.backup

# Or use state pull
terraform state pull > state-backup-$(date +%Y%m%d).json

3. Use Version Control (But Not for State!)

Add to .gitignore:

cat > .gitignore << 'EOF'
# Terraform files
.terraform/
*.tfstate
*.tfstate.backup
.terraform.lock.hcl

# Variable files with secrets
*.tfvars
*.auto.tfvars
EOF

Why not commit state?

Contains sensitive data (passwords, keys)
Can cause merge conflicts
Not designed for version control

What to commit:

✅ .tf files
✅ .gitignore
✅ README.md
✅ Documentation

4. Use Remote State for Teams

Problem: Local state doesn't work for teams.

Solution: Remote state (we'll cover in Article 8).

Preview:

terraform {
  backend "s3" {
    bucket = "my-terraform-state"
    key    = "prod/terraform.tfstate"
    region = "us-east-1"
  }
}

🐛 Common State Issues and Solutions

Issue 1: State File Locked

Error:

Error: Error acquiring the state lock

Cause: Another terraform apply is running, or previous run crashed.

Solution:

# Wait for other operation to finish, or
terraform force-unlock <lock-id>

⚠️ Only force-unlock if you're sure no other process is running!

Issue 2: State Out of Sync

Error:

Error: Resource already exists

Solution:

# Refresh state
terraform apply -refresh-only

# Or import the resource
terraform import <resource_type>.<name> <id>

Issue 3: Lost State File

Problem: Accidentally deleted terraform.tfstate.

Solutions:

Option 1: Restore from backup

cp terraform.tfstate.backup terraform.tfstate

Option 2: Rebuild state by importing all resources

terraform import aws_s3_bucket.logs my-app-logs-yourname-2026
terraform import aws_s3_bucket.backups my-app-backups-yourname-2026
# ... import all resources

Option 3: If using remote state, re-initialize

terraform init

Issue 4: Corrupted State

Symptoms: Strange errors, resources not found, plan shows unexpected changes.

Solution:

# Restore from backup
cp terraform.tfstate.backup terraform.tfstate

# Or pull from remote state
terraform state pull > terraform.tfstate

🧪 Hands-On Challenge

Before moving to the next article, complete these challenges:

Challenge 1: State Inspection

List all resources in your state
Show detailed info for the backups bucket
Pull the raw state and save it to a file

Challenge 2: Import Practice

Create a new S3 bucket manually in AWS Console
Write Terraform code for it
Import it into your state
Verify with terraform plan

Challenge 3: Drift Detection

Manually add a tag to one of your buckets in AWS Console
Run terraform plan to detect the drift
Decide: keep the manual change or revert it

Challenge 4: State Manipulation

Rename one of your resources in code
Use terraform state mv to update state
Verify no resources will be destroyed

Challenge 5: Cleanup

Remove one resource from state (but keep in AWS)
Verify it's no longer tracked
Manually delete it from AWS Console

🧹 Cleanup

Let's clean up the resources we created:

cd ~/terraform-state-demo

# Destroy all resources
terraform destroy

# Verify in AWS Console
aws s3 ls | grep yourname

# Remove directory
cd ~
rm -rf terraform-state-demo

📊 State File Structure Explained

Key Sections

{
  "version": 4,                    // State format version
  "terraform_version": "1.14.4",   // Terraform version used
  "serial": 3,                     // Increments with each change
  "lineage": "abc-123-def",        // Unique ID for this state
  "outputs": {},                   // Output values
  "resources": []                  // All managed resources
}

Important fields:

version: State format version (currently 4)
serial: Increments with each state change (helps detect conflicts)
lineage: Unique identifier for this state file
resources: Array of all managed resources

🎓 What You Learned

State Fundamentals:

✅ State maps Terraform code to real resources
✅ State is the single source of truth
✅ State enables Terraform to calculate changes

State Commands:

✅ terraform state list - List all resources
✅ terraform state show - Show resource details
✅ terraform state mv - Rename resources
✅ terraform state rm - Remove from state
✅ terraform import - Import existing resources

State Management:

✅ Detect and handle state drift
✅ Import existing infrastructure
✅ Backup and restore state
✅ Troubleshoot common issues

Best Practices:

✅ Never edit state manually
✅ Always backup before major changes
✅ Never commit state to Git
✅ Use remote state for teams

🎬 What's Next?

In Article 5: Variables and Outputs - Making Code Reusable, you'll learn:

How to parameterize your Terraform code
Using input variables for flexibility
Creating reusable configurations
Managing different environments (dev/staging/prod)
Working with variable files
Sensitive data handling

You'll build:

A reusable S3 bucket module
Environment-specific configurations
Dynamic resource naming
Secure variable management

📚 Resources

Terraform State Documentation: https://developer.hashicorp.com/terraform/language/state
State Commands: https://developer.hashicorp.com/terraform/cli/commands/state
Import Command: https://developer.hashicorp.com/terraform/cli/import

📌 Wrapping Up

Thank you for reading. I hope this article provided practical insights and a clearer understanding of the topic.

If you found this useful:

❤️ Like if it added value
🦄 Unicorn if you’re applying it today
💾 Save it for your next optimization session
🔄 Share it with your team

💡 What’s Next

More deep dives are coming soon on:

Cloud Operations
GenAI & Agentic AI
DevOps Automation
Data & Platform Engineering

Follow along for weekly insights and hands-on guides.

🌐 Portfolio & Work

You can explore my full body of work, certifications, architecture projects, and technical articles here:

👉 Visit My Website

🛠️ Services I Offer

If you're looking for hands-on guidance or collaboration, I provide:

Cloud Architecture Consulting (AWS / Azure)
DevSecOps & Automation Design
FinOps Optimization Reviews
Technical Writing (Cloud, DevOps, GenAI)
Product & Architecture Reviews
Mentorship & 1:1 Technical Guidance

🤝 Let’s Connect

I’d love to hear your thoughts. Feel free to drop a comment or connect with me on:

🔗 LinkedIn

For collaborations, consulting, or technical discussions, reach out at:

📧 simplynadaf@gmail.com

Found this helpful? Share it with your team.

⭐ Star the repo • 📖 Follow the series • 💬 Ask questions

Made by Sarvar Nadaf

🌐 https://sarvarnadaf.com

Your First Infrastructure as Code in Four Commands

Sarvar Nadaf — Mon, 16 Mar 2026 23:31:39 +0000

👋 Hey there, tech enthusiasts!

"The journey from clicking to coding starts with a single resource."

🎯 Welcome Back!

Remember in Article 1 when I showed you the pain of clicking through AWS Console for hours? And in Article 2 we got Terraform installed?

Today, you'll experience the magic moment - creating real AWS infrastructure with just a few lines of code.

By the end of this article, you'll:

✅ Understand the Terraform workflow (the 4 commands you'll use forever)
✅ Write your first .tf file
✅ Create an actual S3 bucket on AWS
✅ Understand what a state file is and why it matters
✅ Clean up resources properly

Time Required: 20 minutes

Cost: $0 (S3 bucket is free tier)

Difficulty: Beginner-friendly

Let's create some infrastructure! 🚀

💡 Quick Theory: The Terraform Workflow

Before we dive in, you need to know the 4 commands that form the core Terraform workflow. You'll use these commands for every project, whether you're creating a simple S3 bucket or a complex multi-tier application.

The Magic 4 Commands:

terraform init      # Initialize - Download providers
terraform plan      # Preview - See what will change
terraform apply     # Execute - Create the resources
terraform destroy   # Cleanup - Remove everything

Think of it like cooking:

init = Get your ingredients and tools ready
plan = Read the recipe and imagine the final dish
apply = Actually cook the meal
destroy = Clean up the kitchen

That's it! Master these 4 commands, and you've mastered the Terraform workflow.

📝 Understanding Terraform Files

Terraform uses files with .tf extension written in HCL (HashiCorp Configuration Language). Don't worry - it's much simpler than it sounds!

Basic Structure:

# This is a comment

block_type "label" "name" {
  argument = "value"
  another_argument = 123
}

Three main block types you'll use:

terraform - Configuration settings
provider - Which cloud (AWS, Azure, GCP)
resource - What to create (S3, EC2, etc.)

Let's see them in action!

🛠️ Hands-On: Create Your First S3 Bucket

Step 1: Create Your Project Directory

# Create a directory for your first Terraform project
mkdir -p ~/terraform-first-resource
cd ~/terraform-first-resource

Step 2: Write Your First Terraform Code

Create a file called main.tf:

nano main.tf
# Or use your favorite editor: vim, VS Code, etc.

Copy this code:

# Terraform configuration block
# Specifies Terraform version and required providers
terraform {
  required_version = ">= 1.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

# Provider configuration
# Tells Terraform to use AWS in us-east-1 region
provider "aws" {
  region = "us-east-1"
}

# Resource block - This creates the actual S3 bucket
resource "aws_s3_bucket" "my_first_bucket" {
  # Bucket name must be globally unique across ALL AWS accounts
  # Replace 'yourname' with your actual name or username
  bucket = "terraform-first-bucket-yourname-2026"

  # Tags help organize and track resources
  tags = {
    Name        = "My First Terraform Bucket"
    Environment = "Learning"
    ManagedBy   = "Terraform"
    CreatedBy   = "YourName"
    Updated     = "Yes"
  }
}

# Output block - Displays information after creation
output "bucket_name" {
  value       = aws_s3_bucket.my_first_bucket.id
  description = "The name of the S3 bucket"
}

output "bucket_arn" {
  value       = aws_s3_bucket.my_first_bucket.arn
  description = "The ARN of the S3 bucket"
}

output "bucket_region" {
  value       = aws_s3_bucket.my_first_bucket.region
  description = "The region where bucket is created"
}

⚠️ Important: Change yourname in the bucket name to something unique (your name, username, or random string). S3 bucket names must be globally unique across ALL AWS accounts!

Save the file (Ctrl+X, then Y, then Enter if using nano)

🚀 The 4 Commands in Action

Now comes the exciting part! Let's run our 4 magic commands.

Command 1: terraform init

This downloads the AWS provider plugin.

terraform init

You'll see:

Initializing the backend...
Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 5.0"...
- Installing hashicorp/aws v5.x.x...
- Installed hashicorp/aws v5.x.x

Terraform has been successfully initialized!

What just happened?

Terraform downloaded the AWS provider plugin
Created a .terraform/ directory (hidden folder)
Created .terraform.lock.hcl file (locks provider versions)

Pro Tip: You only need to run init once per project, or when you add new providers.

Command 2: terraform plan

This shows you what Terraform will create (without actually creating it).

terraform plan

You'll see:

Terraform will perform the following actions:

  # aws_s3_bucket.my_first_bucket will be created
  + resource "aws_s3_bucket" "my_first_bucket" {
      + bucket                      = "terraform-first-bucket-yourname-2026"
      + bucket_domain_name          = (known after apply)
      + id                          = (known after apply)
      + region                      = (known after apply)
      + tags                        = {
          + "CreatedBy"   = "YourName"
          + "Environment" = "Learning"
          + "ManagedBy"   = "Terraform"
          + "Name"        = "My First Terraform Bucket"
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Understanding the output:

+ means "will be created"
~ means "will be modified" (you'll see this later)
- means "will be destroyed"
(known after apply) means AWS will generate this value

This is your safety check! Always review the plan before applying.

Command 3: terraform apply

This actually creates the resources on AWS.

terraform apply

Terraform will show the plan again and ask for confirmation:

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value:

Type yes and press Enter.

You'll see:

aws_s3_bucket.my_first_bucket: Creating...
aws_s3_bucket.my_first_bucket: Creation complete after 2s [id=terraform-first-bucket-yourname-2026]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

bucket_arn = "arn:aws:s3:::terraform-first-bucket-yourname-2026"
bucket_name = "terraform-first-bucket-yourname-2026"
bucket_region = "us-east-1"

🎉 Congratulations! You just created your first AWS resource with code!

Verify in AWS Console

Let's confirm it's really there:

Go to AWS S3 Console
You should see your bucket: terraform-first-bucket-yourname-2026
Click on it and check the Tags tab - you'll see all the tags you defined!

This is the "aha!" moment - you created AWS infrastructure without clicking through the console!

🗂️ Understanding the State File

After running terraform apply, you'll notice a new file: terraform.tfstate

ls -la

You'll see:

.terraform/
.terraform.lock.hcl
main.tf
terraform.tfstate

What is the State File?

The state file is Terraform's memory. It stores:

What resources Terraform created
Current configuration of those resources
Metadata and dependencies

Let's peek inside:

cat terraform.tfstate

You'll see JSON with details about your S3 bucket - its name, ARN, region, tags, etc.

Why Does State Matter?

When you run terraform plan or terraform apply again, Terraform:

Reads the state file to know what exists
Compares it with your .tf files
Calculates what needs to change

Without state, Terraform wouldn't know what it created!

⚠️ Important State Rules:

Never edit state files manually
Never delete state files (you'll lose track of resources)
Never commit state files to Git (they contain sensitive data)
For teams, use remote state (we'll cover this in Article 8)

🧪 Let's Make a Change

Now let's see Terraform's power - making changes to existing infrastructure.

Edit your main.tf file and add a new tag:

resource "aws_s3_bucket" "my_first_bucket" {
  bucket = "terraform-first-bucket-yourname-2026"

  tags = {
    Name        = "My First Terraform Bucket"
    Environment = "Learning"
    ManagedBy   = "Terraform"
    CreatedBy   = "YourName"
    Updated     = "Yes"  
    LastUpdated = "Today"  # ← Add this new tag
  }
}

Run plan again:

terraform plan

You'll see:

  # aws_s3_bucket.my_first_bucket will be updated in-place
  ~ resource "aws_s3_bucket" "my_first_bucket" {
      ~ tags     = {
          + "Updated"     = "Yes"
            # (5 unchanged elements hidden)
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Notice the ~ symbol - it means "modify existing resource"!

Apply the change:

terraform apply

Type yes when prompted.

Check AWS Console - your bucket now has the new tag!

This is Infrastructure as Code magic - you changed infrastructure by editing a text file!

🧹 Command 4: terraform destroy

Always clean up resources you're not using (to avoid unexpected charges).

terraform destroy

Terraform will show what it will delete:

  # aws_s3_bucket.my_first_bucket will be destroyed
  - resource "aws_s3_bucket" "my_first_bucket" {
      - bucket = "terraform-first-bucket-yourname-2026"
      ...
    }

Plan: 0 to add, 0 to change, 1 to destroy.

Do you really want to destroy all resources?
  Enter a value:

Type yes to confirm.

aws_s3_bucket.my_first_bucket: Destroying...
aws_s3_bucket.my_first_bucket: Destruction complete after 1s

Destroy complete! Resources: 1 destroyed.

Verify in AWS Console - your bucket is gone!

🎓 What You Just Learned

Let's recap what you accomplished:

✅ The Terraform Workflow:

init - Initialize project
plan - Preview changes
apply - Create resources
destroy - Clean up

✅ HCL Basics:

terraform block (configuration)
provider block (cloud provider)
resource block (what to create)
output block (display information)

✅ State Management:

State file tracks resources
Never edit state manually
State enables change detection

✅ Real Infrastructure:

Created actual AWS S3 bucket
Modified existing resource
Destroyed resources cleanly

🐛 Common Issues and Solutions

Issue 1: Bucket Name Already Exists

Error:

Error: creating S3 Bucket: BucketAlreadyExists

Solution: S3 bucket names are globally unique. Change your bucket name to something more unique:

bucket = "terraform-first-bucket-yourname-12345-2026"

Issue 2: Invalid Bucket Name

Error:

Error: invalid bucket name

Solution: S3 bucket names must:

Be 3-63 characters long
Use only lowercase letters, numbers, and hyphens
Start with a letter or number
Not use uppercase or underscores

Bad: My_Bucket_Name

Good: my-bucket-name

Issue 3: AWS Credentials Not Found

Error:

Error: No valid credential sources found

Solution: Configure AWS credentials (from Article 2):

aws configure

Enter your AWS Access Key ID and Secret Access Key.

Issue 4: Permission Denied

Error:

Error: AccessDenied: Access Denied

Solution: Your AWS user needs S3 permissions. Attach the AmazonS3FullAccess policy to your IAM user in AWS Console.

💡 Best Practices You Should Follow

1. Always Use Version Control

git init
echo ".terraform/" >> .gitignore
echo "terraform.tfstate*" >> .gitignore
echo "*.tfvars" >> .gitignore
git add main.tf .gitignore
git commit -m "Add first Terraform resource"

2. Use Meaningful Names

Bad:

resource "aws_s3_bucket" "b" {
  bucket = "bucket1"
}

Good:

resource "aws_s3_bucket" "application_logs" {
  bucket = "myapp-logs-production-2026"
}

3. Always Add Tags

Tags help with:

Cost tracking
Resource organization
Automation
Compliance

Minimum tags:

tags = {
  Name        = "Resource purpose"
  Environment = "dev/staging/prod"
  ManagedBy   = "Terraform"
  Owner       = "team-name"
}

4. Run Plan Before Apply

Never skip terraform plan! It's your safety net.

5. Use Descriptive Outputs

Outputs make it easy to get important information:

output "bucket_name" {
  value       = aws_s3_bucket.my_first_bucket.id
  description = "The name of the S3 bucket"
}

🔍 Understanding What Happened Behind the Scenes

When you ran terraform apply, here's what happened:

Terraform read your main.tf file
- Parsed the HCL syntax
- Understood you want an S3 bucket
Terraform checked the state file
- Found it was empty (first run)
- Knew it needed to create the bucket
Terraform called AWS API
- Used your AWS credentials
- Made API call: CreateBucket
- Set the tags with PutBucketTagging
AWS created the bucket
- Generated unique bucket ID
- Returned bucket details
Terraform updated state file
- Saved bucket information
- Recorded all attributes
Terraform displayed outputs
- Showed the values you defined

This is the power of Infrastructure as Code - complex API calls abstracted into simple, readable code!

🎯 Challenge: Try It Yourself

Before moving to the next article, try these challenges:

Challenge 1: Create Multiple Buckets

Create 2 different S3 buckets in the same main.tf file.

Hint:

resource "aws_s3_bucket" "bucket_one" {
  bucket = "my-first-bucket-yourname"
}

resource "aws_s3_bucket" "bucket_two" {
  bucket = "my-second-bucket-yourname"
}

Challenge 2: Use Different Region

Change the provider region to us-west-2 and create a bucket there.

Challenge 3: Add More Tags

Add 3 more custom tags to your bucket (Project, CostCenter, Department).

Challenge 4: Format Your Code

Run terraform fmt to automatically format your code:

terraform fmt

Challenge 5: Validate Your Code

Run terraform validate to check for syntax errors:

terraform validate

📚 Additional Terraform Commands

Beyond the core 4 commands, here are useful ones:

# Format your code (auto-fix indentation)
terraform fmt

# Validate syntax
terraform validate

# Show current state
terraform show

# List resources in state
terraform state list

# Get detailed resource info
terraform state show aws_s3_bucket.my_first_bucket

# View outputs without applying
terraform output

# View specific output
terraform output bucket_name

🔗 Useful Resources

Terraform AWS Provider Docs: https://registry.terraform.io/providers/hashicorp/aws/latest/docs
S3 Bucket Resource: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket
HCL Syntax: https://developer.hashicorp.com/terraform/language/syntax/configuration
Terraform CLI Commands: https://developer.hashicorp.com/terraform/cli/commands

🎬 What's Next?

In Article 4: Understanding Terraform State, we'll dive deeper into:

How state files work internally
State commands (state list, state show, state mv)
Importing existing AWS resources
State locking and remote state (preview)
Recovering from state issues

You'll learn:

Why state is Terraform's most important file
How to inspect and manipulate state safely
What to do when state gets out of sync
Preparing for team collaboration

📝 Summary

Today you learned the foundation of Terraform:

The 4 Core Commands:

terraform init     # Get ready
terraform plan     # Preview
terraform apply    # Execute
terraform destroy  # Cleanup

Key Concepts:

.tf files contain your infrastructure code
State files track what Terraform created
HCL syntax is simple and readable
Always plan before applying
Tags help organize resources

You created real infrastructure with code! No more clicking through AWS Console for simple tasks.

📌 Wrapping Up

Thank you for reading. I hope this article provided practical insights and a clearer understanding of the topic.

If you found this useful:

❤️ Like if it added value
🦄 Unicorn if you’re applying it today
💾 Save it for your next optimization session
🔄 Share it with your team

💡 What’s Next

More deep dives are coming soon on:

Cloud Operations
GenAI & Agentic AI
DevOps Automation
Data & Platform Engineering

Follow along for weekly insights and hands-on guides.

🌐 Portfolio & Work

You can explore my full body of work, certifications, architecture projects, and technical articles here:

👉 Visit My Website

🛠️ Services I Offer

If you're looking for hands-on guidance or collaboration, I provide:

Cloud Architecture Consulting (AWS / Azure)
DevSecOps & Automation Design
FinOps Optimization Reviews
Technical Writing (Cloud, DevOps, GenAI)
Product & Architecture Reviews
Mentorship & 1:1 Technical Guidance

🤝 Let’s Connect

I’d love to hear your thoughts. Feel free to drop a comment or connect with me on:

🔗 LinkedIn

For collaborations, consulting, or technical discussions, reach out at:

📧 simplynadaf@gmail.com

Found this helpful? Share it with your team.

⭐ Star the repo • 📖 Follow the series • 💬 Ask questions

Made by Sarvar Nadaf

🌐 https://sarvarnadaf.com

Installing Terraform and Setting Up Your Environment

Sarvar Nadaf — Sat, 14 Mar 2026 20:22:21 +0000

👋 Hey there, tech enthusiasts!

Let's dive in and explore the fascinating world of cloud technology together! 🚀

In this guide, we will walk through the step-by-step process of installing Terraform and preparing your local environment for infrastructure automation.

What You'll Learn

Install Terraform on Linux (Ubuntu/Amazon Linux)
Install AWS CLI
Configure AWS credentials
Verify your setup
Set up VS Code for Terraform development

Prerequisites

A Linux server or local machine (Ubuntu 20.04+ or Amazon Linux 2)
AWS account with IAM user credentials
Basic command line knowledge

Step 1: Install Terraform

For Ubuntu/Debian

# Update package list
sudo apt-get update

# Install required packages
sudo apt-get install -y gnupg software-properties-common wget

# Add HashiCorp GPG key
wget -O- https://apt.releases.hashicorp.com/gpg | \
gpg --dearmor | \
sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg

# Add HashiCorp repository
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \
https://apt.releases.hashicorp.com $(lsb_release -cs) main" | \
sudo tee /etc/apt/sources.list.d/hashicorp.list

# Update and install Terraform
sudo apt-get update
sudo apt-get install -y terraform

For Amazon Linux 2 (Recommended)

# Install yum-config-manager
sudo yum install -y yum-utils

# Add HashiCorp repository
sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo

# Install Terraform
sudo yum -y install terraform

Verify Installation

terraform version

Expected Output:

Terraform v1.14.x
on linux_amd64

Step 2: Install AWS CLI (Recommended)

For Ubuntu/Debian

# Download AWS CLI installer
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

# Install unzip if not present
sudo apt-get install -y unzip

# Unzip the installer
unzip awscliv2.zip

# Run the installer
sudo ./aws/install

# Clean up
rm -rf aws awscliv2.zip

For Amazon Linux 2

# Download and install
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
rm -rf aws awscliv2.zip

Verify Installation

aws --version

Expected Output:

aws-cli/2.x.x Python/3.x.x Linux/x.x.x

Step 3: Create IAM User for Terraform

In AWS Console:

Go to IAM → Users → Create User
Username: terraform (or your preferred name)
Select "Provide user access to the AWS Management Console" (optional)
Attach policies: AdministratorAccess (for learning; use restricted policies in production)
Create user
Go to Security Credentials → Create Access Key
Select "Command Line Interface (CLI)"
Download or copy:
- Access Key ID
- Secret Access Key

⚠️ Important: Never share or commit these credentials to version control!

Step 4: Configure AWS Credentials

aws configure

You'll be prompted for:

AWS Access Key ID [None]: <YOUR_ACCESS_KEY_ID>
AWS Secret Access Key [None]: <YOUR_SECRET_ACCESS_KEY>
Default region name [None]: us-east-1 (or your preferred region)
Default output format [None]: json

Verify Configuration

aws sts get-caller-identity

Expected Output:

{
    "UserId": "AIDASRZSGHJSDC6XXXXX",
    "Account": "123456789012",
    "Arn": "arn:aws:iam::123456789012:user/terraform"
}

Step 5: Set Up Your Working Directory

# Create project directory
mkdir -p ~/terraform-projects
cd ~/terraform-projects

# Create your first project folder
mkdir my-first-terraform
cd my-first-terraform

Step 6: (Optional) Install VS Code with Terraform Extension

Install VS Code

# Ubuntu/Debian
wget -qO- https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > packages.microsoft.gpg
sudo install -D -o root -g root -m 644 packages.microsoft.gpg /etc/apt/keyrings/packages.microsoft.gpg
sudo sh -c 'echo "deb [arch=amd64,arm64,armhf signed-by=/etc/apt/keyrings/packages.microsoft.gpg] https://packages.microsoft.com/repos/code stable main" > /etc/apt/sources.list.d/vscode.list'
sudo apt-get update
sudo apt-get install -y code

Install Terraform Extension

Open VS Code
Go to Extensions (Ctrl+Shift+X)
Search for "HashiCorp Terraform"
Click Install

Extension Features:

Syntax highlighting
Auto-completion
Formatting (terraform fmt)
Validation

Step 7: Test Your Setup

Create a simple test file:

cat > test.tf << 'EOF'
terraform {
  required_version = ">= 1.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

# This is just a test - we won't create any resources yet
output "account_id" {
  value = data.aws_caller_identity.current.account_id
}

data "aws_caller_identity" "current" {}
EOF

Run Terraform commands:

# Initialize Terraform
terraform init

# Validate configuration
terraform validate

# See what would happen (no resources created)
terraform plan

Expected Output:

Terraform has been successfully initialized!
Success! The configuration is valid.
Changes to Outputs:
  + account_id = "123456789012"

Clean up test file:

rm test.tf
rm -rf .terraform .terraform.lock.hcl

Troubleshooting Common Issues

Issue 1: "terraform: command not found"

Solution: Add Terraform to PATH or reinstall

Issue 2: "Unable to locate credentials"

Solution: Run aws configure again and verify credentials

Issue 3: "Error: No valid credential sources found"

Solution: Check ~/.aws/credentials file exists and has correct format

Issue 4: Permission denied errors

Solution: Verify IAM user has necessary permissions

What's Next?

In the next article, we'll:

Create our first AWS resource (S3 bucket)
Understand Terraform workflow (init, plan, apply, destroy)
Learn about Terraform state
Explore basic Terraform syntax

Key Takeaways

✅ Terraform is installed and working
✅ AWS CLI is configured with credentials
✅ You can verify your AWS identity
✅ Your development environment is ready
✅ You understand the basic Terraform commands

Additional Resources

Next Article: Part 3: Provisioning Your First AWS Resource

📌 Wrapping Up

Thank you for reading. I hope this article provided practical insights and a clearer understanding of the topic.

If you found this useful:

❤️ Like if it added value
🦄 Unicorn if you’re applying it today
💾 Save it for your next optimization session
🔄 Share it with your team

💡 What’s Next

More deep dives are coming soon on:

Cloud Operations
GenAI & Agentic AI
DevOps Automation
Data & Platform Engineering

Follow along for weekly insights and hands-on guides.

🌐 Portfolio & Work

You can explore my full body of work, certifications, architecture projects, and technical articles here:

👉 Visit My Website

🛠️ Services I Offer

If you're looking for hands-on guidance or collaboration, I provide:

Cloud Architecture Consulting (AWS / Azure)
DevSecOps & Automation Design
FinOps Optimization Reviews
Technical Writing (Cloud, DevOps, GenAI)
Product & Architecture Reviews
Mentorship & 1:1 Technical Guidance

🤝 Let’s Connect

I’d love to hear your thoughts. Feel free to drop a comment or connect with me on:

🔗 LinkedIn

For collaborations, consulting, or technical discussions, reach out at:

📧 simplynadaf@gmail.com

Found this helpful? Share it with your team.

⭐ Star the repo • 📖 Follow the series • 💬 Ask questions

Made by Sarvar Nadaf

🌐 https://sarvarnadaf.com

Why Every Developer Should Learn Terraform in 2026 (And How to Start)

Sarvar Nadaf — Fri, 13 Mar 2026 23:54:58 +0000

👋 Hey there, tech enthusiasts!

Let's dive in and explore the fascinating world of cloud technology together! 🚀

"The best time to learn Infrastructure as Code was 5 years ago. The second best time is now."

🎯 Why You're Here

If you're reading this, chances are:

You're tired of manually clicking through AWS console for hours
You've accidentally deleted a production server and wished you had a backup "recipe"
Your team struggles to replicate environments consistently
You want to level up your DevOps skills and increase your market value

Good news: You're in the right place. This series will take you from zero to confidently managing cloud infrastructure with code.

The Old Way: Manual Infrastructure Management

The Painful Reality

Imagine this scenario (maybe you've lived it):

Monday Morning:

Boss: "We need a new staging environment by EOD."
You: *Opens AWS Console*
     *Clicks through 47 different screens*
     *Creates VPC, subnets, security groups, EC2 instances...*
     *Takes 4 hours*
You: "Done! "

Tuesday Morning:

Boss: "Great! Now create the same setup for production."
You: *Realizes you forgot what you clicked*
     *Tries to remember the configuration*
     *Spends 5 hours recreating it*
     *Something is different but you're not sure what*

Wednesday Morning:

Boss: "The staging environment crashed. Can you rebuild it?"
You: *Panic intensifies*

Problems with Manual Infrastructure:

❌ No Documentation - "How did I configure that security group again?"

❌ Human Errors - One wrong click = production down

❌ Time-Consuming - Hours of repetitive clicking

❌ Not Reproducible - Each environment is slightly different

❌ No Version Control - Can't rollback changes

❌ Team Collaboration Nightmare - "Who changed the firewall rules?"

❌ Disaster Recovery - If it's gone, it's GONE

Sound familiar? This is where Infrastructure as Code (IaC) comes to the rescue.

🚀 The Modern Way: Infrastructure as Code (IaC)

What is Infrastructure as Code?

Simple Definition:

Infrastructure as Code means managing and provisioning your servers, networks, and cloud resources using code files instead of manual processes.

Think of it like this:

Traditional Way: Following a recipe in your head (error-prone, not shareable)
IaC Way: Writing down the recipe in a cookbook (repeatable, shareable, version-controlled)

The IaC Approach:

You: *Writes a configuration file (5 minutes)*
     resource "aws_instance" "web_server" {
       ami           = "ami-12345678"
       instance_type = "t2.micro"
     }

You: *Runs one command*
     $ terraform apply

Terraform: *Creates entire infrastructure in 2 minutes*
           ✅ VPC created
           ✅ Subnets created
           ✅ Security groups configured
           ✅ EC2 instance launched
           ✅ Everything documented in code

Benefits of IaC:

✅ Version Control - Track every change with Git

✅ Reproducible - Same code = Same infrastructure, every time

✅ Fast - Deploy in minutes, not hours

✅ Documented - Your code IS your documentation

✅ Collaborative - Team can review changes before applying

✅ Disaster Recovery - Rebuild everything with one command

✅ Cost Savings - Automate, optimize, and destroy unused resources easily

🌟 Enter Terraform: Your Infrastructure Automation Superpower

What is Terraform?

Terraform is an open-source Infrastructure as Code tool created by HashiCorp that lets you define, provision, and manage cloud infrastructure using simple, human-readable configuration files.

Why Terraform? (Not CloudFormation, Ansible, or Others)

1. Cloud-Agnostic 🌍

Works with AWS, Azure, GCP, and 3000+ providers
Not locked into one cloud vendor
Manage multi-cloud infrastructure from one tool

2. Declarative Syntax 📝

# You declare WHAT you want, not HOW to create it
resource "aws_s3_bucket" "my_bucket" {
  bucket = "my-awesome-bucket"
}
# Terraform figures out the steps

3. State Management 🗂️

Terraform tracks what's deployed
Knows what changed and what needs updating
Prevents configuration drift

4. Plan Before Apply 🔍

$ terraform plan
# Shows you EXACTLY what will change before you apply
# No surprises, no accidents

5. Massive Ecosystem 🌐

3000+ providers (AWS, Kubernetes, GitHub, Datadog, etc.)
Huge community support
Thousands of pre-built modules

Terraform vs Other Tools

Feature	Terraform	CloudFormation	Ansible	Pulumi
Multi-Cloud	✅ Yes	❌ AWS Only	✅ Yes	✅ Yes
Learning Curve	Easy	Medium	Easy	Hard
State Management	✅ Built-in	✅ Built-in	❌ No	✅ Built-in
Language	HCL (Simple)	JSON/YAML	YAML	Real Programming Languages
Community	Huge	Large	Huge	Growing
Best For	Infrastructure	AWS-only projects	Configuration Management	Developers who prefer code

Bottom Line: Terraform is the industry standard for multi-cloud infrastructure automation.

🎯 Real-World Use Cases: Where Terraform Shines

1. Multi-Environment Management

Same code → Different variables → Dev, Staging, Prod environments
No manual errors, perfect consistency

2. Disaster Recovery

Server crashed? Region down?
Run: terraform apply
Infrastructure restored in minutes

3. Cost Optimization

# Destroy dev environment after work hours
$ terraform destroy -target=aws_instance.dev_servers

# Recreate next morning
$ terraform apply

Savings: $500/month on unused resources

4. Team Collaboration

Developer: Creates pull request with infrastructure changes
Team: Reviews the terraform plan
Manager: Approves
CI/CD: Automatically applies changes
Everyone: Knows exactly what changed and when

5. Compliance & Auditing

Every infrastructure change is:
- Version controlled in Git
- Reviewed and approved
- Documented automatically
- Auditable for compliance

🏆 Why Learning Terraform Will Boost Your Career

Market Demand 📈

Job Postings with "Terraform":

2020: 15,000 jobs
2024: 85,000+ jobs
2026: Still growing rapidly

Average Salary Increase:

DevOps Engineer without IaC: $90K
DevOps Engineer with Terraform: $120K+
That's a $30K+ difference!

Skills You'll Gain 🎓

By learning Terraform, you'll also learn:

✅ Cloud architecture (AWS, Azure, GCP)
✅ Infrastructure best practices
✅ Version control workflows
✅ CI/CD pipelines
✅ Security and compliance
✅ Cost optimization strategies

Career Paths 🚀

Terraform opens doors to:

DevOps Engineer - Automate everything
Cloud Architect - Design scalable systems
Site Reliability Engineer (SRE) - Ensure uptime
Platform Engineer - Build internal platforms
Infrastructure Engineer - Manage cloud resources

🚦 Prerequisites: What You Need to Start

Required ✅

Basic Command Line Knowledge - Know how to navigate directories, run commands
AWS Account - Free tier is enough (Create one here)
Willingness to Learn - That's it!

Nice to Have (But Not Required) 💡

Basic understanding of cloud concepts (we'll explain as we go)
Git basics (we'll cover what you need)
Any programming experience (helpful but not necessary)

What You DON'T Need ❌

❌ DevOps experience
❌ AWS certification
❌ Programming expertise
❌ Expensive tools or subscriptions

If you can open a terminal and follow instructions, you can learn Terraform!

🎬 How to Follow This Series

Step 1: Bookmark This Series

📖 dev.to Series: https://dev.to/sarvar_04/series/36958

Step 2: Star the GitHub Repository

⭐ GitHub Repo: https://github.com/simplynadaf/terraform-by-sarvar

All code examples are here
Tested and ready to use
Updated with each article

Step 3: Set Up Your Environment

👉 Next Article: Installation & Setup

Install Terraform
Configure AWS CLI
Verify your setup
Ready to write your first Terraform code!

Step 4: Join the Community

💬 Ask questions in GitHub Issues
🔗 Connect on LinkedIn
📧 Email: simplynadaf@gmail.com

🎯 Your Learning Path Starts Now

What Happens Next?

In the next article, you'll:

✅ Install Terraform on your machine (Linux/Mac/Windows)
✅ Set up AWS CLI and credentials
✅ Configure your development environment
✅ Run your first Terraform command
✅ Verify everything is working

Time Investment: 30 minutes

Difficulty: Beginner-friendly

Result: Ready to write Terraform code!

The Journey Ahead

Week 1-2:  Foundation (Articles 1-3)
           ↓
Week 3-4:  Intermediate Concepts (Articles 4-8)
           ↓
Week 5-6:  Advanced Techniques (Articles 9-15)
           ↓
Result:    Production-Ready Terraform Skills 🎉

💭 Final Thoughts: Why Start Today?

The Infrastructure Revolution is Here

Companies are moving to cloud at an unprecedented rate. Infrastructure as Code is no longer optional it's expected.

The Best Investment You Can Make

⏰ Time: 6-8 weeks of learning
💰 Cost: Free (just AWS free tier)
📈 Return: Career advancement, higher salary, in-demand skills

You're Not Alone

Thousands of developers have learned Terraform and transformed their careers. You're joining a massive, supportive community.

The Hardest Step is the First One

You've already taken it by reading this article. Now keep the momentum going!

📚 Resources & Links

Official Documentation

This Series

📖 Series Home: https://dev.to/sarvar_04/series/36958
💻 Code Repository: https://github.com/simplynadaf/terraform-by-sarvar

🎓 Conclusion

Infrastructure as Code isn't just a trend it's the foundation of modern cloud operations. Terraform has become the industry standard, and learning it will open countless opportunities in your career.

Remember:

Every expert was once a beginner
The best way to learn is by doing
This series will guide you every step of the way

In the next article, we'll get your hands dirty by installing Terraform and setting up your development environment. You'll run your first Terraform command and be ready to start building real infrastructure.

👉 What's Next?

Next Article: Part 2: Installing Terraform and Setting Up Your Environment

In the next article, you'll learn:

✅ How to install Terraform on Linux, Mac, and Windows
✅ Setting up AWS CLI and credentials
✅ Configuring VS Code for Terraform development
✅ Running your first Terraform commands
✅ Verifying your setup is working correctly

See you in the next article! Let's start building! 🚀

📌 Wrapping Up

Thank you for reading. I hope this article provided practical insights and a clearer understanding of the topic.

If you found this useful:

❤️ Like if it added value
🦄 Unicorn if you’re applying it today
💾 Save it for your next optimization session
🔄 Share it with your team

💡 What’s Next

More deep dives are coming soon on:

Cloud Operations
GenAI & Agentic AI
DevOps Automation
Data & Platform Engineering

Follow along for weekly insights and hands-on guides.

🌐 Portfolio & Work

You can explore my full body of work, certifications, architecture projects, and technical articles here:

👉 Visit My Website

🛠️ Services I Offer

If you're looking for hands-on guidance or collaboration, I provide:

Cloud Architecture Consulting (AWS / Azure)
DevSecOps & Automation Design
FinOps Optimization Reviews
Technical Writing (Cloud, DevOps, GenAI)
Product & Architecture Reviews
Mentorship & 1:1 Technical Guidance

🤝 Let’s Connect

I’d love to hear your thoughts. Feel free to drop a comment or connect with me on:

🔗 LinkedIn

For collaborations, consulting, or technical discussions, reach out at:

📧 simplynadaf@gmail.com

Found this helpful? Share it with your team.

⭐ Star the repo • 📖 Follow the series • 💬 Ask questions

Made by Sarvar Nadaf

🌐 https://sarvarnadaf.com