Forem: Akhilesh Mishra

Every Kubernetes Concept Has a Story

Akhilesh Mishra — Sat, 21 Mar 2026 22:26:23 +0000

Kubernetes Concepts are not random; each one has a beautiful problem-solving story behind it.

Most people learn Kubernetes the wrong way.

They see it as a list of concepts. Pods. Deployments. Services. Ingress. ConfigMaps. Secrets. HPA. They memorize them without understanding why they exist.

Every concept in Kubernetes exists because something broke. Someone ran it in production, something failed, and a new concept was born to fix it.

Let me show you the full story in order.

You start with a Pod.

A pod runs your container. Simple. Clean. Done.

Until it crashes.

Nobody restarts it. It is just gone.

In production, that is not acceptable.

So you use a Deployment.

A Deployment watches your pods.

One dies and it creates another.

You want 3 running, it keeps 3 running.

You want to scale to 10, one command does it.

Pods were too fragile for production. Deployment fixed that.

But now you have a new problem.

Every pod gets a new IP when it restarts.

You have 3 pods running your app.

Another service needs to talk to them.

Which IP do you use? They keep changing.

You cannot hardcode them. You cannot track them at scale.

So you use a Service.

A Service gives your app one stable IP address.

It finds your pods using labels, not IPs.

Pods die and come back with new IPs. The Service does not care.

It always finds them. It also load balances.

Traffic coming in gets distributed across all healthy pods automatically.

Pods had unstable IPs. Service fixed that.

But your app still needs to be accessible from the internet.

So you use a LoadBalancer Service.

This creates a real cloud load balancer. AWS ALB. Azure LB. GCP LB.

Your app gets a public endpoint.

Works perfectly. Until you have 10 services.

Now you have 10 load balancers. Each one costs money every single month.

Your cloud bill does not care that 6 of them handle almost no traffic.

LoadBalancer Services solved external access. But one per service does not scale.

So you use Ingress.

One load balancer. All your services behind it.

Ingress routes traffic based on rules.

Request comes in for /api, goes to the API service.

Request comes in for /dashboard, goes to the frontend service.

One entry point. Smart routing. One cloud load balancer on your bill.

But Ingress is just a set of rules. Something has to execute those rules.

So you use an Ingress Controller.

Nginx. Traefik. AWS Load Balancer Controller.

These are the actual engines that read your Ingress rules and make the routing happen.

Ingress without a controller is just a config file nobody reads.

Ingress Controller made the rules actually work.

If you want to learn Kubernetes the way it works in production, I am starting a 9-week advanced Kubernetes on AWS bootcamp where we build real production projects, troubleshoot live incidents, write RCAs, and implement AIops on Kubernetes. Checkout 👇

9-week Advanced Kubernetes Bootcamp on AWS (EKS) + AIops

Now your app is running. But it needs configuration.

Database URL. API keys. Environment name. Feature flags.

So you do what feels natural. You hardcode them inside the container.

It works on your laptop.

You deploy to staging. Wrong database URL.

You deploy to production. Wrong API key.

You fix it by rebuilding the image every time config changes.

In production, rebuilding an image to change a config value is not acceptable.

So you use a ConfigMap.

A ConfigMap holds your configuration outside the container.

You inject it into your pod at runtime as environment variables or a mounted file.

Change the ConfigMap. Redeploy. Your app picks up the new values.

The image never changes. The config does.

Same image runs in dev, staging and production with different configs.

Hardcoded config made your image environment-specific. ConfigMap fixed that.

But now you have a new problem.

Your database password is sitting in a ConfigMap.

ConfigMaps are not encrypted.

Anyone with basic kubectl access can read them.

You just stored your production database credentials in plain text inside your cluster.

That is not a mistake. That is a security incident.

So you use a Secret.

A Secret holds sensitive data. Passwords. Tokens. Certificates. API keys.

Stored separately from ConfigMaps with its own access controls.

Your app reads it at runtime. Your image never sees it.

You control who in the cluster can access which Secret.

ConfigMaps were not safe for sensitive data. Secrets fixed that.

Now traffic starts growing. And manual scaling breaks you.

Some days 100 users. Some days 10,000.

You are running 3 pods. On a busy day all three are maxed out.

Users are seeing slow responses. Requests are timing out.

You jump on, run a scale command, bump it to 8 pods. Crisis over.

Traffic drops at night. 8 pods sitting idle. Wasting money.

Next spike you do it all over again.

You cannot babysit your cluster every time traffic changes.

So you use HPA. Horizontal Pod Autoscaler.

HPA watches your pods continuously.

CPU goes above 70 percent. It adds more pods automatically.

Traffic drops. It scales back down automatically.

You define the minimum and maximum. Kubernetes does the rest.

Your app handles the spike. You are not woken up at 2am to manually scale.

Manual scaling could not keep up with real traffic. HPA fixed that.

But scaling pods created a new problem.

HPA adds pods during a traffic spike.

Your nodes are full.

The new pods sit in Pending state.

They cannot be scheduled because there is no capacity.

HPA did its job. But your cluster had nowhere to put the pods.

Scaling pods without scaling nodes is half a solution.

So you use Cluster Autoscaler or Karpenter.

They watch for pods stuck in Pending state.

Not enough capacity. They add a new node automatically.

Pending pods get scheduled. Traffic is handled.

Load drops. Nodes sit underutilized. They remove them automatically.

You only pay for the compute you actually need.

On EKS, Karpenter is the better choice. It is faster and more cost efficient. It provisions the exact right node for your workload instead of waiting for a fixed node group to scale.

HPA scaled your pods. Karpenter scaled your nodes. Together they make your cluster truly elastic.

But one last problem. And it is the one that takes things down silently.

Everything is scaling. Pods are coming up. Nodes are being added.

One pod starts consuming 4GB of memory. It was never supposed to.

But nobody told Kubernetes that. So it keeps consuming.

It starves every other pod on that node.

Those pods start failing. A cascade begins.

One rogue pod. No limits. Your entire node is affected.

An unpredictable cluster is an unreliable cluster.

So you use Resource Requests and Limits.

Requests tell Kubernetes the minimum your pod needs to be scheduled on a node.

Limits tell Kubernetes the maximum it is ever allowed to consume.

The scheduler places pods intelligently across nodes using requests.

Limits make sure one noisy pod cannot take down everything around it.

Your cluster runs predictably. Every pod gets what it needs. Nothing more.

Uncontrolled resource usage made your cluster unpredictable. Requests and Limits fixed that.

To summarize the full story:

Pod ran your app but had no resilience. Deployment fixed that.

Pods had unstable IPs. Service fixed that.

One load balancer per service was too expensive. Ingress fixed that.

Ingress needed something to execute its rules. Ingress Controller fixed that.

Hardcoded config made images inflexible. ConfigMap fixed that.

ConfigMaps were not safe for sensitive data. Secrets fixed that.

Manual scaling could not keep up with traffic. HPA fixed that.

Pod scaling without node scaling left pods pending. Karpenter fixed that.

Uncontrolled resource usage made clusters unpredictable. Requests and Limits fixed that.

Each concept exists because the previous one was not enough.

That is how you stop memorizing Kubernetes and start understanding it.

Deploy Secure Google Cloud VMs with Terraform and GitHub Actions in 2025

Akhilesh Mishra — Wed, 23 Jul 2025 09:57:44 +0000

Automate VM provisioning, networking, and CI/CD pipelines using Infrastructure as Code best practices

Part 2 of our comprehensive Terraform on Google Cloud series - Building on your VPC foundation with compute resources and automation

Ready to automate your VM deployments like a DevOps pro? This tutorial builds directly on our Part 1 foundation, adding secure Compute Engine instances with automated GitHub Actions workflows. You'll master firewall rules, Cloud NAT setup, and production-ready CI/CD patterns.

By the end of this guide, you'll have a complete automated pipeline that deploys VMs to Google Cloud every time you push code to your GitHub repository.

Perfect for teams wanting to eliminate manual infrastructure deployments and embrace true DevOps automation.

What You'll Learn in This Guide

Secure VM provisioning with proper service accounts and networking
Firewall rule management for SSH access and internet connectivity
Cloud NAT configuration for private VM internet access
GitHub Actions automation with service account authentication
Production CI/CD patterns for infrastructure deployment

Prerequisites

Before starting this tutorial, ensure you have:

✅ Completed Part 1 of this series (VPC and storage setup)

✅ GitHub account with repository access

✅ Git installed locally with basic knowledge

✅ GitHub personal access token created

✅ Same GCP project from Part 1 with billing enabled

Understanding Google Cloud Compute Components

What are Firewall Rules in GCP?

Firewall rules in Google Cloud are security policies that control inbound and outbound traffic to your VM instances and other resources within a VPC network. They work at the network level and are stateful - meaning return traffic is automatically allowed.

Key concepts:

Direction: INGRESS (incoming) or EGRESS (outgoing) traffic
Priority: Lower numbers = higher priority (0-65534)
Targets: Which resources the rule applies to (tags, service accounts)
Sources/Destinations: Where traffic can come from or go to
Protocols and Ports: What type of traffic is allowed

What is Cloud NAT?

Cloud NAT (Network Address Translation) provides outbound internet connectivity for VM instances that only have private IP addresses. This is essential for:

Security: VMs don't need public IPs to access the internet
Cost optimization: Fewer external IP addresses needed
Compliance: Keeps resources private while allowing necessary internet access

What is Cloud Router?

Cloud Router is a networking service that provides dynamic routing capabilities. It's required for Cloud NAT and enables:

Dynamic routing between VPCs and on-premises networks
BGP support for advanced networking scenarios
NAT gateway functionality when paired with Cloud NAT

Extending Your Terraform Configuration

Let's add the new variables and resources to your existing Terraform setup from Part 1.

Add VM Variables (`variables.tf`)

Add these variables to your existing variables.tf file:

variable "zone" {
  type        = string
  description = "GCP zone for VM deployment"
  default     = "us-central1-b"
  validation {
    condition     = can(regex("^[a-z]+-[a-z]+[0-9]+-[a-z]$", var.zone))
    error_message = "Zone must be a valid GCP zone format (e.g., us-central1-b)."
  }
}

variable "vm_machine_type" {
  type        = string
  description = "Machine type for the VM instance"
  default     = "e2-standard-2"
  validation {
    condition = contains([
      "e2-micro", "e2-small", "e2-medium", "e2-standard-2", 
      "e2-standard-4", "n1-standard-1", "n1-standard-2"
    ], var.vm_machine_type)
    error_message = "Machine type must be a valid GCP machine type."
  }
}

variable "prefix" {
  type        = string
  description = "Prefix for resource naming"
  default     = "main"
  validation {
    condition     = can(regex("^[a-z][a-z0-9-]{1,10}$", var.prefix))
    error_message = "Prefix must start with a letter, contain only lowercase letters, numbers, and hyphens, and be 2-11 characters long."
  }
}

variable "vm_disk_size" {
  type        = number
  description = "Boot disk size in GB"
  default     = 50
  validation {
    condition     = var.vm_disk_size >= 10 && var.vm_disk_size <= 2000
    error_message = "VM disk size must be between 10 and 2000 GB."
  }
}

variable "vm_image" {
  type        = string
  description = "VM boot disk image"
  default     = "debian-cloud/debian-12"
  validation {
    condition = contains([
      "debian-cloud/debian-12", "ubuntu-os-cloud/ubuntu-2204-lts",
      "centos-cloud/centos-7", "rhel-cloud/rhel-8"
    ], var.vm_image)
    error_message = "VM image must be a supported OS image."
  }
}

Update Variable Values (`terraform.tfvars`)

Add these values to your existing terraform.tfvars file:

# Existing variables from Part 1
project_id  = "your-gcp-project-id"
environment = "dev"
region      = "us-central1"
vpc_name    = "main-vpc"
subnet_name = "primary-subnet"
subnet_cidr = "10.0.1.0/24"

# New VM variables
zone            = "us-central1-b"
vm_machine_type = "e2-standard-2"
prefix          = "demo"
vm_disk_size    = 50
vm_image        = "debian-cloud/debian-12"

Create VM Resources (`vm.tf`)

Create a new file vm.tf for your compute resources:

# Service account for VM instances
resource "google_service_account" "vm_service_account" {
  project      = var.project_id
  account_id   = "${var.prefix}-vm-sa"
  display_name = "VM Service Account for ${var.environment}"
  description  = "Service account for VM instances with minimal required permissions"
}

# IAM roles for VM service account
resource "google_project_iam_member" "vm_sa_logging" {
  project = var.project_id
  role    = "roles/logging.logWriter"
  member  = "serviceAccount:${google_service_account.vm_service_account.email}"
}

resource "google_project_iam_member" "vm_sa_monitoring" {
  project = var.project_id
  role    = "roles/monitoring.metricWriter"
  member  = "serviceAccount:${google_service_account.vm_service_account.email}"
}

# Compute Engine VM instance
resource "google_compute_instance" "main_vm" {
  name                      = "${var.prefix}-${var.environment}-vm"
  project                   = var.project_id
  machine_type              = var.vm_machine_type
  zone                      = var.zone
  tags                      = ["ssh-allowed", "http-server"]
  allow_stopping_for_update = true

  # Enable deletion protection in production
  deletion_protection = var.environment == "prod" ? true : false

  # Boot disk configuration
  boot_disk {
    initialize_params {
      image = var.vm_image
      type  = "pd-standard"
      size  = var.vm_disk_size
      labels = {
        environment = var.environment
        managed-by  = "terraform"
      }
    }
    auto_delete = true
  }

  # Network configuration - private IP only
  network_interface {
    network    = google_compute_network.main_vpc.self_link
    subnetwork = google_compute_subnetwork.main_subnet.self_link

    # No external IP - will use Cloud NAT for internet access
    # Uncomment below for external IP:
    # access_config {
    #   nat_ip = google_compute_address.vm_external_ip.address
    # }
  }

  # Service account and scopes
  service_account {
    email  = google_service_account.vm_service_account.email
    scopes = [
      "https://www.googleapis.com/auth/cloud-platform",
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring.write"
    ]
  }

  # Metadata for VM configuration
  metadata = {
    enable-oslogin = "TRUE"
    startup-script = <<-EOF
      #!/bin/bash
      apt-get update
      apt-get install -y nginx
      systemctl start nginx
      systemctl enable nginx
      echo "<h1>Hello from ${var.prefix}-${var.environment}-vm</h1>" > /var/www/html/index.html
      echo "<p>Deployed with Terraform and GitHub Actions</p>" >> /var/www/html/index.html
    EOF
  }

  # Labels for resource management
  labels = {
    environment = var.environment
    managed-by  = "terraform"
    team        = "devops"
  }

  # Depends on the subnet being ready
  depends_on = [
    google_compute_subnetwork.main_subnet
  ]
}

# Optional: Static external IP (commented out for security)
# resource "google_compute_address" "vm_external_ip" {
#   name    = "${var.prefix}-${var.environment}-vm-ip"
#   project = var.project_id
#   region  = var.region
# }

Create Firewall Rules (`firewall.tf`)

Create a new file firewall.tf for network security:

# Ingress firewall rule for SSH access
resource "google_compute_firewall" "allow_ssh" {
  name    = "${var.environment}-allow-ssh"
  network = google_compute_network.main_vpc.name
  project = var.project_id

  description = "Allow SSH access to instances with ssh-allowed tag"
  direction   = "INGRESS"
  priority    = 1000

  allow {
    protocol = "tcp"
    ports    = ["22"]
  }

  # Restrict source ranges for better security
  # For production, use your specific IP ranges
  source_ranges = ["0.0.0.0/0"]  # Change this in production!
  target_tags   = ["ssh-allowed"]

  # Log firewall activity
  log_config {
    metadata = "INCLUDE_ALL_METADATA"
  }
}

# Ingress firewall rule for HTTP traffic
resource "google_compute_firewall" "allow_http" {
  name    = "${var.environment}-allow-http"
  network = google_compute_network.main_vpc.name
  project = var.project_id

  description = "Allow HTTP access to web servers"
  direction   = "INGRESS" 
  priority    = 1000

  allow {
    protocol = "tcp"
    ports    = ["80", "8080"]
  }

  source_ranges = ["0.0.0.0/0"]
  target_tags   = ["http-server"]

  log_config {
    metadata = "INCLUDE_ALL_METADATA"
  }
}

# Egress firewall rule for internet access
resource "google_compute_firewall" "allow_egress" {
  name    = "${var.environment}-allow-egress"
  network = google_compute_network.main_vpc.name
  project = var.project_id

  description = "Allow outbound internet access"
  direction   = "EGRESS"
  priority    = 1000

  allow {
    protocol = "tcp"
    ports    = ["80", "443", "53"]
  }

  allow {
    protocol = "udp"
    ports    = ["53", "123"]
  }

  destination_ranges = ["0.0.0.0/0"]
  target_tags        = ["ssh-allowed", "http-server"]
}

# Internal communication firewall rule
resource "google_compute_firewall" "allow_internal" {
  name    = "${var.environment}-allow-internal"
  network = google_compute_network.main_vpc.name
  project = var.project_id

  description = "Allow internal communication within VPC"
  direction   = "INGRESS"
  priority    = 1000

  allow {
    protocol = "tcp"
    ports    = ["0-65535"]
  }

  allow {
    protocol = "udp"
    ports    = ["0-65535"]
  }

  allow {
    protocol = "icmp"
  }

  # Allow traffic from the VPC CIDR
  source_ranges = [var.subnet_cidr, "10.1.0.0/16", "10.2.0.0/16"]
}

Update Networking with Cloud NAT (`networking.tf`)

Add these resources to your existing networking.tf file:

# Cloud NAT for private VM internet access
resource "google_compute_router_nat" "main_nat" {
  name   = "${var.environment}-nat-gateway"
  router = google_compute_router.main_router.name
  region = var.region

  # NAT IP allocation
  nat_ip_allocate_option = "AUTO_ONLY"

  # Which subnets to provide NAT for
  source_subnetwork_ip_ranges_to_nat = "LIST_OF_SUBNETWORKS"

  subnetwork {
    name                    = google_compute_subnetwork.main_subnet.name
    source_ip_ranges_to_nat = ["ALL_IP_RANGES"]
  }

  # Logging for monitoring
  log_config {
    enable = true
    filter = "ERRORS_ONLY"
  }

  # Timeout settings
  min_ports_per_vm               = 64
  udp_idle_timeout_sec          = 30
  icmp_idle_timeout_sec         = 30
  tcp_established_idle_timeout_sec = 1200
  tcp_transitory_idle_timeout_sec  = 30
}

Update Outputs (`outputs.tf`)

Add these VM-related outputs to your existing outputs.tf:

# Existing outputs from Part 1...

# VM outputs
output "vm_name" {
  value       = google_compute_instance.main_vm.name
  description = "Name of the created VM instance"
}

output "vm_internal_ip" {
  value       = google_compute_instance.main_vm.network_interface[0].network_ip
  description = "Internal IP address of the VM"
}

output "vm_zone" {
  value       = google_compute_instance.main_vm.zone
  description = "Zone where the VM is deployed"
}

output "vm_service_account" {
  value       = google_service_account.vm_service_account.email
  description = "Service account email used by the VM"
}

output "ssh_command" {
  value       = "gcloud compute ssh ${google_compute_instance.main_vm.name} --zone=${var.zone} --project=${var.project_id}"
  description = "Command to SSH into the VM instance"
}

# NAT gateway output
output "nat_gateway_name" {
  value       = google_compute_router_nat.main_nat.name
  description = "Name of the Cloud NAT gateway"
}

Setting Up GitHub Actions Automation

Now let's create an automated CI/CD pipeline that will deploy your infrastructure whenever you push code to GitHub.

Create GitHub Secrets

First, you need to store your service account credentials in GitHub secrets:

Get your service account key content:

cat terraform-sa-key.json

In your GitHub repository:
- Go to Settings > Secrets and variables > Actions
- Click New repository secret
- Name: GCLOUD_SERVICE_ACCOUNT_KEY
- Value: Paste the entire contents of your service account key file

Security Note: This approach using service account keys is for learning purposes. In Part 3, we'll upgrade to Workload Identity Federation for keyless authentication, which is the production-recommended approach.

Create GitHub Actions Workflow

Create the workflow directory and file:

mkdir -p .github/workflows
touch .github/workflows/deploy.yml

Add this workflow configuration to deploy.yml:

name: Terraform Infrastructure Deployment

on:
  push:
    branches:
      - main
      - develop
  pull_request:
    branches:
      - main

env:
  PROJECT_ID: your-gcp-project-id  # Replace with your project ID
  TF_VERSION: 1.6.0

jobs:
  terraform:
    name: Terraform Plan and Apply
    runs-on: ubuntu-latest

    permissions:
      contents: read
      id-token: write
      pull-requests: write

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Google Cloud SDK
        uses: google-github-actions/setup-gcloud@v1
        with:
          version: 'latest'
          service_account_key: ${{ secrets.GCLOUD_SERVICE_ACCOUNT_KEY }}
          project_id: ${{ env.PROJECT_ID }}

      - name: Configure Terraform authentication
        run: |
          echo '${{ secrets.GCLOUD_SERVICE_ACCOUNT_KEY }}' > /tmp/gcp-key.json
          gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
          gcloud config set project ${{ env.PROJECT_ID }}
          gcloud config set compute/region us-central1
          export GOOGLE_APPLICATION_CREDENTIALS=/tmp/gcp-key.json

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: ${{ env.TF_VERSION }}

      - name: Terraform Format Check
        id: fmt
        run: terraform fmt -check -diff -recursive
        continue-on-error: true

      - name: Terraform Initialize
        id: init
        run: terraform init

      - name: Terraform Validation
        id: validate
        run: terraform validate

      - name: Terraform Plan
        id: plan
        run: |
          terraform plan -input=false -out=tfplan -detailed-exitcode
        continue-on-error: true

      - name: Comment PR with Plan
        if: github.event_name == 'pull_request'
        uses: actions/github-script@v7
        with:
          script: |
            const output = `#### Terraform Format and Style 🖌\`${{ steps.fmt.outcome }}\`
            #### Terraform Initialization ⚙️\`${{ steps.init.outcome }}\`
            #### Terraform Validation 🤖\`${{ steps.validate.outcome }}\`
            #### Terraform Plan 📖\`${{ steps.plan.outcome }}\`

            <details><summary>Show Plan</summary>

            \`\`\`terraform
            ${{ steps.plan.outputs.stdout }}
            \`\`\`

            </details>

            *Pusher: @${{ github.actor }}, Action: \`${{ github.event_name }}\`*`;

            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: output
            })

      - name: Terraform Apply
        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
        run: terraform apply -input=false tfplan

      - name: Clean up credentials
        if: always()
        run: |
          rm -f /tmp/gcp-key.json
          rm -f tfplan

      - name: Output VM Information
        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
        run: |
          echo "## 🚀 Deployment Complete!" >> $GITHUB_STEP_SUMMARY
          echo "| Resource | Value |" >> $GITHUB_STEP_SUMMARY
          echo "|----------|-------|" >> $GITHUB_STEP_SUMMARY
          echo "| VM Name | $(terraform output -raw vm_name) |" >> $GITHUB_STEP_SUMMARY
          echo "| Internal IP | $(terraform output -raw vm_internal_ip) |" >> $GITHUB_STEP_SUMMARY
          echo "| SSH Command | \`$(terraform output -raw ssh_command)\` |" >> $GITHUB_STEP_SUMMARY

Deploying Your Infrastructure

Local Deployment First

Before pushing to GitHub, test your configuration locally:

# Format and validate
terraform fmt -recursive
terraform validate

# Plan and apply
terraform plan -out=tfplan
terraform apply tfplan

# Verify deployment
terraform output

Expected Output

You should see output similar to:

vm_internal_ip = "10.0.1.2"
vm_name = "demo-dev-vm"
vm_zone = "us-central1-b"
ssh_command = "gcloud compute ssh demo-dev-vm --zone=us-central1-b --project=your-project-id"

Test VM Connectivity

# SSH into your VM (if you have gcloud configured)
gcloud compute ssh demo-dev-vm --zone=us-central1-b --project=your-project-id

# Test internet connectivity from VM
curl -I http://google.com

# Check nginx is running
curl localhost

Push to GitHub for Automated Deployment

Once local testing works:

git add .
git commit -m "Add VM deployment with GitHub Actions automation"
git push origin main

GitHub Actions will:

Validate your Terraform code
Plan the infrastructure changes
Apply changes automatically on main branch
Comment on pull requests with plan details

Monitoring Your Deployment

Check GitHub Actions

Go to your repository on GitHub
Click the Actions tab
Watch your workflow run in real-time
Check the job summary for deployment details

Verify in GCP Console

Compute Engine > VM instances - see your VM
VPC network > Firewall - check your firewall rules
Network services > Cloud NAT - verify NAT configuration
Logging > Logs Explorer - monitor VM and firewall logs

Test Your VM

# List your VMs
gcloud compute instances list

# SSH into the VM
gcloud compute ssh demo-dev-vm --zone=us-central1-b

# Inside the VM, test internet access
curl -I https://www.google.com

# Check nginx status
sudo systemctl status nginx
curl localhost

Production Considerations

Security Enhancements

Firewall Rules:

Restrict SSH access to specific IP ranges
Use IAP (Identity-Aware Proxy) for SSH access
Implement network tags consistently

Service Accounts:

Follow principle of least privilege
Use separate service accounts for different workloads
Regularly rotate service account keys

Monitoring and Logging

Enable Monitoring:

# Add to vm.tf
metadata = {
  enable-oslogin = "TRUE"
  google-monitoring-enabled = "TRUE"
  google-logging-enabled = "TRUE"
}

Set up Alerts:

VM instance health checks
High CPU or memory usage
Network connectivity issues
Unusual SSH access patterns

Cost Optimization

Right-sizing:

Monitor VM performance and adjust machine types
Use preemptible instances for non-critical workloads
Implement auto-shutdown for development VMs

Storage Optimization:

Use appropriate disk types (pd-standard vs pd-ssd)
Enable disk auto-deletion with VMs
Implement disk snapshots for backups

Troubleshooting Common Issues

Authentication Failures

# Re-authenticate gcloud
gcloud auth application-default login

# Verify project setting
gcloud config get-value project

# Check service account permissions
gcloud projects get-iam-policy YOUR_PROJECT_ID

VM Won't Start

# Check VM status
gcloud compute instances describe VM_NAME --zone=ZONE

# View serial console output
gcloud compute instances get-serial-port-output VM_NAME --zone=ZONE

# Check quotas
gcloud compute project-info describe --project=PROJECT_ID

Network Connectivity Issues

# Test firewall rules
gcloud compute firewall-rules list

# Check routes
gcloud compute routes list

# Verify NAT configuration
gcloud compute routers get-nat-mappings ROUTER_NAME --region=REGION

What's Next in Part 3

In our next tutorial, we'll enhance this setup with:

🔜 Secure PostgreSQL deployment with Cloud SQL

🔜 Workload Identity Federation for keyless GitHub Actions

🔜 Private service connections for database security

🔜 Advanced networking with VPC peering

Key Takeaways

✅ Automated infrastructure reduces human error and increases deployment speed

✅ GitHub Actions integration enables true Infrastructure as Code workflows

✅ Proper firewall rules are essential for both security and functionality

✅ Cloud NAT provides secure internet access without public IPs

✅ Service account best practices improve security posture from the start

Ready to level up your infrastructure automation? This foundation of automated VM deployment sets you up perfectly for the advanced database and security patterns we'll cover in Part 3!

Connect with me:

LinkedIn for more Google Cloud and DevOps content

Tags: #Terraform #GoogleCloud #GCP #GitHubActions #DevOps #InfrastructureAsCode #ComputeEngine #Automation

Build Production-Ready Google Cloud Infrastructure with Terraform in 2025

Akhilesh Mishra — Wed, 23 Jul 2025 09:39:26 +0000

Complete step-by-step guide to creating VPC networks, subnets, and storage buckets using Infrastructure as Code

Part 1 of our comprehensive 6-part Terraform on Google Cloud series - from beginner setup to advanced DevOps automation

Ready to transform your Google Cloud infrastructure management? This comprehensive guide kicks off our Terraform on Google Cloud series, where you'll master professional-level cloud automation, GitHub Actions CI/CD, advanced security patterns, and production-ready DevOps practices.

By the end of this tutorial series, you'll have hands-on experience with:

Infrastructure as Code fundamentals and best practices
Automated CI/CD pipelines with GitHub Actions and Workload Identity Federation
Production security with Secret Manager and key rotation
Serverless computing with Cloud Functions and Cloud Run
Big data processing with Cloud SQL and Dataproc

🚀 Perfect for cloud engineers, DevOps practitioners, and developers ready to level up their infrastructure automation game.

What You'll Build in This Series

Part 1 (This Post): Foundation - VPC, Subnets, and Storage

Part 2: Compute Engine VMs with GitHub Actions CI/CD

Part 3: Secure PostgreSQL with Workload Identity Federation

Part 4: Secret Management and Automated Key Rotation

Part 5: Serverless FastAPI with Cloud Run

Part 6: Big Data Processing with Dataproc (Advanced)

Understanding Google Cloud Networking Fundamentals

What is a VPC in Google Cloud?

A Virtual Private Cloud (VPC) serves as your isolated network environment within Google Cloud:

🔒 Network Isolation: Complete separation from other projects and tenants
🛡️ Security Control: Fine-grained access control and firewall rules
🌍 Global Resource: Automatically spans multiple regions worldwide
📍 IP Management: Custom IP addressing, subnets, and routing control
🔗 Connectivity: VPN, interconnect, and peering capabilities

What is a Subnet in Google Cloud?

Subnets are regional network segments within your VPC that:

Define IP address ranges (CIDR blocks) for your resources
Enable regional resource deployment and isolation
Provide traffic segmentation and security boundaries
Support private Google API access for enhanced security
Allow custom routing and firewall rule application

Pro Tip: Unlike AWS, Google Cloud subnets are regional (not zonal), giving you automatic high availability across zones within a region.

Prerequisites: Setting Up Your Environment

Before diving into Terraform automation, ensure you have:

✅ Google Cloud Platform account (Sign up here)

✅ Active GCP project with billing enabled

✅ Terraform installed (Installation guide) - Version 1.0+ recommended

✅ Google Cloud SDK (Download here)
✅ Basic understanding of cloud networking concepts

Verify Your Installation

# Check Google Cloud SDK
gcloud --version
gcloud init 

# Verify Terraform
terraform --version

# Check your active project
gcloud config get-value project

Google Cloud Authentication Setup

Creating a Service Account (Development Setup)

For this tutorial series, we'll create a service account with owner permissions. Important: In production environments, always follow the principle of least privilege and use keyless authentication with Workload Identity Federation (covered in Part 3).

# Set your project ID (replace with your actual project ID)
export PROJECT_ID="your-gcp-project-id"
gcloud config set project $PROJECT_ID

# Create service account for Terraform
gcloud iam service-accounts create terraform-automation \
    --description="Service account for Terraform infrastructure automation" \
    --display-name="Terraform Automation SA"

# Grant owner role (development only - we'll improve this in later parts)
gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:terraform-automation@${PROJECT_ID}.iam.gserviceaccount.com" \
    --role="roles/owner"

# Generate service account key
gcloud iam service-accounts keys create terraform-sa-key.json \
    --iam-account terraform-automation@${PROJECT_ID}.iam.gserviceaccount.com

Set Environment Variables

# Set credentials for Terraform
export GOOGLE_APPLICATION_CREDENTIALS="terraform-sa-key.json"

# Verify authentication
gcloud auth application-default print-access-token

Project Structure: Organizing Your Terraform Code

Create a clean, maintainable project structure that will scale throughout our series:

mkdir terraform-gcp-foundation && cd terraform-gcp-foundation

# Create core Terraform files
touch main.tf variables.tf outputs.tf providers.tf terraform.tfvars

# Create resource-specific files
touch networking.tf storage.tf

# Create backend configuration
touch backend.tf

# Initialize git for version control
git init
echo "*.tfvars" >> .gitignore
echo "terraform-sa-key.json" >> .gitignore
echo ".terraform/" >> .gitignore
echo "*.tfstate*" >> .gitignore

Why this structure?

Separation of concerns: Each file has a specific purpose
Scalability: Easy to add new resources in dedicated files
Team collaboration: Clear organization for multiple developers
Version control ready: Proper .gitignore for sensitive files

Terraform Configuration Files Explained

1. Provider Configuration (`providers.tf`)

terraform {
  required_version = ">= 1.0"

  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 5.0"
    }
    google-beta = {
      source  = "hashicorp/google-beta" 
      version = "~> 5.0"
    }
  }
}

provider "google" {
  project = var.project_id
  region  = var.region
}

provider "google-beta" {
  project = var.project_id
  region  = var.region
}

2. Input Variables (`variables.tf`)

variable "project_id" {
  type        = string
  description = "Google Cloud Project ID"
  validation {
    condition     = length(var.project_id) > 6
    error_message = "Project ID must be more than 6 characters."
  }
}

variable "region" {
  type        = string
  description = "GCP region for resources"
  default     = "us-central1"
  validation {
    condition = contains([
      "us-central1", "us-east1", "us-west1", "us-west2",
      "europe-west1", "europe-west2", "asia-east1"
    ], var.region)
    error_message = "Region must be a valid GCP region."
  }
}

variable "environment" {
  type        = string
  description = "Environment name (dev, staging, prod)"
  default     = "dev"
  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "Environment must be dev, staging, or prod."
  }
}

variable "vpc_name" {
  type        = string
  description = "Name for the VPC network"
  default     = "main-vpc"
}

variable "subnet_name" {
  type        = string
  description = "Name for the subnet"
  default     = "main-subnet"
}

variable "subnet_cidr" {
  type        = string
  description = "CIDR range for the subnet"
  default     = "10.0.1.0/24"
  validation {
    condition     = can(cidrhost(var.subnet_cidr, 0))
    error_message = "Subnet CIDR must be a valid IPv4 CIDR block."
  }
}

3. Networking Resources (`networking.tf`)

# VPC Network
resource "google_compute_network" "main_vpc" {
  name                    = "${var.environment}-${var.vpc_name}"
  project                 = var.project_id
  auto_create_subnetworks = false
  routing_mode           = "REGIONAL"
  mtu                    = 1460

  description = "Main VPC network for ${var.environment} environment"

  # Enable deletion protection in production
  delete_default_routes_on_create = false
}

# Subnet
resource "google_compute_subnetwork" "main_subnet" {
  name                     = "${var.environment}-${var.subnet_name}"
  project                  = var.project_id
  region                   = var.region
  network                  = google_compute_network.main_vpc.name
  ip_cidr_range           = var.subnet_cidr
  private_ip_google_access = true

  description = "Primary subnet in ${var.region} for ${var.environment}"

  # Enable flow logs for security monitoring
  log_config {
    aggregation_interval = "INTERVAL_10_MIN"
    flow_sampling       = 0.5
    metadata           = "INCLUDE_ALL_METADATA"
  }

  # Secondary IP ranges for future use (GKE, etc.)
  secondary_ip_range {
    range_name    = "pods"
    ip_cidr_range = "10.1.0.0/16"
  }

  secondary_ip_range {
    range_name    = "services"
    ip_cidr_range = "10.2.0.0/16"
  }
}

# Cloud Router for Cloud NAT (we'll use this in Part 2)
resource "google_compute_router" "main_router" {
  name    = "${var.environment}-cloud-router"
  project = var.project_id
  region  = var.region
  network = google_compute_network.main_vpc.id

  description = "Cloud Router for NAT gateway"
}

4. Storage Resources (`storage.tf`)

# Storage bucket for Terraform state (remote backend)
resource "google_storage_bucket" "terraform_state" {
  name     = "${var.project_id}-${var.environment}-terraform-state"
  project  = var.project_id
  location = var.region

  # Force destroy for development (disable in production)
  force_destroy = var.environment == "dev" ? true : false

  # Enable versioning for state file safety
  versioning {
    enabled = true
  }

  # Uniform bucket-level access for better security
  uniform_bucket_level_access = true

  # Encryption at rest
  encryption {
    default_kms_key_name = null # We'll add KMS in Part 4
  }

  # Lifecycle management to control costs
  lifecycle_rule {
    condition {
      age = 30
    }
    action {
      type = "Delete"
    }
  }

  lifecycle_rule {
    condition {
      age                   = 7
      with_state           = "ARCHIVED"
    }
    action {
      type = "Delete"
    }
  }

  # Labels for resource management
  labels = {
    environment = var.environment
    purpose     = "terraform-state"
    managed-by  = "terraform"
  }
}

# General purpose storage bucket
resource "google_storage_bucket" "app_storage" {
  name     = "${var.project_id}-${var.environment}-app-storage"
  project  = var.project_id
  location = var.region

  # Storage class optimization
  storage_class = "STANDARD"

  uniform_bucket_level_access = true

  # CORS configuration for web applications
  cors {
    origin          = ["*"]
    method          = ["GET", "HEAD", "PUT", "POST", "DELETE"]
    response_header = ["*"]
    max_age_seconds = 3600
  }

  labels = {
    environment = var.environment
    purpose     = "application-storage"
    managed-by  = "terraform"
  }
}

# IAM binding for service account access to state bucket
resource "google_storage_bucket_iam_member" "terraform_state_access" {
  bucket = google_storage_bucket.terraform_state.name
  role   = "roles/storage.admin"
  member = "serviceAccount:terraform-automation@${var.project_id}.iam.gserviceaccount.com"
}

5. Outputs (`outputs.tf`)

output "vpc_name" {
  value       = google_compute_network.main_vpc.name
  description = "Name of the created VPC"
}

output "vpc_id" {
  value       = google_compute_network.main_vpc.id
  description = "ID of the created VPC"
}

output "vpc_self_link" {
  value       = google_compute_network.main_vpc.self_link
  description = "Self-link of the VPC (useful for other resources)"
}

output "subnet_name" {
  value       = google_compute_subnetwork.main_subnet.name
  description = "Name of the created subnet"
}

output "subnet_cidr" {
  value       = google_compute_subnetwork.main_subnet.ip_cidr_range
  description = "CIDR range of the subnet"
}

output "subnet_gateway_address" {
  value       = google_compute_subnetwork.main_subnet.gateway_address
  description = "Gateway IP address of the subnet"
}

output "terraform_state_bucket" {
  value       = google_storage_bucket.terraform_state.name
  description = "Name of the Terraform state storage bucket"
}

output "app_storage_bucket" {
  value       = google_storage_bucket.app_storage.name
  description = "Name of the application storage bucket"
}

output "cloud_router_name" {
  value       = google_compute_router.main_router.name
  description = "Name of the Cloud Router (for NAT in Part 2)"
}

# Network details for use in subsequent parts
output "network_details" {
  value = {
    vpc_name    = google_compute_network.main_vpc.name
    subnet_name = google_compute_subnetwork.main_subnet.name
    region      = var.region
    project_id  = var.project_id
  }
  description = "Network configuration details for other Terraform configurations"
}

6. Variable Values (`terraform.tfvars`)

# Project Configuration
project_id  = "your-gcp-project-id"  # Replace with your actual project ID
environment = "dev"

# Network Configuration
region      = "us-central1"
vpc_name    = "main-vpc"
subnet_name = "primary-subnet"
subnet_cidr = "10.0.1.0/24"

7. Backend Configuration (`backend.tf`)

# Local backend for initial setup
terraform {
  backend "local" {
    path = "terraform.tfstate"
  }
}

# After creating the storage bucket, uncomment below and migrate:
# terraform {
#   backend "gcs" {
#     bucket = "your-project-id-dev-terraform-state"
#     prefix = "foundation/state"
#   }
# }

Understanding Terraform State Management

Terraform state is the crucial component that tracks your infrastructure:

📊 Resource Metadata: Current state, IDs, and configurations
🔗 Dependencies: Relationships and creation order between resources
📝 Performance: Caches resource attributes for faster operations
🎯 Drift Detection: Identifies manual changes made outside Terraform

State Storage Options

Local State (Development):

Stored on your local machine
Simple for learning and testing
Not suitable for team collaboration

Remote State (Production):

Google Cloud Storage (recommended for GCP)
Terraform Cloud (HashiCorp's managed solution)
Amazon S3 (for multi-cloud scenarios)

Migrating to Remote State

After your first deployment, migrate to remote state:

# 1. Update backend.tf with your bucket name
# 2. Run migration command
terraform init -migrate-state

Deploying Your Infrastructure

Step 1: Initialize Terraform

terraform init

Downloads provider plugins and initializes the working directory

Step 2: Format and Validate

# Format code for consistency
terraform fmt -recursive

# Validate configuration
terraform validate

Step 3: Plan Your Deployment

terraform plan -out=tfplan

Creates an execution plan and saves it to a file

Understanding the Plan Output:

+ Resources to be created
- Resources to be destroyed
~ Resources to be modified
<= Resources to be read (data sources)

Step 4: Apply Changes

terraform apply tfplan

Executes the plan - no additional confirmation needed since plan was saved

Step 5: Verify Deployment

# Check outputs
terraform output

# Verify in GCP Console
gcloud compute networks list
gcloud compute networks subnets list --network=dev-main-vpc
gsutil ls gs://your-project-id-dev-terraform-state

Step 6: Clean Up (When Needed)

terraform destroy

⚠️ Removes all managed infrastructure - be extremely careful in production!

Production-Ready Best Practices

🔧 Code Organization & Standards

Modularize configurations for reusability across environments
Use consistent naming conventions with environment prefixes
Implement proper file structure with clear separation of concerns
Version pin providers and modules to avoid breaking changes

🔒 Security & Access Management

Never commit sensitive data or credentials to version control
Use remote state backend with proper access controls and encryption
Implement state locking to prevent concurrent modifications
Follow least privilege principle for service account permissions

📚 Documentation & Team Collaboration

Add meaningful descriptions to all resources and variables
Use version control (Git) with proper branching strategies
Implement peer reviews for all infrastructure changes
Document runbooks for common operations and troubleshooting

🚀 Automation & CI/CD (Coming in Part 2)

Integrate with GitHub Actions for automated deployments
Use environment-specific variable files and workspaces
Implement automated testing for infrastructure code
Set up monitoring and alerting for infrastructure changes

💰 Cost Optimization

Use labels and tags for resource tracking and cost allocation
Implement lifecycle policies for storage resources
Choose appropriate machine types and disk sizes
Monitor and set up budget alerts for cost control

What's Coming Next in This Series

🔜 Part 2: Compute Engine VMs with GitHub Actions

Deploy secure VM instances with proper networking
Set up automated CI/CD pipelines with GitHub Actions
Implement firewall rules and Cloud NAT for secure internet access
Master service account management and authentication

🔜 Part 3: PostgreSQL with Workload Identity Federation

Deploy Cloud SQL PostgreSQL in private networks
Implement keyless authentication with Workload Identity Federation
Set up VPC peering and private service connections
Database security and backup configuration

🔜 Part 4: Secret Management and Key Rotation

Secure credential storage with Secret Manager
Automated service account key rotation with Cloud Functions
Production-grade security patterns
Integration with existing infrastructure

🔜 Part 5: Serverless FastAPI with Cloud Run

Containerize and deploy Python applications
Implement proper artifact registry workflows
Configure custom domains and SSL certificates
Performance optimization and scaling strategies

🔜 Part 6: Big Data with Dataproc (Advanced)

Set up managed Hadoop and Spark clusters
Configure preemptible instances for cost optimization
Data processing pipelines and job orchestration
Integration with Cloud Storage and BigQuery

Key Takeaways

✅ Infrastructure as Code provides consistency, repeatability, and version control

✅ Proper project structure makes maintenance and collaboration easier

✅ Variable validation catches errors early in the development cycle

✅ Remote state management is essential for team collaboration and production use

✅ Security best practices should be implemented from day one, not as an afterthought

Ready to continue? This foundation sets you up for success in the remaining parts of our series. Each subsequent tutorial builds upon this infrastructure, adding more sophisticated patterns and production-ready features.

Troubleshooting Common Issues

Authentication Problems

# Re-authenticate if needed
gcloud auth application-default login
gcloud config set project YOUR_PROJECT_ID

API Enablement

# Enable required APIs
gcloud services enable compute.googleapis.com
gcloud services enable storage-api.googleapis.com

Permission Issues

# Verify service account permissions
gcloud projects get-iam-policy YOUR_PROJECT_ID \
  --flatten="bindings[].members" \
  --filter="bindings.members:serviceAccount:terraform-automation@YOUR_PROJECT_ID.iam.gserviceaccount.com"

Connect with me:

Tags: #Terraform #GoogleCloud #GCP #DevOps #InfrastructureAsCode #CloudAutomation #NetworkingSeries

Managing AWS Lambda Layers Was a Nightmare Until I Did This

Akhilesh Mishra — Fri, 27 Jun 2025 12:17:15 +0000

Automation That Saved My Team Months of Toil

Automate Lambda Layer management with Terraform and GitHub Actions - A complete guide from pain to automation

I started my DevOps career with the Google Cloud platform and used Cloud Functions for event-driven serverless workloads. It was pretty straightforward — package the Python code with its dependencies, i.e. requirements.txt and everything else will be taken care of by Cloud Function (i.e. GCP).

Then I moved to AWS and discovered Lambda, their popular serverless offering. It was great, except for one headache: managing code dependencies.

I was used to just using a requirements.txt file, but that doesn't work with Lambda. Instead, you have two options. You can either bundle all your libraries with your code in a big zip file (which is a pain to manage) or use something called Lambda Layers.

Before I explain Layers and how to use them effectively, let me give you a quick rundown on Lambda and serverless computing in general.

What is Serverless?
What is AWS Lambda?
What are Lambda Layers?
How to Build Lambda Layers
The Automation Solution
Implementation Guide
Running the Automation

What is Serverless?

Don't let the term "serverless" fool you — it doesn't mean servers have vanished into thin air. Instead, think of it as "servers you don't see." In this model, your cloud provider manages the resources for you, handling all the nitty-gritty of server maintenance.

At its core, serverless computing is about simplifying your life as a developer or business owner. It's a world where you can focus on what truly matters — your code and business logic — without getting bogged down in the complexities of managing the machines that run it.

That's the promise of serverless computing — it's about freeing you to innovate, while your cloud provider handles the heavy lifting of infrastructure management.

What is AWS Lambda?

Lambda is the Function as a Service (serverless) offering by AWS. It uses an event-driven architecture, meaning it runs a piece of code when some event occurs. It integrates well with other AWS services such as DynamoDB, S3, SQS, API Gateway, etc.

Your code may be just a couple of lines, or spread across 2 files but if it requires 15 different packages to tie everything together, it's referred to as a "deployment package."

It is best practice to manage the code dependencies separately and this is where Lambda Layers come in handy.

What are Lambda Layers?

To quote AWS documentation:

A Lambda layer is a .zip file archive that contains supplementary code or data. Layers usually contain library dependencies, a custom runtime, or configuration files.

In simple words, if your Python code requires 5, 10, 15 libraries to run the code, you build them separately and enable Lambda to use them on runtime.

Key benefits of Lambda Layers:

📦 Separate dependencies from your application code
🔄 Share layers across multiple Lambda functions
🌐 Share layers across multiple AWS accounts
💾 Reduce deployment package size
⚡ Faster deployments and cold starts

How to Build Lambda Layers

Manual Method (The Old Way)

The simplest way of building Layers is to install the Python dependencies in your local machine inside a Python directory, zip it, and upload it to AWS layers. Your Lambda function can use the layer.

⚠️ Important Note: Creating a zip package from a non-Linux environment (even Mac) will NOT WORK with Lambda.

Docker Method

Alternatively, you can use Docker with --platform linux/amd64 flag to build the layer with the help of sam/build-python Docker images.

docker run --platform linux/amd64 -v \
"$PWD":/var/task "public.ecr.aws/sam/build-python3.10" /bin/sh \
-c "pip install -r requirements.txt -t python/; exit"

Cross-Account Sharing

If you want to share your Layer in a different AWS account, add permissions by running:

aws lambda add-layer-version-permission \
    --layer-name <Layer_name> \        
    --statement-id xaccount \
    --action lambda:GetLayerVersion  \
    --principal <AWS_ACCOUNT_ID> \
    --version-number <Layer_Version>

The Problem: Managing Lambda Layers manually can be a lot of pain when every other month you need to update the libraries to use new versions of libraries to take your security team off your back.

The Automation Solution

Automating Layer Build with Terraform and GitHub Actions

Before we start writing Terraform, create the file structure in your VSCode as shown below:

lambda-python-layers/
├── .github/
│   └── workflows/
│       └── build-lambda-layers.yml
├── layers/
│   ├── layer1/
│   │   └── requirements.txt
│   ├── layer2/
│   │   └── requirements.txt
│   └── common-scripts/
│       ├── logger.py
│       └── db_connection.py
├── providers.tf
├── variables.tf
├── config.tf
├── s3.tf
└── main.tf

Defining Python Dependencies for Layers

layers/layer1/requirements.txt

aws-psycopg2
requests==2.32.3
urllib3==2.2.2

layers/layer2/requirements.txt

pyzipper==0.3.6
pycryptodome==3.20.0
cryptography==41.0.7
PyPDF2==3.0.1

Sometimes we also need to use common scripts such as logger, and db-connection scripts used by Lambda functions. We can package them with the dependencies to avoid packaging them with Lambda code or building separate layers for these scripts.

Place those Python scripts in layers/common-scripts/ path.

Implementation Guide

Writing Terraform Configuration

💡 Note: You can find the complete code for this blog in my public GitHub repo.

providers.tf

Setting up Terraform providers and remote backend using S3 bucket I have already created to store the Terraform state files:

terraform {
  required_version = ">= 1.6"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
  backend "s3" {
    bucket = "your-terraform-state-bucket"
    key    = "lambda-layers/terraform.tfstate"
    region = "ap-south-1"
  }
}

provider "aws" {
  region = var.aws_region
}

variables.tf

We will store default values for Python layers such as runtime, and compatible runtimes:

variable "aws_region" {
  description = "AWS region"
  type        = string
  default     = "ap-south-1"
}

variable "python_runtime" {
  description = "Python runtime version"
  type        = string
  default     = "python3.10"
}

config.tf

We will create local variables to store the layer details such as layer name and path to the dependencies:

locals {
  layer_definitions = [
    {
      identifier = "layer1"
      path       = "${path.module}/layers/layer1"
    },
    {
      identifier = "layer2"
      path       = "${path.module}/layers/layer2"
    }
  ]
}

s3.tf

Use the data source to get the AWS account ID and use it as a prefix to ensure the unique name of the S3 bucket, and use it to store the layers packages in .zip format:

data "aws_caller_identity" "current" {}

resource "aws_s3_bucket" "lambda_layers" {
  bucket = "${data.aws_caller_identity.current.account_id}-lambda-layers-bucket"
}

resource "aws_s3_bucket_versioning" "lambda_layers" {
  bucket = aws_s3_bucket.lambda_layers.id
  versioning_configuration {
    status = "Enabled"
  }
}

main.tf

It will use the Terraform module (terraform-aws-modules/lambda/aws) for Lambda to build Python layers. I will package the custom scripts along with the layers for usability:

module "layers" {
  source   = "terraform-aws-modules/lambda/aws"
  version  = "~> 6.0"
  for_each = { for i in local.layer_definitions : i.identifier => i }

  create_layer = true

  layer_name          = each.value.identifier
  description         = "Lambda layer for ${each.value.identifier}"
  compatible_runtimes = [var.python_runtime]

  source_path = [
    {
      path             = each.value.path
      pip_requirements = true
      prefix_in_zip    = "python"
    },
    {
      path          = "${path.module}/layers/common-scripts"
      prefix_in_zip = "python"
    }
  ]

  store_on_s3 = true
  s3_bucket   = aws_s3_bucket.lambda_layers.bucket
}

Cross-Account Layer Sharing (Optional)

If you want to share the layers across multiple accounts, add these configurations:

Append to config.tf:

locals {
  # List of layer names
  layer_names = [for i in local.layer_definitions : i.identifier]
  # Accounts that should have permission on layer version
  allowed_accounts = ["AWS_ACCOUNT_ID_1", "AWS_ACCOUNT_ID_2"]
  # Mapping layers -> aws accounts for permission on layer version
  layers_to_accounts = flatten([
    for layer in local.layer_names : [
      for account in local.allowed_accounts : {
        id      = "${layer}-${account}"
        layer   = layer
        account = account
      }
    ]
  ])

  # Map to be used by for_each loop for resource
  layers_to_accounts_map = { for item in local.layers_to_accounts : item.id => item }
}

Append to main.tf:

resource "aws_lambda_layer_version_permission" "lambda_layer_permission" {
  for_each       = local.layers_to_accounts_map
  layer_name     = module.layers[each.value.layer].lambda_layer_layer_arn
  version_number = module.layers[each.value.layer].lambda_layer_version
  principal      = each.value.account
  action         = "lambda:GetLayerVersion"
  statement_id   = "${each.value.layer}-${each.value.account}-${random_integer.random.result}"
}

resource "random_integer" "random" {
  min = 1
  max = 1000
}

GitHub Actions Workflow

Here is the GitHub Action workflow that will build the layers:

.github/workflows/build-lambda-layers.yml

name: Build Lambda Layers with Terraform
on:
  # On push to main branch when there is a change in dependencies
  push:
    branches:
      - main
    paths:
      - lambda-python-layers/layers/
  # Run manually
  workflow_dispatch:

jobs:
  terraform:
    runs-on: ubuntu-latest
    env:
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      AWS_REGION: ap-south-1
      TERRAFORM_VER: 1.6.6
      TERRAFORM_PATH: lambda-python-layers/
      PYTHON_VERSION: '3.10'

    steps:
      - name: Checkout Repository
        uses: actions/checkout@v4

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: ${{ env.TERRAFORM_VER }}

      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: ${{ env.PYTHON_VERSION }}

      - name: Terraform Init
        working-directory: ${{ env.TERRAFORM_PATH }}
        run: terraform init

      - name: Terraform Plan
        working-directory: ${{ env.TERRAFORM_PATH }}
        run: terraform plan -out=tfplan

      - name: Terraform Apply
        working-directory: ${{ env.TERRAFORM_PATH }}
        run: terraform apply -auto-approve tfplan

Workflow Explanation

This GitHub Action workflow is designed to deploy Terraform to build the layers for a specific path (layers/) within a repository.

Triggers:

On push to the main branch, specifically if files within the lambda-python-layers/layers/ directory are updated
Can be triggered manually via workflow_dispatch

Job Configuration:

Job runs on the ubuntu-latest runner
AWS credentials and region are sourced from GitHub Secrets
Terraform version (1.6.6) and Python version (3.10) are specified

Steps:

Checkout repository using actions/checkout@v4
Install Terraform using hashicorp/setup-terraform@v3
Install Python 3.10 using actions/setup-python@v5
Run Terraform commands to deploy the layers

Running the Automation

Step-by-Step Setup

You can clone the code from my Public GitHub Repo:

# Clone my GitHub repo
git clone https://github.com/akhileshmishrabiz/Devops-zero-to-hero.git

# Create a folder/directory 
mkdir lambda-layer-example
cd lambda-layer-example

# Copy the terraform from Devops-zero-to-hero/lambda-python-layers
cp -r Devops-zero-to-hero/lambda-python-layers .

# Create a .github/workflows folder to store the GitHub Action workflow
mkdir -p .github/workflows

# Create a .yml file 
touch .github/workflows/layer-build.yml

# Copy the workflow code from the cloned repo
cp Devops-zero-to-hero/.github/workflows/build-lambda-layer.yml .github/workflows/layer-build.yml

# Create a github repository and push the code from lambda-layer-example to your repo

Configure GitHub Secrets

Create access and secret keys and store them as secrets in your GitHub repo:

Go to your repo → Settings → Secrets and variables → Actions
Create two secrets:
- AWS_ACCESS_KEY_ID = Your AWS Access Key
- AWS_SECRET_ACCESS_KEY = Your AWS Secret Key

Deploy the Automation

Push the code to trigger the workflow, or run it manually from the Actions tab.

You can see the layers in your AWS account: Lambda → Layers

AWS Lambda Layers in Console

Benefits of This Automation

Before Automation	After Automation
❌ Manual zip creation	✅ Automated builds
❌ Platform compatibility issues	✅ Consistent Linux builds
❌ Version management pain	✅ Automated versioning
❌ Manual uploads	✅ Automatic deployment
❌ Cross-account sharing complexity	✅ Terraform-managed permissions

Conclusion

This automation solution has saved my team countless hours of manual Lambda Layer management. Instead of spending time on repetitive packaging and uploading tasks, we can focus on writing better Lambda functions.

The combination of Terraform and GitHub Actions provides:

Consistency across environments
Reproducibility of builds
Version control for dependencies
Automated security updates

The Terraform code for this is available here, and the GitHub Action workflow is here.

About Me

I am Akhilesh Mishra, a self-taught DevOps engineer with 11+ years working on private and public cloud (GCP & AWS) technologies.

Connect with me:

You Are Making A Big Mistake By Not Scanning Your Terraform Code

Akhilesh Mishra — Fri, 27 Jun 2025 09:55:32 +0000

Stop infrastructure misconfigurations before they become security nightmares - Master Terrascan for bulletproof IaC security

Complete DevSecOps Guide to Terrascan for Infrastructure as Code Security

Last year, someone I know deployed a Kubernetes cluster that accidentally exposed internal APIs to the internet because of a misconfigured network policy. They caught it during a routine scan, but it could have been much worse.

Infrastructure misconfigurations aren't just embarrassing — they're potentially catastrophic.

Infrastructure misconfigurations have been behind some of the most notorious security incidents in recent years. From the Capital One breach caused by a misconfigured WAF to the thousands of Elasticsearch instances leaking sensitive data, the pattern is clear: the security of your cloud depends heavily on your configuration, not just your code.

Security in infrastructure code often feels like an afterthought. We're so focused on making things work that we forget to make them secure.

At my company, we already use Checkov for basic security scanning, but we've found limitations in its policy coverage and customization options. That's why we're adding Terrascan to our security toolkit — it offers deeper analysis, better policy flexibility, and more comprehensive coverage across different cloud providers.

Why Infrastructure Security Scanning Matters
What is Terrascan?
Installation and Setup
Basic Usage and Commands
Advanced Scanning Techniques
Docker Integration
CI/CD Pipeline Integration
Custom Security Policies
Best Practices and Recommendations
Troubleshooting Common Issues

Why Infrastructure Security Scanning Matters

The Hidden Cost of Misconfigurations

Real-world impact of IaC security failures:

🏦 Capital One (2019): Misconfigured WAF led to 100+ million customer records exposed
📊 Elasticsearch incidents: Thousands of instances leaked sensitive data due to default configurations
☁️ AWS S3 breaches: Countless incidents from misconfigured bucket permissions
🔐 Kubernetes exposures: APIs accidentally exposed to the internet via network policies

Common Infrastructure Security Risks

Risk Category	Examples	Impact Level
Network Security	Open security groups, exposed databases	🔴 Critical
Access Control	Overprivileged IAM roles, public buckets	🔴 Critical
Encryption	Unencrypted storage, data in transit	🟠 High
Compliance	Missing logging, audit trails	🟡 Medium
Resource Management	Untagged resources, cost overruns	🟡 Medium

What is Terrascan?

Terrascan is an open-source static code analyzer created by Tenable, specially designed for Infrastructure as Code security. It's your first line of defense against configuration vulnerabilities.

Key Features That Set Terrascan Apart

✅ Multi-Cloud Support: AWS, Azure, GCP, Kubernetes

✅ Multiple IaC Tools: Terraform, Helm, Kubernetes, Docker

✅ Custom Policies: Write organization-specific security rules

✅ CI/CD Integration: Seamless DevOps workflow integration

✅ Compliance Frameworks: SOC 2, PCI DSS, GDPR, HIPAA

✅ Active Community: Regular updates and policy additions

Terrascan vs Competitors

Feature	Terrascan	Checkov	tfsec	Bridgecrew
Open Source	✅ Free	✅ Free	✅ Free	❌ Paid
Custom Policies	✅ Rego	✅ Python	❌ Limited	✅ Advanced
Multi-Cloud	✅ Excellent	✅ Good	✅ Good	✅ Excellent
Docker Scanning	✅ Yes	✅ Yes	❌ No	✅ Yes
Learning Curve	🟡 Moderate	🟢 Easy	🟢 Easy	🔴 Steep

Installation and Setup

Method 1: Homebrew (Mac/Linux)

# Install via Homebrew (recommended for Mac)
brew install terrascan

# Verify installation
terrascan version

Method 2: Manual Installation

# Download latest release
curl -L "$(curl -s https://api.github.com/repos/tenable/terrascan/releases/latest | grep -o -E "https://.+?_Darwin_x86_64.tar.gz")" > terrascan.tar.gz

# Extract and install
tar -xf terrascan.tar.gz terrascan && rm terrascan.tar.gz
sudo install terrascan /usr/local/bin && rm terrascan

# Verify installation
terrascan version

Method 3: Docker

# Run as Docker container
docker run --rm tenable/terrascan version

# Scan current directory
docker run --rm -it -v "$(pwd):/iac" -w /iac tenable/terrascan scan

Basic Usage and Commands

Your First Scan

Navigate to any directory containing Infrastructure as Code files and run:

# Basic scan - scans all supported files in current directory
terrascan scan

Example output:

Violated Policies (7):
  LOW     | Ensure S3 bucket has versioning enabled
  MEDIUM  | Ensure S3 bucket has access logging enabled  
  HIGH    | Ensure S3 bucket is not publicly readable
  HIGH    | Security group allows ingress from 0.0.0.0/0 to port 22

I ran this command with my code and this is how the results look like

Essential Command Flags

# Get comprehensive help
terrascan-help

terrascan scan --help

Flags:
      --config-only               will output resource config (should only be used for debugging purposes)
      --config-with-error         will output resource config and errors (if any)
      --find-vuln                 fetches vulnerabilities identified in Docker images
  -h, --help                      help for scan
  -d, --iac-dir string            path to a directory containing one or more IaC files (default ".")
  -f, --iac-file string           path to a single IaC file
  -i, --iac-type string           iac type (arm, cft, docker, helm, k8s, kustomize, terraform, tfplan)
      --iac-version string        iac version (arm: v1, cft: v1, docker: v1, helm: v3, k8s: v1, kustomize: v2, v3, v4, terraform: v12, v13, v14, v15, tfplan: v1)
      --non-recursive             do not scan directories and modules recursively
  -p, --policy-path stringArray   policy path directory
  -t, --policy-type strings       policy type (all, aws, azure, docker, gcp, github, k8s) (default [all])
  -r, --remote-type string        type of remote backend (git, s3, gcs, http, terraform-registry)
  -u, --remote-url string         url pointing to remote IaC repository
      --repo-ref string           branch of the repo being scanned
      --repo-url string           URL of the repo being scanned, will be reflected in scan summary
      --scan-rules strings        one or more rules to scan (example: --scan-rules="ruleID1,ruleID2")
      --severity string           minimum severity level of the policy violations to be reported by terrascan
      --show-passed               display passed rules, along with violations
      --skip-rules strings        one or more rules to skip while scanning (example: --skip-rules="ruleID1,ruleID2")
      --use-colors string         color output (auto, t, f) (default "auto")
      --use-terraform-cache       use terraform init cache for remote modules (when used directory scan will be non recursive, flag applicable only with terraform IaC provider)
      --values-files strings      one or more values files to scan(applicable for iactype=helm) (example: --values-files="file1-values.yaml,file2-values.yaml")
  -v, --verbose                   will show violations with details (applicable for default output)
      --webhook-token string      optional token used when sending authenticated requests to the notification webhook
      --webhook-url string        webhook URL where Terrascan will send JSON scan report and normalized IaC JSON


Global Flags:
  -c, --config-path string      config file path
  -l, --log-level string        log level (debug, info, warn, error, panic, fatal) (default "info")
      --log-output-dir string   directory path to write the log and output files
  -x, --log-type string         log output type (console, json) (default "console")
  -o, --output string           output type (human, json, yaml, xml, junit-xml, sarif, github-sarif) (default "human")
      --temp-dir string         temporary directory path to download remote repository,module and templates

Target-Specific Scanning

# Scan specific cloud provider
terrascan scan -t aws           # AWS only
terrascan scan -t azure         # Azure only  
terrascan scan -t gcp           # GCP only
terrascan scan -t k8s           # Kubernetes only

# Scan specific IaC type
terrascan scan -i terraform     # Terraform files
terrascan scan -i helm          # Helm charts
terrascan scan -i docker        # Dockerfiles
terrascan scan -i k8s           # Kubernetes manifests

Output Format Options

# JSON output for automation
terrascan scan -o json

# YAML output
terrascan scan -o yaml

# SARIF format for GitHub integration
terrascan scan -o sarif

# JUnit XML for CI reporting
terrascan scan -o junit-xml

Try out different flags with the command

terrascan scan -t gcp -i terraform  -o json --severity "High"

Advanced Scanning Techniques

Severity-Based Filtering

# Only show high severity issues
terrascan scan --severity "High"

# Show medium and high severity
terrascan scan --severity "Medium"

# Verbose output with details
terrascan scan -v

File and Directory Targeting

# Scan specific file
terrascan scan -f main.tf

# Scan specific directory
terrascan scan -d /path/to/terraform/modules

# Non-recursive scan (current directory only)
terrascan scan --non-recursive

Rule Management

# Skip specific rules
terrascan scan --skip-rules="AWS.S3.DS.High.1041,AWS.EC2.NetworkACL.Medium.0506"

# Scan only specific rules
terrascan scan --scan-rules="AWS.S3.DS.High.1041,AWS.RDS.DS.High.1025"

# Show passed rules along with violations
terrascan scan --show-passed

Remote Repository Scanning

# Scan remote Git repository
terrascan scan -r git -u https://github.com/user/repo.git

# Scan specific branch
terrascan scan -r git -u https://github.com/user/repo.git --repo-ref develop

# Scan Terraform modules from registry
terrascan scan -r terraform-registry -u terraform-aws-modules/vpc/aws

Configuration Debugging

# Output resource configuration for debugging
terrascan scan --config-only -o json

# Show configuration with errors
terrascan scan --config-with-error

# Enable debug logging
terrascan scan -l debug

Docker Integration

Scanning Dockerfiles

# Scan Dockerfiles for vulnerabilities
terrascan scan -i docker


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wn8vko9mi99c9mc18sxd.webp)

# Find vulnerabilities in container images
terrascan scan -i docker --find-vuln

Running Terrascan in Docker

# Basic Docker usage
docker run --rm -v "$(pwd):/iac" -w /iac tenable/terrascan scan

# With specific parameters
docker run --rm -v "$(pwd):/iac" -w /iac tenable/terrascan scan -i terraform -t aws -o json

CI/CD Pipeline Integration

GitHub Actions Integration

Create .github/workflows/terrascan.yml:

name: Terrascan Security Scan
on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

jobs:
  terrascan_job:
    runs-on: ubuntu-latest
    name: Terrascan Security Analysis
    steps:
    - name: Checkout repository
      uses: actions/checkout@v3

    - name: Run Terrascan
      id: terrascan
      uses: tenable/terrascan-action@main
      with:
        iac_type: 'terraform'
        iac_version: 'v15'
        policy_type: 'aws'
        only_warn: true
        sarif_upload: true

    - name: Upload SARIF file
      uses: github/codeql-action/upload-sarif@v2
      with:
        sarif_file: terrascan.sarif

GitLab CI Integration

Add to .gitlab-ci.yml:

terrascan_security:
  stage: security
  image: tenable/terrascan:latest
  script:
    - terrascan scan -i terraform -t aws -o json --config-path terrascan-config.toml
  artifacts:
    reports:
      sast: terrascan-results.json
  only:
    - merge_requests
    - main

Jenkins Integration

pipeline {
    agent any
    stages {
        stage('Terrascan Security Scan') {
            steps {
                script {
                    docker.image('tenable/terrascan:latest').inside {
                        sh 'terrascan scan -i terraform -o junit-xml > terrascan-results.xml'
                    }
                }
            }
            post {
                always {
                    publishTestResults testResultsPattern: 'terrascan-results.xml'
                }
            }
        }
    }
}

Pre-commit Hooks

Create .pre-commit-config.yaml:

repos:
- repo: https://github.com/tenable/terrascan
  rev: v1.18.0
  hooks:
  - id: terrascan
    args: ['scan', '-i', 'terraform', '--severity', 'High']
    verbose: true

Install and use:

# Install pre-commit
pip install pre-commit

# Install hooks
pre-commit install

# Run manually
pre-commit run terrascan --all-files

Custom Security Policies

Understanding Rego Policy Language

Terrascan uses Rego (from Open Policy Agent) for custom policies. This allows you to create organization-specific security rules.

Example: Prevent Public SSH Access

Create custom-policies/terraform/no_public_ssh/rule.rego:

package accurics

# Rule to deny SSH access from 0.0.0.0/0
deny_public_ssh[msg] {
    resource := input.aws_security_group[_]
    rule := resource.config.ingress[_]
    rule.from_port <= 22
    rule.to_port >= 22
    rule.cidr_blocks[_] == "0.0.0.0/0"

    msg := sprintf("Security group '%s' allows SSH (port 22) from 0.0.0.0/0", [resource.name])
}

Example: Enforce Resource Tagging

Create custom-policies/terraform/require_tags/rule.rego:

package accurics

required_tags := ["Environment", "Owner", "Project"]

# Check if EC2 instances have required tags
deny_missing_tags[msg] {
    resource := input.aws_instance[_]
    missing_tags := required_tags[_]
    not resource.config.tags[missing_tags]

    msg := sprintf("EC2 instance '%s' is missing required tag '%s'", [resource.name, missing_tags])
}

Using Custom Policies

# Scan with custom policies
terrascan scan -p ./custom-policies

# Combine with built-in policies
terrascan scan -p ./custom-policies -t aws

Policy Development Best Practices

Start Simple: Begin with basic rules and iterate
Test Thoroughly: Use different scenarios to validate policies
Document Rules: Include clear descriptions and examples
Version Control: Store policies in Git with your infrastructure code
Peer Review: Have security and DevOps teams review custom policies

Best Practices and Recommendations

1. Implement Shift-Left Security

# Run in development environment
terrascan scan -i terraform --severity High

# Integrate with IDE plugins
# Use VS Code Terrascan extension

2. Create Security Gates

# GitHub Actions example with failure conditions
- name: Terrascan Security Gate
  run: |
    terrascan scan -i terraform -o json > results.json
    HIGH_ISSUES=$(jq '.results.violations[] | select(.severity == "HIGH") | length' results.json)
    if [ "$HIGH_ISSUES" -gt 0 ]; then
      echo "❌ High severity security issues found. Blocking deployment."
      exit 1
    fi

3. Configure Organization Policies

Create terrascan-config.toml:

[rules]
    skip-rules = [
        "AWS.S3.DS.High.1041",  # Allow specific exceptions
    ]

[severity]
    level = "MEDIUM"

[output]
    format = "json"

[notifications]
    webhook-url = "https://hooks.slack.com/your-webhook"

4. Regular Policy Updates

# Stay updated with latest policies
docker pull tenable/terrascan:latest

# Check for new rule releases
terrascan scan --help | grep -A 20 "policy-type"

Troubleshooting Common Issues

Issue 1: False Positives

Problem: Legitimate configurations flagged as violations

Solution:

# Skip specific rules for false positives
terrascan scan --skip-rules="RULE_ID_1,RULE_ID_2"

# Use inline comments in Terraform
resource "aws_s3_bucket" "example" {
  # ts:skip=AWS.S3.DS.High.1041 Business requirement for public access
  bucket = "my-public-bucket"
}

Issue 2: Performance with Large Codebases

Problem: Slow scanning on large repositories

Solution:

# Use non-recursive scanning
terrascan scan --non-recursive

# Scan specific directories
terrascan scan -d terraform/modules/security

# Use .terrascignore file
echo "*.backup" > .terrascignore
echo "test/" >> .terrascignore

Issue 3: Custom Policy Errors

Problem: Rego policy compilation errors

Solution:

# Test policy syntax
opa fmt custom-policies/

# Validate policy structure
opa test custom-policies/

Advanced Configuration Options

Webhook Integration

# Send results to webhook
terrascan scan --webhook-url https://your-webhook.com/endpoint --webhook-token your-token

Multi-Environment Scanning

# Development environment
terrascan scan -d environments/dev --severity Low

# Production environment  
terrascan scan -d environments/prod --severity High --config-path prod-config.toml

Security Scanning Metrics and KPIs

Track your infrastructure security improvements:

Metric	Target	Measurement
Critical Issues	0	High severity violations
Scan Coverage	100%	% of IaC files scanned
Policy Compliance	>95%	Passed rules / Total rules
Mean Time to Fix	<24h	Time from detection to resolution

Conclusion

Infrastructure security can't be an afterthought anymore. With the rise in cloud misconfigurations leading to major breaches, scanning your Infrastructure as Code is no longer optional — it's essential.

Key takeaways:

🔒 Prevention is cheaper than remediation - Catch issues before deployment

⚡ Automation is key - Integrate scanning into your CI/CD pipeline

📋 Custom policies matter - Tailor security rules to your organization

📊 Measure and improve - Track security metrics and continuously improve

Terrascan makes infrastructure security scanning accessible and powerful, automatically checking your code against hundreds of security policies while allowing complete customization for your organization's specific needs.

Start implementing Terrascan today, and transform your infrastructure security from reactive to proactive.

Resources and References

How to Auto-Generate requirements.txt for Python Projects: Complete Guide to pipreqs and pigar

Akhilesh Mishra — Thu, 26 Jun 2025 11:29:20 +0000

Stop manually hunting for Python dependencies - automate requirements.txt generation with these powerful tools

As a DevOps engineer, I frequently collaborate with multiple teams, diving into various projects and repositories. One common challenge I face is encountering Python code without a requirements.txt file.

Every time I run such code, I need to figure out and manually install the necessary dependencies. I found it a waste of time and effort. I am sure you guys also feel the same way.

So I dug around, did some Google searches, experimented with a few tools, and found many good ones. My favorites are pipreqs and pigar.

In this comprehensive guide, I'll show you how to use these Python dependency management tools to generate requirements.txt files automatically for any Python project.

Why Not Use pip freeze?
Method 1: Using pipreqs
Method 2: Using pigar
Comparison and Best Practices

Why Not Use pip freeze?

Before diving into the solutions, let's address why pip freeze isn't always the best choice for generating requirements.txt:

❌ pip freeze problems:

Only saves packages installed with pip install in your current environment
Includes ALL packages in the environment, even those not used in your project (if you don't use virtualenv)
Sometimes you need to create requirements.txt for a new project without installing modules first
Can include system-wide packages that aren't relevant to your project

✅ What we need instead:

Tools that analyze your actual Python code
Generate requirements based on import statements
Create clean, project-specific dependency lists

Method 1: Using pipreqs - The Import-Based Analyzer

What is pipreqs?

pipreqs is a Python tool that analyzes your Python files and generates requirements.txt based on the actual imports in your code. It's smart, fast, and doesn't require installing packages beforehand.

Setting Up Your Environment

I recommend working with isolated Python virtual environments for better dependency management:

# Create a project directory
mkdir python-requirements-demo
cd python-requirements-demo

# Create and activate virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

Installing pipreqs

pip install pipreqs

Practical Example: Word Cloud Generator

Let's create a real Python project to demonstrate pipreqs. Create a directory and file:

mkdir project
cd project

Create word_cloud_generator.py with the following code:

import matplotlib.pyplot as plt
from wordcloud import WordCloud
from collections import Counter
import requests
from PIL import Image
import numpy as np
import argparse
from io import BytesIO

def get_mask_image(url):
    """Download and process mask image from URL"""
    response = requests.get(url)
    img = Image.open(BytesIO(response.content))
    return np.array(img)

def create_word_cloud(text, mask_url=None):
    """Generate word cloud from text with optional mask"""
    if mask_url:
        mask = get_mask_image(mask_url)
    else:
        mask = None

    wordcloud = WordCloud(
        width=800, 
        height=400, 
        background_color='white', 
        mask=mask, 
        contour_width=1, 
        contour_color='black'
    ).generate(text)

    plt.figure(figsize=(10, 5))
    plt.imshow(wordcloud, interpolation='bilinear')
    plt.axis('off')
    plt.show()

def parse_arguments():
    """Parse command line arguments"""
    parser = argparse.ArgumentParser(description="Generate a word cloud from input text")
    parser.add_argument('--text', '-t', required=True, help="Input text for word cloud")
    parser.add_argument('--mask_url', '-m', help="URL of mask image for word cloud shape")
    return parser.parse_args()

if __name__ == "__main__":
    args = parse_arguments()
    create_word_cloud(args.text, args.mask_url)

Generating requirements.txt with pipreqs

Now comes the magic! Instead of manually figuring out dependencies, run:

# Generate requirements.txt for the project directory
pipreqs project

Output in requirements.txt:

matplotlib==3.9.1
numpy==2.0.0
Pillow==10.4.0
Requests==2.32.3
wordcloud==1.9.3

Installing and Testing Dependencies

# Install all dependencies at once
pip install -r project/requirements.txt

# Test the script
python project/word_cloud_generator.py --text "Python dependency management made easy"

# With mask image
python project/word_cloud_generator.py --text "pipreqs is awesome" --mask_url "https://example.com/mask.jpg"

Advanced pipreqs Options

# Generate requirements for specific directory
pipreqs /path/to/your/project

# Specify output file name
pipreqs . --savepath requirements-dev.txt

# Force overwrite existing requirements.txt
pipreqs . --force

# Use local database for package names
pipreqs . --use-local

# Debug mode for troubleshooting
pipreqs . --debug

Method 2: Using pigar - The Advanced Dependency Scanner

What is pigar?

pigar is another excellent tool for generating Python requirements. It offers more advanced features and better handling of complex projects.

Installing pigar

# Using pip
pip install pigar

# From source (latest features)
pip install git+https://github.com/damnever/pigar.git@main --upgrade

# Using conda
conda install -c conda-forge pigar

Real-World Example: Flask Application

Let's test pigar with a Flask application:

# Clone a sample Flask project
git clone https://github.com/akhileshmishrabiz/Devops-zero-to-hero
cd Devops-zero-to-hero/project5

# Remove existing requirements.txt for demo
rm requirements.txt

Generating requirements with pigar

# Generate requirements.txt for current directory
pigar generate

# Check the generated file
cat requirements.txt

Output:

# Automatically generated by https://github.com/damnever/pigar.

Flask==3.0.3
Flask-SQLAlchemy==3.1.1

Advanced pigar Usage

# Generate requirements for specific directory with custom filename
pigar gen -f custom-requirements.txt /path/to/project

# Example with Python Lambda project
pigar gen -f python-lambda-requirements.txt python-for-devops/python-lambda-runtime/

Output for Lambda project:

# Automatically generated by https://github.com/damnever/pigar.

boto3==1.34.142
colorlog==6.8.2
packaging==24.1

pigar Advanced Features

# Check for outdated packages
pigar check

# Search for packages
pigar search numpy

# Show package information
pigar show requests

Comparison: pipreqs vs pigar

Feature	pipreqs	pigar
Speed	⚡ Very Fast	🔄 Moderate
Accuracy	✅ High	✅ Very High
Local imports	❌ Limited	✅ Excellent
Complex projects	✅ Good	✅ Excellent
Additional features	❌ Basic	✅ Package search, outdated checks
File size	📦 Lightweight	📦 Slightly larger

Best Practices for Python Dependency Management

1. Use Virtual Environments

# Always create isolated environments
python -m venv project-env
source project-env/bin/activate

2. Pin Exact Versions

# Good - specific versions
requests==2.32.3
numpy==2.0.0

# Avoid - unpinned versions
requests
numpy

3. Separate Development Dependencies

# Create different requirement files
pigar gen -f requirements.txt .           # Production
pigar gen -f requirements-dev.txt .       # Development

4. Regular Dependency Audits

# Check for security vulnerabilities
pip-audit -r requirements.txt

# Update outdated packages
pip list --outdated

Troubleshooting Common Issues

pipreqs Not Finding All Imports

# Use debug mode
pipreqs . --debug

# Force scan all files
pipreqs . --scan-notebooks

pigar Missing Local Modules

# Include local packages
pigar gen --with-referenced-comments

Version Conflicts

# Generate without version pins
pipreqs . --no-version

# Then manually specify versions

Automation with CI/CD

GitHub Actions Example

name: Check Requirements
on: [push, pull_request]

jobs:
  check-deps:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Generate requirements
      run: |
        pip install pipreqs
        pipreqs . --force
        git diff --exit-code requirements.txt

Conclusion

Both pipreqs and pigar are excellent tools for automating Python dependency management:

Choose pipreqs for: Simple projects, fast generation, lightweight solution
Choose pigar for: Complex projects, advanced features, comprehensive analysis

Stop wasting time manually managing Python dependencies. These tools will save you hours and reduce deployment errors significantly.

Next Steps

Try both tools on your existing Python projects
Integrate dependency generation into your development workflow
Set up automated checks in your CI/CD pipeline
Share this guide with your team to standardize dependency management

I Ditched Python Built-In Logging For Loguru — You Should Too

Akhilesh Mishra — Wed, 25 Jun 2025 21:13:58 +0000

A Complete Guide to better logging in Python with loguru

I have been using Python's built-in logging module for years, I have always found it quite boring. It cannot be customized beyond a point.

I wanted to include some color coding in my log messages, but had to use a third-party package, for it. I wished Python built-in logging module has options to customise logs with ease.

Then I stumbled upon loguru and found it interesting. I spent some time with this yesterday and fell in love with it.

By the end of this blog post, you will fall for it too.

Understanding Log Levels

When talking about logging, loglevel is an important term, they are like a severity scale for your messages. By assigning different levels to log messages, it becomes easier to focus on critical issues while reducing noise from less important events while troubleshooting or monitoring.

While the Python builtin module comes with DEBUG, INFO, WARNING, ERROR, and CRITICAL, loguru include 2 more log levels, TRACE and SUCCESS.

Here is the table to remember log levels in order of increasing priority:

Level	Priority
TRACE	5
DEBUG	10
INFO	20
SUCCESS	25
WARNING	30
ERROR	40
CRITICAL	50

Enough with the talking, let me show you what we can do with loguru and why I fell in love with it.

Installing Loguru

pip install loguru

Basic Usage

To use Loguru, import the logger from loguru module and use it.

from loguru import logger

logger.trace("Hi, This is Akhilesh Mishra")
logger.debug("I will show how to use loguru for better logging in python")
logger.info(" I love how the logs look")
logger.warning("Different color of each section of log")
logger.error("Easy to get started with")
logger.critical( "So many options to choose from")
logger.success(" You see what i am talking about")

Output format: date | level | file location: scope: line number - message

You can see that the trace output is not printed. Default log level for loguru is debug.

Changing the Default Log Level

We can use logger's add() function to change the default loglevel, update the log's formatting.

import sys
from loguru import logger

logger.add(sys.stderr, level="TRACE")
logger.trace("Hi, This is Akhilesh Mishra")

Change the Formatting with Loguru

Unlike Python's built-in logging module, you can add a handler, update formatting, and change the log level in just one line — with add() function.

import sys
from loguru import logger

logger.remove() # remove the old formatting 
logger.add( sys.stdout, format="{time}::{level} --- {message}", level="INFO")

logger.debug(" Add a handler, update formatting, and change the loglevel")
logger.info(" one function to rule them all")
logger.success(" logger.add()")

Pretty Logging with Colors

You can change the color of these output messages by using HTML like syntax.

logger.remove()
logger.add(
    sys.stdout,
    format=" <yellow>{time} </yellow>:: <green> <bold> {level} </bold> </green>--- <blue> {message} </blue>",
)

logger.info("Set the colors you want to use for logs")
logger.success(" How easy it was???")

Change the Time Formatting

import sys
from loguru import logger

logger.remove()
# MMMM D, YYYY > HH:mm:ss!UTC : UTC time
logger.add(sys.stderr, format="{time:MMMM D, YYYY > HH:mm:ss!UTC} | {level} | <level>{message} </level>")

logger.warning(" Use time format-> {time:MMMM D, YYYY > HH:mm:ss!UTC}")
logger.success("Use the default color from level: <level>{message} </level>")

You can use more options with time, check the loguru documentation for more time formatting options.

Sending Logs to a File

Loguru sends logs to the console by default, but you can configure it to send logs to a file.

logger.add("log_file_demo.log")

Let's use formatted logs:

from loguru import logger

logger.remove()
logger.add("file_{time}.log", format="{time:MMMM D, YYYY > HH:mm:ss} | {level} | <level>{message} </level>")

logger.info("using file logging")
logger.success(" will send logs to the file")

Log Rotation, Retention, and Compression

Loguru also allows you to rotate/retain/compress logs with time and size filters.

logger.add("log_rotate.log", rotation="500 MB")    # Automatically rotate too big file
logger.add("log_rotate2.log", rotation="12:00")     # New file is created each day at noon
logger.add("log_rotate3.log", rotation="1 week")    # Once the file is too old, it's rotated

logger.add("log_retention.log", retention="10 days")  # Cleanup after some time

logger.add("log_retention2.log", compression="zip")    # Save some loved space

JSON Logging

Loguru supports logging in JSON format with serialize=True option.

import sys
from loguru import logger

logger.remove(0)
logger.add(
    sys.stderr,
    format="{time:MMMM D: YYYY:: HH:mm:ss!UTC} | {level} | {message}",
    serialize=True,
)
logger.warning(" Its addictive, use with caution !")
logger.success("I know you started liking loguru")

Adding Context to Log Messages

Suppose you want to add some additional information to the log message for context, you can use the bind() method.

You can add directive extra to the format in your logger add() method to add custom entries to the logs output.

import sys
from loguru import logger

# Remove the default logger
logger.remove(0)

# Add a new logger that outputs to sys.stderr
logger.add(
    sys.stderr,
    format=" {level} | <level>{message}</level> | {extra} ",
)

# Create a new logger with some initial context
context_logger = logger.bind(author="Akhilesh", type="demo")

# Log an info message with the current context
context_logger.info("You can pass context with logs!")

You can further customize the context:

# Bind additional context to the logger and log a warning message
context_logger.bind(blog_type="Tutorial").warning(
    "You can use extra attributes to bind context!"
)
# Log a success message with additional context provided during formatting
context_logger.success(
    "Use kwargs to add context during formatting: {platform}", platform="Medium"
)

Using Context Managers

We can also use the Python context manager to modify a context-local state temporarily with contextualize():

import sys
from loguru import logger

logger.remove(0)

logger.add(
    sys.stderr,
    format=" {level} | <level>{message}</level> | {extra} ",
)

context_logger = logger.bind(blog_id=45)

def do_something():
    context_logger.debug("doing something")

with logger.contextualize(scope="From context manager"):
    do_something()

do_something()

Filtering Logs

Combine bind() and filter for fine-grained control over your logs.

from loguru import logger

logger.add("special.log", filter=lambda record: "special" in record["extra"])
logger.debug("This message is not logged to the file")
logger.bind(special=True).info("This message, though, is logged to the file!")

Dynamic Values with patch()

The patch() method allows you to attach dynamic values to be attached to each new message:

import sys
from loguru import logger
from datetime import datetime

logger.remove(0)
logger.add(sys.stderr, format="{extra[utc]} - {level}- {message}")
logger = logger.patch(lambda record: record["extra"].update(utc=datetime.now()))

logger.info("using patch method from loguru")

Create Custom Log Levels

Loguru allows you to create your own log level with level() function.

import sys
from loguru import logger

logger.remove(0)

m_level = logger.level("Medium", no=45, color="<yellow><bold>", icon="/\\/\\")
n_level = logger.level("Nedium", no=45, color="<blue><bold>", icon="|\\|")

logger.add(sys.stderr, format=" <level> {level.icon} :: {message} </level>")

logger.log("Medium", "This is my custom log level")
logger.log("Nedium", "I like having optional log levels")

Logging Exceptions

Logging exceptions is crucial for tracking bugs, but it's not helpful if you don't know the cause. Loguru makes it easier by showing the entire stack trace, including variable values so you can identify the problem.

from loguru import logger

logger.remove(0)

# Caution, "diagnose=True" is the default and may leak sensitive data in prod
logger.add("loguru.log", backtrace=True, diagnose=True)

def func(a, b):
    return a / b

def nested(c):
    try:
        func(5, c)
    except ZeroDivisionError:
        logger.exception("Did you just??")

nested(0)

You can also use it with context manager by using logger.catch() decorator.

from loguru import logger

logger.remove(0)

# Caution, "diagnose=True" is the default and may leak sensitive data in prod
logger.add("loguru.log", backtrace=False, diagnose=False)

@logger.catch()
def func(a, b):
    return a / b

func(5,0)

By default the level you will see ERROR level, but you can customize it to use different levels.

Final Words

If you got to this point, you know how nice loguru is. If you love customization and making things pretty, you will love it.

Let me know what you think about it in the comments. If you found it useful, don't forget to like and follow for more Python tutorials!

Connect with the author on LinkedIn

Step-by-Step Guide to Setting OIDC With Terraform for GitHub Actions Workflows with AWS

Akhilesh Mishra — Sun, 22 Jun 2025 10:47:22 +0000

Your GitHub Actions Secrets Are the Weakest Link in Your AWS Security Chain

Are you still using AWS access keys and secrets to authenticate your GitHub Actions with AWS in 2025?

Please don't. Unless you want to wake up someday with 1000 GPU machines mining Bitcoin in your account at your expense, footing a million-dollar bill.

Using long-term secrets can be a security nightmare that could expose your cloud account to a possible security incident.

You might be just one exposed GitHub secret away from an AWS billing catastrophe.

Then what should you do?

You should use OpenID Connect (OIDC), a modern, more secure way to authenticate your GitHub Actions workflows with AWS without storing long-lived credentials.

You can configure the OIDC to work with certain GitHub org and GitHub repos, or you can go more granular and allow it with certain branches, tag formats, etc.

What the heck is OIDC?

OIDC enables token-based authentication between GitHub Actions and AWS, eliminating the need to store long-lived access keys. Establishing a trusting relationship with temporary credentials significantly enhances security while simplifying the authentication process.

The better way to authenticate GitHub Actions with AWS is with OpenID Connect (OIDC).

Setting up OIDC for your AWS account is very simple,

Step 1: Create an OIDC Identity Provider in AWS
Step 2: Create an IAM Role for GitHub Actions
Step 3: Attach the Permissions your CICD workflow needs to the Role
Step 4: Update Your GitHub Actions Workflow to use the Identity provider ARN instead of access keys and secrets

You can configure it from the AWS console, with AWS CLI commands or use Terraform to configure it.

I will use AWS CLI to configure the OIDC first and later with Terraform(with more flexible options).

Real-time implementation

Scenario

I have a 3-tier application running on an EKS cluster.
I want to build and deploy new versions of applications using the GitHub Action workflow. For simplicity, I will run kubectl commands from the workflow to deploy the new version of the application.

Workflow

Look for the change in frontend/backend code and only build the images upon the code change and push them to their respective ECR repos.

Configure the kubectl and authenticate to the EKS cluster
Use the newly built images (for backend/frontend) and deploy them to the frontend/backend service.

Setting Up OIDC Between GitHub Actions and AWS: A Step-by-Step Guide

Create the file structure below

Run the below command to create the directory structure
touch configure-oidc-github.sh eks-policy.json trust-policy.json
Copy the code to their respective files.
trust-policy.json

{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/$OIDC_PROVIDER"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringLike": {
            "$OIDC_PROVIDER:sub": "repo:$GITHUB_ORG/$GITHUB_REPO:*"
          }
        }
      }
    ]
  }

eks-policy.json

{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "eks:DescribeCluster",
          "eks:ListClusters"
        ],
        "Resource": "*"
      },
      {
        "Effect": "Allow",
        "Action": [
          "ecr:GetAuthorizationToken",
          "ecr:BatchCheckLayerAvailability",
          "ecr:GetDownloadUrlForLayer",
          "ecr:BatchGetImage",
          "ecr:InitiateLayerUpload",
          "ecr:UploadLayerPart",
          "ecr:CompleteLayerUpload",
          "ecr:PutImage"
        ],
        "Resource": "*"
      }
    ]
  }

configure-oidc-github.sh

#!/bin/bash
set -e

export OIDC_PROVIDER="token.actions.githubusercontent.com"
export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text)
export GITHUB_ORG="akhileshmishrabiz" # GitHub Org/Owner
export GITHUB_REPO="DevOpsDojo"  # Repo you want to allow to use OIDC with AWS

# Create the OIDC provider
aws iam create-open-id-connect-provider \
  --url https://$OIDC_PROVIDER \
  --client-id-list sts.amazonaws.com \
  --thumbprint-list "6938fd4d98bab03faadb97b34396831e3780aea1"

aws iam create-role \
  --role-name GitHubActionsEKSDeployRole \
  --assume-role-policy-document file://trust-policy.json

aws iam create-policy \
  --policy-name GitHubActionsEKSPolicy \
  --policy-document file://eks-policy.json

# Attach the policy to the role
aws iam attach-role-policy \
  --role-name GitHubActionsEKSDeployRole \
  --policy-arn arn:aws:iam::$AWS_ACCOUNT_ID:policy/GitHubActionsEKSPolicy

This script will configure the OIDC to allow my GitHub workflows in the repo https://github.com/akhileshmishrabiz/DevOpsDojo to use OIDC to authenticate with AWS.

Let me explain the steps.

Setting the environment variable

export OIDC_PROVIDER="token.actions.githubusercontent.com"
export AWS_ACCOUNT_ID=$(aws sts get-caller-identity - query "Account" - output text)
export GITHUB_ORG="akhileshmishrabiz"
export GITHUB_REPO="DevOpsDojo"

Creating the OIDC IAM provider

aws iam create-open-id-connect-provider \
--url https://$OIDC_PROVIDER \
--client-id-list sts.amazonaws.com \
--thumbprint-list "6938fd4d98bab03faadb97b34396831e3780aea1"

Creating an IAM Role for GitHub Actions

aws iam create-role \
 --role-name GitHubActionsEKSDeployRole \
--assume-role-policy-document file://trust-policy.json

Creating an IAM policy for the GitHub Action role

aws iam create-policy \
--policy-name GitHubActionsEKSPolicy \
--policy-document file://eks-policy.json

Attaching IAM policy with the role

aws iam attach-role-policy \
--role-name GitHubActionsEKSDeployRole \
--policy-arn arn:aws:iam::$AWS_ACCOUNT_ID:policy/GitHubActionsEKSPolicy

Configuring the OIDC provider for GitHub Action with AWS CLI.

Install the AWS CLI if you haven't already done

curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
sudo installer -pkg AWSCLIV2.pkg -target /

Configure the AWS credentials - with Access key and security key

aws configure# This will as you access and secret key
Run the script to

chmod u+ x configure-oidc-github.sh
./configure-oidc-github.sh

Note: You can find the entire code for OIDC with AWS CLI here.

This will allow us to use the role arn:aws:iam::366140438193:role/GitHubActionsEKSDeployRole
to authenticate with the GitHub repo https://github.com/akhileshmishrabiz/DevOpsDojo

Terraform implementation

In a production environment, we mostly use Terraform to deploy the OIDC provider for GitHub Action.

Let me show how you can set up the OIDC provider with the help of Terraform and how to allow more than one repository.

Create aproviders.tf file and paste the below code into it. This will configure the Terraform version and AWS provider.
Create the files,variables.tf, output.tf, main.tf, iam.tfand paste the below code into them.
Create a terraform.tfvars file to create a map of repositories and branches.
Deploy the Terraform code, it will return the Role ARN that we can use with GitHub Action to authenticate with AWS.
This will create an IAM role to use with GitHub Action, IAM policies with permissions you want your GitHub Action workflow to have, and attach them to the role.

IAM roles's trust policy will allow the particular repositories and branches to use this role to authenticate with AWS.

providers.tf

terraform {
  required_version = "1.8.1"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "ap-south-1"
}

variables.tf

# Variables
variable "aws_region" {
  description = "AWS region"
  type        = string
  default     = "ap-south-1"
}

variable "github_repositories" {
  description = "List of GitHub repositories to grant access to"
  type = list(object({
    org    = string
    repo   = string
    branch = optional(string, "*")
  }))
  default = [
    {
      org    = "akhileshmishrabiz"
      repo   = "DevOpsDojo"
      branch = "*"
    }
  ]
}

**main.tf **

# OIDC Provider
resource "aws_iam_openid_connect_provider" "github_actions" {
  url = "https://token.actions.githubusercontent.com"

  client_id_list = [
    "sts.amazonaws.com"
  ]

  thumbprint_list = [
    "6938fd4d98bab03faadb97b34396831e3780aea1"
  ]

  tags = {
    Name = "GitHub-Actions-OIDC-Provider"
  }
}

# IAM Role
resource "aws_iam_role" "github_actions_eks_deploy_role" {
  name = "GitHubActionsEKSDeployRole"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Federated = aws_iam_openid_connect_provider.github_actions.arn
        }
        Action = "sts:AssumeRoleWithWebIdentity"
        Condition = {
          StringLike = {
            "token.actions.githubusercontent.com:sub" = [
              for repo in var.github_repositories : 
                "repo:${repo.org}/${repo.repo}:${repo.branch}"
            ]
          }
        }
      }
    ]
  })

  tags = {
    Name = "GitHub-Actions-EKS-Deploy-Role"
  }
}

iam.tf

# IAM Policy
resource "aws_iam_policy" "github_actions_eks_policy" {
  name        = "GitHubActionsEKSPolicy"
  description = "Policy for GitHub Actions to access EKS and ECR"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "eks:DescribeCluster",
          "eks:ListClusters"
        ]
        Resource = "*"
      },
      {
        Effect = "Allow"
        Action = [
          "ecr:GetAuthorizationToken",
          "ecr:BatchCheckLayerAvailability",
          "ecr:GetDownloadUrlForLayer",
          "ecr:BatchGetImage",
          "ecr:InitiateLayerUpload",
          "ecr:UploadLayerPart",
          "ecr:CompleteLayerUpload",
          "ecr:PutImage"
        ]
        Resource = "*"
      }
    ]
  })

  tags = {
    Name = "GitHub-Actions-EKS-Policy"
  }
}

# Policy Attachment
resource "aws_iam_role_policy_attachment" "github_actions_eks_policy_attachment" {
  role       = aws_iam_role.github_actions_eks_deploy_role.name
  policy_arn = aws_iam_policy.github_actions_eks_policy.arn
}

terraform.tfvars

github_repositories = [
  {
    org    = "akhileshmishrabiz"
    repo   = "DevOpsDojo"
    branch = "*"  # All branches
  },
  {
    org    = "akhileshmishrabiz"
    repo   = "Devops-zero-to-hero" 
    branch = "main"  # Only main branch
  }
]

You can find the Terraform code to set up OIDC for Github Action here.

Apply the Terraform

This example uses the Terraform version 1.8.1 , make sure you have the same Terraform version installed.

If you have some other version of Terraform installed then update the providers.tf

If you want to use multiple versions of Terraform then use tfenv CLI.

terraform init
terraform apply

This will allow us to use the role arn:aws:iam::366140438193:role/GitHubActionsEKSDeployRole
to authenticate with the GitHub repo https://github.com/akhileshmishrabiz/DevOpsDojo on all branches.
And https://github.com/akhileshmishrabiz/Devops-zero-to-hero on the main branch.

Let's use this OIDC provider in our GitHub Action workflow

Using the OIDC provider to authenticate GitHub Action workflow with AWS to push the docker images to ECR repositories.

For this example, I will use my public GitHub Repo https://github.com/akhileshmishrabiz/DevOpsDojo
This repo has a 3-tier app:

Flask backend
React frontend
RDS Postgres database I will use GitHub Action to build the backend and frontend images and push them to AWS ECR repositories. GitHub Action will use OIDC to authenticate with AWS.

Let me create 2 ECR repositories

Create ECR repositories for frontend and backend

aws ecr create-repository --repository-name devopsdozo/frontend
aws ecr create-repository --repository-name devopsdozo/backend

I want to use the workflow to build the backend and frontend docker images and push them to ECR repositories.

If you look at the below GitHub Action workflow, you can see that I have used GitHubActionsEKSDeployRole instead of access and secret keys.
role-to-assume:arn:aws:iam::366140438193:role/GitHubActionsEKSDeployRole
instead of AWS access and security keys.

When GitHub sends an auth request to AWS, it verifies the trust policy attached to the role to check if it is allowed to authenticate the workflow for the specific GitHub repo, and branch.

If it finds the conditions to be true, it will generate a short-term token that GH workflow uses to authenticate with AWS.

name: Build and Push Docker Images

on:
  push:
    branches: [ main ]
  workflow_dispatch:  # Enable manual triggering

env:
  AWS_REGION: ap-south-1
  ECR_REPOSITORY_FRONTEND: devopsdozo/frontend
  ECR_REPOSITORY_BACKEND: devopsdozo/backend
  IMAGE_TAG: latest

jobs:
  build-and-push:
    name: Build and Push Images
    runs-on: ubuntu-latest

    permissions:
      id-token: write
      contents: read

    steps:
      - name: Checkout repository
        uses: actions/checkout@v3

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v3
        with:
          aws-region: ${{ env.AWS_REGION }}
          role-to-assume: arn:aws:iam::366140438193:role/GitHubActionsEKSDeployRole

      - name: Login to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v1

      - name: Build, tag, and push frontend image to Amazon ECR
        working-directory: ./frontend
        env:
          ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
        run: |
          docker build -t $ECR_REGISTRY/$ECR_REPOSITORY_FRONTEND:$IMAGE_TAG \
                     --platform=linux/amd64 .
          docker push $ECR_REGISTRY/$ECR_REPOSITORY_FRONTEND:$IMAGE_TAG
          echo "Frontend image pushed to ECR: $ECR_REGISTRY/$ECR_REPOSITORY_FRONTEND:$IMAGE_TAG"

      - name: Build, tag, and push backend image to Amazon ECR
        working-directory: ./backend
        env:
          ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
        run: |
          docker build -t $ECR_REGISTRY/$ECR_REPOSITORY_BACKEND:$IMAGE_TAG \
                     --platform=linux/amd64 .
          docker push $ECR_REGISTRY/$ECR_REPOSITORY_BACKEND:$IMAGE_TAG
          echo "Backend image pushed to ECR: $ECR_REGISTRY/$ECR_REPOSITORY_BACKEND:$IMAGE_TAG"

      - name: Summary
        run: |
          echo "### Docker Images Built and Pushed" >> $GITHUB_STEP_SUMMARY
          echo "✅ Frontend: ${{ steps.login-ecr.outputs.registry }}/${{ env.ECR_REPOSITORY_FRONTEND }}:${{ env.IMAGE_TAG }}" >> $GITHUB_STEP_SUMMARY
          echo "✅ Backend: ${{ steps.login-ecr.outputs.registry }}/${{ env.ECR_REPOSITORY_BACKEND }}:${{ env.IMAGE_TAG }}" >> $GITHUB_STEP_SUMMARY

Note: You can find a better version of this workflow here.

You can see from the screenshot above that GH workflows are authenticated with AWS using the OIDC, build the Docker images, and push to ECR.

This is all for this blog post.
Let's connect

Connect with me on LinkedIn ** **- Connect with me on Twitter (@livingdevops)

Stop Writing Ugly Docker Build Commands Today - Do This Instead

Akhilesh Mishra — Tue, 17 Jun 2025 13:54:53 +0000

Why Docker Bake is the Smarter Way to Build and Manage Docker Images

Do you enjoy remembering the docker build commands with all the different flags, and arguments?

Well, I never did. I had to google the flag and arguments whenever I needed to build images.

It gets worse when I have to build multiple docker images for mono repositories with different platforms.

Then, you will be using way more flags. That means way more googling or prompting AI bots.

What if I tell you a better way that allows you to write docker build as code and build everything with just one short command?

Well, you are in luck as good people at Docker finally released their long-awaited, build orchestration tool, Docker Bake in General Availability.

It is available with the Docker Desktop version 4.38.

Introducing Docker bake

Docker Bake allows you to define build stages and deployment environments in a declarative file, making complex builds easier to manage. It also leverages BuildKit's parallelization and optimization features to speed up build times.

Let me show you why I love Docker Bake

Let's take the example of a mono repo with a dummy frontend and backend application code with its Dockerfile.

If I had to build the docker images for both of these applications and push these two, I would have to run these commands run 2 commands

# Frontend build
docker build --build-arg NODE_VERSION=20 -t \
366140438193.dkr.ecr.ap-south-1.amazonaws.com/frontend:latest \
-f frontend/frontend.Dockerfile frontend

# Backend Build
docker build --build-arg GO_VERSION=1.21 -t \
366140438193.dkr.ecr.ap-south-1.amazonaws.com/backend:latest \
-f backend/backend.Dockerfile backend

And if you want to push these images to ECR repos, you will be running

docker push 366140438193.dkr.ecr.ap-south-1.amazonaws.com/frontend:latest
docker push 366140438193.dkr.ecr.ap-south-1.amazonaws.com/backend:latest

I used multiple flags to build the app, and 4 commands to push the image and you don't have these commands version-controlled as these are just commands.

Now what if I can tell you that you don't need to remember these flags and store these commands just like Terraform infra templates?

Yes, you can do it all with just one command - Docker buildx bake
Let me show you how Docker Bake makes all this a piece of cake.

Create a file docker-bake.hcl into the root path and paste the below into that file.

# docker-bake.hcl
group "default" {
  targets = ["frontend", "backend"]
}

target "frontend" {
  context = "./frontend"
  dockerfile = "frontend.Dockerfile"
  args = {
    NODE_VERSION = "20"
  }
  tags = ["366140438193.dkr.ecr.ap-south-1.amazonaws.com/frontend:latest"]
}

target "backend" {
  context = "./backend"
  dockerfile = "backend.Dockerfile"
  args = {
    GO_VERSION = "1.21"
  }
  tags = ["366140438193.dkr.ecr.ap-south-1.amazonaws.com/backend:latest"]
}

Your project will look like this.

Now just run one command

docker buildx bake

Checking if the new Docker images exist by running the docker images command.

You can see that the docker build bake command built both images using the HCL config file.

You can push the docker-bake.hcl to your GitHub repo, no one ever needs to use remember docker build commands or its flags.

You can also push the images to remote ECR repositories with --push argument

docker buildx bake --push

Digging deep into Docker bake

If you have a docker command like below:

docker build \
  -f Dockerfile \
  -t myapp:latest \
  --build-arg foo=bar \
  --no-cache \
  --platform linux/amd64,linux/arm64 \
  .

It's equivalent Docker Bake file will be like this:

# docker-bake.hcl
target "myapp" {
  context = "."
  dockerfile = "Dockerfile"
  tags = ["myapp:latest"]
  args = {
    foo = "bar"
  }
  no-cache = true
  platforms = ["linux/amd64", "linux/arm64"]
}

A target in a Bake file represents a build invocation.

It holds all the information you would normally pass on to a docker build command using flags.

To build a target with Bake, pass the name of the target to the bake command -
docker buildx bake webapp
If you don't define any target, the bake command will build the default target.

target "default" {
  dockerfile = "webapp.Dockerfile"
  tags = ["docker.io/username/webapp:latest"]
  context = "https://github.com/username/webapp"
}

You can run
docker buildx bake
and it will build the image

Let's consider this bake config

group "all" {
  targets = ["webapp", "api", "tests"]
}

target "webapp" {
  dockerfile = "webapp.Dockerfile"
  tags = ["docker.io/username/webapp:latest"]
  context = "https://github.com/username/webapp"
}

target "api" {
  dockerfile = "api.Dockerfile"
  tags = ["docker.io/username/api:latest"]
  context = "https://github.com/username/api"
}

target "tests" {
  dockerfile = "tests.Dockerfile"
  contexts = {
    webapp = "target:webapp",
    api = "target:api",
  }
  output = ["type=local,dest=build/tests"]
  context = "."
}

To build multiple targets at once, run

docker buildx bake webapp api tests
You can group targets using the group block

docker buildx bake all

If you noticed in the example I talked about at the start, I used a group block with a default target and I used docker build bake to build both images.


group "default" {
  targets = ["frontend", "backend"]
}

Inheritance in Bake

Inheritance allows you to define common configurations and reuse them across multiple targets. Consider the below config

target "common" {
    context = "."
    platforms = ["linux/amd64", "linux/arm64"]
}

target "backend" {
    inherits = ["common"]
    dockerfile = "backend.Dockerfile"
    args = {
        GO_VERSION = "1.21"
    }
}

target "frontend" {
    inherits = ["common"]
    dockerfile = "frontend.Dockerfile"
    args = {
        NODE_VERSION = "20"
    }
}

Common target sets context and multi-platform builds.

backend and frontend inherit from common, ensuring they both build for multiple platforms.

Overriding Values in Inheritance

When a target inherits another target, it can override any of the inherited attributes.

For example, the following target overrides the args attribute from the inherited target:

target "base" {
    context = "."
    dockerfile = "Dockerfile"
    args = {
        APP_ENV = "development"
    }
}

target "production" {
    inherits = ["base"]
    args = {
        APP_ENV = "production"
    }
}

Using Variables in Bake - Just like Terraform

You can define and use variables in a Bake file to set attribute values, interpolate them into other values, and perform arithmetic operations.

Consider the below config

group "default" {
  targets = [ "frontend" ]
}

variable "NODE_VERSION" {
    default = "20"
}

variable "tag" {
    default = "latest"
}

target "frontend" {
    context = "."
    dockerfile = "frontend.Dockerfile"
    args = {
        NODE_VERSION = NODE_VERSION
    }
    tags = ["myapp-frontend:${tag}"]
}

Printing the Bake file with the --print flag shows the interpolated value in the resolved build configuration.

docker buildx bake --print

You can do a lot of things like Validating multiple conditions, Escaping variable interpolation

Airthematic expression and Ternary conditional with Docker bake

variable "FOO" {
  default = 3
}

variable "IS_FOO" {
  default = true
}

target "app" {
  args = {
    v1 = FOO > 5 ? "higher" : "lower"
    v2 = IS_FOO ? "yes" : "no"
  }
}

Using built-in and user-defined functions with Docker bake

We can use a user-defined function to generate a tag with a timestamp(with a built-in function)

// User-defined function to generate a tag with a timestamp
function "generate_tag" {
    params = [version]
    result = format("%s-%s", version, replace(timestamp(), "[-:TZ]", "")) // Correct concatenation
}

// Define a variable for version
variable "APP_VERSION" {
    default = "1.0.0"
}

// Define a target using the custom function and built-in function
target "myapp" {
    context = "."
    dockerfile = "Dockerfile"
    tags = ["myapp:${generate_tag(APP_VERSION)}"]
    args = {
        BUILD_DATE = timestamp()
    }
}

Remote Bake file definition

You can build Bake files directly from a remote Git repository or an HTTPS URL

Bake file reference

You can write Bake files in HCL, YAML (Docker Compose files), or JSON.

I used docker-bake.hcl as a file name but you can use any filename you want and pass that file as an argument to the bake command.

docker buildx bake --file ../docker/bake.hcl
By default, Bake uses the following lookup order to find the configuration file.

Docker bake is a powerful tool, there are so many things you can do with it. It offers great flexibility with options like using a matrix, docker-compose like config, overriding configuration, cache, and so many things.

Go check it out and try to use it in your workflow. I am sure you will love it.

Thanks for reading, do share your thoughts in the comments. Would love to hear any feedback/suggestions

Building a production Lambda function that monitors IAM access keys and sends automated email alerts using boto3 and AWS SES.

Akhilesh Mishra — Tue, 17 Jun 2025 13:12:00 +0000

Scenario

In my project, we have over 10+ users with access keys. Most of the access keys were 300–400 days old.

Having access keys lying around for a long period poses a security risk, which is unacceptable. They’re not just a minor security issue — they’re multimillion-dollar liabilities.

The issue I can see is that no one remembers to rotate the access keys, so I thought of building a Lambda function that could remind the team to rotate the keys after a certain number of days.

Different organizations enforce varying rotation policies — some require rotation every 30 days, others allow 60 or 90 days before access keys must be refreshed.

My solution needed to accommodate these different requirements while being easy to deploy and maintain.

What to expect in this blog post

In this blog post, I’ll walk you through the complete implementation of an automated access key rotation reminder system. You’ll learn how to build a Lambda function that monitors key ages, sends targeted notifications, and helps your team maintain robust security hygiene without the overhead of manual tracking.

✅ You can find the entire code for this blog post on my public GitHub repo.

Writing the Python code

I used AWS Python SDK, boto3, for the automation. Here is the approach I took.

List the users and access keys
Calculate the age
Compare it with age standards, and determine if this key should be rotated.
If the key should be rotated, build the email body that includes the link to the user
Send the email using AWS SES(Simple Email Service)
Building the code
Before we deploy a fully functioning Lambda, I will build and test the code locally.

Note: Also, configure the AWS credentials -> aws configure locally

I will use boto3, datetime, and email Python modules. datetime and email are Python built-in modules, but we need to install boto3, which is the AWS-managed library.

Install the boto3 I will use a Python virtual environment and install boto3 with that

python3 -m venv .venv

# activate the virtual environemnts on my mac. There is a different commnad
#  windows. 
source .venv/bin/activate

Install boto3 with pip

pip install boto3

Now that we have the installation done, let's write the code step by step.

1. List the users

import boto3
def get_users():
    iam_client = boto3.client('iam')
    response = iam_client.list_users()
    return [user['UserName'] for user in response['Users']]

print(get_users())

2. Get the access keys details

I will use the datetime module to calculate the age from the created date parameter of the access key

def get_access_keys_age(username):
    iam_client = boto3.client('iam')
    response = iam_client.list_access_keys(UserName=username).get('AccessKeyMetadata', [])

    access_keys_info = []
    for item in response:
        if item['Status'] == 'Active':
            access_key_id = item['AccessKeyId']
            create_date = item['CreateDate'].date()
            age = (date.today() - create_date).days
            access_keys_info.append((access_key_id, age))

    return access_keys_info

print(get_access_keys_age('cliuser-akhilesh'))

As you can see, the user cliuser-akhilesh has 2 access keys, one with 22 days' age and the other with 2 days.

For this use case, I will set the access key expiry age as 20 days, and I would want to send email from 15 days (I would want to have 5 days as a buffer to ensure people responsible for rotating email get enough time to address this)

3.Check if the expired key

I want a function that returns an HTML message if the keys are expired. I would include a dynamic link of the AWS IAM user for which the keys have expired.

username = "cliuser-akhilesh" 
Expiry_days = 20
reminder_email_age = Expiry_days - 5

def if_key_expired(access_key_id, age, reminder_email_age):
    if age >= reminder_email_age:
        return f'''
    <p>Reminder: Access key <strong>{access_key_id}</strong> is <strong>{age}</strong> days old. Please rotate it.</p>
    <p>For more details, visit the <a href="https://us-east-1.console.aws.amazon.com/iam/home?region=us-east-1#/users/details/{username}?section=security_credentials"> Rotate this key here</a>.</p>
    '''
    return None

print(if_key_expired("AKIA4ZPZU3T7QIPRKR5X", 22, reminder_email_age))

That HTML will look like this

4.Process users

This will return the email body, only for users with access keys about to expire. We will only build an email for these users/access_keys and send an email

def process_users():
    email_body_list = []
    users = get_users()
    for user in users:
        access_keys_info = get_access_keys_age(user)
        for keys in access_keys_info:
            access_key_id, age = keys   
            email_body = if_key_expired(access_key_id, age, reminder_email_age)
            if email_body:
                email_body_list.append(email_body)       
    return email_body_list

print(type(process_users()))
print(process_users())

5. Build an email with the Python email library

from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText

def build_email_message(to_email, from_email, subject, body):
    msg = MIMEMultipart()
    msg['From'] = from_email
    msg['To'] = to_email
    msg['Subject'] = subject

    body_part = MIMEText(body, 'html')
    msg.attach(body_part)

    return msg

6. Send an email using the AWS SES service

# Create ses clinet
ses_client = boto3.client('ses')

def send_email(msg, to_emails):
    response = ses_client.send_raw_email(
        Source=msg["From"],
        Destinations=to_emails,
        RawMessage={"Data": msg.as_string()},
    )
    return response.get('MessageId', None)

7. Now let’s put it all together

This function will find all the users with expiring access keys and send an email. I used my emails for this demo; in real scenarios, you can send an email to the whole team.

def main():
    subject = f"AWS Access Key Rotation Reminder -user {username}"
    to_email = "akhileshmishratoemail@gmail.com"
    from_email = "akhileshmishrafromemail@gmail.com"
    for email_body in process_users():
        email_msg = build_email_message(to_email, from_email, subject, email_body)
        email_sent =send_email(email_msg, [to_email])
        print(f"Email sent with Message ID: {email_sent}")

I will go to Lambda and test it. I should get an email.
And here it is

Now that we have tested the code on the local machine, we are ready to deploy it on Lambda.

Only one part will change on Lambda, the main function will look something like this.

def main(event, context):
    subject = f"AWS Access Key Rotation Reminder -user {username}"
    to_email = "aditiyamishranit@gmail.com"
    from_email = "akhileshmishra121990@gmail.com"
    for email_body in process_users():
        email_msg = build_email_message(to_email, from_email, subject, email_body)
        email_sent =send_email(email_msg, [to_email])
        print(f"Email sent with Message ID: {email_sent}")

The main function will be the entry function for lambda.

AWS SES

You need to validate the identities in AWS before you can send an email to them. Since you will be using the SES sandbox account, you need to validate to_email and from_email.

Go to Amazon SES > Configuration: Identities

Create and validate identities

Terraform implementation

I will follow the steps below to write a Terraform function to deploy the Lambda function.

Lambda will expect zipped code, so I will be archiving the code in a zip format using the Terraform archive_file data source
Lambda will need access to AWS IAM to get the access key details, so I will be creating an IAM role with an IAM policy with access to list users, get access key data, and send email
Create the Lambda function
Create a cron job that will run this Lambda daily

- Archiving the code

# zip the code
data "archive_file" "lambda_zip" {
  type        = "zip"
  source_dir  = "${path.module}/lambda/iam-key-rotation"
  output_path = "${path.module}/iam-key-rotation.zip"
}

path.module reference to the path of the Terraform config. We use it to write the relative path for the code files.

- IAM role for the Lambda

# iam role
resource "aws_iam_role" "lambda_role" {
  name = "iam-key-rotation-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "lambda.amazonaws.com"
        }
      }
    ]
  })    

}

- IAM policy for the Lambda

resource "aws_iam_policy" "lambda_policy" {
  name        = "iam-key-rotation-policy"
  description = "Policy for Lambda function to rotate IAM keys"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "iam:ListAccessKeys",
          "iam:ListUsers",
        ]
        Effect   = "Allow"
        Resource = "*"
      },
      {
        Action = [
          "ses:SendEmail",
          "ses:SendRawEmail",
        ]
        Effect   = "Allow"
        Resource = "*"
      }
    ]
  })
}

Attach the policy to the role

#### # attach policy to role
resource "aws_iam_role_policy_attachment" "lambda_policy_attachment" {
  role       = aws_iam_role.lambda_role.name
  policy_arn = aws_iam_policy.lambda_policy.arn
}

- Lambda function

# lambda 
resource "aws_lambda_function" "my_lambda_function" {
    function_name    = "iam-key-rotation"
    role             = aws_iam_role.lambda_role.arn
    handler          = "main.main"
    runtime          = "python3.13"
    timeout          = 60
    memory_size      = 128

    # Use the Archive data source to zip the code
    filename         = data.archive_file.lambda_zip.output_path
    source_code_hash = data.archive_file.lambda_zip.output_base64sha256
}

source_code_hash will enable Lambda to update the Lambda code whenever a change happens in the Python code.
timeout is set to 60 seconds, if not set, will fall back to the default 3 seconds
handler Use the format python_code.python_function. In this use case, main.py is the code that runs when Lambda is invoked, and main() is the entry-level function. Hence, handler = main.main
To enable the cron job trigger, we use an AWS eventbridge rule.

resource "aws_cloudwatch_event_rule" "cron_lambdas" {
  name                = "cronjob"
  description         = "to triggr lambda daily 7.15 pm ist"
  schedule_expression = "cron(40 13 * * ? *)"
}
resource "aws_cloudwatch_event_target" "cron_lambdas" {
  rule = aws_cloudwatch_event_rule.cron_lambdas.name
  arn  = aws_lambda_function.my_lambda_function.arn
}

Also, this cron will need permissions to invoke the lambda on schedule

# Invoke lambda permission
resource "aws_lambda_permission" "cron_lambdas" {
  statement_id  = "key-rotation-lambda-allow"
  action        = "lambda:InvokeFunction"
  principal     = "events.amazonaws.com"
  function_name = aws_lambda_function.my_lambda_function.arn
  source_arn    = aws_cloudwatch_event_rule.cron_lambdas.arn
}

Lambda code is set. I will use the Terraform provider AWS and a remote state file

*providers.tf
*

terraform {
  required_version = "1.8.1"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.32.0"
    }
  }
}

provider "aws" {
  region = "ap-south-1"
}

# remote backend
terraform {
  backend "s3" {
    bucket         = "state-bucket-879381234673"
    key            = "lambda-blog/terraform.tfstate"
    region         = "ap-south-1"
    encrypt        = true
  }
}

Now we can deploy the lambda function

terraform init
terraform plan
terraform apply

Better version of Python and Lambda code

To ensure the automation is flexible and maintainable, I’ll implement a configuration-driven approach using environment variables.

This eliminates hardcoded values and allows dynamic configuration management through Terraform, making the solution easily adaptable across different environments and organizations.

Here is the final version of the code.

main.py

import boto3
from datetime import date
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
import os

from_email = os.environ.get('FROM_EMAIL')
to_email = os.environ.get('TO_EMAIL')
Expiry_days = int(os.environ.get('EXPIRY_DAYS', 90))  # Default to 90 days if not set
reminder_email_age = Expiry_days - 5

def get_users():
    iam_client = boto3.client('iam')
    response = iam_client.list_users()
    return [user['UserName'] for user in response['Users']]

def get_access_keys_age(username):
    iam_client = boto3.client('iam')
    response = iam_client.list_access_keys(UserName=username).get('AccessKeyMetadata', [])

    access_keys_info = []
    for item in response:
        if item['Status'] == 'Active':
            access_key_id = item['AccessKeyId']
            create_date = item['CreateDate'].date()
            age = (date.today() - create_date).days
            access_keys_info.append((access_key_id, age))

    return access_keys_info

def if_key_expired(username, access_key_id, age, reminder_email_age):
    if age >= reminder_email_age:
        return f'''
    <p>Reminder: Access key <strong>{access_key_id}</strong> is <strong>{age}</strong> days old. Please rotate it.</p>
    <p>For more details, visit the <a href="https://us-east-1.console.aws.amazon.com/iam/home?region=us-east-1#/users/details/{username}?section=security_credentials"> Rotate this key here</a>.</p>
    return None

def process_users():
    email_body_list = []
    users = get_users()
    for user in users:
        access_keys_info = get_access_keys_age(user)
        for keys in access_keys_info:
            access_key_id, age = keys   
            email_body = if_key_expired(user, access_key_id, age, reminder_email_age)
            if email_body:
                email_body_list.append(email_body)       
    return email_body_list

def build_email_message(to_email, from_email, subject, body):
    msg = MIMEMultipart()
    msg['From'] = from_email
    msg['To'] = to_email
    msg['Subject'] = subject

    body_part = MIMEText(body, 'html')
    msg.attach(body_part)
    return msg

def send_email(msg, to_emails):
    ses_client = boto3.client('ses')
    response = ses_client.send_raw_email(
        Source=msg["From"],
        Destinations=to_emails,
        RawMessage={"Data": msg.as_string()},
    )
    return response.get('MessageId', None)

def main(event, context):
    subject = f"AWS Access Key Rotation Reminder"
    for email_body in process_users():
        email_msg = build_email_message(to_email, from_email, subject, email_body)
        email_sent =send_email(email_msg, [to_email])
        print(f"Email sent with Message ID: {email_sent}")

And updated the Terraform resource for Lambda

# lambda 
resource "aws_lambda_function" "my_lambda_function" {
    function_name    = "iam-key-rotation"
    role             = aws_iam_role.lambda_role.arn
    handler          = "lambda_function.lambda_handler"
    runtime          = "python3.13"
    timeout          = 60
    memory_size      = 128

    # Use the Archive data source to zip the code
    filename         = data.archive_file.lambda_zip.output_path
    source_code_hash = data.archive_file.lambda_zip.output_base64sha256
    environment {
      variables = {
        "to_email" = "akhileshmishratoemail@gmail.com"
        "from_email" = "akhileshmishra1from@gmail.com"
        "Expiry_days" = 20
      }
    }

✅ You can find the entire code for this blog post on my public GitHub repo.

This blog is originally published on my personal blog, livingdevops

Let's connect

- Connect with me on Linkedin
**
**- Connect with me on twitter (@livingdevops)

Forem: Akhilesh Mishra

Every Kubernetes Concept Has a Story

To summarize the full story:

Deploy Secure Google Cloud VMs with Terraform and GitHub Actions in 2025

Automate VM provisioning, networking, and CI/CD pipelines using Infrastructure as Code best practices

What You'll Learn in This Guide

Prerequisites

Understanding Google Cloud Compute Components

What are Firewall Rules in GCP?

What is Cloud NAT?

What is Cloud Router?

Extending Your Terraform Configuration

Add VM Variables (variables.tf)

Update Variable Values (terraform.tfvars)

Create VM Resources (vm.tf)

Create Firewall Rules (firewall.tf)

Update Networking with Cloud NAT (networking.tf)

Update Outputs (outputs.tf)

Setting Up GitHub Actions Automation

Create GitHub Secrets

Create GitHub Actions Workflow

Deploying Your Infrastructure

Local Deployment First

Expected Output

Test VM Connectivity

Push to GitHub for Automated Deployment

Monitoring Your Deployment

Check GitHub Actions

Verify in GCP Console

Test Your VM

Production Considerations

Security Enhancements

Monitoring and Logging

Cost Optimization

Troubleshooting Common Issues

Authentication Failures

VM Won't Start

Network Connectivity Issues

What's Next in Part 3

Key Takeaways

Build Production-Ready Google Cloud Infrastructure with Terraform in 2025

Complete step-by-step guide to creating VPC networks, subnets, and storage buckets using Infrastructure as Code

What You'll Build in This Series

Understanding Google Cloud Networking Fundamentals

What is a VPC in Google Cloud?

What is a Subnet in Google Cloud?

Prerequisites: Setting Up Your Environment

Verify Your Installation

Google Cloud Authentication Setup

Creating a Service Account (Development Setup)

Set Environment Variables

Project Structure: Organizing Your Terraform Code

Terraform Configuration Files Explained

1. Provider Configuration (providers.tf)

2. Input Variables (variables.tf)

3. Networking Resources (networking.tf)

4. Storage Resources (storage.tf)

5. Outputs (outputs.tf)

6. Variable Values (terraform.tfvars)

7. Backend Configuration (backend.tf)

Understanding Terraform State Management

State Storage Options

Migrating to Remote State

Deploying Your Infrastructure

Step 1: Initialize Terraform

Step 2: Format and Validate

Step 3: Plan Your Deployment

Step 4: Apply Changes

Step 5: Verify Deployment

Step 6: Clean Up (When Needed)

Production-Ready Best Practices

🔧 Code Organization & Standards

🔒 Security & Access Management

📚 Documentation & Team Collaboration

🚀 Automation & CI/CD (Coming in Part 2)

💰 Cost Optimization

What's Coming Next in This Series

🔜 Part 2: Compute Engine VMs with GitHub Actions

🔜 Part 3: PostgreSQL with Workload Identity Federation

🔜 Part 4: Secret Management and Key Rotation

Add VM Variables (`variables.tf`)

Update Variable Values (`terraform.tfvars`)

Create VM Resources (`vm.tf`)

Create Firewall Rules (`firewall.tf`)

Update Networking with Cloud NAT (`networking.tf`)

Update Outputs (`outputs.tf`)

1. Provider Configuration (`providers.tf`)

2. Input Variables (`variables.tf`)

3. Networking Resources (`networking.tf`)

4. Storage Resources (`storage.tf`)

5. Outputs (`outputs.tf`)

6. Variable Values (`terraform.tfvars`)

7. Backend Configuration (`backend.tf`)