Forem: Sri

Building Secure Jenkins-Slack Integration with AWS Lambda - Part 2: Troubleshooting Real-World Issues

Sri — Tue, 14 Oct 2025 11:08:28 +0000

Welcome back! In Part 1, we built the foundation of our secure Jenkins-Slack integration. Now it's time to tackle the real-world challenges that make or break production deployments.

This is Part 2 of our series, where we'll dive deep into troubleshooting common issues, implementing critical fixes, and adding production-ready enhancements.

The Reality Check

After deploying the initial setup from Part 1, you'll likely encounter several roadblocks. Based on real-world experience, here are the most common issues and their battle-tested solutions:

🚨 Critical Issues and Solutions

Issue 1: Jenkins Authentication Failure

Problem: Jenkins login with admin/ not working

Symptoms:

Jenkins stuck in initial setup mode
"Invalid username or password" errors
Cannot access Jenkins dashboard

Root Cause: Jenkins was in initial setup wizard mode, not accepting default credentials

Solution: Complete Jenkins Reset with Programmatic Admin Creation

# Create the final-fix.sh script
cat > final-fix.sh << 'EOF'
#!/bin/bash
echo "🚀 Starting Jenkins reset and setup..."

# Stop Jenkins
sudo systemctl stop jenkins
echo "✅ Jenkins stopped"

# Clean existing user files
sudo rm -rf /var/lib/jenkins/users /var/lib/jenkins/init.groovy.d
echo "✅ Cleaned existing user files"

# Create proper admin user script
sudo mkdir -p /var/lib/jenkins/init.groovy.d
sudo tee /var/lib/jenkins/init.groovy.d/create-admin.groovy > /dev/null << 'GROOVY'
#!/usr/bin/env groovy
import jenkins.model.*
import hudson.security.*

def instance = Jenkins.getInstance()
def hudsonRealm = new HudsonPrivateSecurityRealm(false)
hudsonRealm.createAccount("admin", "your-secure-password")
instance.setSecurityRealm(hudsonRealm)

def strategy = new FullControlOnceLoggedInAuthorizationStrategy()
strategy.setDenyAnonymousReadAccess(false)
instance.setAuthorizationStrategy(strategy)

instance.setCrumbIssuer(null)  // Disable CSRF
instance.save()
println "Admin user created and CSRF disabled"
GROOVY

sudo chown jenkins:jenkins /var/lib/jenkins/init.groovy.d/create-admin.groovy
echo "✅ Created admin user script"

# Start Jenkins
sudo systemctl start jenkins
echo "✅ Jenkins started"

# Wait for initialization
echo "⏳ Waiting for Jenkins initialization..."
sleep 30

# Test authentication
echo "🧪 Testing authentication..."
curl -u admin:your-secure-password http://localhost:8080/api/json > /dev/null 2>&1
if [ $? -eq 0 ]; then
    echo "✅ Authentication successful!"
else
    echo "❌ Authentication failed. Check Jenkins logs."
fi

echo "🎉 Setup complete! Access Jenkins at http://localhost:8080 with admin/your-secure-password"
EOF

chmod +x final-fix.sh
./final-fix.sh

Key Files Created:

final-fix.sh - Complete Jenkins setup script
create-admin.groovy - Programmatic admin user creation
CSRF protection disabled for API access

Issue 2: CSRF Token Errors

Problem: "403 No valid crumb was included in the request" errors

Symptoms:

Lambda getting 401 Unauthorized for CSRF endpoints
Jenkins API calls failing with crumb errors
Authentication works but job triggers fail

Root Cause: Jenkins CSRF protection blocking API calls from Lambda

Solution: Disable CSRF Protection Programmatically

# Create CSRF disable script
sudo tee /var/lib/jenkins/init.groovy.d/disable-csrf.groovy > /dev/null << 'EOF'
#!/usr/bin/env groovy
import jenkins.model.*
def instance = Jenkins.getInstance()
instance.setCrumbIssuer(null)
instance.save()
println "CSRF protection disabled"
EOF

sudo chown jenkins:jenkins /var/lib/jenkins/init.groovy.d/disable-csrf.groovy
sudo systemctl restart jenkins

Why This Works: For API-only access scenarios, CSRF protection can be safely disabled since we're using proper authentication and the API is not exposed to browsers.

Issue 3: Slack Slash Command dispatch_unknown_error

Problem: dispatch_unknown_error when using /run_test

Symptoms:

Slack command returns error immediately
No response from API Gateway
Lambda function not being invoked

Root Cause: Slack app not properly installed or missing permissions

Solution: Proper Slack App Configuration

Verify App Installation:

Go to https://api.slack.com/apps
Select your app → "Install App"
Ensure it shows "Installed" with green checkmark
If not installed, click "Install to Workspace"

Check Request URL:

Go to "Slash Commands" → Edit /run_test
Verify URL is exactly: https://your-api-gateway-url/prod/trigger
No extra spaces or characters

Test with Webhook:

   # Use webhook.site for testing
   # 1. Go to https://webhook.site
   # 2. Copy unique URL
   # 3. Temporarily update Slack command to use webhook URL
   # 4. Test command - should see request in webhook.site

Reinstall App:
- Uninstall app from workspace
- Reinstall with proper permissions
- Grant "Send messages" permission

Issue 4: Lambda Function Evolution

Problem: Lambda function needs optimization for Slack integration

Evolution Path: Basic → CSRF-aware → Slack-optimized

Final Solution: Slack-Optimized Lambda Function

// main_slack_fixed.go - Final optimized version
package main

import (
    "context"
    "encoding/json"
    "fmt"
    "net/http"
    "net/url"
    "strings"

    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/ssm"
)

type SlackRequest struct {
    Text        string `json:"text"`
    UserName    string `json:"user_name"`
    ChannelName string `json:"channel_name"`
}

type JenkinsCredentials struct {
    Username string `json:"username"`
    Password string `json:"password"`
}

func handler(ctx context.Context, request events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
    // Parse Slack request (handles both JSON and form data)
    var slackReq SlackRequest

    if request.Headers["Content-Type"] == "application/json" {
        json.Unmarshal([]byte(request.Body), &slackReq)
    } else {
        // Handle form data
        values, _ := url.ParseQuery(request.Body)
        slackReq.Text = values.Get("text")
        slackReq.UserName = values.Get("user_name")
        slackReq.ChannelName = values.Get("channel_name")
    }

    // Parse parameters from text
    env := "dev"
    testSuite := "smoke"

    if slackReq.Text != "" {
        parts := strings.Fields(slackReq.Text)
        if len(parts) >= 1 {
            env = parts[0]
        }
        if len(parts) >= 2 {
            testSuite = parts[1]
        }
    }

    // Get Jenkins credentials from SSM
    sess := session.Must(session.NewSession())
    ssmClient := ssm.New(sess)

    jenkinsURLParam := "/jenkins-slack-demo/jenkins_url"
    jenkinsCredsParam := "/jenkins-slack-demo/jenkins_credentials"

    jenkinsURLResult, err := ssmClient.GetParameter(&ssm.GetParameterInput{
        Name: aws.String(jenkinsURLParam),
    })
    if err != nil {
        return createErrorResponse("Failed to get Jenkins URL"), nil
    }

    credsResult, err := ssmClient.GetParameter(&ssm.GetParameterInput{
        Name:            aws.String(jenkinsCredsParam),
        WithDecryption:  aws.Bool(true),
    })
    if err != nil {
        return createErrorResponse("Failed to get Jenkins credentials"), nil
    }

    var creds JenkinsCredentials
    json.Unmarshal([]byte(*credsResult.Parameter.Value), &creds)

    // Trigger Jenkins job
    jobURL := fmt.Sprintf("%s/job/run_test/buildWithParameters", *jenkinsURLResult.Parameter.Value)
    data := url.Values{}
    data.Set("ENVIRONMENT", env)
    data.Set("TEST_SUITE", testSuite)

    client := &http.Client{}
    req, err := http.NewRequest("POST", jobURL, strings.NewReader(data.Encode()))
    if err != nil {
        return createErrorResponse("Failed to create request"), nil
    }

    req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
    req.SetBasicAuth(creds.Username, creds.Password)

    resp, err := client.Do(req)
    if err != nil {
        return createErrorResponse("Failed to trigger Jenkins job"), nil
    }
    defer resp.Body.Close()

    // Create Slack-compatible response
    var status string
    var emoji string
    if resp.StatusCode == 201 || resp.StatusCode == 200 {
        status = "Successfully triggered"
        emoji = "🚀"
    } else {
        status = "Failed to trigger"
        emoji = "❌"
    }

    response := map[string]interface{}{
        "response_type": "in_channel",
        "text": fmt.Sprintf("%s Jenkins job triggered!\n• Environment: %s\n• Test Suite: %s\n• Status: %s",
            emoji, env, testSuite, status),
    }

    responseBody, _ := json.Marshal(response)
    return events.APIGatewayProxyResponse{
        StatusCode: 200,
        Headers: map[string]string{
            "Content-Type": "application/json",
        },
        Body: string(responseBody),
    }, nil
}

func createErrorResponse(message string) events.APIGatewayProxyResponse {
    response := map[string]interface{}{
        "response_type": "ephemeral",
        "text": "❌ " + message,
    }
    responseBody, _ := json.Marshal(response)
    return events.APIGatewayProxyResponse{
        StatusCode: 500,
        Headers: map[string]string{
            "Content-Type": "application/json",
        },
        Body: string(responseBody),
    }
}

func main() {
    lambda.Start(handler)
}

🛠️ Production Enhancements

Security Improvements

1. Slack Signature Verification

Add proper Slack request signature validation:

import (
    "crypto/hmac"
    "crypto/sha256"
    "encoding/hex"
)

func verifySlackSignature(signature, timestamp, body, signingSecret string) bool {
    if signature == "" {
        return false
    }

    // Remove "v0=" prefix
    signature = strings.TrimPrefix(signature, "v0=")

    // Create signature base string
    sigBasestring := "v0:" + timestamp + ":" + body

    // Compute HMAC
    mac := hmac.New(sha256.New, []byte(signingSecret))
    mac.Write([]byte(sigBasestring))
    expectedSignature := hex.EncodeToString(mac.Sum(nil))

    return hmac.Equal([]byte(signature), []byte(expectedSignature))
}

2. Enhanced IAM Roles

# terraform/iam.tf - Enhanced IAM configuration
resource "aws_iam_role" "lambda_execution_role" {
  name = "${var.project_name}-lambda-execution-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "lambda.amazonaws.com"
        }
      }
    ]
  })
}

resource "aws_iam_role_policy" "lambda_policy" {
  name = "${var.project_name}-lambda-policy"
  role = aws_iam_role.lambda_execution_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ]
        Resource = "arn:aws:logs:*:*:*"
      },
      {
        Effect = "Allow"
        Action = [
          "ssm:GetParameter",
          "ssm:GetParameters"
        ]
        Resource = [
          "arn:aws:ssm:${var.aws_region}:*:parameter/${var.project_name}/*"
        ]
      },
      {
        Effect = "Allow"
        Action = [
          "ec2:CreateNetworkInterface",
          "ec2:DescribeNetworkInterfaces",
          "ec2:DeleteNetworkInterface"
        ]
        Resource = "*"
      }
    ]
  })
}

Monitoring and Observability

1. CloudWatch Alarms

# terraform/monitoring.tf
resource "aws_cloudwatch_metric_alarm" "lambda_errors" {
  alarm_name          = "${var.project_name}-lambda-errors"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = "2"
  metric_name         = "Errors"
  namespace           = "AWS/Lambda"
  period              = "300"
  statistic           = "Sum"
  threshold           = "0"
  alarm_description   = "Lambda function errors"
  alarm_actions       = [aws_sns_topic.alerts.arn]

  dimensions = {
    FunctionName = aws_lambda_function.jenkins_trigger.function_name
  }
}

resource "aws_cloudwatch_metric_alarm" "jenkins_response_time" {
  alarm_name          = "${var.project_name}-jenkins-response-time"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = "2"
  metric_name         = "Duration"
  namespace           = "AWS/Lambda"
  period              = "300"
  statistic           = "Average"
  threshold           = "10000"  # 10 seconds
  alarm_description   = "Jenkins response time too high"
  alarm_actions       = [aws_sns_topic.alerts.arn]

  dimensions = {
    FunctionName = aws_lambda_function.jenkins_trigger.function_name
  }
}

2. Structured Logging

import (
    "log"
    "os"
)

type Logger struct {
    *log.Logger
}

func NewLogger() *Logger {
    return &Logger{
        Logger: log.New(os.Stdout, "", log.LstdFlags|log.Lshortfile),
    }
}

func (l *Logger) LogRequest(user, channel, text string) {
    l.Printf("REQUEST: user=%s channel=%s text=%s", user, channel, text)
}

func (l *Logger) LogJenkinsTrigger(env, testSuite string, statusCode int) {
    l.Printf("JENKINS_TRIGGER: env=%s testSuite=%s status=%d", env, testSuite, statusCode)
}

func (l *Logger) LogError(operation string, err error) {
    l.Printf("ERROR: operation=%s error=%v", operation, err)
}

Advanced Features

1. Dynamic Parameter Parsing

type CommandParser struct{}

func (p *CommandParser) ParseCommand(text string) (map[string]string, error) {
    params := make(map[string]string)

    // Default values
    params["ENVIRONMENT"] = "dev"
    params["TEST_SUITE"] = "smoke"

    if text == "" {
        return params, nil
    }

    // Parse key=value pairs
    pairs := strings.Split(text, " ")
    for _, pair := range pairs {
        if strings.Contains(pair, "=") {
            parts := strings.SplitN(pair, "=", 2)
            if len(parts) == 2 {
                params[strings.ToUpper(parts[0])] = parts[1]
            }
        }
    }

    return params, nil
}

2. Job Status Callbacks

type JobStatusCallback struct {
    WebhookURL string
    Channel    string
}

func (j *JobStatusCallback) SendStatus(jobName, status, details string) error {
    message := map[string]interface{}{
        "channel": j.Channel,
        "text":    fmt.Sprintf("Job %s: %s", jobName, status),
        "attachments": []map[string]interface{}{
            {
                "color": getStatusColor(status),
                "fields": []map[string]interface{}{
                    {
                        "title": "Details",
                        "value": details,
                        "short": false,
                    },
                },
            },
        },
    }

    return j.sendToSlack(message)
}

func getStatusColor(status string) string {
    switch strings.ToLower(status) {
    case "success":
        return "good"
    case "failure":
        return "danger"
    default:
        return "warning"
    }
}

🔧 Troubleshooting Guide

Common Debugging Commands

# Check Lambda logs
aws logs tail /aws/lambda/jenkins-slack-demo-jenkins-trigger --follow

# Test API Gateway directly
curl -X POST https://your-api-gateway-url/prod/trigger \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "text=dev smoke&user_name=testuser&channel_name=general"

# Check Jenkins connectivity from Lambda subnet
aws ec2 describe-instances --filters "Name=tag:Name,Values=jenkins-slack-demo-*"

# Verify SSM parameters
aws ssm get-parameter --name "/jenkins-slack-demo/jenkins_url" --region us-east-1
aws ssm get-parameter --name "/jenkins-slack-demo/jenkins_credentials" --with-decryption --region us-east-1

# Test Jenkins API directly
curl -u admin:your-secure-password http://localhost:8080/api/json

Performance Optimization

1. Connection Pooling

var httpClient *http.Client

func init() {
    transport := &http.Transport{
        MaxIdleConns:        100,
        MaxIdleConnsPerHost: 100,
        IdleConnTimeout:     90 * time.Second,
    }

    httpClient = &http.Client{
        Transport: transport,
        Timeout:   30 * time.Second,
    }
}

2. SSM Parameter Caching

type ParameterCache struct {
    cache map[string]string
    mutex sync.RWMutex
}

func (c *ParameterCache) GetParameter(name string) (string, error) {
    c.mutex.RLock()
    if value, exists := c.cache[name]; exists {
        c.mutex.RUnlock()
        return value, nil
    }
    c.mutex.RUnlock()

    // Fetch from SSM and cache
    c.mutex.Lock()
    defer c.mutex.Unlock()

    // SSM fetch logic here
    // ...

    c.cache[name] = value
    return value, nil
}

🎯 Key Lessons Learned

What Worked Well

Infrastructure as Code: Terraform made the setup reproducible and version-controlled
Private Jenkins: Keeping Jenkins in private subnets provided excellent security
SSM Parameter Store: Centralized secret management eliminated hardcoded credentials
Go Lambda Functions: Fast, efficient, and easy to debug

What Didn't Work Initially

Jenkins Initial Setup: Manual setup wizard was unreliable for automation
CSRF Tokens: Added unnecessary complexity for API-only access
Slack App Installation: Required careful attention to permissions and URLs
Lambda VPC Configuration: ENI management required proper cleanup

Best Practices Discovered

Automate Everything: Use scripts for Jenkins setup, not manual configuration
Test Each Layer: Verify infrastructure, Lambda, Jenkins, and Slack separately
Document Everything: Keep detailed notes of fixes and workarounds
Monitor Early: Set up CloudWatch alarms from day one

🚀 Production Deployment Checklist

Before going live, ensure you have:

[ ] Security Review: All secrets in SSM, no hardcoded credentials
[ ] Monitoring: CloudWatch alarms for errors and performance
[ ] Backup Strategy: Jenkins configuration and job definitions
[ ] Disaster Recovery: Terraform state in S3 with DynamoDB locking
[ ] Access Control: Proper IAM roles with least privilege
[ ] Network Security: Security groups reviewed and tightened
[ ] Documentation: Runbooks for common operations
[ ] Testing: End-to-end tests for all critical paths

📊 Performance Metrics

After implementing all optimizations:

Lambda Execution: ~2-3 seconds (down from 5-8 seconds)
Jenkins Job Trigger: ~1-2 seconds
End-to-End Response: ~5 seconds (down from 10-15 seconds)
Error Rate: <1% (down from 15-20%)
Infrastructure Deployment: ~8 minutes (consistent)

🎉 Conclusion

Building a production-ready Jenkins-Slack integration is more than just connecting the dots. The real value comes from understanding the challenges, implementing robust solutions, and continuously improving the system.

Key takeaways from this journey:

✅ Automation is Critical: Manual setup processes are error-prone and don't scale

✅ Security by Design: Private subnets, SSM parameters, and least privilege IAM

✅ Monitoring Matters: You can't fix what you can't see

✅ Documentation Saves Time: Detailed troubleshooting guides prevent future headaches

✅ Iterative Improvement: Start simple, then add complexity as needed

🔗 Resources and Next Steps

Repository: jenkins-slack-aws-integration
Part 1: Complete Setup Guide
AWS Documentation: Lambda in VPC
Jenkins Documentation: REST API
Slack API: Slash Commands

🤝 Community Contributions

This project is open source and welcomes contributions! Areas for improvement:

Additional CI/CD Tools: Support for GitLab, GitHub Actions, or Azure DevOps
Enhanced Monitoring: Prometheus metrics, Grafana dashboards
Multi-Environment: Support for multiple Jenkins instances
Advanced Features: Job scheduling, parameter validation, approval workflows

Ready to build something amazing? Start with Part 1 if you haven't already, then implement these production enhancements. Have questions or want to share your own solutions? Drop a comment below!

Remember: The best DevOps solutions are built through iteration, collaboration, and learning from real-world challenges. Happy building! 🚀

Cover image by @dlxmedia.hu from unsplash

Building Secure Jenkins-Slack Integration with AWS Lambda - Part 1: Complete Setup Guide

Sri — Tue, 14 Oct 2025 11:08:20 +0000

Building Secure Jenkins-Slack Integration with AWS Lambda - Part 1: Complete Setup Guide

Ever wished you could trigger your CI/CD pipelines directly from Slack without exposing Jenkins to the public internet? In this comprehensive two-part series, we'll build a production-ready solution that lets your team trigger Jenkins jobs using Slack slash commands while maintaining enterprise-grade security.

This is Part 1 of our series, focusing on architecture design, infrastructure setup, and initial implementation. Part 2 will dive deep into troubleshooting real-world issues and production enhancements.

What We'll Build

This solution enables teams to trigger Jenkins CI/CD pipelines from Slack using a secure, serverless architecture. The key innovation is keeping Jenkins completely private while providing a public API endpoint through AWS Lambda and API Gateway.

Tech Stack Overview

Infrastructure: Terraform (IaC), AWS VPC, EC2, Lambda, API Gateway, ALB, NAT Gateway
Backend: Go 1.20 (Lambda functions), Jenkins (CI/CD orchestration)
Storage/Secrets: AWS SSM Parameter Store
Integration: Slack API (Slash Commands, Webhooks)
Monitoring: CloudWatch Logs
OS: Amazon Linux 2

Architecture Deep Dive

Let's start with the complete architecture flow:

┌─────────────────────────────────────────────────────────────────┐
│                        Architecture Flow                        │
└─────────────────────────────────────────────────────────────────┘

👥 Slack User                    🌐 API Gateway
/run_test environment=staging ──→ /prod/trigger
                                        │
                                        ▼
                                ⚡ AWS Lambda
                                jenkins-trigger
                                        │
                    ┌───────────────────┼───────────────────┐
                    │                   │                   │
                    ▼                   ▼                   ▼
            🔐 AWS SSM Parameter    🔧 Jenkins (Private EC2)    🛡️ Bastion Host
            Store                   CI/CD Orchestration        SSH Access Gateway
            Jenkins URL &           ◄─────────────────────────┘
            Credentials             │
                    │               │
                    └───────────────┼───────────────────────────┐
                                    │                           │
                                    ▼                           ▼
                            ⚖️ Internal ALB              Job Status
                            Load Balancer                Notifications
                                    │                           │
                                    └───────────────────────────┘
                                                              │
                                                              ▼
                                                        👥 Slack User
                                                        (Status Updates)

Legend:
🟠 AWS Services (API Gateway, Lambda, SSM, ALB, Bastion)
🟣 Slack Integration
🔴 Jenkins (Private EC2)

Key Components Explained

🔒 VPC with Public/Private Subnets: Jenkins runs in a private subnet with no internet access, while the bastion host provides secure SSH access from the public subnet.

⚡ Lambda Function in VPC: The Lambda function runs in the private subnet to communicate with Jenkins while being invoked by API Gateway from the internet.

🌐 API Gateway: Provides a public HTTPS endpoint that Slack can reach, while the Lambda function handles the secure communication with Jenkins.

🔐 SSM Parameter Store: Stores Jenkins credentials and configuration securely, eliminating the need for hardcoded secrets.

🛡️ Bastion Host: Provides secure SSH access to Jenkins for administrative tasks without exposing Jenkins directly.

⚖️ Internal Load Balancer: Distributes traffic to Jenkins instances and provides health checks.

Security Benefits

Zero Public Exposure: Jenkins never faces the internet
Encrypted Secrets: All credentials stored in SSM with encryption
Network Isolation: Private subnets with controlled access
Least Privilege: IAM roles with minimal required permissions

Prerequisites

Before we start, ensure you have:

AWS account with appropriate permissions (EC2, Lambda, API Gateway, SSM, VPC)
Terraform >= 1.0.0 installed
AWS CLI configured with credentials
Go 1.20+ installed (for Lambda function builds)
Slack workspace with admin privileges
SSH key pair for EC2 access
Basic understanding of Jenkins, Lambda, and Terraform

Step 0: Initial Setup

Configure AWS CLI

First, configure your AWS CLI with proper credentials:

# Configure AWS CLI
aws configure

# Enter your AWS Access Key ID, Secret Access Key, region (us-east-1), and output format (json)
# Verify configuration
aws sts get-caller-identity

Create SSH Key Pair

Create an EC2 key pair for SSH access:

# Option 1: Create via AWS CLI
aws ec2 create-key-pair --key-name jenkins-slack-key --query 'KeyMaterial' --output text > jenkins-slack-key.pem
chmod 400 jenkins-slack-key.pem

# Option 2: Create via AWS Console
# Go to EC2 → Key Pairs → Create Key Pair → Name: jenkins-slack-key → Download .pem file

Step-by-Step Implementation

Step 1: Clone and Setup

git clone https://github.com/kasukur/jenkins-slack-aws-integration
cd jenkins-slack-aws-integration

Step 2: Build Lambda Functions

Build the Lambda function binaries:

# Navigate to Lambda function directories
cd lambda/jenkins_trigger
go mod tidy
GOOS=linux GOARCH=amd64 go build -o bootstrap main.go
zip jenkins_trigger.zip bootstrap

cd ../slack_verification
go mod tidy
GOOS=linux GOARCH=amd64 go build -o bootstrap main.go
zip slack_verification.zip bootstrap

# Copy zips to terraform directory
cp jenkins_trigger.zip ../../terraform/
cp slack_verification.zip ../../terraform/

The repository structure:

├── terraform/           # Infrastructure as Code
│   ├── main.tf         # Provider and key pair
│   ├── vpc.tf          # VPC, subnets, security groups
│   ├── jenkins.tf      # Jenkins EC2 and ALB
│   ├── lambda.tf       # Lambda functions
│   ├── api_gateway.tf  # API Gateway configuration
│   ├── iam.tf          # IAM roles and policies
│   └── variables.tf    # Variable definitions
├── lambda/             # Lambda function source code
│   ├── jenkins_trigger/    # Main Lambda function
│   └── slack_verification/ # Slack signature verification
└── jenkins/            # Jenkins pipeline definitions
    └── Jenkinsfile     # Test pipeline configuration

Step 3: Configure Slack App

Create Slack App

Go to Slack API: https://api.slack.com/apps
Click "Create New App" → "From scratch"
App Name: Jenkins CI/CD Integration
Workspace: Select your workspace
Click "Create App"

Configure Slash Command

In your app settings, go to "Slash Commands"
Click "Create New Command"
Fill in the details:
- Command: /run_test
- Request URL: https://your-api-gateway-url/prod/trigger (we'll get this after deployment)
- Short Description: Trigger Jenkins test pipeline
- Usage Hint: [environment] [test_suite]
Click "Save"

Enable Incoming Webhooks

Go to "Incoming Webhooks" in your app settings
Toggle "Activate Incoming Webhooks" to On
Click "Add New Webhook to Workspace"
Select a channel (e.g., #devops)
Click "Allow"
Copy the webhook URL (starts with https://hooks.slack.com/services/)

Get App Credentials

Go to "Basic Information" in your app settings
In "App Credentials" section, copy:
- App-Level Token (starts with xapp-)
- Signing Secret (long random string)

Install App to Workspace

Go to "Install App" in your app settings
Click "Install to Workspace"
Review permissions and click "Allow"
Copy the Bot User OAuth Token (starts with xoxb-)

Save these values for the Terraform configuration:

App-Level Token (xapp-...)
Signing Secret
Webhook URL
Bot User OAuth Token (xoxb-...)

Step 4: Prepare Terraform Variables

Copy the example configuration:

cp terraform/terraform.tfvars.example terraform/terraform.tfvars

Edit terraform/terraform.tfvars with your values:

# Basic Configuration
aws_region = "us-east-1"
project_name = "jenkins-slack-demo"

# Network Configuration
vpc_cidr = "10.0.0.0/16"
public_subnet_cidrs = ["10.0.1.0/24", "10.0.2.0/24"]
private_subnet_cidrs = ["10.0.101.0/24", "10.0.102.0/24"]

# SSH Access
key_name = "your-key-pair-name"
public_key = "ssh-rsa AAAAB3NzaC1yc2E... your-email@example.com"

# Jenkins Configuration
jenkins_admin_username = "admin"
jenkins_admin_password = "your-secure-password"

# Slack Configuration
slack_app_token = "xapp-your-app-token"
slack_signing_secret = "your-signing-secret"
slack_webhook_url = "https://hooks.slack.com/services/your/webhook/url"
slack_bot_token = "xoxb-your-bot-token"
slack_channel = "#devops"

Step 5: Deploy Infrastructure

cd terraform

# Initialize Terraform
terraform init

# Review the plan (should show ~25 resources)
terraform plan

# Deploy the infrastructure
terraform apply

Important: Save the outputs displayed after successful deployment:

api_gateway_url = "https://<your-api-gateway-id>.execute-api.<region>.amazonaws.com/prod/trigger"
bastion_public_ip = "<bastion-public-ip>"
jenkins_url = "http://<internal-alb-url>"

Step 6: Update Slack Configuration

Go back to your Slack app settings
Update the slash command Request URL with the API Gateway URL from Step 5
Save the changes

Step 7: Configure Jenkins

Access Jenkins

Set up SSH tunnel through the bastion host:

# SSH to bastion host with port forwarding
ssh -i your-key.pem -L 8080:<jenkins_private_ip>:8080 ec2-user@<bastion_public_ip> -N

# In another terminal, access Jenkins via localhost
# Open browser: http://localhost:8080

Initial Jenkins Setup

Wait for Jenkins initialization (5-10 minutes), then:

Get the initial admin password:

   # SSH to Jenkins instance
   ssh -i your-key.pem ec2-user@<bastion_public_ip>
   ssh ec2-user@<jenkins_private_ip>

   # Get initial password
   sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Access Jenkins at http://localhost:8080
Enter the initial admin password
Install suggested plugins
Create admin user:
- Username: admin
- Password: <your-secure-password> (use a strong password)
- Full name: Administrator
- Email: admin@yourcompany.com

Install Slack Plugin

Go to: Manage Jenkins → Manage Plugins → Available
Search for: "Slack Notification"
Check the box and click "Install without restart"
Wait for installation to complete
Go to: Manage Jenkins → Configure System
Find "Slack" section:
- Workspace: Your workspace name
- Default Channel: #devops (or your preferred channel)
- Integration Token: Paste your Bot User OAuth Token (xoxb-...)
Click "Test Connection" - should show "Success"
Click "Save"

Create Pipeline Job

Click "New Item"
Enter item name: run_test
Select "Pipeline" and click "OK"
In Pipeline section, select "Pipeline script"
Copy the content from jenkins/Jenkinsfile into the script area
Click "Save"

Step 8: Update SSM Parameters

Verify the Jenkins URL in SSM Parameter Store:

aws ssm get-parameter --name "/jenkins-slack-demo/jenkins_url" --region us-east-1

If needed, update it with the correct internal ALB URL:

aws ssm put-parameter \
  --name "/jenkins-slack-demo/jenkins_url" \
  --value "http://<internal-alb-url>" \
  --overwrite \
  --region us-east-1

Verification and Testing

Test 1: Verify Infrastructure

# Check EC2 instances
aws ec2 describe-instances --filters "Name=tag:Name,Values=jenkins-slack-demo-*" --region us-east-1

# Check Lambda functions
aws lambda list-functions --region us-east-1

# Check API Gateway
curl -X POST https://your-api-gateway-url/prod/trigger

Test 2: Verify Lambda Function

Test the Lambda function directly:

# Test Lambda function
aws lambda invoke \
  --function-name jenkins-slack-demo-jenkins-trigger \
  --payload '{}' \
  /tmp/lambda-response.json && cat /tmp/lambda-response.json

Test 3: Verify Jenkins Connectivity

Test Jenkins accessibility from Lambda:

# SSH to Jenkins and verify it's running
ssh -i your-key.pem ec2-user@<bastion_public_ip>
ssh ec2-user@<jenkins_private_ip>

# Check Jenkins status
sudo systemctl status jenkins
curl -u admin:<your-secure-password> http://localhost:8080/api/json

Test 4: Test Slack Command

In any Slack channel where your app is installed, type:

/run_test

Expected behavior:

Immediate response in Slack: "Job triggered successfully!"
Jenkins job appears in the dashboard
Job runs with default parameters (ENVIRONMENT=dev, TEST_SUITE=smoke)

Test 5: Verify Jenkins Job Execution

Check Jenkins dashboard for the run_test job
Verify job parameters (ENVIRONMENT=dev, TEST_SUITE=smoke)
Check job logs for successful execution
Confirm Slack receives success/failure notifications

Test 6: End-to-End Test with Parameters

Test with custom parameters:

/run_test environment=staging test_suite=regression

Monitor the complete flow:

Slack → API Gateway → Lambda
Lambda → Jenkins (job trigger with parameters)
Jenkins → Slack (status notification)

Key Code Components

Lambda Function (Go)

The Lambda function handles the core integration logic:

func handler(ctx context.Context, request events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
    // Get Jenkins URL and credentials from SSM
    jenkinsURL, err := getSSMParameter(ctx, jenkinsURLParam, false)
    credsJSON, err := getSSMParameter(ctx, jenkinsCredsParam, true)

    // Get CSRF crumb for Jenkins security
    crumbField, crumb, err := getJenkinsCrumb(client, jenkinsURL, creds.Username, creds.Password)

    // Trigger Jenkins job
    jobURL := fmt.Sprintf("%s/job/run_test/buildWithParameters", jenkinsURL)
    data := url.Values{}
    data.Set("ENVIRONMENT", "dev")
    data.Set("TEST_SUITE", "smoke")

    // Make authenticated request with crumb
    req.Header.Set("Jenkins-Crumb", crumb)
    req.SetBasicAuth(creds.Username, creds.Password)

    return events.APIGatewayProxyResponse{
        StatusCode: 200,
        Body:       `{"message": "Job triggered! Status: 201"}`,
    }, nil
}

Jenkins Pipeline

The Jenkinsfile defines the test pipeline:

pipeline {
    agent any

    parameters {
        string(name: 'ENVIRONMENT', defaultValue: 'dev', description: 'Environment to run tests against')
        string(name: 'TEST_SUITE', defaultValue: 'smoke', description: 'Test suite to run')
    }

    stages {
        stage('Prepare') {
            steps {
                echo "Preparing to run tests in ${params.ENVIRONMENT} environment"
                slackSend(color: '#FFFF00', message: "STARTED: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]'")
            }
        }

        stage('Run Tests') {
            steps {
                echo "Running ${params.TEST_SUITE} tests against ${params.ENVIRONMENT}"
                // Your actual test commands here
            }
        }
    }

    post {
        success {
            slackSend(color: '#00FF00', message: "SUCCESS: Job completed successfully")
        }
        failure {
            slackSend(color: '#FF0000', message: "FAILED: Job failed")
        }
    }
}

What's Next?

Congratulations! You've successfully built the foundation of a secure Jenkins-Slack integration. At this point, you should have:

✅ Working Infrastructure: VPC, Jenkins, Lambda, API Gateway

✅ Basic Integration: Slack can trigger Jenkins jobs

✅ Security: Private Jenkins with secure access patterns

✅ Monitoring: CloudWatch logs for observability

Coming Up in Part 2

In Part 2 of this series, we'll dive deep into:

🔧 Real-World Troubleshooting: Common issues you'll encounter and battle-tested solutions

🛠️ Production Enhancements: Security improvements, monitoring, and scaling considerations

🚨 Critical Fixes: Jenkins authentication issues, CSRF token problems, and Slack integration challenges

📊 Advanced Features: Parameter parsing, job status callbacks, and multi-environment support

Key Takeaways

This architecture demonstrates several important DevOps and AWS best practices:

✅ Security First: Jenkins is never exposed to the internet, following zero-trust principles

✅ Infrastructure as Code: Everything is defined in Terraform, making it reproducible and version-controlled

✅ Serverless Integration: Lambda provides cost-effective, scalable integration without managing servers

✅ Secret Management: SSM Parameter Store eliminates hardcoded credentials

✅ Network Isolation: VPC with public/private subnets provides proper network segmentation

✅ Observability: CloudWatch logs provide visibility into system behavior

Ready to tackle the real-world challenges? Check out Part 2 where we'll solve the tricky issues that make or break production deployments!

Have questions or suggestions for improvements? Feel free to reach out or contribute to the project repository!

Cover image by @dlxmedia.hu from unsplash

Setting up AWS Client VPN with CloudFormation and AWS CLI

Sri — Sat, 26 Jul 2025 03:16:24 +0000

After I published my blog on How to Set Up an AWS Client VPN, someone asked if I could share a CloudFormation template to automate the setup. In response, I’ve put together this follow-up post to walk you through deploying an AWS Client VPN endpoint using CloudFormation and AWS CLI. You’ll learn how to associate the VPN with your VPC, configure access rules, and handle the critical manual step of updating security group rules. I’ve also included a handy shell script to simplify and automate the entire deployment process.

What is AWS Client VPN?
Prerequisites
Step-by-Step Deployment with CloudFormation
- 1. Generate Client and Server Certificates
- 2. Deploy VPC and EC2 Instance Optional
- 3. Deploy the Client VPN CloudFormation Stack
- 4. Manual Step Update Security Group Rule
- 5. Authorize Clients to Access the Network
- 6. Optional Add Route to Enable Traffic from VPN to VPC
Deploying with AWS CLI Alternate Method
- 1. Create Client VPN Endpoint
- 2. Associate Client VPN Endpoint with a Subnet
- 3. Authorize Client VPN Ingress
- 4. Revoke Client VPN Ingress Optional
Deployment Script
Cleanup
Conclusion
Referrals

What is AWS Client VPN?

AWS Client VPN is a managed client-based VPN service that enables you to securely access your AWS resources and on-premises networks from any location. It provides a highly available and scalable solution for remote access, allowing your users to connect to your private networks using a VPN client.

Prerequisites

Before you begin, ensure you have the following:

An AWS account.
AWS CLI installed and configured with appropriate permissions.
OpenSSL installed for certificate generation (if not using existing certificates).
A VPC where you want to deploy the Client VPN endpoint. The provided CloudFormation template assumes a new VPC will be created.

Step-by-Step Deployment with CloudFormation

AWS CloudFormation allows you to define your AWS infrastructure as code, making it easy to deploy and manage resources.

1. Generate Client and Server Certificates

AWS Client VPN requires mutual authentication, which means both the server and client need certificates. You can generate these using OpenSSL. The provided generate_certs.sh script in the attached zip file can help you with this, as detailed in the original blog post: How to Set Up an AWS Client VPN. Make sure to upload these certificates to AWS Certificate Manager (ACM).

# Example command to generate certificates (from generate_certs.sh)
# This is a simplified example, refer to the actual script for full details.
# You will need to import these into ACM.

# Generate server certificate and key
openssl genrsa -out server.key 2048
openssl req -new -key server.key -out server.csr
openssl x509 -req -in server.csr -signkey server.key -out server.crt

# Generate client certificate and key
openssl genrsa -out client.key 2048
openssl req -new -key client.key -out client.csr
openssl x509 -req -in client.csr -signkey client.key -out client.crt

# Import server certificate to ACM
aws acm import-certificate --certificate fileb://server.crt --private-key fileb://server.key --certificate-chain fileb://ca.crt

# Import client certificate to ACM
aws acm import-certificate --certificate fileb://client.crt --private-key fileb://client.key --certificate-chain fileb://ca.crt

Important: Note down the ARNs of the imported server and client certificates. You will need them for the CloudFormation deployment.

2. Deploy VPC and EC2 Instance (Optional)

If you don't have an existing VPC and an EC2 instance in a private subnet, you can use the vpc-ec2.yaml CloudFormation template (provided in the attached zip file) to set up a basic environment. This template creates a VPC, a private subnet, a route table, a security group, and an EC2 instance within that private subnet.

AWSTemplateFormatVersion: '2010-09-09'
Description: VPC, subnet, security group, and EC2 instance for AWS Client VPN demo.

Parameters:
  KeyName:
    Type: String
    Description: Name of an existing EC2 KeyPair to enable SSH access
  LatestAmiId:
    Type: 'AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>'
    Description: SSM parameter for the latest Amazon Linux 2 AMI

Resources:
  MyVPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 192.168.0.0/16
      EnableDnsSupport: true
      EnableDnsHostnames: true
      Tags:
        - Key: Name
          Value: MyClientVPNVPC

  MyPrivateSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref MyVPC
      CidrBlock: 192.168.1.0/24
      MapPublicIpOnLaunch: false
      AvailabilityZone: !Select [0, !GetAZs '']
      Tags:
        - Key: Name
          Value: MyPrivateSubnet

  MyRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref MyVPC
      Tags:
        - Key: Name
          Value: MyPrivateRouteTable

  MySubnetRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref MyPrivateSubnet
      RouteTableId: !Ref MyRouteTable

  MySecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Allow SSH from Client VPN
      VpcId: !Ref MyVPC
      SecurityGroupIngress: []
      Tags:
        - Key: Name
          Value: MyPrivateSG

  MyInstanceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      Roles: [!Ref MyInstanceRole]

  MyInstanceRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: ec2.amazonaws.com
            Action: sts:AssumeRole
      Path: '/'
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

  MyEC2Instance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: t3.micro
      KeyName: !Ref KeyName
      SubnetId: !Ref MyPrivateSubnet
      SecurityGroupIds: [!Ref MySecurityGroup]
      IamInstanceProfile: !Ref MyInstanceProfile
      ImageId: !Ref LatestAmiId
      Tags:
        - Key: Name
          Value: MyPrivateEC2

Outputs:
  VpcId:
    Description: VPC ID
    Value: !Ref MyVPC
  SubnetId:
    Description: Private Subnet ID
    Value: !Ref MyPrivateSubnet
  SecurityGroupId:
    Description: Security Group ID
    Value: !Ref MySecurityGroup
  InstanceId:
    Description: EC2 Instance ID
    Value: !Ref MyEC2Instance

To deploy this stack, you can use the AWS CLI:

aws cloudformation deploy \
    --stack-name MyVPCAndEC2Stack \
    --template-file vpc-ec2.yaml \
    --capabilities CAPABILITY_NAMED_IAM \
    --parameter-overrides \
        KeyName=YOUR_KEY_PAIR_NAME \
        LatestAmiId=/aws/service/ami-amazon-linux-2/latest/amzn2-ami-hvm-x86_64-gp2

Note: Replace YOUR_KEY_PAIR_NAME with your actual key pair name. If you deploy this stack, you will need to retrieve the VpcId, SubnetId, and SecurityGroupId from its outputs to use in the next step.

3. Deploy the Client VPN CloudFormation Stack

The aws-client-vpn.yaml CloudFormation template (provided in the attached zip file) defines the necessary AWS resources for the Client VPN endpoint. This template assumes you have an existing VPC and subnet, or have deployed them using the previous step. It sets up the Client VPN endpoint itself, and includes a security group for the EC2 instance.

AWSTemplateFormatVersion: '2010-09-09'
Description: >
  AWS Client VPN and EC2 setup.
  - VPC: 192.168.0.0/16
  - Private Subnet: 192.168.1.0/24
  - EC2 Instance in private subnet
  - Security group for EC2
  - Client VPN endpoint (client CIDR: 10.0.0.0/22, does not overlap with VPC)

Parameters:
  KeyName:
    Type: String
    Description: Name of an existing EC2 KeyPair to enable SSH access
  ServerCertificateArn:
    Type: String
    Description: ARN of the ACM server certificate for the Client VPN endpoint
  ClientCertificateArn:
    Type: String
    Description: ARN of the ACM client certificate for mutual authentication
  LatestAmiId:
    Type: 'AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>'
    Description: SSM parameter for the latest Amazon Linux 2 AMI

Resources:
  MyVPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 192.168.0.0/16
      EnableDnsSupport: true
      EnableDnsHostnames: true
      Tags:
        - Key: Name
          Value: MyClientVPNVPC

  MyPrivateSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref MyVPC
      CidrBlock: 192.168.1.0/24
      MapPublicIpOnLaunch: false
      AvailabilityZone: !Select [0, !GetAZs '']
      Tags:
        - Key: Name
          Value: MyPrivateSubnet

  MyRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref MyVPC
      Tags:
        - Key: Name
          Value: MyPrivateRouteTable

  MySubnetRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref MyPrivateSubnet
      RouteTableId: !Ref MyRouteTable

  MySecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Allow SSH from Client VPN
      VpcId: !Ref MyVPC
      SecurityGroupIngress: []
      Tags:
        - Key: Name
          Value: MyPrivateSG

  MyInstanceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      Roles: [!Ref MyInstanceRole]

  MyInstanceRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: ec2.amazonaws.com
            Action: sts:AssumeRole
      Path: '/'
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

  MyEC2Instance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: t3.micro
      KeyName: !Ref KeyName
      SubnetId: !Ref MyPrivateSubnet
      SecurityGroupIds: [!Ref MySecurityGroup]
      IamInstanceProfile: !Ref MyInstanceProfile
      ImageId: !Ref LatestAmiId
      Tags:
        - Key: Name
          Value: MyPrivateEC2

  MyClientVPNEndpoint:
    Type: AWS::EC2::ClientVpnEndpoint
    Properties:
      AuthenticationOptions:
        - Type: certificate-authentication
          MutualAuthentication:
            ClientRootCertificateChainArn: !Ref ClientCertificateArn
      ConnectionLogOptions:
        Enabled: false
      ServerCertificateArn: !Ref ServerCertificateArn
      ClientCidrBlock: 10.0.0.0/22
      DnsServers:
        - 8.8.8.8
        - 8.8.4.4
      SplitTunnel: true
      TagSpecifications:
        - ResourceType: client-vpn-endpoint
          Tags:
            - Key: Name
              Value: MyClientVPNEndpoint

  MyClientVPNAuthorizationRule:
    Type: AWS::EC2::ClientVpnAuthorizationRule
    Properties:
      ClientVpnEndpointId: !Ref MyClientVPNEndpoint
      TargetNetworkCidr: 192.168.1.0/24 # Your VPC subnet
      AuthorizeAllGroups: true
      Description: Allow access to VPC subnet

Outputs:
  VpcId:
    Description: VPC ID
    Value: !Ref MyVPC
  SubnetId:
    Description: Private Subnet ID
    Value: !Ref MyPrivateSubnet
  SecurityGroupId:
    Description: Security Group ID
    Value: !Ref MySecurityGroup
  InstanceId:
    Description: EC2 Instance ID
    Value: !Ref MyEC2Instance
  ClientVpnEndpointId:
    Description: Client VPN Endpoint ID
    Value: !Ref MyClientVPNEndpoint

To deploy this stack, you can use the AWS CLI:

aws cloudformation deploy \
    --stack-name MyClientVPNStack \
    --template-file aws-client-vpn.yaml \
    --capabilities CAPABILITY_NAMED_IAM \
    --parameter-overrides \
        KeyName=YOUR_KEY_PAIR_NAME \
        ServerCertificateArn=YOUR_SERVER_CERT_ARN \
        ClientCertificateArn=YOUR_CLIENT_CERT_ARN \
        LatestAmiId=/aws/service/ami-amazon-linux-2/latest/amzn2-ami-hvm-x86_64-gp2

Note: Replace YOUR_KEY_PAIR_NAME, YOUR_SERVER_CERT_ARN, and YOUR_CLIENT_CERT_ARN with your actual values.

4. Manual Step: Update Security Group Rule

After the CloudFormation stack is deployed, you need to manually update the security group associated with your EC2 instance to allow ingress from the Client VPN. This step is crucial because the Client VPN endpoint assigns IP addresses from its own CIDR block (e.g., 10.0.0.0/22 in our template), and your EC2 instance's security group needs to explicitly allow traffic from this range.

Navigate to the EC2 console and find the security group created by the CloudFormation stack (e.g., MyPrivateSG).
Go to the Inbound Rules tab and click Edit inbound rules.
Add a new rule:
- Type: All TCP (or specific ports like SSH if preferred)
- Source: Custom, and enter the Client VPN CIDR block (e.g., 10.0.0.0/22).
- Description: Allow traffic from Client VPN.
Save the rules.

This manual step is necessary because CloudFormation does not inherently know the dynamically assigned Client VPN CIDR block at the time of stack creation, and it's a best practice to explicitly define ingress rules for security groups.

5. Authorize Clients to Access the Network

6. (Optional) Add Route to Enable Traffic from VPN to VPC

To allow VPN clients to route traffic to your VPC, you need to add a route:

aws ec2 create-client-vpn-route \
  --client-vpn-endpoint-id <VPN_ENDPOINT_ID> \
  --destination-cidr-block 192.168.1.0/24 \
  --target-vpc-subnet-id <SUBNET_ID> \
  --region ap-south-1

Replace <VPN_ENDPOINT_ID> and <SUBNET_ID> with your actual values. This is essential for traffic to flow from VPN clients to private resources in the VPC.

Even after associating the Client VPN endpoint with a subnet, clients cannot access resources until authorization rules are defined. This rule specifies which network resources clients can access.

aws ec2 authorize-client-vpn-ingress \
  --client-vpn-endpoint-id <VPN_ENDPOINT_ID> \
  --target-network-cidr 192.168.1.0/24 \
  --authorize-all-groups \
  --description "Allow access to VPC subnet" \
  --region ap-south-1

Replace <VPN_ENDPOINT_ID> with the actual ID of your Client VPN endpoint, which you can get from the CloudFormation stack outputs or the AWS console. The target-network-cidr should be the CIDR block of the subnet you want your VPN clients to access (e.g., 192.168.1.0/24 for the private subnet in our CloudFormation template).

Deploying with AWS CLI (Alternate Method)

For those who prefer a more granular approach or need to integrate into existing scripts, here are the AWS CLI commands to set up an AWS Client VPN. This section assumes you have already generated and imported your server and client certificates into ACM.

1. Create Client VPN Endpoint

aws ec2 create-client-vpn-endpoint \
    --client-cidr-block "10.0.0.0/22" \
    --server-certificate-arn "arn:aws:acm:ap-south-1:123456789012:certificate/your-server-cert-id" \
    --authentication-options Type=certificate-authentication,MutualAuthentication={ClientRootCertificateChainArn="arn:aws:acm:ap-south-1:123456789012:certificate/your-client-cert-id"} \
    --connection-log-options Enabled=false \
    --dns-servers "8.8.8.8" "8.8.4.4" \
    --split-tunnel \
    --tag-specifications 'ResourceType=client-vpn-endpoint,Tags=[{Key=Name,Value=MyClientVPNEndpointCLI}]'

Note: Replace the ARN values with your actual certificate ARNs.

2. Associate Client VPN Endpoint with a Subnet

aws ec2 associate-client-vpn-target-network \
    --client-vpn-endpoint-id <VPN_ENDPOINT_ID> \
    --subnet-id <SUBNET_ID>

Replace <VPN_ENDPOINT_ID> and <SUBNET_ID> with your respective IDs.

3. Authorize Client VPN Ingress

aws ec2 authorize-client-vpn-ingress \
    --client-vpn-endpoint-id <VPN_ENDPOINT_ID> \
    --target-network-cidr 192.168.1.0/24 \
    --authorize-all-groups \
    --description "Allow access to VPC subnet" \
    --region ap-south-1

4. Revoke Client VPN Ingress (Optional)

aws ec2 revoke-client-vpn-ingress \
    --client-vpn-endpoint-id <VPN_ENDPOINT_ID> \
    --target-network-cidr 192.168.1.0/24 \
    --revoke-all-groups \
    --region ap-south-1

Deployment Script

To automate the deployment process, you can use a shell script. This script provides options to deploy an optional VPC and EC2 instance, deploy the Client VPN CloudFormation stack, automatically add the necessary security group rules, and includes a cleanup function to tear down the deployed resources. Make sure you have the aws-client-vpn.yaml, vpc-ec2.yaml, and generate_certs.sh files in the same directory as this script.

#!/bin/bash

# Variables (customize these)
STACK_NAME="MyClientVPNStack"
KEY_PAIR_NAME="YOUR_KEY_PAIR_NAME"
SERVER_CERT_ARN="YOUR_SERVER_CERT_ARN"
CLIENT_CERT_ARN="YOUR_CLIENT_CERT_ARN"
REGION="ap-south-1"
TARGET_NETWORK_CIDR="192.168.1.0/24"

# 1. Deploy CloudFormation Stack
echo "Deploying CloudFormation stack..."
aws cloudformation deploy \
    --stack-name ${STACK_NAME} \
    --template-file aws-client-vpn.yaml \
    --capabilities CAPABILITY_NAMED_IAM \
    --parameter-overrides \
        KeyName=${KEY_PAIR_NAME} \
        ServerCertificateArn=${SERVER_CERT_ARN} \
        ClientCertificateArn=${CLIENT_CERT_ARN} \
        LatestAmiId=/aws/service/ami-amazon-linux-2/latest/amzn2-ami-hvm-x86_64-gp2 \
    --region ${REGION}

if [ $? -ne 0 ]; then
    echo "CloudFormation deployment failed. Exiting."
    exit 1
fi

echo "CloudFormation stack deployed successfully. Waiting for resources..."
aws cloudformation wait stack-create-complete --stack-name ${STACK_NAME} --region ${REGION}

# Ensure Client VPN endpoint is in available state
echo "Waiting for VPN endpoint to become available..."
sleep 15

# Get Client VPN Endpoint ID from stack outputs
VPN_ENDPOINT_ID=$(aws cloudformation describe-stacks \
    --stack-name ${STACK_NAME} \
    --query "Stacks[0].Outputs[?OutputKey==\'ClientVpnEndpointId\'].OutputValue" \
    --output text \
    --region ${REGION})

if [ -z "${VPN_ENDPOINT_ID}" ]; then
    echo "Could not retrieve Client VPN Endpoint ID. Exiting."
    exit 1
fi

echo "Client VPN Endpoint ID: ${VPN_ENDPOINT_ID}"

# 2. Authorize Client VPN Ingress Rule
echo "Authorizing Client VPN ingress rule..."
aws ec2 authorize-client-vpn-ingress \
  --client-vpn-endpoint-id ${VPN_ENDPOINT_ID} \
  --target-network-cidr ${TARGET_NETWORK_CIDR} \
  --authorize-all-groups \
  --description "Allow access to VPC subnet" \
  --region ${REGION}

if [ $? -ne 0 ]; then
    echo "Failed to authorize client VPN ingress. Exiting."
    exit 1
fi

echo "Client VPN ingress rule authorized successfully."

# 3. Get Security Group ID from stack outputs
SECURITY_GROUP_ID=$(aws cloudformation describe-stacks \
    --stack-name ${STACK_NAME} \
    --query "Stacks[0].Outputs[?OutputKey==\'SecurityGroupId\'].OutputValue" \
    --output text \
    --region ${REGION})

if [ -z "${SECURITY_GROUP_ID}" ]; then
    echo "Could not retrieve Security Group ID. Exiting."
    exit 1
fi

echo "Security Group ID: ${SECURITY_GROUP_ID}"

# 4. Get Client VPN CIDR Block
CLIENT_VPN_CIDR=$(aws ec2 describe-client-vpn-endpoints \
    --client-vpn-endpoint-ids ${VPN_ENDPOINT_ID} \
    --query "ClientVpnEndpoints[0].ClientCidrBlock" \
    --output text \
    --region ${REGION})

if [ -z "${CLIENT_VPN_CIDR}" ]; then
    echo "Could not retrieve Client VPN CIDR Block. Exiting."
    exit 1
fi

echo "Client VPN CIDR Block: ${CLIENT_VPN_CIDR}"

# 5. Add Ingress Rule to EC2 Security Group
echo "Adding ingress rule to EC2 Security Group..."
aws ec2 authorize-security-group-ingress \
    --group-id ${SECURITY_GROUP_ID} \
    --protocol tcp \
    --port 22 \
    --cidr ${CLIENT_VPN_CIDR} \
    --description "Allow SSH from Client VPN" \
    --region ${REGION}

if [ $? -ne 0 ]; then
    echo "Failed to add ingress rule to security group. Exiting."
    exit 1
fi

echo "Ingress rule added to security group successfully."

echo "Deployment complete!"

Usage:

Save the above script as deploy_client_vpn.sh.
Make it executable: chmod +x deploy_client_vpn.sh.
Customize the variables at the beginning of the script (STACK_NAME, KEY_PAIR_NAME, SERVER_CERT_ARN, CLIENT_CERT_ARN, REGION, TARGET_NETWORK_CIDR).
Run the script: ./deploy_client_vpn.sh.

Cleanup

To avoid incurring unnecessary costs, it is important to clean up the AWS resources created by these CloudFormation stacks when they are no longer needed. You can delete the stacks using the AWS CLI.

Deleting the Client VPN Stack

aws cloudformation delete-stack \
    --stack-name MyClientVPNStack \
    --region ap-south-1

Wait for the stack deletion to complete:

aws cloudformation wait stack-delete-complete \
    --stack-name MyClientVPNStack \
    --region ap-south-1

Deleting the VPC and EC2 Stack (if deployed)

If you deployed the optional VPC and EC2 stack, you should delete it as well:

aws cloudformation delete-stack \
    --stack-name MyVPCAndEC2Stack \
    --region ap-south-1

Wait for the stack deletion to complete:

aws cloudformation wait stack-delete-complete \
    --stack-name MyVPCAndEC2Stack \
    --region ap-south-1

Note: Ensure that any associated resources (like EC2 instances, security groups, etc.) are properly terminated before attempting to delete the VPC, as CloudFormation might not delete resources that have dependencies.

Conclusion

Setting up an AWS Client VPN provides a secure and flexible way to access your AWS resources remotely. Whether you prefer the infrastructure-as-code approach with CloudFormation or the direct control of AWS CLI, both methods allow you to establish a robust VPN solution. Remember the crucial step of updating the security group ingress rules to ensure proper connectivity from your VPN clients to your target resources. The provided deployment script automates this process, streamlining your setup.

Happy VPNing!

Referrals

My AWS Certified Security Specialty Certification Study Guide

Sri — Sat, 22 Mar 2025 04:34:13 +0000

Important: Having a solid foundation in AWS core services is essential. If you're new to AWS, I recommend completing the AWS Certified Solutions Architect - Associate certification first.

1. Study Guide for AWS Security Specialty

No	Course/Resource	Time	Status	Comments
1	Stephane Maarek's Ultimate AWS Certified Security Specialty	4 weeks	Completed	Comprehensive course with hands-on labs
2	Tutorial Dojo Practice Exams	2 weeks	Completed	Highly recommended for exam preparation
3	AWS Security Specialty Exam Readiness Course	1 week	Completed	Essential AWS official prep material
4	AWS Certified Security - Specialty Study Notes	1 week	Completed	Comprehensive community study notes

2. Essential AWS Security Workshops:

Practice hands-on with these official AWS workshops:

3. Key Focus Areas:

Identity and Access Management (IAM)
- IAM policies and permissions
- IAM roles and federation
- AWS Organizations
- AWS SSO/IAM Identity Center
Data Protection
- KMS and CloudHSM
- AWS Certificate Manager
- Secrets Manager
- AWS Shield and WAF
Infrastructure Security
- VPC Security
- Network ACLs and Security Groups
- AWS Network Firewall
- AWS Security Hub
Incident Response
- CloudWatch
- CloudTrail
- AWS Config
- Amazon Detective
Compliance and Audit
- AWS Audit Manager
- AWS Control Tower
- AWS Artifact
- Amazon Macie

FAQs:

How long does it take to prepare?
With prior AWS experience, expect 2-3 months of dedicated study. Without significant AWS security experience, 4-6 months might be more realistic.

What prerequisites are recommended?
While not mandatory, having an AWS Associate-level certification (particularly Solutions Architect) provides a strong foundation for the security concepts.

How is this exam different from Associate-level exams?
This is a specialty exam focusing deeply on security services and concepts. Questions are typically more scenario-based and require detailed understanding of security implementations.

What's the exam format?
The exam consists of 65 questions to be completed in 170 minutes. Questions are multiple choice and multiple response.

Exam Tips:

Focus on understanding security services in-depth, particularly IAM, KMS, and security-related aspects of VPC.

Practice with real-world scenarios through AWS workshops and labs.

Pay special attention to compliance frameworks and AWS's shared responsibility model.

Review AWS security whitepapers and best practices documentation.

For non-native English speakers, you can request ESL+30 minutes accommodation through certmetrics.

Additional Resources:

Note: Keep in mind that AWS regularly updates its services and exam content. Always verify the latest exam guide from the official AWS Certification page before starting your preparation.

Deploying a Simple Static Website on AWS with CDK and TypeScript

Sri — Wed, 01 Jan 2025 02:27:55 +0000

You may have come across several blogs on hosting static websites on S3. This guide demonstrates how to create and deploy a simple static website using AWS CDK with TypeScript. The project will cover:

Setting up a basic AWS CDK project structure.
Defining infrastructure as code (IaC) with TypeScript.
Creating AWS resources necessary for hosting a static website.
Deploying the website to AWS.

The website source code is hosted on GitHub: kasukur/cdk-s3-website.

Prerequisites
Step 1: Initialize the Project
Step 2: Create a Simple Website
Step 3: Write CDK Code to Host the Website
Step 4: Deploy the Website
Full Command History
Clean Up
Conclusion
Referrals

Prerequisites

Before starting, ensure you have the following installed:

Node.js (>= 14.x)
AWS CLI (configured with credentials)
AWS CDK (>= 2.x)

To install AWS CDK globally:

npm install -g aws-cdk

Step 1: Initialize the Project

Create a project directory and navigate to it:

 mkdir cdk-s3-website && cd cdk-s3-website

Initialize a new CDK app:
```
 cdk init app --language typescript
```

Install dependencies for the CDK constructs we'll use:

 npm install @aws-cdk/aws-s3 @aws-cdk/aws-s3-deployment

Verify the setup by synthesizing the stack:
```
 cdk synth
```

This command generates the CloudFormation template for your stack.

Step 2: Create a Simple Website

Download the website tempalte from html5up.net:
unzip the file, navigate to the folder and copy all the files to the website folder.

cp -r ~/Downloads/html5up-dimension ~/Documents/cdk-s3-website/website/.

Step 3: Write CDK Code to Host the Website

Edit the lib/cdk-s3-website-stack.ts file to define the resources:

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as s3 from 'aws-cdk-lib/aws-s3';
import * as s3deploy from 'aws-cdk-lib/aws-s3-deployment';

export class CdkS3WebsiteStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Create an S3 bucket for website hosting
    const websiteBucket = new s3.Bucket(this, 'WebsiteBucket', {
      websiteIndexDocument: 'index.html',
      publicReadAccess: true,
      removalPolicy: cdk.RemovalPolicy.DESTROY, // Allow deletion of non-empty bucket
      autoDeleteObjects: true, // Automatically delete objects when bucket is removed
    });

    // Deploy the website content to the S3 bucket
    new s3deploy.BucketDeployment(this, 'DeployWebsite', {
      sources: [s3deploy.Source.asset('./website')],
      destinationBucket: websiteBucket,
    });

    // Output the website URL
    new cdk.CfnOutput(this, 'WebsiteURL', {
      value: websiteBucket.bucketWebsiteUrl,
    });
  }
}

Step 4: Deploy the Website

Run the following commands to deploy your website:

Bootstrap the environment:
```
 cdk bootstrap
```

You might noticed the following error:

 Error: Cannot use 'publicReadAccess' property on a bucket           without allowing bucket-level public access through 'blockPublicAccess' property.
at new Bucket (/Users/sridharkasukurthy/Documents/cdk-s3-website/node_modules/aws-cdk-lib/aws-s3/lib/bucket.js:1:24460)
     ...

Troubleshooting: The error occurs because the publicReadAccess: true property is used without explicitly allowing public access by disabling the blockPublicAccess feature. AWS CDK enforces stricter controls to prevent unintended public access.

Solution: To fix this, we need to configure the bucket to allow public access explicitly by setting the blockPublicAccess feature.

Updated Code:

 import * as cdk from 'aws-cdk-lib';
 import { Construct } from 'constructs';
 import * as s3 from 'aws-cdk-lib/aws-s3';
 import * as s3Deploy from 'aws-cdk-lib/aws-s3-deployment';

 export class CdkS3WebsiteStack extends cdk.Stack {
   constructor(scope: Construct, id: string, props?: cdk.StackProps) {
     super(scope, id, props);

     // Create an S3 bucket for the website with public access explicitly allowed
     const websiteBucket = new s3.Bucket(this, 'SriSriWebsiteBucket', {
       websiteIndexDocument: 'index.html',
       publicReadAccess: true,
       removalPolicy: cdk.RemovalPolicy.DESTROY, // Optional: Remove the bucket on stack deletion
       blockPublicAccess: s3.BlockPublicAccess.BLOCK_ACLS, // Allows public access but blocks ACLs
     });

     // Deploy website contents to the bucket
     new s3Deploy.BucketDeployment(this, 'DeployWebsite', {
       sources: [s3Deploy.Source.asset('./website')],
       destinationBucket: websiteBucket,
     });

     // Output the website URL
     new cdk.CfnOutput(this, 'WebsiteURL', {
       value: websiteBucket.bucketWebsiteUrl,
     });
   }
 }

Bootstrap the environment again:
```
 cdk bootstrap
```
Deploy the stack:
```
 cdk deploy
```
Do you wish to deploy these changes (y/n)? y

During deployment, the CDK will output the URL for the hosted website. You can access your site using this URL.

Full Command History

Here’s a reference of all commands used:

1  mkdir cdk-s3-website && cd cdk-s3-website
2  cdk init app --language typescript
3  npm install @aws-cdk/aws-s3 @aws-cdk/aws-s3-deployment
4  git clone https://github.com/kasukur/s3-website.git website
5  cdk synth
6  cdk bootstrap
7  cdk deploy

Clean Up

To avoid incurring unnecessary charges, delete the stack. If the cdk destroy command fails due to non-empty buckets, empty the bucket manually or use the autoDeleteObjects property in your stack as shown above:

Manually empty the bucket:

   aws s3 rm s3://<bucket-name> --recursive

Destroy the stack:

   cdk destroy

Conclusion

This project demonstrated how to use AWS CDK with TypeScript to deploy a simple static website on AWS. The focus was on showcasing the CDK’s capabilities for infrastructure provisioning and deployment automation. You can now customize and expand upon this setup for more complex use cases.

Referrals

My AWS DevOps Engineer Professional Study Guide

Sri — Sat, 27 Apr 2024 12:56:10 +0000

Important: Complete both the courses and study material suggested in the My AWS SysOps Associate study guide and My AWS Developer Associate study guide first as this has some overlap. After you have completed these, then follow the study guide below.

Please review the following in addition to my
AWS DevOps Engineer Professional Notes.

1. My Study Guide for AWS DevOps Engineer Professional

No	Course	Time	Status	Comments
1	Stephane's AWS Certified DevOps Engineer Professional	15 days	Completed	Preferable if you have all three associate certifications
2	Tutorial Dojo Practice Exams	7 days	Completed	Highly recommend
3	Stephane Maarek Practice Exams	1 day	Completed	Highly recommend
4	Neal Davis Practice Exams	2 days	Completed	Highly recommend
5	AWS SkillBuilder - Exam Prep Standard Course: AWS Certified DevOps Engineer - Professional	1 week	Completed	You MUST COMPLETE this course before taking the exam.
6	AWS SkillBuilder - AWS Exam Preparation Official Practice Question Sets Overview and Instructions.pdf	1 hour	Completed	You MUST COMPLETE this course before taking the exam.
7	Whizlabs - Hands-on Labs	7 days	some labs	Optional

2. AWS Workshops for DevOps:

AWS DevOps Engineer Professional Exam tests your hands on experience, So I highly recommend you gain some hands on experience using the following workshops:

Introduction to AWS Code Family
Building event-driven architectures on AWS
AWS CloudFormation Workshop
One Observability Workshop - Focus on AWS CloudWatch

FAQs:

Which video courses do I need to take?
If you haven't completed all three associate certifications, then I recommend completing Stephane's Sys Ops and Developer Associate courses first.

How long does it take to prepare?
If you have completed Sys Ops and Developer Associate exams then probably a month or two. Otherwise you might need at least 4-6 months.
It took me six months in total (part time and a bit on and off due to some other activities).
I suggest taking the exam as soon as you can after completing the Associate level certifications as the concepts will be fresh in your mind and you would have built up the momentum to complete the next certification level.

What if I do not feel confident to take the exam?
I didn't feel confident even after completing the practice exams, so I took AWS SkillBuilder - Exam Prep Standard Course: AWS Certified DevOps Engineer - Professional which helped me gain confidence. This course has precisely everything that is needed for the Professional exam (re:Invent videos, FAQs, blogs, etc.)

How is the Professional exam different to an Associate exam?
The Professional exam is very different to the Associate exam as some of the questions are longer and some questions have two correct answers. It is important to pick the most appropriate answer based on the requirement in the question. Most importantly, AWS DevOps Engineer Professional exam tests your hands on experience.

What happens if I don't pass the practice exams?
Try to score close to 70% in any practice exam in your first attempt. Don't be worried if you don't pass on your first attempt.

What do I do if I discover that I still have some weak areas after doing the practice exams?
During the practice exams, if you find any areas that you seem to be weak in, then read more about those particular services in the AWS documentation.

What tools do I get to use during the exam at the test centre?
I was given a pen and erasable sheet.

Where should I take the exam - online or at the test centre?
You can take the exam anywhere but my preference is taking the test at a test center.

Note: I haven't included white-papers, FAQ's and workshops in this guide as I completed them during my preparation for the Associate exam.

3. AWS DevOps Engineer Professional Notes

kasukur / AWS_CCP_Notes

AWS Certification Notes

AWS Certification Preparation Notes

AWS Certified Cloud Practioner Notes

AWS Solution Architect Associate Notes

AWS Certified Developer Associate Notes

AWS Solution Architect Professional Notes

AWS DevOps Engineer Professional Notes

AWS Workshops/Labs

More to follow...

Disclaimer:

The information has been taken from the content in acloudguru for AWS Solution Architect Associate.

I highly recommend you to purchase the course from acloudguru

The purpose of these notes is to help students with their revision prior to taking the certification.

View on GitHub

4. Exam Tips

During the exam, I marked the questions I was unsure of for review. At the end of the exam, I had about 20 questions for review, which made me feel slightly confident that I might pass the exam.

Make sure that you don't leave any questions unanswered as there is no negative marking.

It is also very important to know the basics, the exam will test you on the basics as well.

DO NOT attempt to answer the questions based on key words alone, ensure you read all the answers as well.
For example: Whenever we see cost, we tend to think of 'Reserved' but that may not always be the case for some scenarios.
You will notice this when you do the live practice exams.

If English is your second language, you can request for ESL+30 mins before booking the exam from certmetrics > Request Exam Accomodations and wait for the approval. It might take a couple of days for the approval to come through.

5. More AWS Labs

AWS Labs/Workshops

This is a list of labs which you can use for further practice, using your own AWS account. This is optional as some of these labs are quite advanced.

Auto Scale EC2 using SQS

Sri — Sat, 14 Jan 2023 21:00:00 +0000

Introduction
Create an EC2 Instance Role
Create a Key pair
Create a Security Group
Create an S3 Bucket
Create an SQS queue
Create a Cloud9 instance
Create CloudWatch Alarms
Create a Launch Template
Create an Auto Scaling Group
Verification and Monitoring
Clean Up
Summary
Referrals

Introduction

In this blog we are going to set up auto scaling of EC2 instances using the SQS ApproximateNumberOfMessagesVisible metric.

AWS Auto Scaling monitors your applications and automatically adjusts capacity to maintain steady, predictable performance at the lowest possible cost. Using AWS Auto Scaling, it’s easy to set up application scaling for multiple resources across multiple services in minutes.

Benefits

Set up Scaling quickly.
Make SMART scaling decisions.
Automatically maintain performance.
Pay only for what you need.

Demo

Let's get started with the demo.

Step 1. Create an EC2 Instance Role

Navigate to IAM > Roles > Click on Create role.
Select EC2 under Common use case and Click Next.
Select AmazonS3ReadOnlyAccess and AmazonSQSFullAccess and Click Next.
Enter Role name as EC2InstanceRoleForSQS and Click Create role.

Note: you may create a custom policy for SQS with the required permissions instead of using the managed policy AmazonSQSFullAccess to provide the least amount of privileges.

AmazonSQSFullAccess

{
"Version": "2012-10-17",
"Statement": [
    {
        "Action": [
            "sqs:*"
        ],
        "Effect": "Allow",
        "Resource": "*"
    }
]
}

AmazonS3ReadOnlyAccess

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:List*",
                "s3-object-lambda:Get*",
                "s3-object-lambda:List*"
            ],
            "Resource": "*"
        }
    ]
}

Trusted entities

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "ec2.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Step 2. Create a key pair

Navigate to EC2 > Key Pairs (under Network & Security).
Click Create key pair.
Enter Name as auto-scale.
Navigate to the folder where the key pair is downloaded and run.

   chmod 400 <key_pair_name>.pem

Auto_Scale_Keypair

Step 3. Create a Security Group

We are going to create a Security Group for SSH.
Navigate to EC2 > Security Groups > Create a new security group for your ALB, and set the following values:
- Name: MyIPSSH-SG.
- Add an Inbound rule to allow SSH (TCP 22) traffic from My IP.
Click Create security group.

MyIPSSH-SG

Step 4. Create an S3 Bucket

Navigate to EC2 > Click Create Bucket
Enter the Bucket name as autoscalescripts, add some random numbers to make the bucket name unique.
Leave the rest of the settings as default and click Create Bucket
Download sendMessages.sh and receiveMessages.sh from the github and upload them to the S3 bucket.

Step 5. Create an SQS queue

Navigate to Simple Queue Service and click Create queue
Enter Name as MyMessages, leave the rest as defaults and click Create queue

SQS_1

SQS_2

SQS_3

SQS_4

Step 6. Create a Cloud9 instance

Navigate to Cloud9 > Create Environment.
Enter Name as awscli and click Create.
Click Open under AWS Cloud9 > Environments.
When Cloud9 is ready, Click File > Upload Local Files... and then upload sendMessages.sh.

Let's check the AWS CLI version.

Sri:~/environment $ aws --version
aws-cli/1.19.112 Python/2.7.18 Linux/4.14.301-224.520.amzn2.x86_64 botocore/1.20.112

The script uses jq, so we need to install jq.
Open a Terminal and execute sudo yum install jq -y.
Execute chmod +x sendMessages.sh to provide the execute permissions.
Now run the script to send 2000 messages to MyMessages queue.

Note: Alternatively you could use AWS CLI on your personal computer to send the messages.

Step 7. Create CloudWatch Alarms

We are going to create a ScaleOut Alarm to launch new instances when the ApproximateNumberOfMessagesVisible is greater than 500.

Navigate to CloudWatch > Alarms > Click Create alarm.
Click Select Metric, search for SQS, select SQS > Queue Metrics.
Select MyMessages > ApproximateNumberOfMessagesVisible and Click Select Metric.
Change Statistic to Sum, Period to 1 minute.
- Threshold type: Static.
- Whenever ApproximateNumberOfMessagesVisible is...: Greater.
- Define the threshold value: 500.
Click Next.
Click Remove under Notification and Click Next.
Enter Alarm name as ScaleOut and Click Next.
Click Create Alarm.

Similar to ScaleOut, we also need to create a ScaleIn Alarm to launch new instances when the ApproximateNumberOfMessagesVisible is lesser than 300.

Select ScaleOut from CloudWatch > Alarms, Click on Actions and Copy.
Change the following:
- Whenever ApproximateNumberOfMessagesVisible is...: Lower.
- Define the threshold value: 300.
Click Next.
Click Remove under Notification and Click Next.
Enter Alarm name as ScaleIn and Click Next.
Click Create Alarm.

👉 It is important to not have the same value for scale-in and scale-out thresholds. We should leave a gap between them to prevent oscillation.
For example: let's say we have 3 instances, and the CPU goes to 60%, which triggers the +1 step scaling policy. If the load stays constant, it will now be distributed to all 4 instances and the average CPU will drop to around 45% and the scale-in alarm will go off. This will then keep happening in a loop until the load goes up or down sufficiently for one of the alarms to stay in the alarm state and the ASG to reache the min or max.

ScaleOut_CreateAlarm1

ScaleOut_CreateAlarm2

ScaleOut_CreateAlarm3

ScaleOut_CreateAlarm4

ScaleOut_CreateAlarm5

ScaleOut_CreateAlarm6

ScaleOut_CreateAlarm7

ScaleOut_CreateAlarm7.1

ScaleIn_CreateAlarm1

ScaleIn_CreateAlarm2

ScaleIn_CreateAlarm3

ScaleIn_CreateAlarm4

ScaleIn_CreateAlarm4.1

Step 8. Create a Launch Template

We can use a Launch Template or Launch Configurations. Launch Template is preferred over Launch Configurations as we can have different versions of the template. Also we can't modify a Launch Configuration after we have created it.

Create a Launch Template that will be used by the Auto Scaling group. The Launch Template defines what the instances are and how they are created.

Navigate to EC2 > Instances > Launch Templates.
Create a new template, and name it AutoScale-SQS.
Search for AMI, and pick the Amazon Linux.
Set the instance type as t2.micro.
Select key pair you created earlier.
Select the MyIPSSH-SG security group you created earlier.
Expand Advanced Details, and select EC2InstanceRoleForSQS Role under IAM instance profile.
Paste the following script under User data.
- Note: These are commands to install jq, AWS CLI, copy scripts from the S3 bucket and to execute receiveMessages.sh.
Click Create Launch Template.
Click View Launch templates.

User data

Note: Please update the bucket name in the script.

#!/bin/bash
# install jq
sudo yum install jq -y
# Update aws cli version
cd /home/ec2-user
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
sudo ./aws/install --bin-dir /usr/local/bin --install-dir /usr/local/aws-cli --update
which aws
ls -l /usr/local/bin/aws
aws --version
# copy sendMessages.sh and receiveMessages.sh from S3
aws s3 cp s3://sqssri/ . --recursive --exclude "*" --include "*.sh"
sudo chmod +x /home/ec2-user/*.sh
nohup ./receiveMessages.sh &

👉 As Launch Templates with User data are slow, it is recommended to create an AMI with the required software to improve the speed of instance initialisation.

LaunchTemplate1

LaunchTemplate1.1

LaunchTemplate1.2

LaunchTemplate1.3

LaunchTemplate1.4

Step 9. Create an Auto Scaling Group

Navigate to EC2 > Auto Scaling > Auto Scaling Groups
Click Create Auto Scaling group.
Call the group ASG-SQS.
Select Launch Template, and choose the template named AutoScale-SQS.
We are using default VPC, which will be selected, so select us-east-1a as subnet.
Click Next.
Leave the default for Health checks, which is EC2.
Leave the default for Configure advanced options and Click Next.
For Group Size, enter the following values:
- Desired Capacity: 1
- Minimum Capacity: 1
- Maximum Capacity: 4
We will not be adding Scaling policies here, so leave the default None and Click Next.
Click Next at Add Notifications.
Click Next at Add tags.
Click Create Auto Scaling Group.
Navigate to EC2 > Auto Scaling > click ASG-SQS and then click Automatic scaling
We are going to add two dynamic scaling policies, one for scale out and another one for scale in. Click Create dynamic scaling policy and enter the following values and then Click Create:
- Policy type: Simple Scaling
- Scaling policy name: ScaleOut
- CloudWatch alarm: ScaleOut
- Take the action: Add with 1 capacity units
- And then wait 60 seconds before allowing another scaling activity
Click Create dynamic scaling policy and enter the following values and then Click Create:
- Policy type: Simple Scaling
- Scaling policy name: ScaleIn
- CloudWatch alarm: ScaleIn
- Take the action: Remove with 1 capacity units
- And then wait 120 seconds before allowing another scaling activity

👉 It is best practice to scale up fast and scale down slow. Hence we used 60 seconds to scale out and 120 seconds to scale in.

ASG1

ASG2

ASG3

ASG4

ASG5

ASG6

ASG7

ASG7.1

ASG7.2

ASG8

ASG8.1_ScaleOut

ASG8.2_ScaleIn

ASG8.3_Dynamic_Scaling

Step 10. Verification and Monitoring

We have populated the queue with 2000 messages and have an auto scaling group launch with a maximum of 4 instances. We will now be verifying it using Cloud Watch Alarms and Auto Scaling group's Activity.

Navigate to CloudWatch > Alarms > ScaleOut, the state of ScaleOut Alarm status will be In alarm and the ScaleIn Alarm status will be OK.
After a period of 10 mins or so, you will notice 4 EC2 instances launched under EC2 > Auto Scaling groups > ASG-SQS > Activity.
Monitor the messages in MyMessages queue, the number of messages will reduce as they are processed by the EC2 instances.
Navigate to CloudWatch > Alarms > ScaleIn, the state of ScaleIn Alarm status will be In alarm and the ScaleOut Alarm status will be OK.
You will notice 3 EC2 instances terminated under EC2 > Auto Scaling groups > ASG-SQS > Activity when the messages are less than 300.

How to verify that the SQS messages are being processed

Log on to the EC2 Instance and then switch to the root user.

[ec2-user@ip-172-31-47-21 ~]$ sudo su -

Execute the following command, which will show the log file name.

ps xf

[root@ip-172-31-47-21 ~]# tail -f /var/log/cloud-init-output.log
Sleep for 1 second...
Sleep for 1 second...
Sleep for 1 second...
Sleep for 1 second...
^C

Messages_in_the_queue

ScaleOut_Alarm

ScaleOut_Activity

Processing_Messages

ScaleIn_Alarm

ScaleIn_Activity

Queue_is_Empty

👉 Auto Scaling Group Tip: When you do not want to have instances running or for Disaster Recovery purposes or to save costs, you may set the following inputs to Zero.

Desired Capacity: 0
Minimum Capacity: 0
Maximum Capacity: 0

ASG_Tip

Clean Up

Terminate Cloud9 EC2 instance.
Delete ASG-SQS under Auto Scaling groups.
Delete AutoScale-SQS under Launch Templates
Delete auto-scale under Key Pairs
Delete Security Group MyIPSSH-SG.
Delete EC2InstanceRoleForSQS Role under IAM.
Delete S3 bucket autoscalescripts.
Delete SQS queue MyMessages.

Summary

👉 It is important to not have the same value for scale-in and scale-out thresholds. We should leave a gap between them to prevent oscillation.
👉 The best practice is to scale up fast, and scale down slow.
👉 Launching an EC2 instance might be slow if we have to install software, configure..etc during the scale out. One way to speed up the process is by creating an AMI with all the required software and then use that AMI in the Launch template.

Hope you learnt something new from the above demo.

See you next time 👋

Referrals

Scaling based on Amazon SQS

AWS Simple Queue Service (SQS) 101 with a demo using AWS CLI

Sri — Sat, 07 Jan 2023 22:00:00 +0000

Introduction
Standard vs FIFO Queue
Queue Configuration
Create a SQS queue
Create a Cloud9 instance
Challenge
Summary
Referrals

Introduction

In this blog we are going to create a SQS queue using the AWS Console and then interact with it using AWS CLI, bash and jq.

What is SQS?

Amazon Simple Queue Service (SQS) lets you send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available.

Some of the use cases of SQS

Increase application reliability and scale.
Decouple microservices and process event-driven applications.
Ensure work is completed cost-effectively and on time.
Maintain message ordering with deduplication

Standard vs FIFO Queue

SQS offers two types of queues – Standard & FIFO queues
Some of the major differences between the standard and the FIFO queues are:

https://dev.to/aws-builders/sagemaker-and-amazon-stack-2jaa

Message Order

Standard queues provide best-effort ordering which ensures that messages are generally delivered in the same order as they are sent. Occasionally (because of the highly-distributed architecture that allows high throughput), more than one copy of a message might be delivered out of order.
FIFO queues offer first-in-first-out delivery and exactly-once processing: the order in which messages are sent and received is strictly preserved.

Delivery

Standard queues guarantees that a message is delivered at least once and duplicates can be introduced into the queue.
FIFO queues ensure a message is delivered exactly once and remains available until a consumer processes and deletes it. The duplicates are not introduced into the queue.

Transactions Per Second (TPS)

Standard queues allow nearly-unlimited number of transactions per second.
FIFO queues are limited to 300 transactions per second per API action. However It can be increased to 3000 by using batching.

Queue Configuration

Note: The following is from AWS Console

Visibility timeout: Should be between 0 seconds and 12 hours (default: 30 seconds).

Visibility timeout sets the length of time that a message received from a queue (by one consumer) will not be visible to the other message consumers.
The visibility timeout begins when Amazon SQS returns a message. If the consumer fails to process and delete the message before the visibility timeout expires, the message becomes visible to other consumers. If a message must be received only once, your consumer must delete it within the duration of the visibility timeout.
The default visibility timeout setting is 30 seconds. This setting applies to all messages in the queue. Typically, you should set the visibility timeout to the maximum time that it takes your application to process and delete a message from the queue.

Message retention period: Should be between 1 minute and 14 days (default: 4 days).

The message retention period is the amount of time that Amazon SQS retains a message that does not get deleted. Amazon SQS automatically deletes messages that have been in a queue for more than the maximum message retention period. The default retention period is 4 days. The retention period has a range of 60 seconds to 1,209,600 seconds (14 days).
The expiration of a message is always based on its original enqueue timestamp. When a message is moved to a dead-letter queue, the enqueue timestamp remains unchanged. For example, if a message spends 1 day in the original queue before being moved to a dead-letter queue, and the retention period of the dead-letter queue is set to 4 days, the message is deleted from the dead-letter queue after 3 days. For this reason, we recommend that you always set the retention period of a dead-letter queue to be longer than the retention period of the original queue.

Delivery delay: Should be between 0 seconds and 15 minutes (default: 0 seconds).

If your consumers need additional time to process messages, you can delay each new message coming to the queue. The delivery delay is the amount of time to delay the first delivery of each message added to the queue. Any messages that you send to the queue remain invisible to consumers for the duration of the delay period. The default (minimum) delay for a queue is 0 seconds. The maximum is 15 minutes.
For standard queues, the per-queue delay setting is not retroactive; changing the setting doesn't affect the delay of messages already in the queue.
For FIFO queues, the per-queue delay setting is retroactive; changing the setting affects the delay of messages already in the queue.

Maximum message size: Should be between 1 KB and 256 KB (default: 256 KB).

You can set the maximum message size for your queue. The smallest supported message size is 1 byte (1 character). The largest size is 262,144 bytes (256 KB). To send messages larger than 256 KB, you can use the Amazon SQS Extended Client Library for Java (https://github.com/awslabs/amazon-sqs-java-extended-client-lib). This library allows you to send an Amazon SQS message that contains a reference to a message payload in Amazon S3. The maximum payload size is 2 GB.

Receive message wait time: Should be between 0 and 20 seconds (default: 0 seconds).

The receive message wait time is the maximum amount of time that polling will wait for messages to become available to receive. The minimum value is zero seconds and the maximum value is 20 seconds.
Long polling helps reduce the cost of using Amazon SQS by eliminating the number of empty responses (when there are no messages available for a ReceiveMessage request) and false empty responses (when messages are available but aren't included in a response). If a receive request collects the maximum number of messages, it returns immediately. It does not wait for the polling to time out.
If you set the receive message wait time to zero, the receive requests use short polling.

Demo

Let's get started with the demo.

Create a SQS queue

Navigate to Simple Queue Service and click Create queue
Enter Name as MyMessages, leave the rest as defaults and click Create queue

SQS_1

SQS_2

SQS_3

SQS_4

Cloud9

Navigate to Cloud9 > Create Environment
Enter Name as SQS and click Create
Click Open under AWS Cloud9 > Environments

Let's check aws cli version.

Sri:~/environment $ aws --version
aws-cli/1.19.112 Python/2.7.18 Linux/4.14.301-224.520.amzn2.x86_64 botocore/1.20.112

jq is not installed by default on Cloud9.

Sri:~/environment $ jq
bash: jq: command not found

Install jq by executing the following command.

sudo yum install jq -y

How to check if jq is installed.

Sri:~/environment $ rpm -qa|grep jq
jq-1.5-1.amzn2.0.2.x86_64

jq -help

Sri:~/environment $ jq
jq - commandline JSON processor [version 1.5]
Usage: jq [options] <jq filter> [file...]

        jq is a tool for processing JSON inputs, applying the
        given filter to its JSON text inputs and producing the
        filter's results as JSON on standard output.
        The simplest filter is ., which is the identity filter,
        copying jq's input to its output unmodified (except for
        formatting).
        For more advanced filters see the jq(1) manpage ("man jq")
        and/or https://stedolan.github.io/jq

        Some of the options include:
         -c             compact instead of pretty-printed output;
         -n             use `null` as the single input value;
         -e             set the exit status code based on the output;
         -s             read (slurp) all inputs into an array; apply filter to it;
         -r             output raw strings, not JSON texts;
         -R             read raw strings, not JSON texts;
         -C             colorize JSON;
         -M             monochrome (don't colorize JSON);
         -S             sort keys of objects on output;
         --tab  use tabs for indentation;
         --arg a v      set variable $a to value <v>;
         --argjson a v  set variable $a to JSON value <v>;
         --slurpfile a f        set variable $a to an array of JSON texts read from <f>;
        See the manpage for more options.

Send a SQS message using AWS CLI

In order to send a message to SQS, we need queue-url.
We could either hard code the queue-url or retreive the queue-url and then pass it to send-message, receive-message or delete-message

# queueurl is required to send or receive messages from SQS

# Note: AWS CLI and boto3 need to use legacy endpoint
# If we use the AWS CLI or SDK for Python, we need to use the legacy endpoints from https://docs.aws.amazon.com/general/latest/gr/sqs-service.html.
# Otherwise, you will get the following error
# botocore.exceptions.ClientError: An error occurred (InvalidAddress) when calling the ReceiveMessage operation: The address https://eu-central-1.queue.amazonaws.com/ is not valid for this endpoint.

export queueurl=$(aws sqs get-queue-url --queue-name "MyMessages"|jq ".QueueUrl"|sed "s/sqs.us-east-1.amazonaws.com/queue.amazonaws.com/g"|sed 's/"//g')
echo "queueurl:" $queueurl

Sri:~/environment $ aws sqs send-message --queue-url $queueurl --message-body Message --region us-east-1
{
    "MD5OfMessageBody": "4c2a8fe7eaf24721cc7a9f0175115bd4", 
    "MessageId": "614ec45a-82b8-4788-aed6-6f3d80051780"
}

Sri:~/environment $ aws sqs receive-message --queue-url $queueurl
{
    "Messages": [
        {
            "Body": "Message", 
            "ReceiptHandle": "AQEBxxUZvU5S1OP2VFBIY5bpg267ZQvSkX3n7jfV505VumHa2JCHo5BR/trRSCKOlhcLSyvJzBUBhq9TPHlQ4F4nQu839HruurRMMYCaaZBiiEdFnADm3EwEi8+Hj9YsmmGf0YL4ftlTyMHnJFjL9l06e7qZZyV8lWKv+q/mnEU8v69AumtvcS7OhmgAtUzQUqG+4/enJgt3WrN1eJiJRlmiUbDpV3VCGr7wQbU5C+GAmYjVNs+vNrMSzOceKFs3XQH6vwhNnnp1d3dJZzn+JEjpfx8g6INrq1LYShgg6cTUVygc2jXuQNBoSHP4Jx6szscdLm2P4gUD515UJ948nX/Zlsu90GZtfyGY5S4wzgl9oYpJcPLRV7z6ISnte0WKuK9cHPCEzmyGeERS1yC7YKXRNA==", 
            "MD5OfBody": "4c2a8fe7eaf24721cc7a9f0175115bd4", 
            "MessageId": "614ec45a-82b8-4788-aed6-6f3d80051780"
        }
    ]
}

👉 If you run the command again with in the Visibility timeout of 30 secs, SQS will not send the message again.
However if the message is not processed and deleted by the consumer, the message becomes visible to other consumers. If a message must be received only once, the consumer must delete it within the duration of the visibility timeout.

Sri:~/environment $ aws sqs receive-message --queue-url $queueurl
Sri:~/environment $

So in order to delete the message, we need to retreive the receipt-handle and then pass the --receipt-handle to sqs delete-message
Since the output message is a JSON message and we only need receipt-handle, we can use jq, which is like sed for JSON data - you can use it to slice, filter, map and transform structured data in the same way that you would with sed, awk, grep and other linux tools.

What does get-queue-url do?
It retreives QueueUrl and then replaces the endpoint with a legacy endpoint and also removes double quotes from the QueueUrl.

The following code snippet retrieves and deletes the message.

export queueurl=$(aws sqs get-queue-url --queue-name "MyMessages"|jq ".QueueUrl"|sed "s/sqs.us-east-1.amazonaws.com/queue.amazonaws.com/g"|sed 's/"//g')
echo "queueurl:" $queueurl

# receiptHandle is required to delete the message from SQS after the message is retreived
export receiptHandle=$(aws sqs receive-message --queue-url $queueurl|jq .Messages[0].ReceiptHandle|sed 's/"//g')
echo "receiptHandle:" $receiptHandle

#echo -e "aws sqs delete-message --queue-url $queueurl --receipt-handle $receiptHandle --region us-east-1"
aws sqs delete-message --queue-url $queueurl --receipt-handle $receiptHandle --region us-east-1;

Let's send a 100 messages to the SQS queue, retrieve them and then delete the messages using bash scripts.
Open up two terminal windows on Cloud9 and use one to execute ./sendMessages.sh and the other for ./receiveMessages.sh.

sendMessages.sh

#!/bin/bash

# queueurl is required to send or receive messages from SQS

# Note: AWS CLI and boto3 need to use legacy endpoint
# If we use the AWS CLI or SDK for Python, we need to use the legacy endpoints from https://docs.aws.amazon.com/general/latest/gr/sqs-service.html.
# Otherwise, you will get the following error
# botocore.exceptions.ClientError: An error occurred (InvalidAddress) when calling the ReceiveMessage operation: The address https://eu-central-1.queue.amazonaws.com/ is not valid for this endpoint.

queueurl=$(aws sqs get-queue-url --queue-name "MyMessages"|jq ".QueueUrl"|sed "s/sqs.us-east-1.amazonaws.com/queue.amazonaws.com/g"|sed 's/"//g')
echo "queueurl:" $queueurl

echo "aws sqs send-message --queue-url $queueurl --message-body Message --region us-east-1"
for i in {1..100} ; do aws sqs send-message --queue-url $queueurl --message-body "Message-$i" --region us-east-1 ; done;

receiveMessages.sh

#!/bin/bash

# queueurl is required to send or receive messages from SQS

# Note: AWS CLI and boto3 need to use legacy endpoint
# If we use the AWS CLI or SDK for Python, we need to use the legacy endpoints from https://docs.aws.amazon.com/general/latest/gr/sqs-service.html.
# Otherwise, you will get the following error
# botocore.exceptions.ClientError: An error occurred (InvalidAddress) when calling the ReceiveMessage operation: The address https://eu-central-1.queue.amazonaws.com/ is not valid for this endpoint.

queueurl=$(aws sqs get-queue-url --queue-name "MyMessages"|jq ".QueueUrl"|sed "s/sqs.us-east-1.amazonaws.com/queue.amazonaws.com/g"|sed 's/"//g')
echo "queueurl:" $queueurl

for i in {1..100};
  do
        # receiptHandle is required to delete the message from SQS after the message is retreived
        receiptHandle=$(aws sqs receive-message --queue-url $queueurl|jq .Messages[0].ReceiptHandle|sed 's/"//g')
        #echo "receiptHandle:" $receiptHandle

        #echo -e "aws sqs delete-message --queue-url $queueurl --receipt-handle $receiptHandle --region us-east-1"
        aws sqs delete-message --queue-url $queueurl --receipt-handle $receiptHandle --region us-east-1;

        echo "Sleep for 1 second..."
        sleep 1
done;

Challenge

There are some improvements which we can make to the above. I will leave this as a challenge for now.
Please let me know in the comments if you have managed to solve it.
I will provide an update in a comment at some point.

We can use --max-number-of-messages (integer).

The maximum number of messages to return. Amazon SQS never returns more messages than this value (however, fewer messages might be returned). Valid values: 1 to 10. Default: 1.
jq query in the command also need to be updated to retrieve an array of --receipt-handle.
bash script also need to store the returned array and loop through the array to delete all the messages returned by --max-number-of-messages.

Clean Up

Delete the MyMessages Queue.
Terminate Cloud9 EC2 instance.

Summary

We learned about SQS.
We also learned about bash scripting and jq.

See you next time 👋

Referrals

How to set up Jenkins and a Pipeline on AWS

Sri — Sat, 31 Dec 2022 23:00:00 +0000

Introduction
Create an IAM User
Create a Key pair
Create a Security Group
Create an EC2 instance
Install and Configure Jenkins
Create a Pipeline
Clean Up
Summary
Referrals

Introduction

In this blog we are going to set up Jenkins on an EC2 instance and then set up a pipeline to copy a file from S3.
We are going to start something very basic and enhance it as we move along.

What is Jenkins?

Jenkins offers a simple way to set up a continuous integration or continuous delivery (CI/CD) environment for almost any combination of languages and source code repositories using pipelines, as well as automating other routine development tasks. While Jenkins doesn’t eliminate the need to create scripts for individual steps, it does give you a faster and more robust way to integrate your entire chain of build, test, and deployment tools than you can easily build yourself.

What is Jenkins Pipeline?

Jenkins Pipeline is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins.

Pipeline provides an extensible set of tools for modeling simple-to-complex delivery pipelines as code via the Pipeline domain-specific language (DSL) syntax.

Demo

Let's get started with the demo.

Step 1. Create an IAM User

Navigate to IAM > Users > Click on Add users
Enter User Name as JenkinsUser
Select Access key - Programmatic access and Click Next: Permissions
Click Attach existing policies directly and then Create policy
In the Create policy window, Click on JSON and add the following JenkinsEC2Policy and save the policy as JenkinsEC2Policy
Select AmazonS3ReadOnlyAccess and JenkinsEC2Policy under Attach existing policies directly
Click Next:Tags, Next:Review and then Create user

Download the credentials .csv file as we will need this during Jenkins configuration at Step 6.

JenkinsEC2Policy

{
"Version": "2012-10-17",
"Statement": [
    {
        "Sid": "Stmt1312295543082",
        "Action": [
            "ec2:DescribeSpotInstanceRequests",
            "ec2:CancelSpotInstanceRequests",
            "ec2:GetConsoleOutput",
            "ec2:RequestSpotInstances",
            "ec2:RunInstances",
            "ec2:StartInstances",
            "ec2:StopInstances",
            "ec2:TerminateInstances",
            "ec2:CreateTags",
            "ec2:DeleteTags",
            "ec2:DescribeInstances",
            "ec2:DescribeInstanceTypes",
            "ec2:DescribeKeyPairs",
            "ec2:DescribeRegions",
            "ec2:DescribeImages",
            "ec2:DescribeAvailabilityZones",
            "ec2:DescribeSecurityGroups",
            "ec2:DescribeSubnets",
            "iam:ListInstanceProfilesForRole",
            "iam:PassRole",
            "ec2:GetPasswordData"
        ],
        "Effect": "Allow",
        "Resource": "*"
    }
]
}

IAM_JenkinsUser1

IAM_JenkinsUser2

IAM_JenkinsUser3

Step 2. Create a key pair

Navigate to EC2 > Key Pairs (under Network & Security).
Click Create key pair.
Navigate to the folder where the key pair is downloaded and run.

   chmod 400 <key_pair_name>.pem

Jenkins_Keypair

Step 3. Create a Security Group

We are going to create a Security Group for SSH and Jenkins web access.
Navigate to EC2 > Security Groups > Create a new security group for your ALB, and set the following values:
- Name: JenkinsSG.
- Add an Inbound rule to allow SSH (TCP 22) traffic from My IP.
- Add another Inbound rule to allow Custom (TCP 8080) traffic from My IP.

Jenkins_SG

Step 4. Create an EC2 instance

Navigate to EC2 > EC2 Dashboard > Click on Launch instance.
Launch an instance with the following values as shown in the screenshots.
When the instance state is Running, select the instance and click on Connect and then copy the connections details.

Connect to the instance.

ssh -i "Jenkins-Keypair.pem" ec2-user@ec2-54-211-70-130.compute-1.amazonaws.com

Jenkins_EC1

Jenkins_EC2

Step 5. Install and Configure Jenkins

Install Jenkins

Ensure that the software packages are up to date on the instance by executing the following command:
```
[ec2-user ~]$ sudo yum update –y
```

Add the Jenkins repo using the following command:

[ec2-user ~]$ sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo

Import a key file from Jenkins-CI to enable installation from the package:

[ec2-user ~]$ sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key

[ec2-user ~]$ sudo yum upgrade

Install Java:

[ec2-user ~]$ sudo amazon-linux-extras install java-openjdk11 -y

Install Jenkins:

[ec2-user ~]$ sudo yum install jenkins -y

Enable Jenkins service to auto start at boot:

[ec2-user ~]$ sudo systemctl enable jenkins

Start Jenkins as a service:

[ec2-user ~]$ sudo systemctl start jenkins

You can check the status of the Jenkins service using the command:
```
[ec2-user ~]$ sudo systemctl status jenkins
```
Just in case we ever need to restart Jenkins during this setup/configuration.
```
[ec2-user ~]$ sudo systemctl restart jenkins
```

Configure Jenkins

Copy the Public IPv4 DNS of EC2 instance and paste the URL as following in the browser For Example: http://ec2-54-211-70-130.compute-1.amazonaws.com:8080

Enter the initialAdminPassword from /var/lib/jenkins/secrets/initialAdminPassword and Click Continue

[ec2-user@ip-172-31-89-157 ~]$ sudo cat /var/lib/jenkins/secrets/initialAdminPassword
    671x0x5x3xxx46xxxxxx099x1xf0149x

Select Install suggested plugins.
Once the installation is complete, Create First Admin User will open. Enter your information, and then select Save and Continue.
Click Dashboard, select Manage Jenkins, and then select Manage Plugins.
Click Available plugins, Search and Select Amazon EC2 and then **Install without restart*
Once the installation is complete, Navigate back to Dashboard, select Manage Jenkins, select Manage nodes and clouds and then click on Configure Clouds.
Select Add a new cloud, and select Amazon EC2. A new pop up window opens with more fields.
Click Add under Amazon EC2 Credentials and Select Jenkins
- From the Jenkins Credentials Provider: Jenkins, select AWS Credentials as the Kind.
- Enter Access Key ID, Secret Access Key from the key pair and Click Add.
Scroll down to Region and select your Region.
Click Add under EC2 Key Pair's Private Key and Select Jenkins.
- From the Jenkins Credentials Provider: Jenkins, select SSH Username with private key as the Kind and set the Username to ec2-user.
- Select Enter Directly under Private Key, then select Add.
- Open the private key pair you created in the creating a key pair step and paste in the contents from -----BEGIN RSA PRIVATE KEY----- to -----END RSA PRIVATE KEY-----. Select Add when completed.
Scroll down to Test Connection and ensure it states Success and then Click Save.

Jenkins_Configure1

Jenkins_Configure2

Jenkins_Configure3

Jenkins_Configure4

Jenkins_Configure5

Jenkins_Configure6

Jenkins_Configure7

Jenkins_Configure8

Jenkins_Configure9

Jenkins_Configure10

Jenkins_Configure10.1

Jenkins_Configure10.2

Jenkins_Configure11

Jenkins_Configure11.1

Jenkins_Configure11.2

Jenkins_Configure12

Step 6. Create a Pipeline

Navigate to Dashboard > Manage Jenkins and copy the ID of IAM User created in Step 1, which we need to replace xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx with the ID.
Navigate to Dashboard > Manage Jenkins > Plugin Manager > Click on Available plugins
Search for AWS Steps and Install without restart.
Navigate to Dashboard > New Job > Enter download-a-file-from-s3 > Select Pipeline and Click OK.
Scroll down to Pipeline, add the following Pipeline script and Click Save
Create an S3 bucket and copy some files to the bucket.
Replace the s3bucket with your bucket name and filename with one of the files in your S3 bucket.
Navigate to Dashboard > download-a-file-from-s3 and Click on Build Now.

Navigate to the latest Build History link and check out the Console Output.

pipeline 
{
agent any
    stages
    {
    stage('S3download') 
        {      
            steps {
                withAWS(region:'us-east-1',credentials:'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx')\
                {
                    s3Download(file: "filename", bucket: 's3bucket', path: '')
                }
            }
        }
    }
}

Jenkins_Pipeline1

Jenkins_Pipeline2

Jenkins_Pipeline3

Jenkins_Pipeline4

The following is the console log, which shows the pipeline has been successful. ✅

Started by user Sri
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/download-a-file-from-s3
[Pipeline] {
[Pipeline] stage
[Pipeline] { (S3download)
[Pipeline] withAWS
Constructing AWS CredentialsSetting AWS region us-east-1 
[Pipeline] {
[Pipeline] s3Download
Downloading s3://s3bucket/ to file:/var/lib/jenkins/workspace/download-a-file-from-s3/receiveMessages.sh 
Finished: Downloading from s3bucket/
Download complete
[Pipeline] }
[Pipeline] // withAWS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

The file that was copied from S3 to the EC2 instance. ✅

[ec2-user@ip-172-31-89-157 ~]$ ls -lrt /var/lib/jenkins/workspace/download-a-file-from-s3/
total 0
drwxr-xr-x 2 jenkins jenkins 55 Dec 31 03:45 receiveMessages.sh

Let's run the build again, this time the build has failed with the following error ❓
The error message suggests to use set force=true.

Started by user Sri
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/download-a-file-from-s3
[Pipeline] {
[Pipeline] stage
[Pipeline] { (S3download)
[Pipeline] withAWS
Constructing AWS CredentialsSetting AWS region us-east-1 
[Pipeline] {
[Pipeline] s3Download
Downloading s3://s3bucket/ to file:/var/lib/jenkins/workspace/download-a-file-from-s3/receiveMessages.sh/ 
Download failed due to existing target file; set force=true to overwrite target file
[Pipeline] }
[Pipeline] // withAWS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
java.lang.RuntimeException: Target exists: file:/var/lib/jenkins/workspace/download-a-file-from-s3/receiveMessages.sh/
    at de.taimos.pipeline.aws.S3DownloadStep$Execution.run(S3DownloadStep.java:146)
    at de.taimos.pipeline.aws.S3DownloadStep$Execution.run(S3DownloadStep.java:113)
    at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Finished: FAILURE

So we need to add force:true as per the Pipeline: AWS Steps documentation

pipeline 
{
agent any
    stages
    {
    stage('S3download') 
        {      
            steps {
                withAWS(region:'us-east-1',credentials:'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx')\
                {
                    s3Download(file: "filename", bucket: 's3bucket', path: '', force:true)
                }
            }
        }
    }
}

The build succeeded after adding force:true and now we can run the build multiple times.

Now let's parameterise the destination foldername.

Navigate to Dashboard > download-a-file-from-s3 and click on Configure
Select This project is parameterised and then Add String parameter
Enter Name as foldername, Default Valu as foldername and Save
Click Build with Parameters
Navigate to the latest Build History link and check out the Console Output.

pipeline 
{
agent any
    stages
    {
    stage('S3download') 
        {      
            steps {
                withAWS(region:'us-east-1',credentials:'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx')\
                {
                    echo "${foldername}" 
                    s3Download(file: "${foldername}", bucket: 's3bucket', path: '', force:true)
                }
            }
        }
    }
}

Jenkins_Pipeline_Parameter

Jenkins_Pipeline_Build-with-Parameters

The following is the console log, which shows the pipeline has been successful. ✅

Started by user Sri
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/download-a-file-from-s3
[Pipeline] {
[Pipeline] stage
[Pipeline] { (S3download)
[Pipeline] withAWS
Constructing AWS CredentialsSetting AWS region us-east-1 
[Pipeline] {
[Pipeline] echo
s3files
[Pipeline] s3Download
Downloading s3://s3bucket/ to file:/var/lib/jenkins/workspace/download-a-file-from-s3/s3files 
Finished: Downloading from sqssri/
Download complete
[Pipeline] }
[Pipeline] // withAWS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

The files were copied to s3files directory on the EC2 instance. ✅

[ec2-user@ip-172-31-89-157 ~]$ ls -lrt /var/lib/jenkins/workspace/download-a-file-from-s3/s3files 
total 8
-rw-r--r-- 1 jenkins jenkins 952 Dec 31 03:40 receiveMessages.sh
-rw-r--r-- 1 jenkins jenkins 646 Dec 31 03:40 sendMessages.sh

Clean Up

Delete the EC2 instance.
Delete the JenkinsUser.
Delete the key pair.
Delete the Security Group JenkinsSG.

Summary

We learned how to install and configure Jenkins.
We also learned about setting up the pipeline.

Referrals

Elastic Beanstalk Demo: Auto Scaling with a shared load balancer

Sri — Sun, 25 Dec 2022 12:42:41 +0000

Introduction
Create a Security Group
Create an Application Load Balancer
Create Elastic Beanstalk Admin App
Create Elastic Beanstalk Forum App
Auto Scaling Verification
Clean Up
Summary
Referrals

Introduction

One of the users of the AWS Certified Global Community had this query which caught my attention and made me want to check it out and see If I could help solve their problem. While doing this, I actually learned about shared load balancer with Elastic Beanstalk application and found it quite interesting, so I would like to share my learning with you.

With Elastic Beanstalk, you can quickly deploy and manage applications in the AWS Cloud without having to learn about the infrastructure that runs those applications. Elastic Beanstalk reduces management complexity without restricting choice or control. You simply upload your application, and Elastic Beanstalk automatically handles the details of capacity provisioning, load balancing, scaling, and application health monitoring. However, it becomes a bit difficult when we need to customise configurations such as Application Load Balancer, Listeners...etc

The user wanted to deploy two Elastic Beanstalk applications, let's say one for an Admin application and another one for Forum application. The problem the user was facing was particularly during auto-scaling - there seemed to be two instances created instead of one when she terminated one instance.

I'd like to share the demo below to show how to configure two different elastic Beanstalk applications using a shared load balancer and verify auto-scaling.

Note: We can only have a maximum of 5 Target Groups per Action per Application Load Balancer.

Demo

Let's get started with the demo.

Step 1. Create a Security Group

We are going to create a Security Group for Elastic Beanstalk
Navigate to EC2 > Security Groups > Create a new security group for your ALB, and set the following values:
- Name: EBSSG
- Add two Inbound rules to allow HTTP traffic from 0.0.0.0/0 (IPV4) and ::/0 (IPV6)
EBS Security Group

Step 2. Create an Application Load Balancer

Navigate to EC2 > Load Balancers.
Click Create Load Balancer.
Click the Create button under the Application Load Balancer and set the following values:
- Name: SharedALB
- Scheme: internet-facing
- IP address type: ipv4
- Load Balancer Protocol: HTTP
- Port: 80
- Leave the default VPC.
- Select us-east-1a and us-east-1b AZs.
Click Next: Configure Security Settings.
- Note: Ignore the warning as we are not using HTTPS.
Select EBSALB and Click Next
Configure Routing, enter the following values and then Click Next:
- Name: EBS-TG
- Target type: Instance
- Protocol: HTTP
- Port: 80
- Path: /index.html
Click Next: Register Targets.
Click Next: Review.
Click Create
Wait until the Application Load Balancer SharedALB is Active.

Shared Application Load Balancer

Configure Security Options

Configure Security Groups

Configure Routing

Register Targets

Review

Application Load Balancer is Active

Initial Listener Rules

Step 3. Create Elastic Beanstalk Admin App

Navigate to Elastic Beanstalk > Environments.
Click Create a new environment.
Select Web server environment
Enter the following values:
- Name: Admin
- Platform: Node.js
- Select Upload your code and upload nodejs_admin.zip
Click Configure more options
- Note: Step 6, 7 and 8 should be done in the same order.
Select High availability under Presets.
Click Edit next to Network and select same two Availability Zones for Load Balancer, Instance settings and Database settings.
Click Edit next to Load Balancer
- Load balancer type: Application Load Balancer
- Type: Shared
- Load balancer ARN: Select the ARN of SharedALB
- Select default under Processes and click Actions > Edit > enter Path as /admin.html and click Save
- Click Add Rule under Rules, Enter Name as admin, PathPattern as /*, Process as default and click Add
Click Save.
Click Create Environment.
Wait for the environment to be create and then click on the application URL.

Note: If you need to make any changes to the code and create a zip on a Mac OS



cd /nodejs_admin
zip ../nodejs_admin -r * .[^.]*

EBS Admin 1

EBS Admin 2.1

EBS Admin 2.2

EBS Admin 3.1

EBS_Admin 3.2

EBS_Admin 3.3

EBS Admin 4.1

EBS Admin 4.2

you might notice this error if you do not select the shared load balancer

Step 4. Create Elastic Beanstalk Forum App

Navigate to Elastic Beanstalk > Environments.
Click Create a new environment.
Select Web server environment
Enter the following values:
- Name: Forum
- Platform: Node.js
- Select Upload your code and upload nodejs_forum.zip
Click Configure more options
- Note: Step 6, 7 and 8 should be done in the same order.
Select High availability under Presets.
Click Edit next to Network and select same two Availability Zones for Load Balancer, Instance settings and Database settings.
Click Edit next to Load Balancer
- Load balancer type: Application Load Balancer
- Type: Shared
- Load balancer ARN: Select the ARN of SharedALB
- Select default under Processes and click Actions > Edit > enter Path as /forum.html and click Save
- Click Add Rule under Rules, Enter Name as admin, PathPattern as /*, Process as default and click Add
Click Save.
Click Create Environment.
Wait for the environment to be create and then click on the application URL.

Note: If you need to make any changes to the code and create a zip on a Mac OS



cd /nodejs_forum
zip ../nodejs_forum -r * .[^.]*

EBS Forum 4.3

Step 5. Auto Scaling Verification

Navigate to EC2 > Instances
Select Admin-env instance
Click Actions > Instance State and Terminate
Wait for a few minutes, the auto scaling group will launch a new instance for Admin application, which can be checked under Auto Scaling Groups > Activity Note: There should only be one instance launched but not two.
Similarly Terminate the Forum-env and there should new instance created in a few minutes.

EBS Environments

EBS Admin Application

Application Load Balancer Listener Rules

EC2 Instances

EBS Admin Auto Scaling Activity

EBS Forum Auto Scaling Activity

EC2 Instances: Only one new Admin instance

EC2 Instances: Forum Instance terminated manually and new instance created by Auto Scaling

EBS Forum Auto Scaling Activity

Clean Up

Select each environment under Elastic Beanstalk > Environments, Select Actions > Terminate Environment.
Select each application under Elastic Beanstalk > Applications, Select Actions > Delete Application.
Delete SharedALB under Load Balancers
Delete Security Group EBSSG.
After both environments are terminated, Empty and Delete S3 bucket prefixed with elasticbeanstalk-us-east-1-...

Summary

We learned how to deploy and configure different settings using Elastic Beanstalk.
We also learned about shared application load balancer, which can save costs.

Referrals

Configuring a shared Application Load Balancer using the Elastic Beanstalk console
The application code is from Tutorials and samples and I have added admin.html and forum.html.

AWS Community Builders – Christmas and New year 2023

Sri — Mon, 19 Dec 2022 13:00:00 +0000

What surprises you most about the community builders program?

I always knew that it was an honour to be selected to the Community Builders program, but what really surprised me were the many benefits of being a part of this program such as access to learning resources, support and most importantly, networking and learning from other community builders.

What’s your background and your experience with AWS?

About two years ago, there was a cloud project at work that I really wanted to be a part of, however I didn't have any cloud skills and therefore couldn't be considered for this role. As AWS was a pioneer as a cloud provider, I decided that I wanted to learn as much I could about AWS.

I started my journey with AWS Certified Cloud Practitioner. On completing this certification, I felt encouraged to try my hand at other certifications. I decided to pursue all the Associate certifications and then continue on to the Solution Architect Professional.

Since I have started my journey, I have grown significantly in my AWS skills. I feel very happy that I have been able to mentor people in the community who are just starting out in the cloud and are looking for support, like I was two years ago. On the work front, I made a presentation to my team about how they could get certified. I am also part of the Cloud Champion Network in my company.

What’s the biggest benefit you see from the program?

The biggest benefit is having access to the Slack channel and to builders worldwide.
The best part about having access to other builders is the promptness in replying to questions we post.

What’s the next swag item that you would like to get?

I would love to request an AWS Skill Builder subscription-based access, if possible.

What are you eating for dinner today? Share the recipe!

Kale salad

Chop up kale and massage it with some salt, lime and apple cider vinegar.
Add some dried cranberries.
Slice a green apple and add it to the salad and enjoy! 🥗

Is there anything else you would like to say about the community builders program in 2022?

It is a one of a kind program and I absolutely love being part of this program.

Challenge: Display EC2 instances from all regions using Python and Boto3

Sri — Sun, 18 Dec 2022 01:38:37 +0000

This was an idea from the AWS Certified Global Community to create challenges by breaking a certain part of code, so the users can learn by fixing the code.

I have written this second challenge for the AWS Certified Global Community and have also used it to mentor some of my mentees.
I thought that it might be a good idea to share this with the broader community as well.

Boto3 is the name of the Python SDK for AWS. It allows you to directly create, update, and delete AWS resources from your Python scripts.

I agree that we have EC2 Global View, which shows what we are trying to achieve. However this is more of a programming challenge on how to code using Python and Boto3.

Let's start the challenge!

Pre-Requisites:

Lambda with Python 3.9
Lambda Execution role with access to EC2 and CloudWatch access (IAM)
It is preferable to have at least one EC2 instance in any of the regions (not mandatory though)

Step 1. Create an IAM Policy:

You could either let Lambda create an execution role and add the AmazonEC2ReadOnlyAccess policy or you could create the following IAM policy and a role.

Note: Replace [Region], [LambdaFunctionName] and [AccountID] in the Policy

IAM Policy with the following access:

AWSLambdaBasicExecution

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": "arn:aws:logs:[Region]:[AccountID]:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "*"
        }
    ]
}

AmazonEC2ReadOnlyAccess

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "ec2:Describe*",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "elasticloadbalancing:Describe*",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "cloudwatch:ListMetrics",
                "cloudwatch:GetMetricStatistics",
                "cloudwatch:Describe*"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "autoscaling:Describe*",
            "Resource": "*"
        }
    ]
}

Step 2. Create a role for Lambda and attach the policy created in Step 1

Trusted entities:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "lambda.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Step 3. Broken Code: Create a Lambda function with the following code and execute the function.

Note: Please increase the Lambda Function Timeout under General Configuration from 3 sec to 2 min or more as required.

import json

ec2list = []

def lambda_handler(event, context):

    # Get list of regions
    ec2 = boto3.client('ec2')
    regions = ec2.describe_regions().get('Regions',[] )

    # Iterate over regions
    for region in regions:

        print ("* Checking region  --   %s " % region['RegionName'])
        reg=region['RegionName']

        client = boto3.client('ec2', region_name=reg)
        response = client.describe_instances()

        for reservation in response["Instances"]:
            for instance in reservation["Instances"]:
                print ("  ---- Instance %s in %s" % (instance['InstanceId'], region['RegionName']))
                ec2list.append(instance['InstanceId'])

    return {
        "statusCode": 200,
        "body": ec2list
    }

Note: DO NOT forget to deploy the Lambda function every time you make a change.

First Error: We get the following error when we execute the code❓

Response
{
  "errorMessage": "name 'boto3' is not defined",
  "errorType": "NameError",
  "requestId": "6926f216-e0e8-4f25-bb16-761ada972e3f",
  "stackTrace": [
    "  File \"/var/task/lambda_function.py\", line 8, in lambda_handler\n    ec2 = boto3.client('ec2')\n"
  ]
}

Hint 1:
The first one is easy to solve by executing the code in Lambda, the hint appears in the error message.

Second Error: We get the following error after fixing the first one❓

Response
{
  "errorMessage": "'Instances'",
  "errorType": "KeyError",
  "requestId": "c78a6211-1d73-4d72-b83a-4c51bfd4e181",
  "stackTrace": [
    "  File \"/var/task/lambda_function.py\", line 21, in lambda_handler\n    for reservation in response[\"Instances\"]:\n"
  ]
}

Hint 2:

Please refer to the Response Syntax of describe_instances(**kwargs) at Boto3 Documentation

Step 4. Final Code: ✅

import json
import boto3

ec2list = []

def lambda_handler(event, context):

    # Get list of regions
    ec2 = boto3.client('ec2')
    regions = ec2.describe_regions().get('Regions',[] )

    # Iterate over regions
    for region in regions:

        print ("* Checking region  --   %s " % region['RegionName'])
        reg=region['RegionName']

        client = boto3.client('ec2', region_name=reg)
        response = client.describe_instances()

        for reservation in response["Reservations"]:
            for instance in reservation["Instances"]:
                print ("  ---- Instance %s in %s" % (instance['InstanceId'], region['RegionName']))
                ec2list.append(instance['InstanceId'])

    return {
        "statusCode": 200,
        "body": ec2list
    }

Output

{
  "statusCode": 200,
  "body": [
    "I-[instanceid]"
  ]
}

START RequestId: f6a1c381-51e2-43c9-8490-a62b8a6394dd Version: $LATEST
* Checking region  --   eu-north-1
* Checking region  --   ap-south-1
* Checking region  --   eu-west-3
* Checking region  --   eu-west-2
* Checking region  --   eu-west-1
* Checking region  --   ap-northeast-3
* Checking region  --   ap-northeast-2
* Checking region  --   ap-northeast-1
* Checking region  --   sa-east-1
* Checking region  --   ca-central-1
* Checking region  --   ap-southeast-1
* Checking region  --   ap-southeast-2
* Checking region  --   eu-central-1
* Checking region  --   us-east-1
---- Instance [instanceid] in us-east-1
* Checking region  --   us-east-2
* Checking region  --   us-west-1
* Checking region  --   us-west-2
END RequestId: f6a1c381-51e2-43c9-8490-a62b8a6394dd
REPORT RequestId: f6a1c381-51e2-43c9-8490-a62b8a6394dd Duration: 15667.85 ms Billed Duration: 15668 ms Memory Size: 128 MB Max Memory Used: 90 MB Init Duration: 246.67 ms

Note: The actual instance id is replaced with [instanceid] in the output.

So what have I done to break the code

I removed import boto3 at line no 2
Replaced response["Reservations"]: with response["Instances"]: at line no 21

Hope you liked the challenge! Please let me know your thoughts in the comments section.

Tip: Please do check out the reference link as you will learn something about global services too with Boto3. 👇

See you next time 👋

References:

I learnt this from stackoverflow when I asked a question about Retrieve list of buckets from all regions using boto3

Hope you have also learnt something about retrieving S3 buckets from all regions using Boto3.

Forem: Sri

Building Secure Jenkins-Slack Integration with AWS Lambda - Part 2: Troubleshooting Real-World Issues

The Reality Check

🚨 Critical Issues and Solutions

Issue 1: Jenkins Authentication Failure

Issue 2: CSRF Token Errors

Issue 3: Slack Slash Command dispatch_unknown_error

Issue 4: Lambda Function Evolution

🛠️ Production Enhancements

Security Improvements

1. Slack Signature Verification

2. Enhanced IAM Roles

Monitoring and Observability

1. CloudWatch Alarms

2. Structured Logging

Advanced Features

1. Dynamic Parameter Parsing

2. Job Status Callbacks

🔧 Troubleshooting Guide

Common Debugging Commands

Performance Optimization

1. Connection Pooling

2. SSM Parameter Caching

🎯 Key Lessons Learned

What Worked Well

What Didn't Work Initially

Best Practices Discovered

🚀 Production Deployment Checklist

📊 Performance Metrics

🎉 Conclusion

🔗 Resources and Next Steps

🤝 Community Contributions

Building Secure Jenkins-Slack Integration with AWS Lambda - Part 1: Complete Setup Guide

Building Secure Jenkins-Slack Integration with AWS Lambda - Part 1: Complete Setup Guide

What We'll Build

Tech Stack Overview

Architecture Deep Dive

Key Components Explained

Security Benefits

Prerequisites

Step 0: Initial Setup

Configure AWS CLI

Create SSH Key Pair

Step-by-Step Implementation

Step 1: Clone and Setup

Step 2: Build Lambda Functions

Step 3: Configure Slack App

Create Slack App

Configure Slash Command

Enable Incoming Webhooks

Get App Credentials

Install App to Workspace

Step 4: Prepare Terraform Variables

Step 5: Deploy Infrastructure

Step 6: Update Slack Configuration

Step 7: Configure Jenkins

Access Jenkins

Initial Jenkins Setup

Install Slack Plugin

Create Pipeline Job

Step 8: Update SSM Parameters

Verification and Testing

Test 1: Verify Infrastructure

Test 2: Verify Lambda Function

Test 3: Verify Jenkins Connectivity

Test 4: Test Slack Command

Test 5: Verify Jenkins Job Execution

Test 6: End-to-End Test with Parameters

Key Code Components

Lambda Function (Go)

Jenkins Pipeline

What's Next?

Coming Up in Part 2

Key Takeaways

Setting up AWS Client VPN with CloudFormation and AWS CLI

Table of Contents

What is AWS Client VPN?

Prerequisites

Step-by-Step Deployment with CloudFormation

1. Generate Client and Server Certificates