Forem: Omolayo Victor

This came to mind after reading some articles. I had to come right in and correct some fundamental mistakes. - Placement groups work within a region and AZ. This flexibility provides huge advantage to your infrastructure in terms of Latency or Reliability.

Omolayo Victor — Fri, 30 May 2025 20:19:37 +0000

Omolayo Victor

Sep 6 '23

AWS Placement Groups: The Lego Analogy

#aws #placementgroup #architecture

Comments

4 min read

Introducing vulne-soldier: A Modern AWS EC2 Vulnerability Remediation Tool

Omolayo Victor — Tue, 14 Jan 2025 16:19:56 +0000

Introduction

As cloud computing platforms like AWS become increasingly widespread, organisations are embracing them for their flexibility and autonomy in managing workloads and services. AWS, in particular, offers a robust infrastructure, flexible migration services that allows businesses to take control of their infrastructure destiny either on-site, hybrid or in the cloud. However, with the growing adoption of cloud services, the threat landscape also expands, necessitating effective vulnerability management tools.

Most existing vulnerability management tools require manual intervention, where engineers must address each vulnerability individually. As workloads grows, the more effort is required to perform these actions. AWS provides tools like AWS Inspector and AWS Systems Manager (SSM) amongst to assess and manage software vulnerabilities and unintended network exposures. Amazon Inspector, for instance, uses the SSM agent to collect software inventory from connected resources (EC2, ECR, and Lambda), scans this data, and identifies software vulnerabilities, a crucial step in vulnerability management.

The Need for Automation

In today's fast-paced digital environment, manual vulnerability management is not only time-consuming but also prone to human error. As organizations scale their cloud infrastructure, the number of vulnerabilities that need to be managed grows exponentially, this is where automation becomes essential. Automating the vulnerability remediation process ensures that security patches are applied consistently and promptly, reducing the risk of exploitation.

Introducing vulne-soldier

Here we present vulne-soldier, an AWS EC2 vulnerability remediation tool designed to automate the process of patching nodes managed by AWS Systems Manager. With a cup of coffee in hand, we package vulne-soldier as a gift to every organization and cloud professional concerned about the security of their systems.
Take for example security issues like the CrowdStrike outage (due to software updates) or Log4j vulnerability (CVE-2021-44228), they were critical vulnerabilities that affected many applications, and the need to patch then were urgent. With a tool like vulne-soldier via Amazon Inspector, the process of identifying and remediating such vulnerabilities would have been automated, reducing the risk of exploitation.

vulne-soldier leverages Amazon Inspector findings for EC2 instances, using resource tags and finding severity to group and address vulnerabilities. It automates the remediation process by applying patches only to the affected EC2 instances, making vulnerability management as simple as possible.

Key Features

Automated Remediation: Uses AWS Systems Manager Patch Manager to automate the patching process.
Integration with Amazon Inspector: Gathers findings from Amazon Inspector and groups them by severity.
Targeted Patching: Applies patches only to affected EC2 instances based on resource tags and severity levels.
Terraform Integration: Provisions all necessary resources using Terraform, ensuring a seamless deployment process.

How It Works

AWS Inspector Findings: Amazon Inspector scans EC2 instances and identifies vulnerabilities.
Grouping by Severity: vulne-soldier groups the findings by severity levels (e.g., CRITICAL, HIGH).
Automated Patching: Uses AWS Systems Manager Patch Manager to apply patches to the affected instances.
Terraform Provisioning: Deploys the necessary resources using Terraform, ensuring a consistent and repeatable setup.

Using vulne-soldier

Download lambda

To apply the terraform module, the compiled lambdas (.zip files) need to be available locally. They can either be downloaded from the GitHub release page or built locally.

The lambdas can be downloaded manually from the release page or by building the Lambda folder using Node.

For local development you can build the lambdas at once using /lambda or individually using npm zip.

Here is an example configuration for deploying the vulne-soldier module:

module "remediation" {
  source  = "iKnowJavaScript/vulne-soldier/aws"
  version = "1.0.2"

  name             = "vulne-soldier-compliance-remediate"
  environment      = "dev"
  aws_region       = "us-east-1"
  account_id       = "2132323212"
  lambda_log_group = "/aws/lambda/vulne-soldier-compliance-remediate"
  lambda_zip       = "../../lambda.zip"
  remediation_options = {
    region                                     = "us-east-1"
    reboot_option                              = "NoReboot"
    target_ec2_tag_name                        = "AmazonECSManaged"
    target_ec2_tag_value                       = "true"
    vulnerability_severities                   = ["CRITICAL", "HIGH"]
    override_findings_for_target_instances_ids = []
  }
}

provider "aws" {
  region = "us-east-1"
}

Triggers Remediation Process

On successful deployment, navigate to the AWS Systems Manager console and search for the SSM document created by the module (vulne-soldier-compliance-remediate-inspector-findings) or similar. You can trigger the remediation process by running the document on the affected EC2 instances. You can also create an AWS CloudWatch event rule to automate the process based on AWS Inspector findings.

Conclusion

vulne-soldier simplifies the process of managing and remediating vulnerabilities in AWS EC2 instances. By automating the patching process and integrating seamlessly with AWS Inspector, it enables you to scale your cloud security as your infrastructure grows with minimal manual intervention. Deploy vulne-soldier today and take control of your cloud security.

Links;

Deploying AI Models with Amazon Web Services: A Practical Guide

Omolayo Victor — Wed, 11 Dec 2024 12:22:43 +0000

Introduction

This is my first rodeo on AI model development, and it has been an incredible learning journey. As part of Blackthorn’s just-concluded company-wide hackathon on AI and Agent Force, I embarked on a chosen project that involved deploying an AI model that generates a niche(event-related) image. This project offered an opportunity to gain in-depth knowledge about AI development, models, datasets, and the infrastructure required to support them. The results were both enlightening and rewarding, showcasing the power of modern AI and cloud technologies.

If you’re here for the source code, it’s available on GitHub: GitHub Repo

Why Control Your AI Infrastructure?

One of the core advantages of deploying our AI model on AWS was gaining complete control over data handling and retention. By hosting the model on our infrastructure, we were able to:

Maintain strict control over sensitive data, ensuring secure storage and retention policies.
Implement data TTL (time-to-live) mechanisms to comply with compliance requirements.
Tailor the environment for optimal performance, resource allocation, and cost efficiency.

This approach highlighted the importance of balancing privacy, performance, and scalability in AI solutions.

Choosing an AI Model

We are all familiar with popular AI tools like ChatGPT, Gemini, and Claude, which showcase the power of conversational AI. While browsing the vast ocean of datasets and models available on Hugging Face was tempting, we decided to focus on leveraging an open-source model for our hackathon project. This led us to explore Stable Diffusion—a remarkable latent text-to-image diffusion model.

Stable Diffusion (GitHub Repository)

Stable Diffusion stood out for its versatility as a latent text-to-image diffusion model pre-trained on a subset of the LAION-5B dataset. Some key features include:

Text Encoder: It uses a text encoder to condition the model on text prompts, enabling intuitive image generation from descriptions.
Resource Efficiency: Lightweight enough to run on GPUs with at least 10GB VRAM, making it accessible for medium-scale deployments.
Default Model: The model "CompVis/stable-diffusion-v1-4" is pre-trained and ready for adaptation, although other versions offer varying trade-offs in terms of fidelity and inference time.

Hugging Face (Hugging Face Hub)
played a significant role in this journey. As a leading platform for sharing pre-trained AI models and datasets, Hugging Face provided access to a wide range of resources. From discovering datasets to fine-tuning models, the platform proved invaluable for quickly iterating and adapting Stable Diffusion to our project’s needs.

Infrastructure on AWS

To host the AI model, we chose the Deep Learning OSS Nvidia Driver AMI (Amazon Linux 2) with the AMI ID ami-002a53be89c7bb5de. This decision was driven by the need for:

High GPU Performance: The AMI’s compatibility with Nvidia drivers ensures efficient usage of GPUs for model inference.
Flexibility with Docker: Using the stable-diffusion-docker repository (GitHub Repository), we adapted the model for containerized deployment.
Cost Efficiency: EC2’s on-demand pricing allowed us to scale resources as needed.

Additionally, we explored Amazon SageMaker for internal model training and deploying models directly within the AWS ecosystem. This service provided a seamless integration for training and inference, leveraging AWS’s robust infrastructure. Further explore AWS Batch to efficiently run AI tasks as jobs for batch processing, which are invaluable for handling workloads at scale.

Diving into Hugging Face

Hugging Face is a platform that provides a repository of pre-trained models, datasets, and tools for AI development. We used it to:

Discover Datasets: Identify relevant datasets for fine-tuning Stable Diffusion.
Create Custom Datasets: Curate and upload datasets with selective questions and answers, tailored to our project needs.
Train the Model: Fine-tune Stable Diffusion to align more closely with our domain-specific requirements.

Challenges and Solutions

The project wasn’t without hurdles. Some notable challenges and how we addressed them include:

API Gateway Timeout:

Problem: The default API Gateway timeout caused issues when EC2 took longer to generate images.
Solution: We implemented an S3-based placeholder system where:
- The AI-generated image was stored in an S3 bucket.
- A response was sent back to the client with a reference to the S3 location.
  - Alternative Approaches: Bidirectional communication with WebSockets, queues like SQS, or real-time protocols could have mitigated this issue further.

Fine-Tuning Stable Diffusion:

Problem: Achieving accurate and domain-specific image generation required additional fine-tuning.
Solution: Leveraged Hugging Face datasets to train the model with targeted data, iterating to improve outcomes.

Latency Optimization:

Problem: Initial inference times averaged 32 seconds per banner, which may not scale well for high-volume usage.
Solution: Optimized Docker configurations, utilized larger GPU instances during high-load periods, and explored model quantization.

Open Source Contribution

The entire infrastructure-as-code for this project has been made open source. The Terraform scripts used to create necessary AWS resources, pull the model, and set up datasets are available at the following repository: GitHub Repo

Lessons Learned

The project was a crash course in AI and cloud engineering. Key takeaways include:

Model Choice Matters: Different versions of Stable Diffusion offer varying benefits; understanding these trade-offs is essential.
Infrastructure Optimization: Balancing cost and performance is critical when scaling AI workloads.
System Design: Asynchronous processing with S3 helped circumvent API limitations, emphasizing the need for resilient architectures.
Collaboration Tools: Platforms like Hugging Face streamline model development and dataset curation.

Future Directions

For the POC, additional considerations include:

Scaling Infrastructure: Implement autoscaling to handle varying demand.
Real-Time Communication: Explore WebSocket-based communication for live updates.
Monitoring and Observability: Integrate CloudWatch to monitor GPU usage, latency, and system health.
Enhanced Security: Implement stricter IAM roles and encryption mechanisms for data in transit and at rest.

Conclusion

Deploying AI models with AWS provides unparalleled flexibility and control, making it an ideal choice for custom AI projects. This journey, from Stable Diffusion exploration to creating an optimized cloud-based infrastructure, has been both challenging and rewarding. The experience has laid a strong foundation for tackling future AI endeavors and scaling them to production-ready solutions.

As I look forward, I’m excited to continue exploring AI models, refining cloud-based architectures, and driving innovation in AI-powered solutions.

Building High-Performance, Secure Static Websites on a Budget with AWS and Terraform

Omolayo Victor — Wed, 15 May 2024 17:50:08 +0000

Introduction

In the ever-evolving landscape of the internet, websites have transitioned from simple information pages to complex systems that power businesses and personal platforms. This journey has been marked by numerous technological advancements, from the early days of static web pages served via Netscape to PHP admin to contemporary frameworks brimming with features that make development a breeze.

Today's digital age demands not only functionally rich websites but also ones that are secure, performant, and cost-effective. Amazon Web Services (AWS) has emerged as a leading cloud provider offering tools to meet these needs.

Herein lies the charm of this guide: we'll embark on a clear and concise journey to deploy a robust frontend architecture using AWS, all orchestrated with the mighty Terraform—an open-source Infrastructure as Code (IaC) tool that simplifies and automates deployment.

This walkthrough is tailored for individuals or businesses striving for an efficient and secure online presence without breaking the bank. We will meticulously set up storage with S3, manage domain names with Route 53, handle access with IAM, accelerate content delivery with CloudFront, and shield the site with WAF.

The best part? No prior extensive knowledge is required—as long as you grasp the basics of AWS services and Terraform, you're good to go. I bet it will be the easiest way for anyone to create a secure, compliant, highly available, and highly performant website from scratch before it becomes a task for AI assistance.

Feel free to leap over to the completed code repository on GitHub (and don't forget to star it!) if you're eager to get your hands on the code right away.

Prerequisites

Basic knowledge of AWS services and Terraform.
Terraform installed on your local machine. If not, you can follow this guide to install it.
A domain name, if you choose to use a custom domain

Terraform Configuration Explained

Step 1: Define Variables

First, we define variables that will be used throughout our Terraform configuration. These include the name of the application, the environment, and optional custom domain settings. Add these to a new inputs.tf file add the values to inputs.auto.tfvars file

variable "name" {
  description = "Name of the application"
  type        = string
}

variable "environment" {
  description = "Name of the environment"
  type        = string
}

variable "hosted_zone_domain" {
  type        = string
  description = "Hosted zone to add domain and CloudFront CNAME to"
  nullable    = true
}

variable "create_custom_domain" {
  type        = bool
  description = "Whether to use a custom domain or not"
  default     = false
}

variable "custom_domain_name" {
  type        = string
  description = "Custom domain name"
  nullable    = true
}

Step 2: Create S3 Bucket

Next, create a main.tf file, we'll create an S3 bucket to store our static website content.

resource "aws_s3_bucket" "static_bucket" {
  bucket = "${var.name}-${var.environment}"
}

Step 3: Create CloudFront Origin Access Identity

The create cloudfront.tf file, we then create a CloudFront Origin Access Identity (OAI). This allows CloudFront to get objects from our S3 bucket on our behalf.

resource "aws_cloudfront_origin_access_identity" "newOAI" {
  comment = "OAI for ${var.name} S3 bucket"
}

Step 4: Create CloudFront Distribution

We create a CloudFront distribution to deliver our content to users. We configure it to use our S3 bucket as the origin and our OAI for access. We also set up caching, HTTPS redirection, and custom error responses. In cloudfront.tf file.

resource "aws_cloudfront_distribution" "static_content_distribution" {
  origin {
    domain_name = aws_s3_bucket.static_bucket.bucket_regional_domain_name
    origin_id   = "S3Origin"

    s3_origin_config {
      origin_access_identity = aws_cloudfront_origin_access_identity.newOAI.cloudfront_access_identity_path
    }
  }


  enabled             = true
  is_ipv6_enabled     = true
  default_root_object = "index.html"
  comment             = "${var.name} - frontend deployment"

  default_cache_behavior {
    allowed_methods  = ["DELETE", "GET", "HEAD", "OPTIONS", "PATCH", "POST", "PUT"]
    cached_methods   = ["GET", "HEAD"]
    target_origin_id = "S3Origin"

    forwarded_values {
      query_string = false

      cookies {
        forward = "none"
      }
    }

    viewer_protocol_policy = "redirect-to-https"
    min_ttl                = 0
    default_ttl            = 3600
    max_ttl                = 86400
    compress               = true
  }

  custom_error_response {
    error_code            = 404
    response_page_path    = "/index.html"
    response_code         = 200
    error_caching_min_ttl = 300
  }

  custom_error_response {
    error_code            = 403
    response_page_path    = "/index.html"
    response_code         = 200
    error_caching_min_ttl = 300
  }

  price_class = "PriceClass_100"

  restrictions {
    geo_restriction {
      restriction_type = "none"
    }
  }

  aliases = [var.create_custom_domain ? var.custom_domain_name : null]


  dynamic "viewer_certificate" {
    for_each = var.create_custom_domain ? [1] : []

    content {
      acm_certificate_arn      = module.dns[0].certificate_arn
      ssl_support_method       = "sni-only"
      minimum_protocol_version = "TLSv1.2_2018"
    }
  }


  dynamic "viewer_certificate" {
    for_each = var.create_custom_domain ? [] : [1]

    content {
      cloudfront_default_certificate = true
    }
  }

  web_acl_id = aws_wafv2_web_acl.web_acl.arn
}

The CloudFront distribution is configured with cache behavior, custom error responses, and TLS settings. It serves content over HTTPS, redirects HTTP traffic, and compresses content for better performance.
We also depend on the variable create_custom_domain to know whether to use a custom domain or a Cloudfront-provisioned random domain name.

Step 5: DNS Module for Custom Domain

In cloudfront.tf file. Add the following.

module "dns" {
  count = var.create_custom_domain ? 1 : 0

  source = "./modules/dns"

  hosted_zone_domain = var.hosted_zone_domain
  custom_domain_name = var.custom_domain_name
  cloudflare_domain  = aws_cloudfront_distribution.static_content_distribution.domain_name
  cloudflare_zone_id = aws_cloudfront_distribution.static_content_distribution.hosted_zone_id
}

If a custom domain is preferred, this module sets up the necessary DNS records and TLS certificate using ACM for SSL/TLS. It associates the custom domain with the CloudFront distribution. code on GitHub

Step 6: Create IAM User and Policy

In iam-user.tf file, We create an IAM user and policy to allow full access to our S3 bucket. This user can be used to upload content to the bucket via AWS CLI or GitHub action - Please comment bellow if you need an article on either one.

resource "aws_iam_user" "s3_user" {
  name = "s3_full_access_user_for_${var.name}"
}

resource "aws_iam_access_key" "s3_user_key" {
  user = aws_iam_user.s3_user.name
}

resource "aws_iam_user_policy" "s3_full_access" {
  name = "s3_full_access"
  user = aws_iam_user.s3_user.name

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "s3:*"
        Effect = "Allow"
        Resource = [
          aws_s3_bucket.static_bucket.arn,
          "${aws_s3_bucket.static_bucket.arn}/*"
        ]
      }
    ]
  })
}

Step 7: Apply S3 Bucket Policy

In policy.tf file, We apply a policy to our S3 bucket to allow our CloudFront OAI to get objects and to enforce server-side encryption for all uploaded objects.

resource "aws_s3_bucket_policy" "s3policyforOAI" {
  bucket = aws_s3_bucket.static_bucket.id

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Action   = ["s3:GetObject"],
        Effect   = "Allow",
        Resource = "${aws_s3_bucket.static_bucket.arn}/*",
        Principal = {
          AWS = "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity ${aws_cloudfront_origin_access_identity.newOAI.id}"
        }
      },
      {
        "Sid" : "enforce-encryption-method",
        "Effect" : "Deny",
        "Principal" : "*",
        "Action" : "s3:PutObject",
        "Resource" : "${aws_s3_bucket.static_bucket.arn}/*",
        "Condition" : {
          "StringNotEquals" : {
            "s3:x-amz-server-side-encryption" : "AES256"
          }
        }
      }
    ]
  })
}

Step 8: Create WAF Web ACL

Finally, In waf.tf file, we create a WAF Web ACL and associate it with our CloudFront distribution to protect our website from common web exploits, managed by AWS rulesets.

resource "aws_wafv2_web_acl" "web_acl" {
  name        = "${var.name}-waf"
  description = "WAF ACL for ${var.name} CloudFront distribution"
  scope       = "CLOUDFRONT"

  default_action {
    allow {}
  }

  visibility_config {
    cloudwatch_metrics_enabled = true
    metric_name                = "${var.name}-web-acl-metric"
    sampled_requests_enabled   = true
  }

  rule {
    name     = "AWS-AWSManagedRulesCommonRuleSet"
    priority = 1

    override_action {
      none {}
    }

    statement {
      managed_rule_group_statement {
        name        = "AWSManagedRulesCommonRuleSet"
        vendor_name = "AWS"
        rule_action_override {
          name = "SizeRestrictions_BODY"
          action_to_use {
            count {}
          }
        }
      }
    }

    visibility_config {
      cloudwatch_metrics_enabled = true
      metric_name                = "AWS-AWSManagedRulesCommonRuleSet"
      sampled_requests_enabled   = true
    }
  }
}

# Associate the WAF Web ACL with the CloudFront distribution
# resource "aws_wafv2_web_acl_association" "waf_assoc" {
#   resource_arn = aws_cloudfront_distribution.static_content_distribution.web_acl_id
#   web_acl_arn  = aws_wafv2_web_acl.web_acl.arn
# }

The creation of a WAF ACL adds a strong layer of security to our CloudFront distribution. We configure it with AWS Managed Rules for common threats, which is an excellent starting point for protecting against a wide range of attacks.

Terraform CMD

To complete these configurations and have everything running, apply your Terraform configuration by executing terraform apply in your terminal. This command will provision all the defined resources in your AWS account.

Remember to review the changes before applying them, ensuring that you understand what resources will be created or modified.

# To initialize Terraform and install required providers
terraform init

# To plan and review the infrastructure changes
terraform plan

# To apply changes and create the infrastructure
terraform apply

After you confirm and apply the changes, Terraform will provide outputs with the necessary information such as your website URL, S3 bucket names, and IAM user credentials which you should secure.

Conclusion

Deploying a high-performing and secure static website on AWS using Terraform can significantly simplify the process of infrastructure management. The power of Infrastructure as Code (IaC) allows you to version control your infrastructure, track changes, and quickly replicate or destroy environments as needed.

In this article, we've outlined the steps to set up your static hosting environment with security and performance best practices in mind. Our approach ensures that your website remains highly available, performant under load, and resilient against common web vulnerabilities at an optimized cost.

Now that you have your website deployed, you can focus on uploading your content, monitoring performance, and enhancing user experience. As your needs evolve, you can update your Terraform configurations to scale your infrastructure or integrate additional services.

Be sure to check out the GitHub repository for the complete code and leave a star behind if you found it helpful. If you encounter any issues or have questions, don't hesitate to comment below or open an issue on GitHub. Your contributions to improving the code are welcome!

Update:
Checkout a simplified Terraform package based on this here https://github.com/iKnowJavaScript/terraform-aws-complete-static-site

Links;

Happy Terraforming!!!

Building the Self-Hosted On-Demand Runner Infrastructure with Terraform

Omolayo Victor — Thu, 23 Nov 2023 05:57:19 +0000

In the previous installment, we established a robust foundation of components that will power our self-hosted on-demand runner infrastructure. These components, including the GitHub App, API Gateway, Lambda functions, SQS, S3, EC2, SSM Parameters, Amazon EventBridge, and CloudWatch, work in concert to provide a scalable and cost-effective solution for GitHub runners.

Now, we'll configure the GitHub App to serve as a nexus between GitHub and AWS, triggering the creation or removal of EC2 instances based on webhook events. This dynamic mechanism ensures that the number of available runners always aligns with the current workload, optimizing resource utilization and costs.

We'll also delve into the practical implementation of this infrastructure using Terraform, an infrastructure as code (IaC) tool that streamlines the provisioning and management of AWS resources. With Terraform, we'll automate the deployment of EC2 instances, VPCs, and IAM roles, ensuring consistent and repeatable infrastructure setups.

Create GitHub App

To begin, navigate to GitHub and establish a new app. Bear in mind that you have the option to create apps for either your organization or a specific user. For the time being, we'll use an organization-level app.

Step 1: Create a GitHub App

Access GitHub and navigate to the "Settings" section for your organization.
Select "Developer settings" from the left-hand sidebar.
Click on "New GitHub App" in the "GitHub Apps" section.
Provide a name for your app, such as "Self-Hosted Runner App".
Enter a website URL for your app (mandatory, but not required for this module).
Uncheck the "Enable webhook" option for now, as we will configure this later or create an alternative webhook.

Step 2: Define App Permissions

Scroll down to the "Permissions" section and define the following permissions for all runners:
- Repository: Actions: Read-only (check for queued jobs)
- Repository: Checks: Read-only (receive events for new builds)
- Repository: Metadata: Read-only (default/required)
Next, define the following permissions specifically for repo-level runners:
- Repository: Administration: Read & write (to register runner)
Finally, define the following permissions specifically for organization-level runners:
- Organization: Self-hosted runners: Read & write (to register runner)

Step 3: Save and Note App Details

Click the "Save" button to finalize the app creation process.
On the General page, make a note of the "App ID" and "Client ID" parameters. These will be used later in the process. Step 4: Generate Private Key
Generate a new private key using an appropriate tool or command-line utility.
Save the generated private key as app.private-key.pem. This file will be used later to authenticate the app with GitHub.

Create Infrastructure with Terraform

Prerequisites

The following tools are required to perform this step:

A GitHub account with an organization or personal access token
An AWS account with VPC and subnets already created
Terraform installed and configured on your system
Node.js and Yarn (for Lambda development)
Bash shell or compatible shell

Once you have installed and configured all the required tools, you are ready to proceed to the next step, where we will create the necessary AWS resources using Terraform.

Creating Resources

We'll utilize a highly configurable and exceptional Terraform module terraform-aws-github-runner
to streamline the implementation of our infrastructure. This module boasts stellar maintenance and offers various approaches through internal modules, allowing you to seamlessly adapt the infrastructure to your project's specific needs. In this article, we'll focus on implementing the simple runner configuration provided by the module, which aligns perfectly with the article's requirements.

All Terraform code is available here.

Before diving into the intricacies of the runner infrastructure, we'll begin by downloading the essential lambda function code required by the module to dynamically create and destroy our resources as needed.

Then, we'll create our runner module which will create Lambda functions(webhook, scale-up, scale-down, syncer), SQS(workflow-queue, queue-builds), EC2 Launch template(e), s3(s) among others.

Lambda Functions: The Heart of the Infrastructure

Lambda functions serve as the brains of our infrastructure, orchestrating various operations and ensuring seamless responsiveness to changing demands. Each Lambda function fulfills a specific role:

webhook: This function acts as the gateway for incoming webhook events, meticulously verifying their authenticity and ensuring they align with the expected criteria. It only processes events related to workflow_job, status queued, and matching the runner labels.
scale-up: Continuously monitoring the SQS queue, this Lambda function eagerly awaits incoming events. Upon receiving an event, it performs a series of checks to determine whether a new EC2 spot instance is required to accommodate the workload. If deemed necessary, it utilizes the predefined EC2 Launch template(e) to spin up new EC2 instances, expanding the runner pool.
scale-down: Monitors the SQS queue triggered at an interval. This Lambda function diligently monitors the runner pool, identifying idle runners that have been removed from GitHub. Once an idle runner is detected, it prompts the termination of the corresponding EC2 instance, optimizing resource utilization and minimizing costs.
syncer: Downloading the GitHub Action Runner distribution can occasionally be a time-consuming process, sometimes exceeding ten minutes. To alleviate this bottleneck, a dedicated Lambda function is introduced. This function meticulously synchronizes the action runner binary from GitHub to a designated S3 bucket(s). Subsequently, EC2 instances(e) seamlessly fetch the distribution from the S3 bucket(s) rather than relying on slower internet downloads.

GitHub App Module: Bridging the Gap Between GitHub and AWS

To establish seamless communication between GitHub and AWS, a GitHub app module is meticulously implemented. This module integrates seamlessly with our GitHub app, enabling the creation of an API gateway. This gateway serves as the intermediary, securely handling webhook events sent by the GitHub App over HTTPS. Additionally, it relays responses back to the GitHub app, ensuring a robust and reliable communication channel.

Conclusion

In this comprehensive guide, we have embarked on a journey to build a scalable and cost-effective self-hosted runner infrastructure on AWS using Terraform, an infrastructure as code (IaC) tool. Through this exploration, we have delved into the key components that power this infrastructure, including the GitHub App, API Gateway, Lambda functions, SQS, S3, EC2, SSM Parameters, Amazon EventBridge, and CloudWatch.

By leveraging these components, we ensure optimal resource utilization and cost-efficiency. We have also explored the practical implementation of this infrastructure using Terraform, automating the provisioning and management of AWS resources for consistent and repeatable setups.

The self-hosted runner infrastructure we have created provides a powerful foundation for organizations seeking to enhance their CI/CD capabilities and streamline their software development processes as we currently do at Blackthorn. By harnessing the scalability and flexibility of AWS, organizations can effectively manage runner capacity and costs, ensuring that their CI/CD infrastructure aligns with their evolving needs.

As you embark on your own journey to build self-hosted runner infrastructures, remember the key takeaways from this guide:

Utilize Infrastructure as Code (IaC): Leverage IaC tools like Terraform to automate the provisioning and management of AWS resources, ensuring consistent and repeatable infrastructure setups.
Design for Scalability: Build your infrastructure with scalability in mind, employing components like Lambda functions and SQS to dynamically adjust runner capacity based on workload demands.
Optimize Resource Utilization: Monitor resource utilization metrics and proactively scale your infrastructure to avoid resource bottlenecks and unnecessary costs.
Embrace Continuous Improvement: Continuously evaluate and refine your infrastructure to adapt to changing requirements and optimize performance.

By adhering to these principles, you can effectively build and maintain a self-hosted runner infrastructure that empowers your organization to achieve continuous delivery excellence.

Links:

All Terraform code is available here
terraform-aws-github-runner

Scaling GitHub Actions Runners on AWS: A Cost-Effective and Scalable Approach

Omolayo Victor — Tue, 14 Nov 2023 16:51:41 +0000

In the realm of software development, continuous integration (CI) and continuous delivery (CD) have become indispensable practices for ensuring the quality and timely release of software applications. GitHub Actions, a cloud-based CI/CD platform, has emerged as a popular choice among developers for its ease of use and flexibility. However, as the number of repositories and workflows under management grows, the need for scalable and cost-effective runner infrastructure becomes increasingly important.

To address this challenge, we have developed a self-hosted on-demand runner infrastructure on AWS that utilizes a combination of GitHub, Amazon Web Services (AWS), and other tools. This infrastructure enables us to scale our runner capacity up or down based on demand, ensuring that we have enough runners to handle the workload without incurring unnecessary costs.

Key Design Considerations

In designing the self-hosted on-demand runner infrastructure, we focused on several key considerations:

Cost-effectiveness: The infrastructure should minimize cloud resource consumption and avoid unnecessary costs when not in use.

Scalability: The infrastructure should be able to handle fluctuating workloads by scaling up or down the number of runners dynamically.

Reliability: The infrastructure should be highly available and ensure consistent execution of workflows.

Ease of management: The infrastructure should be easy to deploy, manage, and maintain.

Key Components of the Infrastructure

The key components of our self-hosted on-demand runner infrastructure include:

GitHub App: This GitHub App acts as a bridge between GitHub and AWS, receiving webhook events from GitHub repositories and triggering the creation or removal of EC2 instances based on those events.

API Gateway: API Gateway serves as an HTTP endpoint for the webhook events sent by the GitHub App, providing a secure and reliable channel for communication.

Lambda Functions: Lambda functions are the workhorses of the infrastructure, handling the incoming webhook events, verifying their authenticity, and triggering the scaling up or scaling down of EC2 instances.

SQS (Simple Queue Service): SQS acts as a message queue, decoupling the incoming webhook events from processing these events. This ensures that events are not lost if there are temporary delays in processing.

S3 (Simple Storage Service): S3 serves as a repository for storing the runner binaries that are downloaded from GitHub. This allows EC2 instances to fetch the runner binaries locally instead of downloading them from the internet, improving performance.

EC2 (Elastic Compute Cloud): EC2 instances provide the computational resources for running GitHub Actions workflows. The number of EC2 instances is dynamically scaled up or down based on the demand for runners.

SSM Parameters: SSM Parameters store configuration information for the runners, registration tokens, and secrets for the Lambdas. This centralized approach simplifies management and access control.

Amazon EventBridge: Amazon EventBridge schedules Lambda functions to execute at regular intervals, ensuring that idle runners are detected and terminated when no longer needed.

CloudWatch: CloudWatch provides real-time monitoring of the resources and applications in the AWS environment, enabling us to collect and track metrics for debugging and performance optimization.

Workflow and Scalability

The self-hosted on-demand runner infrastructure operates seamlessly to handle the scaling of runners based on workflow demands. When a workflow is triggered on a pull request action, the GitHub App sends a webhook event to the API Gateway, which in turn triggers the webhook Lambda function. The Lambda function verifies the event authenticity, processes it, and posts it to an SQS queue.

The scale-up Lambda function monitors the SQS queue for new events and evaluates various conditions to determine if a new EC2 spot instance needs to be created. If a new instance is required, the Lambda function requests a JIT configuration or registration token from GitHub, creates an EC2 spot instance using the launch template and user data script, and fetches the runner binary from the S3 bucket for installation. The runner registers with GitHub and starts executing workflows once it is fully configured.

In contrast, the scale-down Lambda function is triggered by Amazon EventBridge at regular intervals to check for idle runners. If a runner is not busy, the Lambda function removes it from GitHub and terminates the corresponding EC2 instance, ensuring efficient resource utilization and cost savings.

What's Next?

So far, we've defined a solid foundation of components that are crucial for building a cost-effective and scalable solution for GitHub runners. These components include the GitHub App, API Gateway, Lambda functions, SQS, S3, EC2, SSM Parameters, Amazon EventBridge, and CloudWatch. These components work together to provide a robust and dynamic infrastructure that can seamlessly handle fluctuating workloads.

Next, we'll embark on the practical implementation of this infrastructure using Terraform, an infrastructure as code (IaC) tool. Terraform will enable us to automate the provisioning of AWS resources, ensuring consistency and repeatability in our infrastructure setup. We'll delve into the process of creating the necessary AWS resources, including EC2 instances, VPCs, and IAM roles.
We'll also configure the GitHub App to act as a bridge between GitHub and AWS, triggering the creation or removal of EC2 instances based on webhook events. This will ensure that we always have the right number of runners available to handle the current workload.

Finally, we'll set up Lambda functions to orchestrate the scaling of runner instances. Lambda functions will be responsible for verifying the authenticity of incoming webhook events, processing them, and triggering the scaling up or scaling down of EC2 instances based on demand. This will ensure that our infrastructure is always optimized for cost and performance.
By the end of this series, you'll have a comprehensive understanding of how to build and deploy a scalable and cost-effective runner infrastructure on AWS using Terraform. You'll be able to leverage this infrastructure to improve your CI/CD performance and reduce your infrastructure costs.

Stay tuned for Part 2: Building the Self-Hosted On-Demand Runner Infrastructure with Terraform

AWS Placement Groups: The Lego Analogy

Omolayo Victor — Wed, 06 Sep 2023 19:21:53 +0000

AWS has a lot of exciting features which you can utilise while building highly optimised applications, in this article, we’ll explore Placement Groups and it’s strategies to architect better optimised application, looking at it from Lego perspective.

What is AWS Placement Group?

Placement groups let you choose the logical placement of your EC2 instances to achieve a more optimised communication, performance or fault tolerance within a Region.

At heart, working with placement groups is similar to creating intricate structures with Lego blocks. Each block represents a component of your design, and how you place these blocks together can impact the overall stability and performance of your creation.
In the world of AWS, similar principles apply when it comes to deploying your infrastructure. AWS Placement Groups offer a way to control how Amazon EC2 instances are placed within a region, much like the way you strategically position Lego blocks for optimal structural integrity.
By understanding this Lego analogy, you can grasp the concept of AWS Placement Groups more easily and make informed decisions when it comes to architecting your AWS environment.

Choosing the appropriate Placement Group strategy may result in optimal performance and reliability in your AWS environment. AWS provides three placement strategies which you can use based on the type of your workload:

Cluster Placement Groups - A logical grouping of instances into a single AZ
Partition Placement Groups - Spread instances across logical partitions to reduce likelihood of correlated hardware failure for your application.
Spread Placement Groups - Layout instances across distinct underlying hardware.

Cluster Placement Group

Imagine constructing a massive Lego tower where each block is tightly integrated with one another. This technique optimizes communication between blocks and ensures stability. In AWS, Cluster Placement Groups offer similar benefits by maximizing network performance and minimizing inter-instance latency within a single AZ.

Cluster Placement Groups is recommended when instances between the group benefits from close proximity, vast network traffic to achieve low latency and high network throughput.

Partition Placement Group

Picture creating a partitioned Lego structure where each section represents a specific function or workload. Similarly, Partition Placement Groups in AWS enable you to create logical partitions, ensuring instances within the same partition share the same rack, which can be beneficial for applications that require low latency and high throughput in multiple AZs in the same Region

Partition placement is typical for large distributed and replicated workloads like Kafka as it distribute the instances evenly across the number of partitions that you specify. You can also launch instances into a specific partition to have more control over where the instances are placed.

Spread Placement Group

Imagine creating a Lego structure by distributing each block evenly across multiple plates. This technique helps minimize the impact of hardware failures, ensuring that instances are placed on separate underlying infrastructure. In AWS, Spread Placement Groups follow a similar logic, maximizing availability by spreading instances across distinct hardware.

In this case, all the EC2 instances are located in different hardware, With the placement of different group across isolated AZs, it can help reduce simultaneous failures within the application while also promoting independent scalability across the group.

Conclusion

At Blackthorn, we utilize the Cluster Placement Group to enhance the performance and reliability of our Server, Worker, Database, and Caching components. By ensuring close physical proximity and minimising network latency, they optimize communication and coordination among these components, resulting in improved application stability, scalability, and user experience.

Understanding AWS Placement Groups can be simplified by using the Lego analogy. By visualizing how Lego structures are built, you can grasp the concept of grouping EC2 instances together for better performance and reliability. Just as master builders carefully select and position Lego blocks, AWS users can make informed decisions about their infrastructure design by leveraging the power of AWS Placement Groups.

More Resources

Placement group rules and Limitation
Creating Placement Group - Follow through

I hope this has been ineresting for you to read, let me know what you think, and share some other insightful features too!