Forem: Aviator

🚀 5 Javascript concepts every web developer should know! 👨‍💻

Sumit Saurabh — Mon, 02 Dec 2024 17:43:44 +0000

TL;DR

This article contains a list of 10 Javascript concepts that can take you from a Javascript newbie to a pro.

Hey Folks! 👋

This article contains top 10 Javascript concepts for web developers. It contains something for everyone so whether you've just started your journey or are a seasoned developer, you'll still find something of use here.

Join our exclusive developer community! 🎉

I'm starting a new community for new developers.

It is my aim to help early-age developers to level up in their careers, make meaningful connections and grow.

I intend to make it one of the best developer community out there and early members will have exclusive perks!

Wanna join? Here's how to:

Click on this invite link
Write a brief introduction of yourself and post in the #general-chat channel

1. Call stack:

Call stack is a data structure that keeps track of where the program is currently in its execution journey. Every time a function is called, an entry (called 'stack frame') is added to the call stack.

When the function completes execution, its corresponding stack frame is removed (or 'popped') from the call stack, and control returns to the previous function in the stack.

Key Points:

LIFO (Last In, First Out): The call stack operates on this principle, where the last function pushed is the first one popped.
Every active function call resides here, maintaining execution order.
If too many function calls are added without being removed (e.g., infinite recursion), the stack runs out of memory, causing a stack overflow.
Call Stack Trace: Debugging tools use the stack to show the order of function calls leading to an error.

2. Execution Context:

Execution context defines the environment within which JavaScript code is executed. It consists of everything needed to execute the code, such as variable definitions, the 'this' keyword, and function references.

There are three primary types of execution contexts in JavaScript:

i) Global Execution Context: Created by default when the script starts. It contains global variables and functions.
ii) Function Execution Context: Created whenever a function is called. Each function call gets its unique context.
iii) Eval Execution Context: Created when code is executed inside an eval() function.

Call stack and execution context work together. Call stack manages the order of execution, while the execution context manages the details of execution.
When a function is invoked, a new execution context is created and pushed onto the stack.
The code runs in the current execution context.
When the function finishes, its context is removed from the stack.

3. typeof, loose and strict equality:

While most developers do know what these are, pro developers, in addition to knowing the basics, also know the limitations and when to use what.

typeof:

The typeof operator returns a string indicating the type of its operand. It’s the easiest way to know the data type of a variable or an expression.

Key Points about typeof:

The output is always a string, such as "number", "string", "boolean", "object", "function", or "undefined".
Despite being an object reference, typeof null returns "object".
You can pass any value or expression to typeof.

typeof 42; // "number"
typeof "Hello"; // "string"
typeof {}; // "object"

loose equality:

The == operator checks if two values are loosely equal, meaning it compares their values after performing type coercion (converting both operands to the same type).

Key Points about ==:

Converts one or both operands to the same type before comparing.
The implicit type conversion can lead to unexpected results.
Use === instead for clarity and reliability.

42 == "42"; // true (string is coerced to a number)
null == undefined; // true (special case of loose equality)
0 == false; // true (type coercion happens)

Strict equality:

The === operator checks if two values are strictly equal, meaning it evaluates both value and type without performing type coercion.

Key Points about ===:

Both value and type must match exactly.
Ensures clarity and prevents unexpected behavior.
Handles everything from numbers and strings to arrays and objects, and is hence more reliable.

42 === "42"; // false (different types)
null === undefined; // false (different values and types)
0 === false; // false (different types)

When to Use Each Operator:

Use typeof when inspecting or debugging variable types.
Use === for comparisons to ensure both value and type match.
Avoid == unless you’re confident type coercion will lead to the intended outcome (e.g., when comparing null and undefined).

4. Pure and Impure functions:

A pure function is a function that has no side effects and always returns the same output for a given input (i.e., it has deterministic output).

// pure function
function add(a, b) {
  return a + b; // Only depends on input arguments
}

// impure function
let counter = 0;  
function increment() {
  counter++; // Modifies external state (impure)
}

Benefits of Pure Functions

Easier to Test: Since the output only depends on the input, you don’t need to mock external dependencies. Example: Testing add(2, 3) always results in 5.
Predictability: No unintended consequences from hidden state or side effects.
Improved Debugging: Pure functions are isolated, making it easier to trace issues.
Reusability: Pure functions are self-contained and can be reused confidently in different contexts.
Facilitates Functional Programming: Encourages immutability and declarative programming paradigms.

5. Event propagation:

Event propagation refers to the way in which events are handled in a program.

In JavaScript, events can bubble up through the DOM (Document Object Model) or they can be captured and handled at each level.

Event Handling Efficiency: Use event delegation to manage events on many child elements with a single parent listener.
Debugging Propagation Issues: Understanding propagation helps resolve issues where unintended elements respond to events.
Fine Control: Choose capturing or bubbling phases to handle events at specific points in the DOM hierarchy.
event.stopPropagation(): Stops further propagation in both capturing and bubbling phases.
event.stopImmediatePropagation(): Stops propagation and prevents other listeners on the same element from being called.

With that, I'm concluding this article today. Don't forget to check out my other articles and remember to join the community.

Also, this isn't a definitive list and I'm sure I've missed many concepts that ought to be here, please add them in the comments down below.

Thanks for reading and see you in the next one!

Bye! 👋

~ SS

How to improve DORA metrics as a release engineer

Ibrahim Salami — Tue, 01 Oct 2024 11:03:23 +0000

Ensuring efficient, reliable, high-quality software releases is crucial in software development. This is where release engineering comes into play. This blog will explore release engineering, its importance, and how release engineers can significantly influence key DevOps Research and Assessment (DORA) metrics.

What is Release Engineering?

Release engineering is a specialized discipline within software development focused on the processes and practices that ensure software is built, packaged, and delivered efficiently and reliably. It involves coordinating various aspects of software creation, from source code management to deployment.

A release engineer ensures that software releases are smooth and efficient, maintaining high standards of quality and reliability. They manage the build and deployment pipelines, automate repetitive tasks, and work closely with development, operations, and QA teams.

Key Components of Release Engineering

Version Control: Manage code changes using systems like Git and implement branching strategies.

Build Automation: Utilizing tools like Maven, Gradle, or Make to automate the build process alongside CI tools like Jenkins or GitHub Actions.

Artifact Management: Storing build artifacts in repositories such as JFrog Artifactory, Nexus, or AWS S3.

Testing: Implementing automated testing strategies, including unit, integration, and end-to-end tests.

Deployment Automation: Using CD tools like Spinnaker or ArgoCD to automate deployments, managed with IaC tools like Terraform or Ansible.

Configuration Management: Handling environment-specific configurations with tools like HashiCorp Consul or AWS Parameter Store.

Monitoring and Logging: Employing tools like Prometheus, Grafana, or the ELK Stack to monitor performance and centralized logging.

Importance of Release Engineering

Release engineering is crucial for:

Ensuring efficient and reliable software releases – Streamlined processes reduce downtime and ensure consistent releases.
Reducing human error through automation – Automation minimizes the risk of errors, ensuring more predictable outcomes.
Enhancing collaboration – Bridging gaps between development, operations, and QA teams improves overall workflow.
Quick Rollback and Recovery Mechanisms-Effective release engineering ensures that issues can be swiftly addressed and systems restored.

DORA (DevOps Research and Assessment) metrics are essential performance indicators used to check the effectiveness of software delivery and operational practices. They provide insights into the performance and health of DevOps processes.

Importance of DORA Metrics

DORA metrics are essential because they help organizations understand their software delivery performance, identify areas for improvement, and drive continuous improvement. They offer a data-driven approach to enhancing efficiency and reliability.

Key DORA Metrics

Deployment Frequency: Deployment frequency measures how often new code is deployed to production. Higher frequency indicates a more agile and responsive development process.
Lead Time for Changes: Lead time for changes measures the duration from when a code change is committed until it is deployed to production. Shorter lead times indicate a more efficient development pipeline.
Change Failure Rate: The duration from a code commit to its successful deployment in production. Change failure rate indicates the percentage of deployments that lead to a failure in the production environment. Lower rates indicate more reliable releases.
Mean Time to Recovery (MTTR): MTTR calculates the duration required to restore service following a failure. A lower MTTR signifies a more resilient and responsive system.

Real-world Implementation

We will use a Powershell script to calculate the four critical metrics from Azure DevOps pipelines. The computed result will be stored in a Log Analytics Workspace. We will use Grafana as the data visualization tool to plot the Dashboard.

Below is the sample dashboard we can see after adding Azure data sources in Grafana. Snippets from the PowerShell scripts used to compute each metric are also below.

The complete code can be found at:

https://github.com/rajputrishabh/DORA-Metrics

Calculating Mean Time to Recovery

To calculate MTTR, sum up the time taken to recover from all incidents over time and divide by the number of incidents.

MTTR = Total downtime / Number of incidents

#calculate MTTR per day
  if($maintainencetime -eq 0){
    $maintainencetime=1
  }
  if($failureCount -gt 0 -and $noofdays -gt 0){
    $MeanTimetoRestore=($maintainencetime/$failureCount)
  }
  $dailyDeployment=1
  $hourlyrestoration=(1/24)
  $weeklyDeployment=(1/7)
 
  #calculate Maturity
  $rating=""

  if($MeanTimeToRestore -eq 0){
  $rating=" NA"
  }
  elseif($MeanTimeToRestore -lt $hourlyrestoration){
    $rating="Elite"
  }
  elseif($MeanTimeToRestore -lt $dailyDeployment){
    $rating="High"
  }
  elseif($MeanTimeToRestore -lt $weeklyDeployment){
    $rating ="Medium"
  }
  elseif($MeanTimeToRestore -ge $weeklyDeployment){
  $rating="Low"
  } 
  if($failureCount -gt 0 -and $noofdays -gt 0){
    Write-Output "Mean Time to Restore of $($pipelinename) for $($stgname) for release id $($relid)
 over last $($noofdays) days, is $($displaymetric) $($displayunit), with DORA rating of '$rating'"
  }
  else{
    Write-Output "Mean Time to Restore of $($pipelinename) for $($stgname) for release id $($relid) 
 over last $($noofdays) days ,is $($displaymetric) $($displayunit), with DORA rating of '$rating'"
}

Calculating Deployment Frequency

Count the number of deployments to production over a specific period to calculate deployment frequency.

Deployment Frequency = Number of deployments / Time period

#calculate DF per day
  $deploymentsperday=0
  if($releasetotal -gt 0 -and $noofdays -gt 0){
  $deploymentsperday=$timedifference/$releasetotal
  }

  $dailyDeployment=1
  $weeklyDeployment=(1/7)
  $monthlyDeployment=(1/30)
  $everysixmonthDeployment=(1/(6*30))
  $yearlyDeployment=(1/365)
 
  #calculate Maturity
  $rating=""
  if($deploymentsperday -eq 0){
    $rating=" NA"
  }
  elseif($deploymentsperday -lt $dailyDeployment){
    $rating="Elite"
  }
  elseif($deploymentsperday -ge $dailyDeployment -and  $deploymentsperday -gt 
 $weeklyDeployment){
    $rating="High"
  }
  elseif($deploymentsperday -ge $weeklyDeployment -and $deploymentsperday -gt 
 $monthlyDeployment){
      $rating ="Medium"
  }
  elseif($deploymentsperday -ge $monthlyDeployment -and  $deploymentsperday -ge 
 $everysixmonthDeployment){
      $rating="Low"
  }
  if($releasetotal -gt 0 -and $noofdays -gt 0){
    Write-Output "Deployment frequency of $($pipelinename) for $($stgname)  for release id $($relid) 
 over last $($noofdays)  days, is $($displaymetric) $($displayunit), with DORA rating of  '$rating'"
  }
  else{
    Write-Output "Deployment frequency of $($pipelinename)  for $($stgname)  for release id $($relid)
 over last $($noofdays)  days, is $($displaymetric) $($displayunit), with DORA rating of '$rating'"
  }

Calculating Change Failure Rate

To calculate the change failure rate, divide the number of failed deployments by the total number of deployments and multiply by 100 to get a percentage.

Change Failure Rate (%) = (Failed deployments / Total deployments) * 100

The PowerShell script to calculate CFR is in the repository linked above.

Calculating Lead Times for Changes

To calculate the lead time for changes, measure the time from code commit to deployment for each change and calculate the average.

Lead Time for Changes = Sum of (Deployment time – Commit time) / Number of changes

The PowerShell script to calculate LTC can be found in the repository linked above.

How Release Engineers Can Influence DORA Metrics

Release engineers play a pivotal role in shaping and improving key DORA metrics, which are crucial for assessing the efficiency and reliability of software delivery. Below, we delve into practical strategies with real-world examples from companies like Etsy, Google, Netflix, and Amazon to illustrate how release engineers can positively impact Deployment Frequency, Change Failure Rate, Lead Time for Changes, and Mean Time to Recovery.

Improving Deployment Frequency – Etsy

Example: Implementing CI/CD Pipelines at Etsy

Strategy: To enhance deployment frequency, Etsy adopted continuous integration and continuous deployment (CI/CD) practices and several tools, such as Try and Deployinator.

Implementation

Automation: They automated their build, test, and deployment processes using Jenkins and custom scripts, enabling multiple daily deployments.

Feature Toggles: Introduced feature toggles to safely deploy incomplete features without affecting end users.

Outcome

Etsy achieved the capability to deploy code changes to production around 50 times a day, significantly increasing their deployment frequency.

Reducing Change Failure Rate – Google

Example: Comprehensive Testing at Google

Strategy: Google emphasizes comprehensive automated testing to reduce the change failure rate.

Implementation

Testing: Google integrated unit tests, integration tests, and end-to-end tests into its CI pipeline. It uses tools like GoogleTest and Selenium for various levels of testing.

Code Reviews: Established a rigorous code review process where peers review each change before it is merged, ensuring high code quality.

Outcome

By catching issues early in the development process, Google reduced the number of failed deployments, lowering their change failure rate.

https://testing.googleblog.com/

Shortening Lead Time for Changes – Netflix

Example: Streamlined Build Process at Netflix

Strategy: Netflix optimized its build and deployment processes to shorten the lead time for changes.

Implementation

Optimized Pipelines: Netflix used Spinnaker, an open-source multi-cloud continuous delivery platform, to streamline their deployment pipelines.

Microservices Architecture: Adopted a microservices architecture, which allowed smaller, more manageable changes to be deployed independently.

Outcome

Netflix reduced its lead time for changes from days to minutes, allowing for rapid iteration and deployment.

Reducing Mean Time to Recovery (MTTR) – Amazon

Example: Robust Monitoring and Quick Rollback at Amazon

Strategy: Amazon focuses on robust monitoring and quick rollback mechanisms to minimize MTTR.

Implementation

Monitoring: Extensive monitoring was implemented using AWS CloudWatch, enabling proactive detection of issues.

Rollback Mechanisms: Developed automated rollback procedures using AWS Lambda functions and CloudFormation scripts to revert to a previous stable state quickly.

Outcome

Amazon reduced their MTTR significantly, ensuring quick recovery from incidents and maintaining high service availability.

https://aws.amazon.com/documentation/

Deployment Frequency and Lead Time for Changes evaluate the speed of delivery, whereas Change Failure Rate and Time to Restore Service evaluate stability. By tracking and continuously improving these metrics, teams can achieve significantly better business results. Based on these metrics, DORA categorizes teams into Elite, High, Medium, and Low performers, finding that Elite teams are twice as likely to achieve or surpass their organizational performance goals.

Pitfalls of DORA Metrics for Release Engineers

While DORA metrics provide valuable insights into software delivery performance and operational practices, they come with challenges and potential pitfalls. Understanding these can help release engineers avoid common mistakes and make more informed decisions.

Overemphasis on Metrics Over Quality

Pitfall: Focusing solely on improving DORA metrics can lead to overlooking the overall quality of the software. Teams might rush changes to increase deployment frequency or reduce lead time, compromising the product’s robustness and security. This is a classic case of “Goodhart’s Law, which states that when a measure becomes a target, it ceases to be a good measure”.

Solution: Balance the focus on metrics with a commitment to maintaining high-quality standards. Implement thorough testing and code review processes to ensure quality is not sacrificed for speed.

Misinterpreting Metrics

Pitfall: DORA metrics can be misinterpreted without context. For example, a high deployment frequency might seem optimistic but could indicate frequent hotfixes for recurring issues, highlighting underlying problems rather than improvements.

Solution: Analyze metrics within the context of overall performance and other relevant data. Use complementary metrics and qualitative insights to view the team’s effectiveness comprehensively.

Neglecting Team Morale

Pitfall: Intense focus on improving DORA metrics can result in burnout and decreased morale among team members. Pushing for more frequent deployments or faster lead times without considering workload can negatively impact the team’s well-being.

Solution: Foster a healthy work environment by setting realistic goals and ensuring adequate support and resources for the team. Encourage open communication about workloads and stress levels.

Lack of Actionable Insights

Pitfall: Collecting and reporting DORA metrics without deriving actionable insights can lead to data without purpose. Teams might track metrics but fail to implement changes based on the findings.

Solution: Review and analyze DORA metrics regularly to identify trends and areas for improvement. Using the insights obtained from the metrics, develop and execute action plans.

Insufficient Tooling and Automation

Pitfall: Inadequate tooling and automation can hinder efforts to improve DORA metrics. Manual processes and outdated tools can slow down deployments and increase lead times.

Solution: Invest in modern CI/CD tools, automated testing frameworks, and infrastructure as code solutions. Continuously evaluate and update the toolchain to ensure it supports efficient workflows.

Conclusion

Release engineering is a cornerstone of modern software development, ensuring that software is released efficiently, reliably, and with high quality. Release engineers can significantly enhance their software delivery performance by understanding and effectively utilizing DORA metrics. However, it’s essential to be mindful of the potential pitfalls and to balance metric improvement with maintaining overall quality and team morale. Best practices and utilizing appropriate tools can help release engineers drive meaningful improvements and achieve better outcomes.

To effectively influence these metrics, release engineers should focus on:

Automation: Automate build, test, and deployment processes using robust CI/CD pipelines to increase deployment frequency and reduce lead times.
Comprehensive Testing: Implement comprehensive automated testing to catch issues early and lower the change failure rate.
Efficient Rollback Mechanisms: Establish quick rollback strategies and robust monitoring to minimize MTTR.
Continuous Improvement: Regularly review and iterate on processes based on DORA metrics to foster continuous improvement and ensure high-quality software delivery.

Frequently Asked Questions

Q1: What are DORA metrics?

DORA (DevOps Research and Assessment) metrics are essential performance indicators for evaluating the effectiveness of software delivery and operational practices. The four main DORA metrics are Deployment Frequency (DF), Lead Time for Change, Change Failure Rate, and Mean Time to Recovery (MTTR).

Q2: Why are DORA metrics important?

DORA metrics provide valuable insights into the performance and health of software delivery processes. They help identify bottlenecks, measure improvements, and drive continuous improvement in DevOps practices, leading to more efficient and reliable software delivery.

Q3: How often should I review and analyze DORA metrics?

Regularly review DORA metrics, ideally on a weekly or bi-weekly basis, to continuously monitor performance and identify areas for improvement. Use these reviews to inform decisions and drive ongoing enhancements in the software delivery process.

Q4: What tools can help improve DORA metrics?

CI/CD Tools: Jenkins, GitHub Actions, GitLab CI, CircleCI
Build Automation Tools: Maven, Gradle, Make, Ant
Artifact Management: JFrog Artifactory, Nexus, AWS S3
Configuration Management: HashiCorp Consul, Spring Cloud Config, AWS Parameter Store
Monitoring and Logging: Prometheus, Grafana, New Relic, ELK Stack

Q5: How can I measure the current state of my DORA metrics?

Deployment Frequency: Measure the number of deployments within a defined timeframe.
Lead Time for Changes: Measure the time from code commit to production deployment.
Change Failure Rate: Divide the number of failed deployments by the total deployments.
Mean Time to Recovery: Track and average the time from incident detection to resolution.

Q6: What is release engineering?

Release engineering is a discipline within software development focused on the processes and practices for building, packaging, and delivering software efficiently and reliably. It involves coordinating various aspects of software creation, from source code management to deployment.

How to configure IAM using Terraform

Ibrahim Salami — Fri, 06 Sep 2024 15:36:29 +0000

Organizations or individuals typically manage IAM using consoles and hesitate to use Infrastructure-as-code (IaC) as it is complex and sensitive to define IAM policies due to security risks. With frequent dynamic changes, you do not get immediate feedback. And more expertise is needed to configure and manage IAM rules with IaC. However, configuring IAM though IaC also have several benefits.

In this blog we’ll explore those benefits, discuss strategies for IAM management via Terraform, explain why implementing Zero Trust policies within IAM is crucial for security, and how to enforce IAM best practices, Policy-as-code, and IAM governance.

Why manage IAM through Infrastructure-as-code?

Automation and consistency – It offers automation, consistency, repeatability, and versioning to IAM policies and role management.
Audit trails – It allows you to maintain a comprehensive audit trail of changes to IAM configurations. This helps with compliance requirements and allows you to easily track who made changes when they were made, and why.
Least Privileges – Terraform’s expressive language allows for defining complex IAM policies with fine-grained control over permissions. Teams can more easily provision their own access in a controlled manner through pull requests, which then undergo a review process before being applied, fostering a self-service infrastructure model.

Zero Trust policies within IAM

Identity and Access Management (IAM) is a critical component of Zero-Trust security, assuming you do not trust anybody. Zero trust in IAM means:

Never Trust, Always Verify – There is no automatic trust. Always verify everyone who is trying to access the resource.
Least Privilege Access – Limits access to resources to the minimum necessary to perform a specific task and reduce the blast radius. This is to stop people from allowing unnecessary permissions to users/roles, which can cause breaking changes.
MFA – Multi-factor authentication enables the extra security layer on IAM users/roles.

Setting Up AWS IAM Policies with Terraform

Setting up AWS IAM policies with Terraform involves defining your IAM resources in Terraform configuration files, applying best practices for security and organization, and using Terraform’s capabilities to manage these resources as code. Below, we’ll outline a basic approach to setting up IAM policies in AWS using Terraform, including an example configuration.

In this blog post, we will cover the Terraform configs that are also compatible with OpenTofu.

Prerequisites

Before you start, ensure you have the following:

Terraform is installed on your machine.
An AWS account and AWS CLI configured with access credentials.
Basic knowledge of IAM concepts (e.g., policies, roles, users) and Terraform syntax.

Step 1: Initialize Terraform Project

Create a new directory for your Terraform project and initialize it with a main.tf file. Then, run terraform init in the project directory to prepare your directory for Terraform operations.

Step 2: Define the AWS Provider

In main.tf, start by defining the AWS provider. This specifies which version of the AWS provider to use and configures the region and other provider settings.

provider “aws” {
  region = “us-east-1”
  #AWS account access keys credentials
  access_key = “A***************”
  secret_key = “U******************”
}

Step 3: Define IAM Policy

Define an IAM policy using the aws_iam_policy resource. You need to provide a name and a policy document. The policy document can be defined inline using the <<EOF … EOF syntax, or it can be loaded from a file using the file() function.

Example of an inline policy definition:

# Create IAM policy to allow S3 read access
resource “aws_iam_policy” “s3_read_policy” {
  name        = “s3_read_policy”
  description = “Allows read access to files in the specified S3 bucket”
  policy      = <<EOF
{
  “Version”: “2012-10-17”,
  “Statement”: [
    {
      “Effect”: “Allow”,
      “Action”: [
        “s3:GetObject”,
        “s3:ListBucket”
      ],
      “Resource”: [
        “arn:aws:s3:::your-bucket-name/*”,
        “arn:aws:s3:::your-bucket-name”
      ]
    }
  ]
}
EOF
}

We can see the iam_policy creation by running the terraform plan command.

terraform plan command

output from terraform plan

Step 4: Create IAM user

Define an IAM user

# Create IAM user
resource “aws_iam_user” “iam_user” {
  name = “iam_user”
}

We can see the iam_user creation by running terraform plan command.

output for terraform plan

Step 5: Create an IAM Role and Attach the Policy

Define an IAM role and set assume role policy to allow the IAM user to assume the role

# Create an IAM role
resource “aws_iam_role” “iam_role” {
  name = “iam-role”
  assume_role_policy = jsonencode({
    Version = “2012-10-17”,
    Statement = [{
      Action = “sts:AssumeRole”,
      Principal = {
        AWS = “arn:aws:iam::AWS_ACCOUNT_ID:user/${aws_iam_user.iam_user.name}” # Replace [AWS_ACCOUNT_ID] with Account’s AWS account ID
      },
      Effect = “Allow”,
      Sid    = “ ”
    }]
  })
}

We can see the iam_role creation by running terraform plan command.

output for terraform plan

Step 6: Attach IAM policy to IAM role

# Attach IAM policy to IAM role
resource “aws_iam_policy_attachment” “s3_read_attach” {
  roles       = [aws_iam_role.iam_role.name]
  policy_arn = aws_iam_policy.s3_read_policy.arn
  name     = “Attaching s3 policy to iam role”
}

We can see the iam_role creation by running terraform plan command.

iam s3 policy – terraform plan output

Step 7: Apply Configuration

The Terraform configuration has been defined. Apply it using the command terraform apply in your project directory. This command will prompt you to review the proposed changes and confirm them. Upon confirmation, Terraform will create the resources in AWS according to your configuration.

After running the final terraform apply command, we can see the iam-role, iam_user, and s3ReadPolicy resources.

Utilizing Terraform’s templatefile function for dynamic policy generation

The templatefile() function in Terraform allows you to dynamically generate configuration files using templates. You can use this function to generate IAM policy documents dynamically, which can be helpful in cases where policies need to be customized based on dynamic inputs.

Here’s an example of how you can use the templatefile() function to dynamically generate an IAM policy document for S3 read access:

provider “aws” {
  region = “us-east-1”
  #AWS account access keys credentials
  access_key = “A***************”
  secret_key = “U******************”
}

# Define IAM policy template. This works prior to terraform 0.12
data “template_file” “s3_read_policy_template” {
  template = <<EOF
{
  “Version”: “2012-10-17”,
  “Statement”: [
    {
      “Effect”: “Allow”,
      “Action”: [
        “s3:GetObject”,
        “s3:ListBucket”
      ],
      “Resource”: [
        “${bucket_arn}/*”,
        “${bucket_arn}”
      ]
    }
  ]
}
EOF
  vars = {
    bucket_arn = “arn:aws:s3:::your-bucket-name”
  }
}

You can also store the above template snippet into a separate s3_read_policy.tmpl template file for Terraform above version 0.12 and reference it as shown below:

# Content of s3_read_policy.tmpl file
 {
  “Version”: “2012-10-17”,
  “Statement”: [
    {
      “Effect”: “Allow”,
      “Action”: [
        “s3:GetObject”,
        “s3:ListBucket”
      ],
      “Resource”: [
        “${bucket_arn}/*”,
        “${bucket_arn}”
      ]
    }
  ]
}

# Create IAM policy using the template
resource “aws_iam_policy” “s3_read_policy” {
  provider = aws.account_a
  name     = “s3_read_policy”
  policy   = templatefile( “${path.module}/template_file.tpl”,
{
  bucket_arn = “arn:aws:s3:::BUCKET_NAME” #provide the s3 bucket name
} )
}

We can see the s3_read_policy creation by running terraform plan command.

iam s3 read policy – terraform plan output

# Create IAM user
resource “aws_iam_user” “iam_user” {
  name     = “iam_user”
}

We can see the iam_user creation by running terraform plan command.

output for terraform plan

# Create an IAM role
resource “aws_iam_role” “reader_role” {
  name = “reader_role”
  assume_role_policy = jsonencode({
    Version   = “2012-10-17”,
    Statement = [{
      Action    = “sts:AssumeRole”,
      Principal = {
        AWS = “arn:aws:iam::AWS_ACCOUNT_ID:user/${aws_iam_user.iam_user.name}” # Replace [AWS_ACCOUNT_ID] with Account’s AWS account ID
      },
      Effect    = “Allow”,
      Sid       = “AssumeRole“
    }]
  })
}

We can see the iam_role creation by running terraform plan command.

output for terraform plan

# Attach IAM policy to IAM role
resource “aws_iam_policy_attachment” “s3_read_attach” {
  name       = “s3_read_attach”
  roles      = [aws_iam_role.reader_role.name]
  policy_arn = aws_iam_policy.s3_read_policy.arn
}

We can see the iam_policy_attachment creation by running terraform plan command.

iam s3 policy – terraform plan output

This configuration reads the content of the s3_read_policy.tmpl file using the file function and then using it to create the IAM policy. You can adjust the file path, policy name, and bucket ARN per your use case.

This approach allows you to generate IAM policies dynamically based on inputs or variables, providing flexibility in your Terraform configurations. Adjust the template and variables as needed for your specific use case.

After running the final terraform apply command, we can see the iam_user, reader-role, and attached s3_read_policy resources have been created in the AWS, as shown below.

Cross-Account Management

Cross-account access is particularly beneficial when organizations maintain multiple AWS accounts for different purposes, such as development, staging, and production.

Benefits of cross-account management

Without cross-account access, managing user access across these accounts can become complex and cumbersome.
It helps to enforce least privilege principles by allowing administrators to define granular access controls using IAM roles. This ensures users can only access the resources required for their roles or responsibilities.
Cross-account access facilitates centralized user management and auditing.
Administrators can create and manage users centrally in an AWS management account, reducing the administrative overhead of managing user identities across multiple accounts.
Auditing and tracking user access become more straightforward as all access requests and actions are logged centrally in the AWS management account.
Cross-account access is crucial for ensuring streamlined operations in large organizations where a security team manages multiple AWS accounts centrally.

Unlocking Cross-Account Access in AWS with Terraform

Let’s use Terraform to create an IAM user in AWS Account A and establish cross-account access with the “AssumeRole” action.

In this example, we’ll create an IAM user in AWS Account A and configure a cross-account role in AWS Account B that allows the IAM user in AWS Account A to assume it and allow the CrossAccountUser in AWS Account A to read files from buckets in AWS Account B. We’ll need to define an IAM policy granting the necessary permissions and attach that policy to the cross-account role in AWS Account B.

Here’s how you can achieve this:

Set alias for Account A

provider “aws” {
  region = “us-east-1”
#AWS account A access key credentials
 access_key = “A***************”
 secret_key = “U******************”
  alias  = “account_a”
}

Create an IAM user in AWS Account A

resource “aws_iam_user” “cross_account_user” {
  provider = aws.account_a
  name = “CrossAccountUser”
}

Setup AWS Account B

provider “aws” {
  region = “us-east-1”
 #AWS account B access key credentials
 access_key = “A***************”
 secret_key = “U******************”
  alias  = “account_b”
}

Define an IAM role in AWS Account B

resource “aws_iam_role” “cross_account_role” {
  provider = aws.account_b
  name = “CrossAccountRole”
  assume_role_policy = jsonencode({
    Version   = “2012-10-17”,
    Statement = [{
      Effect    = “Allow”,
      Principal = {
        AWS = “arn:aws:iam::Account_A_ID:user/CrossAccountUser”  # Replace [Account A ID] with AWS Account A’s AWS account ID
      },
      Action    = “sts:AssumeRole”,
    }]
  })
}

Define IAM policy to allow reading files from S3 buckets in Account B

# Replace "bucket-name" with the name of your bucket in AWS Account B
resource “aws_iam_policy” “s3_read_policy” {
  provider = aws.account_b
  name = “S3ReadPolicy”
  policy = <<EOF
{
  “Version”: “2012-10-17”,
  “Statement”: [{
    “Effect”: “Allow”,
    “Action”: [
      “s3:GetObject”,
      “s3:ListBucket”
    ],
    “Resource”: [
      “arn:aws:s3:::bucket-name/*”,  
      “arn:aws:s3:::bucket-name”
    ]
  }]
}
EOF
}

Attach IAM policy to the IAM role in AWS Account B

resource “aws_iam_role_policy_attachment” “s3_read_attach_policy” {
  provider = aws.account_b
  role       = aws_iam_role.cross_account_role.name
  policy_arn = aws_iam_policy.s3_read_policy.arn
}

Make sure to replace Account_A_ID with the Account A AWS account ID and “bucket-name” with the name of the s3 bucket in AWS Account B that you want to grant access to. Also, ensure that the bucket policy in AWS Account B allows access from the role CrossAccountRole.

With this setup, the CrossAccountUser in Account A can assume the CrossAccountRole in Account B and access files from the specified s3 bucket in Account B.

Enforcing IAM Best Practices with Policy-as-Code

Enforcing IAM Best Practices with Policy-as-Code ensures that security policies are consistently applied across an organization’s cloud infrastructure. By codifying IAM policies, teams can automate enforcing security controls, reducing the risk of misconfigurations and unauthorized access.

checkov is one of the Policy-as-Code tools available in cloud security. It is an open-source static code analysis tool developed by Bridgecrew.

It scans infrastructure as code (IaC) templates like Terraform and CloudFormation to detect security and compliance issues early. By analyzing configurations against predefined policies and industry standards, Checkov helps identify misconfigurations, vulnerabilities, and compliance violations. It focuses on cloud security, particularly in AWS, Azure, and GCP environments, and integrates seamlessly into CI/CD pipelines for proactive issue remediation before deployment.

Utilizing Checkov, highlighting its ability to detect IAM configuration issues early, focusing on preventing overly permissive policies.

Regarding IAM configuration issues, Checkov plays a crucial role in detecting overly permissive policies early in the development process.

Here’s how Checkov helps in detecting IAM configuration issues, mainly focusing on preventing overly permissive policies:

Static Analysis with Checkov: Configuring Checkov for IAM policy scans.

Let us set up checkov to scan this setup for potential security risks and misconfigurations using the Terraform code example

Example Terraform code we’ll be analyzing:

AWS Provider Configuration – Set the AWS region to us-east-1.

provider “aws” {
  region = “us-east-1”
  access_key = “A*********************”
  secret_key =   “U****************************”
}

IAM Policy for S3 Read Access – Create an IAM policy named s3_read_policy that allows read access (s3:GetObject, s3:ListBucket) to a specified S3 bucket.

resource “aws_iam_policy” “s3_read_policy” {
  name        = “s3_read_policy”
  description = “Allows read access to files in the specified S3 bucket”
  policy      = <<EOF
{
  “Version”: “2012-10-17”,
  “Statement”: [
    {
      “Effect”: “Allow”,
      “Action”: [
        “s3:GetObject”,
        “s3:ListBucket”
      ],
      “Resource”: [
        “arn:aws:s3:::your-bucket-name/*”,
        “arn:aws:s3:::your-bucket-name”
      ]
    }
  ]
}
EOF
}

IAM User Creation – Define an IAM user named iam_user.

resource “aws_iam_user” “iam_user” {
  name = “iam_user”
}

IAM Role and Assume Role Policy – Set up an IAM role named iam-role with an assume role policy that allows the iam_user to assume this role.

resource “aws_iam_role” “iam_role” {
  name = “iam-role”
  assume_role_policy = jsonencode({
    Version = “2012-10-17”,
    Statement = [{
      Action = “sts:AssumeRole”,
      Principal = {
        AWS = “arn:aws:iam::AWS_ACCOUNT_ID:user/${aws_iam_user.iam_user.name}”
      },
      Effect = “Allow”,
      Sid    = “AssumeRole”
    }]
  })
}

Policy Attachment – Attaches the s3_read_policy to the iam_role.

resource “aws_iam_policy_attachment” “s3_read_attach” {
  roles       = [aws_iam_role.iam_role.name]
  policy_arn = aws_iam_policy.s3_read_policy.arn
  name     = “Attaching s3 policy to iam role”
}

Configuring Checkov for IAM Policy Scans

Install Checkov – First, ensure that checkov is installed in your environment. If not, install it via pip by running

pip install checkov

Run checkov Scan -checkov will provide a detailed report of any issues, including security vulnerabilities, best practice violations, and compliance issues. In our example, checkov can help identify potential risks in IAM policies, such as overly broad permissions, and suggest mitigations, etc.

For a file – You can run it for one single file using checkov --file file_name.tf command.

For a directory – You can go to the directory containing your Terraform files to scan all files within the directory and run command.

checkov –d .

checkov output

Refining the Scan – If you want to focus specifically on IAM-related checks, you can use checkov’s –check flag to include or exclude certain checks based on their IDs, tailoring the scan to your specific needs. For example, to ensure IAM policies that allow full “*-*” administrative privileges are not created, you can run:

checkov -d . --check CKV_AWS_62

Output for Terraform code example scan

Results – By running Checkov as described, we can identify potential security issues such as:

Excessive permissions in the IAM policy.
IAM policies are attached directly to users instead of roles.
Missing or overly permissive assume role policies.

Addressing these issues involves modifying the Terraform code to adhere to best practices, such as implementing least privilege, using roles for cross-account access, and ensuring policies are scoped appropriately.

Custom Policies – checkov allows you to enforce specific security or compliance requirements that Checkov’s built-in checks might not cover. This is helpful if your organization has specific compliance requirements that the default checks do not cover.

Integrating Policy Scans in CI/CD: Automating IAM policy compliance checks before deployment.

policy compliance with CI/CD

Wrapping with IAM governance & best practices

Focusing on Identity and Access Management (IAM) governance and best practices is essential for ensuring the security and compliance of cloud environments. This approach helps systematically manage digital identities, their authentication, authorization, roles, and privileges within or across system and enterprise boundaries.

Integrating IAM Governance

IAM governance should be an integral part of any organization’s security strategy. It involves several key components:

Organizations should strive for centralized management of user identities and their access across all systems and platforms. This simplifies the enforcement of access policies and compliance with regulatory requirements.
Assigning permissions based on roles tightly aligned with organizational structures and job functions streamlines access management and enforces the principle of least privilege.
Conducting regular audits of IAM policies and practices helps identify and remediate unused or excessive permissions and ensures compliance with relevant standards and regulations.
Implementing robust processes for the entire lifecycle of user identities – from creation through management, to deletion – ensures that access rights are always up to date and reduces the risk of orphaned accounts.

IAM Best Practices

To enhance IAM governance, organizations should adhere to a set of best practices:

Ensuring that users have only the minimum levels of access required to perform their functions minimizes potential damage from errors or malicious intent.
Use multi-factor authentication (MFA) and strong password policies to enhance security. For critical resources, consider additional authentication factors and stringent authorization checks.
Separate roles and responsibilities to prevent conflicts of interest or fraud. This is crucial in preventing any single point of compromise.
Automate the process of granting and revoking access to minimize the risk of oversight.
Leverage managed policies for easier administration and reuse of standard permission sets.

Methods to perform compliance audits on IAM configurations

Open Policy Agent (OPA) is an open-source, general-purpose policy engine that unifies policy enforcement across the cloud-native stack. It can be incorporated into IaC workflows. OPA enables you to craft policies that govern and secure your cloud environments without embedding policy logic within your applications and enhances their security, compliance, and governance.

OPA Policy Example for Terraform

For creating an Open Policy Agent (OPA) policy relevant to the provided Terraform code example (which involves IAM policies for s3 read access, IAM user, and IAM role creations), we’ll focus on enforcing a rule that IAM policies should specify a specific s3 bucket and not allow broad access.

OPA Policy for Specific S3 Bucket Access in IAM Policies

OPA policies are written in a high-level declarative language called Rego. This policy aims to ensure that any IAM policy granting access to s3 buckets explicitly specifies the bucket name, rather than allowing access to all buckets.

Define a Rego policy file, e.g., iam_policy.rego, that includes the rule to check IAM policy statements for specific S3 bucket access.

package terraform.analysis
default allow = false
# Rule to check for specific bucket access in IAM policies
allow {
    some i
    input.resource.aws_iam_policy[i].policy
    policy := json.unmarshal(input.resource.aws_iam_policy[i].policy)
    policy.Statement[_].Effect == “Allow”
    action_allowed(policy.Statement[_].Action)
    not wildcard_bucket_access(policy.Statement[_].Resource)
}

# Helper to check if actions related to S3 read are allowed
action_allowed(actions) {
    allowed_actions := [“s3:GetObject”, “s3:ListBucket”]
    allowed_actions[_] == actions[_]
}

# Helper to check for wildcard bucket access
wildcard_bucket_access(resources) {
    resources[_] == “arn:aws:s3:::*”
}

This Rego policy does the following:

Checks IAM policies: It looks for aws_iam_policy resources in the Terraform plan.
Parses the policy JSON: It unmarshals the JSON policy document to inspect the policy statements.
Evaluate policy statements: It checks if any “Allow” statements permit s3:GetObject or s3:ListBucket actions.
Ensures specific bucket access: It ensures that resources do not include a wildcard (arn:aws:s3:::*), indicating that the policy specifies particular buckets.

Using the OPA Policy:

To use this policy, you would typically evaluate it against your Terraform plan output in JSON format, using the opa eval command. First, generate the Terraform plan:

terraform plan -out=tfplan.binary
terraform show -json tfplan.binary > tfplan.json

Then, evaluate your policy with OPA:

opa eval --format pretty --data iam_policy.rego --input tfplan.json “data.terraform.analysis.allow”

This OPA policy scrutinizes your Terraform plan, specifically checking whether IAM policies for S3 access are narrowly scoped to specific buckets. Enforcing such policies ensures that your cloud environment adheres to security best practices, significantly mitigating potential risks.

Conclusion

We’ve explored the nuances of managing AWS IAM through Terraform, highlighting its significance in bolstering cloud security, IAM configurations as Infrastructure-as-code, and the critical role of Zero-Trust policies within IAM. Setting up IAM policies, creating users and roles, and managing cross-account access and trust relationships.

The exploration into enforcing IAM best practices through policy-as-code with tools like checkov underscored the transformative impact of static code analysis in preempting configuration errors and security risks.

Finally, we touched upon IAM governance and compliance, underscoring methods like Rego policy definitions with OPA for performing compliance audits on IAM configurations. This ensures alignment with security best practices and regulatory standards, cementing IAM’s role in securing cloud environments.

Commonly Asked Questions

1) What are the best practices for managing AWS IAM policies with Terraform?

Use Least Privilege Principle – Grant only the permissions necessary for a user, group, or role to perform their intended tasks.
Separation of Concerns – Organize IAM policies logically by separating them based on roles, responsibilities, or permissions.
Enable Policy Testing – Implement automated tests to validate IAM policies for correctness and compliance with organizational policies and regulatory requirements.
Rotate IAM Credentials Regularly – Reduce risk and enhance security by automating the rotation of IAM access keys and credentials using AWS Secrets Manager or AWS IAM Access Analyzer.
Use IaC to maintain the IAM policies and changes using Infrastructure-as-a-code to maintain consistency.
Monitor and Audit IAM Changes – Implement and review logging and monitoring of IAM actions and changes using AWS CloudTrail and AWS Config.

2) Can Terraform manage dynamic IAM policies for temporary access?

Yes, Terraform can manage dynamic IAM policies for temporary access using AWS IAM Roles with session policies and AWS Security Token Service (STS)

3) How do I create and manage AWS IAM users and their access keys with Terraform?

Terraform’s AWS provider, iam users, and IAM policies allow you to create and manage AWS IAM users and their access keys.

4) What are instance profiles, and how do they relate to IAM roles in AWS?

Instance profiles associate IAM roles with EC2 instances, allowing the instances to inherit the role’s permissions. When an IAM role is attached to an EC2 instance, the corresponding instance profile is attached. This mechanism enables EC2 instances and other services to securely access AWS resources without requiring long-term credentials like access keys or passwords.

Rethinking code reviews with stacked PRs

Ibrahim Salami — Fri, 06 Sep 2024 15:33:14 +0000

The peer code review process is an essential part of software development. It helps maintain software quality and promotes adherence to standards, project requirements, style guides, and facilitates learning and knowledge transfer.

Code review effectiveness

While the effectiveness is high for reviewing sufficiently small code changes, it drops exponentially with the increase in the size of the change. To sustain the necessary level of mental focus to be effective, large code reviews are exhausting. Usually, the longer the review duration gets, the less effective the overall review becomes:

So why can’t we just restrict the size of the pull requests (PRs)? While many changes can start small, suddenly a small two-line change can grow into a 500-line refactor including multiple back-and-forth conversations with reviewers. Some engineering teams also maintain long-running feature branches as they continue working, making it hard to review.

So, how do we strike the right balance? Simple. Use stacked PRs.

What are stacked PRs?

Stacked pull requests make smaller, iterative changes and are stacked on top of each other instead of bundling large monolith changes in a single pull request. Each PR in the stack focuses on one logical change only, making the review process more manageable and less time-consuming.

We also wrote a post last year explaining how this help represents code changes as a narrative instead of breaking things down by files or features.

Why stacked PRs?

Other than building a culture of more effective code reviews, there are a few other benefits of stacked PRs:

Early code review feedback

Imagine that you are implementing a large feature. Instead of creating the entire feature and then requesting a code review, consider carving out the initial framework and promptly putting it up for feedback. This could potentially save you countless hours by getting early feedback on your design.

Faster CI feedback cycle

Stacked PRs support the shift-left practice because changes are continuously integrated and tested, which allows for early detection and rectification of issues. The changes are merged in bits and pieces catching any issues early vs merging one giant change hoping it does not bring down prod!

Knowledge sharing

Code reviews are also wonderful for posterity. Your code changes are narrating your thought process behind implementing a feature, therefore, the breakdown of changes creates more effective knowledge transfer. It’s easier for team members to understand the changes, which promotes better knowledge sharing for the future.

Staying unblocked

Waiting on getting code reviewed and approved can be a frustrating process. With stacked PRs, the developers can work on multiple parts of a feature without waiting for reviewers to approve previous PRs

What’s the catch?

So, why don’t more developers use stacked PRs for code reviews?

Although this stacked PR workflow addresses both the desired practices of keeping code reviews manageable and developers productive, unfortunately, it is not supported very well natively by either git or GitHub. As a result, several tools have been developed across the open-source community to enable engineers to incorporate this stacking technique into the existing git and GitHub platforms. But stacking the PRs is only part of the story.

Updating

As we get code review feedback and we make changes to part of the stack, we have to now rebase and resolve conflicts at all subsequent branches.

Let’s take an example. Imagine that you are working on a change that requires making a schema change, a backend change, and a frontend change. With that, you can now send a simple schema change for review first, and while that’s being reviewed you can start working on the backend and frontend. Using stacked PRs, all these 3 changes can be reviewed by 3 different reviews.

In this case, you may have a stack that looks like this where demo/schema, demo/backend and demo/frontend represents the 3 branches stacked on top of each other.

So far this makes sense, but what if you got some code review comments on the schema change that requires creating a new commit? Suddenly your commit history looks like this:

Now you have to manually rebase all subsequent branches and resolve conflicts at every stage. Imagine if you have 10 stacked branches where you may have to resolve the conflicts 10 times.

Merging

But that’s not all, merging a PR in the stack can be a real nightmare. You have 3 options squash, merge and rebase to merge a PR. Let’s try to understand what goes behind the scenes in each one.

In the case of a squash commit, Git takes changes from all the existing commits of the PR and rewrites them into a single commit. In this case, no history is maintained on where those changes came from
A merge commit is a special type of Git commit that is represented by a combination of two or more commits. So, it works very similar to a squash commit but it also captures information about its parents. In a typical scenario, a merge commit has two parents: the last commit on the base branch (where the PR is merged) and the top commit on the feature branch that was merged. Although this approach gives more context to the commit history, it inadvertently creates non-linear git-history that can be undesirable.
Finally, in case of a rebase and merge, Git will rewrite the commits onto the base branch. So similar to squash commit option, it will lose any history associated with the original commits.

Typically if you are using the merge commit strategy while stacking PRs, your life will be a bit simpler, but most teams discourage using that strategy to keep the git-history clean. That means you are likely using either a squash or a rebase merge. And that creates a merge conflict for all subsequent unmerged stacked branches.

In the example above, let’s say we squash merge the first branch demo/schema into mainline. It will create a new commit D1 that contains changes of A1 and A2. Since Git does not know where D1 came from, and demo/backend is still based on A2, trying to rebase demo/backend on top of the mainline will create merge conflicts.

Likewise, rebasing demo/frontend after rebasing demo/backend will also cause the same issues. So if you had ten stacked branches and you squash merged one of them, you would have to resolve these conflicts nine times.

We are still just scratching the surface, there are many other use cases such as reordering commits, splitting, folding, and renaming branches, that can create huge overhead to manage when dealing with stacked PRs.

That’s why we built stacked PRs management as part of Aviator.

Why Aviator CLI is different

Think of Aviator as an augmentation layer that sits on top of your existing tooling. Aviator connects with GitHub, Slack, Chrome, and Git CLI to provide an enhanced developer experience.

Aviator CLI works seamlessly with everything else! The CLI isn’t just a layer on top of Git, but also understands the context of stacks across GitHub. Let’s consider an example.

Creating a stack

Creating a stack is fairly straightforward. Except in this case, we use av CLI to create the branches to ensure that the stack is tracked. For instance, to create your schema branch and corresponding PR, follow the steps below.

av stack branch demo/schema
# make schema changes
git commit -a -m "[demo] schema changes"
av pr create

Since Aviator is also connected to your GitHub, it makes it easy for you to visualize the stack.

Or if you want to visualize it from the terminal, you can still do that with the CLI commands:

Updating the stack

Using the stack now becomes a cakewalk. You can add new commits to any branch, and simply run av stack sync from anywhere in the stack to synchronize all branches. Aviator automatically rebases all the branches for you, and if there’s a real merge conflict, you just have to resolve it once.

Merging the stack

This is where Aviator tools easily stand out from any existing tooling. At Aviator, we have built one of the most advanced MergeQueue to manage auto-merging thousands of changes at scale. Aviator supports seamless integration with the CLI and stacked PRs. So to merge partial or full stack of PRs, you can assign them to Aviator MergeQueue using CLI av pr queue or by posting a comment in GitHub: /aviator stack merge.

Aviator automatically handles validating, updating, and auto-merging all queued stacks in order.

Now when the PRs are merged, you can this time run av stack sync --trunk to update all PRs and clean out all merged PRs.

Shift-Left is the future

Stacked PRs might initially seem like more work due to the need to break down changes into smaller parts. However, the increase in code review efficiency, faster feedback loops, and enhanced learning opportunities will surely outweigh this overhead. As we continue embracing the shift-left principles, stacked PRs will become increasingly useful.

The Aviator CLI provides a great way to manage stacked PRs with a lot less tedium. The CLI is open-source and completely free. We would love for you to try it out and share your feedback on our discussion board.

At Aviator, we are building developer productivity tools from first principles to empower developers to build faster and better.

Scanning AWS S3 Buckets for Security Vulnerabilities

Ibrahim Salami — Tue, 27 Aug 2024 18:55:23 +0000

All cloud providers offer some variations of file bucket services. These file bucket services allow users to store and retrieve data in the cloud, offering scalability, durability, and accessibility through web portals and APIs. For instance, AWS offers Amazon Simple Storage Service (S3), GCP offers Google Cloud Storage, and DigitalOcean provides Spaces. However, if unsecured, these file buckets pose a major security risk, potentially leading to data breaches, data leakages, malware distribution, and data tampering. For example, the United Kingdom Council’s data on member’s benefits was exposed by an unsecured AWS bucket. In another incident in 2021, an unsecured bucket belonging to a non-profit cancer organization exposed sensitive images and data for tens of thousands of individuals.

Thankfully, S3Scanner can help. S3Scanner is a free and easy-to-use tool that can help you identify and fix unsecured file buckets in all major cloud providers: Amazon S3, Google Cloud Storage, and Spaces:

In this article, you’ll learn all about S3Scanner and how it can help identify unsecured file buckets on multiple cloud providers.

Common Security Risks in Amazon S3 Buckets

Amazon S3 buckets offer a simple and scalable solution for storing your data in the cloud. However, just like any other online storage platform, there are security risks you need to be aware of.

Following are some of the most common security risks associated with Amazon S3 buckets:

Unintentional public access: Misconfiguration, such as overly permissive permissions (ie granting public read access), can cause insecure bucket policies and permissions, which can result in unauthorized users being able to access and perform actions on your S3 bucket.
Insecure bucket policies and permissions: S3 buckets use identity and access management (IAM) to control access to data. This allows you to define permissions for individual users and groups using bucket policies. If your bucket policies are not properly configured, it can give unauthorized users access to your data (eg policies using wildcard). Poorly configured IAM settings can also result in compliance violations due to unauthorized data access or modification, which impacts regulatory requirements and can expose the organization to legal consequences.
Data exposure and leakage: Even if your S3 bucket isn’t public, data can still be exposed. For instance, data can be exposed if you accidentally share the URL of an object with someone else or if there are overly permissive permissions for that bucket. Additionally, data exposure can occur if you download data from your S3 bucket to an insecure location.
Lack of encryption: The lack of encryption for data stored in S3 buckets is another significant security risk. Without encryption, intercepted data during transit or compromised storage devices may expose sensitive information.

Managing AWS access control and encryption options can be difficult. For instance, AWS has numerous tools, ranging from intricate access controls to robust encryption options, that help to protect your data and accounts from unauthorized access. Navigating this wide range of tools can be daunting, especially for individuals who don’t have a background in security. A single policy misconfiguration or permission can leave sensitive data exposed to unintended audiences.

This is where S3Scanner could be useful.

What Is S3Scanner

S3Scanner is an open source tool designed for scanning and identifying security vulnerabilities in Amazon S3 buckets:

S3Scanner supports many popular platforms including:

AWS (the subject platform of this article)
GCP
Digital Ocean
Linode
Scaleway

You can also use S3Scanner with custom providers such as your own bespoke bucket solution. This makes it a versatile solution for various organizations.

Please note, for non AWS services, S3Scanner currently only supports scanning for anonymous user permissions.

The following command shows S3Scanner basic usage to scan for buckets listed in a file called names.txt and enumerate the objects.

$ s3scanner -bucket-file names.txt -enumerate

The following are some of S3Scanner’s key features:

Multithreaded Scanning

S3Scanner uses multithreading capabilities to concurrently assess multiple S3 buckets, optimizing the speed of vulnerability detection. To specify the number of threads to use, you can use the -threads flag and then provide the number of threads you want to use.

For instance, if you want to use ten threads, you’ll use the following command:

s3scanner -bucket my_bucket -threads 10

Config File

If you’re using flags that require config options like custom providers, you’ll need to create a config file. To do so, create a file named config.yml and put it in one of the following locations where S3Scanner will look for it:

(current directory)
/etc/s3scanner/
$HOME/.s3scanner/

Built-In and Custom Storage Provider Support

As previously stated, S3Scanner seamlessly integrates with various providers. You can use the -provider option to specify the object storage provider when checking buckets.

For instance, if you use GCP, you’d use the following command:s3scanner -bucket my_bucket -provider gcp

To use a custom provider when working with currently unsupported or a local network storage provider, the provider value should be custom like this:s3scanner -bucket my_bucket -provider custom

Please note that when you’re working with a custom provider, you also need to set up config file keys under providers.custom, as listed in the config file. Some examples include address_style, endpoint_format, and insecure. Here’s an example of a custom provider config:

# providers.custom required by `-provider custom`
#   address_style - Addressing style used by endpoints.
#     type: string
#     values: "path" or "vhost"
#   endpoint_format - Format of endpoint URLs. Should contain '$REGION' as placeholder for region name
#     type: string
#   insecure - Ignore SSL errors
#     type: boolean
# regions must contain at least one option
providers:
  custom: 
    address_style: "path"
    endpoint_format: "https://$REGION.vultrobjects.com"
    insecure: false
    regions:
      - "ewr1"

Comprehensive Permission Analysis

S3Scanner provides access scans by examining bucket permissions. It identifies misconfigurations in access controls, bucket policies, and permissions associated with each S3 bucket.

PostgreSQL Database Integration

S3Scanner can save scan results directly to a PostgreSQL database. This helps maintain a structured and easily accessible repository of vulnerabilities. Storing results in a database also enhances your ability to track historical data and trends.

To save all scan results to a PostgreSQL, you can use the -db flag, like this:s3 scanner -bucket my_bucket -db

This option requires the db.uri config file key in the config file. This is what your config file should look like:

# Required by -db
db:
uri: "postgresql://user:password@db.host.name:5432/schema_name"

RabbitMQ Connection for Automation

You can also integrate with RabbitMQ, which is an open source message broker for automation purposes. This allows you to set up automated workflows triggered by scan results or schedule them for regular execution. Automated responses can include alerts, notifications, or further actions based on the identified vulnerabilities, ensuring proactive and continuous security.

The -mq flag is used to connect to a RabbitMQ server, and it consumes messages that contain the bucket names to scan:s3scanner -mq

The -mq flag requires mq.queue_name and mq.uri keys to be set up in the config file.

Customizable Reporting

With S3Scanner, you can generate reports tailored to your specific requirements. This flexibility ensures that you can communicate findings effectively and present information in a format that aligns with your organization’s reporting standards.

For instance, you can use the -json flag to output the scan results in JSON format:s3scanner -bucket my-bucket -json

Once the output is in JSON, you can pipe it to jq, a command-line JSON processor, or other tools that accept JSON, and format the fields as needed.

How S3Scanner Works

To use S3Scanner, you need to install it on your system. The tool is available on GitHub, and the installation instructions vary based on your platform. Currently, supported platforms include Windows, Mac, Kali Linux, and Docker.

The installation steps for the various platforms and version numbers are shown below:

Platform: Homebrew (MacOS)
- Version: v3.0.4
- Steps: brew install s3scanner
Platform: Kali Linux
- Version: 3.0.0
- Steps: apt install s3scanner
Platform: Parrot OS
- Version: –
- Steps: apt install s3scanner
Platform: BlackArch
- Version: 464.fd24ab1
- Steps: pacman -S s3scanner
Platform: Docker
- Version: v3.0.4
- Steps: docker run ghcr.io/sa7mon/s3scanner
Platform: Winget (Windows)
- Version: v3.0.4
- Steps: winget install s3scanner
Platform: Go
- Version: v3.0.4
- Steps: go install -v github.com/sa7mon/s3scanner@latest
Platform: Other (build from source)
- Version: v3.0.4
- Steps: git clone git@github.com:sa7mon/S3Scanner.git && cd S3Scanner && go build -o s3scanner .

For instance, on a Windows system, you would use winget and run the following command: winget install s3scanner. Your output would look like this:

Found S3Scanner [sa7mon.S3Scanner] Version 3.0.4
This application is licensed to you by its owner.
Microsoft is not responsible for, nor does it grant any licenses to, third-party packages.
Downloading https://github.com/sa7mon/S3Scanner/releases/download/v3.0.4/S3Scanner_Windows_x86_64.zip
  ██████████████████████████████  6.52 MB / 6.52 MB
Successfully verified installer hash
Extracting archive...
Successfully extracted archive
Starting package install...
Command line alias added: "S3Scanner"
Successfully installed

The last sentence shows that S3Scanner was successfully installed.

If you want to avoid installing S3Scanner via the above methods, you can also use the Python Package Index (PyPI). To do so, search for S3Scanner on PyPI:

And select the first option that appears (ie S3Scanner):

Create and navigate to a directory of your choosing (eg s3scanner_directory) and run the command pip install S3Scanner to install it.

Please note that you need to have Python and pip installed on your computer to be able to run the pip command.

Your output looks like this:

Collecting S3Scanner
  Downloading S3Scanner-2.0.2-py3-none-any.whl (15 kB)
Requirement already satisfied: boto3>=1.20 in c:\python\python39\lib\site-packages (from S3Scanner) (1.34.2)
Requirement already satisfied: botocore<1.35.0,>=1.34.2 in c:\python\python39\lib\site-packages (from boto3>=1.20->S3Scanner) (1.34.2)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in c:\python\python39\lib\site-packages (from boto3>=1.20->S3Scanner) (1.0.1)
Requirement already satisfied: s3transfer<0.10.0,>=0.9.0 in c:\python\python39\lib\site-packages (from boto3>=1.20->S3Scanner) (0.9.0)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in c:\python\python39\lib\site-packages (from botocore<1.35.0,>=1.34.2->boto3>=1.20->S3Scanner) (2.8.2)
Requirement already satisfied: urllib3<1.27,>=1.25.4 in c:\python\python39\lib\site-packages (from botocore<1.35.0,>=1.34.2->boto3>=1.20->S3Scanner) (1.26.18)
Requirement already satisfied: six>=1.5 in c:\python\python39\lib\site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.35.0,>=1.34.2->boto3>=1.20->S3Scanner) (1.16.0)
Installing collected packages: S3Scanner
Successfully installed S3Scanner-2.0.2

This confirms that the S3Scanner was successfully installed.

Configure Scanning Parameters

Before running any scans, you need to make sure everything is working and configure your scanning parameters.

Run one of the following command to make sure S3Scanner is configured correctly:

s3scanner -h

s3scanner --help

You should receive some information about the various options you can use when scanning buckets:

usage: s3scanner [-h] [--version] [--threads n] [--endpoint-url ENDPOINT_URL]
                 [--endpoint-address-style {path,vhost}] [--insecure]
                 {scan,dump} ...

s3scanner: Audit unsecured S3 buckets
           by Dan Salmon - github.com/sa7mon, @bltjetpack

optional arguments:
  -h, --help            show this help message and exit
  --version             Display the current version of this tool
  --threads n, -t n     Number of threads to use. Default: 4
  --endpoint-url ENDPOINT_URL, -u ENDPOINT_URL
                        URL of S3-compliant API. Default: https://s3.amazonaws.com
  --endpoint-address-style {path,vhost}, -s {path,vhost}
                        Address style to use for the endpoint. Default: path
  --insecure, -i        Do not verify SSL

mode:
  {scan,dump}           (Must choose one)
    scan                Scan bucket permissions
    dump                Dump the contents of buckets

If you have the AWS Command Line Interface (AWS CLI) installed and have AWS credentials specified in the .aws folder, S3Scanner will pick up these credentials for use when scanning. Otherwise, you have to install the AWS CLI to be able to pick buckets in your environment:

Run Scans and Interpret Results

To run a scan, you need to run s3scanner and provide the flags, such as scan or dump, and the name of the bucket. For example, to scan for permissions on a bucket called my-bucket, you would run s3scanner scan --bucket my-bucket.

This gives you a similar output to the following (the columns are delimited by the pipe character, |):

my-bucket | bucket_exists | AuthUsers: [], AllUsers: []

The first portion of the output gives you the name of the bucket, and it tells you if that bucket exists in the S3 universe. The last portion of the output shows you the permissions attributable to authenticated users (anyone with an AWS account) as well as all users.

Run a scan command for a bucket that is in your AWS environment, such as ans3scanner-bucket, like this:

You should get the following output:

ans3scanner-bucket | bucket_exists | AuthUsers: [Read], AllUsers: []

This output shows that the bucket has authenticated users granted [Read] rights.

Scan Your GCP Buckets

To test your GCP buckets, create a bucket in your GCP account and make sure it doesn’t have public access:

Inside the bucket, add a text file:

To scan the bucket, run the previously mentioned command: s3scanner -bucket s3scanner-demo -provider gcp. You have to provide the -provider gcp flag to tell S3Scanner that you want to scan a GCP bucket. If you don’t provide this flag, S3Scanner uses AWS (the default option).

Your output shows that a bucket exists:

level=info msg="exists    | s3scanner-demo | default | AuthUsers: [] | AllUsers: []"

Now, change the GCP bucket access to “public” and grant all users access:

Then, scan the GCP bucket. Your output will show that the bucket is available to all users:

level=info msg="exists    | s3scanner-demo | default | AuthUsers: [] | AllUsers: [READ, READ_ACP]"

Best Practices for Remediation

After you review the results of your scan, make sure to prioritize the identified issues based on their severity. Some common remediations are as follows:

Adjust bucket permissions: You can restrict access to buckets by adjusting permissions and policies to adhere to the principle of least privilege. Make sure to remove unnecessary public access and ensure that only authorized entities have the required permissions.
Regularly audit and monitor your S3 bucket configurations: Establish a routine for auditing and monitoring your S3 bucket configurations. You can also set up alerts for any changes to permissions or policies, enabling timely detection and response to potential security incidents. Additionally, you can utilize tools and services such as AWS Config, which helps you assess, audit, and evaluate the configuration of your resources. Moreover, AWS Trusted Advisor helps inspect your environment and provides recommendations to improve security, performance, and cost.
Encrypt data: Securing data through encryption involves implementing measures for both in transit and at rest. For data that is in transit, employing secure communication channels like HTTPS during transfer ensures that information remains encrypted between clients and servers. On the server side, AWS S3 offers different options for encrypting data at rest.

Conclusion

In this article, you learned about some of the common security risks associated with Amazon S3 buckets and how S3Scanner can help.

S3Scanner is a valuable tool for anyone leveraging cloud storage through buckets because it helps you scan for vulnerabilities in your environment. With multithreaded scanning, comprehensive permission analysis, custom storage provider support, PostgreSQL database integration, and customizable reporting, S3Scanner is definitely worth exploring.

The irrational fear of deployments

Ibrahim Salami — Fri, 09 Aug 2024 21:38:41 +0000

A recent outage involving CrowdStrike impacted 8.5 million Windows operating systems, leading to disruptions in various global services, including airlines and hospitals. Multiple analyses have examined the root cause of this incident itself.

However, as a software engineer, I think we are missing the aspect of human emotions related to deployments, specifically the fear of breaking production. That’s what we will try to dive into in this article. We will cover:

Understanding the function of release engineering.
What software engineers care about and what they don’t.
Impact of continuous delivery (CD).
A look at manual deployments.
Problems with manual deployment and the solution to these problems.

Release Engineering

Before delving into the fear of deployments from a software engineer’s perspective, let’s first understand the role of a release engineer.

Release engineering has evolved considerably in recent years, thanks to the modern CI and CD tools and standardization of Kubernetes. Despite these advancements, the primary responsibilities remains the same:

Consistent and repeatable deployments: Standardizing release processes, reduces the risk of bad deployments to production.
Reducing service disruptions: Standardized processes also ensure teams are equipped to tackle harmful production environment incidents—for example, a rollback strategy for scenarios where a release causes problems.
Monitor and Optimize Performance: Look for performance improvements for faster and reliable deployments.
Collaborate with engineering: Work closely with developers, QA, and DevOps teams to ensure all new and existing services have a well defined deployment process.

What Software Engineers Care About

Unlike the release engineers, as a software engineer working in the product team we may only care about certain aspects of deployments:

Quick code merges: Merging quickly allows them to validate their work and move on to new tasks or unblock dependent tasks.
Production incidents: Although engineers may not care about all production incidents, they definitely care about their code changes causing any production outages.
Deployment schedule: Engineers also like to track when their changes go live or have gone live, so that they can have access to real-time feedback on their changes.

What Software Engineers Don’t Care About

Although there are things we care about, there are also those we don’t:

Deployment methodology: Although we know the need for an efficient and reliable deployment process, they don’t care how it is performed.
Effect of other changes: Unless things go wrong, we don’t worry about unrelated changes from other developers.
Deployment management: An engineer is indifferent to who manages deployment in a software team. For instance, we would only care about managing deployment if tasked with doing so.

Impact of Continuous Deployments (CD)

So what does the fear have to do with Continuous deployments?

A lot.

Studies have proven several benefits of Continuous Deployment (CD), and unsurprisingly many of which are psychological in nature. Continuous deployments removes “human-in-the-loop”, therefore it requires a strong trust in the test infrastructure.

In other words, automated tests are not only ensuring reliability of production, but also providing psychological safety, sometimes irrationally, reducing the fear of deployments. As a developer, I’m more comfortable making changes in a CD process vs if I’m asked to verify the changes manually.

However, despite the popularity of these CD strategies, a lot of companies still trigger deployments manually (have a human-in-the-loop), indicating a cautious approach to CD implementations. This behavior suggests that teams prefer to retain supervision on the release process and intervene where necessary.

This is important to understand from a psychological safety perspective. Manual deployments imply that someone is overseeing the process and handling issues when things go wrong. While this provides a sense of security, it can also induce fear in the person deploying and is prone to human error.

Manual deployments

Despite the drawbacks most teams manage deployments manually. A typical manual deployment may include a few steps:

Supervision

Someone babysits the entire deployment process before a release goes out. This person is tasked with intervening when and if there are signs of trouble. Teams maintain an oncall person who manages their deployments and handles problems when they arise.

Dedicated Release Teams

Some teams have a dedicated release engineering team, which ensures releases go smoothly. Since this means a high degree of specialization, the deployment process could be more efficient and reliable.

Spreadsheets

Some companies maintain a spreadsheet to validate any changes made. This allows companies to systematically review and approve these changes, ensuring they meet predefined quality standards.

Manual QA

In addition to spreadsheets, manual QA is another layer companies add. Manual QA tests new releases in staging environments before deploying them to production. However, a testing environment isn’t foolproof, so that some real-life scenarios won’t be accounted for.

Where Do Things Go Wrong With Manual Deployments?

Many things can go wrong for any software development team relying solely on manual deployments:

Dependence on a small group

This can create bottlenecks, which lead to release delays and human error in some instances. Also, a team could have problems when this specific person leaves or can’t deliver on the required tasks.

No risk-mitigation strategy

There is no strategy for following through in an unfavorable production incident. When an incident happens, the release team has to grapple to find the relevant stakeholders to help resolve and make decisions.

Prone to human error

Typographical errors in commands or scripts, or forgot to run the pre-deployment or post-deployment steps.

High effort

Since the deployments require babysitting the process,it becomes a time consuming effort. Also causing the frequency of deployments to drop significantly. For instance, if it requires an hour to monitor the entire deployment, the release team may decide to skip deployments on the days with minor changes to save that time.

Communication Breakdown

It’s unclear from product teams on the state of the releases and when their changes are getting into production.

Looking at these challenges, it’s easy to understand why engineers dread deployments. The risk of deployment failures, the high stakes, and the pressure to keep downtime low also contribute to this fear.

These failures can be minimized by increasing test automation. Still, since these tests are carried out in a test environment, you should not expect an automated test to catch every possible error. Failures are to be expected but at a reduced rate.

What can we do about it?

Simply set up Continuous Deployments? Easier said than done. Despite the drawbacks, manual deployments are still okay if managed well. The goals should be:

provide guardrails to avoid production incidents
reduce human errors
enable anyone to trigger deploys
ensure deployments happen frequently

Guardrails – Canary and Rollbacks

Canary and Rollback strategies can help reduce the impact of an outage and in many cases avert the crisis automatically.

A canary release exposes your new release to a small portion of production environment traffic. This gives teams insight into issues that might not have come up during testing.

On the other hand, a rollback strategy helps engineers revert a release to its previous stable version state. It is done when new problems arise after deployments to the production environment.

Reduce human errors – Standardization

Define standard deployment methodologies that result in efficiency, consistency, reliability, and high software quality. In their state of DevOps report, DORA shows that reliability predicts better operational performance. Furthermore, having a standardized process allows repeatability in release processes, which can be automated. Automating this process helps a team keep production costs lower.

Democratize deployment process

Democratizing the deployment process removes the reliance on specific individuals. If we empower any software engineer to deploy, it slowly reduces the fear. “If “anyone can deploy it should not be too hard”. Share your legos!

Frequent deployments

To reduce deployment anxiety, we need to deploy more frequently, not less. The DORA report also highlights that smaller batch deployments are less likely to cause issues and help lower the psychological barrier for developers.

Improve developer experience

Clarifying what is being deployed enhances the developer experience. Make it easy for developers to know when deployments occur and what changes are included. This transparency helps developers track when their changes go live and simplifies incident investigations.

Defined risk-mitigation strategies

There should be defined steps to follow for rollbacks and hotfixes, as this helps eliminate any indecision with production incidents. For instance, there should be separate build and deploy steps for teams to follow for easy rollbacks

Similarly, standardizing how to deal with hotfixes and cherrypicks can make it simple to operate when the stakes are high.

Feature flags

Feature flags are like kill-switches that can turn off a new feature that caused an incident in production. This can enable engineers to resolve production incidents quickly.

Conclusion

Software teams must treat release engineering as a priority from the outset of product development to avoid costly mistakes. And we should not let incidents like the Crowdstrike outage cripple our development practices. Addressing the fear of deployment and preventing production incidents involves several key strategies:

Invest in the standardization of deployment processes
Set up well-defined risk-mitigating strategies, such as canary releases, strategic rollouts, rollbacks, and hotfixes.
Simplify the developer experience by democratizing deployments, and encourage everyone to participate.

Comparing Flux CD, Argo CD, and Spinnaker

Ibrahim Salami — Fri, 26 Jul 2024 19:02:53 +0000

Continuous delivery (CD) tools play a crucial role in modern software development workflows, enabling teams to automate the process of deploying applications. Among the available CD tools, Flux CD, Argo CD, and Spinnaker stand out for their unique features and capabilities. This article provides an in-depth comparison of these three tools. In it, we’ll explore their architectures, key features, integration capabilities, and ideal use cases, and we’ll go into each tool’s basic implementation.

Comparing Flux CD, Argo CD, and Spinnaker is essential for organizations seeking the right CD tool to fit their specific requirements. By understanding the architectural differences, key features, and integration capabilities of each tool, teams can make informed decisions and optimize their deployment workflows.

Brief introduction to Flux CD, Argo CD, and Spinnaker

Flux CD, Argo CD, and Spinnaker are prominent players in the field of CD tools — each offers a unique approach to application deployment and management.

- Flux CD: Flux CD, or Flux, is an open-source tool that follows the GitOps methodology, where the desired state of the system is controlled in Git repositories. It continuously monitors these repositories for changes and automatically applies them to the Kubernetes cluster.
- Argo CD: Argo CD is another open-source tool designed for Kubernetes-native continuous deployment. It utilizes declarative YAML manifests in a GitHub repository to define the desired application state and synchronizes that with the actual state in the Kubernetes cluster.
- Spinnaker: Spinnaker is a more compact CD platform that provides support for multicloud deployments. It offers advanced features such as automated canary analysis and pipeline orchestration, making it suitable for complex deployment scenarios.

Architecture

Flux CD

Flux is constructed with GitOps Toolkit components. In the Flux ecosystem, those components are Flux Controllers, composable APIs, and reusable Go packages. They’re used for developing CD workflows on Kubernetes using GitOps principles.

Key components of Flux CD include the source controller, which establishes a collection of Kubernetes entities, enabling cluster administrators and automated operators to manage Git and Helm repository tasks through a dedicated controller.

You have the option of using the toolkit for expanding Flux capabilities and creating custom systems tailored for continuous delivery. A recommended starting point for this is the source-watcher guide.

Argo CD

Argo CD operates as a Kubernetes controller, continually monitoring active applications and comparing their existing operational state with the intended target state defined in a Git repository. Applications that do not match the desired state are flagged as out of sync. After that, Argo CD provides reporting and visualization of these disparities, offering options for automatic or manual synchronization to bring the operational state in line with the desired target state.

Any modifications made to the desired target state in the Git repository are automatically applied and reflected in the specified target environments (usually a Kubernetes cluster). All the changes made are also displayed in the Argo CD UI.

This architecture ensures automated application deployment and lifecycle management, aligning with the GitOps pattern of using Git repositories as the source of truth for defining application states. Argo CD supports various methods of specifying plain directories of YAML/JSON manifests, Kubernetes manifests, including kustomize applications, Helm charts, and Jsonnet files.

Argo CD provides a CLI for automation and integration with CI pipelines, webhook integration with version control systems, and so on.

Spinnaker

Spinnaker employs a microservices architecture comprising several components that interact to facilitate the deployment process. Core components of Spinnaker include the Deck UI for user interaction, the Gate API for authentication and authorization, and various cloud-specific Clouddriver services for interacting with cloud providers.

The diagram below illustrates the interdependencies among microservices. The green rectangles denote “external” elements, such as the Deck UI, a single-page JavaScript application operating within your web browser. The gold rectangles signify Halyard components, which are utilized solely during the configuration of Spinnaker.

Key features

Flux CD

- GitOps-based continuous delivery: Flux CD leverages Git repositories as the source of truth for defining the desired state of the system.
- Automated deployments: *Flux CD automates the deployment process based on changes detected in Git repositories.
*- Git repository synchronization: Flux CD synchronizes Kubernetes resources with Git repositories, ensuring consistency between environments.

Argo CD

- Declarative GitOps application deployment: Argo CD enables declarative application deployments using YAML manifests stored in Git repositories.
- Rollback and version control: Argo CD supports rollback functionality and maintains version control for application configurations.
- SSO integration: Argo CD provides integration with single sign-on (SSO) systems for authentication and access control.

Spinnaker

- Multi-cloud support: Spinnaker offers native support for multiple cloud providers, allowing easy deployment across heterogeneous environments.
- Automated canary analysis: Spinnaker facilitates automated canary analysis for evaluating new versions of applications before pushing them to production.
- Pipeline orchestration: Spinnaker provides robust pipeline orchestration capabilities, enabling complex deployment workflows.

Integration and extensibility

Flux CD

- Integration with Kubernetes and Helm: Flux CD integrates easily with Kubernetes and Helm for managing containerized applications.
- Extensibility through custom controllers: Flux CD allows extending the Kubernetes API with custom resource definitions and validation webhooks.

Argo CD

- Kubernetes native integration: Argo CD is tightly integrated with Kubernetes, leveraging custom resource definitions (CRDs) for managing application deployments.

Spinnaker

- Integration with major cloud providers: Spinnaker provides out-of-the-box integration with major cloud providers such as AWS, Google Cloud Platform (GCP), and Microsoft Azure.
- Extensibility through custom stages and plugins: It supports extensibility through custom stages and plugins, allowing users to integrate with additional services and tools.

Use cases and best practices

Flux CD

Flux CD is suitable for small- to medium-scale Kubernetes deployments. It’s ideal for teams practicing GitOps methodologies, where the entire deployment process is managed through version-controlled Git repositories. It’s more flexible than Argo CD.

Argo CD

Argo CD is good for DevOps teams looking for Kubernetes-native continuous deployment solutions. It’s recommended for CI/CD pipelines requiring declarative application definitions stored in Git repositories.

Spinnaker

Spinnaker is recommended for enterprises with complex, multi-cloud deployment requirements because of its robust multi-cloud support. It’s ideal for organizations needing advanced CD workflows, including canary deployments and automated analysis. It’s more flexible than Flux CD and Argo CD but harder to get started with.

Examples of how to use Flux CD, Argo CD, and Spinnaker

This section will cover the basics of how to set up and use Flux CD, Argo CD, and Spinnaker — it’s meant to give you an idea of what you’re getting into before you implement a CD tool in a real project. To follow the steps, you should have a cluster running.

How to use Flux CD

Using Flux CD involves setting up a Git repository to store your Kubernetes manifests and configuring Flux CD to synchronize these manifests with your Kubernetes cluster. Here’s a step-by-step guide:

Step 1: Install Flux CD

You need to install the Flux CLI to run commands on. With Bash for macOS and Linux, you can use the following command (you can get other installation methods in the CLI install documentation):

curl -s https://fluxcd.io/install.sh | sudo bash

You can check whether it installed properly with the following command:

flux check --pre # use sudo if you get error like "connection refused"

Step 2: Configure GitHub credentials

Flux needs your GitHub credentials in order to log in and perform some actions on your repository. Export your GitHub personal access token and username:

`export GITHUB_TOKEN=

export GITHUB_USER=
`

Step 3: Install Flux CD onto your cluster

The flux bootstrap github command sets up the Flux controllers on a Kubernetes cluster and sets them to synchronize the cluster’s state with a Git repository. It also uploads the Flux manifests to the Git repository and sets up Flux CD to automatically update itself based on changes in the Git repository.

To do do this run the following command:

`echo $GITHUB_TOKEN | flux bootstrap github \

--owner=$GITHUB_USER \

--repository= \

--branch=main \

--path=./flux-clusters \

--personal

--private=false
`

The bootstrap command above does the following:

Creates a Git repository (in my case, flux-test-app ) on your GitHub account.
Adds Flux component manifests to the repository.
Deploys Flux components to your Kubernetes cluster. You can run kubectl get all -n flux-system to check out the components.
Configures Flux components to track the path /flux-clusters in the repository.
–private=false flag is used to create a public repository.

Your output will look like this:

Step 4: Add Podinfo repository to Flux CD (or any repository you want)

First, clone the repository you created (in my case, flux-test-app) to your local machine:

`git clone https://github.com/$GITHUB_USER/flux-test-app

cd flux-test-app`

Now run the following to create a GitRepository manifest pointing to the github.com/stefanprodan/podinfo master branch. Podinfo is a web application written in Go.

`flux create source git podinfo \

--url=https://github.com/stefanprodan/podinfo \

--branch=master \

--interval=2m \

--export > ./flux-cluster/podinfo-source.yaml
`

In the command above:

A GitRepository named podinfo is created.
The source-controller checks the Git repository every two minutes, as indicated by the –interval flag.
It clones the master branch of the https://github.com/stefanprodan/podinfo repository.
When the current GitRepository revision differs from the latest fetched revision, a new Artifact is archived.

After the command is run, you should have the corresponding file podinfo-source.yaml.

Step 5: Deploy the podinfo application using GitOps

Configure Flux CD to build and apply the kustomize directory located in the podinfo repository. This directory contains the Kubernetes deployment files.

Use the following flux create command to create a Kustomization that applies the podinfo deployment:

`flux create kustomization podinfo \

--target-namespace=default \

--source=podinfo \

--path="./kustomize" \

--prune=true \

--wait=true \

--interval=10m \

--retry-interval=2m \

--export > ./flux-cluster/podinfo-kustomization.yaml
`

In the command above:

A Flux GitRepository named podinfo is created that clones the master branch and makes the repository content available as an Artifact inside the cluster.
A Flux Kustomization named podinfo is created that watches the GitRepository for Artifact changes.
The Kustomization builds the YAML manifests located at the specified location in –path=”./kustomize”, validates the objects against the Kubernetes API, and applies them on the cluster.
The –interval=10m flag, every ten minutes, sets the Kustomization to run a server-side dry-run to detect and correct drift inside the cluster.
The –retry-interval=2m specifies the interval (two minutes) at which to retry a failed reconciliation.
When the Git revision changes, the manifests are reconciled automatically. If previously applied objects are missing from the current revision, these objects are deleted from the cluster when enabled with –prune=true.

After the command is run you should have the corresponding file podinfo-kustomization.yaml.

Now commit and push the manifests to the repository:

`git add -A && git commit -m "Add podinfo manifests"

git push`

After about ten minutes, your application should be running on your cluster. You can check with the following command:

sudo kubectl -n default get deployments,services

Output:

How to use Argo CD

To use Argo CD, you typically install Argo CD onto your Kubernetes cluster, deploy your applications to Kubernetes, configuring Argo CD to watch your application manifests in a Git repository, and then let Argo CD synchronize the desired state of your applications with the actual state running in your cluster.

Here’s a basic guide to get started:

Step 1: Install Argo CD onto your Kubernetes cluster

You can install Argo CD using Kubernetes manifests. Below is an example of how you can install Argo CD using kubectl:

`kubectl create namespace argocd

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml`{% endraw %}

Also install the Argo CD CLI to run the argocd commands in later steps.

Now change the argocd-server service type to LoadBalancer with the following command:

{% raw %}kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'

Step 2: Access the Argo CD UI

Once it’s installed, you can access the Argo CD UI via a port forward or by exposing the service externally. Here’s how to port forward:

kubectl port-forward svc/argocd-server -n argocd 8080:443

You can then access the Argo CD UI by navigating to http://localhost:8080 in your web browser.

The initial password for the admin (login username) account is automatically generated and saved as plain text in the password field within a secret named argocd-initial-admin-secret in your Argo CD installation namespace. To easily obtain this password, you can run the following argocd admin command:

argocd admin initial-password -n argocd

Using the username admin and the password from above, log in to Argo CD’s host:

argocd login https://localhost:8080/

Step 3: Creating an app on Kubernetes

First, you need to set the current namespace from default to argocd by running the following command:

kubectl config set-context --current --namespace=argocd

Next, deploy a sample application to the Kubernetes cluster using YAML manifests. This manifest is on GitHub so you can check out the content.

Create the example application with the following command:

argocd app create guestbook --repo https://github.com/khabdrick/argocd-example-apps.git --path . --dest-server https://kubernetes.default.svc --dest-namespace default

If you’re using a different repository, update https://github.com/khabdrick/argocd-example-apps.git –path . in the code as appropriate.

In the Argo CD UI, you will see that your app has been deployed and synchronized successfully.

Argo CD will now start monitoring the Git repository for changes and automatically synchronize the application to the desired state specified in the manifests. It takes about three minutes for Argo CD to refresh automatically and synchronize and apply changes in the repository.

This is a basic guide to get started with Argo CD. Depending on your specific use case and requirements, you may need to explore more advanced features and configurations.

How to use Spinnaker

To install Spinnaker, you need Halyard. Halyard is a tool used to configure and manage Spinnaker deployments. This section outlines the process of setting up Spinnaker with a MySQL database on Kubernetes. We’ll start by running Halyard in a Docker container.

Note: For this section, I will use a Kubernetes cluster from Docker Desktop.

Setting up a MySQL database

To begin, deploy a MySQL database using Kubernetes and the MariaDB Docker image.

(Try to use a more secure password.)

kubectl run mysql --image=mariadb:10.2 --env="MYSQL_ROOT_PASSWORD"="123" --env="MYSQL_DATABASE"="front50"

This command creates a MySQL instance named mysql, setting the root password and creating a database named front50. This will be used to configure Front50. Front50 serves as the persistent storage and retrieval mechanism for Spinnaker’s pipeline configurations, application details, and other metadata.

Configuring Halyard

Next, we configure Halyard by creating a container that runs Halyard:

`docker run --name halyard --rm \

-v ~/.kube:/home/spinnaker/.kube \

-it us-docker.pkg.dev/spinnaker-community/docker/halyard:stable`

In another terminal window, enter the Halyard container:

docker exec -it halyard bash

Once inside the Halyard container, configure the Spinnaker version:

`hal config version

hal config version edit --version `

Enable Kubernetes as a provider:

hal config provider kubernetes enable

Add a Kubernetes account; docker-desktop in the command below is the context of the cluster running on Docker Desktop:

hal config provider kubernetes account add my-account --context docker-desktop

Now associate your Kubernetes account (my-account) with Halyard:

hal config deploy edit --type distributed --account-name my-account

Configure storage using Redis. This will be changed later, since Halyard doesn’t allow setting MySQL directly:

hal config storage edit --type redis

Now enable artifacts. The Artifacts feature in Spinnaker allows the system to manage and deploy artifacts (such as Docker images, JAR files, and Debian packages) as part of your deployment pipelines:

hal config features edit --artifacts true

Configuring Spinnaker to use MySQL

Next, you have to configure Spinnaker to use the MySQL database. Create the /home/spinnaker/.hal/default/profiles/front50-local.yml file and insert the following configurations:

sql:
  enabled: true
  connectionPools:
    default:
      default: true
      jdbcUrl: jdbc:mysql://MYSQL_IP_ADDRESS:3306/front50
      user: root
      password: 123
  migration:
    user: root
    password: 123
    jdbcUrl: jdbc:mysql://MYSQL_IP_ADDRESS:3306/front50
spinnaker:
  redis:
    enabled: false

Replace MYSQL_IP_ADDRESS with the appropriate IP address. Also make sure that other credentials match with what you used to deploy MySQL earlier.

You can get the MYSQL IP by running the following command (outside the Hayland container):

kubectl get pods -o wide

Apply the deployment (in the Hayland container). This command is used to apply the changes made to the Spinnaker configuration and deploy or update Spinnaker in the target environment:

hal deploy apply

Now you can check to see whether the pods are running completely:

kubectl get pods -n spinnaker

We need the deck and gate pods to be running so we can access the Spinnaker UI.

Now we can forward the deck and gate pods so that we can access it on the browser. Do this with the following command:

kubectl -n spinnaker port-forward <spin-deck-pod-name> 9000

On another terminal, run the gate :

kubectl -n spinnaker port-forward <spin-gate-pod-name> 8084

Now you can access the Spinnaker UI at http://localhost:9000/ and start developing your pipelines.

Conclusion

Flux CD, Argo CD, and Spinnaker offer distinct advantages and cater to different use cases within the realm of continuous delivery. By evaluating their architectures, features, and integrations, you can make informed decisions about the best way to automate your deployment and delivery processes.

Adopting OpenTofu as an Alternative to Terraform

Ankit Jain — Mon, 18 Mar 2024 22:20:21 +0000

Infrastructure as code (IaC) is a concept for deploying infrastructure like a software application. Ongoing advancements in infrastructure virtualization have made this possible. The rise of cloud computing transformed physical servers, networks, and hardware infrastructure into virtualized services. Add assets like databases, domain name service (DNS) providers, and authentication systems, and you get infrastructure that can be deployed like software.

With a few lines of configuration script, you can deploy operating systems, networks, and storage to a remote data center with a single command. The system can be kept in sync with updates to the configuration, which makes changing or redeploying the infrastructure a breeze.

IaC should not be confused with configuration management. Tools in this category, such as Chef, Puppet, or Ansible, typically deploy and configure software components on existing infrastructure. Terraform, on the other hand, is a provisioning tool that specializes in setting up infrastructure components like operating systems, networks, databases, users, and permissions on bare server hardware. Terraform can play hand-in-hand with configuration management tools by providing the infrastructure necessary for deploying software components.

When Terraform's licensing changed, an open-source fork called OpenTofu was spun off. In this article, you'll learn about OpenTofu's features and how OpenTofu compares to Terraform.

The goal of this article is to help you make an informed decision about using OpenTofu to manage your infrastructure requirements as code.

What Is OpenTofu

OpenTofu provides a simple but powerful configuration language to describe the infrastructure that you want to deploy. You specify the number and size of virtual servers to deploy to, the operating system to use, the virtual network overlays, the DNS settings, and anything else your infrastructure needs.

Then you run a tool that reads configuration scripts written in a specialized configuration language and interacts with infrastructure resources called providers to build and deploy the required parts of the infrastructure.

If anything needs to be changed, you update the script and run the tool again. The tool then updates the actual state of your infrastructure to match the description.

OpenTofu and Terraform

You may have come across the name Terraform in the context of IaC. Terraform and OpenTofu are closely related: OpenTofu is a fork of Terraform 1.5.

Years ago, a company called HashiCorp created Terraform, among other DevOps tools like Vagrant, Packer, Nomad, Vault, and Consul. All these tools aim at automating software orchestration and infrastructure management.

Terraform's approach to provisioning infrastructure consists of a three-step workflow: write, plan, and apply. In the write step, you define the resources needed to run the infrastructure. In the plan step, Terraform compares the written infrastructure definition against the existing infrastructure and creates an execution plan for creating, updating, or destroying resources as needed. Finally, in the apply step, Terraform applies the planned changes to the target infrastructure.

To utilize as many infrastructure resources as possible, Terraform uses a plugin system that allows resource providers, such as DNS services, container orchestration systems, database services, or logging systems, to be included.

Terraform's License Change

Terraform had become a popular IaC tool when HashiCorp, its maker, decided to change the license for their tools from the Mozilla Public License (MPL) 2.0 to the Business Source License (BSL). This move aimed to protect HashiCorp from competitors who could set up hosted Terraform offers just like HashiCorp had and reap the benefits without contributing back.

HashiCorp is not the first company to make this move. Earlier, companies like MongoDB, Redis, and Cockroach Labs also decided to restrict the ability to resell their (otherwise open source) code for similar reasons.

Nevertheless, HashiCorp's announcement raised concerns among Terraform users about the possible legal implications of the license switch. It didn't take long until the latest MPL-licensed Terraform version got forked. The fork was originally named OpenTF but was later renamed to OpenTofu due to trademark concerns.

Because Terraform 1.6.x and beyond will be under the new BSL, the OpenTofu project cannot take over changes made to Terraform anymore. While OpenTofu aims to keep feature parity with Terraform, and both projects come with a backward compatibility promise (1) and (2), the two projects may slowly diverge.

Changes Introduced after the Fork

After the fork, the changelog for OpenTofu 1.6.0 lists several significant changes, including the following:

Conditional GNU Privacy Guard (GPG) validation bypass for the default registry
Changes to the cloud and remote backends and the login and logout commands that no longer default to app.terraform.io
A new tofu test command that significantly changes how tests are written and executed
Several enhancements and bug fixes

Additionally, the OpenTofu community continues to add changes to its roadmap.

Is OpenTofu a Viable Alternative to Terraform

If you're in search of an IaC tool, you'll need to decide between Terraform and OpenTofu. Existing Terraform users even have three options to consider: Whether to stay with Terraform, switch to OpenTofu now, or stick with Terraform 1.5.x for a while and watch future developments closely.

For both audiences, there are numerous reasons for choosing the free and open source (FOSS) alternative.

Compatibility

As of this writing, OpenTofu is fully compatible with Terraform 1.5.x, but the list of features will increasingly diverge in the future. The OpenTofu FAQ has a clear statement about a community-driven approach shaping OpenTofu's future: "The community will decide what features OpenTofu will have."

However, OpenTofu will remain backward-compatible so that existing Terraform configurations (made with Terraform up to 1.5.x) continue to work with future versions of OpenTofu (per the OpenTofu Manifesto).

Additionally, the OpenTofu core team confidently predicts that "the large number of developers pledging their resources to help develop OpenTofu will accelerate the development of features and enable faster releases than Terraform managed previously." In other words, features that Terraform users have been anticipating for some time may come to life faster with OpenTofu than with Terraform.

A Free and Open License

The compatibility question may fade in the future when OpenTofu and Terraform are established as two independent projects and interested parties have a decent list of features to compare before settling on one of the tools. Switching between the two will become less frequent.

What will remain is the question of the license under which each of the projects is offered. Terraform's BSL is a source-available license that does not guarantee that users can use the code in their preferred manner. Notably, users may not include Terraform in offerings to third parties that compete with HashiCorp's offerings. (See the "Additional Use Grant" in the HashiCorp BSL.)

In contrast, OpenTofu is licensed under the MPL 2.0, a true FOSS license that grants users the right to use the code as they wish. Having a true FOSS license means that development is driven by the community that was built around OpenTofu, which is large and thriving. The list of supporters includes more than 150 companies and more than 780 individuals.

Additionally, OpenTofu has been adopted by the Linux Foundation, a step that guarantees vendor-neutral governance of the project. The Linux Foundation maintains several projects that are used worldwide, including Linux, Kubernetes, and Node.js.

When to Choose OpenTofu over Terraform

Reasons for choosing OpenTofu vary based on the user.

Companies may have the largest incentive to choose OpenTofu. Use cases for an IaC tool are constantly at risk of conflicting with the BSL. Moreover, the Terraform project changed its license unexpectedly, and it may do so again. With a true open source license and under the umbrella of a nonprofit foundation, OpenTofu provides a far more predictable future.

Digital agencies and consultants should strive to offer their clients a solution whose legal basis is predictably stable and immune to the opaque decisions of a single company. OpenTofu is not only that. It's also backed by a large list of supporters who joined forces to enhance and extend OpenTofu based on the wishes of the community.

Individual users may feel unaffected by Terraform's switch to the BSL. But then there is little that speaks against using OpenTofu instead, which comes under a more open license. While the BSL might have little effect on individual, noncommercial users, nobody knows what possible future changes to the Terraform license will bring. Choosing an open source project now, when switching costs are low, is the sensible choice.

Can Terraform Still Be a Good Choice?

With all the obvious advantages of truly open-source projects, it would seem that no one would choose to use Terraform. Don't judge too quickly, though. It's not all black and white.

For instance, Terraform’s availability as a cloud service when paired with Hashicorp’s enterprise-level support, could be a combo that's difficult for other competitors to match.
Moreover, some customers might conclude that their Terraform use cases are not negatively affected by Terraform's switch from an open-source license to a source-available license.

Granted, these advantages are rather specific to particular customers and use cases. OpenTofu might catch up on these aspects quickly—never underestimate the power of open source.

OpenTofu Continues to Evolve

As mentioned previously, OpenTofu and Terraform are likely to take different paths in the future. With hundreds of supporters committed to shaping the future of OpenTofu, the open source path is likely the one that leads to a more feature-rich and mature product than Terraform.

For example, in August 2023, OpenTofu announced an experimental implementation of end-to-end encryption for state files, a feature that Terraform users have been waiting for since 2014.

Chances are that there will be more of these initiatives to implement long-awaited features soon.

The Future of IaC Tools Is Open Source

The power of open source should not be underestimated. For any company, consultant, or developer whose business model might be threatened by the sudden license change of Terraform, the OpenTofu project provides a solid open source alternative.

OpenTofu is compatible with Terraform and aims to stay that way. It's a Linux Foundation project free of business tactics and legal insecurity. And it's backed by a large and active community.

While the BSL forbids certain kinds of uses of Terraform, the MPL 2.0 grants users complete freedom in using OpenTofu. The Linux Foundation has a track record of widely successful open source projects, and OpenTofu is a worthy addition.

Existing Terraform users may envision a few risks when making the switch to OpenTofu, but those risks are more than mitigated by the stability that an open source, foundation-led project provides. The best time for making the switch is now, while the two projects are still similar enough to enable a smooth transition.

Pre and post-merge tests using a merge queue

Ankit Jain — Thu, 14 Mar 2024 22:43:06 +0000

One of the key ingredients making developers productive is faster feedback loops. A fast feedback loop allows developers to identify and address issues promptly, leading to higher-quality code and faster release cycles.

Pre-merge and Post-merge tests

To maintain the good health of your system, there are a few types of tests that may be required. Validating these tests can take anywhere from a sub-second to several hours.

In the traditional waterfall model, testing is often a phase that occurs after development. However, the emphasis is on catching issues early in agile development and CI/CD workflows. But running all the tests can be extremely time-consuming, and provides slow feedback to the developers.

Instead, consider dividing tests into “pre-Merge” and “post-Merge” buckets. For instance, explore how LinkedIn manages pre-merge and post-merge workflows.

Splitting the tests

There are a couple of considerations to split the tests effectively:

The pre-merge tests should be faster to execute ensuring a fast feedback cycle for the developers
On the other hand, the post-merge tests should be generally more stable. If these tests fail often, we end up in a constant state of failed mainline. But, we should still expect these tests to fail occasionally. If a test doesn’t ever fail, it’s not worth running!

So let’s apply that to some types of tests, these answers may vary depending on your setup but should be generally agreeable.

Code linting

Pre-merge: Linting could catch a bug early in the developer lifecycle, and is both cheap and fast to run.

Unit tests

Pre-merge: They help catch bugs at the smallest level, ensuring that each piece of the puzzle works independently.

Integration testing

Pre-merge: Integration tests verify the interactions between various modules, ensuring a cohesive and functional application.

Automated regression testing

Post-merge: Running tests to identify anything that was previously working that may have regressed. We expect these would fail less often but are still essential to maintain code health.

Performance testing

Post-merge: As changes are deployed over time, the performance of the software may be impacted, this is where performance testing is important. We do not expect every change to impact the performance, and these tests should not fail often.

User Acceptance Testing (UAT)

Post-merge: Any type of manual QA or UAT should be handled post-merge as we expect these to be extremely slow. In most cases, if we identify a failure, it is typically resolved in future releases.

Managing the split

Now that we have the tests split, we can still catch most of the issues in the development cycle, and provide fast feedback to the developer. But what happens when the post-merge tests fail?

In most cases, you coordinate with the release team to roll back the change that caused the failure or roll forward a fix manually. This manual process can be annoying and can also cause poor developer experience.

This is where MergeQueue could play an interesting role.

Understanding the MergeQueue

Before delving into the intricacies of managing pre and post-merge tests with the queue, let’s take a moment to understand the role of MergeQueue in the CI/CD pipeline. MergeQueue acts as a gatekeeper, managing code merges and orchestrating the deployment process. If you are unfamiliar with merge queues in general, here’s a good primer.

Most of the modern merge queues offer some variance of an optimistic parallel mode. In such a mode, as changes are submitted to the queue, it creates an optimistic batch that contains all queued changes including the most recent submitted one. This batch is then validated again for all the tests to ensure that it does not break the mainline before merging the PRs.

Fast forwarding merge

A small variation of this optimistic parallel mode is called a fast-forwarding merge. The main difference in the case of fast-forwarding is that you are fast-forwarding the mainline (e.g. master) to these validated commit SHAs instead of creating new commit SHAs when the PRs are actually merged post-validation.

Post-merge is actually post-queue

Using the image above, you can think of the queued PRs as part of a “staging” branch. So as new commits are being added, those get “merged” into this staging. branch before they land on mainline. If we use that lens, we can run the “post-merge” test in the queue instead of running them after the changes are merged in mainline.

From the developer feedback viewpoint, the experience of “handing over” the PR to the queue is the same as merging the PR. But now, your rollbacks can be automated. Since the queue validates the changes before forwarding the mainline, any failure detected gets force pushed out.

Developer experience

As a developer, the workflow would be:

Open a PR, run the pre-merge test, and request a code review
Once the changes are approved and the tests pass, instead of merging the PR, they can enqueue it
MergeQueue will create a new branch with the squash commit of changes and run the CI on it. Developers can still access this staging branch if they want to access the latest codemix, knowing that it may still not have all the tests validated.
If all the tests pass, the mainline is fast-forwarded to this commit SHA, the original PR is flagged as merged.
If any of the required tests fail, the mainline is not impacted, and the developer is notified about the failed tests and the PR remains open for the developer to fix the issue and submit again.

Behind the scenes

The queue runs all the post-merge tests, which may be slow but usually pass. The mainline is always green because the changes are completely validated before they reach mainline. You would want to configure separate test execution and validation for the PRs created by developers and the branches created by the queue. Aviator MergeQueue provides a simple branching structure to split the test execution in all the common CI platforms for efficient CI usage.

Conclusion

In one of our old posts, we also talked about the death by a thousand papercuts due to the broken mainline, but maintaining a faster feedback cycle is also critical for an improved developer experience.

Using MergeQueue to split the tests (pre-queue and post-queue) is a great way to balance both sides of the problem while improving developer productivity.

SonarQube vs Fortify

Brian Neville-O'Neill — Wed, 06 Dec 2023 17:31:50 +0000

SonarSource SonarQube and OpenText Fortify are popular software security and code analysis tools. In this article, we will focus on the following:

SonarQube and Fortify’s features, capabilities, and functionalities.
A comparison between SonarQube and Microfocus Fortify

SonarQube

Sonarqube is a platform used for continuous code inspection and static code analysis. You can use it early in your software development cycle to identify and address code issues. It helps you improve your code quality and reduce build failure rates.

SonarQube has a lower barrier for fast use because it has a user-friendly interface, community support, and easy setup.

SonarQube features

Let’s take a deep dive into the features of SonarQube:

*Code coverage and Testing: * It integrates with many popular testing frameworks and tools that help identify what part of your code hasn’t been tested. It helps with an extensive range by highlighting areas that need test cases.
Code Quality Analysis: SonarQube analyzes code according to predefined standards and alerts you when your code doesn’t meet these standards or doesn’t meet some of the rules. It checks for code quality, like code smells, bugs, and vulnerabilities.
Complex Analysis Of Code: SonarQube analyses your code and lets you know the part of your code that might be hard to maintain or understand. This insight can make your complex code more readable and easily understood.
CI/CD Integration and Reporting: SonarQube integrates with different Continuous Integration and Continuous Delivery (CI/CD) tools, and you can easily add them to your development pipeline. It provides you with centralized reporting that allows you to make data-driven decisions that can improve your software development process.

SonarQube benefits

There are several strengths you enjoy when you use SonarQube, and they are:

Great support for many programming languages
Interactive community support
A detailed set of rules for code quality and detection
It is user-friendly and easy to use
You can integrate with popular CI/CD tools

SonarQube limitations

Despite the benefits you might enjoy when you use SonarQube in your development process, there are certain limitations you should be aware of. They are:

There is limited support for particular programming languages
It lacks advanced code security features
False positives in security vulnerabilities

Fortify

Fortify helps you identify and remedy security vulnerabilities in your software development process. You get a comprehensive approach during your development process with software composition analysis (SCA), dynamic application security testing (DAST), and static application security testing (SAST) it integrates.

Using these features, you can detect vulnerabilities early on and fix them before deploying your application. It supports programming languages from Apex, Java, and others.

Fortify features

Let’s dive into the features of Microfocus Fortify:

Advanced Security Testing: Suppose you use Fortify for your software development process. In that case, you enjoy advanced code security testing that would help your overall efforts because it enables you to understand the issues or potential threats better and can help you address these critical bottlenecks. Using Fortify means picking up problems you might miss using other tools.
Static Code Analysis: Fortify analyses for code structure and logic, which helps identify coding flaws in your source code. Fortify checks your code against predefined rules and notifies you of an issue, allowing you to fix your code before deploying. In addition, Fortify lets you set your own rules and policies based on your software development requirements.
Integration with Build Sytems: Fortify integrates with other build systems and CI/CD pipelines. It allows you to implement security testing as an essential part of your software development process by allowing you to incorporate security testing into existing workflows.

Fortify benefits

There are several benefits of Fortify, and they are:

It allows customizable rules and standards for static code analysis
It has comprehensive security code testing capabilities
It uses advanced vulnerability testing techniques and methods
Easy integration with development environments and CI/CD tools

Fortify limitations

Here are several limitations you have when using Fortify:

It takes a lot of work to set up and a steep learning curve.
Compared to SonarQube, it needs more language support.
It is expensive for enterprise-level usage.

Comparison: SonarQube vs Fortify

There are some differences when you use both tools for your software development process. However, you must know their weakness and strengths to help you make an ideal and better decision.

SonarQube beats Fortify because it has the best-suited features regarding quality code analysis. When you use SonarQube for software development builds, you can get comments from code coverage measurement, a predefined rules-based analysis, complexity analysis, and code duplication detection.
Fortify beats SonarQube regarding security vulnerabilities because it is more suited for this purpose. Fortify offers you in-depth reporting, customizable rules, and data flow analysis. It is specifically designed to deal with security issues in your code.
In terms of integration with CI/CD tools and development workflows, SonarQube and Fortify offer a seamless workaround for developers. They provide detailed reporting for coding and security vulnerabilities to aid your development process.
Regarding operating costs, SonarQube is less expensive than Fortify for enterprise purposes.

Conclusion

Your choice of software for your development process should depend on your project needs, requirements, and available capital for operation. In the article, we looked at the features of both, their benefits, and the limitations you would face when you use them. Furthermore, by comparing both, you can reach a conclusion for which to use when appropriate, assuming it meets your needs.

Aviator: Automate your cumbersome processes

Aviator automates tedious developer workflows by managing git Pull Requests (PRs) and continuous integration test (CI) runs to help your team avoid broken builds, streamline cumbersome merge processes, manage cross-PR dependencies, and handle flaky tests while maintaining their security compliance.

There are 4 key components to Aviator:

MergeQueue – an automated queue that manages the merging workflow for your GitHub repository to help protect important branches from broken builds. The Aviator bot uses GitHub Labels to identify Pull Requests (PRs) that are ready to be merged, validates CI checks, processes semantic conflicts, and merges the PRs automatically.
ChangeSets – workflows to synchronize validating and merging multiple PRs within the same repository or multiple repositories. Useful when your team often sees groups of related PRs that need to be merged together, or otherwise treated as a single broader unit of change.
TestDeck – a tool to automatically detect, take action on, and process results from flaky tests in your CI infrastructure.
Stacked PRs CLI – a command line tool that helps developers manage cross-PR dependencies. This tool also automates syncing and merging of stacked PRs. Useful when your team wants to promote a culture of smaller, incremental PRs instead of large changes, or when your workflows involve keeping multiple, dependent PRs in sync.

Try it for free.

The post SonarQube vs Fortify first appeared on Aviator Blog.

The post SonarQube vs Fortify appeared first on Aviator Blog.

What is a monorepo and why use one?

Brian Neville-O'Neill — Wed, 29 Nov 2023 18:39:26 +0000

Managing a sprawling codebase across multiple repositories can be a logistical nightmare. Developers often find themselves juggling various versions, wrestling with incompatible dependencies, and navigating a maze of pull requests and merges.

This chaos not only hampers productivity but also increases the risk of errors and inconsistencies. Are you tired of this disarray and looking for a streamlined way to manage your projects?

The answer lies in adopting a monorepo (aka a monolithic repository). One of the most compelling benefits of a monorepo is its ability to simplify version control.

In a traditional multirepo setup, each project or component has its own repository, often leading to versioning conflicts and making it difficult to keep track of changes across projects. With a monorepo, all your code lives in one place, making it easier to manage versions and maintain a coherent history.

This centralized approach ensures that everyone on the team is working with the same codebase, reducing the likelihood of versioning issues and making rollbacks more straightforward.

In this comprehensive guide, you’ll gain insights into what a monorepo is and how it differs from traditional multirepo strategies. You’ll also learn about the advantages of using a monorepo, particularly for larger teams dealing with complex projects.

What is a monorepo?

A monorepo is a software development strategy where the code for multiple projects is stored in a single version control system (VCS) repository:

This differs from the more traditional approach where each project or module has its own separate repository (aka a multirepo):

The projects within a monorepo can be interconnected libraries, services, applications, or even documentation.

The central idea of a monorepos is to consolidate the codebase, ensuring more streamlined version control, code reuse, and improved collaboration. For larger teams, this means better code visibility, simplified dependency management, and the possibility of atomic changes across multiple projects.

Why you should use monorepos

One of the main advantages of using a monorepo is unified versioning. In a traditional multirepo setup, each project has its own version history, making it challenging to understand how changes in one project affect others. With a monorepo, all projects share a single version history, making it easier to understand their interdependencies.

For example, if Project A depends on a feature in Project B, both can be updated simultaneously in a single commit, making it easier to track changes and dependencies.

Following are a few more advantages of using a monorepo:

Reusable code across projects

While it’s true that package managers can help sync dependencies across multiple repositories, having all code in a single repository makes it even easier to share and reuse code. There’s no need to publish internal packages just to share common utilities or components.

This is particularly beneficial for large teams where multiple projects often have overlapping requirements. Code reusability in a monorepo ensures that developers can easily leverage existing code, reducing duplication and accelerating development cycles.

Easier refactoring ensures consistency

In a monorepo, refactoring becomes a less daunting task. Changes can be made once and propagated across all dependent projects in a single commit.

This ensures that improvements or fixes are consistently applied, reducing the risk of one project lagging behind in terms of code quality or features.

Enhanced collaboration through visibility

Monorepos offer improved visibility, allowing teams to better communicate and collaborate. In a large team, this is especially beneficial. Developers can see the entire codebase, understand the context of their work better, and make cross-project changes effortlessly.

This holistic view eliminates the need for special permissions to access different repositories, making it easier for team members to assist each other and encourage code reuse.

Streamlined dependency management

Managing dependencies in a large team can be cumbersome with multiple repositories. A monorepo ensures that there’s a single version of each dependency, reducing conflicts and making updates more predictable.

This centralized approach to dependency management eliminates the “it works on my machine” type of problem, as every team member works with the same set of standardized tools and configurations.

Atomic changes for better version control

In large teams, coordinating releases and updates can be a complex task. Monorepos enable atomic changes, allowing related modifications across multiple projects to be committed at once. This ensures that features or fixes affecting multiple projects are released cohesively, making version control more straightforward and reliable.

Optimized CI/CD pipelines

One of the benefits of monorepos is that continuous integration, continuous deployment (CI/CD) are more streamlined. There’s no need to sync multiple repositories or ensure cross-repo compatibility.

The unified nature of a monorepo allows build and test tools to be standardized, ensuring that everyone is testing and deploying based on the same criteria.

This is particularly advantageous for large teams, where maintaining consistency in CI/CD practices is crucial for efficient and reliable software delivery.

By understanding these benefits in the context of large teams, it becomes clear why monorepos are becoming increasingly popular. They offer a unified, streamlined, and efficient approach to software development that is especially advantageous in complex, multiproject environments.

Monorepo challenges and tools that can help

Monorepos have surged in popularity, especially among large tech giants, due to their myriad advantages. But they’re not a one-size-fits-all solution and come with their own set of challenges. Explore some of these challenges and learn about tools that can help.

Scaling issues

As the codebase within a monorepo grows, so does the build time. Every time a change is made, the CI system might try to rebuild and retest the entire codebase, making the process slow and cumbersome.

To help with these scaling issues, build tools like Bazel, Pants, and Buck2 are specifically designed to optimize the build process through a technique known as incremental builds. Incremental builds minimize the strain on system resources, allowing for more efficient use of hardware, whether you’re working on a local machine or in a cloud-based development environment.

Unlike traditional build systems that recompile the entire codebase every time a change is made, these tools are smart enough to identify which parts of the codebase are affected by recent changes.

These tools are built to seamlessly integrate into your existing development workflow. Once configured, they can automatically detect changes in the codebase and trigger the appropriate incremental builds. This automation is particularly beneficial in a CI/CD environment, where rapid and frequent builds are the norm.

While these tools offer powerful capabilities, they do come with an initial learning curve. Each tool has its own set of configurations, syntax, and best practices that you need to familiarize yourself with. However, the investment in learning is often justified by the significant gains in build speed and efficiency.

Another advantage of using these specialized build tools is their flexibility. They allow for a high degree of customization, enabling you to tailor the build process to meet the specific needs of your project or team. This is especially useful in large teams or complex projects where generic build configurations may not be sufficient.

High complexity

For newcomers or even seasoned team members, navigating a huge codebase can be daunting. Understanding the interdependencies, finding the right modules, or even simply knowing where to start can be overwhelming.

Code navigation tools such as Sourcegraph and integrated features within platforms like GitHub serve as invaluable aids for developers navigating extensive codebases. These tools go beyond basic text search to offer a range of advanced functionalities designed to make code exploration more efficient and insightful.

One of the primary features of these tools is advanced code search, which allows developers to perform complex queries to find specific code snippets, functions, or even documentation within a large codebase. This is particularly useful when you’re trying to understand how a particular piece of code interacts with other components or when you’re debugging.

Another powerful feature is cross-referencing, which enables developers to easily find where a particular function or variable is used across different files or projects. This is incredibly helpful for understanding the impact of potential changes or for tracking down the root cause of a bug. It eliminates the need to manually search through multiple files, saving both time and effort.

These tools also offer intelligent code mapping, which provides a visual representation of how different parts of the code are interconnected. This can be especially useful for new team members who are trying to get a grasp of a complex project or for any developer who wants to understand the architecture and dependencies within the codebase.

Potential for conflicts

With many developers working simultaneously on the same repository, the chances of conflicting changes or merge conflicts increase. This can hamper the development speed and lead to errors if not resolved correctly.

VCS like Git offer robust mechanisms to handle merge conflicts. Features like pull requests in platforms like GitHub or Bitbucket allow for code review, helping spot and resolve conflicts before they’re merged into the main branch.

Additionally, automated testing tools like Jenkins, Travis CI, or CircleCI can automatically run tests on branches before they’re merged. This ensures that any breaking changes or conflicts get flagged early.

As you can see, while monorepos have their disadvantages, there’s a range of tools designed to mitigate these challenges.

Building a monorepo culture

The decision to use a monorepo goes beyond just tools and technical considerations; it requires a cultural shift in how developers work and collaborate. This culture is foundational to effectively managing and scaling a monorepo environment, ensuring that the benefits outweigh the challenges.

Take a look at a few different aspects of building a monorepo culture.

Shared responsibility

In a monorepo setting, boundaries between projects or components become blurred. Instead of viewing projects as isolated entities, team members should see the entire repository as their domain. That’s why it’s important to encourage collaboration across teams. Cross-team code reviews, pair programming, and team rotations can break silos and foster a holistic view of the codebase.

Additionally, you should regularly organize internal workshops, tech talks, or code walkthroughs. This can help team members familiarize themselves with different parts of the codebase and understand its intricacies.

For instance, Google fosters an environment in which developers have the freedom to access and contribute to any section of the codebase. This approach to code ownership has led to standardized coding practices, enhanced collaboration among team members, and a simplified process of reusing code.

Early merging to catch integration Issues

Consistently merging code changes is a proactive approach to software development that helps catch integration issues at an early stage. By integrating changes frequently, you can identify conflicts or bugs sooner rather than later, making them easier to resolve.

This practice minimizes the risk of encountering larger, more complicated issues in the future, which could require significant time and effort to fix. For example, if two developers are working on features that affect the same piece of code, early merging will reveal any incompatibilities between their changes, allowing for quicker adjustments.

To manage these merges in a more organized fashion, implementing branching strategies like feature branching or trunk-based development is highly recommended.

In feature branching, each new feature or bug fix is developed in its own branch. This allows developers to work on different features simultaneously without affecting the main codebase. Once the feature is complete and tested, it can be merged back into the main branch.

Feature branching is particularly useful for teams that have multiple developers working on different aspects of a project, as it allows for parallel development without the risk of one feature negatively impacting another.

In comparison, trunk-based development encourages developers to merge their changes directly into the trunk or main codebase as quickly as possible, often multiple times a day. This approach is beneficial for catching integration issues early and ensures that the codebase remains in a consistently deployable state. It’s especially effective for large teams where rapid integration is crucial for maintaining a smooth development workflow.

Take Facebook’s example, where the codebase is designed to empower engineers to “move fast and break things,” signifying a culture that values swift innovation along with ongoing refinement and iteration.

Thorough documentation

A monorepo’s vastness makes it challenging to navigate and understand. Comprehensive documentation acts as a map, guiding developers through the code.

Make sure you establish clear standards for documenting code. This might include things like comments, READMEs, and architecture diagrams.

Additionally, use tools like Doxygen, Javadoc, or Sphinx to automatically generate documentation from source code comments.

Continual refinement for a healthy codebase

As your codebase grows and evolves, it’s essential to periodically revisit and fine-tune existing code. This practice ensures that your code stays clean, efficient, and in line with current best practices. For instance, an algorithm that was efficient a year ago may now have a more optimized version, or a library you’re using might have received updates that you can take advantage of.

To systematically address this, consider dedicating specific sprints or time periods exclusively to code refactoring and reducing technical debt. For example, you could allocate the last week of every development cycle to revisit sections of the code that have been flagged for optimization or refactoring. This focused effort ensures that your codebase doesn’t accumulate quick fixes or workarounds that can make it harder to maintain and scale over time.

In addition, encourage a culture of detailed code reviews that go beyond just assessing functionality. These reviews should also scrutinize the quality of the code, examining factors like readability, efficiency, and adherence to coding standards. Peer feedback during these reviews can be invaluable for identifying areas that may require refactoring. For example, a team member might notice that a particular function is overly complex and suggest breaking it down into smaller, more manageable functions, thereby improving both readability and maintainability.

By continually refining your code, dedicating time to tackle technical debt, and fostering a culture of thorough code reviews, you can maintain a high-quality, efficient codebase that is easier to work with and less prone to issues in the long run.

Monorepo culture at tech giants

Monorepo culture has been adopted by many tech giants and renowned companies due to the myriad advantages it offers. Take a quick look at how Google, Facebook, and Microsoft have adopted a monorepo culture:

Google

Google is often credited for popularizing the monorepo approach through its massive monolithic codebase known as Piper, which contains billions of lines of code and thousands of projects.

At Google, a culture of shared ownership encourages developers to access and contribute to any part of the codebase. This collaborative approach has led to consistent coding standards, enhanced collaboration, and easier code reuse.

In conjunction with this, Google created Bazel, a build tool designed to work with large codebases like theirs. Bazel supports incremental builds, ensuring only affected components are rebuilt, significantly speeding up the build process.

Facebook

Facebook also employs a monorepo for its vast collection of projects, including the main Facebook app, Instagram, and WhatsApp.

Facebook’s codebase encourages engineers to “move fast and break things,” meaning they actively engage in rapid innovation while also continuously refining and iterating.

In conjunction, Facebook uses Buck, a build system tailored for their monorepo. It ensures efficient and reproducible builds, which is vital given the scale and pace of their development.

Microsoft

Microsoft famously transitioned the Windows codebase to a monorepo using Git, the largest Git repo on the planet. With the move, Microsoft aimed to increase developer productivity, improve code sharing, and streamline the engineering system.

To manage the massive repository, Microsoft developed the Virtual File System for Git (VFS for Git). It allows the Git client to operate at a scale previously thought impossible by virtualizing the filesystem beneath the repo and making it appear as though all the files are present when, in reality, they are not.

These companies not only showcase the technical adaptability of monorepos but also emphasize the cultural shift essential for such a model’s success.

The benefit of monorepos

Deciding between monorepos and multirepos isn’t solely a technical decision—it encapsulates a team’s collaboration dynamics, accountability distribution, and holistic view toward software creation. When complemented with the right tools and a strong culture emphasizing shared ownership and ongoing refinement, monorepos can create a vibrant, streamlined, and unified framework for software initiatives, particularly for larger teams.

Beyond technical merits, monorepos foster an enhanced collaborative environment. They dissolve barriers between developers, promoting shared responsibility, comprehensive code reviews, and a unified development environment.

Together, these features make monorepos a compelling choice for teams seeking both technical efficiency and collaborative synergy.

Aviator: Automate your cumbersome processes

There are 4 key components to Aviator:

MergeQueue – an automated queue that manages the merging workflow for your GitHub repository to help protect important branches from broken builds. The Aviator bot uses GitHub Labels to identify Pull Requests (PRs) that are ready to be merged, validates CI checks, processes semantic conflicts, and merges the PRs automatically.
ChangeSets – workflows to synchronize validating and merging multiple PRs within the same repository or multiple repositories. Useful when your team often sees groups of related PRs that need to be merged together, or otherwise treated as a single broader unit of change.
TestDeck – a tool to automatically detect, take action on, and process results from flaky tests in your CI infrastructure.
Stacked PRs CLI – a command line tool that helps developers manage cross-PR dependencies. This tool also automates syncing and merging of stacked PRs. Useful when your team wants to promote a culture of smaller, incremental PRs instead of large changes, or when your workflows involve keeping multiple, dependent PRs in sync.

Try it for free.

The post What is a monorepo and why use one? first appeared on Aviator Blog.

The post What is a monorepo and why use one? appeared first on Aviator Blog.

Building a CI/CD pipeline for a Google App Engine site using CircleCI

Brian Neville-O'Neill — Mon, 27 Nov 2023 18:21:14 +0000

In this article, we will build a CI/CD pipeline for a Google App Engine Site using CircleCI.

Prerequisite

Python installed on your system
Google Cloud CLI installed

What we are building

Documentation site and connect it to GCP (Google Cloud Platform)
Using CircleCI for automation

What is CircleCI?

CircleCI is a popular choice for software engineers, particularly DevOps engineers when working on automation and overall CI/CD integrations. The CI/CD platform helps software teams automate the process of building, testing, and deploying code. As a cloud-based platform, CircleCI allows you to seamlessly integrate with any version control system you choose, such as GitHub, Bitbucket, or GitLab. However, we will be working with GitHub on this article.

One cool thing about CircleCI is that it lets developers define pipelines that automate the process of building, testing, and deploying code. Pipelines are composed of jobs, which are individual steps in the CI/CD process. Jobs can be configured to run on various platforms, including Linux, macOS, and Windows.

Why Circle CI?

CircleCI is a friendly tool for teams of all sizes, ranging from small startups to large enterprises. No wonder the tool is usually “top of the ladder” during CI/CD integrations. It is a powerful tool that can help teams improve the quality and speed of their software development process.

Improved code quality: CircleCI can help enhance code quality by automating the testing process.
Reduced deployment time: CircleCI can help reduce the time it takes to deploy code by automating the process of building and deploying.
Increased confidence in releases: CircleCI can help increase confidence in releases by ensuring that code is thoroughly tested before deployment.
Improved team communication: CircleCI can help to improve team communication by providing a central location for monitoring the progress of builds and tests.

Relevant CircleCI features

Some of the core features of CircleCI include:

Parallelism: Jobs can be run in parallel to improve the speed of the CI/CD process.
Caching: CircleCI can cache build artifacts and test results to improve the speed of subsequent builds.
Notifications: CircleCI can notify team members when builds fail or pass.
Monitoring: CircleCI provides a dashboard that allows teams to monitor the progress of their builds and tests.

Getting started

To get started, we first need to create a free GitHub repo (I assume you already know how to do that). The next step is to clone the empty repo. After this, let’s create a Python virtual environment by running the following command on your terminal:

pipenv shell

This is what it should look like after a successful installation:

Next is to run the command:

pip install sphinx

Sphinx is a popular documentation generator written in Python that is widely used for creating high-quality documentation for Python projects. It is known for its ease of use, comprehensive features, and extensive support for various output formats.

The next step is to get a Sphinx quick start. To do this, head over to the get started section of Sphinx’s official site and run this command:

sphinx-quickstart

Once you run the command, you get asked a series of questions, exactly the ones in the image below:

Respond to these questions until the whole process is complete.

This whole process creates a build and source directory. We are also going to install Sphinx make files by running this command:

make html

After running the command, your project should look like this in your code editor:

Within the build directory, we have our website files, which look this:

📦build
 ┣ 📂doctrees
 ┃ ┣ 📜environment.pickle
 ┃ ┗ 📜index.doctree
 ┗ 📂html
 ┃ ┣ 📂_sources
 ┃ ┃ ┗ 📜index.rst.txt
 ┃ ┣ 📂_static
 ┃ ┃ ┣ 📜alabaster.css
 ┃ ┃ ┣ 📜basic.css
 ┃ ┃ ┣ 📜custom.css
 ┃ ┃ ┣ 📜doctools.js
 ┃ ┃ ┣ 📜documentation_options.js
 ┃ ┃ ┣ 📜file.png
 ┃ ┃ ┣ 📜language_data.js
 ┃ ┃ ┣ 📜minus.png
 ┃ ┃ ┣ 📜plus.png
 ┃ ┃ ┣ 📜pygments.css
 ┃ ┃ ┣ 📜searchtools.js
 ┃ ┃ ┗ 📜sphinx_highlight.js
 ┃ ┣ 📜.buildinfo
 ┃ ┣ 📜genindex.html
 ┃ ┣ 📜index.html
 ┃ ┣ 📜objects.inv
 ┃ ┣ 📜search.html
 ┃ ┗ 📜searchindex.js

When we run our project on localhost:8000, this is what it looks like on the browser:

Congratulations, we have our documentation site live!

Creating a GCP project

In this section, we will create a brand new GCP project to figure out which settings need to be tweaked or updated from the base. The next thing is to create our app engine app.yaml. Google provides a walkthrough on how to host a static website using GAE. Here, we can copy this YAML file:

runtime: python27
api_version: 1
threadsafe: true

handlers:
- url: /
  static_files: www/index.html
  upload: www/index.html

- url: /(.*)
  static_files: www/1
  upload: www/(.*)

Create an app.yaml file on your editor and paste this code. We then have to edit the YAML file to point it to the proper location where the website files live. To point your Gcloud command line install to this project, use this command:

gcloud init --project=<"project ID">

Here, you will prompted to log in to Google Cloud like this:

Follow the link provided to get your authorization code.

On the GCP dashboard, navigate to “App Engine” and run the command shown there on your terminal:

glcloud app deploy

You will get a prompt asking you to choose the location you would like your app to be deployed:

Choose anyone! and your app will successfully deploy. It’s time to push our code to Github! Alternatively, you can clone my GitHub repo here.

Link Github repo to CircleCI

The first thing you need to do is create a CircleCI account and link your Github to it. The process is pretty straightforward. Our dashboard should look like this after creating and connecting our project to CircleCI.

Next, in our code editor, we will create a folder named .circleci and a config.yaml file inside it which contains a code that works like this: first, it defines a workflow, then, the workflow will say; each time we push to the main branch, run this set of jobs. We will also define that job, which will contain the logic of where we build our documentation site and deploy it to the Google App engine.

workflows:
  version: 2
  build_and_deploy:
    jobs:
      - build_and_deploy:
        filters:
          branches:
            only:
              - main

CircleCI will only run this workflow when we push to the main branch. Now, to define the job:

jobs:
  build_and_deploy:
    docker:
      - image: busybox
    steps:
      - run:
          name: hello world
          command: |
            echo "Hello world"

You can check our formatting with CircleCI. To do this, first install CircleCI CLI and run this command:

circleci config validate

Response:

In our CircleCI, you can see the tests, processes, and workflows whenever we push to the main branch on GitHub:

Awesome! We have successfully created a CI/CD pipeline. Now, whenever we make changes to our code base or documentation, we can just push to the main, and CircleCI will pick up that change (as demonstrated in the image above) and the job or workflow and make deployments a few minutes later.

Conclusion

This article provided a step-by-step guide on building a CI/CD pipeline for a Google App Engine site using CircleCI. We covered setting up a Python environment, using Sphinx for documentation, and integrating the project with Google Cloud Platform.

The process demonstrated the benefits of automating deployments via CircleCI, including enhanced code quality, reduced deployment time, and improved team communication. This guide highlights the efficiency and effectiveness of CircleCI in streamlining development processes, making it an invaluable tool for modern software development teams.

Aviator: Automate your cumbersome processes

There are 4 key components to Aviator:

MergeQueue – an automated queue that manages the merging workflow for your GitHub repository to help protect important branches from broken builds. The Aviator bot uses GitHub Labels to identify Pull Requests (PRs) that are ready to be merged, validates CI checks, processes semantic conflicts, and merges the PRs automatically.
ChangeSets – workflows to synchronize validating and merging multiple PRs within the same repository or multiple repositories. Useful when your team often sees groups of related PRs that need to be merged together, or otherwise treated as a single broader unit of change.
TestDeck – a tool to automatically detect, take action on, and process results from flaky tests in your CI infrastructure.
Stacked PRs CLI – a command line tool that helps developers manage cross-PR dependencies. This tool also automates syncing and merging of stacked PRs. Useful when your team wants to promote a culture of smaller, incremental PRs instead of large changes, or when your workflows involve keeping multiple, dependent PRs in sync.

Try it for free.

The post Building a CI/CD pipeline for a Google App Engine site using CircleCI first appeared on Aviator Blog.

The post Building a CI/CD pipeline for a Google App Engine site using CircleCI appeared first on Aviator Blog.