Testing your Infrastructure in CI/CD Pipeline

Yuncheng Yang — Wed, 08 Mar 2023 06:39:25 +0000

Testing Infrastructure ad Code

These days, everyone's using Infrastructure as Code (IaC) to manage their infrastructure.

🛠️ Some popular Infrastructure as Code (IaC) tools include:

Terraform: Open-source IaC tool for managing infrastructure across multiple cloud platforms and providers. Flexible and reliable.
AWS CloudFormation: IaC tool to manage and automate AWS resources. Allows previewing changes before deployment and rolling back infrastructure if needed.
Azure Resource Manager: Robust IaC tool for managing infrastructure using ARM templates. Supports RBAC and provides excellent organization tools.
Google Cloud Deployment Manager: Native infrastructure deployment service for Google Cloud Platform. Uses declarative language and offers built-in console and UI support.
Pulumi: Flexible IaC tool that supports various programming languages. Allows for automation options for deployment delivery, quality assurance, easy auditing, and comprehensive identity control.
Ansible: Red Hat's orchestration and configuration tool that uses YAML-based playbooks to configure infrastructure. Easy to use with robust default configuration and allows for custom modules.

It's great because it's scalable, automated, and ensures consistency across environments.

But when it comes to testing that IaC, a lot of us fall back on the old "deploy and see" method: deploying your infrastructure and then checking to see if it's actually doing what it's supposed to be doing.
It's a time-consuming and error-prone process, and it's not enough to catch all the potential issues. Especially when you're dealing with complex infrastructures with multiple environments.

To make the process more efficient and reliable,** you can create tests for infrastructure that can be run with a CI/CD pipeline.**

Standardize the deployment environment
Automate many of the manual tasks involved in testing and deployment

Use 🌟SQL🌟 with Cloud

If you prefer not to learn and use a domain-specific language to manage resources on the cloud, there are several tools available.

Using SQL also enables the use of relational algebra concepts and operations, such as joins, selections, and projections.

Steampipe: a Postgres-based open-source tool that uses foreign data wrappers and SDK-based provider plugins to query your cloud infrastructure using SQL. Steampipe supports multiple cloud providers, including AWS, Azure, and Google Cloud Platform.
CloudQuery: an open-source tool that collects inventory and configuration data from cloud providers using the provider SDKs. CloudQuery allows you to query your cloud infrastructure using SQL. CloudQuery supports multiple cloud providers, including AWS, Azure, Google Cloud Platform, and Kubernetes.
IaSQL: a Postgres-based tool that allows you to query and provision resources in AWS.
StackQL: an open-source query engine that allows you to query, provision, and interact with cloud and SaaS resources using SQL.
StackQL cloud provider interfaces are extensions of the provider's OpenAPI specification, exposing all operations available in a provider's API using SQL grammar. StackQL's document-based provider definitions make bringing your providers or APIs, and extending to other cloud or SaaS platforms easy.

Testing tools

There are several IaC testing tools available, including Terratest, Kitchen-Terraform, and Chef InSpec.

These tools can help automate the process of testing your infrastructure code and catch issues.

Terratest

Terratest is a Go library that provides patterns and helpers for testing infrastructure.

Terratest allows you to execute your real IaC tools (Terraform, Packer, etc.) to deploy real infrastructure in a real environment.
You can then use the tools built into Terratest to validate that the infrastructure works correctly in that environment by making HTTP requests, API calls, SSH connections, etc. Terratest can undeploy everything at the end of the test.

Here's an example of how to use Terratest to test a Terraform module:

package test

import (
  "fmt"
  "testing"

  "github.com/gruntwork-io/terratest/modules/terraform"
  "github.com/stretchr/testify/assert"
)

func TestTerraformModule(t *testing.T) {
  terraformOptions := &terraform.Options{
    TerraformDir: "../examples/simple",
  }

  defer terraform.Destroy(t, terraformOptions)

  terraform.InitAndApply(t, terraformOptions)

  output := terraform.Output(t, terraformOptions, "output_name")
  assert.Equal(t, "expected_output", output)
}

This test will run Terraform to create the infrastructure, then check that the output matches the expected output.

Chef InSpec

Chef InSpec is an open-source testing and compliance automation framework. It allows you to write tests for your infrastructure code and check that it meets the desired state.

You can use InSpec to test various infrastructure components, such as operating systems, cloud providers, and databases. InSpec can also test compliance with industry standards like PCI DSS, HIPAA, and CIS benchmarks.

Here's an example of how to use Chef InSpec to test a Terraform module:

control "example-control" do
  title "Example Control"
  desc "An example InSpec control"

  describe aws_instance('example-instance') do
    it { should be_running }
    its('instance_type') { should eq 't2.micro' }
  end

  describe aws_security_group('example-security-group') do
    it { should allow_in(port: 22, ipv4_range: '0.0.0.0/0') }
  end
end

This test will check that an AWS EC2 instance named example-instance is running and is of the correct instance type, and that the security group example-security-group allows incoming traffic on port 22 from any IP address.

Kitchen-Terraform

Kitchen-Terraform is a Ruby-based test harness for Terraform modules.

Kitchen-Terraform allows automated testing of Terraform code, including syntax and linting checks.
It can also create and destroy infrastructure in a testing environment and run tests against it. It can be integrated with Chef InSpec to perform additional testing of infrastructure code for a comprehensive testing process.

Here's an example of how to use Kitchen-Terraform to test a Terraform module:

---
driver:
  name: terraform

provisioner:
  name: terraform

verifier:
  name: terraform

platforms:
  - name: terraform

suites:
  - name: default
    verifier:
      rakefile: test/integration/default/default_test.rb

This test will create a Terraform instance and apply the Terraform configuration in the default suite. After that, it will run the tests in default_test.rb, which can include any number of tests.

GitHub Action: stackql-assert

The stackql-assert is a GitHub action that allows you to run a StackQL query with minimal setup.
It is used to test assertions against the results of a StackQL query, which can validate the state of a resource after an IaC or lifecycle operation has been performed, or validate the system (e.g., CSPM or compliance queries).

For example, you can use the stackql-assert action in a Terraform deployment pipeline:

- name: Terraform Apply
        env:
          TF_VAR_google_credentials: ${{ secrets.GOOGLE_CREDS }}
        id: apply
        run: cd terraform; terraform apply -no-color -var-file=stackql-demo.tfvars -auto-approve

- name: check terraform deployment using stackql-assert - should fail
  uses: stackql/stackql-assert@v1.0.2
  with:
    auth_obj_path: './stackql/auth.json'
    test_query_file_path: './stackql/scripts/check-terraform-instances.iql'
    expected_results_str: '[{"name":"terraform-test-1","name":"terraform-test-2"}]'

This example run a StackQL SELECT query and compare the actual result with an expected result after a Terraform deployment.
This can test specific configuration properties of the resource (for compliance or policy enforcement) or just the existence of the resource.

Reference

https://bluelight.co/blog/best-infrastructure-as-code-tools#what-are-the-best-infrastructure-as-code-tools-for-2022

https://qxf2.com/blog/testing-infrastructure-as-code-beginner/

Forem: Yuncheng Yang