Forem: Adam Connelly

Spacelift Module Registry – What It is and How to Use It

Adam Connelly — Wed, 01 Feb 2023 09:51:43 +0000

What is the Module Registry and Why Do I Care?

The Spacelift module registry provides a private Terraform module registry that is fully compatible with Terraform. The module registry allows you to define reusable Terraform modules and share them with various different audiences. This includes sharing modules with multiple Stacks in your Spacelift account but also includes sharing modules with other accounts.

The module registry is available to all Spacelift accounts regardless of tier, although certain functionality, like the ability to share modules with other Spacelift accounts, is only available on the Enterprise tier.

Using the Registry

Using the module registry involves the following steps:

Declaring the module source.
Adding the module to your Spacelift account.
Deploying a version of that module.
Consuming that version.

1. Declaring the module source

Creating a module is simple, and if you already have an existing Terraform module, you’re almost ready to go. The only additional steps you need to take are defining a config.yml file that describes the module as well as pointing Spacelift at your module source.

The basic folder structure of a Spacelift module looks like this:

- /
 + - .spacelift/
   |
   + - config.yml
 + - examples/
   |
   + - example-1/
   + - example-2/
 + - main.tf
 + - ...
 + - something-else.tf

As you can see, as well as any Terraform definitions your module requires, there’s also a .spacelift folder. This folder contains a single file: config.yml. This config file defines certain pieces of information about your module, including its version, along with any test cases.

A minimum example of a config.yml file looks like this:

# version defines the version of the config.yml configuration format. Currently the only option is 1.
version: 1

# module_version defines the version of the module to publish.
module_version: 0.0.1

The folder structure above also includes an examples folder. This folder is not required, and also does not need to be called examples. It would typically contain one or more test cases that can double as usage examples for your module.

2. Adding the module to your Spacelift account

There are two main ways to add a module to your Spacelift account: using the UI, or using the Terraform provider. The following steps will show how to add the module via the UI, but if you prefer to manage your account via Terraform, take a look at the spacelift_module resource for more information.

The first step is to navigate to the modules section of the Spacelift UI and click the Add module button:

Next, choose the repository containing your module along with your main branch.

For now, we’ll assume that your repo only contains a single module, so the project root can be left empty:

For simple use-cases, you should be able to use the default behavior settings:

Finally, fill out the description settings:

For simple situations where your repository contains a single module and is named using HashiCorp’s module repository naming scheme of terraform-<PROVIDER>-<NAME>, Spacelift will automatically infer the Name and Provider for you, meaning that you don’t need to enter anything on this screen.

For example, the repository used in the screenshots above was named terraform-blog-8ball, giving it a name of 8ball and a provider of blog.

Congratulations – you now have a module:

3. Publishing a version

Before you can actually use your module, you need to publish a version. To publish the initial version of your module, click on the Trigger button. Once the version has been published, it should look something like this:

Further versions can be published simply by pushing changes to your Git repository. The only thing to be aware of is that new versions will only be published if you update the module_version in your config.yml file. Another approach to module versioning using git tags is described later in this post.

4. Consuming that version

OK great – we’ve got a version! Now what?

Now that you have a version published, you need to consume that version from a stack. Consuming a Spacelift module is exactly the same as consuming a module from the Terraform registry. The only difference is you need to specify a different source:

module "8ball" {
 source  = "spacelift.io/adamconnelly/8ball/blog"
 version = "0.0.1"
}

In fact, if you click the Show Instructions button in the Spacelift UI, it will give you a usage example:

NOTE: in order to be able to access your module, Terraform needs to be authenticated with the Spacelift module registry. This happens automatically for you when running Terraform inside a Spacelift stack.

If you want to be able to use a module from outside Spacelift, for example, if you want to test a module out locally, there are a few more steps involved. A few approaches are outlined later in this post.

Testing

One of the pieces of functionality that the Spacelift registry provides that isn’t part of the standard Terraform module registry is testing functionality. Each module can contain one or more test cases.

Test cases serve two purposes: they provide you with confidence that any changes you make haven’t broken your module, and they serve as usage examples for any consumers.

Defining a test case

Each test case is like a mini stack that uses your module, making sure that it can be applied and then destroyed correctly. To add a test case, create a Terraform file that references your module:

module "basic_example" {
 source  = "../../"
}

And then reference that test case in your config.yml file:

version: 1
module_version: 0.0.2

tests:
 - name: Basic Example
   project_root: examples/basic-example

Note, when referencing the module, you should use a relative reference (for example source = “../../”) and avoid specifying a version constraint. This ensures that your tests are run against the current version of your module code rather than a previously published version.

Now that your test case has been defined commit your code, push and create a pull request. After opening your PR, you should be able to see it along with any test cases defined in the PRs tab for your module:

If you click on a test case you can view the Spacelift run, showing the full plan, apply and destroy lifecycle:

Note: test cases will only be triggered if the version number of your module is increased. Please remember to do this, otherwise, your changes will be ignored!

Extending test cases via hooks

Although the module registry gives you the ability to run one or more test cases, it doesn’t provide a particular test framework out of the box. However, you can extend your test cases using Workflow Customization (a.k.a. hooks) to run any additional checks you need.

Let’s pretend that we want to extend the magic 8-ball module to allow a list of custom outcomes to be supplied. The first step would be to add a new variable to our module to support this:

variable "custom_outcomes" {
 type        = list(string)
 default     = null
 description = "A set of custom outcomes that should be used by the 8-ball"
}

At this point we can define our new test case by extending our config.yml file to look like this:

version: 1
module_version: 0.0.3

tests:
 - name: Basic example
   project_root: examples/basic-example

 - name: Custom outcomes
   project_root: examples/custom-outcomes
   environment:
     TF_VAR_custom_outcomes: '["definitely","likely","surely"]'
   after_apply:
     # Get the value of our Terraform output
     - ACTUAL_OUTCOME=$(terraform output -raw outcome)
     # Check it's in our set of custom outcomes
     - echo "definitely likely surely" | grep "$ACTUAL_OUTCOME"

As you can see, a new test case has been defined named Custom outcomes. It defines an environment variable TF_VAR_custom_outcomes to allow us to define the set of outcomes via the config.yml file, and defines some after_apply hooks that run the additional checks.

In this simple example, all we want to do is get the value of a Terraform output (terraform output -raw outcome), and then check whether it exists in our list of expected outcomes (the echo and grep commands). The reason that this works is that an exit code of 0 from our custom hook will allow the test case to pass, whereas a non-zero exit code causes a failure.

Now that our config.yml file has been updated, it’s time to create the Terraform definition for the test case. It should look something like the following:

# Define an input variable that can be supplied via the test case definition
variable "custom_outcomes" {
 type        = list(string)
 description = "A set of custom outcomes that should be used by the 8-ball"
}

# Invoke our module, and pass through the custom_outcomes variable
module "custom_outcomes_example" {
 source          = "../../"
 custom_outcomes = var.custom_outcomes
}

# Define an output to pass the `outcome` output from our module out of our test case's root
# module so that we can assert on it via the `after_apply` hook
output "outcome" {
 value = module.custom_outcomes_example.outcome
}

If we try to run this test case now, it should result in a failure since we haven’t actually implemented the changes. You can see that in the following screenshot:

We could then add an implementation like the following to our module to get the test case to pass:

# Allow a set of custom outcomes to be specified
variable "custom_outcomes" {
 type        = list(string)
 default     = null
 description = "A set of custom outcomes that should be used by the 8-ball"
}

locals {
 default_outcomes = ["probably", "maybe", "unlikely", "outcome not looking good", "ask me again"]

 # Use the custom outcomes if they are provided, otherwise use the default outcomes
 outcomes = var.custom_outcomes != null ? var.custom_outcomes : local.default_outcomes
}

resource "random_integer" "result" {
 min = 0
 max = length(local.outcomes) - 1
}

# Output the result
output "outcome" {
 value = local.outcomes[random_integer.result.result]
}

Environment and Cloud Credentials

Spacelift modules support environment variables, context attachments and Cloud credential integrations, just like a stack.

The difference between module and stack environments is that with stacks, the environment is made available to each run, whereas with a module, the environment is made available to each test case that runs.

Cloud Credentials

Since each module test-case goes through a full plan-apply-destroy lifecycle, it’s important to understand that Terraform resources will be created by test case executions even though they will be destroyed as part of the test run. If your module creates resources in a cloud provider, we would recommend that you use a completely separate account dedicated for testing to avoid accidentally impacting other environments.

On a related note, if you have multiple test-cases defined for your module, be aware that they could execute simultaneously, leading to conflicts with resource names. For example, if you created a module that contained the following S3 bucket definition, it could fail if two module tests attempted to create the resource at the same time:

resource "aws_s3_bucket" "conflicting_bucket" {
 bucket = "test-bucket"
}

Instead, it would make more sense to use a bucket_prefix, or to allow the bucket name to be parameterized via a Terraform variable:

resource "aws_s3_bucket" "conflicting_bucket" {
 bucket_prefix = "test-bucket"

 # or:
 # bucket = var.bucket_name
}

Overriding test case environment

The environment for test cases can be overridden on a case-by-case basis via the config.yml file. The config file allows a set of default environment variables to be specified and then allows that environment to be overridden by each test:

version: 1
module_version: 0.0.1

test_defaults:
 environment:
   TF_VAR_bucket: "test-bucket"

tests:
 - name: Test case 1
   project_root: examples/test-case-1

 - name: Test case 2
   project_root: examples/test-case-2
   environment:
     TF_VAR_bucket_name: "test-bucket-2"

 - name: Test case 3
   project_root: examples/test-case-3
   environment:
     TF_VAR_bucket_name: "test-bucket-3"

Please note, Terraform variables don’t automatically propagate through to the underlying module used by each test case. This means that for the example above to work, each test case would need to contain Terraform code similar to the following:

# Declare a variable that will be populated by our test case config
variable "bucket_name" {
 type = string
}

module "test" {
 source = "../../"

 # Reference that variable, and pass it through to our module
 bucket_name = var.bucket_name
}

Integration with VCS Systems

The module registry integrates with your VCS system to provide feedback while working on changes to modules in PRs, and also to connect published versions back to their commits. After publishing a new version, you should see a status against your commit that looks something like this:

Clicking on details shows more information about the version that was published:

If your module has test cases associated with it, and you push a PR, the result of the checks will be displayed:

Storing Multiple Modules in a Single Repo

Spacelift also supports using a single Git repo to store multiple modules. For example, you might have a structure like this:

- /
 + - vpc/
   + - .spacelift/
     |
     + - config.yml
   + - main.tf
 + - eks/
 + - sqs/
 + - s3/

In this example, we’re defining four modules: vpc; eks; sqs; and s3. The contents of the eks, sqs and s3 modules have been omitted for simplicity, but each would also contain a config.yml file along with any required Terraform definitions.

To support this, simply set the project root of each module. Here’s what the vpc module’s settings might look like:

Versioning

Manually via the config.yml file

The simplest way to version your modules is to manually update the module_version property in your config.yml file:

version: 1

# Update module_version before publishing a new version
module_version: 0.0.1

A typical workflow might look like this:

Make any required changes to your module.
Edit config.yml and increase module_version as required.
Create a pull request and make sure any tests pass.
Once your PR has been approved, merge it to publish the new version.

Automatically via tags

If you don’t want to have to manually edit your config.yml file each time you make a change, another option is to use a tag based approach. To do this, create a new push policy with the following contents:

package spacelift

module_version := version {
   version := trim_prefix(input.push.tag, "v")
}

propose { true }
track { input.push.tag != "" }

This policy tells Spacelift to trigger version publishing whenever a new tag in the format vx.y.z is pushed to your repo and also tells Spacelift to use the tag as the version number. Because Terraform module versions don’t support the v prefix, the policy removes that from the start of the tag.

To enable the policy, attach it to your module:

Now that your policy is in place, pushes to your main branch will not trigger new versions to be published. Instead a new version will be published when a new tag is pushed to your repository.

Automatically Triggering Consumers

Spacelift keeps track of the stacks that consume particular modules. This means that you can automatically trigger any consumers when a new version of the module is published. To do this, you need to do two things:

Make sure that any Terraform version constraints allow new versions to be used.
Attach a trigger policy to your module telling it to trigger any consumers.

Version constraints

When referencing a Terraform module, you can specify an optional version constraint. This defines the set of versions that your stack is able to use. For example, if you published version 2.0.0 of a module, but had the following module reference, the new version would not be picked up automatically:

module "version_example" {
 source  = "spacelift.io/adamconnelly/8ball/blog"
 version "~> 1.0.0"
}

Trigger policy

Trigger policies for modules accept the list of consumers of the current latest version of the module as an input to the policy. This allows you to write a simple trigger policy like the following to automatically re-trigger any consumers:

package spacelift

trigger[stack.id] {
 stack := input.stacks[_]
}

Once you have attached this trigger policy to your module and released a new version of the module, you should see runs triggered on any stacks that are currently using the latest version of the module. It should look something like this:

NOTE: Please be aware that the list of stacks provided to the trigger policy only includes those that are using the _latest _ version of the module at the time that the new version is triggered. This means that if the latest version of a module was 0.0.5, but a stack is currently only using version 0.0.4 of your module, it will not automatically be included in the list of consumers. To fix this, you will need to manually trigger your stack once to make sure it’s using the latest version.

For more information, see our trigger policy documentation.

Sharing Modules with Other Accounts

Spacelift modules can optionally be used by other Spacelift accounts than the one the module is defined in. This can be useful, for example, if you have multiple Spacelift accounts or if you have clients who also use Spacelift who use your custom Terraform modules.

The list of accounts that can access a module can be configured via the sharing section of the module settings:

For more information, see our documentation. Also, please note that module sharing is only available to customers on the Enterprise tier.

Using Modules Outside of Spacelift

Using modules locally

If you want to use Spacelift modules while running Terraform locally, you need to tell Terraform how to authenticate with the Spacelift registry. The easiest way to do this is via the terraform login command. To do this, run the following command and then follow the instructions to login to your Spacelift account via your web browser:

terraform login spacelift.io

Once authentication has completed successfully, you should see a message similar to the following in your console:

Now that you are authenticated, you can add a reference to a Spacelift module and run terraform commands locally as normal.

Using modules in an automated environment

If you need to use a module in an automated environment outside of Spacelift, the best bet is to use an API key. Please see our documentation for more information on generating and using API keys.

Wrapping Up

For more information about the Spacelift module registry, please see the documentation available at docs.spacelift.io.

💡 You might also like:

Monitoring Your Spacelift Account via Prometheus

Adam Connelly — Wed, 17 Aug 2022 11:53:47 +0000

For a while now, we have received feedback from customers that they would like to be able to connect their Spacelift account to external monitoring systems. Today we are excited to announce the Prometheus Exporter for Spacelift! We also have plans to add support for , so if you’re using another system, we should have you covered soon.

What Can It Do?

The Prometheus exporter allows you to monitor various metrics about your Spacelift account over time. You can then use tools like Grafana to visualize those changes and Alertmanager to take actions based on account metrics. Several metrics are available, and you can find the complete list of available metrics here. Below are a few examples of the information the exporter currently provides:

The number of runs pending and currently executing in both public and private worker pools.
The number of workers in a pool.
Usage information, including the number of public and private worker minutes used during the current billing period.

Once you have that information, it opens a number of possibilities, including visualizing information about your account via Grafana dashboards, alerting on events like a lack of private workers, as well as using that information to autoscale worker pools via the Horizontal Pod Autoscaler.

To give you a taste, here’s an example Grafana dashboard showing some of the information available from the exporter:

How Does It Work?

The Prometheus exporter is an adaptor between Prometheus and the Spacelift GraphQL API. Whenever Prometheus asks for the current metrics, the exporter makes a GraphQL request and converts it into the metrics format Prometheus expects.

The following diagram gives an overview of this process:

Getting Started

Ok, great, but how do I use the Prometheus Exporter?

To help with setup and deployment, we are going to guide you through the following:

Deploy a Prometheus stack to a Kubernetes cluster.
Deploy the exporter.
Configure Prometheus to monitor it.

The Quick Start section in the exporter repo outlines several options for deploying the exporter. Review the best deployment option that makes sense, depending on your setup and requirements.

If you already have a Prometheus stack setup and have plenty of experience, feel free to skip to the “Installing the Prometheus Exporter” section.

We also assume that you already have a Kubernetes cluster provisioned (hopefully via Spacelift!) and available to complete the following steps. If not, a local installation like Minikube will work fine for illustration purposes.

Installing kube-prometheus-stack

The first step is to install the kube-prometheus-stack Helm chart. You can do that with the following commands:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install --create-namespace -n monitoring kube-prometheus prometheus-community/kube-prometheus-stack

That will install a Prometheus, Grafana, and Alertmanager stack into your cluster’s namespace called monitoring. Run the kubectl get pods -n monitoring command to see the installed components.

$ kubectl get pods -n monitoring
NAME                                                     READY   STATUS    RESTARTS       AGE
alertmanager-kube-prometheus-kube-prome-alertmanager-0   2/2     Running   14 (58m ago)   20d
kube-prometheus-grafana-5cd6d47467-2rt88                 3/3     Running   21 (58m ago)   20d
kube-prometheus-kube-prome-operator-54b7488f58-7fmfv     1/1     Running   7 (58m ago)    20d
kube-prometheus-kube-state-metrics-8ccff67b4-zlfx9       1/1     Running   10 (58m ago)   20d
kube-prometheus-prometheus-node-exporter-zv8zn           1/1     Running   7 (58m ago)    20d
prometheus-kube-prometheus-kube-prome-prometheus-0       2/2     Running   14 (58m ago)   20d

Getting API Credentials

The Prometheus exporter authenticates to the Spacelift GraphQL API using an API key. Follow the guide to create a new API key required by the explorer. After you create your key, please note the API Key ID and API Key Secret – you’ll need both when configuring the exporter.

Installing the Prometheus Exporter

The exporter is available via a Docker image published to the public.ecr.aws/spacelift/promex container registry. To deploy the exporter to Kubernetes, we need to create the following resources:

A Deployment – to run the exporter container.
A Service – to allow Prometheus to scrape the exporter.
A ServiceMonitor – to let Prometheus know that it needs to scrape the exporter.

The following is an example Deployment definition for running the exporter. Make sure to replace the <account name>, <API Key> and <API Secret> placeholders with the correct values:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: spacelift-promex
 labels:
   app: spacelift-promex
spec:
 replicas: 1
 selector:
   matchLabels:
     app: spacelift-promex
 template:
   metadata:
     labels:
       app: spacelift-promex
   spec:
     containers:
     - name: spacelift-promex
       image: public.ecr.aws/spacelift/promex:latest
       ports:
       - name: metrics
         containerPort: 9953
       readinessProbe:
         httpGet:
           path: /health
           port: metrics
         periodSeconds: 5
       env:
       - name: "SPACELIFT_PROMEX_API_ENDPOINT"
         value: "https://<account name>.app.spacelift.io"
       - name: "SPACELIFT_PROMEX_API_KEY_ID"
         value: "<API Key>"
       - name: "SPACELIFT_PROMEX_API_KEY_SECRET"
         value: "<API Secret>"
       - name: "SPACELIFT_PROMEX_LISTEN_ADDRESS"
         value: ":9953"

Note: The example above defines the API key and secret as normal environment variables. We would recommend that you use Kubernetes Secrets for anything other than testing purposes.

Next, create a Service to expose the exporter:

apiVersion: v1
kind: Service
metadata:
 name: spacelift-promex
 labels:
   app: spacelift-promex
spec:
 selector:
   app: spacelift-promex
 ports:
   - name: http-metrics
     protocol: TCP
     port: 80
     targetPort: metrics

And finally create your ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
 name: app-monitor
 labels:
   app: app-monitor
   release: kube-prometheus
spec:
 jobLabel: app-monitor
 selector:
   matchExpressions:
   - {key: app, operator: Exists}
 namespaceSelector:
   matchNames:
   - monitoring
 endpoints:
 - port: http-metrics
   interval: 15s
   path: "/metrics"

The ServiceMonitor definition above tells Kubernetes to scrape any services that contain an app label. The granularity of the metrics can be increased or decreased depending on requirements and organizational standards. The above example is configured for 15-second intervals.

Viewing your Metrics

Once you have your Prometheus stack up and running and have deployed the Spacelift exporter, you can use port-forwarding to access each component. First, let’s port-forward the exporter to port 8080 locally using the following command:

kubectl port-forward service/spacelift-promex -n monitoring 8080:80

Assuming all is well, you should be able to see the raw metrics output by accessing http://localhost:8080/metrics:

You can also port-forward to your Grafana instance to view the metrics in Grafana:

kubectl port-forward service/kube-prometheus-grafana -n monitoring 8081:80

You can then quickly discover the available metrics via the Grafana Explore view:

Alerting

One of the things that I think is amazing about the Prometheus stack is the ability to use PromQL queries not just for monitoring but for defining alerts. For example, if we want to trigger an alert whenever our worker pool has no available workers for a specific time period, we can use a query like this:

max(spacelift_worker_pool_workers) by (worker_pool_id, worker_pool_name) <= 0

Similarly, if we want to alert when the number of queued runs for a pool gets too high, we can use a query like this:

max(spacelift_worker_pool_runs_pending) by (worker_pool_id, worker_pool_name) >= 10

Autoscaling

NOTE: This section is intended to outline some of the possibilities that exist when using the Prometheus exporter and should NOT be used as a guide for a production-ready autoscaling solution. For example, it does not consider properly draining workers before scaling them down to avoid in-progress runs being terminated.

Since we have metrics related to queued runs, we can use them to autoscale private workers. We can use the spacelift_worker_pool_runs_pending metric, which tells us how many runs for a given worker pool are waiting to be scheduled, to detect when we need to add more workers to our pool. Similarly, we can use the spacelift_worker_pool_workers and spacelift_worker_pool_workers_busy metrics to decide when to scale down.

We need to take both sets of metrics into account to avoid scaling down just because there are no queued runs. In this situation, we might still have runs in progress that need workers.

For this to work, we need the following components available:

A working Prometheus stack with the Spacelift Prometheus Exporter running.
A Spacelift worker pool.
A prometheus-adapter installation.
A Horizontal Pod Autoscaler resource to tell Kubernetes how to scale our worker pool.

For the sake of simplicity, I’m going to assume that you already have steps 1 and 2 covered, and let’s assume that you’ve deployed the Spacelift worker pool chart with the default settings. I’ll also assume that you have a standard installation of the kube-prometheus-stack chart.

The first thing we need to do is set up a Prometheus-adapter installation. The Prometheus-adapter works like a bridge between the Kubernetes metrics API and your Prometheus installation. It allows you to use Prometheus metrics to make autoscaling decisions within your cluster. We can visualize it something like this:

The Prometheus-adapter uses a configuration format to map between Prometheus queries and Kubernetes metrics. In our case, we can use something like the following to generate the two metrics we need:

prometheus:
 # The URL points at the Kubernetes Service for Prometheus
 url: "http://kube-prometheus-kube-prome-prometheus"

rules:
 default: false

 external:
 # Define the spacelift_worker_pool_runs_pending metric
 - seriesQuery: '{__name__=~"^spacelift_worker_pool_runs_pending$"}'
   resources:
     template: <<.Resource>>
   name:
     as: "spacelift_worker_pool_runs_pending"
   metricsQuery: max(spacelift_worker_pool_runs_pending) by (worker_pool_id, worker_pool_name)
 # Define the spacelift_worker_pool_utilization metric
 - seriesQuery: '{__name__=~"^spacelift_worker_pool_workers$"}'
   resources:
     template: <<.Resource>>
   name:
     as: "spacelift_worker_pool_utilization"
   metricsQuery: |
     (max(spacelift_worker_pool_workers_busy) by (worker_pool_id, worker_pool_name)
       / max(spacelift_worker_pool_workers) by (worker_pool_id, worker_pool_name))
     or vector(0)

After deploying the adapter, you can query your metrics via the Kubernetes API, using the following commands (assuming you’ve deployed everything to a namespace called monitoring):

kubectl get --raw /apis/external.metrics.k8s.io/v1beta1/namespaces/monitoring/spacelift_worker_pool_runs_pending
kubectl get --raw /apis/external.metrics.k8s.io/v1beta1/namespaces/monitoring/spacelift_worker_pool_utilization

Finally, we can create a Horizontal Pod Autoscaler definition to scale our worker pool Deployment based on those metrics automatically:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
 name: spacelift-worker-hpa
spec:
 scaleTargetRef:
   apiVersion: apps/v1
   kind: Deployment
   name: spacelift-worker
 minReplicas: 1
 maxReplicas: 15
 metrics:
 - type: External
   external:
     metric:
       name: spacelift_worker_pool_runs_pending
       selector:
         matchLabels:
           worker_pool_id: "01G8Y06VGHCT17453VEE9T4YBZ"
     target:
       type: AverageValue
       averageValue: 1
 - type: External
   external:
     metric:
       name: spacelift_worker_pool_utilization
       selector:
         matchLabels:
           worker_pool_id: "01G8Y06VGHCT17453VEE9T4YBZ"
     target:
       type: Value
       value: 0.8

Notice the metrics are grouped by the worker_pool_id; we can use this to target an individual worker pool when creating our HPA!

That’s it – bask in your autoscaling glory:

We are excited to learn how our customers use the exporter and the problems they solve with it! Feel free to provide feedback and comments via Issues in the Prometheus Exporter Github repository.

Running Spacelift CI/CD workers in Kubernetes using DinD

Adam Connelly — Mon, 07 Feb 2022 14:20:03 +0000

For a while now at Spacelift we’ve been hearing from users that they’d like to be able to run Spacelift worker pools in their Kubernetes clusters. As a first step towards that we decided to investigate whether we could run workers in Kubernetes using a Docker-in-Docker sidecar container. The rest of this post gives a rough overview of the architecture of Spacelift workers, and also explains how the sidecar container strategy works.

I’d just like to say thanks to Vadym Martsynovskyy for his great article about running Jenkins agents in Kubernetes. In that article he outlines the general approach taken here, and also describes some of the pros and cons of using Docker-in-Docker. It’s definitely well worth a read.

Background

Before going into more detail, it’s worth explaining what a Spacelift worker is, and giving a quick overview of our architecture. At Spacelift, we provide a hybrid-SaaS model like many other CI/CD systems. What this means in practice is that the control plane that handles scheduling runs is managed by us, but the execution of runs happens in a separate process called a worker. We provide a public worker pool, but also allow you to self-host your own workers, providing complete control over your infrastructure.

Perhaps confusingly, if you follow the instructions to create a private worker pool, you’ll notice that it involves downloading something called the “launcher”. The launcher is responsible for communicating with the Spacelift mothership (control plane), and creating and destroying worker containers using Docker for each Spacelift run:

As you can see, the launcher process needs access to a Docker daemon in order to create a container for each run.

Running in Kubernetes

We found that the launcher architecture maps very nicely to Kubernetes using a sidecar container to run the Docker daemon. For each worker in the pool, we need to create a Kubernetes Pod with two containers: one for the launcher; and another for Docker. The following shows a stripped down Deployment definition for creating a worker pool with 4 workers:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: worker-pool-1-spacelift-worker
spec:
 replicas: 4
 template:
   spec:
     containers:
       - name: launcher
         image: "public.ecr.aws/spacelift/launcher:latest"
         imagePullPolicy: Always
         env:
           - name: DOCKER_HOST
             value: tcp://localhost:2375
           - name: SPACELIFT_TOKEN
             value: "..."
           - name: SPACELIFT_POOL_PRIVATE_KEY
             value: "..."
         volumeMounts:
           - name: launcher-storage
             mountPath: /opt/spacelift
             subPath: spacelift
       - name: dind
         image: "docker:dind"
         imagePullPolicy: Always
         command: ["dockerd", "--host", "tcp://127.0.0.1:2375"]
         securityContext:
           privileged: true
         volumeMounts:
           - name: launcher-storage
             mountPath: /var/lib/docker
             subPath: docker
           - name: launcher-storage
             mountPath: /opt/spacelift
             subPath: spacelift
     volumes:
       - name: launcher-storage
         emptyDir: {}

The launcher communicates with the Docker-in-Docker sidecar container via TCP, which is configured via the DOCKER_HOST environment variable for the launcher container. This works because the containers in a Kubernetes Pod share the same IP address and port space, and can communicate with each other via localhost.

In addition, a shared volume called launcher-storage is mounted into each container. This is used by the launcher to store the workspaces for runs, along with other things like cached tool binaries (for example Terraform). The Docker sidecar needs access to the run workspaces in order to mount them into the worker containers, and it also uses that volume to store its image cache.

The last thing worth pointing out is that the dind container sets securityContext.privileged to true. This is required for Docker-in-Docker to function correctly.

Launcher Image

The Spacelift launcher is distributed as a statically linked binary, making it simple to build an image. As you can see, the Dockerfile simply copies the launcher into an Alpine container, and then sets the startup command:

FROM alpine:3.14

COPY spacelift-launcher /usr/bin/spacelift-launcher
RUN chmod 755 /usr/bin/spacelift-launcher
CMD [ "/usr/bin/spacelift-launcher" ]

This image is rebuilt and published to our public ECR repository any time the launcher binary is updated.

Helm Chart

We provide Terraform modules to make it really easy for users to deploy worker pools to AWS, Azure and GCP, and we wanted to provide a similar experience for Kubernetes. To achieve this, we created a Helm chart that makes it simple to deploy workers to Kubernetes clusters: https://github.com/spacelift-io/spacelift-workerpool-k8s.

Assuming you’ve already configured your worker pool in Spacelift and have access to the credentials for the pool, deploying this chart to your cluster simply requires two steps.

First, add the Spacelift Helm chart repository and update your local chart cache:

helm repo add spacelift https://downloads.spacelift.io/helm
helm repo update

Next, install the chart:

helm upgrade worker-pool-1 spacelift/spacelift-worker --install --set "credentials.token=<worker-pool-token>,credentials.privateKey=<worker-pool-private-key>"

Replace <worker-pool-token> and <worker-pool-private-key> with your own credentials, and make sure to base64-encode the private key.

If all goes well, you should be able to view the pods for your worker pool using kubectl get pods:

kubectl get pods
NAME                                              READY   STATUS    RESTARTS   AGE
worker-pool-1-spacelift-worker-7fcfc9f594-f94tj   2/2     Running   0          22m

You should also be able to view the workers in your pool in Spacelift:

When implementing in a production environment, you may want to use an alternative approach to providing credentials, and may want to configure a custom storage volume for the worker Pods. For more information about configuring the chart, check out the README in the chart GitHub repo.

Closing Thoughts

If you’re interested in running Spacelift workers in Kubernetes, we’d welcome any feedback about the approach, contributions to the Helm chart, and also any issues you encounter so that we can make improvements. Also, if you aren’t already using Spacelift but are interested in trying it, you can signup and start your free evaluation of Spacelift here.

(The original post was published at Spacelift)

Managing Azure Resources using Spacelift

Adam Connelly — Thu, 18 Nov 2021 23:37:00 +0000

In this post I want to show how easy it is to create a Spacelift Stack that manages Azure resources. To keep things simple, I’m going to assume that you already know what Spacelift can do, and the basics of creating and running Stacks, but if not check out our terraform-starter repo that walks you through the process of setting up an account, and getting started.

Overview

In order to allow Spacelift to manage Azure resources, we need to do the following:

Create a Service Principal in Azure with the correct ARM permissions to manage our resources.
Create a Stack in Spacelift that uses the Azure provider.
Give our Stack the credentials for our Service Principal.

All of the information you need to configure the Azure provider can be found here. This post just explains how to combine that information with Spacelift.

Creating our Service Principal

To create our Service Principal, we need to use the az ad sp create-for-rbac command (replacing with your actual subscription ID):

az ad sp create-for-rbac --name spacelift-sp --role="Contributor" --scopes="/subscriptions/<subscription-id>"

This will output something like the following:

Changing "spacelift-sp" to a valid URI of "http://spacelift-sp", which is the required format used for service principal names
Creating 'Contributor' role assignment under scope '/subscriptions/<subscription-id>'
The output includes credentials that you must protect. Be sure that you do not include these credentials in your code or check the credentials into your source control. For more information, see https://aka.ms/azadsp-cli
{
  "appId": "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa",
  "displayName": "spacelift-sp",
  "name": "http://spacelift-sp",
  "password": "A4xFo3z-Hv9BMOFlmHfXP2ESXugHMdaike",
  "tenant": "bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb"
}

The command creates a new Service Principal called spacelift-sp , and grants it the Contributor role on your subscription. It also outputs the appId, password and tenant for the Service Principal. Make a note of these because you’ll need them later.

If you would rather assign permissions yourself later, you can run the following command to just create a Service Principal with no permissions:

az ad sp create-for-rbac --name spacelift-sp --skip-assignment

You should now be able to view your Service Principal in the Azure Active Directory -> App registrations section of the Azure Portal:

Creating a Stack

Now that we’ve got a service principal, we need to create a Stack to actually create some Azure resources. To keep things simple all we’re going to do is add some terraform definitions that specify the version of the Azure Provider we want to use, and create an Azure Resource Group:

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "=2.46.0"
    }
  }
}

# Configure the Microsoft Azure Provider
provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "test-group" {
  name = "test-group"
  location = "West Europe"
}

If you commit that change and try to run your Stack just now, you should see something similar to the following failure:

Here’s the full error message:

Error building AzureRM Client: obtain subscription() from Azure CLI: Error parsing json result from the Azure CLI: Error launching Azure CLI: exec: “az”: executable file not found in $PATH

At first glance this error is slightly confusing, but what it means is that because we didn’t provide any credentials to the Azure provider it attempted to fallback to using the az CLI for authentication. This works fine locally, but won’t work on Spacelift.

To solve this, we need to provide our stack with our credentials, which is explained in the next section.

Passing Credentials to the Stack

If you take a look at the Terraform documentation, you’ll find that there are a number of different ways that you can authenticate Terraform with Azure. What we’re going to do here is authenticate using a Service Principal with a Client Secret, and provide that configuration via environment variables.

To do this we need to set the following environment variables, using the information returned by the Azure CLI when we created our Service Principal:

*ARM_CLIENT_ID — the appId from the response.
*ARM_CLIENT_SECRET — the password from the response.
*ARM_SUBSCRIPTION_ID — your subscription ID.
*ARM_TENANT_ID — the tenant from the response.

You can set these environment variables directly on your Stack, but what we’re going to do here is create a Spacelift Context and attach it to our Stack. This allows us to share the same credentials with multiple Stacks if we want.

Add a new Context via the Contexts screen in Spacelift:

Once your context has been created, edit it and enter your environment variables, making sure to mark the ARM_CLIENT_SECRET as a secret rather than plaintext:

Now that we’ve got a Context containing our credentials, you can attach it to your Stack. To do this, click on the Manage link next to your Stack, go to Contexts, choose your Context, and click Attach:

Now that your Context is attached, you should be able to see the it in the Environment section of your Stack:

If you try to run your Azure Stack again you should get a successful plan:

And applying should now succeed:

If you take a look in the Azure Portal, you should now be able to see the Resource Group that was created:

Credential Expiry

Before finishing up, I wanted to just point out a few things about the Service Principal credential expiry. Although this isn’t really related to Spacelift, it’s something that you need to be aware of.

When you create your Service Principal via the Azure CLI, it automatically creates a Client Secret for you with an expiry of 1 year:

What this means is that if you do absolutely nothing, your Spacelift Stacks will stop working after the credentials expire a year later. To solve this, all you need to do is generate a new Client Secret, and update the ARM_CLIENT_SECRET environment variable in your Context.

Wrapping Up

Hopefully in this post I’ve shown that it’s simple to manage Azure resources using Spacelift. Once you’ve performed a little bit of initial setup, you should be able to rapidly get new Stacks up and running creating Azure resources.

Building a Secure CI/CD Integration with Azure

Adam Connelly — Thu, 21 Oct 2021 12:48:19 +0000

Securing your infrastructure is critical for the success of your business. Failure to take security seriously can result in major damage including fines, loss of customer confidence, or the inability to carry out crucial business functions. The growth of Infrastructure as Code tools and CI/CD systems has allowed developers to integrate infrastructure management into our typical development workflows, improving quality and delivery speed.

At the same time, in order to manage your infrastructure, the CI/CD system used needs access to sensitive credentials. At Spacelift, we aim to give our users the maximum balance between flexibility and security. Because of this, we provide multiple options for connecting your Azure subscriptions to Azure, including setting static credentials, using our fully managed integration, as well as utilizing private workers to avoid sharing credentials at all.

In this post I wanted to give you an overview of how the Spacelift Azure integration works from a technical perspective, as well as discuss some of the issues we encountered and solved while designing and developing it. As a CI/CD system, we do quite a lot of work to integrate with other systems that our users use. Occasionally, like in the case of Azure, things get non-trivial. If you’re keen to learn more, read on, no Azure knowledge required!

Concept

Let’s start with a quick description of how our cloud integrations work. The overall workflow is very simple, and looks something like this:

Breaking the requirements down, we need to be able to get management credentials for a user’s cloud provider account and pass them to Terraform via environment variables, at which point Terraform can run an apply with access to the customer’s infrastructure.

Note: For simplicity this post uses Terraform in all its examples, but this overall approach also applies to the other tools that we support, for example, Pulumi.

The concept behind the Azure integration was to provide a similar experience to our AWS and GCP integrations, but for our Azure customers. The following diagram shows a simplified outline of how the AWS integration works:

AWS provides the ability to temporarily assume a role in another AWS account. This allows our users to create a role in IAM with any permissions they might want to give Spacelift. They can then set up a trust relationship for this role with our AWS account, which will allow our AWS account to assume the role. Role assumption provides us with raw AWS credentials and works seamlessly with any AWS tooling, including the Terraform AWS provider. It additionally allows us to specify the validity duration, so each run can get its own credentials which are constrained to a short period of time.

Although Azure doesn’t have the same capability, it provides another approach called Azure Active Directory Applications, that allow service accounts to be created. Azure AD Applications are the resources that allow Spacelift to seamlessly manage access to a customer’s Azure resources.

Azure Terminology

It’s worth explaining a few pieces of terminology that are used throughout the rest of this post:

Azure Active Directory (Azure AD) – the identity and access management component of Azure.
Directory / tenant – an individual instance of Azure AD owned by a company or individual.
Subscription – the container for any Azure compute resources. This roughly corresponds to an AWS Account. A subscription is linked to a single Azure AD tenant, but multiple subscriptions can be linked to the same tenant.
Azure AD Application – a way of creating external integrations with Azure AD.
Enterprise Application – an instance of an Azure AD application that has been installed in another user’s Azure AD tenant.
Service Principal – a service account that is automatically created when an Azure AD application is installed. This can be used to grant permissions that allow the application to manage Azure resources.
Microsoft Graph API – the main API for managing Azure AD resources.

Goals

We set a number of goals for the integration:

Making it really easy for customers to manage Azure infrastructure using Spacelift.
Automatic handling of credential rotation so that customers don’t have to deal with this themselves, or use very long-lived credentials to avoid it entirely.
Providing a mechanism for customers to configure granular permissions in Azure for different stacks, or different types of runs (e.g. PRs vs Tracked Runs).

Initial Approach

Initially, our idea was to create a single multi-tenant AD Application:

The idea was that we would generate an Access Token that could only be used for a specific customer directory, and pass that token to the Terraform Azure RM provider during runs. In the end, we had to revise our approach because of the following issues:

The Terraform Azure RM provider doesn’t support authentication via an Access Token. Instead, you have to supply the underlying credentials for the account – either a Client Secret or a Client Certificate. In our case, that would have meant passing the credentials for our own multi-tenant application to Spacelift runs. Since that application would have been installed in the Azure AD tenants of any Spacelift user who had setup the integration, this could have allowed users to access other user’s Azure accounts.
The integration would have been less flexible. Using a single multi-tenant AD application would have prevented customers from creating more than one Azure integration per Active Directory tenant. The ability to create multiple integrations per tenant is useful because it allows different Azure permissions to be applied to each integration.

Revised Approach

After days of brainstorming on an alternative approach, we came up with a new architecture. We could programmatically generate a new Azure AD Application on our side for each Azure Integration created by Spacelift users. This way, having access to the credentials for an Azure AD Application would only lead to having access to a single Azure AD Tenant on a user’s side. This approach allows Client Secrets to be passed to Spacelift runs without fear of inter-user permission leakage. The final design ended up as shown in the following diagram:

Applications are installed into a customer’s Active Directory tenant via a process called Admin Consent. After admin consent has been completed, a Service Principal is created in the user’s Azure Active Directory to which the user can grant permissions. This allows users to decide the exact level of access that Spacelift has to their resources.

Credential Generation

The next issue we faced was related to generating credentials for a run. As described in the provider documentation, the Azure RM provider can be configured by setting certain environment variables. Initially, we took a basic approach of attempting to generate credentials during a Spacelift run. This is what we do for our AWS and GCP integrations, so we weren’t expecting major issues. The steps taken looked something like this:

Run triggered.
Generate a new Client Secret with a short expiry time.
Populate the required environment variables.
Execute terraform.

This seemed to work… but only some of the time.

While testing the integration, strange things were happening. As an example, the planning phase for a run would succeed, but the apply would fail with a permissions error from Azure. After investigating, we came to the conclusion that this was being caused by eventual consistency in Azure AD.

You can visualize the problem using the following diagram (note: this is just an illustration, and is not meant to be completely accurate):

In the example above, step 2 may succeed or fail depending on whether the secret has managed to replicate to the Azure AD server that its request is routed to. Initially, we attempted to test whether or not the secret was usable by making an API request, and retrying until the request succeeded using an exponential backoff. What we soon realized was that even then, subsequent requests could be routed to a different Azure AD instance, which still hasn’t received the new Client Secret, and potentially fail.

Even if it was possible to verify when the secret was fully replicated, waiting for replication to complete would have added a minimum of 30 seconds, and potentially another several minutes. Because of this, we decided to move credential generation and rotation out of the run flow, and into a scheduled task:

The scheduled task runs once per hour, generates secrets with an expiry of 24 hours, and attempts to generate a new secret for an integration roughly 2 hours before expiry of the old secret. This allows credential rotation while always keeping a valid secret.

When a new secret is generated, we use AWS’s Key Management Service to encrypt it so that it is never stored in plaintext.

When a run is triggered, we try to find the secret for the integration with the most amount of time until expiry. We also avoid using new secrets until roughly 10 minutes after generation to avoid the eventual consistency issues caused by Azure AD’s architecture.

You can visualize the secret lifecycle using the following diagram:

In addition, when creating a new integration we immediately generate a secret. This helps to ensure that the secret will have successfully propagated within Azure AD by the time a run is triggered.

Credential Rotation for the Integration

The last major issue we faced was figuring out how to implement credential rotation for our own management account. The integration itself uses an Azure AD Service Principal to manage customer AD Applications using the Microsoft Graph API. Because we run most of our own infrastructure in AWS, we didn’t have the option of using a Managed Service Identity, meaning that we needed to handle credential rotation ourselves. In addition, our goal was to automate the process to avoid developers having to periodically perform a manual task, and to reduce the risk of forgetting to renew the credentials before expiry.

In the end, we decided to take the relatively simple approach of storing the certificate in Secrets Manager and writing a scheduled task to periodically check whether the certificate was ready to expire, similar to the approach we took for Client Secrets for the integration. If so the scheduled task generates a new certificate and uploads it to both Secrets Manager and Azure AD:

The parts of the system that need to use the client certificate for authentication periodically check for an updated certificate. As with the client secret rotation, we avoid using the new certificate for approximately 10 minutes to allow time for the certificate to propagate throughout Azure AD.

Similar to what happens with the integration client secrets, Secrets Manager uses AWS Key Management Service to encrypt the certificate at rest.

Wrapping Up

Hopefully, this post has given you a glimpse into the internals of Spacelift’s Azure integration, along with some of the problems we had to solve while implementing it. As you probably noticed, we’re willing to go to great lengths to ensure a secure and pleasant experience for our users. To find out more, take a look at our Azure integration documentation available at Spacelift Documentation.

How does Open Policy Agent (OPA) work?

Adam Connelly — Tue, 19 Oct 2021 16:32:12 +0000

In this article, I want to give an overview of Open Policy Agent, why you would want to use it, as well as showcasing how you can use OPA with your Spacelift account. Although OPA can be used for many purposes, I’m going to focus on how it can be used alongside Infrastructure as Code.

What is OPA?

Open Policy Agent provides a way of declaratively writing policies as code and then using those policies as part of a decision-making process. It uses a policy language called Rego, allowing you to write policies for various different services using the same language.

OPA can be used for a number of purposes, including:

Authorization of REST API endpoints.
Allowing or denying Terraform changes based on compliance or safety rules.
Integrating custom authorization logic into applications.
Implementing Kubernetes Admission Controllers to validate API requests.

OPA was originally created by Styra, and is now a part of the Cloud Native Computing Foundation (CNCF), alongside other CNCF technologies like Kubernetes and Prometheus.

How does OPA work?

We can visualise how OPA works using the following diagram:

As you can see, OPA accepts a policy, input and query, and generates a response based on that. The input can be any valid JSON document, allowing OPA to integrate with any tool that produces JSON output.

Why would I want to use it?

In the following sections, I’ll go into more specific examples of using OPA that should help make things clearer, but before I do here’s a quick list of reasons that I’m interested in using OPA:

Policies as code allow you to follow your standard development lifecycle with PRs, CI, etc, and provide you with a history of changes to your policies.
OPA is designed to work with any kind of JSON input, meaning it can easily integrate with any tool that produces JSON output.
Because OPA integrates with a number of different tools, it allows you to use a standard policy language across many parts of your system, rather than relying on multiple vendor-specific technologies.
OPA supports unit-testing, making it easier and faster to iterate your policies with confidence that they won’t break.

Using OPA with Terraform

Let’s try to make that a bit less theoretical by using a specific example: Terraform. Terraform can produce a plan in JSON format via the terraform show command. This means that we can define policies for our infrastructure, and use OPA to make a decision about whether a plan is safe to apply or not:

For example, say we have the following terraform definition to create an EC2 instance:

provider "aws" {
 region = "eu-central-1"
}

resource "aws_instance" "web" {
 ami           = "ami-00003c1d"
 instance_type = "t3.micro"
}

Now say we want to ensure that every Terraform resource has a Name tag, we could enforce that by creating a file called plan.rego with the following content:

package spacelift

allow {
   resource_change := input.resource_changes[_]

   resource_change.change.after.tags["Name"]
}

In order to use OPA to evaluate our policy, we need to take the following steps:

Generate a Terraform plan as JSON.
Run opa eval to verify whether that plan passes our policy or not.

Step 1 – Generate our Terraform plan as JSON

To get a JSON representation of our plan, we need to output our plan to a file, and then use the terraform show command to output that plan as JSON:

terraform plan -out spacelift.plan
terraform show -json spacelift.plan > spacelift.json

Step 2 – Run opa eval

We can then use opa eval to evaluate our plan against our policy:

$ opa eval --data plan.rego --input spacelift.json "data.spacelift.allow"
{}

As you can see, we’re using the query data.spacelift.allow because our policy is loaded as a data file, and we defined our allow rule in the spacelift namespace. You can also see that opa eval produced empty output ({}). This means that our allow rule didn’t evaluate to true, and so produced no output.

Let’s adjust our terraform definition to include a Name tag:

resource "aws_instance" "web" {
 ami           = "ami-00003c1d"
 instance_type = "t3.micro"

 tags = {
   Name = "my-instance"
 }
}

Now if we generate our plan and evaluate the policy again, we should get a slightly different output:

$ terraform plan -out spacelift.plan && \
 terraform show -json spacelift.plan > spacelift.json
... lots of Terraform output

$ opa eval --data plan.rego --input spacelift.json "data.spacelift.allow"
{
 "result": [
   {
     "expressions": [
       {
         "value": true,
         "text": "data.spacelift.allow",
         "location": {
           "row": 1,
           "col": 1
         }
       }
     ]
   }
 ]
}

This tells us that the allow rule has evaluated true. We can get that in a slightly more concise way using the pretty format option:

$ opa eval --data plan.rego --input spacelift.json --format pretty "data.spacelift.allow"
true

At this stage, you can probably imagine how you could integrate this into your CI/CD pipeline to enforce naming schemes, security rules, and various other organizational policies.

Other Examples

The following examples illustrate some possible use-cases for OPA with Terraform. All of the examples have been taken from the Spacelift plan policy documentation, but have been altered to work with plain vanilla Terraform.

Example 1. Require human review when resources are deleted or updated

Adding new resources is usually a fairly safe operation, but updating or deleting existing resources can carry more risk. You could flag these for human review using the following policy:

package spacelift

warn[sprintf(message, [action, resource.address])] {
  message  := "action '%s' requires human review (%s)"
  review   := {"update", "delete"}
  resource := input.resource_changes[_]
  action   := resource.change.actions[_]
  review[action]
}

Example 2. Require commits to be reasonably sized

Once a PR goes over a certain size it becomes difficult to review without missing things. The same is true for Terraform plans. The following policy can be used to warn when the number of changes goes over a certain threshold:

package spacelift

warn[msg] {
   msg := too_many_changes[_]
}

too_many_changes[msg] {
   threshold := 50
   res := input.resource_changes
   ret := count([r | r := res[_]; r.change.actions != ["no-op"]])
   msg := sprintf("more than %d changes (%d)", [threshold, ret])
   ret > threshold
}

Example 3. Blast radius

The following policy attempts to determine the risk of a particular plan by assigning different weightings to the change type (create, update, delete), along with the affected resource type (ECS cluster, EC2 instance, etc). It takes the approach that an update or delete is more risky than a create because it affects an existing resource:

package spacelift

warn[msg] { msg := blast_radius_too_high[_] }

blast_radius_too_high[sprintf("change blast radius too high (%d/100)", [blast_radius])] {
   blast_radius := sum([blast |
                        resource := input.resource_changes[_];
                        blast := blast_radius_for_resource(resource)])

   blast_radius > 100   
}

blast_radius_for_resource(resource) = ret {
   blasts_radii_by_action := { "delete": 10, "update": 5, "create": 1, "no-op": 0 }

   ret := sum([value | action := resource.change.actions[_]
                   action_impact := blasts_radii_by_action[action]
                   type_impact := blast_radius_for_type(resource.type)
                   value := action_impact * type_impact])
}

# Let's give some types of resources special blast multipliers.
blasts_radii_by_type := { "aws_ecs_cluster": 20, "aws_ecs_user": 10, "aws_ecs_role": 5 }

# By default, blast radius has a value of 1.
blast_radius_for_type(type) = 1 {
   not blasts_radii_by_type[type]
}

blast_radius_for_type(type) = ret {
   blasts_radii_by_type[type] = ret
}

Unit Testing

How can we make sure that our policies work as we expect and that they don’t break over time as we make changes to them? You guessed it: unit testing! Luckily for us, OPA has first-class support for testing via the opa test command.

To create tests for our policy, all we need to do is create another Rego file with a series of rules prefixed with test_. Each rule starting with that prefix defines a separate test.

Let’s go ahead and create a file called plan_test.rego, with the following contents:

package spacelift

test_allow_missing_name_tag {
 not allow with input as {
     "resource_changes": [
       {
         "change": {
           "after": {
             "tags": null,
           },
         }
       }
     ]
   }
}

As you can see, OPA makes it really easy to specify the policy input using the <variable> as <value> syntax. This allows us to create very concise tests by only including values we care about in the policy input, rather than having to use the entire plan output.

Let’s go ahead and run opa test:

$ opa test .
data.spacelift.test_allow_missing_name_tag: PASS (188.503µs)
--------------------------------------------------------------------------------
PASS: 1/1




   resource_change.change.after.tags["Name"]

   resource_change.change.after.tags["Environment"]

}

Unsurprisingly our test passes. That’s not very exciting, so let’s add a new test to ensure our resources include an Environment tag:

test_allow_missing_environment_tag {
 not allow with input as {
     "resource_changes": [
       {
         "change": {
           "after": {
             "tags": { "Name": "my-instance" },
           },
         }
       }
     ]
   }
}

Running opa test again shows us a failure:

$ opa test .
data.spacelift.test_allow_missing_environment_tag: FAIL (118.078µs)
--------------------------------------------------------------------------------
PASS: 1/2
FAIL: 1/2

Which we can fix by updating our policy:

allow {
   resource_change := input.resource_changes[_]
   resource_change.change.after.tags["Name"]
   resource_change.change.after.tags["Environment"]
}

At this stage if you run the test command again it should show 2 passes:

$ opa test .
PASS: 2/2

If you want to know more about OPA testing, the official docs are full of great examples and information about what you can do.

OPA + Spacelift

At this stage, hopefully, you’ve got a pretty good idea of what OPA is as well as how it can be useful to you. It’s not too difficult to see how you could start integrating OPA into your development process or even use it as part of production systems.

Luckily for you, at Spacelift all heavy lifting is done for you, allowing you to get the benefits of using OPA for Policy-as-Code without having to implement everything from scratch for yourself. In this section, I want to showcase some of the functionality that Spacelift provides that relates to OPA.

1) Policy Types

Spacelift allows you to use OPA policies to manage various aspects of your Spacelift account, not just during planning. For example, you can use policies to control who can login to your account, along with what they have access to. For more information see https://docs.spacelift.io/concepts/policy.

2) PR Checks

Spacelift can automatically trigger planning runs whenever you push changes to your VCS provider. For example, here’s what you might see in GitHub after creating a PR:

If all goes well your check will succeed, but if a policy is violated, a failed check will be reported:

You can then view the details of the failure in Spacelift:

3) Manual Approvals

Spacelift plan policies use a slightly different format than we used in our example policies earlier in this post. Not only do they allow you to specify a message to be displayed, but they also have the concept of deny and warn:

package spacelift

deny["you shall not pass"] {
 true
}

warn["hey, you look suspicious"] {
 true
}

The deny rule fails the run completely, while the warn rule just displays a warning in the logs.

warn rules take on another role when a run is going to deploy changes (vs just showing the planned changes against a PR). If any warnings are reported during a deployment, the run will wait for manual approval before applying any changes.

Let’s use the following example policy, designed to warn if a certain set of suggested tags aren’t found on our resources:

suggested_tags := { "Name", "Environment" }

warn[sprintf("resource %q does not have all suggested tags (%s)", [resource.address, concat(", ", missing_tags)])] {
 resource := input.terraform.resource_changes[_]
 tags := resource.change.after.tags

 missing_tags := { tag | suggested_tags[tag]; not tags[tag] }
 count(missing_tags) > 0
}

If we then attempt to add resources that don’t contain all of those tags, Spacelift will block before applying the changes, and give us the chance to manually approve or deny the run:

This allows you to build complex workflows where certain changes are completely blocked, but others are allowed as long as the changes are reviewed first.

Check out the plan policy cookbook for more ideas about what you can do.

Spacelift Terraform Provider

Spacelift provides a Terraform provider for managing your Spacelift account. This means that you can manage the policies available within your account, along with the Stacks they apply to in code.

For example, to add the policy we defined earlier to Spacelift we can use the spacelift_policy resource like this:

resource "spacelift_policy" "plan" {
 type = "PLAN"

 name = "Plan Policy"
 body = file("${path.module}/plan.rego")
}

We can then attach this policy to a Spacelift Stack using the spacelift_policy_attachment resource:

resource "spacelift_policy_attachment" "mystack-plan" {
 policy_id = spacelift_policy.plan.id
 stack_id  = spacelift_stack.mystack.id
}

Taking this a step further, we could create a custom module for defining Spacelift Stacks that ensured that all Stacks had a certain set of policies attached by default.

What's Next?

I hope you’ve enjoyed this post, and can see the value that OPA brings to the table. If you’re interested in trying out Spacelift to see what it has to offer, why not sign up for a free trial? You can setup a Spacelift account in minutes, and get started on your Open Policy Agent journey!

How to Manage Active Directory Objects with Azure AD Provider for Terraform

Adam Connelly — Mon, 18 Oct 2021 17:16:58 +0000

The Azure AD provider for Terraform can be used to manage your Azure Active Directory resources declaratively. This allows you to do things like:

Automatically provision users and make sure they belong to the correct groups.
Manage Azure compute permissions via Azure AD groups.

Example Usage

The following example shows how to use the Azure AD provider to create a group in Azure AD:

terraform {
 required_providers {
   azuread = {
     source  = "hashicorp/azuread"
     version = "= 1.6.0"
   }
 }
}

resource "azuread_group" "test" {
 display_name = "Test Group"
}

The terraform section at the start is used to specify the version of the provider that we want to use, and the resource azuread_group test block defines our group.

Authenticating with Azure

The Azure AD provider allows multiple authentication methods, which are outlined in the provider's documentation. To allow you to get up and running quickly, the AD provider will attempt to get your credentials via the Azure CLI.

While this is fine for experimentation and local testing, for non-interactive scenarios like CI you need to use a Service Principal or a Managed Service Identity.

Permissions

In order to manage your Azure AD objects, the account used by Terraform needs to have the correct permissions to perform its actions. You can manage these permissions via the Roles and administrators section of Azure AD:

For example, to allow a Service Principal to manage groups, you would add it to the Groups administrator role:

API Permissions

Another option that can be used with Service Principals instead of granting an administrator role is to grant specific API permissions to them. To do this, first find the AD Application linked to your Service Principal in the App Registrations section:

Go to the API permissions page for the application, and click on Add a permission:

On the screen that appears, choose the Azure Active Directory Graph API, and then choose the relevant permission you want to add:

Before the Service Principal can actually use the permission you just added, you need to take a final step called granting Admin Consent. You can do this by clicking on the Grant admin consent for button displayed above the permissions table:

NOTES:

When adding permissions to your Service Principal, you need to add Application permissions rather than Delegated permissions. This means that the Service Principal is allowed to perform the specified actions as itself, rather than on behalf of another user.
The set of permissions that you can add via API permissions is quite limited. For example, to create AD groups you need to add the Directory.ReadWrite.All permission, but this will not allow your Service Principal to delete any groups it creates. In order to be able to delete groups, you need to grant it the Group Administrator role, so depending on your requirements there may not be any point in granting API permissions.
The Azure AD Terraform provider is switching to the Microsoft Graph API as of version 2.0.0, so after version 2 is released you will need to grant permissions to the Microsoft Graph API instead of to the Azure Active Directory Graph API.

More Examples

Managing Users and Groups

The following example creates two users and two groups, and assigns each user to a group:

resource "azuread_user" "adamc" {
 user_principal_name   = "adamc@adamrpconnellygmail.onmicrosoft.com"
 display_name          = "Adam Connelly"
 password              = "SuperSecret01@!"
 force_password_change = true
}

resource "azuread_user" "bobd" {
 user_principal_name   = "bobd@adamrpconnellygmail.onmicrosoft.com"
 display_name          = "Bob Dolton"
 password              = "SuperSecret01@!"
 force_password_change = true
}

resource "azuread_group" "development" {
 display_name = "Development"
 members = [
   azuread_user.adamc.id
 ]
}

resource "azuread_group" "sales" {
 display_name = "Sales"
 members = [
   azuread_user.bobd.id
 ]
}

Creating a Service Principal and granting RBAC permissions

The following example combines the Azure AD provider
with the Azure RM provider, allowing you to create a Service Principal and assign it permission to manage certain Azure resources:

# Create an AD Application
resource "azuread_application" "automation" {
 display_name = "sp-automation"
}

# Create a Service Principal from that Application
resource "azuread_service_principal" "automation" {
 application_id               = azuread_application.automation.application_id
 app_role_assignment_required = false
}

# Get information about the configured Azure subscription
data "azurerm_subscription" "primary" {}

# Grant our service principal "Contributor" access over the subscription
resource "azurerm_role_assignment" "automation_contributor" {
 scope                = data.azurerm_subscription.primary.id
 role_definition_name = "Contributor"
 principal_id         = azuread_service_principal.automation.object_id
}

Conclusion

In this post I’ve covered what the Azure AD Terraform provider is used for, how to authenticate and grant the correct permissions, as well as showing a few examples of what you can do with it. Hopefully you’ve found it useful!

If you’re interested in finding out about how you can use Spacelift to manage your Azure resources, check out Spacelift Azure documentation. Also, don’t forget that you can easily give Spacelift a free test drive!

You will find more Terraform Tutorials on our website: