Forem: Christian Lechner

Exploring Terramate Cloud part 2 – A walkthrough

Christian Lechner — Wed, 30 Jul 2025 05:39:41 +0000

Introduction

In the first part of this two-part blog post we covered the basic setup to use Terramate Cloud. In this second part we take a closer look at the Terramate Cloud offering per se.

Let us jump right in.

Terrmate Cloud

Terramate Cloud is a SaaS offering that complements the Terramate CLI with features like monitoring of your stacks.

As we already onboarded three stacks we explore the different sections that are offered by Terramate Cloud and test the setup for configuration drifts and changes in the deployments.

Your "Homework"

The first thing you see once logged into Terramate Cloud is the “Your ‘Homework’” section that gives you a crisp overview about your setup and its state:

Usually when exploring new software, you first must get used to the wording and the semantics that the solution introduces. That should be covered in the documentation, but that also means you must switch between tabs/windows etc.

But do you see that button in the right upper corner labeled “Explain this page”? Pressing this button adds extensive in-place help and explanations to the page and its sections:

That’s cool. I really like this as it helps new joiners right in the place where they need it. Unfortunately, this is the only page where this in-place help is available.

While I do not think that this is a fitting approach for every page or detail section in Terramate Cloud, I would e.g. like to also have that in the Dashboard page. Overall a nice start.

Let’s move on to the “Dashboard” section.

Dashboard

As the name implies this page gives an overview about the metrics of your setup. Besides some KPIs about the stacks the main thing is the display of DORA metrics.

According to the pricing overview this is not included in the free edition, but I see them and I am not sad about that:

Due to the limited setup in my experiments the numbers are not impressive (though I am elite in some areas 🤪).

I think the DORA metrics part is a very valuable addition to the offering. While developing and providing business apps clearly show the impact on business, the management of infrastructure is often not seen by e.g., C-level as an essential building block for this although it is. Having these metrics in place out-of-the-box will help the teams when arguing about the importance of a decent IaC setup and of course also help the team improve. This will also have a positive impact.

The only thing that was a bit irritating about the metrics was that the preconfigured time interval points to the previous month. I would have expected that it points to the current month. Maybe looking at a “completed” month is more reasonable for overall reporting, I would always like to see the current state.

Anyway, this feature is imho a highlight of Terramate Cloud.

Resources

The next section covers the deployed “Resources”. As the name states this section provides an overview of all resources and their state:

There are tons of sorting options, so tailoring the view to your needs is possible.

However, there are some small points on the overview that came to my attention (keeping in mind that I am using a “exotic” setup on SAP BTP):

The column “Account” is empty. I have no idea what this refers to. As there is no entry for any resource, it could also be hidden automatically by the platform. But this is not really an issue.
The column “Name” seems to be fetched from the resource if it has a field called “name”. If there is no such field, it remains empty. Although in my case, this is most of the time empty, it is useful for me.
The column provider triggered my inner “Monk” a bit: the first letter is capitalized, and the organization of the provider is missing. I don't know how this is derived, but I would like to see “SAP/btp” there as defined in the provider.tf file or in the Hashicorp Terraform/OpenTofu registry

From the overview, you can also dive into the details of each resource:

The details also contain the state, and I assume that sensitive data might be displayed, but I did not test that. The detailed view also provides a cross-navigation to the Terramate stack the resource belongs to.

I think this is a useful overview/detail page combination enabling you to get a quick access to the basic information of the resources that are deployed.

As we have a cross-navigation to stacks, let’s take a closer look at that section.

Stacks

The “Stacks” section provides an overview of all stacks available to Terramate Cloud:

Each line of the overview contains some basic stack information. From there you can navigate to the details screen for each stack:

Here we find more stack-related information like deployments or drift runs or policy checks (that are not available for the free edition).

From this view we can navigate to the details per resource which closes the loop to the “Resources” section.

As an intermediate summary until: nice experience, reasonable structuring of the views and detail views, the displayed information and the cross-navigation.

From the short evaluation I would say that for basic overviews my preferred areas are the Dashboard with the DORA metrics and the stacks.

Now let's evaluate the dynamical aspects of Terramate Cloud and by this investigate the remaining sections.

Handling of Configuration Drift

Let’s introduce a drift. For that I manually removed some labels on one SAP BTP subaccount and triggered the GitHub Actions workflow for drift detection (remember: no auto remediation in place).

Once the drift detection was executed, we can immediately see the effect in the “Dashboard” section namely that one stack is unhealthy due to drift:

Same is true for the “Stacks “ where we also can identify the stack that causes the issue:

This is also a scenario when the “Alerts” section comes into play:

This section informs us about the issue. As you can see in the screenshot there are also some useful predefined filters in place as buttons, I like that.

Drilling into the details of the alarm reveals the details of the drift namely the result of the planning:

This already helps a lot to pin down the issue and trigger the consequent actions if you know Terraform and the semantics of the provider that is used.

From several discussions with customers, I know that this result of the terraform plan can cause some uncertainty and questions to the fact that there are also in-place updates. They are all on computed fields and therefore not an issue, but they are often the source of confusion.

And this is where I like to see the new AI explain feature in Terramate Cloud. My opinion: one of the right spots for using AI in contrast to all the fancy marketing promises floating around at the moment.

What is even better: this feature is also available in the free plan. So let's try it out and press the “Explain drift with AI” button. This is what we get:

I like that explanation and I think this is really useful. I understand this cannot be extrapolated to complex setups/drifts.

However, imho if you are facing very complex drifts, I think you have another issue that has nothing to do with IaC: either the platform you are using per se behaves "weird" or (more often the case) your organizational setup and processes are far from where they should be when it comes to IaC and you blindly rely on “IaC will remediate things anyway”.

After fixing the drift manually and redoing the drift run via GitHub Actions, all turned green again in the Terramate Cloud cockpit. This includes the alert that gets the status „fixed“ automagically:

One thing to be aware: no matter if there is a drift or not, the GitHub Action workflow based on the template will show a successful execution. The fair assumption is that you rely on Terramate Cloud and not on monitoring the GitHub Action workflow status when it comes to drift detection, but something to be aware of.

There is one small bug I recognized in that context: if the alert is not assigned to a user and gets fixed you can no longer assign a user to it. If you try to do an assignment you get an error message/pop-up that states „Conflict“:

Small issue, but I think this should be possible even when a issue is fixed.

I did not dive into the depths of organizational workflows on how this alerting needs to be integrated into the overall organizational processes like adding an issue in GitHub or alike and referencing the alert there. I at least did not see a way to reference the alert somewhere else.

Let’s move on and make a change to the configuration via a pull request (PR).

Handling of Configuration Changes

Let us change the security setting of our subaccounts for all stacks. This is a dedicated resource, so we should see three updates of the resource, one for each stack.

I created a PR with the change which triggers the preview workflow in the repository and the PR gets synched to Terramate Cloud. As the repository is connected to Terramate Cloud via the Terramate GitHub app, the PR gets a comment of what will happen when getting applied:

The comment also contains a link to Terramate Cloud namely to the details view of the PR.

First, this connection is great (you do not need to manually navigate to Terramate Cloud) but also that it brings you to the details view. Yes, it is obvious that this is needed, but I know a lot of solutions that would bring you to the overview page and you must navigate on your own.

Let's do it the manual way and open the "Pull Request" section. We see the PR and some basic information:

At a first glance we see how many resources and stacks are affected by the change. Drilling into the details give us more information about the change of the infrastructure per stack:

Some more details including the logs are also available:

You can also display the information in an ASCII view. This might be useful when you think that there is a rendering issue, but other than that I wouldn’t know when to use this ASCII view.

As for drifts explaining the plan is a good use case for AI. This time it is not a button you need to press but a dedicated tab in the details view of the PR:

The explanation is good and contains all relevant information about the change. I like that. As for the drift this was a small change, so this feature will probably really shine in more complex update scenarios.

While I like the feature per se, I think it would make sense to integrate the AI companion or copilot or however you want to call it, consistently in the different screens.

As we are happy with the change let’s merge the PR. This results in a new deployment as we can see in the corresponding section "Deployments" in Terramate Cloud:

As in the other screens we can directly navigate to the details:

Drilling down further is possible, so that we can analyze the deployment details per stack:

Here we have the AI feature again, this time as a button. I am also not sure if it is really needed here as this change already happened and I do not see value in getting an explanation after a successful deployment. If something goes sideways, this might make more sense, but then we see an alert.

Overall, the experience in Terramate Cloud for deploying changes is also quite nice.

Wrap up

I already liked Terramate CLI and its concepts so what is my summary about Terramate Cloud?

Terramate Cloud complements that with several nice features supporting the operations namely the monitoring and troubleshooting part of IaC. The onboarding is smooth, and the experience is very good. The interplay with established GitHub flows works as expected. Function-wise the DORA dashboard as well as the alerting and PR handling look really good.

From the UI perspective I like the clean style and having the right information in the right places. Although I did not work with the tool for weeks, I enjoyed the connection between the views: depending on your way of work or the preferred view to start from I would make the bold statement that you always see the right information or can drill down/navigate to it.

Performance-wise the UI is fast as one would expect but does not always get in such solutions.

The recently added AI features are at the spots where I would expect them and where I see added value. Not overhyped, agentic, MCP, fill in the blanks BS, but fitting into the flow and nicely integrated into the UI (except for some small inconsistencies).

There are some things that I missed during my walk through:

Storing settings and filters for views: as everybody has his own way of working and preferred setups of overviews it might be useful to be able to store some personal settings in that place. I don’t know if that’s just me being already SAP-ified to a large degree or if other see this as a useful feature, too.
Most commonly used CI/CD pipelines are covered that I also know from customers. However, I am missing more content for Azure DevOps that I also see being used at a lot of customers. Maybe there was not yet demand for that from Terramate customers.
After finishing my experiments, I wanted to clean up the setup on Terramate Cloud. Basically, I was looking for a “Delete/Offboard stacks” button or a CLI command like terramate rm stack xyz -- sync-deployment. I did not find something like that, so I asked on the Terramate Discord. I want to stress again, I used the free version, I am not a paying customer. But I got the answer in a few hours - highly appreciated and something I experience with every question to the Terramate team. I was right, this functionality does not exist, only archiving is possible (or asking the Terramate team to manually delete stuff). I think this would be a handy feature especially when doing proof-of-concepts on the platform.
One more conceptual question came to my mind about best practices on how to structure Terramate Cloud organizations in combination with GitHub organizations and repositories to enable a smooth integration. I am sure there is not one perfect setup, but some guidance about the pros and cons on different ways/common scenarios would be appreciated. The “easiest” way is of course to have everything under one GitHub org, but I think then Terramate Cloud overviews become quite crowded and there are maybe other drawbacks. It would be really interesting to learn what the experiences and good practices of the Terramate team are in that area.

I think all those points are already nitpicking to some degree. Terramate Cloud is as it is already a very useful tool and complements the Terramate CLI at the right areas.

I did not wrap my head about integrating Terramate Cloud in your existing workflows, like i shortly mentioned when talking about drifts and alerts. Something that would need further investigation that I did not do, but would certainly be aprt of the evaluation when you are doing a proof-of-concept and evaluate Terramate Cloud for your organization

Long story short: If you are looking for a solution in the IaC area for making life easier in operations and monitoring Terramate Cloud is worth a closer look.

Exploring Terramate Cloud part 1 – Getting things set up

Christian Lechner — Wed, 30 Jul 2025 05:39:39 +0000

Introduction

Exploring Terramate Cloud was on my to do list for a while now. The latest updates of Terramate Cloud about DORA metrics and integration of some AI explain features reminded me of that. A good time to get this thing done.

This is a two parted blog post. The first part you are currently reading is about getting the basic setup for using Terramate Cloud in place. The second part is about how Terramate Cloud itself and what it brings to the table.

Context

I have already written some blog posts about the Terramate CLI and how it can support you with challenges in your day-to-day work with Infrastructure as Code (IaC), in my case Terraform/OpenTofu.

The CLI is open source and free of charge. In my opinion it is a very useful tool with some smart concepts and can help you with some challenges around setup of infrastructure especially on SAP BTP. If you are interested in those blog posts, here they are:

But there is also a SaaS offering of Terramate, namely Terramate Cloud. This offering focuses on the management and observability part of IaC.

There is a free plan for Terramate Cloud which reduces the barrier of trying it out to zero.

Prerequisites

The basic prerequisite is to have a IaC setup in place that consists of Terramate stacks. I created such a setup and you can find it on GitHub: https://github.com/btp-automation-scenarios/btp-terramate-cloud.

It comprises three stacks (dev, test and prod) with a basic setup on SAP BTP. I used the Terramate code generation feature, but that is not needed to use Terramate Cloud. The only prerequisite is to have Terramate stacks in place.

I deployed the stacks from my local machine via the Terramate CLI. My remote state storage is an Azure Blob Storage, but that has no impact on Terramate Cloud.

On the Terramate Cloud side we first need to sign up to Terramate Cloud and create an organization that is used to connect your deployments to Terramate Cloud. No credit card required, which I highly appreciate when trying things out.

With these two things in place we must make Terramate Cloud aware of our deployed stacks. The onboarding process is very well described in the documentation and consists of three steps:

A Terramate-specific configuration must be added to your repository that points to the organization we created in Terramate Cloud as described here: https://terramate.io/docs/cli/on-boarding/terraform#_5-configure-your-repository
We must log into Terramate Cloud via the Terramate CLI as described here: https://terramate.io/docs/cli/on-boarding/terraform#_6-login-from-cli
And as a last step we must trigger an initial sync of our setup to Terramate Cloud. The easiest option is to trigger a drift detection which triggers the sync. As for the previous steps the documentation explains how to do that: https://terramate.io/docs/cli/on-boarding/terraform#_7-sync-stacks-to-terramate-cloud

That’s it. Now the stacks are onboarded to Terramate Cloud.

As we also want to monitor our stacks, we need to define CI/CD workflows to keep Terramate Cloud informed about changes but also get information back e.g, in the case of pull requests.

As my code is on GitHub, I made the necessary configuration to give Terramate access to my GitHub repository. This comprises the enabling of OIDC and adding a Terramate GitHub app to my GitHub repository. While I did that on repository level, you can also do that on GitHub organization level.

To have an interaction with Terramate I added three workflows to my repository:

A workflow for the deployment of the stacks after a PR got merged
A workflow for drift detection
A workflow for executing a preliminary planning of changes that gets triggered whenever a pull request is opened.

Setting up the workflows is also quite easy as the Terramate documentation provides templates for the three workflows:

The only adjustment I made was that I removed the automatic remediation of the drift in the drift detection workflow.

Taking a closer look at the workflow code, you will see that the change detection of Terramate CLI comes into play (no we do not use -X) as well as some Terramate Cloud specific options that are responsible for the synchronization:

There was only one thing I struggled with when using the proposed actions namely the conditional check for changes in the stacks. This is used to execute the action only if there have been changes in the stacks which is proposed like this:

- name: List changed stacks
        id: list
        run: terramate list –changed

The condition for consequnt steps is defined as

  if: steps.list.outputs.stdout

This did not work for me, and I also did not find stdout as a default output for a GitHub Actions workflow step. Maybe I missed something or I made a typo I could not identify, but to get things working I replaced the proposed code with the following code:

- name: List changed stacks
        id: list
        run: |
          terramate list --changed > changed-stacks.txt
          echo "CHANGEDSTACKS=$(awk 'END { print NR }' changed-stacks.txt)" >> $GITHUB_OUTPUT
          rm -rf changed-stacks.txt

Not elegant, but pushing the terramate list command output directly to the GitHUb output gave me some errors in the consequent execution.

The condition then reads:

if: steps.list.outputs.CHANGEDSTACKS > 0

And with that our setup is ready for exploring Terramate Cloud.

Overall, the onboarding experience is straightforward and there is no obstacle to try this offering out, at least none that I came across. The documentation is helpful especially when it comes to the templates for the pipelines which are a huge time saver. By the way: these templates also exist for GitLab and Bitbucket.

This onboarding experience raised the barrier quite a lot, let’s see if Terramate Cloud per se can keep up with it in the next blog post Exploring Terramate Cloud part 2 – A walkthrough.

Test Drive the Terraform MCP Server with GitHub Copilot Chat

Christian Lechner — Thu, 22 May 2025 08:05:41 +0000

Introduction

Hashicorp recently released a first version of an MCP server for Terraform that was announced at Microsoft Build. This allows us to enrich the experience with GitHub Copilot Chat when it comes to Terraform.
Currently the focus is on querying the Terraform Registry for artifact information and request recommendations.

In this blog post I want to guide you through the setup which has some pitfalls and show you some first experiences with the MCP server.

The Setup

To start our journey with the Terraform MCP server we must have some prerequisites in place namely:

Docker is installed and running
VS Code is installed and GitHub Copilot is set up. Make sure that you are running the latest stable version. The last few versions of VS Code came with some improvements and bug fixes when it comes to GitHub Copilot Chat and the integration with MCP servers.

If these prerequisites are in place we will first download the Docker image of the MCP server from Docker Hub via the command line:

docker pull hashicorp/terraform-mcp-server:latest

Now let's switch to VS Code and set up the configuration needed for the MCP server. I struggled a bit with the setup, and I got the impression that the documentation is a bit unclear, but maybe it is only me. Let's go through the steps together.

First, we must make sure that the settings in VS Code are matching the requirements for the MCP setup. There are two things that you need to check:

In the settings of VS Code make sure that the Agent mode for GitHub Copilot Chat is enabled:

Next, we must make sure that the MCP server is enabled:

Next, we want to make GH Copilot aware that there is a MCP server available. There are different options available:

Workspace setting - configuration for the current workspace
User settings - configuration for all workspaces
Autmatic discovery - this enables a auto discovery of an MCP server exposed by tools like Claude Desktop.

We will do the setup for one VS Code workspace (and do not mix it up with Terraform workspaces).

To do so, we create a new folder called test-terraform-mcp. Inside of this folder we create another folder called .vscode
Inside of the .vscode folder we create an empty file called mcp.json. We open the empty JSON file in VS Code:

There is a blue button in the lower right corner that says, "Add Server ...". Do not use this button. This opens some guided procedure that didn't help me at all and also behaves different than documented. We will paste the required configuration directly into the file. The configuration looks like this:

{
  "servers": {
    "terraform": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "hashicorp/terraform-mcp-server"
      ]
    }
  }
}

Now we save the file and wait a bit. Now we see a tiny start link in the line below "servers"

We press the start link to start the MCP server as defined in the mcp.json file. This will start the Docker container. This also changes the available links displayed in the mcp.json file:

Let's do another check that GitHub Copilot is aware of the MCP server. We open GitHub Copilot Chat and make sure that we are in the Agent mode:

Now we click on the tool icon in the Copilot Chat and check if the MCP server is available:

This opens a list in the command palette of VS Code showing all available MCP servers and their toolsets. The result should look like this:

Okay we are ready to go. The MCP server is running, and GitHub Copilot Chat is aware of it. Now we can start asking questions.

The Test Drive

Time to challenge the MCP server with some questions. My home turf is the Terraform provider for SAP BTP. Let's see if the MCP server can help us with some questions around the provider.

Let's start with a question about entitlements and what resources are available in the provider:

What I learned quickly is that you must point the MCP server to the right namespace and provider name. Names like "Terraform provider for SAP BTP" do not work well. GitHub Copilot might be able to finally find the provider, but some steps are needed in the agentic execution to get on the right track.

The agent will execute serveral steps using the MCP server toolsets to get to the result. Everything is transparent for the user which is nice:

And the result looks good:

That's what we expected. Let's go a bit deeper and ask for the details of the entitlements on subaccount level:

Yes that seems to work fine. Let's switch gears and test the other toolsets about modules:

The result is this:

Hmmm ... that's not exactly what we expected. Let's cross check the Terraform Registry to see if there are more modules available:

Okay let's give Copilot another chance and ask if there aren't any other modules available:

It stays confident with the (incomplete) answer. Then let's ask directly for the missing module:

Okay, now it found the module. I have no idea why it missed it as the metadata in the registry seems to connect the dots correctly, but maybe the search query from the toolset is restricting the results to "something with BTP" which is not visible in the module itself. Interesting to see that the metadata doesn't seem to be a part of query.

Conclusion

The MCP server for Terraform adds some nice features to the GitHub Copilot Chat experience. I see some value when starting with Terraform on SAP BTP and have such a integrated experience. Having said that for experienced users I see only limited value (despite of trying it like I did). To be fair, the MCP server for terraform is still in a very early stage (it's beta and version 0.1.0) so I am curious to see how it will evolve in the future. I will keep an eye on it.

When it comes to the setup. I think this could be a bit smoother with more details in the documentation, but overall it is not too complicated, and things are moving fast when it comes to VS Code and GitHub Copilot. I am quite sure that the setup will be easier in the future.

References

Here are some references for more information about the Terraform MCP server and MCP servers in VS Code:

Automating an Open Source Project with GitHub Actions

Christian Lechner — Tue, 25 Feb 2025 13:56:33 +0000

Introduction

If you are working on GitHub and maybe even maintain an open-source project, GitHub Actions are an excellent tool to automate tasks going far beyond "typical" CI/CD automation.

In this blog post I describe the GitHub Action setup that we are running in open-source projects in the context of developing a tool for working with Terraform/OpenTofu namely the import of existing setups.

Project Parameters

In this blog post we will focus on the GitHub repository https://github.com/SAP/terraform-exporter-btp and the setup therein. The project is an open-source project and contains a CLI that is written in Golang.

The CLI handles some Terraform task around the import of existing resources, so it also contains some Terraform configurations in the Hashicorp Configuration Language (HCL) mainly dealing with the setup for integration tests.

All changes to the repository must be made via pull requests. This is safeguarded by protecting the main branch via a ruleset. The configuration is part of the repository settings.

The CLI is published as a release using GoReleaser to build the release.

All interaction with the community is happening via the repository namely via GitHub issues and GitHub Discussions including milestones that are part of the issues.

From a development project perspective i.e., for managing the backlog, we are using GitHub Projects. We work on one central project that contains the issues of all our Terraform specific GitHub repositories.

The documentation of the CLI is also part of the repository and published to a GitHub page of the repository.

As we are part of an organization that belongs to a corporate, two more requirements come into play:

The repository should be attached to Sonar Cloud to have a standardized way of monitoring code quality
The security vulnerability reporting is managed via a central approach of the organization. Hence, we have not activated GitHubs's private vulnerability reporting.

Our Goal

Based on these parameters and requirements, we wanted to automate as many repetitive tasks that we face in our daily work and keep the overall quality of the repository and its contents as high as possible.

In addition, we wanted to make the life of the users that file issues or contribute code as easy as possible.

This comprises of course the code per se but also more generic tasks like handling of issues and PRs.

As we are on GitHub, it is a natural choice to use GitHub Actions for this goal. Let's see what we automated, how we did that and what to consider.

GitHub Action Areas

We use GitHub Actions (or other automation capabilities of GitHub) in different areas. We will discuss the different areas in dedicated sections.

We will cover the following topics:

Basic project hygiene
Issues
Pull request
CI tasks esp. testing
Integration Tests
Releases
Documentation Generation

Let us walk through the different areas and learn what approach we took.

Basic project hygiene

Some basic functionality is always used when we work on GitHub namely:

Dependabot to keep your projects up to date. We configured this via dependabot.yml to cover the ecosystems used in our project like Golang or GitHub Actions.
CodeQL, for vulnerability scanning of the code. We use a configuration defined in a codeql.yml file. CodeQL is triggered upon pull requests, pushes to the main branch as well as periodically

We also have switched on Secret Scanning for generic secrets. This configuration is part of the repository settings and available under the Code Security settings.

As we have markdown files in our repository that contain links to places in the repository or to further external documentation we want to:

have spell checking of the markdown files
have a periodic checking of the links because what is more annoying than clicking on a link and then getting a 404.

Both requirements are covered by GitHub Actions.

Checking the links that are part of the repository is handled by a periodically executed workflow. The core of the check is the lychee link checking action.

A nice feature of this action is that if issues get detected like a site not being available a report gets created by the action.
We use this report to create an automated issue based on this report via the Create Issue From File action. This way we do not need to check the results of the GitHub Action runs but see it immediately in our issues.

Of course we do not want to have multiple issues for that, so if an issue exists and the action runs again it should update the existing issue. We achieve this, by combining Create Issue From File action with the Find Last Issue. To identify the issue, we use issue labels. Luckily the two action work perfectly together to cover this requirement.

With these basic automations in place, let us look at the next topic namely issues.

Issues

We use issues for bug reports as well as for feature requests. There are also freestyle issues that we might use internally, but the main interaction with the users is happening via bug reports and feature requests.

In general, we offer two different issue templates for bugs and feature requests to make the data entry and filing the necessary information as easy as possible.

For the optimal convenience for our users we use issue forms as they make the data entry much easier by offering features like checkboxes, drop down menus etc. The template also comprises the labels that get attached to the issue once created.

What to automate once an issue is filed. First, we do not want to manually add the issues to our central project board and so we trigger a workflow whenever an issue is opened that does this for us.

Not too much work from our end as GitHub provides an action for this called actions/add-to-project. As GitHub projects are separate entities that are not directly linked to the repository a GitHub token is needed that contains the right permissions.

For feature requests we want to make people aware that we prioritize based on the feedback from the community. To make these rules of the game obvious, we add an automated comment with a community note to every feature request using the Create or Update Comment action that adds a predefined text as a comment.

There is one scenario that probably has crossed your path as a maintainer of a project: what if you need to clarify something that was brought up via an issue?

Whenever we need a clarification from a reporter, we add a tag named needs-author-feedback to the issue. This label gets removed whenever a new comment is added via a (you already guess it) workflow that uses the Action Remove Labels.

What if the reporter does not answer? At some point in time, you probably want to get rid of the issue. We handle this by another workflow using a action provided by GitHub called Close Stale Issues and PRs.

We use this in a periodic workflow to first add a stale message after 15 days of no reply and finally close the issue after additional 5 days. Every step as accompanied by a comment on the issue that gets created by the workflow. This keeps our repository clean without the need to continuously check for stale issues manually.

Note - What about questions? We use GitHub Discussions to cover this area, but there is no automation needed for us so far in this context.

That covers the handling of the issues. Let us look next at pull requests.

Pull requests

As mentioned before, we require pull requests (PRs) for any code change in the repository. To have a structured code review process we have a pull request template in place to keep the PRs uniform and help the provider as well as the reviewer with the main points that need to be described.

There are some repetitive tasks that need to be done for each PR and as we are lazy, the following tasks are delegated to a workflow:

Adding the PR to the central project board
Adding the creator of the PR as assignee except for PRs opened by Dependabot
Adding the next open milestone
Setting the default labels based on the PR title

We add the PR to the project board via the same action that we use for issues. The other three tasks are a bit more coding to do as we did not find any fitting Actions on the GitHub Marketplace. Fortunately, it is quite easy to fulfill the requirements with a bit of Bash or Node.js code.

Note - When diving into the GitHub API to find the right endpoints you might be stunned that there is only a very limited number of dedicated endpoints for pull request. The reason for this is that technically pull requests are issues with some features on top. Check out the endpoints for issues and you will probably find what you are looking for.

To make the creator of the PR the assignee, we use the GitHub CLI that is available in the GitHub-hosted runners with this code snippet:

 - name: Add creator as assignee
   if: ${{ github.actor != 'dependabot[bot]' }}
   env:
    GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
   run: |
     gh api \
     --method POST \
     -H "Accept: application/vnd.github+json" \
     -H "X-GitHub-Api-Version: 2022-11-28" \
     /repos/${{github.repository}}/issues/${{github.event.number}}/assignees \
      -f "assignees[]=${{github.actor}}"

We use the information available in the action about the actor (github.actor) and call the API for the assignees. This is straightforward.

What about the next open milestone and the labels?

There is some more logic needed here, so we decided to use a small piece of Node.js code to achieve our goals. As a basis we use the GitHub Script action that contains a pre-authenticated Octokit REST client to call the API endpoints.

For the milestone requirement we must get the next open milestone and assign it to the PR. This is done via:

- name: Add next milestone
  uses: actions/github-script@v7
  with:
   script: |
     const milestones = await github.rest.issues.listMilestones({
       owner: context.repo.owner,
       repo: context.repo.repo,
       state: "open",
       sort: "due_on",
       direction: "asc"
     })

     await github.rest.issues.update({
        owner: context.repo.owner,
        repo: context.repo.repo,
        issue_number: context.issue.number,
        milestone: milestones.data[0].number
     });

The GitHub Script action helps a lot as you can directly interact with the API without wrapping your head around authentication. You also get the context information of the workflow injected. For larger pieces of code, I prefer dedicated files that I then call via Node.js instead of putting the code in the action. For this, this might be a bit too much though.

As a last piece we add the labels derived from the PR title that must follow the conventional commits. We use the keywords of the commit to distinguish which label to set:

- name: Set default labels
  if: ${{ github.actor != 'dependabot[bot]' }}
  uses: actions/github-script@v7
  env:
    TITLE: ${{ github.event.pull_request.title }}
  with:
    script: |
      const title = process.env.TITLE;

      let defaultLabels = [];
      if (title.startsWith("feat:")) {
        defaultLabels.push("enhancement");
      } else if (title.startsWith("fix:")) {
        defaultLabels.push("bug");
      } else if (title.startsWith("docs:")) {
        defaultLabels.push("documentation");
      } else if (title.startsWith("test:")) {
        defaultLabels.push("test setup", "internal", "ignore-for-release");
      } else if (title.startsWith("refactor:")) {
        defaultLabels.push("internal", "ignore-for-release", "refactoring");
      } else {
        defaultLabels.push("internal", "ignore-for-release");
      }

      await github.rest.issues.addLabels({
        owner: context.repo.owner,
        repo: context.repo.repo,
        issue_number: context.issue.number,
        labels: defaultLabels
      });

We exclude PRs filed via Dependabot.

Now the reviewer can focus on the review and does not need to click around to fulfill organizational requirements. Don't get me wrong: the labels and the assignment to the project are important, but this could and should be automated.

Note - Maybe you ask yourself if I am aware of the GraphQL endpoints of GitHub. Yes I am, but up to now I had only one situation where I saw a benefit in using the GraphQL API without breaking my hands to get a result. You can achieve all what is mentioned here with GraphQLs, but I am happy with the REST endpoints ... maybe I am also not smart enough to use GraphQL 😉

As for issues we might run into stale PRs. Their handling uses the same workflow as for stale issues described in the previous section.

When opening a PR you also want to know if the code follows your quality standards, namely if tests pass etc. In the next section we describe what we automated when it comes to this area.

CI

When a PR is opened, the following workflows get triggered:

We execute a spell check to see if any typos made their way into the PR.
We validate the PR title to ensure that it follows the conventional commits logic. This is done via a workflow that uses the action-semantic-pull-request.

In addition we have one main test workflow that bundle several checks, namely:

Can the Go project be built?
Are the unit tests executed successfully?
Is the documentation that is generated for the CLI is up to date?

All these parts are Go CLI commands executed in the runner of the workflow.

As a last step we send the test coverage report from the unit tests to Sonar Cloud using the SonarQube Scan Action.

Note - There is one important thing to mention here when it comes to external tools and Dependabot. You usually use an API key or alike to be able to call external tools like Sonar Cloud. You store this information as a secret in your GitHub repository.

This works fine until a PR gets opened by Dependabot. For security purposes GitHub decided to not propagate the secret if the actor is Dependabot. To make the secret accessible to Dependabot, you must define them as Dependabot secrets.

These checks running for a PR are also defined as required status checks that must pass as part of the main branch protection ruleset.

Integration Test

Besides the unit tests that are part of the test workflow we also implemented a integration test workflow.

This workflow is more complex and interacts with real infrastructure. The idea is to execute the CLI functionality which imports infrastructure into Terraform and compare the created state with the expected state on resource level.

The main flow is:

Install the CLI build from source
Execute the CLI to execute the Terraform code generation
Execute a state import
Transfer the state into JSON format
Compare the newly created state with a reference state

The comparison of the two JSON states is done via a Node.js script. The script is called as part of the GitHub Action and compares the two state files in JSON format.

The workflow configuration to achieve this is mostly around bringing the state files in the right format. What is worth to mention is the storage of the reference state file.

We store this as a secret in the GitHub repository. There is only one small obstacle: you cannot directly store a JSON file as a secret.

Consequently, we use a little trick: we encode the JSON file as a base64 string and store the encoded string as a secret. When the workflow is executed we take the secret, decode it and pass it to the Node.js script.

As we have all tests automated and in place, time for the fun part, releasing a new version.

Release

The moment of truth is for sure the creation of a new release of the CLI. We do so by pushing a new tag to the repository which triggers ...(drumroll) a workflow.
This workflow uses the concept of reusable workflows to trigger the tests before the release via GoReleaser starts. Better safe than sorry!

Be aware that you might need to explicitly propagate secrets to the called workflows (see documentation).

The workflow executes the following steps:

Call of the test workflow
Call of the integration test workflow
Execution of the GoReleaser if the test workflows have been executed successfully
Generation of the documentation and deployment as GitHub page

Here is the screenshot of a release run:

The release notes are also created automatically via the GoReleaser step using GitHub native features. The format is defined in a dedicated configuration mapping the labels to the sections and getting a nice formatting with emojis

Note - There is also a pre-release check workflow that contains the two test workflows. We trigger this one manually when a release date is approaching to avoid unwanted surprises on release day.

Generation of GH Pages

The documentation of the CLI is provided via a GitHub page as part of the repository. We are using MkDocs to generate the content, but I think most of the tools in that area are well integrated with GitHub and GitHub Actions.

The corresponding workflow is therefore short comprising the installation of the MkDocs CLI and then calling it with the gh-deploy option.

This workflow can be triggered manually but is also integrated into the release process as final step. So, no manual hands-on needed for the documentation part.

What about Automating GitHub Projects?

GitHub Projects offer built-in automation that we use to automatically set the status of an issue in the project namely on the events of:

adding
closing
reopening

an issue.

We did not yet run into the need to add GitHub Actions here; however, it would be possible.

Points to Consider

There are two big point to consider when using GitHub Actions:

To execute a workflow the configuration file must land in the main branch. From then on you can run it from a branch. So, whenever you add a new GitHub Action, you are either a GitHub magician or you probably have to have at least two PRs end up with a running action
GitHub does not offer a way to locally run a workflow. There are projects that try to fill this gap. One prominent example is act. However, I had it running on a Windows machine until it did not work anymore (probably a Docker update killed it) and I also had some issues on a Mac with Apple silicon. Again, probably due to Docker, but at the end it is cumbersome for the user, no matter what the root cause is.

These are points that you must take into account when developing workflows. A bit of inconvenience, but from my experience worth it (and maybe you have more luck with act which would remove at least some pain).

Conclusion

Maintaining one or maybe several open-source projects is work. Even if it is part of your job, you probably want to focus on improving the features and functions of the project and not deal with tedious repetitive tasks, that could be automated.

In this blog post, we described the areas where we used GitHub Actions to automate these tasks by either using and combining existing Actions available on the GitHub Marketplace or using some custom bash or Node.js Code in combination with GitHub API to get stuff automated.

Maybe you could get some inspiration on what is possible, maybe you know better ways to achieve this. If this is the case, I am curious to learn about that in the comments!

What comes next for us? We are continuously evaluating if there are other points worth automating. Currently there is no hot take on our list, but let's see what the future will bring

With that - happy automating with GitHub!

Experimenting with Terramate and SAP BTP

Christian Lechner — Tue, 10 Sep 2024 07:34:09 +0000

UPDATE 13.12.2024

I updated several parts of this blog post as I missed one feature of Terramate concerning the output sharing namely the option to mock input. With this feature in place and the corresponding setting in the scripts, the challenge of setting up the stacks with dynamic dependencies worked.

The Challenge

In case you are working with Terraform in a multi-provider setup you certainly also came across the following challenge:

You want to create a resource 1 that depends on provider A and a resource 2 that depends on provider B. So far, so good, but the read-only data of resource 1 contains parts of the information that is needed to configure provider B. Resource 2 has a dependency to resource 1, so from a logical perspective things should work out, as from a runtime perspective provider B is needed once resource 1 and consequently all relevant information is available.

Unfortunately, Terraform as well as OpenTofu require the provider configuration fully in place from the start. A dynamic configuration or a lazy loading of the provider configuration is not possible, and I did not come across any plans to provide such a functionality.

In this blog post I want to present a possible solution leveraging Terramate. As an example, I will be using one example from my "daily life" namely SAP BTP and the Cloud Foundry environment on SAP BTP. This setup perfectly matches the above challenge:

The Cloud Foundry environment (namely the organization) can be created via the Terraform provider for SAP BTP. After the creation, we have access to the data relevant for authenticating against the Cloud Foundry provider namely the API endpoint of the environment.
In addition, we need another information namely the ID of the Cloud Foundry organization to setup further resources in Cloud Foundry.

So, jackpot ... this is a challenge and it is for real. Let's see how we can resolve it or at least make life a bit easier.

Status Quo

Currently if we want to resolve the situation, we must execute Terraform in two steps:

Setup the Cloud Foundry environment via the Terraform Provider for SAP BTP.
Do the consequent setup in Cloud Foundry via a second Configuration, transferring the information from the first step to the second one e.g., via outputs.

While this is doable in CI/CD pipelines e.g. via GitHub Actions, testing things locally gets painful. As a workaround you can write the output of step 1 to a terraform.tfvars file of step 2, which gives some level of convenience, but with this setup the setup in the CI/CD scenario differs from your local setup which is also not desirable.

This is where I think Terramate can help. Although it cannot do some black magic and make things work as we would love them to, we can get a more streamlined setup. What do we need for that? Let's first explore some (experimental) features of Terramate that can help us.

Terramate (Experimental) Features

Terramate as a productivity tool on top of Terraform (or OpenTofu) can support us with the orchestration of the Terraform flow. Our recipe we want to use contains the following ingredients:

Stacks which we will use to define self-contained units for the deployment
Explicit Order of Execution which we use to ensure that in case of a full deployment the self-contained units are executed in the right sequence
Outputs Sharing(experimental) for transferring the output of one stack to the dependent one
Scripts(experimental) to split the Terraform specific commands from the Terramate commands and to avoid multiple commands to be keyed in reducing the chance to make a mistake.

As you can see we will leverage some experimental features which "might still be subject to changes in the future.". We love to live on the edge of new features, so we accept that, right?

Let's see how we can bring these ingredients together.

Terramating the Experience of SAP BTP and Cloud Foundry

Let us shortly recap the setup that we plan to have:

On the SAP BTP side of the house we first want to setup a subaccount and an envrionment of type Cloud Foundry. This is done via the Terraform provider for SAP BTP.
After that we want to create a space in the Cloud Foundry environment and if applicable assign some space roles. This is done via the Terraform provider for ≈Cloud Foundry that needs the output of the previous setup for the configuration.

The code samples that you will see, will focus on the Terramate features mentioned above. I did not use the code generation feature of Terramate to avoid distracting from the main topics of this blog post. Having said that, we would probably add code generation to the setup when bringing things to production.

Defining the stack

First we need to do is define the Terramate stacks. The kind of natural split is to have one stack for the subaccount and one for the Cloud Foundry setup. We achieve this via:

terramate create --name "subacccount-dev" --description "Basics for BTP development setup" --tags "subaccount,dev" stacks/subaccount_dev
terramate create --name "cloudfoundry-dev" --description "CF for BTP dev setup" --tags "cloudfoundry,dev" stacks/cloudfoundry_dev

This results in this directory structure:

| - stacks
|   | - subacount_dev
|   | - cloudfoundry_dev

Each of the stacks contains the stack information in the stack.tm.hcl file.

Excellent. Let's move on to bring them in the right execution order.

Defining the execution order

Both stacks are on the same level, so no implicit dependency and consequently no order is defined. Nevertheless, we would like to have such an order. We could either use sub-stacks or we define the order explicitly. In this blog post we make use of the second option.

We configure the order of execution via the after block in the stack definition. As we want to instruct Terramate to execute the Cloud Foundry stack after the subaccount stack, we adjust the stack.tm.hcl file accordingly:

stack {
  name        = "cloudfoundry-dev"
  description = "CF for BTP dev setup"
  tags        = ["cloudfoundry", "dev"]
  id          = "d5962b3f-3b79-412f-9970-93112741855a"
  after       = ["tag:subaccount:dev"]
}

We can specify the after block via tags as we did or via the path to the stack. Using the tags is considered as best practice and keeps us more flexible. We can validate the sequence via the command:

terramate list --run-order

With that we can add the configuration for the setup.

The Terraform configuration

I do not want to clutter this blog post with a lot of basic Terraform code, so the following snippets focus on the main bits and pieces of our storyline. You find the complete code on GitHub.

In the subaccount_dev stack we define the resources as well as the provider configuration as "usual". Although we need to define an output for the configuration to get access to the API as well as to the organization ID of the Cloud Foundry environment, we do not do that manually.

Terramate will close the gap here. For a better understanding of the following code snippets, this is the main.tf of the subaccount stack:

resource "random_uuid" "uuid" {}

locals {
  random_uuid       = random_uuid.uuid.result
  subaccount_domain = lower("${var.subaccount_name}-${local.random_uuid}")
  subaccount_name   = var.subaccount_name
  subaccount_cf_org = substr(replace("${local.subaccount_domain}", "-", ""), 0, 32)
}

resource "btp_subaccount" "sa_dev_base" {
  name      = var.subaccount_name
  subdomain = join("-", ["sa-dev-base", random_uuid.uuid.result])
  region    = lower(var.region)
}

# Fetch all available environments for the subaccount
data "btp_subaccount_environments" "all" {
  subaccount_id = btp_subaccount.sa_dev_base.id
}

# Take the landscape label from the first CF environment if no environment label is provided
resource "terraform_data" "cf_landscape_label" {
  input = [for env in data.btp_subaccount_environments.all.values : env if env.service_name == "cloudfoundry" && env.environment_type == "cloudfoundry"][0].landscape_label
}

# Create the Cloud Foundry environment instance
resource "btp_subaccount_environment_instance" "cfenv_dev_base" {
  subaccount_id    = btp_subaccount.sa_dev_base.id
  name             = local.subaccount_cf_org
  environment_type = "cloudfoundry"
  service_name     = "cloudfoundry"
  plan_name        = var.cf_plan_name
  landscape_label  = terraform_data.cf_landscape_label.output

  parameters = jsonencode({
    instance_name = local.subaccount_cf_org
  })
}

The resource of interest is the btp_subaccount_environment_instance which delivers the necessary information for the other stack.

When it comes to the cloudfoundry_dev stack, we also define the project with the usual structure, a main.tf with the space creation and the space role assignment:

resource "cloudfoundry_space" "dev_space" {
  name = var.cf_space_name
  org  = var.cf_org_id
}

resource "cloudfoundry_space_role" "space_developer" {
  for_each = toset(var.cf_space_developers)
  username = each.value
  type     = "space_developer"
  space    = cloudfoundry_space.dev_space.id
}

resource "cloudfoundry_space_role" "space_manager" {
  for_each = toset(var.cf_space_managers)
  username = each.value
  type     = "space_manager"
  space    = cloudfoundry_space.dev_space.id
}

and a provider configuration that looks like this:

terraform {
  required_providers {
    cloudfoundry = {
      source  = "sap/cloudfoundry"
      version = "1.0.0-rc1"
    }
  }
}

provider "cloudfoundry" {
  api_url = var.cf_api_url
}

We see that the API URL in the provider configuration as well as the Cloud Foundry org ID is a variable (var.cf_api_url and var.cf_org_id). Defining the variables is a bit counterintuitive now. As for the outputs in the other stack we leave these two variables out of the variables.tf file which looks like this:

variable "cf_space_name" {
  type        = string
  description = "The name of the Cloud Foundry space."
  default     = "dev"
}

variable "cf_space_managers" {
  type        = list(string)
  description = "List of managers for the Cloud Foundry space."
  default     = []
}

variable "cf_space_developers" {
  type        = list(string)
  description = "List of developers for the Cloud Foundry space."
  default     = []
}

As for the outputs this looks weird at a first glance (and also at a second glance) but as the configuration is in place, we can define the necessary bits and pieces for the output sharing and close this gap.

Sharing the Output between stacks

Now we sparkle some Terramate magic dust onto the stacks to connect the output of the subaccount_dev stack with the input of the cloudfoundry_dev stack.

We define a backend (not to mix up with a remote backend of Terraform) for the sharing. To do so we create a file called terramate.tm at the root level (= one level above the stack directories) with the following content:

terramate {
  config {
    experiments = [
      "outputs-sharing"
    ]
  }
}

sharing_backend "default" {
  type     = terraform
  filename = "sharing_generated.tf"
  command  = ["terraform", "output", "-json"]
}

The config block advices Terramate to enable the experimental feature of output sharing. The sharing_backend block defines which filename should be used to generate the content for the sharing namely the variables as well as the output via the filename attribute. The command attribute specifies the command Terramate should use to access the output. It's the usual suspect when it comes to outputs of Terraform.

Next, we configure Terramate what is the output that should be shared. We achieve this by adding a configuration inside of the subaccount_dev stack that we call sa_dev_config.tm with the following content:

output "cf_api_url" {
  backend   = "default"
  value     = "${jsondecode(btp_subaccount_environment_instance.cfenv_dev_base.labels)["API Endpoint"]}"
  sensitive = false
}

output "cf_org_id" {
  backend   = "default"
  value     = btp_subaccount_environment_instance.cfenv_dev_base.platform_id
  sensitive = false
}

In a nutshell we define the output variables of the stack via output blocks. The configuration tells Terramate which backend to use (the default one we created before), the value of the output as well as the sensitivity of the value.

As a counterpart we define the missing variables in the cloudfoundry_dev stack via the file cf_dev_config.tm inside of the stack directory:

input "cf_api_url" {
  backend       = "default"
  from_stack_id = "ca4662d3-b75d-4290-9842-5bb8ef924d97"
  value         = outputs.cf_api_url.value
  mock          = "https://api.cf.ap21.hana.ondemand.com"
}

input "cf_org_id" {
  backend       = "default"
  from_stack_id = "ca4662d3-b75d-4290-9842-5bb8ef924d97"
  value         = outputs.cf_org_id.value
  mock          = "917f57a1-8fee-43b3-b3a8-4bb4ce8259ab"
}

We use input blocks or the definition. Besides the backend we specify the ID of the stack where the values come from (from_stack_id) as well as the value. Beware that we are using terraform output -json, so we must reference the attribute of the JSON object via value.

UPDATE 13.12.2024: In addition, we know that the values won't be availabel during the planning phase. Hence, we add some mock data that should be used in case of an error to make the planning phase pass.

With that we are good to go and can start the code generation of Terramate via:

terramate generate

As a result, a new file will appear in the stacks named sharing_generated.tf. This corresponds to our configuration and the content of the files closes the gap of the Terraform configuration of the step before by creating the missing output

// TERRAMATE: GENERATED AUTOMATICALLY DO NOT EDIT

output "cf_api_url" {
  value = "${jsondecode(btp_subaccount_environment_instance.cfenv_dev_base.labels)["API Endpoint"]}"
}
output "cf_org_id" {
  value = btp_subaccount_environment_instance.cfenv_dev_base.platform_id
}

and input:

// TERRAMATE: GENERATED AUTOMATICALLY DO NOT EDIT

variable "cf_api_url" {
  type = any
}
variable "cf_org_id" {
  type = any
}

From a configuration perspective we are complete. Now we need to execute Terramate i.e. use Terramate to execute Terraform.

Script it!

Taking one step back, our goal is to automate the provisioning flow which means issue several Terraform commands. We do not want to do that manually firing one command at a time but we want to have a pre-defined flow for setting up and also tearing down the infrastructure.

Again, Terramate offers a solution for that namely using scripts.

As the scripting is still in experimental stage, we must activate it in the already existing configuration terramate.tm. We add the value scripts to the experiments list, so the file looks like this:

terramate {
  config {
    experiments = [
      "outputs-sharing",
      "scripts"
    ]
  }
}

sharing_backend "default" {
  type     = terraform
  filename = "sharing_generated.tf"
  command  = ["terraform", "output", "-json"]
}

Next, we define the two scripts. We call them deploy.tm and teardown.tm and store them in the stacks directory.

The deploy.tm contains the command flow from initialization, validation, planning and applying of the Terraform configuration. According to the Terramate documentation the file has the following content:

script "deploy" {
  job {
    name        = "Terraform Deployment"
    description = "Init, validate, plan, and apply Terraform changes."
    commands = [
      ["terraform", "init"],
      ["terraform", "validate"],
      ["terraform", "plan", "-out", "out.tfplan", "-lock=false", {
        enable_sharing = true
        mock_on_fail   = true
      }],
      ["terraform", "apply", "-input=false", "-auto-approve", "-lock-timeout=5m", "out.tfplan", {
        enable_sharing = true
        mock_on_fail   = true
      }],
    ]
  }
}

The structure is intuitive and consistent with other objects in Terramate. The only thing that must not be forgotten is to add the option that the output sharing is enabled when executing the commands via enable_sharing = true.
UPDATE 13.12.2024 In addition we add the option mock_on_fail = true which tells Terramate to fall back on the mock value if the real value from the dependent stack is not yet available.

The script for the teardown follows the same logic and looks like this:

script "teardown" {
  job {
    name        = "Terraform Teardown"
    description = "Destroy Terraform setup."
    commands = [
      ["terraform", "destroy", "-input=false", "-auto-approve", "-lock-timeout=5m", {
        enable_sharing = true
        mock_on_fail   = true
      }],
    ]
  }
}

Let us first check if we did things correctly and Terramate finds the scripts. We do so via

terramate script list

The output should look like this:

Looks good. Now let us check where the scripts can be applied based on their location in the directories. We do so via the following command:

terramate script tree

The output looks like this:

Also looks good. The scripts can be applied to our stacks.

The directory structure with all the implementations in place finally looks like this:

Now time for some action.

3, 2, 1 ... and Action

UPDATE 13.12.2024

Before we can start the fun part we add the required parameters in the stacks via terraform.tfvar files, namely the globalaccount in the SAP BTP-specific stack and the cf_space_managers as well as the cf_space_developers in the Cloud Foundry stack.

Having the mocking in place we can finally overcome the challenge of the dependencies and can spin up everything with one single command.

Let's bring all the bits and pieces together and spin up the infrastructure with the following commands (don't forget to set your authentication information for SAP BTP and Cloud Foundry before):

terramate script run -X deploy

After that you will have all the desired resources up and running on SAP BTP. The combined output of Terramate and Terraform in the console tells exactly what happened and helps a lot if you are running into issues.

Note I use the -X option as I am lazy and do not want to commit every time, I do some changes or fixes (and I did some over the course of setting things up). In a productive setup I would not recommend that

We can also tear things down via the second script:

terramate script run -X --reverse teardown

Mind the --reverse as otherwise things might get funny (I never forget something like that ... I heard of people doing so but not me ;-) ).

And that's it, we did it!

Where to find the code

If you want to take a closer look at the code, you find it on GitHub in this repository.

Summary

Overall, we must state there is no perfect way to solve the challenge we mentioned in the beginning. This is due to the way how Terraform works. One could work around this dependency issues for the provider configuration but it is always cumbersome and often leads to solutions that work differently when trying things out locally compared to CI/CD pipelines and are error-prone and hard to understand.

UPDATE 13.12.2024
Terramate can help us with the setup to make it concise and at least from a user perspective more streamlined. The experimental feature of output sharing as well as scripts support resolve our challenge with a at least from my perspective clean and understandable solution following the paradigm of stacks that I think is a huge strength of Terramate.

As we are using experimental features there are some points that could be improved from a documentation perspective. I could figure out everything after some time, but I think the documentation of the output sharing as well as the one for scripts might need some more love before making them GA.

What also caused some confusion is that the VS Code plugin marked the files using the experimental syntax as having errors. The corresponding error message in the pop-up really helps you with figuring out if something is missing, but I would have expected that it also recognizes the overall configuration. Not a big thing though as you can validate the setup via the Terramate commands to see if everything is where and how it should be.

As the features still evolve it would be great if the scripts would also support filtering for tags inside of the script. That would streamline the setup even more.

With that happy Terraforming and Terramating!

Pulumi with Terraform – the easy way

Christian Lechner — Fri, 30 Aug 2024 11:22:57 +0000

Introduction

Some time ago I did some experimentation with bridging the Terraform Provider for SAP BTP to get a Pulumi provider. While this was not an easy task it was doable, and I could get things up and running.

I learned that some improvements have been made for this exercise at the Kubernetes Community Days Munich 2024, but did not yet follow up on them. And that was good. Why? Because hot of the press a blog post landed in the Pulumi blogs that sounded too. Good to believe: Introducing: Support For Using Any Terraform Provider with Pulumi.

What the blog post basically says is that with one pulumi command you can now create the Pulumi provider by referencing the Terraform provider for the language you want to write your setup with. Can you believe that? Well, I gave a try, and I was stunned. Curious what I saw? Then follow along.

Disclaimer

As I am working for SAP some disclaimer (that you are certainly used to from SAP folks): this blog post should show your options when you would like to use Pulumi for setting up resources on SAP BTP. There is no official support of Pulumi from SAP (at least at the time when writing this blog post).

With this statement let the fun start.

Generating Pulumi from the provider

I had the Pulumi CLI installed and followed a long the blog post. As JavaScript and TypeScript are languages that are heavily used in the SAP ecosystem, I decided to go for Typescript for my Pulumi setup.

After doing the authentication dance with pulumi login I created a new TypeScipt project via:

pulumi new typescript

After the scaffolding complete, the next step was the magic one: create a local Pulumi provider from the Terraform provider

pulumi package add terraform-provider SAP/btp

After that there it was ... the SDK based on the Terraform provider:

The generation just took a few seconds and looking into the code, it looked decent.

What to do next? The output of the command also gives you advice on that namely executing

npm add btp@file:sdks/btp

In addition, the output states to add the import of the SDK to the index.ts. Very convenient experience lowering the entry barrier a lot. Been there done that.

But now the moment of truth has come. Let us give it a try and deploy some resources to SAP BTP.

Giving it a try

First, we need to put some code into the index.ts so that Pulumi knows what to do. I did a simple setup with a directory and a subaccount underneath:

import * as btp from "btp";

const directory = new btp.Directory("my-dir-from-pulumi")

const subacountParameters: btp.SubaccountArgs = {
    description: "my subaccount from pulumi",
    parentId: directory.id,
    region: "us10",
    subdomain: "my-subaccount-from-pulumi"
}

const subaccount = new btp.Subaccount("my-ssubaccount-from-pulumi", subacountParameters)

Nothing fancy, but for a first try that should do the trick. I did not want to do a production grade setup, so I added the necessary configuration parameters via CLI:

pulumi config set btp:globalaccount 'MyGlobalAccountSubdomain'
pulumi config set btp:username 'My Email'
pulumi config set btp:password 'MySecretPassword' --secret

Configuration done, so let't give the deploy a try:

pulumi up

The output of the CLI said that it worked:

and so does the result on SAP BTP:

Awesome stuff!

Summary

The new functionality of Pulumi to create a language-specific SDK based on a Terraform provider is nothing less than breath-taking. It is straightforward to get the SDK and make use of it. The generated code (at least for TypeScript where I tested it) looks decent, so easy to generate and to use. Although the feature is brand-new when publishing this blog post, I can definitely recommend to give it a try.

As to be expected for a new feature there are also some known gaps that probably will be closed over time like generating the documentation. This clearness on what's possible and what is not (yet) is highly appreciated.

I also want to say that the experience with the Pulumi CLI is great. It has very good hints and makes your start especially for Pulumi newbie like me a smooth experience.

With that happy Terrafo .... ahh Pulumiing!

Mind the Terraform Modules

Christian Lechner — Mon, 29 Jul 2024 06:26:25 +0000

Introduction

One important feature of Terraform is the ability to structure your infrastructure configuration into reusable modules. In this blog post I would like to give an overview about modules include digging into potential pitfalls and how to avoid them.

Although I am mentioning Terraform only in this blog post, the statements also apply to OpenTofu.

What are modules?

Let us start with some (maybe confusing) terminology. Formally modules are defined as "[...] containers for multiple resources that are used together." (Source: Terraform documentation). So, every collection *.tf files kept together in a directory is considered a module. Not what would have to come to my mind first, but that is the definition.

Based on that we can further distinguish between two types of modules:

The root module
Child modules

Every Terraform configuration has a root module that contains your Terraform configuration. That's the files that you typically write, when staring your configuration journey. From this root module you can call other modules the so called child modules. In contrast to the root module, a child module can be used multiple times in the same configuration. It can also be used from several configurations. This is not possible for the root module.

For the sake of keeping things simple I will use the term "module" as a reference to a child module from now.

Why should I use modules?

Thinking about the "as Code" aspect a module consequently represents a reusable block of configurations with an interface (the variables you define) and probably an output. This approach enables you to structure your Infrastructure as Code configurations in a well-structured. It also gives you the opportunity to adhere to the Do Not Repeat Yourself (DRY) principle when crafting your configurations.

Modules in Terraform represent the same approach that you would take when working with "classical" application code: you would encapsulate functionality to give your code a clear structure. If there are bits and pieces that could be used in several spots you would probably put them in reusable assets.

Besides the DRY aspect the modularization also helps you to keep your configuration code clean and readable and avoid human errors due to copy-pasting configurations all over the place (and then forgetting to apply fixes in all places). Not to forget, the usage of modules can support in ensuring compliance and governance with respect to the infrastructure configurations starting from ensuring naming conventions to the point of restricting e.g., the allowed VM sizes that you can provision via a module.

Note: Of course modules do not exempt you to safeguarding things like allowed VM sizes also on platform level like via Azure policies on Azure as somebody might not use modules and directly provision resources or not use Terraform at all.

While the advantages of using modules (or modularize your code) are the same as with "traditional" application code, guess what the challenges are so too. I will come to this topic in a second touching the two angles of modules namely the provider and consumer part, but first let us have a look at how modules are defined in Terraform.

The anatomy of a module

Following the definition of modules there is no specific keyword to define a module. However, when calling a module, you must use the keyword module accompanied by the name of the module in your Terraform code. In addition, you must specify the source of the module via the source attribute. There are several source types available like local files, Git repositories, a S3 bucket or the Terraform registry. All the supported sources are listed in the documentation.

Let us look at a simple example. We want to leverage a module to provision an AKS cluster on Azure. This would be quite a bit of work to stitch together all the resources manually. Luckily there is a module available for this in the Terraform registry. To call this module in our configuration the following code snippet would do the trick with a minimalistic configuration:

module "aks" {
  source              = "Azure/aks/azurerm"
  version             = "9.1.0"
  resource_group_name = "rg_my_aks_cluster" 
  }

This module comprises 19 resources, so obviously leveraging the module makes the code more readable and maintainable as you do not have to care about how to set them up. There are also a lot more optional parameters that you can feed into the module (more than 150 to be precise) to further tailor your AKS cluster configuration.

As you can see in the code snippet you can also specify a version for the module. An important topic that we will discuss in the following section when it comes to consuming modules.

Despite that the handling of modules is the same as with regular resources from a configuration perspective including meta-arguments like depends_on or count.

Note - If you are working with Azure, I want to draw your attention to the Azure Verified Modules. The goal of this initiative is to provide a consolidate set of standards on how a good Infrastructure-as-Code module should look like. As it is Azure it does not only cover Terraform, but also Bicep modules. This project is not only of interest for consumption but also getting inspiration when you are building your own modules.

Let us look at the two angles of modules the provider and the consumer, and what we need to keep in mind for each of them.

Building Modules - What to consider

The content of a module

When building a module in Terraform the aspects that we must consider are the same as when implementing a library or any re-useable chunk of code. The first point to consider is the design i.e. what resource combination should be contained in a module that makes the life of the user easier to provision a specific part on a cloud platform. The concrete number of resources that you combine in a module depends on your scenario. Nevertheless, avoid the following pitfalls:

If the module comprises only one resource you should rethink if this is worth a module.
Crafting "god modules" that contain too many resources that are covering a too huge use case or setup. As in programming high cohesion is the name of the game here. Bring together what logically belongs together and separate what does not.

Keep in mind that the number of resources in a module also defines its blast radius when things go sideways. With a proper design namely a reasonable combination of resources that have a clear responsibility you can limit the blast radius at least to a reasonable extent. As an extreme example it makes sense to have a module for setting up an AKS cluster and its components, but it makes in my opinion no sense to encapsulate a complete stage setup like DEV, TEST or PROD in a single module.

Storage and versioning

Once you have determined the boundaries of your module put the module into a dedicated code repository (one per module would be my advice) and make use of versioning of the module via the mechanics of the code repository. The consumer of the module will want to specify a version when calling the module.

You can also push the modules to an official registry like the one of Hashicorp, but from a user perspective this is not a must. It depends on your use case and the requirements and constraints in your setup.

Stick to common naming conventions

When structuring your Terraform files for your module, stick to common conventions when it comes to structure and naming i.e., have something like a main.tf, variables.tf and outputs.tf files (maybe with prefixes). This helps the consumer and the contributor to easily understand your module.

Interface of the module

You should define reasonable input and output variables. When it comes to the input variables also make sure to use default values wherever it makes sense to make the consumer's life easier. Also make sure that the variables are properly validated. Cross-object referencing that landed in Terraform 1.9 is something you should look at in that context.

The validations and checks can be further improved via pre- and post-condition blocks. The goal when designing a module must be to make it as easy and safe for the consumer to use it.

Depending on your scenario you might also have the requirement to create resources conditionally. My advice is to make this explicit via a dedicated Boolean input variable instead of implicit derivations like "if variable X has a value and Variable Y and Z have no value then do create resource C otherwise not". Using a dedicated variable makes it also easier for the consumer to know what to expect and easier for you as provider of the module to validate the input.

Write clean (Infrastructure as) Code

Although the consumer does not see you code, it should be well structured and maintainable. This will help you to maintain the module in the long run and make contributions easier. In that context you might want to look at the dynamic blocks that help you to get rid of repetitive code.

However, take this with caution: making the code better readable and maintainable via dynamic blocks can turn into the opposite. So do not overuse them and especially do not nest them too deep. Of course, this always depends on the scenario, but worth to keep in mind.

If you run into a situation where the nesting gets too deep, it might also be worth to look at tools that allow to generate Terraform code like Terramate.

Documentation

Having good documentation including examples and sample use cases are a must. No difference to any other software artifact. Make it easy for your user to understand what the module does and how to use it.

Testing

"With great power comes great responsibility" - as with every reusable code block this is also true for modules. You have only limited control about who uses your module where and how. Consequently, testing your modules is an important part of the process. You should use all options that Terraform provides (see e.g., this blog post about Testing HashiCorp Terraform) and automate the process via CI/CD pipelines.

One word about local modules

You can of course also use local modules to improve the structure of your code. I would see this as a first step from a monolithic configuration to a modularized monolith. Nevertheless, the lifecycle for your configuration is still tightly coupled.

With local modules there is also no option to reuse this code in other configurations. This can be a valid step to evaluate and get an impression what might make sense to be extracted into a module but is maybe not yet the final step if you want to foster reuse.

Using Modules - What to consider

Let us switch perspective and put ourselves in the shoes of a consumer. Assume that we have found a module that perfectly fits our needs and follows all the best practices described above.

There is one important thing as a caller we must consider: specify the version of the module. This way we make a basic step towards safeguarding the setup by (hopefully) always using the same version and doing version upgrades intentionally. As mentioned before for modules from the Terraform registry this is achieved via the version attribute in the module block:

module "aks" {
  source              = "Azure/aks/azurerm"
  version             = "9.1.0"
  resource_group_name = "rg_my_aks_cluster" 
  }

If you are using a git repository you can use the ref argument and target a released version or (my preference) the SHA hash. I highly recommend doing this to avoid unpleasant surprises.

So, when specifying a version everything is great in terraform country, right? Right ...? Well, not perfectly. Let us dive a bit deeper into the mechanics.

Let us assume the besides the module configuration above we also have specified a provider configuration:

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "3.113.0"
    }
  }
}

The first step we do in our Terraform workflow is to run a terraform init. This command downloads the modules and provider sources and stores them in the .terraform directory. In the very first run it will select the newest available versions specified in your configuration that matches the given version constraint. With this information it will fill another file that landed in your file system, namely the .terraform.lock.hcl file:

The file is the so called dependency lock file. It contains the checksums of the version that you downloaded. Once this lock file is there a consequent terraform init will always fetch the version that was downloaded before even in case that a newer version exists that fulfills the version constraints.

Taking a closer look at the lock file we see that it not only contains the version constraints, but
also a checksum of the downloaded version:

Besides evaluating the existing version, Terraform will execute a checksum verification for the provider version. In case the checksum deviates for a version that exists in the lock file, Terraform will raise an error.

Note: The initial verification of the checksum is up to you. This is not covered by any built-in mechanics of Terraform. There are also scenarios where Terraform might not be capable to verify the checksum. Please refer to the documentation for more details and the solution via terraform providers lock command.

That's good and safeguards your setup. But wait ... what about the modules? They are also downloaded, but they are not reflected in the dependency lock file. If a module owner accidentally (or intentionally) updates a module but releases the update with the same version, this cannot be detected by Terraform at least not by the built-in functionality.

That is not so cool when it comes to secure the supply chain of your infrastructure configuration. Is there a way to mitigate this? Fortunately Terraform has an ecosystem surrounding it and there is one quite new tool that can help you here: terrahash.

This tool fills the gap mentioned before. You can create, store, and validate the checksums for the modules that you use with the terrahash CLI. One word of caution before we take a closer look: terrahash is in its early stages (version 0.1.0 when writing this post). Basic functionalities are there, but there are certainly missing features and maybe bugs. Nevertheless, it is worth to take a look. As it is open source, you can also contribute to it.

With that in mind, let us see what this CLI can do for us.

Using terrahash

The first thing that we do is execute the following command:

terrahash init

This evaluates the current configuration and generates a .terraform.module.lock.hcl file in analogy to the dependency lock file for the providers. The output of the command tells you exactly what it did:

The content of the new file looks familiar:

The file contains the checksums of the modules that you use. Looks good, now how to we validate the checksums? As this is an additional tool the checksum validation is not part of the Terraform CLI and must be triggered manually. However, in a productive setup with CI/CD pipeline this is not an issue as we can integrate this as additional pipeline step. We can trigger the check via the following command:

terrahash check

The output in a success case looks like this:

And here we go in case of an error:

As promised this tool is a good addition to the Terraform setup when using modules and puts another piece in the puzzle of securing your infrastructure configurations with respect to unwanted version changes.

For the sake of completeness as a consumer you should of course also take of care of testing your configurations, but in contrast to a reusable module the blast radius is limited to your configuration (which might already be worse enough if you are not doing it ... nothing gets your heart rate up like accidentally destroying resources on production)

That's it from the perspective of a consumer. So let us summarize what we discussed in this blog post.

Summary

In general. modules are a good approach to structure your IaC configurations. As I have my background in application development is interesting to see that the "as Code" aspects heavily come into play when thinking about modules. The exact same challenges arise as with "traditional" code be it from the perspective of a provider of module and the things you must warp your head around as soon as you provide reusable (Infrastructure as) code. Fortunately, you can use the same strategies and patterns to deal with these challenges.

The same is true from a consumer perspective. Here the Terraform landscape has a gap when it comes to ensure the version integrity of the modules that you use. Luckily there is an ecosystem around Terraform and even more luckily with terrahash a brand-new open-source tool is available that can help you to deal with this gap.

Finally, I want to stress that although modules support you in your IaC journey towards a sustainable and maintainable setup, they are no silver bullet. You need to look at several aspects of your setup and while modules can help you with reuse, there are challenges that you will not be able to solve with them and you should also take other tools into account like e.g., Terramate. One further advice if you are starting your journey: always take one step at a time, work iteratively, and do not try to boil the ocean by targeting the "perfect" setup from the beginning.

With that … happy Terraforming and Terrahashing!

P.S. I am also quite sure that I missed points and best practices to describe, so feel free to add them in the comments.

Terramate this SAP BTP!

Christian Lechner — Mon, 17 Jun 2024 06:26:07 +0000

Introduction

Let us assume that we have a company that wants to standardize the setup of SAP BTP subaccounts.

The company decides to define a standard setup of SAP BTP subaccounts with the following constraints:

A subaccount should be created under a directory
Every subaccount should get assigned some basic entitlements for app development
A Cloud Foundry environment should be created
Emergency subaccount admins should be added
The resources should be labelled according to the company’s standards

Of course, the company wants to leverage Terraform i.e. the Terraform provider for SAP BTP for the setup. It uses a three-stage approach namely a subaccount for development, test, and production.

To get some inspiration it checks the available samples for SAP BTP available on GitHub. It finds a sample that seems to fit to the requirements. Maybe some adoptions are needed, but it is a good starting point.

The sample uses a Terraform module to setup a subaccount. It creates the three stages by iterating over a list containing the stages and executes the module which results in the expected setup.

While the example is a good starting point to get ideas on how to do such a setup and what options you have with Terraform, it also shows some drawbacks that we should address:

Although it makes sense to keep the setup logically together, the current configuration is tightly coupled in one configuration. The blast radius is high as all three stages might be affected by a change.
It is difficult to distinguish environments when it comes to e.g., introducing variants depending on the stage.
Let us say we want to test a new version of the module or a new version of the Terraform provider on the development environment, this is difficult to achieve.
One huge state gets created containing all three environments and this is definitly a too tight coupling. This makes follow-up activities like drift detections hard to do.

Let us be fair, the code on GitHub is just a sample, but it shows that you need to design the setup properly to avoid these drawbacks.

The "usual" solution for this is to split the configuration into three, one for each stage. This makes things more self contained, but now we have another huge drawback: how do we keep the configurations in sync? Do we copy&paste the configurations? This is not a good idea as it is error-prone and not maintainable on the long run.

Terraform per se does not provide a perfect solution for this, but fortunately there is a huge ecosystem around Terraform that we can leverage. One tool that is worth to take a closer look at is Terramate. This tool seems to exactly address the issues we are facing. So let us try it out.

What is Terramate?

Terramate according to its documentation is a productivity tool for Terraform. Its value proposition is to fill the gaps mentioned before (and even more) that come into play when dealing especially with large/complex Terraform configurations by helping you around automation, orchestration, and observation.

Now you might say: "oh no, not another tool" or at least "hopefully I do not need to learn/use another language in addition to the configuration language that comes with Terraform".

I think this is where the Terramate team made some very smart decisions:

Terramate is acting on top of Terraform, so no restrictions concerning standard Terraform functionality. It enhances the features and functionalities of Terraform by covering the gaps. It is "just" another CLI you must install, but it doesn’t try to be a substitute for the Terraform CLI.
It uses the Terraform configuration language to configure the additional features and functions that come with Terramate. No need to leave the realms of the language you are anyway using when working with Terraform.

Before diving into the application of Terramate we must do some homework around the concepts of Terramate.

Note - There is a lot of features and functionality that Terramate provides. We will not cover all of them in this blog post. We will focus on the features that help us to address the issues we are facing with the current setup. If you want to get an overview or want to dig deeper in some topics, you should check the Terramate documentation.

The main concept that will help us is Terramate's concept of stacks. Stacks put an additional "layer" on top of Terraform by making the configuration and state a well-defined unit that can be managed via Terramate based on the Terramate configuration. Using stacks we can define stack-specific or stack agnostic Terramate functionality e.g. code-generation that probably helps us getting rid of copy&pasting activities.

Now you might say "great another layer equals more complexity ... as if the scenario wasn't complex enough". We will see soon that this is not the case. Of course, you must understand the additional concepts, but they fall in place quite well and help you to manage the complexity in a convenient way. Having said that, complex things remain complex (no matter what management tries to tell you), but Terramate helps you to manage the complexity in a more structured way.

Before repeating what is anyway available in the Terramate documentation, let us make things tangible and rebuild the setup for the dev-test-prod scenario leveraging Terramate.

Applying Terramate to the dev-test-prod scenario

The staged setup with dev, test and production is a perfect candidate for applying the stack concept. Let us start from scratch in an empty directory. First, we create the following directory structure to host the stacks on the file system:

| - stacks
|   | - dev
|   | - test
|   | - prod

Next, we create the stack configuration for the three stages via the Terramate CLI in each of the directories via the Terramare CLI. We key in the following commands:

terramate create --name "development" --description "Stack for BTP development setup" --tags "dev" stacks/dev
terramate create --name "testing" --description "Stack for BTP testing setup" --tags "tst" stacks/test
terramate create --name "production" --description "Stack for BTP production setup" --tags "prd" stacks/prod

As you can see, we gave each stack a name, a description, and a tag. I love the concept of tags in general, so great to have this feature in Terramate to flexibly categorize the stacks.

As a result, we find a stack.tm.hcl file in every directory. This file contains the stack-specific metadata. Here an example for the development stack:

stack {
  name        = "development"
  description = "Stack for BTP development setup"
  tags        = ["dev"]
  id          = "29300b60-f0bb-4dda-bb4a-f0320148b0ed"
}

As you can see the content reflects the parameters we defined. In addition, it contains a unique identifier for the stack.

After we have initialized the stacks, we want to put in the necessary Terraform configuration. We do not want to copy&paste the configuration back and forth, but we want to define things once and use the configuration in the different stacks considering the specific requirements for the stages.

To solve that Terramate brings another functionality to the table namely code genration. We will use this feature to generate the basic setup for the stacks. I am a big pro-ponent of code generation instead of generic magic, so this feature is very appealing to me.

Generating the Terraform configuration

Let us now create the stage/stack-specific Terraform configurations. We start with the provider configuration. The layout of the provider should be the same for all three stages. However, we want to allow some flexibility when it comes to the development stage to test new versions of the Terraform CLI as well as new versions of the Terraform provider.

To achieve that we define global variables that can be accessed in the code generation process. We put these variables in the configs.tm.hcl file in the root directory:

// Configure default Terraform version and default providers
globals "terraform" {
  version     = ">= 1.6.0"
  version_dev = ">= 1.8.0"
}

globals "terraform" "providers" "btp" {
  version     = "~> 1.4.0"
  version_dev = "~> 1.4.0"
}

globals "terraform" "modules" "btp_subaccount_module" {
  source = "github.com/btp-automation-scenarios/btp-subaccount-module?ref=67cb61948e19497377fb4e23f01dd301319c6907"
}

Here we specify several version constraints as well as the source of the module on GitHub.
Next, we create a directory called templates where we will put in the code templates for the code generation. We add a file called generate_provider.tm.hcl which serves as template for the code generation resulting in a provider configuration. It contains the following content:

generate_hcl "_terramate_generated_provider.tf" {
  content {
    terraform {
      required_version = tm_ternary(tm_contains(terramate.stack.tags, "dev"), global.terraform.version_dev, global.terraform.version)

      required_providers {
        btp = {
          source  = "SAP/btp"
          version = tm_ternary(tm_contains(terramate.stack.tags, "dev"), global.terraform.providers.btp.version_dev, global.terraform.providers.btp.version)
        }
      }
    }
    provider "btp" {
      globalaccount = var.globalaccount
    }
  }
}

The syntax is good to understand:

The generate_hcl block defines the contents of the configuration to be generated.
The block label "_terramate_generated_provider.tf" defines the file name of the generated file.
The content block contains the actual content of the file. Here we define the provider configuration. As you can see, we make use of some Terramate specific functions to assign the version of the Terraform CLI and the Terraform provider depending on the stage with the help of the global variables and the stack specific information. We check if the stack is tagged as “dev” using the tm_contains function together with the information available via the terramate.stack object. Depending on the availability we decide which version to use leveraging the tm_ternary function.

One last piece is missing that we need for the generation: Teramate needs to know which templates to use for the generation. We define this in the imports.tm.hcl file on root level. The file has the following content:

# Import helper files
import {
  source = "./templates/generate_provider.tm.hcl"
}

Okay, ready to go? Let us kick off our first code generation via:

terramate generate

We see the following output:

And indeed, every stack now contains a generated provider.tf file according to our naming:

Next stop: the variables. We want to make the call of the setup as easy as possible, so we will default the values where possible i.e., we will only leave the globalaccount variable open for the user to provide. We will also make use of the stack specific information to:

Assign the architect role to the emergency admins in the development stage, otherwise the organizationally assigned admins
Assign the entitlement free for HANA Cloud in the development stage, otherwise the entitlement hana.

To achieve this, we create a new template file for the variables in the templates directory called generate_variables.tm.hcl. We put the following code into the file:

generate_hcl "_terramate_generated_variables.tf" {
  content {
    variable "globalaccount" {
      type        = string
      description = "The globalaccount subdomain where the sub account shall be created."
    }

    variable "region" {
      type        = string
      description = "The region where the account shall be created in."
      default     = "us10"
    }

    variable "unit" {
      type        = string
      description = "Defines to which organisation the sub account shall belong to."
      default     = "Sales"
    }

    variable "unit_shortname" {
      type        = string
      description = "Short name for the organisation the sub account shall belong to."
      default     = "sls"
    }

    variable "architect" {
      type        = string
      description = "Defines the email address of the architect for the subaccount"
      default     = "genius.architect@test.com"
    }

    variable "costcenter" {
      type        = string
      description = "Defines the costcenter for the subaccount"
      default     = "1234509874"
    }

    variable "owner" {
      type        = string
      description = "Defines the owner of the subaccount"
      default     = "someowner@test.com"
    }

    variable "team" {
      type        = string
      description = "Defines the team of the sub account"
      default     = "awesome_dev_team@test.com"
    }

    variable "emergency_admins" {
      type        = list(string)
      description = "Defines the colleagues who are added to each subaccount as emergency administrators."
      default     = tm_ternary(tm_contains(terramate.stack.tags, "dev"), ["somearchitect@test.com"], ["jane.doe@test.com", "john.doe@test.com"])
    }

    variable "entitlements" {
      description = "List of entitlements for a BTP subaccount"
      type = list(object({
        group  = string
        type   = string
        name   = string
        plan   = string
        amount = number
      }))

      default = [
        {
          group  = "Audit + Application Log"
          type   = "service"
          name   = "auditlog-viewer"
          plan   = "free"
          amount = null
        },
        {
          group  = "Alert"
          type   = "service"
          name   = "alert-notification"
          plan   = "standard"
          amount = null
        },
        {
          group  = "SAP HANA Cloud"
          type   = "service"
          name   = "hana-cloud"
          plan   = tm_ternary(tm_contains(terramate.stack.tags, "dev"), "free", "hana")
          amount = null
        },
        {
          group  = "SAP HANA Cloud"
          type   = "service"
          name   = "hana"
          plan   = "hdi-shared"
          amount = null
        }
      ]
    }
  }
}

We add the template to the imports.tm.hcl file that now contains two blocks:

# Import helper files
import {
  source = "./templates/generate_provider.tm.hcl"
}

import {
  source = "./templates/generate_variables.tm.hcl"
}

Good to go for a second round of code generation:

terramate generate

The output shows that the variables are generated for every stack:

And we also find the generated files in the stacks:

But hold on a second: the output also shows that only the delta is generated. Files that do not need to be touched are not changed even if we re-run the terramate generate command. I like that!

Last but not least we must generate the main configuration file. The only requirement we have is that the usage attribute of the subaccount is set to USED_FOR_PRODUCTION in the testing and production stage, while the development environment should be set to NOT_USED_FOR_PRODUCTION. In addition, we want to use the label of the stack to feed into the stage parameter of the module that creates the subaccount.

We know the drill by now and create a new template called generate_main.tm.hcl in the templates directory:

generate_hcl "_terramate_generated_main.tf" {

  lets {
    stage = tm_upper(tm_element(terramate.stack.tags, 0))
  }

  content {
    # ------------------------------------------------------------------------------------------------------
    # Creation of directory
    # ------------------------------------------------------------------------------------------------------
    resource "btp_directory" "parent" {
      name        = "${var.unit}-${terramate.stack.name}"
      description = "This is the parent directory for ${var.unit} - ${terramate.stack.name}."
      labels      = { "architect" : ["${var.architect}"], "costcenter" : ["${var.costcenter}"], "owner" : ["${var.owner}"], "team" : ["${var.team}"] }
    }

    # ------------------------------------------------------------------------------------------------------
    # Call module for creating subaccoun
    # ------------------------------------------------------------------------------------------------------
    module "project_setup" {

      source = "${global.terraform.modules.btp_subaccount_module.source}"

      stage  = "${let.stage}"
      region = var.region

      unit                = var.unit
      unit_shortname      = var.unit_shortname
      architect           = var.architect
      costcenter          = var.costcenter
      owner               = var.owner
      team                = var.team
      emergency_admins    = var.emergency_admins
      parent_directory_id = btp_directory.parent.id
      usage               = tm_ternary(tm_contains(terramate.stack.tags, "dev"), "NOT_USED_FOR_PRODUCTION", "USED_FOR_PRODUCTION")
    }
  }
}

The source of the module is defined via the global variable we defined before, and the usage is set depending on the stack label using the known Terramate functions. For the sake of demoing the capabilities we introduced local variables via a lets block to define the stage variable. We make use of two further Terramate functions to get the first element of the stack tags (tm_element) and to convert it to upper case (tm_upper).

We add the import to the imports.tm.hcl file which now finally looks like this:

# Import helper files
import {
  source = "./templates/generate_provider.tm.hcl"
}

import {
  source = "./templates/generate_variables.tm.hcl"
}

import {
  source = "./templates/generate_main.tm.hcl"
}

After the execution of the generation command, we see the main.tf files with the expected content in the stacks:

Shortly summarizing what we did until now:

We created Terramate stacks with some metadata representing the stages of the setup
We used the Terramate code generation feature to generate the provider configuration, the variables, and the main configuration for the Terraform setup including stack specific adjustments. As a consequence, we did not need to repeat ourselves or do some copy&paste exercises. In addition, we can now easily adapt the configuration in the future and regenerate the files.

Now time to get some infrastructure set up!

Using Terramate to run Terraform commands

As we have a complete Terraform configuration in the single directories, we could hop into each of those and execute the Terraform commands there. But can't we do better with Terramate? Yes, we can.

Terramate has the terramate run command that allows you to execute any command in the stacks. Let us try out some things.

First, we want to initialize all stacks. we do so by executing the following command:

terramate run terraform init

In the console we se that the Terraform commands gets executed per stack:

We recognize that the usual suspects appear in the file system:

Initialization successful. Next, we might want to plan the setup. We do so by executing the following command:

terramate run terraform plan -var globalaccount=<YOU GLOBAL ACCOUNT>

In the output we will see that the plan is executed for all three stacks in sequence. The output is a bit lengthy, so I will not provide a screenshot here.

The output reflects that the plan is executed for all three stacks. We will also see a different number of resources that are to be created as we have only one emergency administrator for the development environment, while we defined two for the test and production environment. Things work as expected.

We can also check the execution sequence via:

terramate list --run-order

The output looks like this and as we have no dependencies all stacks are on the same level ordered by the sequence in the file system:

The execution of the commands can also be done in parallel:

Terramate will do the parallelization if the stacks are independent from each other. If there are dependencies between the stacks, Terramate will execute the stacks in the correct order:

terramate run --parallel=3 terraform plan -var globalaccount=<YOU GLOBAL ACCOUNT>

This is then reflected in the output of the command:

We can also restrict the execution to a specific stack e.g., via the path:

terramate run --chdir=stacks/dev terraform plan -var globalaccount=<YOU GLOBAL ACCOUNT>

or even more convenient using tags:

terramate run --tags dev terraform plan -var globalaccount=<YOU GLOBAL ACCOUNT>

The output reflects the filtering:

You can of course also execute all the other commands and apply your setup. But I think the value of executing the commands via Terramate is clear now.

Note - Maybe you stumble across Terramate's built-in safeguarding to prevent executions on uncommitted changes when doing things on your machine. You can switch off all safeguards by adding the -X flag to the terramate run commands. It certainly makes sense though to check the documentation what are the effects before doing so.

Where to find the code

You find the code including a copy of the original monolithic setup on GitHub: https://github.com/btp-automation-scenarios/btp-terramate.

Conclusion and Outlook

When dealing with more complex Terraform setups that usually arise quickly, Terramate is a very valuable tool that you should take a look at. It helps you to manage the complexity in a structured way and comes with useful features to make your life in the Terraform world easier.

The rewriting of the dev-test-prod setup showed where Terramate could shine and helps us tackling the complexity that comes with these setups. Two main ingredients for achieving this is the concept of stacks and the code generation feature.

However, we left out one important aspect of remote backends. Here again the stack-based approach comes in handy as you can see in this sample provided by the Terramate team that showcases how to generate stack specific backend configurations: https://github.com/terramate-io/terramate-examples/blob/main/01-keep-terraform-dry/imports/generate_backend.tm.hcl.

There is so much more to explore in Terramate and in which scenarios it can support your setup. Looking at the documentation I just scratched the surface and need to spend a bit more time to understand all the details. But I am convinced it is worth it.

With that happy Terraforming with Terramate!

Note - I focused on Terraform in this blog post, but you can of course also use OpenTofu. There is no barrier built into Terramate that would prevent you from doing so.

Terraform - Let's keep the quality up!

Christian Lechner — Tue, 11 Jun 2024 06:07:10 +0000

Introduction

Terraform is one variant of managing Infrastructure as Code. Let us take a closer look at the the second part of the term namely at the "as-Code". What are good practices when it comes to the quality of the code when thinking about application development? Formatting and syntax checks as well as automated testing (unit and integration testing).

Shouldn't we apply the same approaches that we do when developing code for applications also when developing Terraform configurations? You can already guess what the answer is ... yes of course we should. And the cool thing is: a lot of support is available out of the box when using Terraform.

In this blog post we will walk through the different aspects that we should consider when writing Terraform configurations. We will cover the following topics:

Code Formatting
Code Validation
Unit Testing
Integration Testing

All the functionality will be demonstrated leveraging HashiCorp Terraform standard functionality using as an example the Terraform provider for SAP BTP. To automate the process, we will use GitHub Actions.

Code Formatting

Let us start with the very first step of ensuring the quality of the Terraform scripts - code formatting. The Terraform CLI provides a dedicated command terraform fmt to format the Terraform configuration files.

This command rewrites the Terraform configuration files to a canonical format and style. The good thing is that no configuration is needed as the formatting rules are predefined by Terraform itself. So no discussion in the teams how to format the code (yes I am looking at you JavaScript and the tons of options and source of infinite discussions).

So as first step to ensure the quality of the Terraform code namely a proper code formatting, run the following command:



terraform fmt -recursive

I added the -recursive flag to format all the files in the directory and its subdirectories. Good if you do that locally, but even better if you add it to your CI/CD pipeline assuming that changes in the Terraform scripts are done via pull requests (PR).

As an example we use GitHub Actions to ensure that whenever a pull request is created the Terraform scripts are checked for proper formatting. If this is not the case we let the pipeline fail. For that we create a GitHub Action that is triggered by a PR and basically does two things:

Check which directories contain changed files
Run the terraform fmt command on these directories to check if the formatting is correct

The GitHub Action workflow looks like this:



name: Terraform Format Check

on:
    pull_request:
      types:
        - opened
        - reopened
        - synchronize
        - ready_for_review

jobs:
    terraform-fmt:
      name: Validate Format of Terraform Files
      runs-on: ubuntu-latest
      steps:
        - name: Checkout code
          uses: actions/checkout@v4

        - name: Setup Terraform
          uses: hashicorp/setup-terraform@v3
          with:
            terraform_wrapper: false

        - name: Get changed directories
          id: changed-files
          uses: tj-actions/changed-files@v44
          with:
              dir_names: 'true'

        - name: Validate Terraform format
          if: steps.changed-files.outputs.any_changed == 'true'
          env:
            ALL_CHANGED_FILES: ${{ steps.changed-files.outputs.all_changed_files }}
          shell: bash
          run: |
              EXIT_CODE=0
              for file in ${ALL_CHANGED_FILES}; do
                echo "Checking format of $file with terraform fmt"
                terraform fmt -check -recursive "$file" || EXIT_CODE=$?
              done
              exit $EXIT_CODE

We use the community action tj-actions/changed-files to get the directories that contain changed files. The terraform fmt command is then run on these directories. Here we used an additional flag -check to check if the files are formatted correctly without formatting them.

Note - There are a few more options available for the terraform fmt that might come handy when executing the checks. You can find them in the official documentation.

The GitHub Actions workflow can be integrated in the required checks for pull requests and this way only proper formatted Terraform scripts will land in your repository. What's next? Let's look at the code validation.

Code Validation

Before executing a Terraform script it makes sense to statically check if the script is syntactically correct. This way we can already sort out some issues before the actual execution.

As for the formatting the Terraform CLI got us covered via the command terraform validate. In contrast to terraform fmt this command has some prerequisites before being applicable namely you must have a initialized workspace. The backend configuration can be omitted. Before executing the validation, you must therefore execute the following command in the directory where your Terraform configuration is located:



terraform init -backend=false

The -backend=false is needed if you have a backend configuration in place, but also doesn't do any harm in case there is no backend configuration provided. After that you can execute the validation. Use the following command to trigger the validation:



terraform validate

This will check the syntax of the Terraform scripts in the directory it is executed in and will return errors if there are any. From a automation perspective it is great that the command can return the result of the validation in a machine-readable format namely JSON by using the -json flag.

Note - You can find more information about the command and in particular the fields of the JSON object in the official documentation.

Let us assume that we want to run this simple validation (without any further jq magic on the JSON response) in our CI/CD pipeline, i.e. just execute the validation and in case it fails, let the pipeline fail. We can do so with the following GitHub Actions workflow:



name: Terraform Validation Check

on:
    pull_request:
      types:
        - opened
        - reopened
        - synchronize
        - ready_for_review

jobs:
    terraform-validate:
      name: Validate Syntax of Terraform Files
      runs-on: ubuntu-latest
      steps:
        - name: Checkout code
          uses: actions/checkout@v4

        - name: Setup Terraform
          uses: hashicorp/setup-terraform@v3
          with:
            terraform_wrapper: false

        - name: Get changed directories
          id: changed-files
          uses: tj-actions/changed-files@v44
          with:
              dir_names: 'true'

        - name: Validate Terraform sytnax
          if: steps.changed-files.outputs.any_changed == 'true'
          env:
            ALL_CHANGED_FILES: ${{ steps.changed-files.outputs.all_changed_files }}
          shell: bash
          run: |
              EXIT_CODE=0
              for file in ${ALL_CHANGED_FILES}; do
                echo "Validating Terraform files in $file with terraform validate"
                cd $file
                terraform init -backend=false || EXIT_CODE=$?
                terraform validate || EXIT_CODE=$?
                rm -rf .terraform/
                cd ${{ github.workspace }}
              done
              exit $EXIT_CODE

As in the case of the formatting check we assume that the change is served via a PR. As before we check which directories are changed, iterate over the list of directories and execute the validation for each of the directories. In case any validation fails, the GitHub Action workflow will fail.

Cool, so now we safeguarded the quality of the Terraform scripts by ensuring that they are properly formatted and that the syntax is correct. However, there is more that can be done using Terraforms testing framework. This framework is available since release 1.6.0 and was since then improved e.g. with test mocking in release 1.7.0.

The good thing in contrast to other Terraform testing frameworks is, that you can write the test in the same syntax that you already know i.e. the HashiCorp configuration language. This lowers the entry barrier and makes the adoption easier. Lets us take a closer look how this framework can help us improve the code quality.

Terraform Testing Framework

If you take a look at the original blog post about the testing framework by HashiCorp, you will recognize that it divides the tests into different categories that you certainly know from application development:

Unit Tests
Contract Tests
Integration Tests
End-to-End Tests

In this blog post we will focus on the categories of "Unit Tests" and "Integration Tests".

Note If you are also using Terraform modules in your company, I highly recommend to take a look at the "Contract Tests" as described in the blog post Testing HashiCorp Terraform.

The Basic Setup

Before we start with the testing let us first look at a basic setup. Assume we have a quite Terraform script that crates a subaccount on the SAP Business Technology Platform and assigns some entitlements to it. The configuration contains several input variables to ensure company guidelines for the naming and labeling of the subaccount. The variables.tf has the following layout:



###
# Provider configuration
###
variable "globalaccount" {
  type        = string
  description = "The subdomain of the SAP BTP global account."
}

variable "region" {
  type        = string
  description = "The region where the project account shall be created in."
  default     = "eu10"
}

###
# Subaccount setup
###
variable "project_name" {
  type        = string
  description = "The subaccount name."
  default     = "proj-1234"

  validation {
    condition     = can(regex("^[a-zA-Z0-9_\\-]{1,200}", var.project_name))
    error_message = "Provide a valid project name."
  }
}

variable "stage" {
  type        = string
  description = "The stage/tier the account will be used for."
  default     = "DEV"

  validation {
    condition     = contains(["DEV", "TST", "SBX", "PRD"], var.stage)
    error_message = "Select a valid stage for the project account."
  }
}

variable "costcenter" {
  type        = string
  description = "The cost center the account will be billed to."
  default     = "1234567890"

  validation {
    condition     = can(regex("^[0-9]{10}", var.costcenter))
    error_message = "Provide a valid cost center."
  }
}

variable "org_name" {
  type        = string
  description = "Defines to which organisation the project account shall belong to."
  default     = "B2C"

  validation {
    condition = contains(concat(
      // Cross Development
      ["B2B", "B2C", "ECOMMERCE"],
      // Internal IT
      ["PLATFORMDEV", "INTIT"],
    ), var.org_name)
    error_message = "Please select a valid org name for the project account."
  }
}

###
# Entitlements for Subaccount
###
variable "entitlements" {
  type = list(object({
    name   = string
    plan   = string
    amount = number
  }))
  description = "List of entitlements for the subaccount."
  default = [
    {
      name   = "alert-notification"
      plan   = "standard"
      amount = null
    },
    {
      name   = "SAPLaunchpad"
      plan   = "standard"
      amount = null
    },
    {
      name   = "hana-cloud"
      plan   = "hana"
      amount = null
    },
    {
      name   = "hana"
      plan   = "hdi-shared"
      amount = null
    },
    {
      name   = "sapappstudio"
      plan   = "standard-edition"
      amount = null
    }
  ]
}

The setup of the subaccount is given by the following main.tf file:



###
# Setup of names in accordance to the company's naming conventions
###
locals {
  project_subaccount_name   = "${var.org_name} | ${var.project_name}: CF - ${var.stage}"
  project_subaccount_domain = lower(replace("${var.org_name}-${var.project_name}-${var.stage}", " ", "-"))
  project_subaccount_cf_org = replace("${var.org_name}_${lower(var.project_name)}-${lower(var.stage)}", " ", "_")
}

###
# Creation of subaccount
###
resource "btp_subaccount" "project" {
  name      = local.project_subaccount_name
  subdomain = local.project_subaccount_domain
  region    = lower(var.region)
  labels = {
    "stage"      = ["${var.stage}"],
    "costcenter" = ["${var.costcenter}"]
  }
  usage = "NOT_USED_FOR_PRODUCTION"
}

###
# Assignment of entitlements
###
resource "btp_subaccount_entitlement" "entitlements" {
  for_each = {
    for index, entitlement in var.entitlements :
    index => entitlement
  }

  subaccount_id = btp_subaccount.project.id
  service_name  = each.value.name
  plan_name     = each.value.plan
}

This file translates the input variables into the right names and creates the resources. In addition we define output variables in the outputs.tf file:



output "subaccount_id" {
  value       = btp_subaccount.project.id
  description = "The ID of the project subaccount."
}

output "subaccount_name" {
  value       = btp_subaccount.project.name
  description = "The name of the project subaccount."
}

While this setup is quite simple, we see that some parts make sense to be tested. Let us therefore start and see how to do a Unit-Test on this configuration.

Unit Tests

Unit Tests per definition should run without any dependencies like external resources or API calls. In the Terraform world that means that we can test the configuration as long as we do not need any active resources and authentication against providers. This means that everything that parses the configuration can be tested in this stage. In our setup there a two great candidates for the tests:

The validation logic in the variables.tf file
The replacement logic of the local values in the main.tf file

We take the recommendations from the Terraform tutorial on testing as a guidance on how to structure our code.

Assume that our Terraform configuration is located in a directory infra we create a new directory tests as a subdirectory to put in all our tests. The structure looks like this:



.
| - infra
|   | - main.tf
|   | - variables.tf
|   | - outputs.tf
|   | - provider.tf
|   | - ...
| - tests

Let us start with testing the input validation. As a first test we want to check if the the variable costcenter is validated as expected. We create a new file called variables_costcenter_validation.tftest.hcl in the test directory. By convention the file name must end with .tftest.hcl to be recognized by Terraform as a test.

As we want an isolated test without any connection to the provider we must mock the provider. To do so we add the following code to the test file:



mock_provider "btp" { }

This will make sure that we do not need to authenticate against the provider. Looking at the variables.tf we see that we have a mandatory parameter that we need to supply namely the globalaccount. As we mocked the provider, we can define any value for this variable as it will not be used. we add a variable to the file which now looks like this:



mock_provider "btp" { }

variables {
    globalaccount = "test"
  }

Now we can finally add the test itself. Terraform tests are executed in a so called run blocks. We add one first block which should test that a valid costcenter is provided:



mock_provider "btp" { }

variables {
    globalaccount = "test"
  }

run "provide_valid_costcenter" {
  command = plan
}

As you can see we can also specify the command that should be executed. The validation is already executed in the plan phase so we added this as command.
Next we define the valid value of a costcenter in the run block:



mock_provider "btp" {}

variables {
  globalaccount = "test"
}

run "provide_valid_costcenter" {
  command = plan

  variables {
    costcenter = "8523652147"
  }


}

Finally we check if the variable is set as expected via an assert block that has the typical layout namely a condition and an error message if the condition is not met:



mock_provider "btp" {}

variables {
  globalaccount = "test"
}

run "provide_valid_costcenter" {
  command = plan

  variables {
    costcenter = "8523652147"
  }

  assert {
    condition     = var.costcenter == "8523652147"
    error_message = "Costcenter is not set correctly"
  }
}

Okay, now we also want to provide an invalid value and check if an error is raised. To do so we add another run block but instead of an assert block we add an expect_failures block resulting in:



mock_provider "btp" {}

variables {
  globalaccount = "test"
}

run "provide_valid_costcenter" {
  command = plan

  variables {
    costcenter = "8523652147"
  }

  assert {
    condition     = var.costcenter == "8523652147"
    error_message = "Costcenter is not set correctly"
  }
}

run "provide_invalid_costcenter_with_letters" {
  command = plan

  variables {
    costcenter = "abc-123"
  }

  expect_failures = [
    var.costcenter
  ]
}

Okay, so time to let terraform test do its magic. In the console we navigate to the infra directory and initialize the workspace:



terraform init -backend=false

Now we are set and can execute the tests:



terraform test

The output should look like this:

This pattern can then be applied to the other validations present in the variables.tf file.

For the sake of exploring further let us switch to the main.tf file and test the local values. We want to test if the local value project_subaccount_name is set correctly. We create a new file called local_project_subaccount_domain.tftest.hcl in the test directory. We want to check that the local project_subaccount_domain has no empty spaces in it.

As before we mock the provider as we just want to test the local values and provide a dummy value for the globalaccount variable:



mock_provider "btp" {}

variables {
  globalaccount = "test"
}

This time we must provide all the variables that are used to construct the local value i.e. org_name, project_name and stage. We add the following variables to the file in a run block:



mock_provider "btp" {}

variables {
  globalaccount = "test"
}

run "validate_project_subaccount_domain" {
  command = plan

  variables {
    org_name     = "B2C"
    project_name = "proj 1234"
    stage        = "DEV"
  }
}

We intentionally keep an empty space in the project_name to validate the replacement. We also keep stage and org_name in capital letters to validate the transformation to lower case. Next we assert that the value for the local variable is set correctly:



mock_provider "btp" {}

variables {
  globalaccount = "test"
}

run "validate_project_subaccount_domain" {
  command = plan

  variables {
    org_name     = "B2C"
    project_name = "proj 1234"
    stage        = "DEV"
  }

  assert {
    condition     = local.project_subaccount_domain == "b2c-proj-1234-dev"
    error_message = "Local variable project_subaccount_domain is not transformed correctly."
  }
}

We can now execute the test as before:



terraform test

We get the following result:

Great, we saw how we can do unit tests in an isolated way without the need to initialize the provider i.e. to authenticate against a real SAP BTP account.

As we want to execute the unit tests in a CI/CD pipeline we rename the tests directory to unit-tests. When doing so we must the testing framework explicitly to the right directory via the option -test-directory:



terraform test -test-directory=unit-tests

This allows us to create a GitHub Action that exclusively executes the unit tests. As the terraform validate command the terraform test command can return the test result in a machine readable format namely JSON. This can be used to further parse the test results in a CI/CD pipeline. A basic GitHub Action Workflow that executes the unit tests looks like this:



name: Terraform Unit Tests

on:
    pull_request:
      types:
        - opened
        - reopened
        - synchronize
        - ready_for_review
    workflow_dispatch:    

jobs:
    terraform-validate:
      name: Validate Syntax of Terraform Files
      runs-on: ubuntu-latest
      steps:
        - name: Checkout code
          uses: actions/checkout@v4

        - name: Setup Terraform
          uses: hashicorp/setup-terraform@v3
          with:
            terraform_wrapper: false

        - name: Execute Unit Tests
          shell: bash
          run: |
              cd ./infra
              terraform init -backend=false
              terraform test -test-directory=unit-tests

We omitted the check for changed files in this workflow. Now let us move on to the next level of testing - the integration tests.

Integration Tests

In contrast to the unit test setup, the integration test will interact with a backend and create real resources on the platform. The goal is to check if a given configuration creates a resource successfully. Be aware that this test should focus on your configuration. It makes no sense to test the provider itself with all its parameters. The test of the provider with all its parameters is done via the acceptance tests of the provider.

Keeping this in mind, we now want to write a test that the subaccount is created successfully by checking the state field of the newly created subaccount. We do not care about the entitlements, so we will mock the resource.

First we create a new directory integration-tests in the infra directory. In this directory we create a new file called subaccount_creation.tftest.hcl. As we want to test the resource for the subaccount we switch of the creation of the entitlements by adding a variable with an empty list of entitlements:



variables {
  entitlements = []
}

Next we add a run block that executes the apply command and provides the variable values for the subaccount:



variables {
  entitlements = []
}

run "test_successful_subaccount_creation" {
  command = apply

  variables {
    costcenter   = "1234567890"
    org_name     = "ECOMMERCE"
    project_name = "proj-0815"
    stage        = "TST"
    region       = "us10"
  }
}

With that we can now check for the state of the subaccount via a corresponding assert block:



variables {
  entitlements = []
}

run "test_successful_subaccount_creation" {
  command = apply

  variables {
    costcenter   = "1234567890"
    org_name     = "ECOMMERCE"
    project_name = "proj-0815"
    stage        = "TST"
    region       = "us10"
  }

  assert {
    condition     = resource.btp_subaccount.project.state == "OK"
    error_message = "The subaccount was not created in the expected state."
  }
}

Now we are good to go. In contrast to the unit test we now must provide valid values for the SAP BTP global account. We do so via the command line option -var. In addition, we must authenticate against the provider, which we will do by setting the corresponding environment variables for BTP_USERNAME and BTP_PASSWORD.

The command to execute the integration test finally looks like this:



terraform test -test-directory=integration-tests -var="globalaccount=yourgasubdomain"

The execution ow takes a bit longer as the subaccount is created on the SAP BTP. The final result should look like this:

The Terraform testing framework will try to remove all resources it created. Nevertheless things could go wrong and some resources might not get destroyed in the teardown of the test. This is clearly displayed in the output of the test, so you closely watch this especially when running the tests in the background via a CI/CD pipeline.

Where is the code?

You find the code of the previous sections in the GitHub repository Terraform Provider for SAP BTP - Quality Aspects.

Conclusion

In this blog post we learned that we can take the "as-Code" part of the "Infrastructure-as-Code" term seriously. We have seen several options how to safeguard the quality of our Terraform scripts by:

Checking for proper formatting
Validating the syntax of the scripts
Adding Unit and Integration tests leveraging the Terraform testing framework

As cherry on top we can automate this in your CI/CD pipeline. The Terraform commands have been created with automation in mind and can return JSON objects that can be used for further action like automatic issue creation.

The terraform test command and the options of mocking resources and data sources enable a lot more than we have tried out here in this blog post. I highly recommend to take a closer look at the documentation and the blog post referenced before and play around with them. Be aware that this is a quite "young" functionality, so maybe you stumble over issues or might miss some features. If this is the case you definitely should open an issue in the corresponding repository.

With that ... happy Terraforming and Testing!

Giving Kyma a little spin ... a SpinKube

Christian Lechner — Tue, 09 Apr 2024 07:15:59 +0000

Introduction

A few weeks ago I stumbled across a webinar about SpinKube that was hosted by and showed a presentation how Zeiss IT validated the project based on their requirements and the promises that come along with the project ("How ZEISS use Wasm in Enterprise - A special livestream").

I was impressed and the talk made me curious as the promise that the project "streamlines developing, deploying and operating WebAssembly workloads in Kubernetes - resulting in delivering smaller, more portable applications and incredible compute performance benefits." is quite appealing.

Although I had the project on my radar, I never took a closer look at it. However, now it was the time to give it a spin ... or try. All I needed was a Kubernetes environment. As I am working in the SAP space the choice of Kubernetes is kind of predefined, namely Kyma in its managed version on the SAP Business Technology Platform.

Making Kyma ready for SpinKube

In general, Kyma provides an opinionated but modular stack of Kubernetes resources that should support especially the SAP ecosystem to build extension and applications. It also allows to bring in own components and this way giving the option to tailor the stack according to the needs of your enterprise. In general, bringing the Spin operator into the stack shouldn't be an issue, but as always: the devil lies in the details, so I wanted to try it out.

In addition, I wanted to check if the Kyma environment on an SAP BTP trial poses some additional constraints on this undertaking, so I decided to use a trial environment for my experiment. Getting a trial environment is straightforward and you find some information about that in the documentation.

Enabling Kyma

To get started you must first manually enable Kyma in your SAP BTP subaccount. The provisioning takes some time. After that you can access the Kyma console via the URL that is provided in the subaccount overview as well as download the kubeconfig file that you need to access the cluster via kubectl. To do so you must have the kubectl installed on your machine. However, this is not sufficient to access the cluster.

Authenticating with Kyma is a (in my opinion) unnecessary challenge as it leverages the OIDC-login plugin for kubectl. You find a description of the setup here. This works fine when on a Mac but can give you some headaches on a Windows and on Linux machine especially when combined with restrictive setups in corporate environments. For Windows I can only recommend installing krew via chocolatey and then install the OIDC plugin via kubectl krew install oidc-login. At least for me that was the only way to get this working on Windows.

So, we do one extra hop and create a service account that we can use to authenticate with the Kyma cluster.

Creating a service account

To get rid of the OIDC flow and the browser-based login, the next step I did was to create a service account and a kubeconfig that contains the necessary roles. The procedure in general is described in a developer tutorial here.

Quite some adjustments are necessary as you need access a lot more resources, API groups and verbs to get the Spin operator and its dependencies up and running.

Let's quickly walk through the steps:

Create a new namespace for the service account. I created a srvc-account namespace
Create the necessary resources (ClusterRole, ClusterRoleBinding, ServiceAccount) in the srvc-account namespace. This was the fun part as I had to jump from error message stating which role I was missing when setting up SpinKube and its dependencies. This is something you can avoid by using this yaml file:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: spinkube-service-account
---
apiVersion: v1
kind: Secret
metadata:
  name: spinkube-service-account
  annotations:
    kubernetes.io/service-account.name: spinkube-service-account
type: kubernetes.io/service-account-token
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: spinkube-role
rules:
  - nonResourceURLs: 
      - /metrics
    verbs:
      - get
  - apiGroups:
      - ""
      - extensions
      - batch
      - apps
      - gateway.kyma-project.io
      - servicecatalog.k8s.io
      - apiextensions.k8s.io
      - node.k8s.io
      - rbac.authorization.k8s.io
      - admissionregistration.k8s.io
      - coordination.k8s.io
      - cert-manager.io
      - acme.cert-manager.io
      - route.openshift.io
      - certificates.k8s.io
      - authorization.k8s.io
      - gateway.networking.k8s.io
      - networking.k8s.io
      - apiregistration.k8s.io
      - authentication.k8s.io
      - apps
      - core.spinoperator.dev
    resources:
      - deployments
      - replicasets
      - pods
      - pods/log
      - pods/portforward
      - jobs
      - configmaps
      - apirules
      - serviceinstances
      - servicebindings
      - services
      - secrets
      - customresourcedefinitions
      - runtimeclasses
      - serviceaccounts
      - clusterroles
      - clusterrolebindings
      - roles
      - rolebindings
      - mutatingwebhookconfigurations
      - validatingwebhookconfigurations
      - namespaces
      - leases
      - secrets
      - issuers
      - issuers/status
      - certificates
      - certificates/finalizers
      - certificates/status
      - certificaterequests
      - certificaterequests/finalizers
      - certificaterequests/status
      - signers
      - clusterissuers
      - clusterissuers/status
      - events
      - challenges
      - challenges/finalizers
      - challenges/status
      - orders
      - orders/finalizers
      - orders/status
      - routes/custom-host
      - certificatesigningrequests
      - certificatesigningrequests/status
      - subjectaccessreviews
      - gateways
      - gateways/finalizers
      - httproutes
      - httproutes/finalizers
      - ingresses
      - ingresses/finalizers
      - apiservices
      - nodes
      - tokenreviews
      - deployments/status
      - spinappexecutors
      - spinappexecutors/status
      - spinappexecutors/finalizers
      - spinapps
      - spinapps/status
      - spinapps/finalizers
    verbs:
      - create
      - update
      - patch
      - delete
      - get
      - list
      - watch
      - deletecollection
      - approve
      - sign
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: spinkube-role-binding
subjects:
  - kind: ServiceAccount
    name: spinkube-service-account
    namespace: srvc-account
roleRef:
  kind: ClusterRole
  name: spinkube-role
  apiGroup: rbac.authorization.k8s.io
---

Create the kubeconfig file that you can use to access the cluster with the service account. Depending on your OS you can create a script that fetches the token from the secret and creates the kubeconfig file. You find the code of the script in step 4 of the tutorial.

With this we have the basic setup to provision SpinKube into the Kyma cluster.

Note - You find all the resources and code snippets in GitHub repository here.

Installing SpinKube

The SpinKube documentation provides tutorials to setup SpinKube locally as well as on clusters. There is one specific tutorial for Azure Kubernetes Service (AKS) that I used as a reference. Besides the first Azure specific section the SpinKube specific steps are the same for any Kubernetes cluster. Fortunately, Kyma makes no difference here, so I could execute the different steps without any issues.

From a tooling perspective you need to have kubectl and helm installed. I created a devcontainer with these tools installed.

To give you a rough idea what will be deployed into the Kyma cluster, here is a list of the components that are part of the SpinKube setup:

Deploy the Spin Operator CRD and the runtime class link
Install the cert manager to automatically provision and manage TLS certificates.
Install the kwasm operator via helm
Install the spin-operator via helm
Deploy the shim executor for the Spin application.

Once you are done the installed components are reflected in the new namespaces:

Time to check if the setup is working.

Deploy the first application

The team behind SpinKube provides some sample apps in their spin operator repository that you can use to test the setup and deploy a simple hello world app. Make sure that you reference the raw file from GitHub via:

kubectl apply -f https://raw.githubusercontent.com/spinkube/spin-operator/main/config/samples/simple.yaml

First we check that the service is available:

Looks good, so after a port-forward via:

kubectl port-forward services/simple-spinapp 8080:80

you can access the sample app via curl:

So full success, SpinKube is up and running in a Kyma environment.

But why ...?

First of all, I wanted to see if there are technical obstacles that the Kyma environment might have that prevent the installation of SpinKube. As you can see there are none even on a trial landscape.

Besides the technical feasibility the question that remains is: Why would you want to use SpinKube in a Kyma environment? The answer is SpinKube's value proposition mentioned in the beginning: a tool to deploy WebAssembly modules into a Kubernetes environment which can be useful in various scenarios, especially when you want to deploy small, portable, and fast applications. This makes it possible to optimize the utilization of your cluster and/or use smaller clusters for the same workload.

Of course, the full prove is still not given, as I just deployed a sample app, but the first step is done. Also keep in mind that the SpinKube project is not yet recommended for productive usage, but I think it is a very good point in time to make your first evaluations with it.

SAP BTP, Terraform and Open Policy Agent

Christian Lechner — Tue, 02 Apr 2024 06:32:23 +0000

The Challenge

When demoing the Terraform Provider for SAP BTP and talking about the advantages of Infrastructure as Code, one important aspect is that Terraform does not only provision the infrastructure but covers the management of the resources like updating configurations. While this is technically straightforward namely you update and apply the configuration, there are some challenges when doing this in a CI/CD setup.

Let us assume that we have a perfect setup in place: the configurations are stored in a source code management system, and we leverage the a pull-based workflow in the code repository. After a successful review and merge, we have the new configuration in the main branch of the repository and will apply it to the target environment. The application via a CI/CD pipeline from a technical perspective is easy. However, what if we missed a glitch in the new configuration and unfortunately some central resources are getting deleted and created again? Let's say for example an SAP HANA Cloud instance. Oopsie! This would not be received well, I guess. Or rephrasing that in a more graphical way:

How can we handle this? Are there any mechanisms to prevent or at least to some extent safeguard this kind of issues without falling back to a manual workflow? There is. One huge advantage of sticking to (de-facto) standards like Terraform is that first we are probably not the first ones to come up with this question and second there is a huge ecosystem around Terraform that might help us with such challenges.
And for this specific scenario the solution is the Open Policy Agent. Let us take a closer look how the solution could look like.

Note This topic is not specific to the SAP BTP but is an open topic for Terraform in general. While this blog post focuses on SAP BTP, you can easily exchange the examples with other Terraform providers.

The Solution

First some words about the Open Policy Agent. The Open Policy Agent (OPA) is a general-purpose policy engine that enables policy enforcement agnostic of the stack you are using. OPA provides a query language called Rego. Citing the official documentation "[...]Rego queries are assertions on data stored in OPA. These queries can be used to define policies that enumerate instances of data that violate the expected state of the system. [...]"

That sounds like what can help us. Thinking about our Terraform scenario it would be great if we could evaluate the terraform plan result of the new configuration and check if some unexpected things are happening. If some limits that we define in the policy are exceeded (like number of deleted resources of type XYZ), we would stop the automatic processing via the CI/CD pipeline and let a human look at the setup.

Luckily this scenario is also covered by a tutorial on the OPA website that you can find here. The tutorial deals with AWS as an example but making the adjustments for SAP BTP are straightforward. In the following section we will walk through the different components of the solution and see how it works in action.

The Components

You find the complete code for this example in the GitHub repository https://github.com/btp-automation-scenarios/btp-terraform-opa. In the following sections take a closer look at the code of the different solution components.

The Repository

For this showcase we will use a repository on GitHub to store our code (Terraform as well as the OPA Policy). The layout for the repository is as shown in the screenshot:

We have a infra folder that contains our Teraform configuration. The policy folder contains the OPA policy. And the .github folder i.e. the workflows folder therein contains the GitHub Actions configuration for the OPA check run.

In addition, we will use GitHub Actions to run the CI/CD pipeline. We will look at the configuration in the corresponding section.

The Terraform configuration

The basis for the setup is a Terraform configuration that you find in the infra folder (you find the complete code here). We do a very simple setup defined in the main.tf file:

###
# Setup of names in accordance with the company's naming conventions
###
locals {
  project_subaccount_name   = "${var.org_name} | ${var.project_name}: CF - ${var.stage}"
  project_subaccount_domain = lower(replace("${var.org_name}-${var.project_name}-${var.stage}", " ", "-"))
  project_subaccount_cf_org = replace("${var.org_name}_${lower(var.project_name)}-${lower(var.stage)}", " ", "_")
}

###
# Creation of subaccount
###
resource "btp_subaccount" "project" {
  name      = local.project_subaccount_name
  subdomain = local.project_subaccount_domain
  region    = lower(var.region)
  labels = {
    "stage"      = ["${var.stage}"],
    "costcenter" = ["${var.costcenter}"]
  }
  usage = "NOT_USED_FOR_PRODUCTION"
}

###
# Assignment of entitlements
###
resource "btp_subaccount_entitlement" "entitlements" {
  for_each = {
    for index, entitlement in var.entitlements :
    index => entitlement
  }

  subaccount_id = btp_subaccount.project.id
  service_name  = each.value.name
  plan_name     = each.value.plan
}

One subaccount and a list of entitlements are created. The entitlements are defined in the variables.tf file:

variable "entitlements" {
  type = list(object({
    name   = string
    plan   = string
    amount = number
  }))
  description = "List of entitlements for the subaccount."
  default = [
    {
      name   = "alert-notification"
      plan   = "standard"
      amount = null
    },
    {
      name   = "SAPLaunchpad"
      plan   = "standard"
      amount = null
    },
    {
      name   = "hana-cloud"
      plan   = "hana"
      amount = null
    },
    {
      name   = "hana"
      plan   = "hdi-shared"
      amount = null
    },
    {
      name   = "sapappstudio"
      plan   = "standard-edition"
      amount = null
    }
  ]
}

Not a very complex setup, but enough to show the concept. For the sake of this example, we do not even create any resources but will execute the check in the initial setup. Let's move on to the heart of the solution, the OPA policy.

Definition of the Policy

We define the policy in the folder policy as terrraform.rego. According to the rego language we define a package and add some basic import:

package terraform.analysis

import rego.v1

import input as tfplan

This the rego.v1 import is a specific opt-in described here. In addition, we define the import of the Terraform plan (result of the terraform plan command) as input for the further processing.

Next, we define some parameters for the policy:

# acceptable score for automated authorization
blast_radius := 30

# weights assigned for each operation on each resource-type
weights := {
    "btp_subaccount": {"delete": 100, "create": 10, "modify": 1},
    "btp_subaccount_entitlement": {"delete": 10, "create": 1, "modify": 5},
}

# Consider exactly these resource types in calculations
resource_types := {"btp_subaccount", "btp_subaccount_entitlement"}

We define a blast radius that represents an acceptable score for automated execution of the Terraform configuration. To be able to calculate the score we define weights for each Terraform operation and resource type.

Now we must calculate the score for the Terraform plan that we provided as input. For that we define some functions that help us to calculate the number of creations, deletions, and modifications for each resource type:

# list of all resources of a given type
resources[resource_type] := all if {
    some resource_type
    resource_types[resource_type]
    all := [name |
        name := tfplan.resource_changes[_]
        name.type == resource_type
    ]
}

# number of creations of resources of a given type
num_creates[resource_type] := num if {
    some resource_type
    resource_types[resource_type]
    all := resources[resource_type]
    creates := [res | res := all[_]; res.change.actions[_] == "create"]
    num := count(creates)
}

# number of deletions of resources of a given type
num_deletes[resource_type] := num if {
    some resource_type
    resource_types[resource_type]
    all := resources[resource_type]
    deletions := [res | res := all[_]; res.change.actions[_] == "delete"]
    num := count(deletions)
}

# number of modifications to resources of a given type
num_modifies[resource_type] := num if {
    some resource_type
    resource_types[resource_type]
    all := resources[resource_type]
    modifies := [res | res := all[_]; res.change.actions[_] == "update"]
    num := count(modifies)
}

There is quite some rego magic going on that you find in the official documentation. As you see we are using the tfplan input to extract the resources and the operations of the Terraform plan to identify the number of create, update, and delete actions.

Finally, we define the policy itself:

# Authorization holds if score for the plan is acceptable, and no changes are made to IAM
default autoexec := false

autoexec if {
    score < blast_radius
}

# Compute the score for a Terraform plan as the weighted sum of deletions, creations, modifications
score := s if {
    all := [x |
        some resource_type
        crud := weights[resource_type]
        del := crud.delete * num_deletes[resource_type]
        new := crud.create * num_creates[resource_type]
        mod := crud.modify * num_modifies[resource_type]
        x := (del + new) + mod
    ]
    s := sum(all)
}

This snippet calculates the overall score and checks if the score is below the defined blast_radius. If the score is below the threshold the autoexec "decision" is set to true and the Terraform plan could be executed automatically. We also have the score available as a "decision" when evaluating the policy.

You find the code of the policy here.

Integration with the CI/CD workflow

We must bring the bits and pieces together. We use a GitHub Actions workflow to run the OPA check. The configuration is as follows:

name: Evaluate Open Policy Agent for Terraform

on:
  workflow_dispatch:
    inputs:
      PROJECT_NAME:
        description: "Name of the project"
        required: true
        default: "sample-proj-opa"
      REGION:
        description: "Region for the sub account"
        required: true
        default: "eu10"
      COST_CENTER:
        description: "Cost center for the project"
        required: true
        default: "1234567890"
      STAGE:
        description: "Stage for the project"
        required: true
        default: "DEV"
      ORGANIZATION:
        description: "Organization for the project"
        required: true
        default: "B2B"

env:
  PATH_TO_TFSCRIPT: 'infra'

jobs:
  execute_base_setuup:
    name: BTP Subaccount Setup
    runs-on: ubuntu-latest
    steps:
    - name: Check out Git repository
      id: checkout_repo
      uses: actions/checkout@v4

    - name: Setup Terraform
      id : setup_terraform
      uses: hashicorp/setup-terraform@v3
      with:
        terraform_wrapper: false
        terraform_version: latest

    - name: Setup Open Policy Agent
      id: setup_opa
      uses: open-policy-agent/setup-opa@v2
      with:
        version: latest

    - name: Terraform Init
      id: terraform_init
      shell: bash
      run: |
        terraform -chdir=${{ env.PATH_TO_TFSCRIPT }} init -no-color

    - name: Terraform plan 
      id: terraform_plan
      shell: bash
      run: |
        export BTP_USERNAME=${{ secrets.BTP_USERNAME }}
        export BTP_PASSWORD=${{ secrets.BTP_PASSWORD }}
        terraform -chdir=${{ env.PATH_TO_TFSCRIPT }} plan -var globalaccount=${{ secrets.GLOBALACCOUNT }} -var region=${{ github.event.inputs.REGION }} -var project_name=${{ github.event.inputs.PROJECT_NAME }} -var stage=${{ github.event.inputs.STAGE }} -var costcenter=${{ github.event.inputs.COST_CENTER }} -var org_name=${{ github.event.inputs.ORGANIZATION }} -no-color --out tfplan.binary
        terraform -chdir=${{ env.PATH_TO_TFSCRIPT }} show -json tfplan.binary > tfplan.json

    - name: Execute OPA policy
      id: execute_opa
      shell: bash
      run: |
        autoexec=$(opa exec --decision terraform/analysis/autoexec --bundle policy/ tfplan.json | jq '.result[].result')
        score=$(opa exec --decision terraform/analysis/score --bundle policy/ tfplan.json | jq '.result[].result')
        echo "Automatic execution possible (true/false): ${autoexec}"
        echo "Score of change: ${score}"

For the sake of the demo, we use the workflow_dispatch event to trigger the workflow. The user can provide input parameters for the Terraform configuration. We use the predefined Actions hashicorp/setup-terraform@v3 and open-policy-agent/setup-opa@v2 to setup Terraform as well as OPA. Make sure to set the terraform_wrapper to false in the Terraform setup to be able to use the terraform command directly.

After initializing the Terraform configuration via terraform init we run the terraform plan command and store the plan via the -out parameter as tfplan.binary. OPA would not be able to work with the plan in this format, so we must convert it to JSON. We do this via the terraform show -json command and store the plan as tfplan.json.

Now we set the stage for the OPA policy execution. We use the opa exec command to run the policy and provide the Terraform plan as input. To be specific we execute two decisions namely the result of the autoexec and the score. The opa exec would return a JSON object. For the sake of readability, we parse the result via jq and print the result to the console.

That's it. What will the result be? Let's see it in action.

Let's see it in action

The execution of the GitHub Action acting looks like this:

So, the initial application of the configuration would be executed automatically as the score is 15 and below the defined threshold of 30. Let us validate if this is correct:

We have the creation of the subaccount which counts as 10 points. In addition, we create 5 entitlements which count as 5 points. The total score is 15. The code is working as expected. Great!

The Conclusion

Terraform is a great tool for provisioning and managing your infrastructure. Automating the corresponding processes is technically quite easy but the management aspects need some more aspects to be considered. The Open Policy Agent comes in quite handy here as a general-purpose policy engine that can help with overcoming challenges in the automation of the setup. The integration of the tools is quite straightforward as the example of the blog post shows. Not being an expert in the rego language it is obviously quite powerful, but another topic to learn; and I am honest, I was quite happy to have a running example that I could take from the tutorials area of OPA and just make minor adjustments.

While the sample I presented here is very simple, it clearly shows how to deal with the challenge of automating your automated infrastructure management.

With that ... happy Terraforming!

Terraform Provider for SAP BTP - Remote State and Drift Detection

Christian Lechner — Tue, 26 Mar 2024 08:53:33 +0000

Introduction

In a recent DSAG (German-speaking SAP User Group) event ("DSAG Betriebstage" - can be translated as Ops Days) I had the opportunity to give a talk about Infrastructure as Code (IaC) and how it can help you with governance topics on the SAP Business Technology Platform (BTP).

One important topic in this context was of course the automated detection of configuration drifts. The scenario basically was that an SAP BTP subaccount was (perfectly) set up using Terraform following all company-specific guidelines. However, a user with the necessary permissions changed the configuration manually in the SAP BTP cockpit. This led to a deviation between the automated configuration and the actual state of the subaccount. This is what we call a configuration drift. How can we find out about this drift?

While I showed how this can be detected via Terraform, we did not go into the technical details of the process. To make this theoretical description more tangible we will take a closer look on how we can achieve this from a technical perspective.

Prerequisites

To make the following examples work I leveraged the following tools and services:

A GitHub repository for storing the Terraform configuration
An Azure Blob Storage for centrally storing the Terraform state
GitHub Actions to automate the Terraform workflow especially the drift detection

You can of course also use different backends like AWS S3. You find a complete list of supported backends in the Terraform documentation. The same is true for the CI/CD pipeline. You can use e.g., GitLab CI/CD or any other CI/CD tool you prefer to enable the automated flow.

The GitHub Repository

I created a GitHub repository that you find at https://github.com/btp-automation-scenarios/btp-terraform-drift. This repository contains the Terraform configuration for an SAP BTP subaccount.

The configuration is very basic and only creates a subaccount with a few entitlements to show the drift detection flow. The infrastructure configuration is stored in the infra directory of the repository.

The sensitive information for the setup like username and password is stored in GitHub Secrets.

Setup for Remote State

The detection of a configuration drift relies on the storage of the Terraform state. This state file contains the configuration of the infrastructure and is used by Terraform to determine what must be changed in the infrastructure.

In a real-life setup this state is stored centrally to make it available to the team of BTP administrators. This can be an Azure Blob Storage, AWS S3 or any other supported backend. One important point here is that the state needs to be encrypted as it might contains sensitive information. In this blog post we will use an Azure Blob Storage.

The first step is to create such a storage on Microsoft Azure. The Microsoft documentation provides a very good guide on how to setup such a storage account for Terraform state. You find the documentation here.

For the sake of simplicity, we will shortly walk through the steps using the Azure CLI:

Create a resource group on Azure:

   az group create --name rg_terraform_state_sapbtp --location westeurope

Create a storage account in the resource group:

   az storage account create --resource-group rg_terraform_state_sapbtp --name sasapbtptfstate --sku Standard_LRS --encryption-services blob

Create a blob container in the storage account:

   az storage container create --name tfstate --account-name sasapbtptfstate

This results in the following setup for the storage account:

and the blob container (that already contains the state file):

You can also do the setup via the Azure Portal or Terraform.

Be aware that:

Azure storage comes with automatic encryption. You can also use customer managed keys for that.
Azure state storage blobs are automatically locked before any operation that writes to the storage account. This prevents concurrent writes to the state file. Details are described in the documentation.

Next, we must make Terraform aware of the fact that should store the state in the Azure Blob Storage. This is done by the following configuration to the provider.tf file:

terraform {
  required_providers {
    btp = {
      source  = "sap/btp"
      version = "~>1.1.0"
    }
  }
  backend "azurerm" {
    resource_group_name  = "rg_terraform_state_sapbtp"
    storage_account_name = "sasapbtptfstate"
    container_name       = "tfstate"
    key                  = "dev.terraform.tfstate"
  }
}

The relevant block is the backend block. We use a predefined backend provided by Terraform for Azure. The resource_group_name, storage_account_name and container_name are the values we used to define the Azure Blob Storage. The key is the name of the state file.

Next, we define the workflows to create and destroy the subaccount.

GitHub Actions for the Setup

To execute the creation and the deletion via GitHub Actions I added two workflows to the repository:

.github/workflows/setup-subaccount.yml: this workflow executed the setup of the subaccount in the SAP BTP based on some input variables. The configuration is given by:

name: Basis Subaccount via Terraform

on:
  workflow_dispatch:
    inputs:
      PROJECT_NAME:
        description: "Name of the project"
        required: true
        default: "sample-proj-drift"
      REGION:
        description: "Region for the sub account"
        required: true
        default: "eu10"
      COST_CENTER:
        description: "Cost center for the project"
        required: true
        default: "1234567890"
      STAGE:
        description: "Stage for the project"
        required: true
        default: "DEV"
      ORGANIZATION:
        description: "Organization for the project"
        required: true
        default: "B2B"

env:
  PATH_TO_TFSCRIPT: 'infra'

jobs:
  execute_base_setuup:
    name: BTP Subaccount Setup
    runs-on: ubuntu-latest
    steps:
    - name: Check out Git repository
      id: checkout_repo
      uses: actions/checkout@v4

    - name: Setup Terraform
      id : setup_terraform
      uses: hashicorp/setup-terraform@v3
      with:
        terraform_wrapper: false
        terraform_version: latest

    - name: Terraform Init
      id: terraform_init
      shell: bash
      run: |
        export export ARM_ACCESS_KEY=${{ secrets.ARM_ACCESS_KEY }}
        terraform -chdir=${{ env.PATH_TO_TFSCRIPT }} init -no-color

    - name: Terraform Apply 
      id: terraform_apply
      shell: bash
      run: |
        export ARM_ACCESS_KEY=${{ secrets.ARM_ACCESS_KEY }}
        export BTP_USERNAME=${{ secrets.BTP_USERNAME }}
        export BTP_PASSWORD=${{ secrets.BTP_PASSWORD }}
        terraform -chdir=${{ env.PATH_TO_TFSCRIPT }} apply -var globalaccount=${{ secrets.GLOBALACCOUNT }} -var region=${{ github.event.inputs.REGION }} -var project_name=${{ github.event.inputs.PROJECT_NAME }} -var stage=${{ github.event.inputs.STAGE }} -var costcenter=${{ github.event.inputs.COST_CENTER }} -var org_name=${{ github.event.inputs.ORGANIZATION }} -auto-approve -no-color

.github/workflows/destroy-subaccount.yml: this workflow destroys the subaccount in the SAP BTP. The configuration is given by:

name: Destroy Subaccount via Terraform

on:
  workflow_dispatch:

env:
  PATH_TO_TFSCRIPT: 'infra'

jobs:
  execute_base_setuup:
    name: BTP Subaccount Deletion
    runs-on: ubuntu-latest
    steps:
    - name: Check out Git repository
      id: checkout_repo
      uses: actions/checkout@v4

    - name: Setup Terraform
      id : setup_terraform
      uses: hashicorp/setup-terraform@v3
      with:
        terraform_wrapper: false
        terraform_version: latest

    - name: Terraform Init
      id: terraform_init
      shell: bash
      run: |
        export export ARM_ACCESS_KEY=${{ secrets.ARM_ACCESS_KEY }}
        terraform -chdir=${{ env.PATH_TO_TFSCRIPT }} init -no-color

    - name: Terraform Apply 
      id: terraform_apply
      shell: bash
      run: |
        export ARM_ACCESS_KEY=${{ secrets.ARM_ACCESS_KEY }}
        export BTP_USERNAME=${{ secrets.BTP_USERNAME }}
        export BTP_PASSWORD=${{ secrets.BTP_PASSWORD }}
        terraform -chdir=${{ env.PATH_TO_TFSCRIPT }} destroy -var globalaccount=${{ secrets.GLOBALACCOUNT }} -auto-approve -no-color

There is one important thing to note here. The Terraform CLI needs to authenticate not only to SAP BTP but also to Azure to access the Blob storage. I used the storage access key for that. This key is stored in the GitHub Secrets along with the other sensitive information. To make the access key available to Terraform we must export it in the Terraform steps as ARM_ACCESS_KEY.

With these GitHub Actions we can setup and destroy the subaccount in the SAP BTP including a central storage of the Terraform state in the Azure Blob Storage. The next step is to automate the detection of a configuration drift.

Drift Detection

The detection of a configuration drift is done by comparing the actual state of the subaccount in the SAP BTP with the state defined in the Terraform configuration. This sounds complicated, but indeed it is quite easy.

The only thing we need to do is to execute a terraform plan inside of a GitHub Action. If the stored state matches the configuration on SAP BTP the planning will basically state "nothing to do". In case there is a deviation the planning will come up with changes that it would like to apply to bring the state and the configuration on SAP BTP back in sync. We use this to find out if any unwanted changes have been made.

To make things easier for the handling in a CI/CD flow we provide an additional parameter to the terraform plan command namely the -detailed-exitcode. This parameter tells the CLI to provide more granular information about what the resulting plan via the exit code:

0 = Succeeded with empty diff, so no drift detected
1 = Error - this should not happen, but at least good to know that things went completely of the rails
2 = Succeeded with non-empty diff, so changes are present, and a drift is detected.

With this we create a new workflow for the drift detection:

name: Check for Subaccount Drift via Terraform

on:
  workflow_dispatch:
    inputs:
      PROJECT_NAME:
        description: "Name of the project"
        required: true
        default: "sample-proj-drift"
      REGION:
        description: "Region for the sub account"
        required: true
        default: "eu10"
      COST_CENTER:
        description: "Cost center for the project"
        required: true
        default: "1234567890"
      STAGE:
        description: "Stage for the project"
        required: true
        default: "DEV"
      ORGANIZATION:
        description: "Organization for the project"
        required: true
        default: "B2B"

env:
  PATH_TO_TFSCRIPT: 'infra'

jobs:
  execute_base_setuup:
    name: BTP Subaccount Drift Check
    runs-on: ubuntu-latest
    steps:
    - name: Check out Git repository
      id: checkout_repo
      uses: actions/checkout@v4

    - name: Setup Terraform
      id : setup_terraform
      uses: hashicorp/setup-terraform@v3
      with:
        terraform_wrapper: false
        terraform_version: latest

    - name: Terraform Init
      id: terraform_init
      shell: bash
      run: |
        export export ARM_ACCESS_KEY=${{ secrets.ARM_ACCESS_KEY }}
        terraform -chdir=${{ env.PATH_TO_TFSCRIPT }} init -no-color

    - name: Terraform Plan 
      id: terraform_plan
      shell: bash
      continue-on-error: true
      run: |
        export ARM_ACCESS_KEY=${{ secrets.ARM_ACCESS_KEY }}
        export BTP_USERNAME=${{ secrets.BTP_USERNAME }}
        export BTP_PASSWORD=${{ secrets.BTP_PASSWORD }}
        terraform -chdir=${{ env.PATH_TO_TFSCRIPT }} plan -var globalaccount=${{ secrets.GLOBALACCOUNT }} -var region=${{ github.event.inputs.REGION }} -var project_name=${{ github.event.inputs.PROJECT_NAME }} -var stage=${{ github.event.inputs.STAGE }} -var costcenter=${{ github.event.inputs.COST_CENTER }} -var org_name=${{ github.event.inputs.ORGANIZATION }} -no-color -detailed-exitcode

    - name: Create issue
      if: steps.terraform_plan.outcome == 'failure'  
      uses: actions/github-script@v7
      env:
        PROJECT_NAME: ${{ github.event.inputs.PROJECT_NAME }}
        REGION: ${{ github.event.inputs.REGION }}
        STAGE: ${{ github.event.inputs.STAGE }}
        COST_CENTER: ${{ github.event.inputs.COST_CENTER }}
        RUN_ID : ${{ github.run_id }}
      with:
        script: |
          const issueTitle = `Configuration Drift Detected for ${process.env.PROJECT_NAME}`
          const issueBody = `A drift has been detected for ${process.env.PROJECT_NAME} in ${process.env.REGION} region. Stage is ${process.env.STAGE} and cost center is ${process.env.COST_CENTER}. Find more information in the run https://github.com/btp-automation-scenarios/btp-terraform-drift/actions/runs/${process.env.RUN_ID}`

          github.rest.issues.create({
            owner: context.repo.owner,
            repo: context.repo.repo,
            labels: [
              'automated issue', 'drift detected'
            ],
            title: issueTitle,
            body: issueBody
          })

    - name: State deviation - Set run to failed
      if: steps.terraform_plan.outcome == 'failure'
      uses: actions/github-script@v7
      with:
        script: |
            core.setFailed('A configuration drift was detected!')

The main part of the workflow is the step with the id terraform plan that executes the planning and provides us the exit code that we use to check if a drift is detected. As follow-up actions we:

Create an issue in the GitHub repository to inform the team about the drift
Set the run to failed to also make it visible in the GitHub Actions overview

The step for the creation is defined as:

  - name: Create issue
      if: steps.terraform_plan.outcome == 'failure'  
      uses: actions/github-script@v7
      env:
        PROJECT_NAME: ${{ github.event.inputs.PROJECT_NAME }}
        REGION: ${{ github.event.inputs.REGION }}
        STAGE: ${{ github.event.inputs.STAGE }}
        COST_CENTER: ${{ github.event.inputs.COST_CENTER }}
        RUN_ID : ${{ github.run_id }}
      with:
        script: |
          const issueTitle = `Configuration Drift Detected for ${process.env.PROJECT_NAME}`
          const issueBody = `A drift has been detected for ${process.env.PROJECT_NAME} in ${process.env.REGION} region. Stage is ${process.env.STAGE} and cost center is ${process.env.COST_CENTER}. Find more information in the run https://github.com/btp-automation-scenarios/btp-terraform-drift/actions/runs/${process.env.RUN_ID}`

          github.rest.issues.create({
            owner: context.repo.owner,
            repo: context.repo.repo,
            labels: [
              'automated issue', 'drift detected'
            ],
            title: issueTitle,
            body: issueBody
          })

This step is executed only if the plan state failed which we ensure via the if condition if: steps.terraform_plan.outcome == 'failure'.

We use the GitHub REST API to create an issue with the corresponding labels. As an example, we provide some additional information, but you can of course also provide more detailed information about the detected drift.

Make sure that within the settings of the repository (or the organization) the GitHub Actions are allowed to create issues by enabling the "Read and write permissions" for GitHub Actions:

In the current state the workflow would be reported as "successful". I personally prefer that the run is marked as failed in case of a drift. This must be done explicitly by the following step:

    - name: State deviation - Set run to failed
      if: steps.terraform_plan.outcome == 'failure'
      uses: actions/github-script@v7
      with:
        script: |
            core.setFailed('A configuration drift was detected!')

With this we have an automated drift detection in place that informs the team about any deviations between the actual state of the subaccount in the SAP BTP and the configuration defined in the Terraform configuration.

Once we introduce a drift, we will see the following in the GitHub Actions overview:

As we can see the exit code is as expected a "2" indicating that changes would be applied.

And consequently, an issue is created in the GitHub repository:

Conclusion and Outlook

With only minor configurations the Terraform built-in functionalities of state storage and planning enable us to detect configuration drifts in the SAP BTP to automatically detect deviations of the configuration with respect to the company-specific guidelines. This ensures that configuration is not changed without the knowledge of the administrator team especially to ensure governance and compliance.

Now that we have detected the drift, how to deal with it? This usually cannot be handled automatically but needs to be analyzed in detail. You have basically two options that bring things back in sync:

Reconcile the state with the configuration in the Terraform configuration. This can be done by executing a terraform apply. This will apply the necessary changes to the infrastructure and bring it back in sync with the state.
The manual changes on the platform were okay (e.g., due to an emergency fix). In this case the state file must be updated to reflect the actual state. This can be done by executing a terraform apply and a terraform apply with the -refresh-only flag. This will update the state file with the actual state of the subaccount and not do any changes to the infrastructure.

Another aspect worth to discuss is the integration into existing processes especially on the SAP side of the house. We implemented the flow and the "handling" (namely the creation of a GitHub issue) in an isolated fashion. However, we also received the ask to integrate the flow into the SAP solutions that might already deal with such governance tasks like SAP Cloud ALM. based on that we have a feature request open (link) in the repository of the Terraform Provider for SAP BTP. If you would like to share your point of view or vote for it, this is the place to go.

With this, nothing more to say than: Happy Terraforming!