Forem: Sefi Genis

Terraform Modules for Advanced Users

Sefi Genis — Wed, 21 Dec 2022 14:59:27 +0000

Terraform (TF) Modules have become an increasingly popular and efficient way to configure a diversity of applications and infrastructure as code. They are a catalog of repeatable templates that help you deploy Terraform code quickly and consistently. The most popular tools, clouds, and platforms have a Terraform Module – some of which have been downloaded tens of millions of times.

The great part about Terraform Modules is that they’re quite flexible and extensible, enabling you to write your own custom modules for proprietary applications, as needed. There are some pretty great tutorials online that extensively cover the basics of getting started and writing your own Terraform Module, you can check out this post as a great reference for how to build your TF module.

When leveraging public-facing modules, like any other resources taken off the public web––whether open source tools and utilities or even container images, there are some good practices to keep in mind when selecting your module of choice, including:

Ensuring it is well-maintained, with good security hygiene (e.g. quick patches for high-severity vulnerabilities)
Maintains backwards compatibility even with upgrades, so that new versions don’t break anything
Has a good release cycle management of major / minor versions
Has a strong community - which is represented in stars, forks, and even how quickly issues are resolved in their repo

These can provide a good understanding of whether these modules are recommended for adoption in your stack, or whether they will cause you future heartache––so be sure to vet the modules you choose before you apply them to your code.

Advanced Use Cases with Terraform Modules

When it comes to creating our own modules, it’s best to start first with a design of your systems and decide the level of coverage you want for your modules. While modules are great for composability, collaboration and automation––like all code, each requires its own share of maintenance.

It is most certainly possible to create a module per component, whether it’s Lambda services, IAM users and roles, policies, and even backend and frontend components. However the decision to create a module should always be driven by the value and ROI in controlling the code being greater than the long-term maintenance and overhead of supporting the modules for perpetuity. If it’s easier to leverage a public resource, and customize it to your need, that may sometimes be sufficient, and is saves having to write the module from scratch.

There are also plenty of modules that bundle components together, such as infrastructure/runtime, a database like RDS and IAM roles. Anton Babenko’s AWS module library is just one example of a well-maintained library of resources you can leverage for nearly every AWS use case and component bundle.

There are also situations where modules are great provisioning tools, but when it comes to connecting multiple and diverse providers there may be additional complexity to take into consideration. Let’s take the use case of MongoDB Atlas on AWS with VPC Peering, in this case you would need to ensure that your modules are connected from both sides, MongoDB to AWS and AWS to MongoDB, to ensure they work properly.

‍The good part is that Terraform Modules have a great community and developer tools, so that even more advanced use cases are well-documented. That said, there is also the reverse scenario where leveraging modules can actually be the best practice, and prevent poor operations hygiene like infrastructure drift and other issues, but may sometimes be an afterthought.

Leveraging Modules for ClickOps Components

All of the examples above are great when we start by creating our resources as code in the cloud. But what happens when we have manually created resources in the cloud via ClickOps? This is a very common practice in operations, particularly with resources created pre-infrastructure as code. What if later we want to leverage a module, particularly a bundled module, that is much more aligned with best practices? How can this be done without causing breakage?

Let's take an example of creating an S3 bucket manually. In our scenario, this manually created resource will not have encryption, versioning, ACL (access lists), or anything else. Therefore, if we'd like to switch to using a TF Module that has these components, it won't match our current, manually created bucket's configuration. In such a scenario, will we now be stuck with our manually provisioned resource forever that has no encryption or versioning?

This is exactly the type of situation where we can create drift, if our code and resources are not aligned, which can cause volatility in our clouds. So what’s a good way to overcome this?

The first step is to actually convert our resource to code, and then upload it to the Terraform state file. Ok - good! But this is still a bare bones resource with none of the additional featuers such as versioning. The next thing you can do is import the module into your newly created Terraform code. This way you import all of the added configuration that you want to apply to your S3 bucket, without having to write it yourself.

The next time you run terraform apply it will prompt you with the many changes you’re making to your bucket, and you will just need to confirm that these are OK. Once the changes apply, your bucket will be upgraded with the additional functionality you wanted to apply to your resource.

That’s great for one-off resources created manually, but how do we do this at scale? What if we have hundreds of manually provisioned S3 buckets with no encryption, ACL or versioning? What now?

EUREKA! This is where a tool like Firefly comes in. It enables you to select all of the resources you’d like to update, select a publicly available module or even upload a custom module, and apply the changes to all of your resources that require upgrading / changing / modification at once.
‍
See it in action:

RUN:

terraform import module.terraform-aws-s3-bucket.aws_s3_bucket.main_bucket[0] BUCKET_NAME

You can see in the code the diff between your existing resource and the imported module:

module "terraform-aws-s3-bucket" {
 source = "github.com/terraform-aws-modules/terraform-aws-s3-bucket"

 object_lock_enabled = false
 bucket       = "<BUCKET_NAME>"
}

These resources are listed in the module but not in the dependencies 
(hence will be created in the next terraform apply): 
# aws_s3_bucket_logging
# aws_s3_bucket_acl
# aws_s3_bucket_website_configuration
# aws_s3_bucket_versioning
# aws_s3_bucket_server_side_encryption_configuration
# aws_s3_bucket_accelerate_configuration
# aws_s3_bucket_request_payment_configuration
# aws_s3_bucket_cors_configuration
# aws_s3_bucket_lifecycle_configuration
# aws_s3_bucket_object_lock_configuration
# aws_s3_bucket_replication_configuration
# aws_s3_bucket_policy
# aws_s3_bucket_public_access_block
# aws_s3_bucket_ownership_controls
# aws_s3_bucket_intelligent_tiering_configuration

Once you click apply (like in the example with the single resource), you can then update all of the selected resources with one single action. This method enables you to write, import, and govern your manually created resources at scale, with minimal pain.

TL;DR - Terraform Modules Made Easy

So just to wrap it up, Terraform Modules have changed the game for codifying resources, bundling services, components, and enabling easy cloud infrastructure governance in the long run. The framework makes it truly simple to connect more common and proprietary resources through publicly available and maintained modules alongside custom-built modules (that take some research, but are eventually pretty easy to create, configure and maintain). Terraform Modules are a great way to ensure your clouds are aligned with best practices, and they can still be leveraged for resources that were not created as code.

It is most certainly recommended to upgrade manually created resources to infrastructure as code, to gain all of the benefits derived from codified resources––security, policy and governance, automation, and extensibility. You can also leverage some great tools to make these transitions at scale, and not leave any resources unmanaged in your clouds.

How to Govern Terraform States Using GitLab Enterprise?

Sefi Genis — Thu, 04 Aug 2022 08:23:00 +0000

TL;DR: With the mass adoption of Terraform and becoming the de facto tool for developers to build, and manage their cloud infrastructure at scale, most companies today, who rely heavily on Terraform for their infrastructure management, choose to do so with an orchestration tool. In this blog, we'll review the way to govern Terraform States using Gitlab Enterprise

Terraform has become the nearly ubiquitous way to provision services in a cloud native era. However, when we start to build our infrastructure using Terraform’s as-code approach, there are a few things we need to consider in order to be able to manage these operations at scale, for a diversity of decentralized services, and for distributed teams. At Firefly, we often encounter the challenges of managing IaC at scale, as part of our effort to help organizations discover and manage their many cloud assets.
‍

Terraform Management at Scale

With the mass adoption of Terraform and becoming the de facto tool for developers to build, and manage their cloud infrastructure at scale, most companies today, who rely heavily on Terraform for their infrastructure management, choose to do so with an orchestration tool. These tools complement the suite of tools Hashicorp provides, to help get a handle on the many modules and providers, frameworks and services being provisioned with the sheer scale of cloud operations, as well as a remote backend to maintain the state for your infrastructure.
‍

The companies that choose a fully managed orchestration tool, oftentimes will select Hashicorp’s very own Terraform Cloud (SaaS solution), however like all tools that gain widespread popularity ––there are many non-Hashicorp alternatives to ride the wave of the tool’s success, as well. Terraform Cloud's benefits are a fully remote backend, native integration with GitHub, State versioning, and advanced features for infrastructure stakeholders, such as platform engineers, DevOps teams and cloud engineers.

‍
However, there is another option that is gaining popularity for large-scale Terraform operations, and that is the GitOps approach, ones who decide to deploy their infrastructure using GitHub Actions or other built-in CI pipelines applications. The most popular tool for this use case is Atlantis. Atlantis is a basic solution that integrates automatically with each pull request (PR) and enforces best practices for infrastructure deployments as they are defined in the company policies such as: the code owner, code reviewers, unit tests using tools like TerraTest, among others.
‍
When you choose the GitOps method, this will not come with the managed backend, and therefore this will still be required for those looking to maintain state for their IaC. Terraform currently supports out-of-the-box integration with: AWS S3, GCS, Hashicorp Consul, Kubernetes, and HTTP.

‍As we all know though, many companies today have chosen to work with Gitlab on-prem, for many reasons, and therefore all of the Github and Github Actions integrations become less relevant with this choice.

‍

Terraform States Using Gitlab Enterprise

Those companies who choose GitLab as their primary source code management (SCM) platform, will also many times choose to deploy their infrastructure using dedicated GitLab pipelines. This leaves us with the question: but what about the Terraform state?
‍
Introducing a new feature for remote backends inside Gitlab. We knew this was just what we needed as a Gitlab shop. However, when we came to try and enable it, we found very little documentation to help us…and so we had to go down the rabbit hole of researching how to configure and setup remote backends with specific requirements dictated in the Gitlab API, and we’d like to share with you some of the excellent intel we uncovered.
‍
We’ll start with some of the challenges we immediately encountered. Configuring the Gitlab backend proved itself quite complex, having to understand the Gitlab configuration syntax in-depth and the various S3 configurations to actually get this set up. Once we managed to configure our S3 bucket as the dedicated data store for our Terraform states, we found that these are all encrypted using AES 256 inside the S3 by Gitlab. What this means is that once encrypted, this state is no longer accessible inside Terraform. This requires you to use Gitlab APIs to download them and be able to use them in your environment.

‍
This is where it gets tricky. So we’ve chosen to deploy and orchestrate our code using Gitlab. Great. Next we want to leverage their new capability of managing state - but this means we can’t actually manage our Terraform State if they are encrypted and not accessible to Terraform.

‍
This adds a particular layer of complexity for this use case, because when you work in the modern engineering format of CI/CD, Gitlab will increase your version number with each deployment, to maintain the log and change history of deployed versions. All of this is fine, and important as an engineering best practice - however this introduces a few gaps when it comes to Terraform state management.
‍

If we take a look at the Gitlab API documentation the way to download the state is as follows:

curl --header "Private-Token: " "https://gitlab.example.com/api/v4/projects//terraform/state//versions/"
‍

This means that in order to be able to download the state you have to have a few critical pieces of information:

Your Access Token
Your Project ID
Your State Name
Your Version Number ‍

Not only does one rarely know the specific name of their deployment, it’s very rare to know the latest version number (Gitlab doesn’t expose this in the UI, only if you hover over the deployment or click on it will you see this number in the URL)––as this is constantly changing with continuous deployment.

‍
In very large scale operations, there are hundreds of environments running Terraform all the time, and news ones constantly being deployed. Not to mention different kinds of environments–– development, staging, production, with all of these having multiple dev accounts. It’s a needle in a haystack.

‍We felt like we hit a wall. We knew there had to be a better way. We went back to researching.
‍

Gitlab GraphQL API for Terraform State Management‍

After digging deeper, we found a gold mine. There IS another way.

‍We found a hidden GraphQL API that reveals all of your Gitlab environments built through GitLab Pipelines which enables you to extract quite simply all of the critical information you will need to be able to download and access the Terraform State.

‍See it in action - below is the GraphQL code snippet that enables you to query and extract the required data.

‍

POST https://{{GITLAB-HOST}}/api/graphql { "operationName": "getStates", "variables": { "projectPath": "sefi/tf-demo", "first": 50, "after": null, "last": null, "before": null }, "query": "query getStates($projectPath: ID!, $first: Int, $last: Int, $before: String, $after: String) {\n project(fullPath: $projectPath) {\n id\n terraformStates(first: $first, last: $last, before: $before, after: $after) {\n count\n nodes {\n ...State\n __typename\n }\n pageInfo {\n ...PageInfo\n __typename\n }\n __typename\n }\n __typename\n }\n}\n\nfragment State on TerraformState {\n id\n name\n lockedAt\n updatedAt\n deletedAt\n lockedByUser {\n ...User\n __typename\n }\n latestVersion {\n ...StateVersion\n __typename\n }\n __typename\n}\n\nfragment User on User {\n id\n avatarUrl\n name\n username\n webUrl\n __typename\n}\n\nfragment StateVersion on TerraformStateVersion {\n id\n downloadPath\n serial\n updatedAt\n createdByUser {\n ...User\n __typename\n }\n job {\n id\n detailedStatus {\n id\n detailsPath\n group\n icon\n label\n text\n __typename\n }\n pipeline {\n id\n path\n __typename\n }\n __typename\n }\n __typename\n}\n\nfragment PageInfo on PageInfo {\n hasNextPage\n hasPreviousPage\n startCursor\n endCursor\n __typename\n}\n" }

This API returns the latest version of all environments in the project.


{ "data": { "project": { "id": "gid://gitlab/Project/", "terraformStates": { "count": 1, "nodes": [ { "id": "gid://gitlab/Terraform::State/", "name": "", "lockedAt": null, "updatedAt": "2022-08-02T19:55:26Z", "deletedAt": null, "lockedByUser": null, "latestVersion": { "id": "gid://gitlab/Terraform::StateVersion/", "downloadPath": "/api/v4/projects//terraform/state//versions/0", "serial": 0, "updatedAt": "2022-08-02T19:55:26Z", "createdByUser": null, "job": null, "__typename": "TerraformStateVersion" }, "__typename": "TerraformState" } ], "pageInfo": { "hasNextPage": false, "hasPreviousPage": false, "startCursor": "", "endCursor": "", "__typename": "PageInfo" }, "__typename": "TerraformStateConnection" },

‍
Using the response, we can download the latest version of the Terraform State leveraging the previously mentioned Gitlab API:

https://{{GITLAB-HOST}}/API/v4/projects//terraform/state//versions/

That’s it! It’s that easy.

For anyone using Gitlab On-Prem or Enterprise, leveraging Gitlab Pipelines, there really is no need to add more tooling to the stack for Terraform orchestration and management. You can leverage the built-in Gitlab support and Terraform’s integration with S3. With the GraphQL API access you can now access the required info to download your state from storage via the Gitlab API.