Forem: drewmullen

Migrating a Terraform Module to Ephemeral Resources Without Breaking Existing Users

drewmullen — Fri, 17 Apr 2026 13:49:16 +0000

Terraform's ephemeral resources are a genuinely useful addition — secrets that are used during an apply but never written to state.

If you're a module maintainer, adopting ephemerals isn't as simple as swapping one block for another. You have existing users. Their state files already have the old resource tracked. Changing to an ephemeral version of the resource will blow away their managed resource.

To illustrate the issue, we'll pretend to be updating a module that generates a TLS private key and stores it in a Vault KV secret. This post documents the problem and a solution with an extra tid-bit that I learned.

The Setup

The original module was straightforward:

resource "tls_private_key" "legacy" {
  algorithm = "RSA"
  rsa_bits  = 4096
}

resource "vault_kv_secret_v2" "legacy" {
  mount     = "kvv2"
  name      = "mytls"
  data_json = jsonencode({ private_key = tls_private_key.legacy.private_key_pem })
}

The private key ends up in state. Vault KV ends up with a copy. Works fine, but the key is visible to anyone with state access.

The goal: keep this working for existing users, while letting new users opt into ephemeral key generation — where the key is used to write to Vault but is never persisted to state.

The Naive Approach (and Why It Fails)

The obvious move is a variable toggle with a local to abstract the logic away from the resource:

variable "use_ephemeral_key" {
  type    = bool
  default = false
}

resource "tls_private_key" "legacy" {
  count     = var.use_ephemeral_key ? 0 : 1
  algorithm = "RSA"
  rsa_bits  = 4096
}

ephemeral "tls_private_key" "ephemeral" {
  count     = var.use_ephemeral_key ? 1 : 0
  algorithm = "RSA"
  rsa_bits  = 4096
}

locals {
  private_key_pem = (
    var.use_ephemeral_key
    ? ephemeral.tls_private_key.ephemeral[0].private_key_pem
    : tls_private_key.legacy[0].private_key_pem
  )
}

resource "vault_kv_secret_v2" "legacy" {
  count     = var.use_ephemeral_key ? 0 : 1
  mount     = "kvv2"
  name      = "mytls"
  data_json = jsonencode({ private_key = local.private_key_pem })  
}

Running this produces:

│ Error: Invalid use of ephemeral value
│
│   with vault_kv_secret_v2.legacy,
│   on updated.tf line 34, in resource "vault_kv_secret_v2" "legacy":
│   34:     data_json = jsonencode({ private_key = local.private_key_pem })
│
│ Ephemeral values are not valid for "data_json", because it is not a
│ write-only attribute and must be persisted to state.

Why This Happens

The fundamental rule is: ephemeral values can only flow into write-only attributes. data_json must be persisted to state, so it can never accept an ephemeral value. That's true whether the ephemeral value arrives directly or through a local.

The shared local makes this worse in a subtle way. Terraform's ephemeral taint propagation is static, not runtime. Because one branch of the local.private_key_pem conditional references an ephemeral value, the entire local is marked ephemeral — regardless of which branch actually executes at runtime. So even if you reasoned "when use_ephemeral_key = false, the local resolves to a non-ephemeral value," Terraform doesn't do branch-aware type narrowing. The local is tainted, and the error fires before it even reaches data_json.

The fix is to never route ephemeral and non-ephemeral values through the same expression. vault_kv_secret_v2 has two relevant attributes: data_json (regular, persists to state) and data_json_wo (write-only, accepts ephemeral values). Use them conditionally on the same resource, keeping each attribute's expression isolated to its own value source:

resource "vault_kv_secret_v2" "legacy" {
  mount        = "kvv2"
  name         = "mytls"
  data_json    = var.use_ephemeral_key ? null : jsonencode({ private_key = one(tls_private_key.legacy[*].private_key_pem) })
  data_json_wo = var.use_ephemeral_key ? jsonencode({ private_key = one(ephemeral.tls_private_key.ephemeral[*].private_key_pem) }) : null
  data_json_wo_version = var.use_ephemeral_key ? var.secret_version : null

}

Why data_json_wo_version: Terraform never reads back write-only attributes, so it has no way to diff them. Without a version signal, the Vault secret is written on initial create and then never updated — even if the key changes. data_json_wo_version is an integer Terraform does track in state; when it changes, Terraform re-writes the secret. Users increment var.secret_version whenever they want to force a rotation.

Why one() instead of [0]: Both key resources use count — only one is active at a time. Referencing resource[0] when count = 0 errors at plan time even when the ternary condition would skip that branch, because Terraform resolves resource instance addresses before evaluating expressions. The splat + one() pattern returns null for an empty collection instead of erroring:

one(tls_private_key.legacy[*].private_key_pem)  # null when count = 0, actual value when count = 1

The outer ternary ensures that null never reaches jsonencode, so neither attribute ends up with an unexpected {"private_key":null} payload.

Rule to internalize: ephemeral values can only ever terminate at ephemeral attributes (write-only, or ephemeral such as provider blocks). Don't route them through any expression that also serves a non-write-only path.

The Second Problem: `moved` Blocks

The original module had no count on either resource. Adding count to tls_private_key.legacy changes its state address:

Before	After
`tls_private_key.legacy`	`tls_private_key.legacy[0]`

Terraform treats these as different resources. Without intervention, it will destroy the old address and create a new one — generating a brand new private key in the process. For existing users, that's an unannounced key rotation.

vault_kv_secret_v2.legacy keeps no count, so its address doesn't change and needs no intervention. Only the TLS key resource needs a moved block:

moved {
  from = tls_private_key.legacy
  to   = tls_private_key.legacy[0]
}

This tells Terraform to update the state address in place — no destroy, no create, no key rotation. On the next plan, existing users will see:

# tls_private_key.legacy has moved to tls_private_key.legacy[0]

Leave the moved block in for at least one release cycle so users who haven't applied yet still get a clean migration. It can be removed in a future major version once the old address no longer exists in any user's state.

The Complete Solution

moved {
  from = tls_private_key.legacy
  to   = tls_private_key.legacy[0]
}

variable "use_ephemeral_key" {
  description = "Use ephemeral key generation (key never written to state). New deployments should set true."
  type        = bool
  default     = false
}

variable "secret_version" {
  description = "Increment to trigger a re-write of the Vault secret. Only relevant when use_ephemeral_key = true."
  type        = number
  default     = 1
}

# Legacy path — key stored in state
resource "tls_private_key" "legacy" {
  count     = var.use_ephemeral_key ? 0 : 1
  algorithm = "RSA"
  rsa_bits  = 4096
}

# Ephemeral path — key never written to state
ephemeral "tls_private_key" "ephemeral" {
  count     = var.use_ephemeral_key ? 1 : 0
  algorithm = "RSA"
  rsa_bits  = 4096
}

resource "vault_kv_secret_v2" "legacy" {
  mount                = "kvv2"
  name                 = "mytls"
  data_json            = var.use_ephemeral_key ? null : jsonencode({ private_key = one(tls_private_key.legacy[*].private_key_pem) })
  data_json_wo         = var.use_ephemeral_key ? jsonencode({ private_key = one(ephemeral.tls_private_key.ephemeral[*].private_key_pem) }) : null
  data_json_wo_version = var.use_ephemeral_key ? var.secret_version : null
}

Versioning Strategy

With this in place, the recommended release approach is:

Current minor version: ship the toggle with default = false. Existing users are unaffected. New users can opt in with use_ephemeral_key = true.
Next major version: flip the default to true. Document the change. Users upgrading from the old version need to either set use_ephemeral_key = false explicitly to preserve legacy behavior, or accept that a new key will be generated on their next apply.

The moved blocks handle the state address migration for everyone regardless of which path they're on.

Migrating a Terraform Module to Ephemeral Resources Without Breaking Existing Users

drewmullen — Thu, 16 Apr 2026 19:35:48 +0000

Terraform's ephemeral resources are a genuinely useful addition — secrets that are used during an apply but never written to state.

The Setup

The original module was straightforward:

resource "tls_private_key" "legacy" {
  algorithm = "RSA"
  rsa_bits  = 4096
}

resource "vault_kv_secret_v2" "legacy" {
  mount     = "kvv2"
  name      = "mytls"
  data_json = jsonencode({ private_key = tls_private_key.legacy.private_key_pem })
}

The private key ends up in state. Vault KV ends up with a copy. Works fine, but the key is visible to anyone with state access.

The goal: keep this working for existing users, while letting new users opt into ephemeral key generation — where the key is used to write to Vault but is never persisted to state.

For solution TL;DR see The Complete Solution below

The Naive Approach (and Why It Fails)

The obvious move is a variable toggle with a local to abstract the logic away from the resource:

variable "use_ephemeral_key" {
  type    = bool
  default = false
}

resource "tls_private_key" "legacy" {
  count     = var.use_ephemeral_key ? 0 : 1
  algorithm = "RSA"
  rsa_bits  = 4096
}

ephemeral "tls_private_key" "ephemeral" {
  count     = var.use_ephemeral_key ? 1 : 0
  algorithm = "RSA"
  rsa_bits  = 4096
}

locals {
  private_key_pem = (
    var.use_ephemeral_key
    ? ephemeral.tls_private_key.ephemeral[0].private_key_pem
    : tls_private_key.legacy[0].private_key_pem
  )
}

resource "vault_kv_secret_v2" "legacy" {
  count     = var.use_ephemeral_key ? 0 : 1
  mount     = "kvv2"
  name      = "mytls"
  data_json = jsonencode({ private_key = local.private_key_pem })  
}

Running this produces:

│ Error: Invalid use of ephemeral value
│
│   with vault_kv_secret_v2.legacy,
│   on updated.tf line 34, in resource "vault_kv_secret_v2" "legacy":
│   34:     data_json = jsonencode({ private_key = local.private_key_pem })
│
│ Ephemeral values are not valid for "data_json", because it is not a
│ write-only attribute and must be persisted to state.

Why This Happens

resource "vault_kv_secret_v2" "legacy" {
  mount        = "kvv2"
  name         = "mytls"
  data_json    = var.use_ephemeral_key ? null : jsonencode({ private_key = one(tls_private_key.legacy[*].private_key_pem) })
  data_json_wo = var.use_ephemeral_key ? jsonencode({ private_key = one(ephemeral.tls_private_key.ephemeral[*].private_key_pem) }) : null
}

one(tls_private_key.legacy[*].private_key_pem)  # null when count = 0, actual value when count = 1

The outer ternary ensures that null never reaches jsonencode, so neither attribute ends up with an unexpected {"private_key":null} payload.

The Second Problem: `moved` Blocks

The original module had no count on either resource. Adding count to tls_private_key.legacy changes its state address:

Before	After
`tls_private_key.legacy`	`tls_private_key.legacy[0]`

vault_kv_secret_v2.legacy keeps no count, so its address doesn't change and needs no intervention. Only the TLS key resource needs a moved block:

moved {
  from = tls_private_key.legacy
  to   = tls_private_key.legacy[0]
}

This tells Terraform to update the state address in place — no destroy, no create, no key rotation. On the next plan, existing users will see:

# tls_private_key.legacy has moved to tls_private_key.legacy[0]

The Complete Solution

moved {
  from = tls_private_key.legacy
  to   = tls_private_key.legacy[0]
}

variable "use_ephemeral_key" {
  description = "Use ephemeral key generation (key never written to state). New deployments should set true."
  type        = bool
  default     = false
}

# Legacy path — key stored in state
resource "tls_private_key" "legacy" {
  count     = var.use_ephemeral_key ? 0 : 1
  algorithm = "RSA"
  rsa_bits  = 4096
}

# Ephemeral path — key never written to state
ephemeral "tls_private_key" "ephemeral" {
  count     = var.use_ephemeral_key ? 1 : 0
  algorithm = "RSA"
  rsa_bits  = 4096
}

resource "vault_kv_secret_v2" "legacy" {
  mount        = "kvv2"
  name         = "mytls"
  data_json    = var.use_ephemeral_key ? null : jsonencode({ private_key = one(tls_private_key.legacy[*].private_key_pem) })
  data_json_wo = var.use_ephemeral_key ? jsonencode({ private_key = one(ephemeral.tls_private_key.ephemeral[*].private_key_pem) }) : null
}

Versioning Strategy

With this in place, the recommended release approach is:

Current minor version: ship the toggle with default = false. Existing users are unaffected. New users can opt in with use_ephemeral_key = true.
Next major version: flip the default to true. Document the change. Users upgrading from the old version need to either set use_ephemeral_key = false explicitly to preserve legacy behavior, or accept that a new key will be generated on their next apply.

The moved blocks handle the state address migration for everyone regardless of which path they're on.

Writing Terraform Resources with Write-Only Parameters

drewmullen — Thu, 22 Jan 2026 22:10:57 +0000

When building Terraform providers that handle sensitive data like passwords, API tokens, or secret keys, you'll eventually encounter the need for write-only parameters. These are values that should be sent to an API but never stored in Terraform state files.

In this post, I'll walk through implementing write-only parameters in a Terraform provider, the challenges involved, and design options to consider.

What Are Write-Only Parameters?

Write-only parameters (introduced in Terraform 1.11) are resource arguments that:

Are sent to the API during creation and updates
Never appear in the Terraform state file

This feature is crucial for security. Without write-only parameters, sensitive values like passwords would be stored in plaintext in your state files.

Here's an example from my fork of the Event Drive Ansible (EDA) provider:

resource "aap_eda_credential" "example" {
  name               = "my-api-credential"
  credential_type_id = 1

  # Write-only: sent to API but NEVER stored in state
  inputs_wo = jsonencode({
    username  = "service-account"
    api_token = var.api_token
  })
}

Design Considerations

The `_version` Argument

Since Terraform's statefile does not manage the value of the secret, Terraform requires way to know if the secret must be updated. The standard solution to detecting changes in write-only parameters is to add a companion _version field that users must manually increment:

resource "aap_eda_credential" "manual" {
  name      = "my-credential"
  inputs_wo = jsonencode({
    password = var.new_password
  })

  # User must increment this to trigger an update
  inputs_wo_version = 1
}

How updates work:

User updates the secret in inputs_wo
User manually increments inputs_wo_version
Terraform detects the version change and triggers an update
The new secret is sent to the API

This works, but it's not user-friendly. Users have to remember to:

Increment the version every time they change secrets
Track version numbers manually

Automatic Change Detection

Ideally, users shouldn't need to manage version numbers. They should just update the secret value and Terraform should detect the change automatically. But here's the problem:

If inputs_wo is write-only and not in state, how do we detect when it changes?

Lucikly, there is a solution: store a deterministic hash in private state.

Using Private State and Hashing

Terraform's plugin framework provides private state - a place to store provider-managed data that's:

Stored in the state file (so it persists)
Not visible to users (doesn't appear in terraform show)
Perfect for storing things like hashes

Here's the implementation approach:

// Create operation
func (r *EDACredentialResource) Create(ctx context.Context, req resource.CreateRequest, resp *resource.CreateResponse) {
    var data EDACredentialResourceModel

    req.Plan.Get(ctx, &data)

    // Calculate SHA-256 hash of the inputs
    inputsHash := calculateInputsHash(data.InputsWO.ValueString())

    // Store hash in private state (JSON-wrapped for validity)
    hashJSON := fmt.Sprintf(`{"hash":"%s"}`, inputsHash)
    resp.Private.SetKey(ctx, "inputs_wo_hash", []byte(hashJSON))

    // Set auto-managed version
    data.InputsWOVersion = tftypes.Int64Value(1)

    // ... send to API and save state ...
}

func calculateInputsHash(inputs string) string {
    h := sha256.New()
    h.Write([]byte(inputs))
    return hex.EncodeToString(h.Sum(nil))
}

In the Update operation, we compare hashes:

func (r *EDACredentialResource) Update(ctx context.Context, req resource.UpdateRequest, resp *resource.UpdateResponse) {
    var data, state EDACredentialResourceModel

    req.Plan.Get(ctx, &data)
    req.State.Get(ctx, &state)

    // Get stored hash from private state
    oldHashBytes, _ := req.Private.GetKey(ctx, "inputs_wo_hash")

    var hashWrapper struct {
        Hash string `json:"hash"`
    }
    json.Unmarshal(oldHashBytes, &hashWrapper)
    oldHash := hashWrapper.Hash

    // Calculate new hash
    newHash := calculateInputsHash(data.InputsWO.ValueString())

    if newHash != oldHash {
        // Inputs changed! Auto-increment version
        data.InputsWOVersion = tftypes.Int64Value(state.InputsWOVersion.ValueInt64() + 1)

        // Update stored hash
        hashJSON := fmt.Sprintf(`{"hash":"%s"}`, newHash)
        resp.Private.SetKey(ctx, "inputs_wo_hash", []byte(hashJSON))
    } else {
        // No change, keep version as-is
        data.InputsWOVersion = state.InputsWOVersion
    }

    // ... send to API and save state ...
}

Now users can just update the secret and Terraform automatically detects it:

resource "aap_eda_credential" "auto" {
  name      = "my-credential"
  inputs_wo = jsonencode({
    password = var.new_password  # Just change this!
  })
  # inputs_wo_version auto-increments (computed field)
}

Best of both worlds

While automatic detection is convenient, some users may not want a hash of their secrets stored in state - even if it's SHA-256 and in private state. The hash is still derived from the secret value.

Solution: Support both modes

func (r *EDACredentialResource) Create(ctx context.Context, req resource.CreateRequest, resp *resource.CreateResponse) {
    var data EDACredentialResourceModel
    req.Plan.Get(ctx, &data)

    var versionToSet tftypes.Int64

    if data.InputsWOVersion.IsNull() || data.InputsWOVersion.IsUnknown() {
        // Auto-managed mode: store hash
        inputsHash := calculateInputsHash(data.InputsWO.ValueString())
        hashJSON := fmt.Sprintf(`{"hash":"%s"}`, inputsHash)
        resp.Private.SetKey(ctx, "inputs_wo_hash", []byte(hashJSON))
        versionToSet = tftypes.Int64Value(1)
    } else {
        // Manual mode: user provided version, don't store hash
        versionToSet = data.InputsWOVersion
    }

    // ... rest of create logic ...
    data.InputsWOVersion = versionToSet
}

Schema definition:

"inputs_wo_version": schema.Int64Attribute{
    Optional: true,
    Computed: true,
    PlanModifiers: []planmodifier.Int64{
        int64planmodifier.UseStateForUnknown(),
    },
    Description: "Version number for managing credential updates. " +
        "If not set, the provider will automatically detect changes to inputs_wo " +
        "using a SHA-256 hash stored in private state. If set manually, you " +
        "control when the credential is updated by incrementing this value yourself.",
}

Users can choose their preferred mode:

# Auto-managed (default)
resource "aap_eda_credential" "auto" {
  name      = "auto-credential"
  inputs_wo = jsonencode({ password = var.pwd })
  # version auto-increments
}

# Manual control
resource "aap_eda_credential" "manual" {
  name              = "manual-credential"
  inputs_wo         = jsonencode({ password = var.pwd })
  inputs_wo_version = 1  # I control this
}

Handling Mode Switching

Unfortunately, a weakness of this design is that its not possible to "switch modes". Why?

Auto → Manual: If a user suddenly sets inputs_wo_version, do we remove the hash? What if they're just trying to force an update?
Manual → Auto: We'd need to create a hash, but we can't compare it to the old inputs (they're write-only and not in state), so we can't tell if inputs changed during the transition

Even if we could find a way to handle this, the logic in your resource becomes complex and error-prone.

Better approach: Prevent mode switching

func (r *EDACredentialResource) Update(ctx context.Context, req resource.UpdateRequest, resp *resource.UpdateResponse) {
    // Determine current mode from private state
    oldHashBytes, _ := req.Private.GetKey(ctx, "inputs_wo_hash")
    wasAutoManaged := oldHashBytes != nil

    // Determine desired mode from config
    var configModel EDACredentialResourceModel
    req.Config.Get(ctx, &configModel)
    isNowManual := !configModel.InputsWOVersion.IsNull() && !configModel.InputsWOVersion.IsUnknown()

    var state EDACredentialResourceModel
    req.State.Get(ctx, &state)
    wasManual := !wasAutoManaged && !state.InputsWOVersion.IsNull()

    // Prevent mode switching
    if wasAutoManaged && isNowManual {
        resp.Diagnostics.AddError(
            "Cannot switch from auto-managed to manual version management",
            "The inputs_wo_version field was previously auto-managed. Once auto-managed, "+
            "it cannot be switched to manual mode. If you need to manually control the version, "+
            "you must recreate the resource with inputs_wo_version set from the start.",
        )
        return
    }

    if wasManual && !isNowManual {
        resp.Diagnostics.AddError(
            "Cannot switch from manual to auto-managed version management",
            "The inputs_wo_version field was previously manually managed. Once manually managed, "+
            "it cannot be switched to auto mode. If you need auto-managed version control, "+
            "you must recreate the resource without setting inputs_wo_version.",
        )
        return
    }

    // ... rest of update logic for the chosen mode ...
}

This gives users a clear error message:

Error: Cannot switch from auto-managed to manual version management

The inputs_wo_version field was previously auto-managed. Once auto-managed,
it cannot be switched to manual mode. If you need to manually control the
version, you must recreate the resource with inputs_wo_version set from the start.

Key Takeaways

When implementing write-only parameters in Terraform providers:

Use private state for hashes - It's stored but not visible to users
Support both auto and manual modes - Give users choice for their security preferences

The complete implementation provides a great user experience while respecting privacy concerns:

# Most users: auto-managed, no version to track
resource "aap_eda_credential" "api" {
  name      = "api-credential"
  inputs_wo = jsonencode({ token = var.token })
}

# Privacy-conscious users: manual control, no hash stored
resource "aap_eda_credential" "secure" {
  name              = "secure-credential"
  inputs_wo         = jsonencode({ token = var.token })
  inputs_wo_version = 1
}

Both approaches are valid. Both are clear. And switching between them requires a resource recreation, which is easy to communicate and reason about.

Have you implemented write-only parameters in your Terraform provider? What challenges did you face? Let me know in the comments!

Making Terraform Workspaces in the same project

drewmullen — Fri, 27 Jun 2025 14:10:43 +0000

I am working on a artifact for helping TFE users provide some self service (to be unveiled soon!). One of the things I needed to do was provide a repeatable way to have a tfe workspace create workspaces in the same project. At first I thought I'd need a new data source for this since the tfe data sources require organization and at the very least.

Turns out I can do it today with some magic runtime env vars:

variable "TFC_WORKSPACE_SLUG" {}
variable "TFC_PROJECT_ID" {}
locals {
  org_name = split("/",var.TFC_WORKSPACE_SLUG)[0]
}

resource "tfe_workspace" "ws" {
  name         = "new-ws"
  organization = local.org_name
  project_id   = var.TFC_PROJECT_ID
}

Executing that code in a TFE workspace will let you create workspaces without knowing the organization name or project! Niche but pretty nifty!

Setting a multi-line variable as an ENV in Terraform Cloud

drewmullen — Wed, 12 Feb 2025 15:55:56 +0000

In this post I will describe the exploration I did after running into a problem trying to setup temporary auth to my kubernetes cluster with Terraform. I will first describe the problem(s) and several ways to solve them.

Preamble

I auth to my homelab k8s cluster using PEM keys. Being a long time Terraform user I almost always auth to a provider using environment variables. This allows me to decouple auth from the HCL, prevent secrets from getting into the statefile, and often allows completely avoiding defining a particular provider block, relying on the "empty default configuration".

Terraform Cloud (TFC) (recently rebranded to HCP TF) allows setting variables outside of the code and in the web ui instead (WS Vars). You can set WS Vars as either type terraform or type env. Type terraform behaves exactly like setting -var myname=drew. Only problem is that type env vars cannot be multi-line which breaks PEM validity.

Solutions

Ephemeral Variables!

Starting Terraform v1.10 HashiCorp introduced ephemeral resource types as well as ephemeral variables. This allows me to set the PEM in a WS Var type terraform but not expose to the statefile. It does require me to hardcode the provider version and auth.

variable "kube_client_key" {
  description = "Private key data for authenticating to k8s."
  ephemeral   = true
}

provider "kubernetes" {
  client_certificate = var.kube_client_certificate
  client_key         = var.kube_client_key
}

Base64 & TF_VAR_

You can pass the multi-line content base64 encoded then leverage terraform's base64decode() function in a TF_VAR_. This allows using an env variable but still requires hardcoding the auth. This solution doesn't make much sense for me but it is an option

provider "kubernetes" {
  client_key = base64decode(var.kubekey)
}

Dynamic K8s Access

You can setup a trust between TFC and your k8s cluster. This is my end game but on the back burner while I figure out how I plan to organize somethings

Doc Link: https://developer.hashicorp.com/terraform/cloud-docs/workspaces/dynamic-provider-credentials/kubernetes-configuration

E1it3 Hax

This last option is totally unexpected but actually quite interesting. Turns out that part of TFC's implementation is that WS Vars that are type terraform are also written to agents as environment variables. This means that you can create a WS Var in TFC that matches the naming convention of the expected provider env var and it will still work. This does not require you to create Terraform HCL for the vars but it does surface a warning. You can suppress the warning by providing a empty variable with the same name and NOT defining it in the provider block.

To suppress the warning just insert a variable with the corresponding name and a default value:

variable "KUBE_CLIENT_KEY_DATA" {
  default = ""
}

Salutation

I hope this is helpful to you. Choose the solution that fits your use-case best!

Deploy Cloudflare Tunnels with Terraform & Kubernetes

drewmullen — Tue, 21 Jan 2025 18:32:07 +0000

This blog will give you a brief overview for exposing services from a private network to the public network using zero trust tool, CloudFlare Zero Trust Tunnels. They have a generous free tier that you can leverage. Please note that this post does not encompass a fully prescriptive set of security best practices, merely it gets the setup working. Getting this right took me longer than I expected. Hopefully its helpful to you.

Due to to my homelab setup, namespaces are created by ArgoCD which kicks off ahead of terraform apply. You may need to make a namespace cloudflared.

Access

Based on the guide from cloudflares docs you'll need to create a API token:

Note: When creating your api token know that "Cloudflare Tunnel" is not the same thing as Zero Trust in their permissions. This was confusing to me because tunnels are now named "zero trust tunnels". For the sake of permissions make sure you provider "Cloudflare Tunnel: Edit".

Terraform

The following code deploys a single tunnel and sets up DNS records for all sites indicated in "services". TLS verify is disabled below so update it if you want.

variable "cloudflare_zone_id" {
  description = "Zone ID for your domain"
  type        = string
  sensitive   = true
}

variable "cloudflare_account_id" {
  description = "Account ID for your Cloudflare account"
  type        = string
  sensitive   = true
}

variable "cloudflare_email" {
  description = "Email address for your Cloudflare account"
  type        = string
  sensitive   = true
}

variable "services" {
  description = "Values for the services to be exposed via Cloudflare Tunnel"
  type = map(object({
    hostname = string # public domain name. ex: "service.example.com"
    service  = string # private service endpoint. ex: service.apps.internaldomain.com
  }))
}

locals {
  cf_tunnel_secret = jsonencode({
    "AccountTag" : "${var.cloudflare_account_id}",
    "TunnelSecret" : "${base64sha256(random_password.tunnel_secret.result)}",
    "TunnelID" : "${cloudflare_zero_trust_tunnel_cloudflared.homelab.id}"
  })

  ingress_rules = [
    for service in var.services : {
      hostname = service.hostname
      origin_request = {
        no_tls_verify = true
      }
      service = service.service
    }
  ]
}

resource "random_password" "tunnel_secret" {
  length = 64
}

resource "cloudflare_zero_trust_tunnel_cloudflared" "homelab" {
  name          = "homelab-tunnel"
  config_src    = "cloudflare"
  account_id    = var.cloudflare_account_id
  tunnel_secret = base64sha256(random_password.tunnel_secret.result)
}

resource "cloudflare_zero_trust_tunnel_cloudflared_config" "homelab" {
  account_id = var.cloudflare_account_id
  tunnel_id  = cloudflare_zero_trust_tunnel_cloudflared.homelab.id
  config = {
    ingress = concat(local.ingress_rules, [
      {
        origin_request = {
          connect_timeout        = 0
          keep_alive_connections = 0
          keep_alive_timeout     = 0
          tcp_keep_alive         = 0
          tls_timeout            = 0
        }
        service = "http_status:503"
      },
    ])
    warp_routing = {
      enabled = false
    }
  }
}

resource "kubernetes_secret" "cloudflare_credentials" {
  metadata {
    name      = "tunnel-credentials"
    namespace = "cloudflared"
  }

  data = {
    "credentials.json" = local.cf_tunnel_secret
  }
}

resource "cloudflare_dns_record" "vault_homelab_tunnel" {
  for_each = var.services

  zone_id = var.cloudflare_zone_id
  comment = "${each.key} tunnel record"
  content = join(".", [cloudflare_zero_trust_tunnel_cloudflared.homelab.id, "cfargotunnel.com"])
  name    = each.value.hostname
  proxied = true
  ttl     = 1
  type    = "CNAME"
}

Kubernetes manifest:

In my setup, k8s resources are mostly deployed via ArgoCD. Below, this manifest deploys cloudflared pod and connects it to the tunnel homelab-tunnel from your account. It authenticates via the public cloudflare API using credentials from a kubernetes secret tunnel-credentials which is created by Terraform.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cloudflared
  namespace: cloudflared
spec:
  selector:
    matchLabels:
      app: cloudflared
  replicas: 1 
  template:
    metadata:
      labels:
        app: cloudflared
    spec:
      containers:
      - name: cloudflared
        image: cloudflare/cloudflared:latest
        args:
        - tunnel
        - --credentials-file
        - /etc/cloudflared/creds/credentials.json
        - --protocol # https://github.com/cloudflare/cloudflared/issues/1176#issuecomment-2404546711
        - http2      # didnt seem to need the sysctl changes
        - --metrics
        - 0.0.0.0:2000
        - run
        - homelab-tunnel
        livenessProbe:
          httpGet:
            path: /ready
            port: 2000
          failureThreshold: 1
          initialDelaySeconds: 10
          periodSeconds: 10
        volumeMounts:
        - name: creds
          mountPath: /etc/cloudflared/creds
          readOnly: true
      volumes:
      - name: creds
        secret:
          secretName: tunnel-credentials

Disclaimer: The following code stores the secret to joining your tunnel in your terraform statefile in 2 locations random_password.tunnel_secret and cloudflare_zero_trust_tunnel_cloudflared.homelab. With 1.10 we have ephemeral resources which will remove that but the random provider doesnt have it yet. Hopefully it will be removed from tunnel resource once available.

Terraform state with native s3 locking

drewmullen — Mon, 02 Dec 2024 20:21:58 +0000

Terraform has many popular mechanisms for storing its state file. While I have grown quite fond of HCP Terraform, there are occasional scenarios where I want to use S3 backend to hold my Terraform state files.

Best practice when sharing a state file among multiple parties is to have a locking mechanism, to avoid concurrent writes from corrupting the Terraform state. This prevents 2 parties from clobbering each other if they happen to be working on the same infrastructure at the same time.

Starting in Terraform v1.10 the S3 backend features S3 native state locking. Prior to this feature state file lock setups required access to a DynamoDB table - which can be completely foregone now!

How?

Lets start by creating a S3 bucket that has the required settings:

Note: It is common to create the S3 state file bucket outside of Terraform. However, you can create the bucket with Terraform, its just generally not advised. Example shell commands below.

$ BUCKET_NAME="<your bucket name>"
$ REGION="us-east-1"

$ aws s3api create-bucket \
    --bucket $BUCKET_NAME \
    --region $REGION

$ aws s3api put-bucket-versioning \
    --bucket $BUCKET_NAME \
    --versioning-configuration Status=Enabled

Using the bucket

Create a new backend.tf file (or add this block to an existing .tf file). Make sure you fill in the bucket name!

Below we use the parameter use_lockfile which is an experimental feature that enforces using the s3 buckets lock file

terraform {
  backend "s3" {
    bucket       = <bucket name>
    key          = "backend/terraform.tfstate"
    region       = "us-east-1"
    use_lockfile = true
  }
}

Wrapping up

For a deep dive on this feature I recommend you read through the initial PR by @bschaatsbergen who wrote this great feature

Optional: Create bucket using Terraform

resource "aws_s3_bucket" "backend" {
  bucket = var.bucket_name
}

resource "aws_s3_bucket_versioning" "backend" {
  bucket = aws_s3_bucket.backend.id
  versioning_configuration {
    status = "Enabled"
  }
}

Edited 12/19 to remove ObjectLock which is not required. Kudos to @philippe_robitaille_f3908

AWS Creds in the CLI via SSO

drewmullen — Thu, 16 Nov 2023 15:59:07 +0000

Problem

Do you often need credentials in your AWS CLI?

Are you overly familiar with this screen (AWS SSO start page)?

If you do not have AWS SSO setup, check out the AWS documentation then come back to this post!

A better credential experience

Yesterday I learned from my colleague @danquack a fun feature built into the AWS CLI. If you already have AWS SSO configured for your Org and are using the GUI to get credentials, follow this post and you can expect an improved AWS CLI credential management experience.

My new, simpler process to get temporary creds from SSO:

Specify which profile I want and login:

$ export AWS_PROFILE=<>
$ aws sso login

Confirm the authorization request in my browser:

Boom! Now my CLI has usable temporary credentials!

Setup

This setup is honestly extremely simple. AWS provides a guided CLI wizard and ill show the examples below.

$ aws configure sso
SSO session name (Recommended):
WARNING: Configuring using legacy format (e.g. without an SSO session).
Consider re-running "configure sso" command and providing a session name.
SSO start URL [https://example.awsapps.com/start#/]:
SSO region [us-east-1]:
There are 2 AWS accounts available to you.
> DeveloperAccount, developer-account-admin@example.com (123456789011) 
  ProductionAccount, production-account-admin@example.com (123456789022)
Using the account ID 123456789011
The only role available to you is: AdministratorAccess
Using the role name "AdministratorAccess"
CLI default client Region [us-east-1]: us-east-2
CLI default output format [None]:
CLI profile name [AdministratorAccess-<>]: providerdev

There are 2 questions regarding region. The first is the region SSO is setup in. The second is the default region you want your CLI setup to use.
Setting a profile name providerdev is now the name ill set for export AWS_PROFILE=providerdev

Once that is complete you can see the configuration:

$ cat -p ~/.aws/config
[profile providerdev]
sso_start_url = https://example.awsapps.com/start#/
sso_region = us-east-1
sso_account_id = 123456789011
sso_role_name = AdministratorAccess
region = us-east-2

Summary

AWS SSO is a great service for providing temporary credentials to known identities in your organization. The new command aws sso login will help you and your engineers get credentials fast, easy, and securely!

Implement Rollback with Terraform

drewmullen — Thu, 12 Jan 2023 14:06:56 +0000

There's no magic here, friends.

$ terraform apply # fails
$ git checkout HEAD~ # checkout the prior commit
$ terraform apply

You can checkout a specific commit sha with:
git checkout <commit sha>

I don't recommend using this with any stateful resource types... RDS, EKS, etc.

Terraform: Variable validation with samples

drewmullen — Fri, 04 Mar 2022 18:57:38 +0000

Terraform allows you to validate variable input in using validation blocks using custom condition and yielding a custom error_message. Below are some examples:

Note: Please share your common validation rules you've written and I'll update here

Update: Please check out this awesome project by @bschaatsbergen that provides built in functions for assertions: https://github.com/bschaatsbergen/terraform-provider-assert

Strings

String may not contain character

Scenario: String may not contain a /.

variable "string_may_not_contain" {
  type = string
  default = "test"

  validation {
    error_message = "Value cannot contain a \"/\"."
    condition = !can(regex("/", var.string_may_not_contain))
  }
}

String with valid options

Scenario: Here we have a string and we only allow to values "approved" or "disapproved". I show 2 examples of the same check using different methods:

variable "string_only_valid_options" {
  type = string
  default = "approved"

  # using regex
  validation {
    condition     = can(regex("^(approved|disapproved)$", var.string_only_valid_options))
    error_message = "Invalid input, options: \"approved\", \"disapproved\"."
  }

  # using contains()
  validation {
    condition     = contains(["approved", "disapproved"], var.string_only_valid_options)
    error_message = "Invalid input, options: \"approved\", \"disapproved\"."
  }
}

Valid AWS Region Name

Scenario: string must be like AWS region

variable "string_like_aws_region" {
  type = string
  default = "us-east-1"

  validation {
    condition     = can(regex("[a-z][a-z]-[a-z]+-[1-9]", var.string_like_aws_region))
    error_message = "Must be valid AWS Region names."
  }

Valid IAM Role Name

Scenario: Your string must be a valid IAM role name

variable "string_valid_iam_role_name" {
    type = string
    default = "MyCoolRole"
    # arn example: "arn:aws:iam::123456789012:role/MyCoolRole"

    validation {
      condition     = can(regex("^[a-zA-Z][a-zA-Z\\-\\_0-9]{1,64}$", var.string_valid_iam_role_name))
      error_message = "IAM role name must start with letter, only contain letters, numbers, dashes, or underscores and must be between 1 and 64 characters."
    }
}

Valid IPv4 CIDR

Scenario: Your string input needs to look like a IPv4 CIDR. Thank you @entscheidungsproblem for reporting and providing a fix for /32.

variable "string_like_valid_ipv4_cidr" {
  type    = string
  default = "10.0.0.0/16"

  validation {
    condition     = can(cidrhost(var.string_like_valid_ipv4_cidr, 0))
    error_message = "Must be valid IPv4 CIDR."
  }
}

Semantic Version

variable "semv1" {
  default = "10.57.123"

  validation {
    error_message = "Must be valid semantic version."
    condition     = can(regex("^(0|[1-9]\\d*)\\.(0|[1-9]\\d*)\\.(0|[1-9]\\d*)(?:-((?:0|[1-9]\\d*|\\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\\.(?:0|[1-9]\\d*|\\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\\+([0-9a-zA-Z-]+(?:\\.[0-9a-zA-Z-]+)*))?$", var.semv1))
  }
}

Maps

Map with optional conflicting keys

Scenario: You have a map variable and 2 keys conflict, in this case, you can only set either cidr or netmask.

variable "only_one_optional_key" {
    type = object({
        name = optional(string)
        cidrs = optional(list(string))
        netmask = optional(number)
    })

    default = {
        cidr = "10.0.0.0/16"
        name = "test"
    }

    validation {
        error_message = "Can only specify either \"cidrs\", or \"netmask\"."
        condition = length(setintersection(keys(var.only_one_optional_key), ["cidrs", "netmask"])) == 1
    }
}

Numbers

Number within a range

Scenario: number must be between 1-16.
Thanks: @tlindsay42

variable "num_in_range" {
  type        = number
  default     = 1

  validation {
    condition     = var.num_in_range >= 1 && var.num_in_range <= 16 && floor(var.num_in_range) == var.num_in_range
    error_message = "Accepted values: 1-16."
  }
}

If you liked this post, please like. If you think it would be helpful in the future as a reference, please bookmark!

Terraform: Common aws_ami Searches

drewmullen — Wed, 02 Feb 2022 20:03:11 +0000

Searching for AMI's is not fun. Here are some you might be searching for:

Amazon Linux 2 (latest):

data "aws_ami" "amazon_linux_2" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["amzn2-ami-hvm-*-x86_64-ebs"]
  }
}

Graviton 2

RHEL 7 (latest)

Thanks: @wsarwariamzn

data "aws_ami" "rhel_7" {
  most_recent = true
  owners      = ["309956199498"]

  filter {
    name = "name"
    values = ["RHEL-7.9_HVM_GA*-x86_64-0-Hourly2-GP2"]
  }
}

Please send me more and I'll include them in this post!

Terraform Boilerplate: Provider block for every AWS region

drewmullen — Wed, 29 Dec 2021 19:31:07 +0000

Boilerplate HCL for your module writing needs! Did I miss any?

provider "aws" {}

#### NORTHERN VIRGINIA : us-east-1
provider "aws" {
  region = "us-east-1"
  alias = "us-east-1"
}

#### OHIO : us-east-2
provider "aws" {
  region = "us-east-2"
  alias = "us-east-2"
}

#### NORTHERN CALIFORNIA : us-west-1
provider "aws" {
  region = "us-west-1"
  alias = "us-west-1"
}

#### OREGON : us-west-2
provider "aws" {
  region = "us-west-2"
  alias = "us-west-2"
}

#### CANADA : ca-central-1
provider "aws" {
  region = "ca-central-1"
  alias = "ca-central-1"
}

#### MUMBAI : ap-south-1
provider "aws" {
  region = "ap-south-1"
  alias = "ap-south-1"
}

//#### OSAKA_LOCAL : ap-northeast-3
provider "aws" {
  region = "ap-northeast-3"
  alias = "ap-northeast-3"
}

#### SEOUL : ap-northeast-2
provider "aws" {
  region = "ap-northeast-2"
  alias = "ap-northeast-2"
}

#### SINGAPORE : ap-southeast-1
provider "aws" {
  region = "ap-southeast-1"
  alias = "ap-southeast-1"
}

#### SYDNEY : ap-southeast-2
provider "aws" {
  region = "ap-southeast-2"
  alias = "ap-southeast-2"
}

#### TOKYO : ap-northeast-1
provider "aws" {
  region = "ap-northeast-1"
  alias = "ap-northeast-1"
}


#### FRANKFHURT : eu-central-1
provider "aws" {
  region = "eu-central-1"
  alias = "eu-central-1"
}

#### IRELAND : eu-west-1
provider "aws" {
  region = "eu-west-1"
  alias = "eu-west-1"
}

#### LONDON : eu-west-2
provider "aws" {
  region = "eu-west-2"
  alias = "eu-west-2"
}

#### MILAN : eu-south-1
provider "aws" {
  region = "eu-south-1"
  alias = "eu-south-1"
}

#### PARIS : eu-west-3
provider "aws" {
  region = "eu-west-3"
  alias = "eu-west-3"
}

#### STOCKHOLM : eu-north-1
provider "aws" {
  region = "eu-north-1"
  alias = "eu-north-1"
}

####  SAO PAULO : sa-east-1
provider "aws" {
  region = "sa-east-1"
  alias = "sa-east-1"
}

Including these but I had trouble if I included

//#### SOUTH AFRICA : af-south-1
provider "aws" {
  region = "af-south-1"
  alias = "af-south-1"
}

//#### HONG KONG : ap-east-1
provider "aws" {
  region = "ap-east-1"
  alias = "ap-east-1"
}

//####  MIDDLE EAST: me-south-1
provider "aws" {
  region = "me-south-1"
  alias = "me-south-1"
}

Forem: drewmullen

Migrating a Terraform Module to Ephemeral Resources Without Breaking Existing Users

The Setup

The Naive Approach (and Why It Fails)

Why This Happens

The Second Problem: moved Blocks

The Complete Solution

Versioning Strategy

Migrating a Terraform Module to Ephemeral Resources Without Breaking Existing Users

The Setup

The Naive Approach (and Why It Fails)

Why This Happens

The Second Problem: moved Blocks

The Complete Solution

Versioning Strategy

Writing Terraform Resources with Write-Only Parameters

What Are Write-Only Parameters?

Design Considerations

The _version Argument

Automatic Change Detection

Using Private State and Hashing

Best of both worlds

Handling Mode Switching

Key Takeaways

Making Terraform Workspaces in the same project

Setting a multi-line variable as an ENV in Terraform Cloud

Preamble

Solutions

Ephemeral Variables!

Base64 & TF_VAR_

Dynamic K8s Access

E1it3 Hax

Salutation

Deploy Cloudflare Tunnels with Terraform & Kubernetes

Access

Terraform

Kubernetes manifest:

Terraform state with native s3 locking

How?

Using the bucket

Wrapping up

Optional: Create bucket using Terraform

AWS Creds in the CLI via SSO

Problem

A better credential experience

Specify which profile I want and login:

Confirm the authorization request in my browser:

Setup

Summary

Implement Rollback with Terraform

Terraform: Variable validation with samples

Strings

String may not contain character

String with valid options

Valid AWS Region Name

Valid IAM Role Name

Valid IPv4 CIDR

Semantic Version

Maps

Map with optional conflicting keys

Numbers

Number within a range

Terraform: Common aws_ami Searches

Amazon Linux 2 (latest):

Graviton 2

RHEL 7 (latest)

Terraform Boilerplate: Provider block for every AWS region

The Second Problem: `moved` Blocks

The Second Problem: `moved` Blocks

The `_version` Argument