Forem: Michael Lin

Deploy Infrastructure using CDK for Terraform with Go

Michael Lin — Wed, 13 Apr 2022 03:23:31 +0000

I joined Sourcegraph earlier this year to the team responsible for our on-prem deployment and cloud offering!

At Sourcegraph, we aim to build the "Google for Code" and make code more accessible to everyone. Many companies leverage Sourcegraph so developers can easily search code, automate code changes at scale, and track code changes, and more! Sourcegraph offers customers two deployment options, cloud multi-tenant offering and on-prem. Although we try really hard to keep your private code safe in multi-tenant offerings, many enterprise customers still prefer high isolation and choose to go with on-prem installation. Sadly, deploying and maintaining a production Sourcegraph instance is not a trivial task. Additionally, not every company has the necessary resources to maintain yet another system (especially if you're maintaining someone else's system).

In a company hackathon, a group of us decided to build a "magic instance maker" that allows anyone to one-click deploy a fully managed single-tenant Sourcegraph instance within shared infrastructure on Google Cloud Platform (GCP). Users only need to provide us with a name and we will magically "make" an instance and return the magic URL of a fresh Sourcegraph deployment.

Architecture

There are a lot of moving pieces to provision a fully functional Sourcegraph deployment, just like any other production web-based system. You need compute resources, storage resources, DNS, HTTP (TLS certificates), and much more.

Kubernetes is one of our supported installation methods for large-scale deployment, and Kubernetes has an amazing ecosystem for various infrastructure automation. We deploy Sourcegraph on a shared Google Kubernetes Engine (GKE) cluster using our experimental Helm chart. For datastores, we utilize managed services on GCP as much as possible, such as Cloud SQL and Google Cloud Storage (GCS). For DNS and TLS, we are mostly relying on Cloudflare. How do we automate so much stuff? Terraform (duh). With Terraform, we can provision all kinds of resources (by providers) and it comes with state management for free.

The Problem

We all like Terraform or Infrastructure as Code (IaC). It's a great tool for declaratively managing infrastructure, and (hopefully) it's reproducible, unlike ClickOps. However, Terraform (HCL) is static and we usually just commit the HCL files in a git repository. In our use case, we need to provision resources dynamically without human intervention. Unfortunately, Terraform doesn't provide any out-of-the-box solution to programmatically create a new module and apply changes.

Wouldn't it be nice to declare terraform modules with one of your favourite programming languages? Additionally, you will have much more control over generated resources, while with plain Terraform, you are bound by the HCL language constraint. CDK for Terraform (cdktf) is an experimental attempt at solveing this problem.

We used cdktf in Go to implement the project. Why Go and Terraform? Go is the go-to language at Sourcegraph and Terraform is something we use everyday (hence we did not go with things like pulumi).

How does it work?

For a complete tutorial, you should check out the Hashicorp's official tutorial. Code snippets below definitely won't compile.

First, you need to create a cdktf.json configuration file. It is used to configure providers and modules.

{
  "language": "go",
  "app": "go run main.go",
  "terraformProviders": [
    {
      "name": "google",
      "source": "hashicorp/google",
      "version": "~> 4.15.0"
    },
    {
      "name": "cloudflare",
      "source": "cloudflare/cloudflare",
      "version": "~> 3.11.0"
    }
  ]
  // ...
}

It's equivalent to the following in HCL,

terraform {
  required_providers {
    google = {
      source = "hashicorp/google"
      version = "4.15.0"
    }
  }
}

Then you need to run cdktf get, which will dynamically generate the go packages for the providers in cdktf.json. You will be able to import these packages later in your code to declare your Terraform module.

What does my module look like?

import (
  "fmt"

  "github.com/aws/constructs-go/constructs/v10"
  jsii "github.com/aws/jsii-runtime-go"
  "github.com/hashicorp/terraform-cdk-go/cdktf"
  "github.com/sourcegraph/magic-instance-maker/generated/google"
  "github.com/sourcegraph/magic-instance-maker/generated/cloudflare"
  "github.com/sourcegraph/magic-instance-maker/generated/helm"
)

func NewStack(scope constructs.Construct, id string) cdktf.TerraformStack {
  stack := cdktf.NewTerraformStack(scope, &id)

  // Configure remote backend to store terraform state
  cdktf.NewGcsBackend(stack, &cdktf.GcsBackendProps{
    Bucket: jsii.String("gcs-bucket-name"),
    Prefix: jsii.String(fmt.Sprintf("tenants/%s", id)),
  })

  // Configure gcp provide, this is equivalent to the `provider` block
  google.NewGoogleProvider(stack, jsii.String("google"), &google.GoogleProviderConfig{
    Zone:    jsii.String("region-name"),
    Project: jsii.String("project-id"),
  })

  // This is equivalent to the data source block `data "google_sql_database_instance" "cloud-sql-instance" {}`
  cloudSqlDatabaseInstance := google.NewDataGoogleSqlDatabaseInstance(stack, jsii.String("cloud-sql-instance"), &google.DataGoogleSqlDatabaseInstanceConfig{
    Project: jsii.String("project-id"),
    Name:    &cloudSqlInstanceId,
  })
  sqlUser := google.NewSqlUser(stack, jsii.String("sql-user"), &google.SqlUserConfig{
    Project:  jsii.String(projectId),
    Name:     jsii.String(fmt.Sprintf("%s-admin", id)),
    Password: cloudSqlAdminPassword.Result(),
    Instance: cloudSqlDatabaseInstance.Name(),
    Type:     jsii.String("BUILT_IN"),
  })
  cloudsqlPgsqlDbDependencies := []cdktf.ITerraformDependable{sqlUser}
  cloudSqlPgsqlDb := google.NewSqlDatabase(stack, jsii.String("pgsql"), &google.SqlDatabaseConfig{
    Project:   jsii.String(projectId),
    Name:      jsii.String(fmt.Sprintf("%s-pgsql", id)),
    Instance:  cloudSqlDatabaseInstance.Name(),
    DependsOn: &cloudsqlPgsqlDbDependencies,
  })

  helm.NewHelmProvider(stack, jsii.String("helm"), &helm.HelmProviderConfig{
    Kubernetes: &helm.HelmProviderKubernetes{
      ConfigPath:    jsii.String("KUBECONFIGPATH"),
      ConfigContext: jsii.String("CLUSTERNAME"),
    },
  })
  // We provision Sourcegraph deployment using our experimental helm chart
  // https://docs.sourcegraph.com/admin/install/kubernetes/helm
  helm.NewRelease(stack, jsii.String("release"), &helm.ReleaseConfig{
    Repository:      jsii.String("https://sourcegraph.github.io/deploy-sourcegraph-helm/"),
    Chart:           jsii.String("sourcegraph"),
    Name:            jsii.String(id),
    Namespace:       jsii.String(id),
    CreateNamespace: jsii.Bool(true),
    Values:          jsii.Strings("values-file-a-yaml-string"),
  })

  // Configure cloudflare provide, this is equivalent to the `provider` block
  cloudflare.NewCloudflareProvider(stack, jsii.String("cloudflare"), &cloudflare.CloudflareProviderConfig{
    ApiToken: jsii.String("cloudflare-api-token"),
  })

  // This is equivalent to `resource "cloudflare_record" "magic-example-com" {}`
  cloudflare.NewRecord(stack, jsii.String("magic-example-com"), &cloudflare.RecordConfig{
    ZoneId:  jsii.String("cloudflare-zone-id"),
    Name:    jsii.String(fmt.Sprintf("magic-%s.example.com", id)),
    Type:    jsii.String("A"),
    Value:   nginxIngressIpAddress.Address(),
    Proxied: jsii.Bool(true),
  })

Above Go code roughly translates to,

variable "tenant_id" { type = string }

terraform {
  required_providers {
    google = {
      source = "hashicorp/google"
      version = "4.15.0"
    }
  }
}

terraform {
  backend "gcs" {
    bucket = "gcs-bucket-name"
    # this actually won't work in terraform
    # backend block doesn't allow interpolations
    prefix = "tenants/${var.tenant_id}"
  }
}

provider "google" { }

data "google_cloud_sql_instance" "cloud-sql-instance" {
  name = ""
}

resource "cloudflare_record" "magic-example-com" {
  name = "magic-${var.tenant_id}.example.com"
  # ...
}

How do you actually apply the module or stack you define in go?

You can do so with cdktf-cli and just run cdktf deploy. But we will need to provision the stack dynamically and we don't want to "shell it out",

func main() {
  tempDir, err := os.MkdirTemp("", "magic-instance-maker-")
  if err != nil {
    return nil, err
  }
  defer os.RemoveAll(tempDir)

  app := cdktf.NewApp(&cdktf.AppOptions{Outdir: jsii.String(tempDir)})
  sharedtenant.NewStack(app, name, cluster)
  app.Synth()
}

Now let's try running go run main.go. Wait, it doesn't do anything? The app.Synth() invocation only synthesizes the module defined in Go into a JSON file without actually applying it. The great thing about HCL is that it is mostly interchangeable with JSON. In fact, if you cd into the tempDir, you can run the usual terraform init and terraform apply commands to apply your terraform module. This is exactly what the cdktf deploy command is doing behind the scene, it simply runs the terraform command for you. Unfortunately, when it comes to applying the terraform module, we still have to fall back to using the terraform CLI.

We will utilize this nice wrapper hashicorp/terraform-exec to apply the synthesized Terraform module from Go.

import (
  "context"
  "log"
  "os"
  "path/filepath"

  jsii "github.com/aws/jsii-runtime-go"
  "github.com/hashicorp/go-version"
  "github.com/hashicorp/hc-install/product"
  "github.com/hashicorp/hc-install/releases"
  "github.com/hashicorp/terraform-cdk-go/cdktf"
  "github.com/hashicorp/terraform-exec/tfexec"
  tfjson "github.com/hashicorp/terraform-json"
)

func main() {
  tempDir, err := os.MkdirTemp("", "magic-instance-maker-")
  if err != nil {
    return nil, err
  }
  defer os.RemoveAll(tempDir)

  app := cdktf.NewApp(&cdktf.AppOptions{Outdir: jsii.String(tempDir)})
  NewStack(app, name, cluster)
  app.Synth()

  installer := &releases.ExactVersion{
    Product: product.Terraform,
    Version: version.Must(version.NewVersion("1.1.4")),
  }

  execPath, err := installer.Install(context.Background())
  if err != nil {
    log.Fatalf("error installing Terraform: %s", err)
  }

  workingDir := filepath.Join(tempDir, "stacks", name)
  tf, err := tfexec.NewTerraform(workingDir, execPath)
  if err != nil {
    log.Fatalf("error running NewTerraform: %s", err)
  }

  err = tf.Init(context.Background(), tfexec.Upgrade(true))
  if err != nil {
    log.Fatalf("error running Init: %s", err)
  }

  err = tf.Apply(context.Background())
  if err != nil {
    log.Fatalf("error running Apply: %s", err)
  }
}

Run go run main.go again, and all your resources should be live.

Thoughts

cdktf is a really cool project that allows us to actually "code" the infrastructure, and it provides a more convenient way to interact with Terraform programmatically. You can utilize the typical flow control or whatever convention you are already familar with, instead of being limited by Terraform's own DSL (HCL).

Now, what's the catch?

Performance Issues with `cdktf get`

We only have a few providers in cdktf.json and the command still takes a good chunk of time to finish. This might have something to do with compiling a lot of providers' code on every run. Below is some unscientific benchmark:

On a maxed out M1 Max MacBook,

Generated go constructs in the output directory: generated
________________________________________________________
Executed in   75.17 secs    fish           external
   usr time   92.10 secs   63.00 micros   92.10 secs
   sys time   12.05 secs  863.00 micros   12.05 secs

Docker for Mac,

[cdktf-builder 6/6] RUN --mount=type=cache,target=/tmp/terraform-plugin-cache cdktf get  854.4s

We also tried caching providers, but it didn't help. Perhaps this can be greatly improved when Hashicorp starts publishing pre-built providers for Go, or if we can maintain our own registry?

Performance Issues at Build Time

Disclaimer: by no means I am an expert in Go and there must be some optimization you can do to the compiler

Prior to introducing cdktf in our Go program, it takes somewhere around 15 seconds to build the binary during the docker image build. After adding cdktf, it takes over two minutes to build.

Moreover, if you want to build a linux/amd64 image, it takes over 15 minutes to compile! Of course, this is mostly due to the poor performance of qemu emulation.

Faster turnaround is key to development velocity and developers' happiness :(

Unintuitive Usage of cdktf in Go

You may notice the usage of jsii everywhere in the module/stack we defined earlier, and it is required for the Go library to interact with the underlying node runtime.

Lack of Go Support

There is not much documentation for Go example. We are mostly relying on reading TypeScript examples and autocomplete of the generated providers' code to write the module. Again, Go is the one of the cdktf supported languages that lack pre-built providers.

Final Words

Just write TypeScript!

cdktf is still an early project and the support for Go is still experimental. The idea is great, but I wouldn't be comfortable using it in a real project just yet. The Go port definitely deserves more love :)

As I mentioned earlier, HCL and JSON are mostly interchangeable, so we can also handcraft the JSON to represent the Terraform module using any programming language or utilize the HCL parser.

How to backup Hashicorp Vault with Raft storage on Kubernetes

Michael Lin — Sun, 27 Jun 2021 15:41:17 +0000

Context

Our team is experimenting with Hashicorp Vault as our new credentials management solution. Thanks to the offical Vault Helm Chart, we are able to get an almost production-ready vault cluster running on our Kubernetes cluster with minimal effort.

Architecture

Our 5-node vault cluster is highly available by using the provided Integrated Storage Raft backend. The vault cluster is run as a Kubernetes StatefulSet and each node has its own data storage. Each data storage is powered by a Block Storage on IBM Cloud via PersistentVolumeClaim.

The Problem

Unfortunately, the open source vault does not provide an out-of-the-box automated backup solution. It is only offer in Vault Enterprise. Apprently, our team doesn't have a deep pocket to pay for the license fee.

That said, the backup feature is still accessible from cli and HTTP API, just not automated. We utilize the snapshot save from vault cli to perform automated backup using a CronJob running along with the vault Kubernetes deployment. The cronjob will periodically take a snapshot of the vault cluster and upload to our S3 storage.

How?

Prerequisite

You have a working vault cluster
You have sufficient access to the cluster
You have a working S3 storage instance

Setup Policy and Authentication

This is mostly stolen from adfinis-sygroup/vault-raft-backup-agent#approle-authentication

Create a minimal policy for our snapshot agent to perform the backup job.

echo '
path "sys/storage/raft/snapshot" {
   capabilities = ["read"]
}' | vault policy write snapshot -

The approle auth method allows machines or apps to authenticate with Vault-defined roles.

AppRole auth method is perfectly suited for the snapshot agent to authenticate with our vault cluster. Notes role-id and secret-id, you will need them later.

vault auth enable approle
vault write auth/approle/role/snapshot-agent token_ttl=2h token_policies=snapshot
vault read auth/approle/role/snapshot-agent/role-id -format=json | jq -r .data.role_id
vault write -f auth/approle/role/snapshot-agent/secret-id -format=json | jq -r .data.secret_id

Prepare Secrets

Let's save all our sensitive information as Secrets. We will use them later.

apiVersion: v1
kind: Secret
metadata:
  name: vault-snapshot-agent-token
type: Opaque
data:
  # we use gotmpl here
  # you can replace them with base64-encoded value
  VAULT_APPROLE_ROLE_ID: {{ .Values.approle.secretId | b64enc | quote }}
  VAULT_APPROLE_SECRET_ID: {{ .Values.approle.secretId | b64enc | quote }}

apiVersion: v1
kind: Secret
metadata:
  name: vault-snapshot-s3
type: Opaque
data:
  # we use gotmpl here
  # you can replace them with base64-encoded value
  AWS_ACCESS_KEY_ID: {{ .Values.backup.accessKeyId | b64enc | quote }}
  AWS_SECRET_ACCESS_KEY: {{ .Values.backup.secretAccesskey | b64enc | quote }}
  AWS_DEFAULT_REGION: {{ .Values.backup.region | b64enc | quote }}

The CronJob

Let's create the CronJob that actually does the work.

We configure VAULT_ADDR environment variable to http://vault-active.vault.svc.cluster.local:8200. Using vault-active Service can make sure the snapshot request is made against the leader node, assuming you have enabled Service Registration, which is the default. The exact url may vary depending on your vault helm chart deployment release name and targer namespace, learn more.

I may have over-engineered the cronjob by using multiple containers to perform a simple backup and upload task. The intention is to avoid building custom images and I don't want to maintain yet another image.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: vault-snapshot-cronjob
spec:
  schedule: "@every 12h"
  jobTemplate:
    spec:
      template:
        spec:
          volumes:
          - name: share
            emptyDir: {}
          containers:
          - name: snapshot
            image: vault:1.7.2
            imagePullPolicy: IfNotPresent
            command:
            - /bin/sh
            args:
            - -ec
            # The offical vault docker image actually doesn't come with `jq`. You can 
            # - install it during runtime (not a good idea and your security team may not like it)
            # - ship `jq` static binary in a standalone image and mount it using a shared volume from `initContainers`
            # - build your custom `vault` image
            - |
              curl -sS https://webinstall.dev/jq | sh
              export VAULT_TOKEN=$(vault write auth/approle/login role_id=$VAULT_APPROLE_ROLE_ID secret_id=$VAULT_APPROLE_SECRET_ID -format=json | /jq/jq -r .auth.client_token);
              vault operator raft snapshot save /share/vault-raft.snap; 
            envFrom:
            - secretRef:
                name: vault-snapshot-agent-token
            env:
            - name: VAULT_ADDR
              valut: http://vault-active.vault.svc.cluster.local:8200
            volumeMounts:
            - mountPath: /share
              name: share
          - name: upload
            image: amazon/aws-cli:2.2.14
            imagePullPolicy: IfNotPresent
            command:
            - /bin/sh
            args:
            - -ec
            # the script wait untill the snapshot file is available
            # then upload to s3
            # for folks using non-aws S3 like IBM Cloud Object Storage service, add a `--endpoint-url` option
            # run `aws --endpoint-url <https://your_s3_endpoint> s3 cp ...`
            # change the s3://<path> to your desired location
            - |
              until [ -f /share/vault-raft.snap ]; do sleep 5; done;
              aws s3 cp /share/vault-raft.snap s3://vault/vault_raft_$(date +"%Y%m%d_%H%M%S").snap;
            envFrom:
            - secretRef:
                name: vault-snapshot-s3
            volumeMounts:
            - mountPath: /share
              name: share
          restartPolicy: OnFailure

Wrapping Up

Now you have all resources needed to automate vault backup for Raft backend. You can either just run kubectl apply -f * or build your own Helm Chart and distribute on your private chart repository.

Forem: Michael Lin

Deploy Infrastructure using CDK for Terraform with Go

Architecture

The Problem

How does it work?

Thoughts

Performance Issues with cdktf get

Performance Issues at Build Time

Unintuitive Usage of cdktf in Go

Lack of Go Support

Final Words

How to backup Hashicorp Vault with Raft storage on Kubernetes

Context

Architecture

The Problem

How?

Prerequisite

Setup Policy and Authentication

Prepare Secrets

The CronJob

Wrapping Up

Performance Issues with `cdktf get`