Forem: Mickaël Canévet

Terraform Project Design — A parallel with Puppet

Mickaël Canévet — Mon, 03 May 2021 11:57:38 +0000

Introduction

Like Puppet, Terraform provides low-level objects written in a programming language that allows you to manage individual resources.

With Puppet, these low-level objects, the Puppet Types and Providers, written in Ruby, allow you to apply the CRUD paradigm to resources located on managed nodes.

With Terraform, these low-level objects, the Terraform Resources, written in Go, allow you to apply the CRUD paradigm to resources in an API.

On top of this, both solutions provide a DSL: the Puppet DSL for Puppet and the HashiCorp Configuration Language (HCL) for Terraform. They are both declarative languages, allowing you to organize your code for higher re-usability and maintainability.

It's all about abstraction and convergence

As in Puppet, HCL is only a wrapper around the low-level resources, the only objects that actually impact your infrastructure.

These resources ensure the convergence of your infrastructure. All other objects only allow you to organize your code in multiple abstraction layers: classes and defined types in Puppet; modules in Terraform.

In Puppet, you could do something like this:

node 'mynode' {
  user { 'foo':
    ensure => present,
  }
}

However, if you want code re-usability and ease maintenance, you should add abstraction layers in between the node and the user resource.

Since 2012 and Craig Dunn's famous blog post about Designing Puppet, the most described (and probably used) pattern in Puppet code is to use public modules from the Puppet forge, then add 2 levels of abstraction on top of it with Roles and Profiles.

In Terraform, you could also do very simple things without any level of abstraction:

data "aws_ami" "ubuntu" {
  most_recent = true

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  owners = ["099720109477"] # Canonical
}

resource "aws_instance" "web" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t3.micro"

  tags = {
    Name = "HelloWorld"
  }
}

However, if you want to maximize re-usability, you probably want to add some abstraction layers, similar to the Roles and Profiles in Puppet. Terraform allows you to group your resources into modules that can than be instantiated multiple times in your code.

The Roles and Profiles pattern in Puppet usually implements the following layers:

node → role → profiles → component module → resources

In Terraform, we do not manage nodes, so the abstraction stack starts with the workspace instead:

workspace → root module → composition module → resource module → resource

Analogies

Puppet Types vs Terraform Resources

Terraform resources are like Puppet Types: they are the only low-level objects that really do something in your API (or on your Puppet node).

Puppet nodes vs Terraform workspaces

With Puppet, the entry point of your code is the node object which automatically includes the Main class.

Terraform implicitly creates a default workspace for your stack, which instantiates the root module where you can declare your infrastructure.

Puppet Roles vs Terraform root modules

The Roles and Profiles design pattern in Puppet suggests to assign one and only one Role class to your node.

In Terraform, you don't really have the choice because you automatically have one and only one root module per workspace.

Puppet Profiles vs Terraform composition modules

In Puppet, you would code your business logic into Profile classes so that you can reuse it, probably with different parameters, in your various Roles.

In Terraform, you could code your business logic into composition modules that you will instantiate in your root module.

Puppet Forge vs Terraform registry

Just like Puppet has its Forge, Terraform provides a registry for anyone to share their resource modules, which represent the first level of abstraction.

Examples

Puppet

Let's provision a node running a Docker engine and a Traefik reverse proxy. The role will include both classes.

Code Entry Point

The code entry point in Puppet is the manifests/site.pp file. There are multiple ways to achieve node classification in Puppet, one of which is to store the node's role in its certificate. It will then be exposed as a trusted fact.

# manifests/site.pp
$role = $trusted['extensions']['pp_role']
node default {
  include "roles::${role}"
}

Role class

# modules/roles/manifests/docker_traefik_dev.pp
class 'roles::docker_traefik_dev' {
  include profiles::docker
  include profiles::traefik
}

Profile classes

# modules/profiles/manifests/docker.pp
class 'profiles::docker' {
  class { 'docker': }
  class { 'docker-compose': }
}

# modules/profiles/manifests/traefik.pp
class 'profiles::traefik'(
  Boolean $enable_docker = true,
) {
  class { 'traefik':
    enable_docker => $enable_docker,
  }
}

Component modules

Here, we are using 3 component modules that could be published on the Puppet Forge:

docker
docker-compose
traefik

Terraform

In this Terraform example, we create an AWS VPC and an EKS cluster.

Root module

The Terraform root module is the entry point of your code, which declares your stack. Here, it only includes a composition module.

# main.tf
module "cluster" {
  src = "registry.example.com/example-corp/k8s-cluster/aws"

  vpc = {
    name            = "my-vpc"
    cidr            = "10.0.0.0/16"
    azs             = ["eu-west-1a", "eu-west-1b", "eu-west-1c"]
    private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
    public_subnets  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]
  }

  eks = {
    cluster_name    = "my-cluster"
    cluster_version = "1.18"
    worker_groups = [
      {
        instance_type = "m5a.large"
        asg_max_size  = 5
      }
    ]
  }
}

Composition module

The composition module is a reusable piece of code that contains your business logic. Here, it creates a VPC and deploys an EKS cluster on every instantiation.

# variables.tf
variable "vpc" {
  type = map(object({
    name            = string
    cidr            = string
    azs             = list(string)
    private_subnets = list(string)
    public_subnets  = list(string)
  }))
}

variable "eks" {
  type = map(object({
    cluster_name    = string
    cluster_version = string
    worker_groups   = list(any)
  }
}


# main.tf`
module "vpc" {
  source = "terraform-aws-modules/vpc/aws"

  name = var.vpc.name
  cidr = var.vpc.cidr

  azs             = var.vpc.azs
  private_subnets = var.vpc.private_subnets
  public_subnets  = var.vpc.public_subnets

  enable_nat_gateway     = true
  single_nat_gateway     = false
  one_nat_gateway_per_az = true

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb" = "1"
  }

  public_subnet_tags = {
    "kubernetes.io/role/elb" = "1"
  }
}

data "aws_eks_cluster" "cluster" {
  name = module.cluster.cluster_id
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.cluster.cluster_id
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster.token
  load_config_file       = false
  version                = "~> 1.9"
}

module "cluster" {
  source          = "terraform-aws-modules/eks/aws"
  cluster_name    = var.eks.cluster_name
  cluster_version = var.eks.cluster_version
  subnets         = module.vpc.private_subnets
  vpc_id          = module.vpc.vpc_id
  worker_groups   = var.eks.worker_groups
}

You probably want maximum testing of this composition module, such as linting, syntax validation, integration tests…

Resource Module

Here, we are using two public modules available on the Terraform registry (but of course you could load any module from any supported source):

Rules

Common rules for the Puppet's Roles and Profiles design pattern are:

A node includes one role, and one only,
A role includes one or more profiles to define the type of server,
A profile includes and manages component modules to define a logical technical stack,
Component modules manage low-level resources,
Modules should only be responsible for managing aspects of the component they are written for.

In Terraform, we could also do some analogies regarding this rules:

A workspace includes one root module, and one only. Here you don't really have the choice because Terraform imposes this to you.
A root module includes one or more composition modules to define the type of infrastructure,
A composition module includes and manages resource modules to define a logical technical stack,
Resource modules manage low-level resources,
Modules should only be responsible for managing aspects of the component they are written for.

Conclusion

Even though Puppet and Terraform have different purposes, they share similarities and can benefit from the same architecture best practices.

Easily deploy an SKS cluster on Exoscale with Terraform

Mickaël Canévet — Thu, 25 Feb 2021 09:26:01 +0000

Following the recent announcement of Exoscale's managed Kubernetes service, we gave it a test run to deploy our standard stack of tools. As usual, we wanted to do it "as Code", so we chose Terraform for the task.

Since the release of Exoscale's Terraform provider v0.22.0, it is now possible to create SKS clusters as code.

To deploy a cluster you'll need to create all these resources:

An exoscale_sks_cluster,
One or more exoscale_sks_nodepools,
An exoscale_affinity per node pool to ensure that all nodes in a pools are in the same anti-affinity group in case of outage in an hypervisor,
An exoscale_security_group for your node pools,
An exoscale_security_group_rule to allow Calico traffic behind your nodes,
An exoscale_security_group_rule to allow nodePorts access from everywhere,
An exoscale_security_group_rule to allow access to logs and exec.

To ease the deployment of all these resources, we decided to write a Terraform module that we published on the Terraform registry.

In order to use it, simply copy this HCL code:

module "sks" {
  source  = "camptocamp/sks/exoscale"
  version = "0.3.1"

  name = "test"
  zone = "de-fra-1"

  nodepools = {
    "router" = {
      instance_type = "medium"
      size          = 2
    },
    "compute" = {
      instance_type = "small"
      size          = 3
    },
  }
}

output "kubeconfig" {
  value     = module.sks.kubeconfig
  sensitive = true
}

Export your API keys:

$ export EXOSCALE_API_KEY=...
$ export EXOSCALE_API_SECRET=...

Then run:

$ terraform apply

This will deploy an SKS cluster with 2 nodepools (one that we'll dedicate for our Ingress Controller and one to host our applications), one anti-affinity group per nodepool and a security group with proper rules so that everything runs properly (you'll still have to open access to http and https ports if needed).

You can retrieve the kubeconfig for the kube-admin user using this command:

$ terraform output -json kubeconfig | jq -r . > ~/.kube/config

NOTE: make sure to not overwrite a previous cluster configuration or prefer working with environment variables with KUBECONFIG=~/path/to/sks-config

You should then be able to connect to the cluster:

$ kubectl get pods --all-namespaces

And voilà.

4 ways to inject secrets into an application

Mickaël Canévet — Mon, 24 Aug 2020 14:27:40 +0000

Most applications require secrets, for example to connect to a database, communicate with another application using tokens or certificates, define an admin password…

Dealing with this is often a headache. Even when you have a proper secret management tool, it's sometimes a nightmare to inject the secrets into the application where it needs to be used.

The 4 ways

First way: build time

This is probably the worst way to do it.

Examples:

Build a WAR file, or a Docker image or any artifact with a configuration file that contains the secret in plain text.

Pros:

Your deployment tool does not need permissions to decrypt the secrets.

Cons:

Your whole artifact becomes a secret;
You have to rebuild and redeploy when you want to renew a secret.

Second way: deploy time

Examples:

Give a Continuous Deployment pipeline access to a secrets store an let it inject the secrets in the application's configuration files;
Give a Continuous Deployment tool, like ArgoCD, access to a secrets store an let it inject the secrets in the application's configuration files.

Pros:

You don't have to rebuild when you want to renew a secret.

Cons:

Your deployment tool needs permissions to retrieve and decrypt the secrets;
You have to redeploy when you want to renew a secret.

Third way: start time

Examples:

Use Kubernetes Secret Store CSI drivers to inject secrets as volumes or secrets;
Use HashiCorp Vault as a secret store and Vault Agent Injector as an init container to create configuration files before the application starts.

Pros:

Your deployment tool does not need permissions to decrypt the secrets;
You don't have to redeploy when you want to renew a secret.

Cons:

You need to restart your application when you want to renew a secret.

Forth way: run time

Examples:

AWS instance profiles;
HashiCorp Vault as secret store and Vault Agent Injector in sidecar mode.

Pros:

You can renew your secret without even having to restart your application, allowing a dynamic secrets mechanism.

Cons:

You probably need to adapt your application (or not if you already use an SDK that supports it).

Conclusion

Do you know any more ways to inject secrets in your applications? Let me know in the comments!

Use Kustomize to post-render Helm charts in ArgoCD

Mickaël Canévet — Tue, 18 Aug 2020 13:50:18 +0000

In an ideal world you wouldn't have to perform multiple steps for the rendering, but unfortunately we don't live in an ideal world...

Kustomize

Nowadays, most applications that are meant to be deployed in Kubernetes provide a Helm chart to ease deployment. Unfortunately, sometimes the Helm chart is not flexible enough to do what you want to do, so you have to fork and contribute and hope that your contribution is quickly merged upstream so that you don't have to maintain your fork.

Instead of pointing to your fork, you could use Kustomize to apply some post-rendering to your templatized Helm release. This is possible natively since Helm 3.1 using the --post-process flag.

Integration in ArgoCD

At Camptocamp, we use ArgoCD to manage the deployment of our objects into Kubernetes. Let's see how we can use Kustomize to do post-rendering of Helm charts in ArgoCD:

At first, declare a new config management plugin into your argocd-cm configMap (the way to do it depends on the way you deployed ArgoCD):

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  configManagementPlugins: |
    - name: kustomized-helm
      init:
        command: ["/bin/sh", "-c"]
        args: ["helm dependency build || true"]
      generate:
        command: ["/bin/sh", "-c"]
        args: ["helm template . --name-template $ARGOCD_APP_NAME --namespace $ARGOCD_APP_NAMESPACE --include-crds > all.yaml && kustomize build"]

Then add a kustomization.yaml file next to your application's Chart.yaml file:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - all.yaml

patchesJson6902:
  - target:
      group: apps
      version: v1
      kind: Deployment
      name: myapplication
    patch: |-
      - op: remove
        path: /spec/template/spec/securityContext

Now configure your Applications object to use this plugin:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapplication
  namespace: argocd
spec:
  project: myproject
  source:
    path: myapplication
    repoURL: {{ .Values.spec.source.repoURL }}
    targetRevision: {{ .Values.spec.source.targetRevision }}
    plugin:
      name: kustomized-helm
  destination:
    namespace: myproject
    server: {{ .Values.spec.destination.server }}

And... voilà!

Integration in App of Apps

One thing that I often do is to use spec.source.helm in my Application object to pass some values that comes from my app of apps. This is not possible using a configuration plugin as the keys helm and plugin are mutually exclusive.

The workaround I found is to use plugin's envs. You have to change your config management plugin configuration to (note the $HELM_ARGS):

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  configManagementPlugins: |
    - name: kustomized-helm
      init:
        command: ["/bin/sh", "-c"]
        args: ["helm dependency build || true"]
      generate:
        command: ["/bin/sh", "-c"]
        args: ["echo \"$HELM_VALUES\" | helm template . --name-template $ARGOCD_APP_NAME --namespace $ARGOCD_APP_NAMESPACE $HELM_ARGS -f - --include-crds > all.yaml && kustomize build"]

You'll then be able to use this in your Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapplication
  namespace: argocd
spec:
  project: myproject
  source:
    path: myapplication
    repoURL: {{ .Values.spec.source.repoURL }}
    targetRevision: {{ .Values.spec.source.targetRevision }}
    plugin:
      name: kustomized-helm
      env:
        - name: HELM_ARGS
          value: "--set targetRevision={{ .Values.spec.source.targetRevision }}"

  destination:
    namespace: myproject
    server: {{ .Values.spec.destination.server }}

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapplication
  namespace: argocd
spec:
  project: myproject
  source:
    path: myapplication
    repoURL: {{ .Values.spec.source.repoURL }}
    targetRevision: {{ .Values.spec.source.targetRevision }}
    plugin:
      name: kustomized-helm
      env:
        - name: HELM_VALUES
          value: |
            targetRevision: {{ .Values.spec.source.targetRevision }}

  destination:
    namespace: myproject
    server: {{ .Values.spec.destination.server }}