Forem: Kubestack

Goodbye Cloud, Hello CLI: Sunsetting Kubestack Cloud

Philipp Strube — Tue, 09 May 2023 19:53:11 +0000

I've recently released a major update for Kubestack, the Terraform framework for Kubernetes platform engineering teams. This update moves all functionality previously provided by Kubestack Cloud into the kbst CLI.

I decided to make this change, because Kubestack Cloud was only able to provide a better developer experience on day one. But, once exported to Terraform, the UI was not helpful any more on day two and all following days.

But my goal is to improve the developer experience and day-to-day lives of platform engineering teams at all times. This latest kbst release is a major step towards achieving this goal.

If this is the first time you hear about Kubestack Cloud: Kubestack Cloud was a browser based UI and allowed users to design a Kubernetes platform by following a step-by-step wizard and then exporting and downloading the designed platform's Terraform code.

However, the disconnect between the UI and the code in the repository on a developer's local machine, diminished the value of Kubestack Cloud on day two and beyond. To address this issue, I moved all this functionality into the kbst CLI, where access to the local code is easier.

The kbst CLI, which previously also only scaffolded new repositories, now has CRUD (create, read, update, delete) functionality for clusters, node-pools, and services. This means users can use the CLI to scaffold Terraform code to add or remove clusters, node-pools, or services inside their existing Kubestack repositories.

If you want to see the new CLI in action, give the updated tutorial a try or read the documentation on adding and removing cluster modules, node pool modules or service modules.

But if you'd like to learn more about how this works under the hood, keep reading.

How this works

If you're already familiar with Kubestack, you know that Kubestack repositories follow a convention-over-configuration approach to define the clusters, node pools, and services that make up a Kubernetes platform in a single Terraform codebase. At the root of each repository, there are several .tf files that follow a specific naming convention. These files contain module calls that define each platform component.

To add or remove components, or update the versions of existing component modules, the kbst CLI parses the necessary subset of Terraform code to understand the components of the platform. You can list the Kubestack component modules it discovered using the kbst list command. By appending --all to the list command, you can also see any non-Kubestack modules.

You can add node pools or services to existing clusters or add more clusters from the same or even a different cloud provider. The CLI will scaffold the additional required .tf files and update the Dockerfile's FROM line to specify the correct image, in case of changing from a single to a multi-cloud environment or vice versa. Likewise, it will also remove module calls and the respective .tf files if you remove a service, a node pool or even a cluster from your platform.

But don't worry, the kbst CLI only changes local files and does never change any cloud or Kubernetes resources.

You can use it to avoid writing repetitive boilerplate code or manually deleting module calls and Terraform files, while still owning your codebase and retaining the ability to extend or modify the code to meet specific needs.

Once you're happy with the code, you can follow the Kubestack GitOps workflow to peer-review, validate, and promote changes to your platform's environments as usual.

In conclusion, the shift from Kubestack Cloud to the kbst CLI provides a better developer experience not only on day one, but also on day two and makes it easier for platform engineering teams to manage their Kubernetes based platforms.

What happened to the platforms I designed with Kubestack Cloud?

If you have previously designed a platform with Kubestack Cloud, you can sign in with your existing user and will see instructions how to scaffold your existing platforms using the new CLI.

Here's an example screenshot of how that will look like.

Getting rigorous about investing in the Kubestack project

Philipp Strube — Sun, 27 Nov 2022 14:52:30 +0000

Sometimes when you spend a long time solving a problem, it makes it harder to see your solution clearly.

In 12 years helping companies adopt modern cloud computing, I saw so many of the same snags repeating across multiple organizations. From these lessons, I built Kubestack as guardrails to make it easier to avoid the pain in the first place. Kubestack has been used in companies large and small for years, but I haven’t always known where it has been most helpful to its users. Without knowing this, it’s hard to bring more people in as both users and contributors. So earlier this year, I contracted with Ana Hevesi to support Kubestack's open source efforts.

Ana operates a developer experience consultancy. After working in technical community building for companies like Stack Overflow and Nodejitsu, Ana now works with devtools founders to create evidence-based approaches for growing their ecosystems.

We started by doing some research into how Kubestack serves your goals.

Research methods

Our objective was to learn about users’ career trajectories and aspirations, and get a clear picture of what role Kubestack plays in your success.

Ana recommended we aim for 5 interviews, citing it as a good “goldilocks zone” for initial quantity of data to work with. I then reached out to a spectrum of new and long-tenured Kubestack users to ask for their time in a 60 minute user interview.

Ana wrote a standard interview script which included bandwidth for conversational “side quests.” After, Ana analyzed recordings, picked out repeat themes, and came to me with conclusions and recommendations.

Areas of positive impact

Kubestack helps careers

Participants attributed their use of Kubestack to positive career outcomes, such as developing a reputation for reliably delivering for users, or scaling on a tight timeframe with limited prior experience. Others reported it was a key learning tool when they were just starting as platform engineers.

Works so well it disappears

The most consistent feedback we received was that users can assume Kubestack is just going to work. Multiple participants had been relying on the framework for many months without needing to give it a second thought.

Areas to improve

Documentation for advanced features needs improvement

Kubestack works great for months on end for most orgs setting up their first K8s cluster, but those who wished to modify Kubestack outside of existing use cases told us error messages and upgrade processes were opaque.

Backwards compatibility and multi-cloud support presents friction to open source contributions

Adding new features requires working knowledge of both Terraform and cloud provider functionality across historical versions, and at times, their interactions with one another. Furthermore, while Kubestack is committed to supporting EKS, AKS, and GKE, a contributor may wish to implement functionality for only one of these cloud providers. Inviting more PRs from a wider array of contributors necessitates a plan for tiered support of legacy versions or defining contributor scope to accommodate this complexity.

How we’re applying these findings

Connecting with the people who need us most

Kubestack makes a huge impact on early-stage teams and emerging professionals. We’re exploring ways to better tailor our communication and outreach to make sure they know about the opportunities this framework provides, improving both adoption and contributions to the project. Kubestack only succeeds because you succeed.

Benefits before features

The current iteration of Kubestack’s landing page assumes a fairly high level of existing knowledge of the platform engineering space. As such, an upcoming iteration of the Kubestack site will aim to engage folks who aren’t already deep in the jargon and progressively bring them into the fold, while still being legible to seasoned professionals.

Open source participation onramps

Since enabling users to learn from each other and communicating where the project is going is an important part of growing an open source community, we’ll be experimenting with office hours and public communication about recent releases. We’ll have scheduling details coming soon.

Leveling up, together

I created Kubestack so that folks coming to Kubernetes for the first time could take immediate advantage of the separation of concerns that containers provide. User research says that this works as intended!

Now comes the iterative task of communicating my own knowledge and experience in ways that make it easier to build together, while learning from your use of the project to fill in its gaps. Ultimately, the intent is a healthy community where we’re all working together to make the project better serve your needs.

Finally, a big thank you to Tomas, AJ, Brendan, Christoph, and Mark for your time and candor. Kubestack is better for it.

A Better Way to Provision Kubernetes Resources Using Terraform

Philipp Strube — Wed, 04 May 2022 18:01:47 +0000

Terraform is immensely powerful when it comes to defining and maintaining infrastructure as code. In combination with a declarative API, like a cloud provider API, it can determine, preview, and apply changes to the codified infrastructure.

Consequently, it is common for teams to use Terraform to define the infrastructure of their Kubernetes clusters. And as a platform to build platforms, Kubernetes commonly requires a number of additional services before workloads can be deployed. Think of ingress controllers or logging and monitoring agents and so on. But despite Kubernetes' own declarative API, and the obvious benefits of maintaining a cluster's infrastructure and services from the same infrastructure as code repository, Terraform is far from the first choice to provision Kubernetes resources.

With Kubestack, the open-source Terraform framework I maintain, I'm on a mission to provide the best developer experience for teams working with Terraform and Kubernetes. And unified provisioning of all platform components, from cluster infrastructure to cluster services, is something I consider crucial in my relentless pursuit of said developer experience.

Because of that, the two common approaches to provision Kubernetes resources using Terraform never really appealed to me.

On the one hand, there's the Kubernetes provider. And while it integrates Kubernetes resources into Terraform, maintaining the Kubernetes resources in HCL is a lot of effort. Especially for Kubernetes YAML you consume from upstream. On the other hand, there are the Helm provider and the Kubectl provider. These two use native YAML instead of HCL, but do not integrate the Kubernetes resources into the Terraform state and, as a consequence, lifecycle.

I believe my Kustomization provider based modules are a better alternative because of three distinct benefits:

Like Kustomize, the upstream YAML is left untouched, meaning upstream updates require minimal maintenance effort.
By defining the Kustomize overlay in HCL, all Kubernetes resources are fully customizable using values from Terraform.
Each Kubernetes resource is tracked individually in Terraform state, so diffs and plans show the changes to the actual Kubernetes resources.

To make these benefits less abstract, let's compare my Nginx ingress module with one using the Helm provider to provision Nginx ingress.

The Terraform configuration for both examples is available in this repository. Let's take a look at the Helm module first.

The Helm-based module

Usage of the module is straightforward. First, configure the Kubernetes and Helm providers.

provider "kubernetes" {
  config_path    = "~/.kube/config"
}

provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
  }
}

Then define a kubernetes_namespace and call the release/helm module.

resource "kubernetes_namespace" "nginx_ingress" {
  metadata {
    name = "ingress-nginx"
  }
}

module "nginx_ingress" {
  source  = "terraform-module/release/helm"
  version = "2.7.0"

  namespace  = kubernetes_namespace.nginx_ingress.metadata.0.name
  repository = "https://kubernetes.github.io/ingress-nginx"

  app = {
    name          = "ingress-nginx"
    version       = "4.1.0"
    chart         = "ingress-nginx"
    force_update  = true
    wait          = false
    recreate_pods = false
    deploy        = 1
  }

  set = [
    {
      name  = "replicaCount"
      value = 2
    }
  ]
}

If you now run a terraform plan for this configuration, you see the resources to be created.

Terraform will perform the following actions:

  # kubernetes_namespace.nginx_ingress will be created
  + resource "kubernetes_namespace" "nginx_ingress" {
      + id = (known after apply)

      + metadata {
          + generation       = (known after apply)
          + name             = "ingress-nginx"
          + resource_version = (known after apply)
          + uid              = (known after apply)
        }
    }

  # module.nginx_ingress.helm_release.this[0] will be created
  + resource "helm_release" "this" {
      + atomic                     = false
      + chart                      = "ingress-nginx"
      + cleanup_on_fail            = false
      + create_namespace           = false
      + dependency_update          = false
      + disable_crd_hooks          = false
      + disable_openapi_validation = false
      + disable_webhooks           = false
      + force_update               = true
      + id                         = (known after apply)
      + lint                       = true
      + manifest                   = (known after apply)
      + max_history                = 0
      + metadata                   = (known after apply)
      + name                       = "ingress-nginx"
      + namespace                  = "ingress-nginx"
      + recreate_pods              = false
      + render_subchart_notes      = true
      + replace                    = false
      + repository                 = "https://kubernetes.github.io/ingress-nginx"
      + reset_values               = false
      + reuse_values               = false
      + skip_crds                  = false
      + status                     = "deployed"
      + timeout                    = 300
      + values                     = []
      + verify                     = false
      + version                    = "4.1.0"
      + wait                       = false
      + wait_for_jobs              = false

      + set {
          + name  = "replicaCount"
          + value = "2"
        }
    }

Plan: 2 to add, 0 to change, 0 to destroy.

And this is the key issue with how Helm is integrated into the Terraform workflow. The plan does not tell you what Kubernetes resources will be created for the Nginx ingress controller. And neither are the Kubernetes resources tracked in Terraform state, as shown by the apply output.

kubernetes_namespace.nginx_ingress: Creating...
kubernetes_namespace.nginx_ingress: Creation complete after 0s [id=ingress-nginx]
module.nginx_ingress.helm_release.this[0]: Creating...
module.nginx_ingress.helm_release.this[0]: Creation complete after 3s [id=ingress-nginx]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Similarly, if planning a change, there's again no way to tell what the changes to the Kubernetes resources will be.

So if you increase the replicaCount value of the Helm chart, the terraform plan will merely show the change to the helm_release resource.

set = [
  {
    name = "replicaCount"
    value = 3
  }
]

What will the changes to the Kubernetes resources be? And more importantly, is it a simple in-place update, or does it require a destroy-and-recreate? Looking at the plan, you have no way of knowing.

Terraform will perform the following actions:

  # module.nginx_ingress.helm_release.this[0] will be updated in-place
  ~ resource "helm_release" "this" {
        id                         = "ingress-nginx"
        name                       = "ingress-nginx"
        # (27 unchanged attributes hidden)

      - set {
          - name  = "replicaCount" -> null
          - value = "2" -> null
        }
      + set {
          + name  = "replicaCount"
          + value = "3"
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

The Kustomize-based module

Now, let's take a look at the same steps for the Kustomize-based module. Usage is similar. First require the kbst/kustomization provider and configure it.

terraform {
  required_providers {
    kustomization = {
      source = "kbst/kustomization"
    }
  }
}

provider "kustomization" {
  kubeconfig_path = "~/.kube/config"
}

Then call the nginx/kustomization module.

module "nginx_ingress" {
  source  = "kbst.xyz/catalog/nginx/kustomization"
  version = "1.1.3-kbst.1"

  configuration_base_key = "default"
  configuration = {
    default = {
      replicas = [{
        name  = "ingress-nginx-controller"
        count = 2
      }]
    }
  }
}

Unlike for the Helm-based module though, when you run terraform plan now you will see each Kubernetes resource and its actual configuration individually. To keep this blog post palatable, I show the details for the namespace only.

Terraform will perform the following actions:

  # module.nginx_ingress.kustomization_resource.p0["_/Namespace/_/ingress-nginx"] will be created
  + resource "kustomization_resource" "p0" {
      + id       = (known after apply)
      + manifest = jsonencode(
            {
              + apiVersion = "v1"
              + kind       = "Namespace"
              + metadata   = {
                  + annotations = {
                      + "app.kubernetes.io/version"      = "v0.46.0"
                      + "catalog.kubestack.com/heritage" = "kubestack.com/catalog/nginx"
                      + "catalog.kubestack.com/variant"  = "base"
                    }
                  + labels      = {
                      + "app.kubernetes.io/component"  = "ingress-controller"
                      + "app.kubernetes.io/instance"   = "ingress-nginx"
                      + "app.kubernetes.io/managed-by" = "kubestack"
                      + "app.kubernetes.io/name"       = "nginx"
                    }
                  + name        = "ingress-nginx"
                }
            }
        )
    }

  # module.nginx_ingress.kustomization_resource.p1["_/ConfigMap/ingress-nginx/ingress-nginx-controller"] will be created
  # module.nginx_ingress.kustomization_resource.p1["_/Service/ingress-nginx/ingress-nginx-controller"] will be created
  # module.nginx_ingress.kustomization_resource.p1["_/Service/ingress-nginx/ingress-nginx-controller-admission"] will be created
  # module.nginx_ingress.kustomization_resource.p1["_/ServiceAccount/ingress-nginx/ingress-nginx"] will be created
  # module.nginx_ingress.kustomization_resource.p1["_/ServiceAccount/ingress-nginx/ingress-nginx-admission"] will be created
  # module.nginx_ingress.kustomization_resource.p1["apps/Deployment/ingress-nginx/ingress-nginx-controller"] will be created
  # module.nginx_ingress.kustomization_resource.p1["batch/Job/ingress-nginx/ingress-nginx-admission-create"] will be created
  # module.nginx_ingress.kustomization_resource.p1["batch/Job/ingress-nginx/ingress-nginx-admission-patch"] will be created
  # module.nginx_ingress.kustomization_resource.p1["networking.k8s.io/IngressClass/_/nginx"] will be created
  # module.nginx_ingress.kustomization_resource.p1["rbac.authorization.k8s.io/ClusterRole/_/ingress-nginx"] will be created
  # module.nginx_ingress.kustomization_resource.p1["rbac.authorization.k8s.io/ClusterRole/_/ingress-nginx-admission"] will be created
  # module.nginx_ingress.kustomization_resource.p1["rbac.authorization.k8s.io/ClusterRoleBinding/_/ingress-nginx"] will be created
  # module.nginx_ingress.kustomization_resource.p1["rbac.authorization.k8s.io/ClusterRoleBinding/_/ingress-nginx-admission"] will be created
  # module.nginx_ingress.kustomization_resource.p1["rbac.authorization.k8s.io/Role/ingress-nginx/ingress-nginx"] will be created
  # module.nginx_ingress.kustomization_resource.p1["rbac.authorization.k8s.io/Role/ingress-nginx/ingress-nginx-admission"] will be created
  # module.nginx_ingress.kustomization_resource.p1["rbac.authorization.k8s.io/RoleBinding/ingress-nginx/ingress-nginx"] will be created
  # module.nginx_ingress.kustomization_resource.p1["rbac.authorization.k8s.io/RoleBinding/ingress-nginx/ingress-nginx-admission"] will be created
  # module.nginx_ingress.kustomization_resource.p2["admissionregistration.k8s.io/ValidatingWebhookConfiguration/_/ingress-nginx-admission"] will be created

Plan: 19 to add, 0 to change, 0 to destroy.

Applying, again, has all the individual Kubernetes resources. And because the modules use explicit depends_on to handle namespaces and CRDs first and webhooks last, resources are reliably applied in the correct order.

module.nginx_ingress.kustomization_resource.p0["_/Namespace/_/ingress-nginx"]: Creating...
module.nginx_ingress.kustomization_resource.p0["_/Namespace/_/ingress-nginx"]: Creation complete after 0s [id=369e8643-ad33-4eb4-95dc-f506cef4a198]
module.nginx_ingress.kustomization_resource.p1["rbac.authorization.k8s.io/RoleBinding/ingress-nginx/ingress-nginx"]: Creating...
module.nginx_ingress.kustomization_resource.p1["batch/Job/ingress-nginx/ingress-nginx-admission-create"]: Creating...

...

module.nginx_ingress.kustomization_resource.p1["batch/Job/ingress-nginx/ingress-nginx-admission-patch"]: Creation complete after 1s [id=58346878-70bd-42f2-af61-2730e3435ca7]
module.nginx_ingress.kustomization_resource.p1["_/ServiceAccount/ingress-nginx/ingress-nginx"]: Creation complete after 0s [id=f009bbb7-7d2e-4f28-a826-ce133c91cc15]
module.nginx_ingress.kustomization_resource.p2["admissionregistration.k8s.io/ValidatingWebhookConfiguration/_/ingress-nginx-admission"]: Creating...
module.nginx_ingress.kustomization_resource.p2["admissionregistration.k8s.io/ValidatingWebhookConfiguration/_/ingress-nginx-admission"]: Creation complete after 1s [id=3185b09f-1f67-4079-b44f-de01bff44bd2]

Apply complete! Resources: 19 added, 0 changed, 0 destroyed.

Naturally, it also means that if you increase the replica count like this...

replicas = [{
  name = "ingress-nginx-controller"
  count = 3
}]

...the terraform plan shows which Kubernetes resources will change and what the diff is.

Terraform will perform the following actions:

  # module.nginx_ingress.kustomization_resource.p1["apps/Deployment/ingress-nginx/ingress-nginx-controller"] will be updated in-place
  ~ resource "kustomization_resource" "p1" {
        id       = "81e8ff18-6c6c-440d-bd8b-bf5f0d016953"
      ~ manifest = jsonencode(
          ~ {
              ~ spec       = {
                  ~ replicas             = 2 -> 3
                    # (4 unchanged elements hidden)
                }
                # (3 unchanged elements hidden)
            }
        )
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Maybe more importantly even, the Kustomization provider will also correctly show if a resource can be changed using an in-place update. Or if a destroy-and-recreate is required because there is a change to an immutable field, for example.

This is the result of two things:

That, as you've just seen, every Kubernetes resource is handled individually in Terraform state, and
that the Kustomization provider uses Kubernetes' server-side dry-runs to determine the diff of each resource.

Based on the result of that dry-run, the provider instructs Terraform to create an in-place or a destroy-and-recreate plan.

So, as an example of such a change, imagine you need to change spec.selector.matchLabels. Since matchLabels is an immutable field, you will see a plan that states that the Deployment resource must be replaced. And you will see 1 to add and 1 to destroy in the plan's summary.

Terraform will perform the following actions:

  # module.nginx_ingress.kustomization_resource.p1["apps/Deployment/ingress-nginx/ingress-nginx-controller"] must be replaced
-/+ resource "kustomization_resource" "p1" {
      ~ id       = "81e8ff18-6c6c-440d-bd8b-bf5f0d016953" -> (known after apply)
      ~ manifest = jsonencode(
          ~ {
              ~ metadata   = {
                  ~ labels      = {
                      + example-selector               = "example"
                        # (6 unchanged elements hidden)
                    }
                    name        = "ingress-nginx-controller"
                    # (2 unchanged elements hidden)
                }
              ~ spec       = {
                  ~ replicas             = 2 -> 3
                  ~ selector             = {
                      ~ matchLabels = {
                          + example-selector               = "example"
                            # (4 unchanged elements hidden)
                        }
                    }
                  ~ template             = {
                      ~ metadata = {
                          ~ labels      = {
                              + example-selector               = "example"
                                # (4 unchanged elements hidden)
                            }
                            # (1 unchanged element hidden)
                        }
                        # (1 unchanged element hidden)
                    }
                    # (2 unchanged elements hidden)
                }
                # (2 unchanged elements hidden)
            } # forces replacement
        )
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Try it yourself

You can find the source code for the comparison on GitHub if you want to experiment with the differences yourself.

If you want to try the Kustomize modules yourself, you can either use one of the modules from the catalog that bundle upstream YAML, like the Prometheus operator, Cert-Manager, Sealed secrets, or Tekton, for example.

But this doesn't only work for upstream services. There is also a module that can be used to provision any Kubernetes YAML in the exact same way as the catalog modules - called the custom manifest module.

Get involved

Currently, the number of services available from the catalog is still limited.

If you want to get involved, you can also find the catalog source on GitHub.

Photo by Vladislav Babienko on Unsplash

Deploying Prometheus Operator via the Kubestack Catalog

Josh Maxwell — Wed, 15 Dec 2021 13:33:17 +0000

Introduction

This walkthrough assumes you have already followed parts one, two, and three of the official Kubestack tutorial and at least have a local development cluster running via kbst local apply.

In this walkthrough we will explore using the Kubestack Catalog to install a Prometheus Operator and collect metrics from an example Go application.

Major Disclaimer:
In the interest of time and reducing technical complexity there is a very strong anti-pattern present in this walkthrough.

Best practice dictates that infrastructure and application manifests be stored in separate repositories so they can be worked on and deployed independently.

In this walkthrough both the Go application and the Kubestack infrastructure manifests we create will be stored in the same repository for simplicity sake while deploying to the local development environment (see the Conclusion for an explanation of how this would look following best practices).

1 - Configure Local Development Environment

Before we install the Prometheus Operator, we need a few additional tools installed in our local development environment in order more easily verify that our configuration is working.

1.1 - Install Go Locally

Follow the instructions at https://go.dev/doc/install to install Go for your local development environment. We need this to build our example application as well as to install another tool below.

1.2 - Install and Configure `kubectl`

kubectl will be used primarily for two reasons. First, to verify which resources are deployed in our k8s cluster. Second, to forward ports and access resources inside the k8s cluster from our local development environment.

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
curl -LO "https://dl.k8s.io/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl.sha256"
echo "$(<kubectl.sha256) kubectl" | sha256sum --check
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
kubectl version --client

At this point if kubectl is successfully installed you should see output similar to the following:

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:41:28Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}

If you're having issues installing kubectl please refer the offical documentation.

1.3 - Install and Configure `kind`

kind (Kubernetes in Docker) will be used to export the cluster configuration file that kubectl needs to access the cluster.

GO111MODULE="on" go get sigs.k8s.io/kind@v0.11.1
kind version
kind get clusters
kind export kubeconfig --name <CLUSTER_NAME>
kubectl get namespaces

At this point if kind has been installed correctly and the kubectl config has been exported successfully, you should see something similar to the following:

NAME                  STATUS   AGE
default               Active   6h27m
ingress-nginx         Active   6h26m
kube-node-lease       Active   6h27m
kube-public           Active   6h27m
kube-system           Active   6h27m
local-path-storage    Active   6h27m

If you're having issues installing kind please refer the official documentation. Otherwise, congrats! You're ready to move on to the next part of the tutorial.

NOTE: kind is only needed for local development environments since that is how Kubestack deploys your environment locally. If you want to follow the rest of the tutorial using your cloud environment instead, you will need to download the kubectl config file from that cluster and import it locally so kubectl can access that cluster instead. Here are some resources about exporting that configuration file: EKS, AKS, GKE.

2 - Deploy an Example Go Application

Now we're going to create an example application to emit some metrics for us to collect with Prometheus.

2.1 - Create a Go Application

Create a new GitHub repository to host your Example Go Application.

Create a new Go Module in the repository with go mod init github.com/<account>/<repo>.

Create a main.go file in the repository with the following content:

package main

import (
    "net/http"
    "time"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

func recordMetrics() {
    go func() {
        for {
            opsProcessed.Inc()
            time.Sleep(2 * time.Second)
        }
    }()
}

var (
    opsProcessed = promauto.NewCounter(prometheus.CounterOpts{
        Name: "app_go_prom_processed_ops_total",
        Help: "The total number of processed events",
    })
)

func main() {
    println("Starting app-go-prom on port :2112")

    recordMetrics()

    http.Handle("/metrics", promhttp.Handler())
    http.ListenAndServe(":2112", nil)
}

With main.go created run go mod tidy to create go.mod and go.sum (these files should be commited to the repo).

2.2 - Create a Dockerfile that Runs Go Application

Create a Dockerfile in the repository with the following content:

FROM golang:1.17-alpine

WORKDIR /app

COPY go.mod ./
COPY go.sum ./
RUN go mod download

COPY *.go ./
RUN go build -o /app-go-prom

EXPOSE 2112

CMD [ "/app-go-prom" ]

2.3 - Create a GitHub Action Pipeline to Build and Publish Docker Image

Create a .github/workflows/docker-publish.yml file in the repository with the following content:

name: Docker Publish

on:
  push:
    branches: [ main ]
    tags: [ 'v*' ]
  pull_request:
    branches: [ main ]

env:
  REGISTRY: ghcr.io
  # github.repository as <account>/<repo>
  IMAGE_NAME: ${{ github.repository }}


jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write

    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      # https://github.com/docker/login-action
      - name: Log into registry ${{ env.REGISTRY }}
        if: github.event_name != 'pull_request'
        uses: docker/login-action@v1
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      # https://github.com/docker/metadata-action
      - name: Extract Docker metadata
        id: meta
        uses: docker/metadata-action@v3
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}

      # https://github.com/docker/build-push-action
      - name: Build and push Docker image
        uses: docker/build-push-action@v2
        with:
          context: .
          push: ${{ github.event_name != 'pull_request' }}
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}

This will build and publish a docker image to the GitHub Registry associated to the repository you created. This image can be used inside your k8s cluster.

In order to trigger this to build an image tagged with latest (rather than the current branch) you will need to tag the commit similar to the following:

git checkout main
git pull
git tag v0.1.1
git push origin v0.1.1

Once the pipeline completes you should be able verify by pulling the image with docker pull ghcr.io/<account>/<repo>:latest.

2.4 - Create an Application Manifest for Kubestack to Deploy in the Cluster

This requires two files to be created in your Kubestack IAC repository:

eks_zero_applications.tf

module "application_custom_manifests" {
  providers = {
    kustomization = kustomization.eks_zero
  }

  source  = "kbst.xyz/catalog/custom-manifests/kustomization"
  version = "0.1.0"

  configuration = {
    apps = {

      resources = [
        "${path.root}/manifests/applications/app-go-prom.yaml"
      ]

      common_labels = {
        "env" = terraform.workspace
      }
    }
    ops = {}
    loc = {}
  }

}

manifests/applications/app-go-prom.yaml

apiVersion: apps/v1
kind: Deployment 
metadata:
  name: app-go-prom
  namespace: default
  labels:
    app: app-go-prom
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-go-prom
  template:
    metadata:
      labels:
        app: app-go-prom
    spec:
      containers:
        - image: ghcr.io/<account>/<repo>:latest
          name: app-go-prom
          ports:
            - containerPort: 2112

---

apiVersion: v1
kind: Service
metadata:
  name: app-go-prom-svc
  namespace: default
  labels:
    app: app-go-prom
spec:
  selector:
    app: app-go-prom
  ports:
    - name: metrics
      port: 2112
      targetPort: 2112
      protocol: TCP
  type: LoadBalancer

With these files created you may need to ^c out of the kbst local apply and run it again if watch doesn't pick up changes in custom manifest files.

2.5 - Verify that the Go Application is Running and Emitting Metrics

Note about this section:
If you ever destroy and re-apply the local cluster you will need to run the kind export kubeconfig --name <CLUSTER_NAME> command from above again to get a fresh kubectl config in order for kubectl to work. The cluster name will probably be the same, but if you need to find it you can always run kind get clusters to figure it out.

Once kbst has finished applying the changes let's verify that the pod is running and that it is emitting metrics as we expect.

kubectl get pods

You should see output similar to the following if everything is working correctly:

NAME                            READY   STATUS    RESTARTS   AGE
app-go-prom-6f9576879d-hvdr9    1/1     Running   0          32h

If STATUS is not "Running" there is an error. You can use kubectl logs <go-app-pod-name> to check the pod logs and fix any errors.

Once your Go application pod is running, lets forward the port from the associated service to our localhost and check for metrics:

kubectl get services

With the service name in hand run:

kubectl port-forward service/app-go-prom-svc 2112
curl localhost:2112/metrics

You should see a bunch of metrics spit out at this point, including the one we created in our Go application.

# HELP app_go_prom_processed_ops_total The total number of processed events
# TYPE app_go_prom_processed_ops_total counter
app_go_prom_processed_ops_total 25058
...

If you run the curl command a few more times you should see our metric increasing steadily as we expect.

You can stop the port forward and move on to the next section now.

3 - Install Prometheus Operator via the Kubestack Catalog

Now we are going to install the Prometheus Operator following the instructions in the Kubestack Catalog.

This consists of 3 steps:

Adding the Prometheus Operator module to the cluster
Configuring read-only access policies to monitoring targets
Specifying which target services to monitor

3.1 - Adding the Prometheus Operator module to the cluster

Create an eks_zero_services.tf file in the root of the repo with the following content:

module "eks_zero_prometheus" {
  providers = {
    kustomization = kustomization.eks_zero
  }

  source  = "kbst.xyz/catalog/prometheus/kustomization"
  version = "0.51.1-kbst.0"

  configuration = {
    apps = {
      additional_resources = [
        "${path.root}/manifests/services/prometheus-default-instance.yaml",
        "${path.root}/manifests/services/prometheus-service-monitors.yaml"
      ]
    }
    ops = {}
    loc = {}
  }
}

This file specifies to add the eks_zero_prometheus module to the eks_zero kustomization provider. If you have customized the name of your provider make sure to update that here as well.

3.2 - Create a Default Instance with permissions to monitor targets

Now we will create first file referenced as additional_resources above.

Create manifests/services/prometheus-default-instance.yaml with the following contents:

---

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: default-instance
  namespace: default
  labels:
    prometheus: default-instance
spec:
  serviceAccountName: prometheus-default-instance
  serviceMonitorSelector:
    matchLabels:
      prometheus-instance: default-instance
  resources:
    requests:
      memory: 2Gi

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: prometheus-default-instance
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus-instance
subjects:
- kind: ServiceAccount
  name: prometheus-default-instance
  namespace: default

---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus-default-instance
  namespace: default

This file contains 3 key components:

The Prometheus default-instance
The RoleBinding permissions
The Service Account

The default-instance is our Prometheus server that collects the metrics and serves the Prometheus UI.
The other components are used to grant the needed permissions to read the metrics we'll specify in the next section.

3.3 - Specify which targets to monitor

Finally, we will create a ServiceMonitor that ties everything together.

Create the manifests/services/prometheus-service-monitors.yaml file with the following contents:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: app-go-prom-monitor
  namespace: default
  labels:
    prometheus-instance: default-instance
spec:
  selector:
    matchLabels:
      app: app-go-prom
  endpoints:
  - port: metrics

There are 3 important pieces here to connect everything:

metadata.labels here needs to exactly match the spec.serviceMonitorSelector.matchLabels from the default-instance
spec.selector.matchLables here needs to exactly match the metadata.labels of the Deployment from the Go application manifest
spec.endpoints.port needs to match the spec.ports.name of the Service from the Go application manifest

Once all those pieces are in place you can once again run kbst local apply to pickup the new manifests.

3.4 - Verify all Prometheus components are Running

Note about this section:
If you ever destroy and re-apply the local cluster you will need to run the kind export kubeconfig --name <CLUSTER_NAME> command from above again to get a fresh kubectl config in order for kubectl to work. The cluster name will probably be the same, but if you need to find it you can always run kind get clusters to figure it out.

Similar to how we verified our Go application was working let's check our Prometheus Operator.

Once kbst has finished applying the changes let's verify that the Prometheus Operator and supporting components have been successfully deployed:

kubectl get all --namespace operator-prometheus

You should see output similar to the following:

NAME                                       READY   STATUS    RESTARTS   AGE
pod/prometheus-operator-775545dc6b-qffng   1/1     Running   0          40h

NAME                          TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
service/prometheus-operator   ClusterIP   None         <none>        8080/TCP   40h

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/prometheus-operator   1/1     1            1           40h

NAME                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/prometheus-operator-775545dc6b   1         1         1       40h

Now let's check that the default-instance pod is running:

kubectl get pods

You should see there is now a prometheus-default-instance-0:

NAME                            READY   STATUS    RESTARTS   AGE
app-go-prom-6f9576879d-hvdr9    1/1     Running   0          33h
prometheus-default-instance-0   2/2     Running   0          33h

If STATUS is not "Running" there is an error. You can use kubectl logs prometheus-default-instance-0 to check the pod logs and fix any errors.

Lastly, let's verify that our ServiceMonitor was created:

kubectl get ServiceMonitors

You should see something similar to:

NAME                  AGE
app-go-prom-monitor   33h

If you don't have any errors, proceed to the final section and let's verify that Prometheus is correctly collecting our metrics.

4 - View the Metrics in the Prometheus UI

Now that we've got everything deployed and Running let's take one final step to verify that everything is working.

Like we did before with our Go application, we now need to forward the Prometheus port to access the UI:

kubectl port-forward prometheus-default-instance-0 9090

Now from your local development environment open a web browser and navigate to localhost:9090.

If everything has been successful to this point you should be greeted with the Prometheus dashboard.

Enter the metric name into the search box that we created in our Go application; app_go_prom_processed_ops_total, and click "Execute".
You will see the metric metadata and count displayed below the search box, similar to the following:

app_go_prom_processed_ops_total{container="app-go-prom", endpoint="metrics", instance="10.244.1.4:2112", job="app-go-prom-svc", namespace="default", pod="app-go-prom-6f9576879d-hvdr9", service="app-go-prom-svc"}  474

Conclusion

Congratulations you have successfully deploy the Prometheus Operator, created an example service emitting metrics, and configured everything to collect those metrics. That is the backbone you'll need to have visibility into the metrics of your new cluster.

From here you could extend your metrics infrastructure by;

adding additional applications / services and their associated ServiceMonitors,
adding a Grafana deployment to create dashboards of your metrics,
configuring an external instance of Prometheus to collect all your metrics,
etc.

The next steps if you'd like would be to go beyond metrics and browse the Kubestack Catalog to install additional helpful services into your cluster.

Initial Disclaimer Explained:

As mentioned in the beginning, we introduced a very strong anti-pattern in this walkthrough by placing our application and infrastructure manifests in the same repository.

This should NEVER be done when deploying to your real cloud infrastructure. Instead, for micro-service architectures such as this, each application would have their own code repo (in whatever language is appropriate). There would also be one additional "deployment repository" containing the Kubernetes manifests of all the applications. A service such as ArgoCD or Flux would then be configured to monitor the deployment repository and deploy changes to the Kubernetes cluster as needed when the applications are updated.

The Prometheus Operator should be deployed as part of the Kubestack infrastructure. The Prometheus Instance and ServiceMonitor (all explained in more details later) should be deployed along side each application. The only exception would be if you plan to have a single Instance monitor all your services, in that case it can be deployed as part of the Kubestack infrastructure.

For more information you can refer to the official documentation regarding infrastructure environments.

What Terraform can learn from PHP

Philipp Strube — Mon, 08 Feb 2021 10:30:47 +0000

TL;DR:

Writing infrastructure as code shows many of the same challenges as writing code for application development, because many of these challenges are not language or use-case specific.
Terraform and its surrounding ecosystem are still evolving and share many similarities with early PHP and the web. Just like PHP evolved by learning from other language ecosystems, Terraform can as well.
Use-case specific frameworks are a major driver of innovation, improved developer experience and productivity on the application development side. But are not yet established parts of the infrastructure as code ecosystem.
The paradigm shift to containers and Kubernetes made use-case specific frameworks possible for infrastructure as code by providing a powerful abstraction between application and infrastructure layer. And the cloud native community is evolving rapidly, extending this abstraction to additional use-cases.
Organizations that adopted application development frameworks for their improved developer experience and productivity, can leverage the same benefits for automating Kubernetes by using an infrastructure as code framework and avoid leaving the cluster the weakest link in their GitOps automation.

Learning from other language ecosystems

PHP’s ease of getting started is widely quoted as the boon and bane of the language. It seems as if making fun of the spaghetti code bases of the early PHP days never gets old. Even in 2021. But there is no doubt that PHP is an extremely successful programming language.

You may ask, what does this have to do with Terraform? Well, hear me out. Terraform and PHP have more in common than you may think. PHP was created when the web was in its infancy and quickly became extremely popular. Don’t forget, PHP is the P in LAMP stack. Similarly, infrastructure as code is still an emerging ecosystem today, and Terraform is by far the most popular language in this ecosystem.

But the modern PHP of today is vastly different from the early PHP we all like to make fun of. And since Terraform today is so similar to where PHP was when it started, there’s a good chance that the Terraform community can learn a lot from how PHP evolved.

Rasmus Lerdorf, the creator of PHP, is famously quoted as never having intended to write a programming language. But PHP got popular and they had to keep going. In addition, the web and its request-response model were new, even to experienced developers. But the endless possibilities of the web got people excited, and the unintentional programming language PHP was easy to get started with. This combination led to the stereotypical poor quality code bases that ended up powering major parts of the early web.

Similarly, infrastructure as code offers huge benefits and gets people excited as well. But it also requires both operations and coding experience, and people coming from either one background have to learn a lot about the respective other, before they can be fully productive.

Languages like Python released a few years before PHP, or Ruby and Java, which were released in the same year as PHP, were intentionally designed programming languages for professional use. While not specific to the web, it is of course possible to build web applications in either one of them. So the self-evident thing was to use these more mature and consistent languages to build web applications, and have more easily maintainable code bases as a result.

And not only were the languages more mature, but so were their ecosystems. The majority of challenges, developers face when writing code, are not language specific. And many are not even use-case specific. You may need different dependencies for building a web application instead of a desktop application for example. But in both cases having dependency management is greatly useful. A feature Python, Ruby and Java all already had.

This led to the creation of frameworks like Django, Ruby on Rails or Spring that made it easy to build web applications in Python, Ruby or Java respectively, leveraging their existing language ecosystems.

A great idea that works in one ecosystem, however is quick to inspire similar development in other languages. And PHP’s wide adoption easily justified major investments to improve the PHP core as well as the surrounding ecosystem. All those teams looking for the best way to maintain their growing PHP code bases were smart to look at other languages and how these same challenges were solved there.

The result are frameworks like Symfony or CakePHP, heavily inspired by Spring and Rails respectively. This is also how Composer brought modern dependency management to PHP. And last but not least, this was when the PHP community adopted Git for version control and slowly moved away from just editing production files directly via FTP.

It’s all about the code

Let's get back to infrastructure as code. Yes, in a lot of ways automating infrastructure is different from application development. But many of the challenges of writing code, that applied across languages and use-cases on the software development side, also apply to infrastructure as code. Code is kind of the keyword here.

So just like PHP learned from other languages, their frameworks and their tooling, Terraform can only benefit from doing so as well.

One area where Hashicorp, the makers of Terraform, recently made major improvements is dependency management. Terraform had the ability to download required providers for quite some time. But it was limited to only Hashicorp’s own providers. Community maintained providers required involving, manual installation. A recent Terraform release introduced support for registry namespaces, which means community providers can now also be installed from the official registry. In addition, required providers and versions can now be specified more explicitly. Even including the ability to vendor providers, and thereby hardening automation runs against failing when the registry is unavailable.

The missing piece

All the language ecosystems we discussed share one key piece that heavily improves the developer experience, but which isn’t a thing yet in the infrastructure as code world. I’m referring to frameworks of course. And concretely use-case specific frameworks. By being use-case specific, the aforementioned software development frameworks drastically reduce upfront and maintenance effort, and provide the best developer experience and workflow possible.

If I’m building a cloud native application in Java, using Spring Boot will make my life much easier. Likewise, if my goal is to build a Jamstack website, a framework like Gatsby will get me there much faster.

But the reason why frameworks are not a thing in the infrastructure as code world yet is not merely that the ecosystem is still evolving. For frameworks to be useful, we also required a strong abstraction layer that kept the infrastructure layer clear from application specific requirements. Containers and Kubernetes are extremely popular because they provide this very abstraction. And this means two things: First, that with using Terraform to manage Kubernetes there is a popular and very specific use-case for an infrastructure as code framework. And second, that because of the powerful abstraction, such a framework makes sense for the first time.

Kubestack is this use-case specific, Terraform GitOps framework. If you’re building GitOps automation for Kubernetes cluster infrastructure and cluster services using Terraform, Kubestack may be the framework for you. Think of Kubestack as the Ruby on Rails of infrastructure automation, the Gatsby of GitOps, or the Spring Boot of Terraform and Kubernetes.

And just like application frameworks copied ideas that worked well from one language to another, Kubestack does the same from application development to infrastructure as code.

Talent borrows, genius steals

One example is Kubestack’s convention over configuration based repository layout. Another one is its inheritance based configuration to prevent drift between environments. A third one is the ability to easily vendor dependencies in the repository, like the Nginx ingress controller or Prometheus monitoring operator. Or, as the last but not the least example, local development environments that automatically update as you make changes to the code.

Slow feedback loops are poison for developer productivity. And infrastructure as code is notoriously known for mandatory, slow pipeline runs. This makes the local development environment the perfect example how Kubestack drastically improves the developer experience, because it’s a use-case specific framework.

The strong abstraction between the application and infrastructure layers is a key mantra of what we know as cloud native. And if you take a look at recent developments from the cloud native community the direction is clear. As more and more organizations shift their workloads and use-cases to cloud native, we continue to see new innovation and iterative improvements that extend this powerful abstraction.

This is both positive for the future of infrastructure as code and Terraform as well as for use-case specific infrastructure as code frameworks.

Terraform loves cloud native

Systems that provide a separation between declaring desired state and current state are the current state-of-the-art. This is a core principle of Kubernetes and high-level managed cloud services, but also of VM auto-scaling groups, as a lower level example of this principle. On the surface there’s an API to declare the desired state. And behind the API are control loops that keep the current state in sync with the desired state.

Terraform shines when being combined with such a system, because it is great at planning and applying changes triggered by a commit in a repository. And it can also be run periodically, to detect drift and either alert or overwrite. But when operating distributed systems, there are various failure scenarios where continuously running controllers, that can take immediate action based on more events than just code changes, are clearly superior. The important thing to understand here is, Terraform is great to provide a way for teams to reason about proposed changes and keeping the committed state and desired state in sync. But keeping desired and current state in sync is, in most cases, better left to a continuously running control loop.

It’s common for teams to hit this limitation when using infrastructure as code to automate legacy systems that don’t provide this separation of concerns. And this frequently leads to automation that only manages the lifecycle partially and causes complex issues for teams to coordinate automation and manual operations. Facing this significantly limits the value of infrastructure as code, and many teams justifiably may hold back on adopting Terraform for this very reason.

But Kubernetes or managed cloud services are not the only systems that rely on declared desired state and reconciliation loops to keep current state in sync. An example doing this for infrastructure automation outside the cloud provider’s walled gardens is ClusterAPI. This cloud native community initiative aims to provide the same separation across on-premise and cloud. And through integration into vSphere, ClusterAPI is readily available to VMware’s vast installed base.

The future of infrastructure is code

As an industry, we’re clearly heading into one direction. And as we continue to adopt this paradigm, the limitations that held infrastructure as code back, when working with legacy systems, do not apply any more. As infrastructure as code becomes more viable for more organizations, more teams can benefit from use-case specific frameworks to get the best possible developer experience and productivity.

Already now, many teams are using Terraform successfully. Yes, there are edge cases to consider and there is a steep learning curve, no matter if your background is in operations or software development. But as the cloud native ecosystem continues to evolve, the benefits of infrastructure as code will be applicable to more teams and more use-cases and just like PHP grew by learning from other language ecosystems, Terraform will too.

As far as Kubernetes is concerned, if you’re already adopting GitOps, the Kubestack framework is an opportunity to implement full-stack GitOps that covers both the cluster infrastructure and cluster services and not just the application workloads on the cluster. This way, you can avoid having the foundation of your system, the cluster, be the weakest link by not managing it manually via UI.

Localhost EKS development environments with EKS-D and Kubestack

Philipp Strube — Tue, 01 Dec 2020 19:04:32 +0000

Today Amazon announced EKS Distro, or EKS-D for short. A Kubernetes distribution making the same release artifacts used by Amazon EKS available to everyone.

This allows teams to use the exact same bits and pieces that power EKS, to build clusters for anything from integration tests to on-premise use-cases. As a launch partner, I got access to EKS-D in advance to integrate it into Kubestack’s local development environments.

Kubestack is about providing the best GitOps developer experience for Terraform and Kubernetes, from local development, all the way to production.

Because I believe platform engineers automating Kubernetes deserve the same great developer experience that application engineers building applications on top of Kubernetes already have.

To achieve this, the Kubestack framework integrates all the moving pieces from Terraform providers, to resources, and modules into a GitOps workflow ready for day-2 operations.

On top of the reliable automation to propose, validate and promote infrastructure changes, Kubestack is focused on giving platform teams a modern developer experience to iterate quickly using local development environments.

Now, whenever Kubestack simulates an EKS cluster locally, it uses EKS-D to do so. Let’s take a look at how Kubestack’s infrastructure automation from local development to production works.

Local Development

Imagine you’re tasked with provisioning the Prometheus operator to deploy a Prometheus instance and configuring it to scrape the metrics from your team’s application for each environment.

If you’re like me, it may take a few iterations to get the label and namespace selectors in the Prometheus resource just right and configure the RBAC for the Prometheus instance’s service account. Especially RBAC is notorious for taking a bit of trial and error for many folks to get right.

Using Kubestack’s local development environment, you can iterate on the exact same manifests that will later be used in production. The local development environment automatically updates as you make changes, and provides immediate feedback, without waiting minutes for CI/CD pipeline runs every time. It’s just like in the infamous XKCD comic, except it’s applying, not compiling.

All you have to do to get started on this task is change into your checkout of the infrastructure repository and run one kbst CLI command. Then you’re all set to work on the Prometheus manifests.

$ kbst local apply
...
Switched to workspace "loc".
...
Apply complete! Resources: 14 added, 0 changed, 0 destroyed.
2020/11/18 12:55:53 #### Watching for changes

The new EKS-D integration means the local environment is now even closer to the EKS production environment. And fewer differences between environments reduce the risk that promoting a change fails. This is also why Kubestack uses inheritance between environments. Differences are sometimes necessary, but configuration inheritance makes them explicit.

Environment Promotion

Eventually, I’ll have the monitoring setup working locally and it’s time to push my changes and ask for a peer-review. This is the first step where the Kubestack GitOps workflow kicks in.

There are two things to review to decide if you want to apply this change. Your code changes of course, and the terraform plan provided by Kubestack’s pipeline for every branch.

If the reviewers requires changes, you can push additional commits to the branch and the pipeline will run terraform plan again. When your team approved, merge the change into master.

This triggers the pipeline and applies the merged changes to the ops environment. A terraform plan is not enough to ensure that the changes will apply correctly. That’s why Kubestack uses the ops environment, to validate the configuration change against real cloud infrastructure. The ops environment does not run applications, so that teams can feel confident to merge infrastructure changes at any time, without worrying about blocking team members or breaking applications.

Finally, if the change to ops applied successfully, the pipeline will additionally provide a terraform plan to show the required changes for the apps environment. The additional plan helps teams decide if they want to promote this change into the apps environment now.

Having a reliable workflow is crucial for teams to trust their automation. Kubestack, by combining purpose built Terraform modules and its proven triggers helps teams build infrastructure automation that is ready for day-2 operations.

Get Started

If you’ve made it this far and want to learn more, you can get started with the Kubestack framework by following the tutorial.