Low-cost Kubernetes cluster on Infomaniak

Teo Stocco — Tue, 04 Jul 2023 22:14:29 +0000

Kubernetes (k8s) is becoming the de facto standard for container orchestration. While it provides convenient abstractions and solves many pain points, it is not as accessible as running a local copy of Docker. This makes it harder to learn and get real hands-on experience without being part of a DevOps team or getting ruined quickly.

When looking at popular Kubernetes providers and without going into their complex pricing structure, the lowest price one can find for a managed Kubernetes is roughly $55/month for 4cpu and 15Go of RAM available (June 2023).

Provider	Region	Cluster fees	4cpu / 15Go	Note
GKE	Frankfurt	$73/month	$126	$74.40/month of free credit
AWS	Frankfurt	$73/month	$112
AKS	West Germany	Free	$112	$73/month for pro cluster
Linode	Frankfurt	$36	$72	Only 8Go RAM
Scaleway	Paris	Free	$73	98% API server availability
OVH	Frankfurt	Free	$55
Exoscale	Frankfurt	Free	$136	$40/month for pro cluster
DigitalOcean	Frankfurt	Free	$126	$40/month for pro cluster

At that price, it becomes interesting to look at reasonably priced VMs providers like Hetzner or Infomaniak, and install Kubernetes directly on it. This comes at the cost of managing the cluster ourselves and maintaining it over time. While this offers a good learning opportunity, it may also be a source of frustration as the setup can be non-trivial. For this reason, the RKE2 distribution of Kubernetes is a great pick as it offers a good balance between features, security and simplicity.

Regarding the hardware provider, Infomaniak offers currently better price than Hetzner and especially gives us with direct access to OpenStack, a well-known open source platform to provision infrastructure (also offered by OVH). This makes it easier to manage the resources using code (IaC) and comes with a good support for Terraform.

Getting started

You will need an account on Infomaniak, a valid credit card and Terraform installed.

Once you have your Infomaniak account, go on the Public Cloud landing page and click on "get started". Pick a name for your public cloud and enter your credit card details in the checkout page. You will be charged at the end of the month for the resources you used. Note that as of the time of writing this article, Infomaniak offers up to CHF 300.— of free credit during the first 3 months.

You should then see an empty list of OpenStack project (also called tenant in the OpenStack jargon). Click on the "create a project" on the top right, choose a project name and a password for the autogenerated user and click on "create". After a few seconds, you should be redirected to the list of projects and see your newly created project.

You can then click on the project name, and you will be directed on Horizon, the user interface of OpenStack which is useful to follow the progress of the cluster creation.

Terraform OpenStack RKE2 module

Now, you are jumping into the interesting part. You will use the Terraform OpenStack RKE2 module that takes care of the heavy lifting and deploy a RKE2 cluster on OpenStack for you. Create a new directory and open a main.tf file with the following content:

variable "project" {
  type = string
}

variable "username" {
  type = string
}

variable "password" {
  type = string
}

# authenticate with OpenStack
provider "openstack" {
  tenant_name = var.project
  user_name   = var.username
  password    = var.password
  auth_url    = "https://api.pub1.infomaniak.cloud/identity"
  region      = "dc3-a"
}

# dependency management
terraform {
  required_version = ">= 0.14.0"

  required_providers {
    openstack = {
      source  = "terraform-provider-openstack/openstack"
      version = ">= 1.49.0"
    }
  }
}

This creates 3 variables that will be used to authenticate to the OpenStack API and configure the OpenStack provider and ensure that the latter is using the expected version. If you would like to avoid having to enter your credentials each time, you can create a terraform.tfvars (make sure to add this file to your gitignore and never accidentally share that file) or check the alternative authentication methods of the provider (e.g. cloud with clouds.yaml):

project=PCP-XXXXXXXX
username=PCU-XXXXXXXX
password=XXXXXXXX

You will now declare a cluster with 1 server and 1 agent nodes in main.tf:

module "rke2" {
  source = "zifeo/rke2/openstack"
  # fixing the version is recommended (follows semantic versioning)
  # version = "2.0.5"

  # must be true for single server cluster or
  # only on the first run for high-availability cluster
  bootstrap = true
  name      = "single-server"

  # path to your public key, in order to connect to the instances
  ssh_authorized_keys = ["~/.ssh/id_rsa.pub"]

  # name of the public OpenStack network to use for the server IP
  floating_pool = "ext-floating1"

  # allow access from any IP
  # it should ideally be restricted to a secure bastion
  rules_ssh_cidr = "0.0.0.0/0"
  rules_k8s_cidr = "0.0.0.0/0"

  # servers hosts Kubernetes control plane + etcd
  # and are the only ones exposed to the internet
  servers = [{
    name = "server"

    # 2 cpu and 4Go of RAM is the minimum recommended per server
    flavor_name = "a2-ram4-disk0"
    image_name  = "Ubuntu 22.04 LTS Jammy Jellyfish"
    system_user = "ubuntu"

    # size of the operating system disk
    boot_volume_size = 4

    # size of the volume for the RKE2 data (persisted on single-server)
    rke2_volume_size = 8
    rke2_version     = "v1.26.4+rke2r1"
  }]

  # agents runs your workloads
  # and are not exposed to the internet (doable with a load balancer)
  agents = [
    {
      name        = "agent-a"
      nodes_count = 1

      # a2-ram4-disk0 is the minimal meaningful config for agents
      # you can also directly go for a4-ram16-disk0 as in the intro
      flavor_name = "a2-ram4-disk0"
      image_name  = "Ubuntu 22.04 LTS Jammy Jellyfish"
      system_user = "ubuntu"

      boot_volume_size = 4

      rke2_volume_size = 8
      rke2_version     = "v1.26.4+rke2r1"
    }
  ]

  # enable automatically agent removal of the cluster
  ff_autoremove_agent = true
  # output the kubeconfig to the current directory
  ff_write_kubeconfig = true

  identity_endpoint = "https://api.pub1.infomaniak.cloud/identity"
}

And you are ready to go! Run the following commands and wait a few minutes for the cluster to be created (a few more are required to have all the pods running after the core is ready):

terraform init
# ...
# Terraform has been successfully initialized!
# ...

terraform apply
# ...
# Plan: 71 to add, 0 to change, 0 to destroy.
#
# Do you want to perform these actions?
#   Terraform will perform the actions described above.
#   Only 'yes' will be accepted to approve.
#
#   Enter a value: yes
# ...
# Apply complete! Resources: 71 added, 0 changed, 0 destroyed.

cat single-server.rke2.yaml
# apiVersion: v1
# kind: config
# ...

export KUBECONFIG=single-server.rke2.yaml

kubectl get pods --all-namespaces
# NAMESPACE     NAME                                                    READY   STATUS              RESTARTS   AGE
# kube-system   helm-install-openstack-cinder-csi-2rp9z                 0/1     Pending             0          2s
# kube-system   helm-install-openstack-cloud-controller-manager-4wdzt   0/1     ContainerCreating   0          2s
# kube-system   helm-install-rke2-cilium-s5skd                          0/1     ContainerCreating   0          2s
# kube-system   helm-install-rke2-coredns-kc4ld                         0/1     ContainerCreating   0          2s
# kube-system   helm-install-rke2-metrics-server-ttt84                  0/1     Pending             0          2s
# kube-system   helm-install-rke2-snapshot-controller-crd-2sdzt         0/1     Pending             0          2s
# kube-system   helm-install-rke2-snapshot-controller-xqzsk             0/1     Pending             0          2s
# kube-system   helm-install-rke2-snapshot-validation-webhook-5w9lw     0/1     Pending             0          2s
# kube-system   helm-install-velero-4zhq7                               0/1     Pending             0          2s
# kube-system   kube-apiserver-single-server-server-1                   1/1     Running             0          12s
# kube-system   kube-controller-manager-single-server-server-1          1/1     Running             0          16s
# kube-system   kube-scheduler-single-server-server-1                   1/1     Running             0          15s

kubectl get pods --all-namespaces
# NAMESPACE     NAME                                                    READY   STATUS      RESTARTS   AGE
# kube-system   cilium-ngcrp                                            1/1     Running     0          2m39s
# kube-system   cilium-operator-b947b9d8d-zc92l                         1/1     Running     0          2m39s
# kube-system   cilium-qt7vr                                            1/1     Running     0          2m39s
# kube-system   etcd-single-server-server-1                             1/1     Running     0          2m42s
# kube-system   helm-install-openstack-cinder-csi-2rp9z                 0/1     Completed   0          2m52s
# kube-system   helm-install-openstack-cloud-controller-manager-4wdzt   0/1     Completed   0          2m52s
# kube-system   helm-install-rke2-cilium-s5skd                          0/1     Completed   0          2m52s
# kube-system   helm-install-rke2-coredns-kc4ld                         0/1     Completed   0          2m52s
# kube-system   helm-install-rke2-metrics-server-ttt84                  0/1     Completed   0          2m52s
# kube-system   helm-install-rke2-snapshot-controller-crd-2sdzt         0/1     Completed   0          2m52s
# kube-system   helm-install-rke2-snapshot-controller-xqzsk             0/1     Completed   0          2m52s
# kube-system   helm-install-rke2-snapshot-validation-webhook-5w9lw     0/1     Completed   0          2m52s
# kube-system   helm-install-velero-4zhq7                               0/1     Completed   0          2m52s
# kube-system   kube-apiserver-single-server-server-1                   1/1     Running     0          3m2s
# kube-system   kube-controller-manager-single-server-server-1          1/1     Running     0          3m6s
# kube-system   kube-scheduler-single-server-server-1                   1/1     Running     0          3m5s
# kube-system   openstack-cinder-csi-controllerplugin-cf5f9869d-xbmtv   6/6     Running     0          2m39s
# kube-system   openstack-cinder-csi-nodeplugin-zghjj                   3/3     Running     0          98s
# kube-system   openstack-cloud-controller-manager-bffxb                1/1     Running     0          2m19s
# kube-system   rke2-coredns-rke2-coredns-autoscaler-597fb897d7-p8k7j   1/1     Running     0          2m41s
# kube-system   rke2-coredns-rke2-coredns-f6f4ff467-6lrgl               1/1     Running     0          83s
# kube-system   rke2-coredns-rke2-coredns-f6f4ff467-shlrv               1/1     Running     0          2m41s
# kube-system   rke2-metrics-server-67d6554d69-8vhrt                    1/1     Running     0          78s
# kube-system   rke2-snapshot-controller-6b9c678c77-2txzn               1/1     Running     0          77s
# kube-system   rke2-snapshot-validation-webhook-6c9d7f868c-qnqdq       1/1     Running     0          77s
# velero        restic-g8mvb                                            1/1     Running     0          30s
# velero        velero-5b67659997-67zgd                                 1/1     Running     0          30s

You can also explore your cluster using Horizon (see instances and components of the network).

Congratulations, you just have installed your first Kubernetes cluster with RKE2! You can find more information about the module and its features (etcd backups, upgrades, volume snapshots, etc.) on the repository. Give a star ⭐️ if you like it or raise an issue there if you find a bug 🐛.

Originally posted on zifeo.com, find more about the architecture and the cost projections there.

Forem: Teo Stocco

Low-cost Kubernetes cluster on Infomaniak

Getting started

Terraform OpenStack RKE2 module