Rancher Kubernetes on Openstack using Terraform

Sami Alhaddad — Wed, 27 May 2020 03:33:00 +0000

In this article we will walk through creating complete infrastructure pieces on OpenStack that are needed to have a fully provisioned Kubernetes cluster using Terraform and Rancher2. In addition to integration with cloud-provider-openstack and cinder-csi-plugin

Getting started with Infrastructure

Clone the repository terraform-rancher2 into a folder.
Go into the openstack folder using cd openstack/
Modify the variables in terraform.tfvars to match your current cloud environment. it is important to uncomment the vars openstack_project , openstack_username and openstack_password or export them as env variables with prefix TF_VAR_* for example:

export TF_VAR_openstack_username=myusername
export TF_VAR_openstack_password=mypassword
export TF_VAR_openstack_project=myproject

Other variables can be obtained from openstack-cli such as rancher_node_image_id , external_network and flavors by invoking

## image list .. pick an ubuntu image
openstack image list
## network name
openstack network list --external
## flavors
openstack flavor list

RKE configuration can be adjusted and customized in rancher2.tf, you can check the provider documentation at rancher_cluster
NOTE: It is really important to keep kubelet extra_args for the external cloudprovider in order to integrate with cloud-provider-openstack
Run terraform init to initialize a working directory containing Terraform configuration files.
To apply the creation of the environment, Run terraform apply --auto-approve and wait for the output after all resources finish the creation

Apply complete! Resources: 25 added, 0 changed, 0 destroyed.

Outputs:

rancher_url = [
  "https://xx.xx.xx.xx/",
]

Up to this point, use the rancher_url from above output and login to rancher instance with username admin and password defined in rancher_admin_password. Wait for all kubernetes nodes to be discovered, registered, and active.

Integration with cloud-provider-openstack

As you may notice, that all the nodes have a taint node.cloudprovider.kubernetes.io/uninitialized. The usage of --cloud-provider=external flag to the kubelet makes it waiting for the clouder-provider to start the initialization. This marks the node as needing a second initialization from an external controller before it can be scheduled work.

Edit the file manifests/cloud-config with the access information to your openstack environment.
Create a secret containing the cloud configuration in the kube-system namespace

kubectl create secret -n kube-system generic cloud-config --from-file=manifests/cloud-config

Create RBAC resources and openstack-cloud-controller-manager deamonset and wait for all the pods in kube-system namespace up and running.

kubectl apply -f manifests/cloud-controller-manager-roles.yaml
kubectl apply -f manifests/cloud-controller-manager-role-bindings.yaml
kubectl apply -f manifests/openstack-cloud-controller-manager-ds.yaml

Create cinder-csi-plugin which are a set of cluster roles, cluster role bindings, statefulsets, and storageClass to communicate with openstack(cinder).

kubectl apply -f manifests/cinder-csi-plugin.yaml

Up to this point, openstack-cloud-controller-manager and cinder-csi-plugin have been deployed, and they're able to obtain valuable information such as External IP addresses and Zone info.

$ kubectl get nodes -o wide

NAME            STATUS   ROLES               AGE     VERSION   INTERNAL-IP     EXTERNAL-IP      OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
demo-master-1   Ready    controlplane,etcd   5h      v1.17.5   192.168.201.6   xx.xx.xx.xx      Ubuntu 18.04.2 LTS   4.15.0-45-generic   docker://19.3.9
demo-worker-1   Ready    worker              4h57m   v1.17.5   192.168.201.4   xx.xx.xx.xx      Ubuntu 18.04.2 LTS   4.15.0-45-generic   docker://19.3.9
demo-worker-2   Ready    worker              4h56m   v1.17.5   192.168.201.5   xx.xx.xx.xx      Ubuntu 18.04.2 LTS   4.15.0-45-generic   docker://19.3.9

Also, as shown in the nodes tab, All nodes are active and labeled by openstack zones.

Scalability

When it comes to scalability with IaaC (infrastructure-as-a-code), it becomes so easy to obtain any desired state in less consumed efforts and time.
All you have to do is to change the number of nodes count_master or count_worker_nodes and run terraform apply again
For example, let's increase the number of count_worker_nodes by 1
A few minutes later, after refreshing states and applying updates:


Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Outputs:

rancher_url = [
  "https://xx.xx.xx.xx",
]

Couple of minutes for the new node to be registered

$ kubectl get nodes -o wide
NAME            STATUS   ROLES               AGE    VERSION   INTERNAL-IP     EXTERNAL-IP      OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
demo-master-1   Ready    controlplane,etcd   28h    v1.17.5   192.168.201.6   xx.xx.xx.xx      Ubuntu 18.04.2 LTS   4.15.0-45-generic   docker://19.3.9
demo-worker-1   Ready    worker              28h    v1.17.5   192.168.201.4   xx.xx.xx.xx      Ubuntu 18.04.2 LTS   4.15.0-45-generic   docker://19.3.9
demo-worker-2   Ready    worker              28h    v1.17.5   192.168.201.5   xx.xx.xx.xx      Ubuntu 18.04.2 LTS   4.15.0-45-generic   docker://19.3.9
demo-worker-3   Ready    worker              2m2s   v1.17.5   192.168.201.7   xx.xx.xx.xx      Ubuntu 18.04.2 LTS   4.15.0-45-generic   docker://19.3.9

NOTE: Scaling down the cluster could be made by decreasing the number of nodes in terrafrom.tfvars. Node gets deleted, moreover cloud-provider-openstack detects that and removes it from the cluster

Cleaning up

To clean up all resources created by this terraform, Just run terraform destroy

rootsami / terraform-rancher2

Terraform manifests to create e2e production grade k8s cluster

terraform-rancher2

Terraform manifests to create e2e production-grade Kubernetes cluster on top of cloud providers

Overview

This repo is intended to be for creating complete infrastructure pieces on OpenStack that are needed to have a fully provisioned Kubernetes cluster using Terraform and Rancher2. In addition to integration with cloud-provider-openstack

Getting started with Infrastructure

Clone the repository terraform-rancher2 into a folder.
Go into the openstack folder using cd openstack/
Modify the variables in terraform.tfvars to match your current cloud environment. it is important to uncomment the vars openstack_project , openstack_username and openstack_password or export them as env variables with prefix TF_VAR_* for example:

export TF_VAR_openstack_username=myusername
export TF_VAR_openstack_password=mypassword
export TF_VAR_openstack_project=myproject

Other variables can be obtained from openstack-cli such as rancher_node_image_id , external_network by invoking

## image list
openstack image list
## network name
openstack network list --external
## flavors
openstack flavor list

Run terraform init to initialize a working directory…

View on GitHub

References

Prometheus blackbox_exporter; Unconventional Way

Sami Alhaddad — Wed, 13 May 2020 01:09:38 +0000

Many of us have different requirements and different complicated setups.
Prometheus has provided us the true power of monitoring and observability, Thus, I'm still learning and figuring things out every single day.

Case

One of the most recent requirements I had, that we have to monitor a certain route of accessibility to maintain applications performing correctly. With out-of-the-box tools' functionality, the monitoring system is always the source, where our case is different that we need to make sure that System-A is reachable from every network in the datacenter.

To have a simple and less complicated example, let's take north-ntw, south-ntw and west-ntw to reach the internet by probing https://wikipedia.com
by this I mean, we are monitoring egress traffic in each network whether it could reach the internet through different routes with reasonable latency.

Strategy

We're going to use Prometheus as a server and Blackbox Exporter
The blackbox exporter allows blackbox probing of endpoints over HTTP, HTTPS, DNS, TCP and ICMP. It's often used to be installed alongside to prometheus server and this makes the prometheus server is always the source.
Here is the trick, we're going to deploy blackbox exporter on every monitored network and instruct prometheus to scrape those exporters as a source of probes to assure that System-A (aka Wikipedia) is reachable from those networks.

Let's go

You can follow as many as guides and tutorial on how to install Prometheus and Prometheus exporters, however, here's a quick win with ansible and playbooks for ansible-roles from the great team at cloudalchemy:
ansible-prometheus and ansible-blackbox-exporter
I will leave the installation part for you to be done in your preferred way

scraping configs

This is the most important part of this journey where we instruct prometheus where to find it's exporters

## /etc/prometheus/prometheus.yml
  - job_name: blackbox_metadata
    params:
      module: [http_2xx]
      target:
        - https://wikipedia.com
    metrics_path: /probe
    scrape_interval: 30s
    scrape_timeout: 10s
    static_configs:
      - targets:
        - south.rootsami.dev:9115
        - north.rootsami.dev:9115
        - east.rootsami.dev:9115
    relabel_configs:
      - source_labels: [__param_target]
        target_label: target
      - source_labels: [__address__]
        separator:     ';'
        regex:         '(.*):.*'
        target_label: instance
        replacement:   '${1}:9115'

target under the params section defines the destination that you want to reach which is https://wikipedia.com, whereas the static_configs targets are the little blackbox-exporter that we are probing from.

As shown above, target endpoints are the deployed exporters which are being scraped by prometheus server and showing that Wikipedia is reachable from that network, as well as latency can be measured from each source.

Conclusion

Monitoring and observability have no limits, tools are there! All you have to do is to find the blind spots to maintain systems up and running at all times.

Forem: Sami Alhaddad

Rancher Kubernetes on Openstack using Terraform

Getting started with Infrastructure

Integration with cloud-provider-openstack

Scalability

Cleaning up

rootsami / terraform-rancher2

Terraform manifests to create e2e production grade k8s cluster

terraform-rancher2

Overview

Getting started with Infrastructure

References

Prometheus blackbox_exporter; Unconventional Way

Case

Strategy

Let's go

scraping configs

Conclusion