Forem: jacobcrawford

Hello Python through Docker and Kubernetes

jacobcrawford — Fri, 17 Sep 2021 11:18:17 +0000

Going from an application to a containerized application can have many benefits. Today we are taking the step further and looking at how we can go from a small application all the way to deploying it on Kubernetes.

We are going to take a small application, build a container image around it, write Kubernetes manifest and deploy the whole thing on Kubernetes. This is a practical guide, so I encourage you to follow along with your own application in your favorite programming language. Here we will use Python.

Application

We start out with a small Hello World python REST api in Flask.

The following dependencies in Python are needed:

flask_restful
flask

which can be installed with pip3 install flask flask_restful.

Our application is a simple python script hello-virtualization.py

from flask import Flask
from flask_restful import Resource, Api

app = Flask(__name__)
api = Api(app)

class HelloWorld(Resource):
    def get(self):
        return "Hello Python!"

api.add_resource(HelloWorld, '/')

if __name__ == '__main__':
    app.run(host='0.0.0.0')

Our app simply exposes an endpoint that returns "Hello Python!" and can be started with python3 hello-virtualization.py. You should see the following in your terminal:

We can see that the server is listening on port 5000, and can confirm this by sending a request: curl 10.0.0.11:5000

Great! We now have a very simple application that responds when we call it. Lets wrap this in a container image.

Docker

First you need to get docker installed on your machine. If you are on Windows or MacOS, Docker Desktop should do the trick.

Next we will wrap the application in a docker image. To do this create a file called Dockerfile in the same folder as your application.

In the Dockerfile we will start by setting a base container image for our own container image. Since we need Python we will use python:3.8-buster by typing: FROM python:3.8-buster.

Next we need to include the packages needed for our application to run. We do this by writing RUN pip3 install flask flask_restful just as when we fetched the packages for our local system, we now fetch the to the image.

We the specify a working directory: WORKDIR /app which is where all commands we run will be executed from.

Copy the source code into the working directory: COPY hello-virtualization.py /app, inform that the container should have port 5000 open with EXPOSE 5000 and finally tell which command will be executed when we run the container image: CMD ["python3", "hello-virtualization.py"]

The final Dockerfile should look like this:

FROM python:3.8-buster

RUN pip3 install flask flask_restful

WORKDIR /app

COPY hello-virtualization.py /app

EXPOSE 5000

CMD ["python3", "hello-virtualization.py"]

We will then need to build the container image and give it the name hello-virtualization for reference:

docker build . -t hello-virtualization

Will build our container image. To run the image we simply type:

docker run hello-virtualization

The command will give an output very similar to when we executed the script using only python. Validate that the application works by curling the endpoint that is printed by the command.

Kubernetes

If you use Docker Desktop, Kubernetes can be enabled through the UI.

If you do not use Docker Desktop, there are multiple alternatives for setting up a Kubernetes cluster for development like:

For this demo all of them will work.

Kubernetes needs to fetch the container image that we just build from a container registry. To make a solution that works for all the different versions, we will push the container image to a remote container registry. The easiest way to do this is to create an account on Docker Hub and authenticate by typing docker login in your terminal. When this is done we need to retag the container image so that the Docker CLI knows that we want to push the image to a container registry owned by the user we just created.

To tag the container image:

docker tag hello-virtualization:latest <DOCKER_HUB_USERNAME>/hello-virtualization:latest

To push the image to Docker Hub:

docker push <DOCKER_HUB_USERNAME>/hello-virtualization:latest

Now we are ready for some Kubernetes.

Kubernetes resources are managed through manifest files written in Yaml. The smallest unit in Kubernetes is called a Pod, which is a wrapper around one or more containers. When you want to deploy a Pod in Kubernetes, you would often use another resource called a Deployment to manage the deployment of a Pod and the replication factor of the Pod.

Create a new file called hello-virtualization-deployment.yaml to create a Deployment for our app:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-virtualization-deployment
  labels:
    app: hello-virtualization
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-virtualization-label
  template:
    metadata:
      labels:
        app: hello-virtualization-label
    spec:
      containers:
      - name: hello-virtualization-container
        image: <DOCKER_HUB_USERNAME>/hello-virtualization:latest
        ports:
        - containerPort: 5000

A few things to notice:

replicas: 1

This tells the Deployment that we want 1 instance of our container running.

  selector:
    matchLabels:
      app: hello-virtualization-label

Tells the Deployment that it shall control the template with the label app: hello-virtualization-label.

  template:
    metadata:
      labels:
        app: hello-virtualization-label

Is where we define that a template and set the label that binds this template to the Deployment.

template:
    ...
    spec:
      containers:
      - name: hello-virtualization-container
        image: <DOCKER_HUB_USERNAME>/hello-virtualization:latest
        ports:
        - containerPort: 5000

We configure that this template is using the container image that we build and that the container listens on 5000 when created.

To deploy this on the Kubernetes we use the tool kubectl:

kubectl apply -f hello-virtualization-deployment.yaml

You should be able to see the Pod being created by typing:

kubectl get pods

Kubernetes needs to pull the image and start the container in the Pod, but after a few seconds the pod should be have a Running status.

The Pod has an IP address within the Kubernetes cluster, that we can use to test that the application works. To get the IP address type kubectl get pods -o wide:

Since this IP address is only reachable from within the Kubernetes cluster we need to jump into a Pod:
kubectl run my-shell --rm -i --tty --image curlimages/curl -- sh
Now we can curl the IP address and see that the application works.

This is not a stable solution for many reasons. First the IP of the Pod is short lived, meaning that the Pod with the application will not have the same IP every time we start it. Try to kill the Pod with kubectl delete pod <POD_NAME>. The pods name is listed when you type kubectl get pods.
If you type kubectl get pods -o wide after having deleted the pod you will se that there is a pod running with a similar name and a new IP address. This happens because we have told Kubernetes that we wanted 1 replica of the Pod, and therefor Kubernetes will make sure that there is always 1 Pod alive with our application. If our application crashed, Kubernetes will simply deploy a new one! Though the IP will not be the same so we need to be able to contact our application another way.

Introducing Kubernetes Services:

Create a file hello-virtualization-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: hello-virtualization-service
  labels:
    app: hello-virtualization-label
spec: 
  ports:
  - port: 5000
    targetPort: 5000
    protocol: TCP
  selector:
    app: hello-virtualization-label

A few things to notice:

spec: 
  ports:
  - port: 5000
    targetPort: 5000
    protocol: TCP

Tells our service that when we contact it on port 5000 it should forward the communication to port 5000 on the target using TCP.

spec: 
  ...
  selector:
    app: hello-virtualization-label

Informs the service that the target is actually the Pods with the label app: hello-virtualization-label, just like we did in the Deployment. Labels and label selectors is how we bind resources in Kubernetes.

We deploy the service just as we did the Deplyment: kubectl apply -f hello-virtualization-service.yaml.

To see that the service was deployed type: kubectl get svc:

We then test that the service work by trying to contact our application through the service.

Once again jump into a pod on the cluster:
kubectl run my-shell --rm -i --tty --image curlimages/curl -- sh

This time we will not use the Pods IP, but the name of the service to contact our application:

Inside the Kubernetes cluster you can always use the names of services to communicate between Pods.

High availability Kubernetes cluster on bare metal - part 2

jacobcrawford — Fri, 26 Mar 2021 11:07:58 +0000

Last week we covered the theory of high availability in a bare-metal Kubernetes cluster, which means that this week is where the magic happens.

First of all, there are a few dependencies that you need to have installed to initialize a Kubernetes cluster. Since this is not a guide on how to set up Kubernetes, I will assume that you have already done this before, and if not you can use the same guide as I used when installing Kubernetes for the first time: guide.

Also, if you did not follow the guide and have already installed Kubernetes and Docker (or your favorite container runtime), you will also have installed a key Kubernetes toolbox kubeadm, which is what we will use to initialize the cluster. First, we need to deal with the problems of high availability, which we discussed last week.

The stable control plane IP

As mentioned, we will use a self-hosted solution where we set up a stable IP with HAProxy and Keepalived as pods inside the Kubernetes cluster. To achieve this, we will need to configure a few files for each master node:

A keepalived configuration.
A keepalived health check script.
A manifest file for the keepalived static pod.
A HAproxy configuration file.
A manifest file for the HAProxy static pod.

Keepalived:

! /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
}
vrrp_script check_apiserver {
  script "/etc/keepalived/check_apiserver.sh"
  interval 3
  weight -2
  fall 10
  rise 2
}

vrrp_instance VI_1 {
    state ${STATE}
    interface ${INTERFACE}
    virtual_router_id ${ROUTER_ID}
    priority ${PRIORITY}
    authentication {
        auth_type PASS
        auth_pass ${AUTH_PASS}
    }
    virtual_ipaddress {
        ${APISERVER_VIP}
    }
    track_script {
        check_apiserver
    }
}

We have some placeholders in bash that we need to fill out manually or through scripting:

STATE Will be MASTER for the node initializing the cluster because it will also be the first one to host the virtual IP address of the control plane.
INTERFACE Is the network interface of the network where the nodes will communicate. For Ethernet connections, this is often eth0, and can be found with the command ifconfig on most Linux operating systems.
ROUTER_ID Needs to be the same for all the hosts. Often set to 51.
PRIORITY A unique number that decides which node should host the virtual IP of the control plane in case the first MASTER node goes down. Often set to 100 for the node initializing the cluster, and then decreasing values for the rest.
AUTH_PASS should be the same for all nodes. Often set to 42.
APISERVER_VIP The virtual IP for the control plane. This will be created.

For the health check script we have the following:

#!/bin/sh

errorExit() {
    echo "*** $*" 1>&2
    exit 1
}

curl --silent --max-time 2 --insecure https://localhost:${APISERVER_DEST_PORT}/ -o /dev/null || errorExit "Error GET https://localhost:${APISERVER_DEST_PORT}/"
if ip addr | grep -q ${APISERVER_VIP}; then
    curl --silent --max-time 2 --insecure https://${APISERVER_VIP}:${APISERVER_DEST_PORT}/ -o /dev/null || errorExit "Error GET https://${APISERVER_VIP}:${APISERVER_DEST_PORT}/"
fi

We see the APISERVER_VIP placeholder again, which is just the same as before. If some variables are repeated I will not repeat the explanation, which means that the only new variable is:

APISERVER_DEST_PORT, which is the front end port on the virtual IP for the API server. This can be any unused port e.g. 4200.

Last, the manifest file for Keepalived:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  name: keepalived
  namespace: kube-system
spec:
  containers:
  - image: osixia/keepalived:1.3.5-1
    name: keepalived
    resources: {}
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
        - NET_BROADCAST
        - NET_RAW
    volumeMounts:
    - mountPath: /usr/local/etc/keepalived/keepalived.conf
      name: config
    - mountPath: /etc/keepalived/check_apiserver.sh
      name: check
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/keepalived/keepalived.conf
    name: config
  - hostPath:
      path: /etc/keepalived/check_apiserver.sh
    name: check
status: {}

This creates a pod that uses the two configuration files.

HAProxy

We have one configuration file for the HAProxy:

# /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    log /dev/log local0
    log /dev/log local1 notice
    daemon

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 1
    timeout http-request    10s
    timeout queue           20s
    timeout connect         5s
    timeout client          20s
    timeout server          20s
    timeout http-keep-alive 10s
    timeout check           10s

#---------------------------------------------------------------------
# apiserver frontend which proxys to the masters
#---------------------------------------------------------------------
frontend apiserver
    bind *:${APISERVER_DEST_PORT}
    mode tcp
    option tcplog
    default_backend apiserver

#---------------------------------------------------------------------
# round robin balancing for apiserver
#---------------------------------------------------------------------
backend apiserver
    option httpchk GET /healthz
    http-check expect status 200
    mode tcp
    option ssl-hello-chk
    balance     roundrobin
        server ${HOST1_ID} ${HOST1_ADDRESS}:${APISERVER_SRC_PORT} check
        server ${HOST2_ID} ${HOST2_ADDRESS}:${APISERVER_SRC_PORT} check
        server ${HOST3_ID} ${HOST3_ADDRESS}:${APISERVER_SRC_PORT} check

Here, we plug in the control plane IPs. Assuming a 3 node cluster we input a symbolic HOST_ID, which is just a unique name, for each as well as the HOST_ADDRESS. The APISERVER_SRC_PORT is by default port 6443, where the apiserver listens for traffic.

The last file is the HAProxy manifest file:

apiVersion: v1
kind: Pod
metadata:
  name: haproxy
  namespace: kube-system
spec:
  containers:
  - image: haproxy:2.1.4
    name: haproxy
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: localhost
        path: /healthz
        port: ${APISERVER_DEST_PORT}
        scheme: HTTPS
    volumeMounts:
    - mountPath: /usr/local/etc/haproxy/haproxy.cfg
      name: haproxyconf
      readOnly: true
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/haproxy/haproxy.cfg
      type: FileOrCreate
    name: haproxyconf
status: {}

This is all we actually need to configure to get a cluster up and running. Some of these are constants that need to be the same for all three master nodes, and some need to vary between nodes. Some are just values you have to input and for some values, you have to make a decision.

Values sanity check

Let us just take a quick sanity check over the variables and what they are by default for each node.

Constants

ROUTER_ID=51
AUTH_PASS=42
APISERVER_SRC_PORT=6443

Variables to input

STATE
MASTER for the node that initializes the cluster, BACKUP for the two others.
PRIORITY
100 for the node that initializes the cluster, 99 and 98 for the two others.

Variables to retrieve

APISERVER_VIP
An IP within your network subnet. If your node has IP 192.168.1.140, this could be 192.168.1.50.
APISERVER_DEST_PORT
A port for your choosing. Must not conflict with other service ports.

INTERFACE
The network interface. Use ifconfig to find it.

HOSTX_ID
Any unique name for each of the 3 master nodes.

HOSTX_ADDRESS
The ip addresses of your machines. Can also be found with ifconfig on each machine.

Files

Now that the files are configured they should be but in the right destination so that kubeadm can find them when the cluster is initializing.

The absolute file paths are:

/etc/keepalived/check_apiserver.sh
/etc/keepalived/keepalived.conf
/etc/haproxy/haproxy.cfg

/etc/kubernetes/manifests/keepalived.yaml
/etc/kubernetes/manifests/haproxy.yaml

Putting manifest files into /etc/kubernetes/manifests/ is what does the magic here. Everything in this folder will be applied when the cluster initializes. Even the control plane pods that are generated by kubeadm will be put in here before the cluster initializes.

Initializing the cluster

When the files are in place, initializing the cluster is as simple as running the kubeadm init command with a few extra pieces of information.

kubeadm init --control-plane-endpoint APISERVER_VIP:APISERVER_DEST_PORT --upload-certs

Will do the trick. The extra arguments tell the cluster that the control plane should not be contacted on the actual nodes IP, but on the virtual IP address. When the other nodes join, this is what makes the cluster highly available. If the node that is currently hosting the virtual IP goes down, the virtual IP will just jump to another available master node.

Last, join the other two nodes to the cluster with the join command output by kubeadm init.

If this even peaked your interest a little bit, you are in for a treat. The whole manual process is being eliminated in an open-source project right here. It is still a work in progress, but feel free to drop in and join the discussion.

High availability Kubernetes cluster on bare metal

jacobcrawford — Fri, 19 Mar 2021 12:33:33 +0000

This blog post is the first in a series concerning deploying a high available Kubernetes cluster on bare metal.

Kubernetes is often used in the comfort of cloud providers, where we can spin up multiple master nodes without even caring about what goes on behind the scene. In this blog post, we will step away from the comforts of cloud-provided architecture and into the dangerous unknown environment of a bare-metal infrastructure.

Using Kubernetes is great and comfortable because you don't have to worry about failing services, discomforts of scaling, downtimes, version upgrades and so much more. Though as always when working with distributed technologies you should count on machine failures, and this is where the term high availability comes up.

When using Kubernetes in a playground environment, we spin up a single master node and run our containers. In a production environment, this is not optimal for a number of reasons, but mostly because it is a single point of failure. Hence, we want a Kubernetes cluster to have multiple master nodes to deal with this problem. This is where it gets tricky on bare metal.

Inside a master node

Taking a step back, we need to understand what goes on in a Kubernetes master node. Normally containers only run on worker nodes and Kubernetes handles deployments, replications, pod communications, etc. If we look into the belly of a master node it contains pods for orchestration:

etcd A key-value storage for cluster data, most importantly the desired state
kube-scheduler A component that watches for newly created pods and decides which node they should run on
kube-controller-manager Manages a lot of Kubernetes functionality like listening for nodes going down, pod creation, etc.
kube-apiserver The frontend that handles communications with all the other pods.

These 4 pods constitute what is commonly known as the control plane of a Kubernetes cluster. As stated, all communication goes through the kube-apiserver. When we execute kubectl commands what happens behind-the-scene is that we send post requests to the kube-apiserver.

The kube-apiserver is also where our problem arises when introducing multiple master nodes. When there are multiple master nodes, what kube-apiserver should I contact? If I just pick my favorite and it dies, what happens?

The control plane

The Kubernetes control plane is an abstraction and users should not have to worry about contacting different kube-apiservers and dealing with the fact that their favorite kube-apiserver might disappear and another one might spin up. This is why Kubernetes will not even let you join multiple master nodes to the same cluster, without handling this problem first.

Users of the cluster should only know one IP address to contact, and this IP address should be stable.

A cloud provider will simply hand you a load balancer with a stable IP address and you are good to go, but this is a manual process in a bare metal setup.

So we need to set up a stable IP for the Kubernetes control plane, a job that can be done in a few ways.

We set up a virtual IP for the control plane by installing HAProxy and Keepalived. In short, HAProxy uses ARP to broadcast that a virtual IP address should be translated to the physical machine's MAC address. This means that when anyone wants to contact the IP address we set up for the control plane the traffic will be redirected to this physical machine. The Keepalived service ensures that this machine can always be contacted, and if it can't the virtual IP will switch to another machine. This makes the virtual IP stable and can therefore be used as the IP address for the control plane.

This approach is great and simple but depending on the implementation we might get into trouble. If we install it besides Kubernetes on our machine, what happens if the HAproxy or Keepalivd service fails? The whole master node will then be considered down, and because we manually need to go in and restart the service, we lose the orchestration benefits of Kubernetes. Well, then let us install it as pods inside Kubernetes. Then if one of the services fails, Kubernetes will just bring them back up again.

Sadly this introduces a chicken and egg situation:

Setup a stable IP in HAProxy and Keepalived to initiate a highly available Kubernetes cluster.
Setup a Kubernetes cluster so that you can host HAproxy and Keepalived in pods.
Cry

Static pods to the rescue

Fortunately, I lied when I said that all communication to the cluster goes through the kube-apiserver. All the pods in the control plane cannot go through the normal process of contacting the kube-apiserver to get deployed, well because the kube-apiserver is not deployed yet. They are what is known as static pods and get deployed simply by putting their YAML files in the right folder. In Linux this is /etc/kubernetes/manifests/ by default. All YAML files in this folder will get deployed with the control plane when we initialize our cluster. This seems like it solved our chicken and egg problem.

Stay tuned

Well, now we have gone through the theory needed to deploy a highly available Kubernetes cluster on bare metal. This means that we can spin up a cluster on a few of our old laptops or raspberry pis.

If this peaked your interest, follow along next week when I actually show how easy it is to do this in practice.