Forem: Calvin Tran

How do I deploy my machine learning model on local kubernetes

Calvin Tran — Mon, 04 Nov 2019 04:29:09 +0000

Introduction

Being a data scientist / data engineer / data wizards …. we don’t want to spend so much our time and efforts to setup a development environment.

With the help of Kind (Kubernetes in Docker), we can run a single cluster kubernetes in docker and even deploy our model into it.

Why kind ?

Yes, why kind while we have other options: minikube, microk8s, kubeadm,..

Kind using docker while minikube requires virtualbox which consume more resources
Kind can run on macbook, ubuntu, linux, window. Basically, if you have docker, you can run kind while microk8s only supports ubuntu.
In Kind, we have the cluster with single command

Installation kind on my local

Kind has available stable binaries. This is the easiest way to run kind on my local machine with CLI. According to the docs, we have to download the binary and and place in to PATH

$ curl -Lo ./kind https://github.com/kubernetes-sigs/kind/releases/download/v0.5.1/kind-$(uname)-amd64
$ chmod +x ./kind
$ mv ./kind /usr/local/bin/kind

Set up kubectl

We also need Kubernetes command-line tool, the installation can be found at the documentation (https://kubernetes.io/docs/tasks/tools/install-kubectl/)

Make sure kubectl is running

$ kubectl version

Once kind is ready, we run these commands to have kubeconfig in our environment

$ export KUBECONFIG=”$(kind get kubeconfig-path)”
$ kubectl cluster-info

Hoooray !! everything is ready. Let’s jump to the deployment our first model.

Deploy iris model

I always use iris model as a “Hello World” example for every project. The deployment.yaml manifest have been prepared

apiVersion: apps/v1
kind: Deployment
metadata:
  name: iris-app
  labels:
    app: iris-app
    tier: backend
    version: v1
spec:
  selector:
    matchLabels:
      app: iris-app
  replicas: 2
  template:
    metadata:
      labels:
        app: iris-app
    spec:
      containers:
      - name: iris-app
        image: canhtran/iris-svm-model-api:latest
        ports:
        - containerPort: 5000

Apply deployment to the cluster.

$ kubectl apply -f https://gist.githubusercontent.com/canhtran/33d7bf2161fe0452b4a0481f6093160a/raw/3600ac616b03fb3e9ff3c81ced52e45a790cdc9b/iris-api-depployment.yaml

deployment.apps/iris-app created

I create a service to expose the deployment.

$ kubectl expose deployment iris-app --type=LoadBalancer --name=iris-svc

service/iris-svc exposed

Finally, we forward the port to our local machine for testing.

$ kubectl port-forward svc/iris-svc 5000:5000

Forwarding from 127.0.0.1:5000 -> 5000
Forwarding from [::1]:5000 -> 500

Test with curl command.

$ curl -i -X POST -H "Content-Type:application/json" http://localhost:5000/iris/predict -d '{"payload":[6.2, 3.4, 5.4, 2.3]}'

HTTP/1.1 200 OK
Server: gunicorn/19.5.0
Date: Mon, 02 Sep 2019 02:51:04 GMT
Connection: close
Content-Type: application/json
Content-Length: 13

{"result":2}

The service is up and running. Everything work perfectly.

Conclusion

In my opinion, KinD (Kubernetes in Docker) is the best alternative so far compare with minikube / microk8s. With a few commands, we’re able to have an environment to deploy our models.

DataOps part 3: deploy Titanic to kubernetes

Calvin Tran — Fri, 01 Nov 2019 03:38:09 +0000

Image Source: Blue Star Line

This is the last part of dataops series, if you haven’t gone through the first and second ones, here is the link: part1, part2

Prerequisite

We have to make sure that our kubernetes cluster has an ingress installed. If you have configured, here is the way how to do it

For vagrant kubernetes

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/service-nodeport.yaml

For microk8s

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml

$ microk8s.enable ingress

Deployment

Before starting with deployment process, let’s go through the process flow.

That are the steps for deploy machine learning model on kubernete the easy way.

Create a deployment (Deployment runs multiple replicas of your application and automatically replaces any instances that fail or become unresponsive — google k8s).
Expose deployment to services (Services for expose your pods into network, there are different type of services like NodePort or LoadBalancer).
Create ingress (Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster, on another way, ingress help outside traffic access to our API).

Create Deployment

Deployment is an object in kubernetes that let you manage a set of identical pods. Without the deployment, you have to create, update, delete the pod manually

apiVersion: apps/v1
kind: Deployment
metadata:
  name: titanic-app
  labels:
    app: titanic-app
    tier: backend
    version: v1
spec:
  selector:
    matchLabels:
      app: titanic-app
  replicas: 2
  template:
    metadata:
      labels:
        app: titanic-app
    spec:
      containers:
      - name: titanic-app
        image: canhtran/titanic-app-demo:latest
        ports:
        - containerPort: 5000

I create two replicas (2 pods) for our api, the default port is flask port 5000.

# Apply deployment
$ kubectl apply -f deployment-titanic.yaml
deployment.apps/titanic-app created

We can check the pods status.

$ kubectl get pod
NAME                           READY   STATUS              RESTARTS   AGE
titanic-app-6f4fdf8668-7nctk   1/1     Running             0          64s
titanic-app-6f4fdf8668-zhghw   0/1     ContainerCreating   0          64s

Expose services

Unlike deployment is used to keep a set of pods running by creating pods from a template, service is used to allow network access to a set of pods.

$ kubectl expose deployment titanic-app --type=LoadBalancer --port=5000
service/titanic-app exposed

Also get our service status

$ kubectl get svc
NAME          TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
titanic-app   LoadBalancer   10.102.5.24   <pending>     5000:32755/TCP   14

Create ingress

Finally, an ingress so that our service can be accessible from outside.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: titanic-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  rules:
  - http:
      paths:
        - path: /
          backend:
            serviceName: titanic-app
            servicePort: 5000

$ kubectl apply -f ingress-titanic.yaml
ingress.extensions/titanic-ingress created

Let's wrap it up and test our API

$ curl -i -X POST -H "Content-Type:application/json" http://10.98.11.201/predict -d '{"Pclass":3,"Sex":1,"Age":2,"Fare":7,"Embarked":0,"Title":1,"FamilySize":2,"IsAlone":0,"Age_Class":6, "FarePerPerson":3}'

HTTP/1.1 200 OK
Server: openresty/1.15.8.1
Date: Mon, 29 Jul 2019 12:00:10 GMT
Content-Type: application/json
Content-Length: 13
Connection: keep-alive

{"result":1}

The API have been up and running as expected :)

DataOps part 2: Sail the Titanic to API

Calvin Tran — Thu, 31 Oct 2019 03:59:40 +0000

Before starting the deployment part, let’s prepare our model. If you haven’t had the running Kubernetes environment, refer to Part 1.

To be short, in this part, I will cover:

Training the model and persistence model using pickle.
Create Restful API to serve model file.

I won’t go details about the EDA or training model in this article however I made a notebook using Titanic dataset (can be found here). I used different machine learning methods to train the model and here is the results.

The winner belongs to Random Forest & Decision Tree with accuracy 93.60%. I exported the random forest model to pickle format model.pkl, we use this file from now onward.

import pickle
pickle.dump(random_forest, open('model.pkl', 'wb'))

Flask framework

One app.py file to serve the titanic model.

import pickle
import numpy as np
from flask import (
    Flask,
    jsonify,
    request
)


app = Flask(__name__)
with open('model.pkl', 'rb') as handle:
    app.model = pickle.load(handle)


@app.route('/heartbeat', methods=['GET'])
def heartbeat():
    return 'ok'


@app.route('/predict', methods=['POST'])
def predict():
    payload = request.get_json()
    np_array = np.expand_dims(list(payload.values()), 0)

    pred = app.model.predict(np_array)
    if pred is None:
        return jsonify({'error': 'Model cannot predict with input'})
    else:
        return jsonify(result=pred[0].item())


if __name__ == "__main__":
    app.run()

Let’s break it down

with open('model.pkl', 'rb') as handle:    
   app.model = pickle.load(handle)

I load the model to app.model variable during starting flask server, the model will be stored in memory so that it doesn’t reload every time we send requests.

I also create a GET endpoint name heartbeat which returned ‘OK’ (200 response code). This using by kubernetes to check the liveness of api.

The main part of service is from line 21. It gets the parameters from POST request and convert it to 2D numpy array

np_array = np.expand_dims(list(payload.values()), 0)

This step is required, machine learning model only accept the 2D numpy array as the input.

Let’s do a quick test for our API

# Run the server
$ python app.py
* Serving Flask app "app" (lazy loading)
* Environment: development
* Debug mode: on
* Running on [http://127.0.0.1:5000/](http://127.0.0.1:5000/) (Press CTRL+C to quit)

And curl on another terminal.

$ curl -i -X POST -H "Content-Type:application/json" [http://localhost:5000/predict](http://localhost:5000/predict) -d '{"Pclass":3,"Sex":1,"Age":2,"Fare":7,"Embarked":0,"Title":1,"FamilySize":2,"IsAlone":0,"Age_Class":6, "FarePerPerson":3}'

HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 18
Server: Werkzeug/0.14.1 Python/3.6.2
Date: Sun, 14 Jul 2019 16:48:57 GMT

{
  "result": 1
}

That’s awesome, our API has been working as expected. One more final step, dockerize our application and push it to dockerhub

FROM python:3.6

LABEL maintainer "Calvin Tran <trandcanh@gmail.com>"

ENV APP_HOME /app

RUN mkdir -p $APP_HOME

WORKDIR $APP_HOME

COPY requirements.txt $APP_HOME

RUN apt-get update && \
    pip install -I --no-cache-dir -r requirements.txt && \
    rm requirements.txt && \
    rm -rf /var/lib/apt/list/*


COPY app.py $APP_HOME
COPY model.pkl $APP_HOME

CMD gunicorn -w 3 -b 0.0.0.0:5000 --timeout=120 --reload "app:app"

VOLUME $APP_HOME

A few commands for pushing our image to docker hub.

$ docker build -t titanic-api .
$ docker tag titanic-api canhtran/titanic-app-demo
$ docker push titanic-api canhtran/titanic-app-demo

The image will be available on https://hub.docker.com. Alternatively, you can use my image.

$ docker pull canhtran/titanic-app-demo

All right! Our preparation have been completed and ready to deploy to kubernetes.

DataOps: Kubernetes on local with vagrant and kubeadm

Calvin Tran — Wed, 30 Oct 2019 15:37:10 +0000

This series is a story about a data guy (which is me) with zero knowledge of Kubernetes pick up devops skill and successfully deploy a machine learning model.

I have been working with Kubernetes for a few months, the cluster is set up by devops team and me myself is only a user. It’s time for me to take a break, build my own local cluster and understand deeply the whole process.

If you guys don’t want to setup a k8s cluster from scratch, Microk8s is the best choice, you can get a full kubernetes system running in under 60s (like snap a finger)

Kubeadm vs Minikube.

After researching, I found two candidates that good for my local cluster:

Minikube is the most popular way to deploy Kubernetes locally, it comes with support for a variety of hypervisors, set up is completely automatic within a few minutes. You really doesn’t need to take care anything.

However, I want to get my hand dirty by doing some installation stuffs as well as understand how to set up Kubernetes cluster. Hence, one of the most frequency criticism that Kubernetes is very hard to install, Kubeadm really makes this easier.

Installation

TL;DR: There are 2 the basic steps that I did:

Spin up a vagrant machine with Ubuntu 16.04 and at least 2GB of memory.
Install and run kubeadm on single master cluster. (official guide can be found at https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/)

Vagrant

A simple vagrant with Ubuntu 16.04 and 2Gb of memory is good enough for this series. You can copy the following code, put it to Vagrantfile and run it with vagrant up .

Vagrant.configure("2") do |config|
  config.vm.box = "bento/ubuntu-16.04"
  config.vm.box_check_update = false
  config.vm.network "private_network", type: "dhcp"
  config.vm.provider "virtualbox" do |vb|
    vb.gui = false
    vb.memory = "2048"
    vb.customize ["modifyvm", :id, "--cableconnected1", "on"]
  end
  config.vm.provision "docker"
end

Inside the VM, I do a couple of prerequisite steps. Turn off swap and make sure it won’t run again whenever I reboot my machine.

$ swapoff -a 
$ sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

Kubeadm

I follow the official guide to install kubernetes on ubuntu.

$ apt-get update && apt-get install -y apt-transport-https curl
$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
$ cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
$ apt-get update
$ apt-get install -y kubelet kubeadm kubectl
$ apt-mark hold kubelet kubeadm kubectl

The most difficult for me during the installation is to config the cgroup driver used by Kubelet. Luckily, A post from Liz Rice saves me, she did the kubeadm on vagrant with a little hack code, I copy it from her.

$ sed -i '0,/ExecStart=/s//Environment="KUBELET_EXTRA_ARGS=--cgroup-driver=cgroupfs"\n&/' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

Initializing the master with IP which generated from DHCP, the reason for it is we can connect to our cluster from outside/host machine.

$ kubeadm init --apiserver-cert-extra-sans=<IP Address>  --node-name $(hostname -s)

To get IP Address, we use ifconfig command, the default network name is eth1

$ ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 08:00:27:1c:68:af
          inet addr:172.28.128.4  Bcast:172.28.128.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe1c:68af/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:21 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2800 (2.8 KB)  TX bytes:2978 (2.9 KB)
          Interrupt:16 Base address:0xd240

I make the kubectl works for vagrant user

$ sudo --user=vagrant mkdir -p /home/vagrant/.kube
$ cp -i /etc/kubernetes/admin.conf /home/vagrant/.kube/config
$ chown $(id -u vagrant):$(id -g vagrant) /home/vagrant/.kube/config

Installing a pod network add-on, we can find the list from third-party pod network providers here. I choose weave net because they support multiple architect (amd64, arm, arm64 and ppc641e)

$ kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

Finally, I make the master isolate. By default, the cluster will not schedule pods on the master for security reason and we have only one node for both master / worker.

$ kubectl taint nodes --all node-role.kubernetes.io/master-

Test our cluster is running with kubectl command.

$ kubectl get nodes
NAME      STATUS   ROLES    AGE     VERSION
vagrant   Ready    master   3h28m   v1.12.1

Below is my sh file for install kubeadm. You also can find the Vagrantfile with full installation process at my repo: https://github.com/canhtran/data-devops-medium/tree/master/vagrant.d

# Install kubernetes
apt-get update && apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl

# kubelet requires swap off
swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

# Configure cgroup driver used by kubelet on Master Node
sed -i '0,/ExecStart=/s//Environment="KUBELET_EXTRA_ARGS=--cgroup-driver=cgroupfs"\n&/' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

IPADDRESS=`ifconfig eth1 | grep "inet addr" | cut -d ':' -f 2 | cut -d ' ' -f 1`

# Set up Kubernetes
kubeadm init --apiserver-cert-extra-sans=$IPADDRESS  --node-name $(hostname -s)

# Make kubectl work for your non-root user
sudo --user=vagrant mkdir -p /home/vagrant/.kube
cp -i /etc/kubernetes/admin.conf /home/vagrant/.kube/config
chown $(id -u vagrant):$(id -g vagrant) /home/vagrant/.kube/config

# Installing a pod network add-on
export KUBECONFIG=~vagrant/.kube/config
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

# Master Isolation
kubectl taint nodes --all node-role.kubernetes.io/master-

Optional

In order to get a kubectl on other computers (e.g. laptop) talk to the cluster, we need to copy the administrator kubeconfig file from master to our workstation.

By default, the SSH access in vagrant is not enable for root. We have to copy the admin.conf file to be accessible by some users.

$ vagrant ssh # Login to vagrant
$ sudo -i # Get root access

The from our machine, we can copy it via scp and test with get nodes command. (We have to change the server ip in admin.conf with the vagrant ip which is eth1 ip).

$ scp -r -P 2222 vagrant@127.0.0.1:/home/vagrant/admin.conf .
##### Edit and change the IP address ####
$ kubectl --kubeconfig ./admin.conf get nodes
NAME      STATUS   ROLES    AGE     VERSION
vagrant   Ready    master   3h28m   v1.12.1

After we have a running cluster, let’s start with helm chart

Helm chart

Helm chart is Kubernetes package management.

Kubernetes can become very complex with a tons of objects that we need to handle: configmaps, services, pods, persistance volumnes. I should mention that the number of release of each service also a big headache for us to manage. These can be solved with Helm, which offers us a simple way to manage, deploy in one simple application with standard template (chart).

Base on the documentation

Helm uses a packaging format called charts. A chart is a collection of files that describe a related set of Kubernetes resources. A single chart might be used to deploy something simple, like a memcached pod, or something complex, like a full web app stack with HTTP servers, databases, caches, and so on.

Charts are created as files laid out in a particular directory tree, then they can be packaged into versioned archives to be deployed

Helm chart can be installed on a specific namespace. In this series, I will create a new namespace call data

$ kubectl create namespace data

Then I can install helm with simple command

$ helm init --tiller-namespace data

Helm will create a Tiller pod in Kubernetes cluster under namespace data. To verify the success of our installation.

$ kubectl get pods -n data 
NAME                             READY     STATUS    RESTARTS   AGE
tiller-deploy-5b9757b87c-s295g   1/1       Running   0          23s

That’s all, everything is ready for the next chapter.