Forem: Guilherme Teixeira

An easy way to a highly available service with Kubernetes probes, Kotlin and Helm

Guilherme Teixeira — Mon, 07 Dec 2020 18:14:41 +0000

A simple explanation with examples of how Kubernetes probes work and help keeping your services always up.

Code is hosted in Github

Cover image from unsplash by Braden Collum

Prerequisites (with the versions used in this tutorial):

Docker (Engine 19.03.13)
Minikube (1.15.1)
Helm (3.4.0)
Kubectl (matching kubernetes version)

About Probes

Probes (sometimes heartbeat or health checks) are a fundamental feature of Microservices. They perform constant checks to know the state of a service and enable recovery whenever things go sour.

You can set up your probes as a really simple endpoint that returns a valid HTTP status code whenever the service is running or they can be more complex and look for things like the state of the database connection or whether the service can communicate with an external dependency, send messages to a queue, etc...

Kubernetes provides out-of-the-box options to set up those checks. If configured together with a multi-replica, multi-node environment, probes allow for a proactive error handling with important features to recover from failures.

They are divided in three categories:

Liveness Probe

Liveness Probe is used to know when a pod/container should be restarted because it is running but not functional. It helps increasing availability by bringing the service back to its original state.

Whenever the initialDelaySeconds value is passed, Kubernetes starts to perform the specified command and if the response is something other than 200-399 it fails. Unless failureThreshold value is changed, the pod will be restarted after 3 failures in a row.

Readiness Probe

Readiness Probe is similar to the liveness probe in terms of options and implementation but has the function to inform Kubernetes when a pod is ready to receive traffic. Sometimes an application has to perform tasks on startup. Maybe load some data in memory, warm up a cache, process information before receiving new requests, you name it. In these scenarios, even if the service is actually running it isn't ready to process any request.

Kubernetes changes the pod condition to ready whenever the ready endpoint returns an HTTP code between 200-399. successThreshold value can be used to determine how many times the return has to be valid before the pod receives traffic.

Startup Probe

Startup Probe provides a delay for both other probes to start. It's specially helpful when there's a slow starting container which guaranteed takes longer to finalize and make the liveness and readiness probe endpoints available. We will skip it in this tutorial.

Implementation starting point

In the last article of this series we created a Kotlin app with Spring Boot and deployed on Kubernetes using Helm. Now we will extend it to include liveness and readiness probes.

Writing probe endpoints

In the Application.kt file, let's create two new endpoints:

    @GetMapping("/health")
    @ResponseStatus(HttpStatus.OK)
    fun health(): String {
        println("Liveness Probe");
        return "Healthy";
    }

    @GetMapping("/ready")
    @ResponseStatus(HttpStatus.OK)
    fun ready(): String {
        println("Readiness Probe");
        return "Ready";
    }

Note that this will always return success, provided that the service is running. Later we will simulate failures.

Extending Helm deployment template

Kubernetes probes are part of the deployment specification. So in your Helm's deployment.yaml template, add the following inside the containers spec:

    livenessProbe:
        httpGet:
            path: /health
            port: 8080
        initialDelaySeconds: 10
        periodSeconds: 3
    readinessProbe:
        httpGet:
            path: /ready
            port: 8080
        initialDelaySeconds: 15
        periodSeconds: 5

In this case we are telling Kubernetes to start performing the liveness checks after 10 seconds and repeating it every 3 seconds, at the /health endpoint on the service port and readiness checks after 15 seconds, every 5 seconds on endpoint /ready.

Build and run app

To test the configuration, just build and run the app on Kubernetes (instructions on how to do this with Minikube are available here or you can use the run.sh script on the root of the repository).

After running kubectl get pods shows that the pod is running and the logs are printing our messages:

~/code/demos/demo-spring-boot master >2 > kubectl get pods
NAME                                READY   STATUS    RESTARTS   AGE
demo-spring-boot-7f7769cfbd-k747j   1/1     Running   0          19s
~/code/demos/demo-spring-boot master >2 > kubectl logs -f demo-spring-boot-7f7769cfbd-k747j

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v2.4.0)

2020-12-06 20:26:05.812  INFO 1 --- [           main] c.g.d.DemoSpringBootApplicationKt        : Starting DemoSpringBootApplicationKt using Java 15-ea on demo-spring-boot-7f7769cfbd-k747j with PID 1 (/app.jar started by root in /)
2020-12-06 20:26:05.816  INFO 1 --- [           main] c.g.d.DemoSpringBootApplicationKt        : No active profile set, falling back to default profiles: default
2020-12-06 20:26:09.290  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2020-12-06 20:26:09.372  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2020-12-06 20:26:09.372  INFO 1 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.39]
2020-12-06 20:26:09.510  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2020-12-06 20:26:09.510  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 3524 ms
2020-12-06 20:26:10.276  INFO 1 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'applicationTaskExecutor'
2020-12-06 20:26:10.705  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
2020-12-06 20:26:10.775  INFO 1 --- [           main] c.g.d.DemoSpringBootApplicationKt        : Started DemoSpringBootApplicationKt in 6.565 seconds (JVM running for 7.716)
2020-12-06 20:26:12.900  INFO 1 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2020-12-06 20:26:12.900  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2020-12-06 20:26:12.901  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 1 ms
Liveness Probe
Liveness Probe
Readiness Probe
Liveness Probe
Readiness Probe
Liveness Probe
Liveness Probe
Readiness Probe

Probe failures

Now in order to show how Kubernetes probes react in case of failures, let's simulate failures on our endpoints. I extended the Kotlin code to have a Counter object which increments a value by 1 whenever a health check happens. When the value reaches 10, our service will not be considered ready anymore because it will return a 503 Service Unavailable. With 15, health check will fail and the pod will be restarted:

package com.gateixeira.demospringboot.controller

import org.springframework.http.HttpStatus
import org.springframework.web.bind.annotation.GetMapping
import org.springframework.web.bind.annotation.RestController
import org.springframework.http.ResponseEntity

@RestController
class Application {
    object Counter {
        public var value = 0
        fun count(): Int = value++
    }

    @GetMapping("/")
    fun home(): String {
        return "Hello World";
    }

    @GetMapping("/health")
    fun health(): ResponseEntity<String> {
        Counter.count();

        if (Counter.value > 15) {
            println("Service not healthy. Count: " + Counter.value);
            return ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE).body("Not healthy");
        } else {
            println("Service is healthy. Count: " + Counter.value);
            return ResponseEntity.status(HttpStatus.OK).body("Healthy");
        }
    }

    @GetMapping("/ready")
    fun ready(): ResponseEntity<String> {

        if (Counter.value > 10) {
            println("Service not ready. Count: " + Counter.value);
            return ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE).body("Not ready");
        } else {
            println("Service is ready. Count: " + Counter.value);
            return ResponseEntity.status(HttpStatus.OK).body("Ready");
        }
    }
}

I also added a couple new properties to the deployment.yaml template just to make sure that the thresholds are set to 1 and it ended up looking like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: {{ .Values.namespace }}
  name: {{ .Values.appName }}
  labels:
    app: {{ .Values.appName }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: {{ .Values.appName }}
  template:
    metadata:
      labels:
        app: {{ .Values.appName }}
    spec:
      containers:
        - name: "{{ .Values.appName }}"
          image: "{{ .Values.image.registry }}/{{ .Values.appName }}:{{ .Values.appVersion }}"
          imagePullPolicy: "{{ .Values.image.pullPolicy }}"
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 3
            successThreshold: 1
            failureThreshold: 1
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 5
            successThreshold: 1
            failureThreshold: 1
      restartPolicy: Always

Now let's build and start our service again and see the probes happening. The terminal window on top shows the pod going from READY 0/1 to READY 1/1 after a successful readiness probe:

After a couple seconds, pod goes back to READY 0/1 because counter already passed 10 so it will not receive traffic anymore:

And finally, pod will be restarted with the failure of a liveness probe:

All these checks caused our pod lifecycle to go from starting up when it's not ready yet to running and ready to receive traffic. Then not ready, to a complete failure causing it to restart so that it could finally be ready again. All within the timespan of a minute:


~/code/demos/demo-spring-boot master > kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
demo-spring-boot-f8bd8cdff-k8csl   0/1     Running   0          9s
~/code/demos/demo-spring-boot master > kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
demo-spring-boot-f8bd8cdff-k8csl   1/1     Running   0          19s
~/code/demos/demo-spring-boot master > kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
demo-spring-boot-f8bd8cdff-k8csl   0/1     Running   0          48s
~/code/demos/demo-spring-boot master > kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
demo-spring-boot-f8bd8cdff-k8csl   0/1     Running   1          60s
~/code/demos/demo-spring-boot master > kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
demo-spring-boot-f8bd8cdff-k8csl   1/1     Running   1          79s

Congratulations! You learned how Kubernetes probes work in practice and might have just won a "9" in your application up time.

Deploying a Spring Boot Kotlin app on Kubernetes with Docker and Helm

Guilherme Teixeira — Mon, 30 Nov 2020 08:37:22 +0000

The easiest hello-world application deployed on Kubernetes with Helm that you can put together.

Code is hosted in Github

Prerequisites (with the versions used in this tutorial):

Docker (Engine 19.03.13)
Minikube (1.15.1)
Helm (3.4.0)
Kubectl (matching kubernetes version)

Spring-boot app setup

On the spring initializr, create your app project. In this case we go with Gradle, Kotlin and Spring Boot 2.4.0 for Java 15 packaged in a JAR file. For the dependencies, let's select Spring Web so that we can initialize a Hello World REST API:

After generating it, you should have your project folder, which is the root to our repository.

Writing a hello-world REST API

for a simple REST API, just create a controller folder on the same folder as your DemoSpringBootApplication.kt, create an Application.kt file inside and add this content:

package com.gateixeira.demospringboot.controller

import org.springframework.web.bind.annotation.GetMapping
import org.springframework.web.bind.annotation.RestController

@RestController
class Application {
    @GetMapping("/")
    fun home(): String {
        return "Hello World";
    }
}

Build and run app

The generated zip file comes with an embedded Gradle executable, so in order to build our app and make sure that everything is working just do a ./gradlew build:

~/code/demos/demo-spring-boot master > ./gradlew build
Starting a Gradle Daemon (subsequent builds will be faster)

> Task :test
2020-11-29 20:44:55.227  INFO 43582 --- [extShutdownHook] o.s.s.concurrent.ThreadPoolTaskExecutor  : Shutting down ExecutorService 'applicationTaskExecutor'

BUILD SUCCESSFUL in 8s
7 actionable tasks: 3 executed, 4 up-to-date

This process generates a jar file under build/libs. This is our executable. You can run it with java -jar build/libs/<your-app>. Your terminal will show the Spring banner and the following message:

DemoSpringBootApplicationKt : Started DemoSpringBootApplicationKt in 1.492 seconds (JVM running for 1.852)

Now if you go to localhost:8080 you see "Hello World".

Creating docker image

Now let's write a very simple dockerfile to generate our image. We are going to use Open JDK's alpine version as our base image, with Java 15 to match what we chose above.

FROM openjdk:15-jdk-alpine
ARG JAR_FILE=build/libs/*.jar
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["java", "-jar", "/app.jar"]

It simply copies the generated JAR file to the container as app.jar and uses this as entrypoint for a Java application.

Writing Helm chart

Helm will be used to manage our application lifecycle inside Kubernetes. This will probably be the simplest chart possible but can easily be extended.

First, create a charts folder on your project's root. Then inside, create a Chart.yaml that specifies basic chart information:

apiVersion: v2
name: helm
description: "A Helm chart for our demo-spring-boot application"
type: application
version: 0.1.0
appVersion: latest

In the charts folder, create a templates folder and add a deployment.yaml inside:

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: {{ .Values.namespace }}
  name: {{ .Values.appName }}
  labels:
    app: {{ .Values.appName }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: {{ .Values.appName }}
  template:
    metadata:
      labels:
        app: {{ .Values.appName }}
    spec:
      containers:
        - name: "{{ .Values.appName }}"
          image: "{{ .Values.image.registry }}/{{ .Values.appName }}:{{ .Values.appVersion }}"
          imagePullPolicy: "{{ .Values.image.pullPolicy }}"
      restartPolicy: Always

Everything defined as {{ variable name }} is a placeholder and will be set in the values file.

This is a very simple deployment file that specifies some metadata, labels match and selectors as well as the name of the container, it's image repository address and pull policy.

Now in the same folder, create a service.yaml file:

apiVersion: v1
kind: Service
metadata:
  namespace: {{ .Values.namespace }}
  name: {{ .Values.appName }}
  labels:
    app: {{ .Values.appName }}
spec:
  type: LoadBalancer
  loadBalancerIP: 172.16.0.1
  sessionAffinity: None
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
      name: http
  selector:
    app: {{ .Values.appName }}

This file defines the LoadBalancer configuration for the application, with an IP Address as entry point and the respective port in which the service is exposed (80) and the app is running (8080). Selector specifies the deployment to which this service relates.

Now in order to connect everything and replace the placeholders, let's create a values.yaml file in the charts folder:

appName: demo-spring-boot
namespace: default
appVersion: latest
replicaCount: 1
image:
  registry: gateixeira
  pullPolicy: Never

Our app will be on the default namespace, have a latest version and just 1 single replica. The container image will be composed of the tag that was added on the docker build command. In this case: gateixeira as the container registry, the app name as image name and concatenated with the version. pullPolicy is set to never since we are building a local image.

Deploying to Kubernetes

Starting Minikube

Our Kubernetes will be running locally with Minikube:

~/code/demos/demo-spring-boot master > minikube start --memory 2048 --cpus 2 --disk-size 10g --kubernetes-version v1.18.8
😄  minikube v1.15.1 on Darwin 10.15.6
❗  Both driver=docker and vm-driver=virtualbox have been set.

    Since vm-driver is deprecated, minikube will default to driver=docker.

    If vm-driver is set in the global config, please run "minikube config unset vm-driver" to resolve this warning.

✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating docker container (CPUs=2, Memory=2048MB) ...
🐳  Preparing Kubernetes v1.18.8 on Docker 19.03.13 ...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass

❗  /usr/local/bin/kubectl is version 1.16.7, which may have incompatibilites with Kubernetes 1.18.8.
    ▪ Want kubectl v1.18.8? Try 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Now our app should be ready to be deployed on Kubernetes. First, let's build the docker image.

Start by switching the docker environment to use Minikube's daemon, otherwise Minikube can't find the image:

eval $(minikube docker-env)

For those running on Windows, the powershell equivalent is

minikube docker-env | Invoke-Expression

Then build the image with docker build -t <image_name>:<image_tag> .

~/code/demos/demo-spring-boot master > docker build -t gateixeira/demo-spring-boot:latest .                                                                                     4s
Sending build context to Docker daemon  23.16MB
Step 1/4 : FROM openjdk:15-jdk-alpine
 ---> f02adfce91a2
Step 2/4 : ARG JAR_FILE=build/libs/*.jar
 ---> Using cache
 ---> 9bb20485adba
Step 3/4 : COPY ${JAR_FILE} app.jar
 ---> e651311f698a
Step 4/4 : ENTRYPOINT ["java", "-jar", "/app.jar"]
 ---> Running in 0ed363cc6d8b
Removing intermediate container 0ed363cc6d8b
 ---> 0033b968a8a9
Successfully built 0033b968a8a9
Successfully tagged gateixeira/demo-spring-boot:latest

To install your application on Minikube, run:
helm upgrade --install demo-spring-boot charts --values charts/values.yaml. Where charts is your charts folder.

~/code/demos/demo-spring-boot master > helm upgrade --install demo-spring-boot charts --values charts/values.yaml                                               INT kube minikube
WARNING: "kubernetes-charts.storage.googleapis.com" is deprecated for "stable" and will be deleted Nov. 13, 2020.
WARNING: You should switch to "https://charts.helm.sh/stable"
Release "demo-spring-boot" does not exist. Installing it now.
NAME: demo-spring-boot
LAST DEPLOYED: Sun Nov 29 21:25:39 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

You can check your application is running with kubectl get pods:

~/code/demos/demo-spring-boot master !1 > kubectl get pods                                                                                                          kube minikube
NAME                               READY   STATUS    RESTARTS   AGE
demo-spring-boot-684cc98cc-zxgfv   1/1     Running   0          101s

With your app running, Minikube can tunnel your application and provide a URL to access it:

~/code/demos/demo-spring-boot master !1 > minikube service demo-spring-boot --url
🏃  Starting tunnel for service demo-spring-boot.
|-----------|------------------|-------------|------------------------|
| NAMESPACE |       NAME       | TARGET PORT |          URL           |
|-----------|------------------|-------------|------------------------|
| default   | demo-spring-boot |             | http://127.0.0.1:61460 |
|-----------|------------------|-------------|------------------------|
http://127.0.0.1:61460

Congratulations! 😄

If you open your browser on http://127.0.0.1:61460 you should see Hello World.

Auto-unseal your Vault Instance on Kubernetes with Azure Key Vault — The Definitive Guide

Guilherme Teixeira — Sun, 18 Oct 2020 13:58:25 +0000

This guide intends to provide a distilled, reasonable, secure and yet simple setup for auto-unsealing Vault on Kubernetes with Azure Key Vault. While I believe the official Hashicorp's guide brings a considerable amount of extra information on how to set up Vault with Terraform, it may not reflect a typical scenario for Vault usage. If you already have an up and running Kubernetes cluster and want to continue to use Helm to install and manage Vault for keeping your application's secrets, this guide is for you.

Prerequisites (with the versions used in this tutorial):

Docker (Engine 19.03.13)
Minikube (1.13.1)
Helm (3.2.1)
Kubectl (matching kubernetes version)
Azure subscription

First, let's start by setting up all required Azure services. On your subscription, create an instance of Azure Key Vault with all default settings:

From this instance you should note down the Subscription ID and your Directory ID, which will be necessary later.
Next, we have to register an App, so that the Service Principal can work as our access layer to the Azure Key Vault. You can do so by going to App Registration:

Now that we have a Service Principal, note its Application (client) ID:

On this Service Principal, we need to create a client secret under Certifications & Secrets:

It's extremely important that you copy the value and save it for later since this is not retrievable after you close the window.

After this is set, switch back to your Azure Key Vault and under IAM add a new Role Assignment as Key Vault Contributor for your Service Principal:

Once your Service Principal is assigned to the Azure Key Vault, we need to provide specific access permissions to it. Vault needs Get permissions at key level and Unwrap Key and Wrap Key at Cryptographic level, so under access policy, create a new one with those selected:

Select your Service Principal in the list:

After you click on "Add", don't forget to save the configuration. (I missed this step and it took me a while to figure out why my Vault complained it had no read permissions).
The last step at Azure portal is to create a key on Key Vault, which will be your unsealing key for Vault:

At this point, you finished all the needed configuration on Azure and are left with some important information that will allow your Vault to communicate and auto-unseal. To set it up, what we will do is to create a values.yaml file for Helm, based on Vault's official helm chart. In our scenario, to have a good trade-off between security, availability and ease of configuration we are going to bootstrap Vault with 2 replicas in a Raft mode as the storage type. Where Raft consensus algorithm supports High Availability and ensures data replication across replicas. The data is stored encrypted in a Kubernetes persistent volume. It's important to highlight that since our Kubernetes setup aims a local Minikube instance, we need to clean up the pod affinity configuration, otherwise the second instance will not start up due to an IP conflict in a single-node environment. If you're configuring this on a multi-node setup, simply remove the "affinity" tag from your values.yaml. You can use the following yaml file and just replace it with your Azure information:

server:
  affinity: "" #important: set affinity empty if you're using a single-node cluster, like Minikube
  ha:
    enabled: true
    replicas: 2
    raft:
      enabled: true
      config: |
        ui = true
        listener "tcp" {
          tls_disable = 1
          address = "[::]:8200"
          cluster_address = "[::]:8201"
        }
        seal "azurekeyvault" {
          tenant_id       = "246dd7ab-f96c-4efb-<last_part_hidden"
          client_id       = "defdac63-6f29-4670-<last_part_hidden"
          client_secret   = "3qT0C1YfnHa_9~kqQf.<last_part_hidden"
          vault_name      = "vault-k8s-data"
          key_name        = "vault-k8s-unsealer-key"
          subscription_id = "f982b9f7-4c2b-4c4c-<last_part_hidden>"
        }

        storage "raft" {
          path = "/vault/data"
        }
        service_registration "kubernetes" {}
  dataStorage:
    enabled: true
    size: 500Mi

What took me a bit of time to figure out is that whatever is given under "config" totally replaces the default configuration from the helm chart. Meaning that besides the "azurekeyvault" config, we need to keep the rest of the configuration existing on values.yaml from the Helm chart repository.

Since you probably noted down all values along the way, it should be quite straightforward to figure out the correct mapping for "azurekeyvault" fields:

tenant_id: Key Vault's Directory ID
client_id: Service Principal's Application ID
client_secret: Service Principal's generated secret (the one that is not retrievable. Hope you saved it!)
vault_name: Name of Azure Key Vault instance
key_name: Name of generated key on Azure Key Vault
subscription_id: ID of the Azure Subscription

With all that filled out, all you need to do is to set up your Vault. Let's assume you are doing a local configuration (not on AKS, which shouldn't differ in terms of Vault commands). First, initialise Minikube:

~/code/demos/vault > minikube start --memory 4096 --cpus 2 --disk-size 10g --kubernetes-version v1.18.8
  😄  minikube v1.13.1 on Darwin 10.15.6
  ❗  Both driver=docker and vm-driver=virtualbox have been set.
  Since vm-driver is deprecated, minikube will default to  driver=docker.
  If vm-driver is set in the global config, please run "minikube config unset vm-driver" to resolve this warning.
  ✨  Using the docker driver based on user configuration
  👍  Starting control plane node minikube in cluster minikube
  🔥  Creating docker container (CPUs=2, Memory=4096MB) ...
  🐳  Preparing Kubernetes v1.18.8 on Docker 19.03.8 ... 
  🔎  Verifying Kubernetes components...
  🌟  Enabled addons: default-storageclass, storage-provisioner
  🏄  Done! kubectl is now configured to use "minikube" by default

Then add the Hashicorp's repository to your helm repos:

~/code/demos/vault > helm repo add hashicorp https://helm.releases.hashicorp.com                                                                                                                                            
"hashicorp" has been added to your repositories

Next, install the Vault Helm chart with your values.yaml as input:

~/code/demos/vault > helm install vault hashicorp/vault -f values.yaml  

  NAME: vault
  LAST DEPLOYED: Sun Oct 18 14:17:45 2020
  NAMESPACE: default
  STATUS: deployed
  REVISION: 1
  TEST SUITE: None
  NOTES:
    Thank you for installing HashiCorp Vault!
  Now that you have deployed Vault, you should look over the docs on  using
  Vault with Kubernetes available here:
  https://www.vaultproject.io/docs/
  Your release is named vault. To learn more about the release, try:
  $ helm status vault
  $ helm get vault

At this point, you should have two replicas of your Vault plus the agent injector pod for communication with your applications. Note that neither of your vault pods are "ready":

~/code/demos/vault > kubectl get pods                                                                                                                                                                                           
NAME                                    READY   STATUS    RESTARTS   
vault-0                                 0/1     Running   0          
vault-1                                 0/1     Running   0          
vault-agent-injector-857cdd9594-b9rdh   1/1     Running   0

This happens because Vault will only try to fetch the unseal key once initialised. But if you check the pods logs, you see that the AzurePublicCloud configuration is in fact enabled:

~/code/demos/vault > kubectl logs -f vault-0                                                                                                                                                                                    
==> Vault server configuration:
Azure Environment: AzurePublicCloud
          Azure Key Name: vault-k8s-unsealer-key
        Azure Vault Name: vault-k8s-data
             Api Address: http://172.18.0.4:8200
                     Cgo: disabled
         Cluster Address: https://vault-0.vault-internal:8201
              Go Version: go1.14.7
              Listener 1: tcp (addr: "[::]:8200", cluster address: "[::]:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
               Log Level: info
                   Mlock: supported: true, enabled: false
           Recovery Mode: false
                 Storage: raft (HA available)
                 Version: Vault v1.5.2
             Version Sha: 685fdfa60d607bca069c09d2d52b6958a7a2febd
2020-10-18T12:17:58.020Z [INFO]  proxy environment: http_proxy= https_proxy= no_proxy=
2020-10-18T12:18:01.355Z [INFO]  core: stored unseal keys supported, attempting fetch
2020-10-18T12:18:01.356Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
==> Vault server started! Log data will stream in below:
2020-10-18T12:18:02.334Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery

Just init your Vault instance directly on pod vault-0 (and keep your generated keys):

~/code/demos/vault > kubectl exec -it vault-0 vault operator init                                                                                                                                                     
Recovery Key 1: FKjt5wkzN5bUBIuR52KrPP1c2Il/f7RZdn5E+ipfNF8s
Recovery Key 2: FCzUyduESPyavh6QtqWZpdnUDKa3bEEpBHbX3NgTrCiU
Recovery Key 3: Tf7FVEpj5tdJLqQqNw/Jt0OytRI5FAZYig/yafSVz3Xg
Recovery Key 4: duLpa/6IozTOR0mkO7sp0CwmnI+1DsC6d2+oZG/A1CIZ
Recovery Key 5: pyVFs/rRFEk9rSn57Ru+KeuAQzW6eurl3j0/pS/JRpXD
Initial Root Token: s.d0LAlSnAerb4a7d6ibkfxrZy
Success! Vault is initialized
Recovery key initialized with 5 key shares and a key threshold of 3. Please
securely distribute the key shares printed above.

Now, for the first time, we see the Vault unsealer in action:

~/code/demos/vault > kubectl get pods                                                                                                                                                                                       
NAME                                    READY   STATUS    RESTARTS   
vault-0                                 1/1     Running   0          
vault-1                                 0/1     Running   0          
vault-agent-injector-857cdd9594-b9rdh   1/1     Running   0

Pod vault-0 has been initialised and came up unsealed, meaning that we configured everything correctly.
However, our replica vault-1 is still not ready. This one will be a follower pod of the leader vault-0. In order to make vault-0 visible, we need to login using our root token (be aware of not overusing and sharing the root token on production):

~/code/demos/vault > kubectl exec -it vault-0 -- vault login s.d0LAlSnAerb4a7d6ibkfxrZy                                                                                                                                     
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key                  Value
---                  -----
token                s.d0LAlSnAerb4a7d6ibkfxrZy
token_accessor       hJEFibLTbUP4sgA8X4tMBqZ8
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

Next, we join vault-1 in the cluster:

~/code/demos/vault > kubectl exec -it vault-1 -- vault operator raft join http://vault-0.vault-internal:8200                                                                                                                    
Key       Value
---       -----
Joined    true

a kubectl get pods shows now that all pods are ready:

~/code/demos/vault > kubectl get pods                                                                                                                                                                                           
NAME                                    READY   STATUS    RESTARTS   
vault-0                                 1/1     Running   0          
vault-1                                 1/1     Running   0          
vault-agent-injector-857cdd9594-b9rdh   1/1     Running   0

Finally, to prove that our goal has been achieved, if you delete one of the pods the new pod starts up unsealed just like the others:

~/code/demos/vault > kubectl delete pod vault-0                                                                                                                                                                                 
pod "vault-0" deleted
~/code/demos/vault > kubectl get pods                                                                                                                                                                                       
NAME                                    READY   STATUS    RESTARTS   
vault-0                                 1/1     Running   0          
vault-1                                 1/1     Running   0          
vault-agent-injector-857cdd9594-b9rdh   1/1     Running   0

I hope this article made it easier to understand how the unsealer setup works for Hashicorp Vault and Azure. In a following article, I will show how to use Azure as storage backend instead of Raft and how to inject Vault secrets directly into Kubernetes pods via an ephemeral side-car container. Stay tuned!

Thanks to Felipe de Almeida who helped along the way.

(cover photo by Jason Pofahl (@jasonpofahlphotography) on Unsplash)

References:
https://learn.hashicorp.com/tutorials/vault/autounseal-azure-keyvault
https://github.com/hashicorp/vault-helm