Forem: Matthew Casperson

Private LLMs for GitHub Actions

Matthew Casperson — Mon, 23 Dec 2024 00:15:34 +0000

GitHub has been an enthusiastic adopter of AI with its Copilot platform to support developers with understanding, coding, and debugging software. However, it is not easy to use Copilot in an automated fashion in your GitHub Actions workflows.

SecondBrain is a new action that supports the use of LLMs inside GitHub Actions workflows. It works by deploying Ollama as a Docker container to host LLMs and then calling Ollama via a custom CLI that automates the process of constructing Retrieval Augmented Generation (RAG) prompts that embed details of git commits.

SecondBrain works like this:

You pass in the git commit SHAs you wish to query
You pass in GitHub token used to query the GitHub REST API to get the details of the commits
You define a prompt to pass to the LLM that can assume access to a summary of the Git commits referenced by the SHAs
SecondBrain queries GitHub for the details of the commits associated with the SHAs, summarizes the commit diffs, places the summaries into the prompt context, and then passes the context and your prompt to the LLM.

The following is a sample workflow YAML file that generates a summary of each commit to the main branch:

name: Summarize the commit

on:
  workflow_dispatch:
  push:
    branches:
      - main

jobs:
  summarize:
    runs-on: ubuntu-latest
    steps:
      - name: SecondBrainAction
        id: secondbrain
        uses: mcasperson/SecondBrain@main
        with:
            prompt: 'Provide a summary of the changes from the git diffs. Use plain language. You will be penalized for offering code suggestions. You will be penalized for sounding excited about the changes.'
            token: ${{ secrets.GITHUB_TOKEN }}
            owner: ${{ github.repository_owner }}
            repo: ${{ github.event.repository.name }}
            sha: ${{ github.sha }}
      - name: Get the diff summary
        env:
            RESPONSE: ${{ steps.secondbrain.outputs.response }}
        run: echo "$RESPONSE"

The output of the Get the diff summary step looks like this:

Here is a summary of the changes:

The README.md file now indicates that the sha input is required and cannot have a default value [2]. The action.yml file has also removed the option to provide a default value for the sha input [1].

- [1]: The action.yml file was also updated to remove the default value for the sha input (52b40e59684d17e5fddc95c4dba3cdc82e4f7b7d)
- [2]: The README.md file was updated to add a note that the sha input is mandatory and has no default value (52b40e59684d17e5fddc95c4dba3cdc82e4f7b7d)

Links:
- [52b40e59684d17e5fddc95c4dba3cdc82e4f7b7d](https://github.com/mcasperson/SecondBrain/commit/52b40e59684d17e5fddc95c4dba3cdc82e4f7b7d)

If you click the link from the report, you will see the commit that generated this summary. The description of the commit is accurate, and much easier to read than inspecting the diff directly.

It is important to note that this action never called any external services except for GitHub itself. There is no need to host your own LLM infrastructure as the entire process is handled by a local, private LLM exposed by Ollama.

Give it a try and let me know if this action was useful.

A first look at Tekton Pipelines

Matthew Casperson — Mon, 11 May 2020 23:08:59 +0000

Kubernetes is quickly evolving from a Docker orchestration platform to a general purpose cloud operating system. With operators Kubernetes gains the ability to natively manage high-level concepts and business processes, meaning you are no longer managing the building blocks of Pods, Services, and Deployments, but instead, describing the things those building blocks can create like web servers, databases, continuous deployments, certificate management, and more.

When deployed to a Kubernetes cluster, Tekton Pipelines expose the ability to define and execute build tasks, inputs and outputs in the form of simple values or complex objects like Docker images, and to combine these resources in pipelines. These new Kubernetes resources, and the controllers that manage them result in a headless CI/CD platform hosted by a Kubernetes cluster.

In this post, we’ll take a look at a simple build pipeline running on MicroK8S.

Preparing the test Kubernetes cluster

For this post, I’m using MicroK8S to provide the Kubernetes cluster. MicroK8S is useful here because it offers a selection of official add-ons, one of which is a Docker image registry. Since our pipeline builds a Docker image, we need somewhere to host it, and the MicroK8S registry add-on gives us that functionality with a single command:

microk8s.enable registry

We also need to enable DNS lookups from within the MicroK8S cluster. This is done by enabling the DNS add-on:

microk8s.enable dns

Installing Tekton Pipelines

Installation of Tekton Pipelines is simple, with a single kubectl (or microk8s.kubectl in our case) command:

microk8s.kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml

We can now create Tekton resources in our Kubernetes cluster.

A "Hello World" task

Tasks contain the individual steps that are executed to do work. In the example below, we have a task with a single step that executes the echo command with the arguments Hello World in a container built from the ubuntu image.

The YAML below shows our helloworldtask.yml file:

apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: echo-hello-world
spec:
  steps:
    - name: echo
      image: ubuntu
      command:
        - echo
      args:
        - "Hello World"

The task resource is created in the Kubernetes cluster with the command:

microk8s.kubectl apply -f helloworldtask.yml

A task describes how work is to be done, but creating the task resource does not result in any action being taken. A task run resource references the task, and the creation of a task run resource triggers Tekton to execute the steps in the referenced task.

The YAML below shows our helloworldtaskrun.yml file:

apiVersion: tekton.dev/v1alpha1
kind: TaskRun
metadata:
  name: echo-hello-world-task-run
spec:
  taskRef:
    name: echo-hello-world

The task run resource is created in the Kubernetes cluster with the command:

microk8s.kubectl apply -f helloworldtaskrun.yml

Building a Docker image

To move beyond this hello world example, we’ll look at the canonical use case of a Tekton build pipeline, which is to compile and push a Docker image. To demonstrate this functionality, we’ll build our RandomQuotes sample application.

We start the pipeline with a pipeline resource. Pipeline resources provide a decoupled method of defining inputs for the build process.

The first input we need is the Git repository that holds our code. Pipeline resources have a number of known types, and here we define a git pipeline resource specifying the URL and branch holding our code:

apiVersion: tekton.dev/v1alpha1
kind: PipelineResource
metadata:
  name: randomquotes-git
spec:
  type: git
  params:
    - name: revision
      value: master
    - name: url
      value: https://github.com/OctopusSamples/RandomQuotes-Java.git

Next, we define the Docker registry holding our compiled image. This is where the MicroK8S registry add-on is useful, as it exposes a Docker registry at http://registry.container-registry.svc.cluster.local:5000.

Here is the pipeline resource of type image defining the Docker image we’ll create as registry.container-registry.svc.cluster.local:5000/randomquotes:

apiVersion: tekton.dev/v1alpha1
kind: PipelineResource
metadata:
  name: randomquotes-image
spec:
  type: image
  params:
    - name: url
      value: registry.container-registry.svc.cluster.local:5000/randomquotes

With the input source code and destination Docker image defined, we can create a task to create the Docker image and push it to the repository.

Traditionally, building Docker images is performed by the Docker client directly on the host operating system. However, in Kubernetes, everything is run inside Docker, which leads to the question: How do you run Docker inside Docker?

Over the last few years, there has been an explosion of tools designed to perform the processes provided by the Docker CLI and daemon, but without any dependency on Docker itself. These include tools like umoci for unpacking and repacking Docker images, Kaniko and Buildah for building Docker images from a Dockerfile, and Podman for running Docker images.

We’ll use Kaniko in our Tekton task to build our Docker image inside the Docker container provided by Kubernetes. The YAML below shows the complete task:

apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: build-docker-image-from-git-source
spec:
  inputs:
    resources:
      - name: docker-source
        type: git
    params:
      - name: pathToDockerFile
        type: string
        description: The path to the dockerfile to build
        default: /workspace/docker-source/Dockerfile
      - name: pathToContext
        type: string
        description:
          The build context used by Kaniko
          (https://github.com/GoogleContainerTools/kaniko#kaniko-build-contexts)
        default: /workspace/docker-source
  outputs:
    resources:
      - name: builtImage
        type: image
  steps:
    - name: build-and-push
      image: gcr.io/kaniko-project/executor:v0.17.1
      # specifying DOCKER_CONFIG is required to allow kaniko to detect docker credential
      env:
        - name: "DOCKER_CONFIG"
          value: "/tekton/home/.docker/"
      command:
        - /kaniko/executor
      args:
        - --dockerfile=$(inputs.params.pathToDockerFile)
        - --destination=$(outputs.resources.builtImage.url)
        - --context=$(inputs.params.pathToContext)

There are some important aspects to this task that are worth pointing out.

There are two properties in this task that relate to the pipeline resources that we created above.

An input resource of type git:

inputs:
    resources:
      - name: docker-source
        type: git

And an output of type image:

outputs:
  resources:
    - name: builtImage
      type: image

There are two additional input parameters that define paths used for the Docker build process:

spec:
  inputs:
    params:
      - name: pathToDockerFile
        type: string
        description: The path to the dockerfile to build
        default: /workspace/docker-source/Dockerfile
      - name: pathToContext
        type: string
        description:
          The build context used by Kaniko
          (https://github.com/GoogleContainerTools/kaniko#kaniko-build-contexts)
        default: /workspace/docker-source

Note that the path /workspace/docker-source is a convention used by git resources, with the docker-source directory matching the name of the input.

We then have a single step that builds the Docker image. The build is executed in a container created from the gcr.io/kaniko-project/executor:v0.17.1 image, that provides Kaniko:

spec:
  steps:
    - name: build-and-push
      image: gcr.io/kaniko-project/executor:v0.17.1
      # specifying DOCKER_CONFIG is required to allow kaniko to detect docker credential
      env:
        - name: "DOCKER_CONFIG"
          value: "/tekton/home/.docker/"
      command:
        - /kaniko/executor
      args:
        - --dockerfile=$(inputs.params.pathToDockerFile)
        - --destination=$(outputs.resources.builtImage.url)
        - --context=$(inputs.params.pathToContext)

Finally, a task run is used to bind the task and pipeline resources together. This resource maps the task docker-source input to the randomquotes-git pipeline resource and the builtImage output to the randomquotes-image pipeline resource.

Creating this resource then triggers the build to take place:

apiVersion: tekton.dev/v1alpha1
kind: TaskRun
metadata:
  name: build-docker-image-from-git-source-task-run
spec:
  taskRef:
    name: build-docker-image-from-git-source
  inputs:
    resources:
      - name: docker-source
        resourceRef:
          name: randomquotes-git
    params:
      - name: pathToDockerFile
        value: Dockerfile
      - name: pathToContext
        value: /workspace/docker-source
  outputs:
    resources:
      - name: builtImage
        resourceRef:
          name: randomquotes-image

Interacting with builds

Tekton itself does not provide any kind of dashboard or GUI for interacting with jobs. However, there is a CLI tool for managing Tekton jobs.

The Tekton CLI tool assumes kubectl is configured, but MicroK8S maintains a separate tool called microk8s.kubectl. The easiest way to configure kubectl is with the following command which copies the MicroK8S configuration file to the standard location for kubectl:

sudo microk8s.kubectl config view --raw > $HOME/.kube/config

At this point, we can get the status of the task with the command:

tkn taskrun logs build-docker-image-from-git-source-task-run

Is Tekton for you?

The idea of a headless build server is an intriguing one.

By composing builds with Docker images, Tekton removes the overhead of maintaining a suite of specialized build agents. Every tool and language provides a supported Docker image these days, making it easier to keep up with the new normal of six month release cycles for major language versions.

Kubernetes is also a natural platform to serve the elastic and short-lived requirements of software builds. Why have ten specialized agents sitting idle when you can have five nodes scheduling builds between them?

However, I suspect Tekton itself is too low-level for most engineering teams. The tkn CLI tool will be familiar to anyone who has used kubectl before, but it is difficult to understand the overall state of your builds from the terminal. Not to mention creating builds with kubectl create -f taskrun.yml gets old quickly.

There is a dashboard available, but it is a bare-bones user interface compared to existing CI tools.

That said, Tekton is a powerful foundation on which to build developer facing tools. Jenkins X and OpenShift Pipelines are two such platforms that leverage Tekton under the hood.

Conclusion

Kubernetes solves many of the requirements for running applications like authentication, authorization, CLI tooling, resource management, health checks, and more. The fact that a Kubernetes cluster can host a fully functional CI server with a single command is a testament to just how flexible Kubernetes is.

With projects like Jenkins X and OpenShift Pipelines, Tekton is at the start of a journey into mainstream development workflows. But as a standalone project, Tekton is a little too close to the metal to be something most development teams could use, if only because few people would have the experience to support it.

The ultimate guide to Tomcat deployments

Matthew Casperson — Tue, 05 May 2020 20:21:56 +0000

Continuous integration and delivery (CI/CD) is a common goal for DevOps teams to reduce costs and increase the agility of software teams. But the CI/CD pipeline is far more than simply testing, compiling, and deploying applications. A robust CI/CD pipeline addresses a number of concerns such as:

High availability (HA)
Multiple environments
Zero downtime deployments
Database migrations
Load balancers
HTTPS and certificate management
Feature branch deployments
Smoke testing
Rollback strategies

How these goals are implemented depends on the type of software being deployed. In this post, I look at how to create the continuous delivery (or deployment) half of the CI/CD pipeline by deploying Java applications to Tomcat. I then build a supporting infrastructure stack that includes the Apache web server for load balancing, PostgreSQL for the database, Keepalived for highly available load balancers, and Octopus for orchestrating the deployments.

A note on the PostgreSQL server

This post assumes that the PostgreSQL database is already deployed in a highly available configuration. For more information on how to deploy PostgreSQL, refer to the documentation.

The instructions in this post can be followed with a single PostgreSQL instance, with the understanding that the database represents a single point of failure.

How to implement HA in Tomcat

When talking about HA, it is important to understand exactly which components of a platform need to be managed to address the unavailability of an individual Tomcat instance, for example, when an instance is unavailable due to routine maintenance or a hardware failure.

For the purpose of this post, I created infrastructure that allows a traditional stateful Java servlet application to continue operating when an individual Tomcat server is no longer available. In practical terms, this means the application session state will persist and be distributed to other Tomcat instances in the cluster when the server that originally hosted the session is no longer available.

As a brief recap, Java servlet applications can save data against a HttpSession instance, which is then available across requests. In the (naïve, as it does not deal with race conditions) example below, we have a simple counter variable that is incremented with each request to the page. This demonstrates how information can be persisted across individual requests made by a web browser:

@RequestMapping("/pageCount")
public String index(final HttpSession httpSession) {
    httpSession.setAttribute("pageCount", ObjectUtils.defaultIfNull((Integer)httpSession.getAttribute("pageCount"), 0) + 1);
    return (Integer)httpSession.getAttribute("pageCount");
}

The session state is held in memory on an individual server. By default, if that server is no longer available, the session data is also lost. For a trivial example like a page count, this is not important, but it is not uncommon for more critical functionality to rely on the session state. For example, a shopping cart may hold the list of items for purchase in session state, and losing that information may result in a lost sale.

To maintain high availability, the session state needs to be duplicated so it can be shared if a server goes offline.

Tomcat offers three solutions to enable session replication:

Using session persistence and saving the session to a shared file system (PersistenceManager + FileStore).
Using session persistence and saving the session to a shared database (PersistenceManager + JDBCStore).
Using in-memory-replication, and using the SimpleTcpCluster that ships with Tomcat (lib/catalina-tribes.jar + lib/catalina-ha.jar).

Because our infrastructure stack already assumes a highly available database, I’ll implement option two. This is arguably the simplest solution for us, as we do not have to implement any special networking, and we can reuse an existing database. However, this solution does introduce a delay between when the session state is modified and when it is persisted to the database. This delay introduces a window during which data may be lost in the case of hardware or network failure. Scheduled maintenance tasks are supported though, as any session data will be written to the database when Tomcat is shutdown, allowing us to patch the operating system or update Tomcat itself safely.

We noted the example code above is naïve as it doesn’t deal with the fact the session cache is not thread safe. Even this simple example is subject to race conditions that may result in the page count being incorrect. The solution to this problem is to use the traditional thread locks and synchronization features available in Java, but these features are only valid within a single JVM. This means we must ensure that client requests are always directed to a single Tomcat instance, which in turn means that only one Tomcat instance contains the single, authoritative copy of the session state, which can then ensure consistency via thread locks and synchronization. This is achieved with sticky sessions.

Sticky sessions provide a way for client requests to be inspected by a load balancer and then directed to one web server in a cluster. By default, in a Java servlet application a client is identified by a JSESSIONID cookie that is sent by the web browser and inspected by the load balancer to identify the Tomcat instance that holds the session, and then by the Tomcat server to associate a request with an existing session.

In summary, our HA Tomcat solution:

Persists session state to a shared database.
Relies on sticky sessions to direct client requests to a single Tomcat instance.
Supports routine maintenance by persisting session state when Tomcat is shutdown.
Has a small window during which a hardware or network failure may result in lost data.

Keepalived for HA load balancers

To ensure that network requests are distributed among multiple Tomcat instances and not directed to an offline instance, we need to implement a load balancing solution. These load balancers sit in front of the Tomcat instances and direct network requests to those instances that are available.

Many load balancing solutions exist that can perform this role, but for this post, we use the Apache web server with the mod_jk plugin. Apache will provide the networking functionality, while mod_jk will distribute traffic to multiple Tomcat instances, implementing sticky sessions to direct a client to the same backend server for each request.

In order to maintain high availability, we need at least two load balancers. But how do we split a single incoming network connection across two load balancers in a reliable manner? This is where Keepalived comes in.

Keepalived is a Linux service run across multiple instances and picks a single master instance from the pool of healthy instances. Keepalived is quite flexible when it comes to determining what that master instance does, but in our scenario, we will use Keepalived to assign a virtual, floating IP address to the instance that assumes the master role. This means our incoming network traffic will be sent to a floating IP address that is assigned to a healthy load balancer, and the load balancer then forwards the traffic to the Tomcat instances. Should one of the load balancers be taken offline, Keepalived will ensure the remaining load balancer is assigned the floating IP address.

In summary, our HA load balancing solution:

Implements Apache with the mod_jk plugin to direct traffic to the Tomcat instances.
Implements Keepalived to ensure one load balancer has a virtual IP address assigned to it.

The network diagram

Here is the diagram of the network that we will create:

Zero downtime deployments and rollbacks

A goal of continuous delivery is to always be in a state where you can deploy (even if you choose not to). This means moving away from schedules that require people to be awake at midnight to perform a deployment when your customers are asleep.

Zero downtime deployments require reaching a point where deployments can be done at any time without disruption. In our example infrastructure, there are two points that we need to consider in order to achieve zero downtime deployments:

The ability for customers to use the existing version of the application to complete their session even after a newer version of the application has been deployed.
Forward and backward compatibility of any database changes.

Ensuring database changes are backward and forward compatible requires some design work and discipline when pushing new application versions. Fortunately, there are tools available, including Flyway and Liquidbase, that provide a way to roll out database changes with the applications themselves, taking care of versioning the changes and wrapping any migrations in the required transactions. We’ll see Flyway implemented in a sample application later in the post.

As long as the shared database remains compatible between the current and the new version of the application, Tomcat provides a feature called parallel deployments that allows clients to continue to access the previous version of the application until their session expires, while new sessions are created against the new version of the application. Parallel deployments allow a new version of the application to be deployed without disrupting any existing clients.

Tomcat has the ability to automatically clean up old versions that no longer have any sessions. We will not enable this feature though, as it may prevent a session for an old version from being migrated to another Tomcat instance.

Ensuring database changes are compatible between the current and new version of the application means we can easily roll back the application deployment. Redeploying the previous version of the application provides a quick fallback in case the new version introduced any errors.

In summary, our zero downtime deployments solution:

Relies on database changes being forward and backward compatible (at least between the new and current versions of the application).
Uses parallel deployments to allow existing sessions to complete uninterrupted.
Provides application rollbacks by reverting to the previously installed application version.

Build the infrastructure

The example infrastructure shown here is deployed to Ubuntu 18.04 virtual machines. Most of the instructions will be distribution agnostic, although some of the package names and file locations may change.

Configure the Tomcat instances

Install the packages

We start by installing Tomcat and the Manager application:

apt-get install tomcat tomcat-admin

Add the AJP connector

Communication between the Apache web server and Tomcat is performed with an AJP connector. AJP is an optimized binary HTTP protocol that the mod_jk plugin for Apache and Tomcat both understand. The connector is added to the Service element in the /etc/tomcat9/server.xml file:

<Server>
  <!-- ... -->
  <Service name="Catalina">
    <!-- ... -->
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443"></Connector>
  </Service>
</Server>

Define the Tomcat instance names

Each Tomcat instance needs a unique name added to the Engine element in the /etc/tomcat9/server.xml file. The default Engine element looks like this:

<Engine name="Catalina" defaultHost="localhost">

The name of the Tomcat instance is defined in the jvmRoute attribute. I’ll call the first Tomcat instance worker1:

<Engine defaultHost="localhost" name="Catalina" jvmRoute="worker1">

The second Tomcat instance is called worker2:

<Engine defaultHost="localhost" name="Catalina" jvmRoute="worker2">

Add a manager user

Octopus performs deployments to Tomcat via the Manager application. This is what we installed with the tomcat-admin package earlier.

In order to authenticate with the Manager application, a new user needs to be defined in the /etc/tomcat9/tomcat-users.xml file. I’ll call this user tomcat with the password Password01!, and it will belong to the manager-script and manager-gui roles.

The manager-script role grants access to the Manager API, and the manager-gui role grants access to the Manager web console.

Here is a copy of the /etc/tomcat9/tomcat-users.xml file with the tomcat user defined:

<tomcat-users xmlns="http://tomcat.apache.org/xml"
              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
              xsi:schemaLocation="http://tomcat.apache.org/xml tomcat-users.xsd"
              version="1.0">
  <role rolename="manager-gui"/>
  <role rolename="manager-script"/>
  <user username="tomcat" password="Password01!" roles="manager-script,manager-gui"/>
</tomcat-users>

Add the PostgreSQL JDBC driver jar

Each Tomcat instance will communicate with a PostgreSQL database to persist session data. In order for Tomcat to communicate with a PostgreSQL database, we need to install the PostgreSQL JDBC driver JAR file. This is done by saving the file https://jdbc.postgresql.org/download/postgresql-42.2.11.jar as /var/lib/tomcat9/lib/postgresql-42.2.11.jar.

Enable session replication

To enable session persistence to a database, we add a new Manager definition in the file /etc/tomcat9/context.xml. This manager uses the org.apache.catalina.session.PersistentManager to save the session details to a database defined in the nested Store element.

The Store element in turn, defines the database that the session information will be persisted to.

We also need to add a Valve loading the org.apache.catalina.ha.session.JvmRouteBinderValve class. This valve is important when a client is redirected from a Tomcat instance that is no longer available to another Tomcat instance in the cluster. We’ll see this valve in action after we deploy our sample application.

Here is a copy of the /etc/tomcat9/context.xml file with the Manager, Store, and Valve elements defined:

<Context>
  <Manager
    className="org.apache.catalina.session.PersistentManager"
    processExpiresFrequency="3"
    maxIdleBackup="1" >
      <Store
        className="org.apache.catalina.session.JDBCStore"
        driverName="org.postgresql.Driver"
        connectionURL="jdbc:postgresql://postgresserver:5432/tomcat?currentSchema=session"
        connectionName="postgres"
        connectionPassword="passwordgoeshere"
        sessionAppCol="app_name"
        sessionDataCol="session_data"
        sessionIdCol="session_id"
        sessionLastAccessedCol="last_access"
        sessionMaxInactiveCol="max_inactive"
        sessionTable="session.tomcat_sessions"
        sessionValidCol="valid_session" />
  </Manager>
  <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
  <WatchedResource>WEB-INF/web.xml</WatchedResource>
  <WatchedResource>WEB-INF/tomcat-web.xml</WatchedResource>
  <WatchedResource>${catalina.base}/conf/web.xml</WatchedResource>
</Context>

Configure the PostgreSQL database

Install the packages

We need to initialize PostgreSQL with a new database, schema, and table. To do this, we use the psql command-line tool, which is installed with:

apt-get install postgresql-client

Add the database, schema, and table

If you look at the connectionURL attribute from the /etc/tomcat9/context.xml file defined above, you will see we are saving session information into:

The database called tomcat.
The schema called session.
A table called tomcat_sessions.

To create these resources in the PostgreSQL server, we run a number of SQL commands.

First, save the following text into a file called createdb.sql. This command creates the database if it does not exist (see this StackOverflow post for more information about the syntax):

SELECT 'CREATE DATABASE tomcat' WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'tomcat')\gexec

Then execute the SQL with the following command, replacing postgresserver with the hostname of your PostgreSQL server:

cat createdb.sql | /usr/bin/psql -a -U postgres -h postgresserver

Next, we create the schema and table. Save the following text to a file called createschema.sql. Note the columns of the tomcat_sessions table match the attributes of the Store element in the /etc/tomcat9/context.xml file:

CREATE SCHEMA IF NOT EXISTS session;

CREATE TABLE IF NOT EXISTS session.tomcat_sessions
(
  session_id character varying(100) NOT NULL,
  valid_session character(1) NOT NULL,
  max_inactive integer NOT NULL,
  last_access bigint NOT NULL,
  app_name character varying(255),
  session_data bytea,
  CONSTRAINT tomcat_sessions_pkey PRIMARY KEY (session_id)
);

CREATE INDEX IF NOT EXISTS app_name_index
  ON session.tomcat_sessions
  USING btree
  (app_name);

Then execute the SQL with the following command, replacing postgresserver with the hostname of your PostgreSQL server:

psql -a -d tomcat -U postgres -h postgresserver -f /root/createschema.sql

We now have a table in PostgreSQL ready to save the Tomcat sessions.

Configure the load balancers

Install the packages

We need to install the Apache web server, the mod_jk plugin, and the Keepalived service:

apt-get install apache2 libapache2-mod-jk keepalived

Configure the load balancer

The mod_jk plugin is configured via the file /etc/libapache2-mod-jk/workers.properties. In this file, we define a number of workers that traffic can be directed to. The fields in this file are documented here.

We start by defining a worker called loadbalancer that will receive all of the traffic:

worker.list=loadbalancer

We then define the two Tomcat instances that were created earlier. Make sure to replace worker1_ip and worker2_ip with the IP addresses of the matching Tomcat instances.

Note, the name of the workers defined here as worker1 and worker2 match the value of the jvmRoute attribute in the Engine element in the /etc/tomcat9/server.xml file. These names must match, as they are used by mod_jk to implement sticky sessions:

worker.worker1.type=ajp13
worker.worker1.host=worker1_ip
worker.worker1.port=8009

worker.worker2.type=ajp13
worker.worker2.host=worker2_ip
worker.worker2.port=8009

Finally, we define the loadbalancer worker as a load balancer that directs traffic to the worker1 and worker2 workers with sticky sessions enabled:

worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=worker1,worker2
worker.loadbalancer.sticky_session=1

Here is a complete copy of the /etc/libapache2-mod-jk/workers.properties file:

# All traffic is directed to the load balancer
worker.list=loadbalancer

# Set properties for workers (ajp13)
worker.worker1.type=ajp13
worker.worker1.host=worker1_ip
worker.worker1.port=8009

worker.worker2.type=ajp13
worker.worker2.host=worker2_ip
worker.worker2.port=8009

# Load-balancing behaviour
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=worker1,worker2
worker.loadbalancer.sticky_session=1

Add an Apache VirtualHost

In order for Apache to accept traffic we need to define a VirtualHost, which we create in the file /etc/apache2/sites-enabled/000-default.conf. This virtual host will accept HTTP traffic on port 80, define some log files, and use the JkMount directive to forward traffic to the worker called loadbalancer:

<VirtualHost *:80>
  ErrorLog ${APACHE_LOG_DIR}/error.log
  CustomLog ${APACHE_LOG_DIR}/access.log combined
  JkMount /* loadbalancer
</VirtualHost>

Configure Keepalived

We have two load balancers to ensure that one can be taken offline for maintenance at any given time. Keepalived is the service that we use to assign a virtual IP address to one of the load balancer services, which Keepalived refers to as the master.

Keepalived is configured via the /etc/keepalived/keepalived.conf file.

We start by naming the load balancer instance. The first load balancer is called loadbalancer1:

vrrp_instance loadbalancer1 {

The state MASTER designates the active server:

state MASTER

The interface parameter assigns the physical interface name to this particular virtual IP instance:

You can find the interface name by running ifconfig.

interface ens5

virtual_router_id is a numerical identifier for the Virtual Router instance. It must be the same on all LVS Router systems participating in this Virtual Router. It is used to differentiate multiple instances of Keepalived running on the same network interface:

virtual_router_id 101

The priority specifies the order in which the assigned interface takes over in a failover; the higher the number, the higher the priority. This priority value must be within the range of 0 to 255, and the load balancing server configured as state MASTER should have a priority value set to a higher number than the priority value of the server configured as state BACKUP:

priority 101

advert_int defines how often to send out VRRP advertisements:

advert_int 1

The authentication block specifies the authentication type (auth_type) and password (auth_pass) used to authenticate servers for failover synchronization. PASS specifies password authentication:

authentication {
    auth_type PASS
    auth_pass passwordgoeshere
}

unicast_src_ip is the IP address of this load balancer:

unicast_src_ip 10.0.0.20

unicast_peer lists the IP addresses of other load balancers. Since we have two load balancers total, there is only one other load balancer to list here:

unicast_peer {
  10.0.0.21
}

virtual_ipaddress defines the virtual, or floating, IP address that Keepalived assigns to the master node:

virtual_ipaddress {
    10.0.0.30
}

Here is a complete copy of the /etc/keepalived/keepalived.conf file for the first load balancer:

vrrp_instance loadbalancer1 {
    state MASTER
    interface ens5
    virtual_router_id 101
    priority 101
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass passwordgoeshere
    }
    # Replace unicast_src_ip and unicast_peer with your load balancer IP addresses
    unicast_src_ip 10.0.0.20
    unicast_peer {
      10.0.0.21
    }
    virtual_ipaddress {
        10.0.0.30
    }
}

Here is a complete copy of the /etc/keepalived/keepalived.conf file for the second load balancer.

Note, the name has been set to loadbalancer2, the state has been set to BACKUP, the priority is lower at 100, and the unicast_src_ip and unicast_peer IP addresses have been flipped:

vrrp_instance loadbalancer2 {
    state BACKUP
    interface ens5
    virtual_router_id 101
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass passwordgoeshere
    }
    # Replace unicast_src_ip and unicast_peer with your load balancer IP addresses
    unicast_src_ip 10.0.0.21
    unicast_peer {
      10.0.0.20
    }
    virtual_ipaddress {
        10.0.0.30
    }
}

Restart the keepalived service on both load balancers with the command:

systemctl restart keepalived

On the first load balancer, run the command ip addr. This will show the virtual IP address assigned to the interface that Keepalived was configured to manage with the output inet 10.0.0.30/32 scope global ens5:

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0e:2b:f9:2a:fa:a7 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.21/24 brd 10.0.0.255 scope global dynamic ens5
       valid_lft 3238sec preferred_lft 3238sec
    inet 10.0.0.30/32 scope global ens5
       valid_lft forever preferred_lft forever
    inet6 fe80::c2b:f9ff:fe2a:faa7/64 scope link
       valid_lft forever preferred_lft forever

If the first load balancer is shutdown, the second load balancer will assume the virtual IP address, and the second Apache web server will act as the load balancer.

Build the deployment pipeline

Our deployment pipeline will involve deploying the Random Quotes sample application. This is a simple stateful Spring Boot application utilizing Flyway to manage database migrations.

When you click the Refresh button, a new quote is loaded from the database, a counter is incremented in the session, and the counter is displayed as the Quote count field on the page. The application version is shown in the Version field.

We can use the Quote count and Version information to verify that existing sessions are preserved as new deployments are performed or Tomcat instances are taken offline.

Get an Octopus instance

If you do not already have Octopus installed, the easiest way to get an Octopus instance is to sign up for a cloud account. These instances are free for up to 10 targets.

Create the environments

We’ll create two environments for this example: Dev and Prod. This means we will configure eight targets in total: four load balancers and four Tomcat instances.

Deploy the Tentacle

We will install a Tentacle on each of our virtual machines to allow us to perform deployments, updates, and system maintenance tasks. The instructions for installing the Tentacle software is found on the Octopus download page. As this example is using Ubuntu as the base OS, we install the Tentacle with the commands:

apt-key adv --fetch-keys https://apt.octopus.com/public.key
add-apt-repository "deb https://apt.octopus.com/ stretch main"
apt-get update
apt-get install tentacle

After the Tentacle is installed we configure an instance with the command:

/opt/octopus/tentacle/configure-tentacle.sh

The installation gives you a choice between polling or listening Tentacles. Which option you choose often depends on your network restrictions. Polling Tentacles require the VM hosting the Tentacle can reach the Octopus Server, while listening Tentacles require the Octopus Server can reach the VM. The choice of communication style depends on whether the Octopus Server or VMs have fixed IP addresses and the correct ports are opened in the firewall. Either option is a valid choice and does not impact the deployment process.

Here is a screenshot of the Dev environment with Tentacles for the Tomcat and Load Balancer instances. The Tomcat instances have a role of tomcat, and the load balancer instances have a role of loadbalancer:

Create the external feed

The Random Quotes sample application has been pushed to Maven Central as a WAR file. This means we can deploy the application directly from a Maven feed.

Create a new Maven feed pointing to https://repo.maven.apache.org/maven2/. A screenshot of this feed is shown below:

Test the feed by searching for com.octopus:randomquotes. Here we can see that our application is found in the repository:

Create the deployment process

Generate a timestamp

In order to support zero downtime deployments, we want to take advantage of the parallel deployments feature in Tomcat. Parallel deployments are enabled by versioning each application when it is deployed.

This version number uses string comparisons to determine the latest version. Typical versioning schemes, like SemVer, use a major.minor.patch format, like 1.23.4, to identify versions. For a lot of cases, these traditional versioning schemes can be compared as strings to determine their order.

However, padding can introduce issues. For example, the version 1.23.40 is lower than 01.23.41, but a direct string comparison returns the opposite result.

For this reason, we use the time of the deployment as the Tomcat version. Because the version needs to be consistent across targets, we start with a script step that generates a timestamp and saves it to an output variable with the code:

$timestamp = Get-Date -Format "yyMMddHHmmss"
Set-OctopusVariable -name "TimeStamp" -value $timestamp

The Tomcat deployment

Our deployment process starts with deploying the application to each Tomcat instance using the Deploy to Tomcat via Manager step. We’ll call this step Random Quotes and run it on the tomcat targets:

We deploy the com.octopus:randomquotes package from the Maven feed we setup earlier:

Because the Tentacle is located on the VM that hosts Tomcat, the location of the Manager API is http://localhost:8080/manager. We then supply the manager credentials, which were the details entered into the tomcat-users.xml file when we configured Tomcat:

The context path makes up the path in the URL that the deployed application is accessible on. Here we expose the application on the path /randomquotes:

The deployment version is set to the timestamp that was generated by the previous step by referencing the output variable as #{Octopus.Action[Get Timestamp].Output.TimeStamp}:

Smoke test the deployment

To verify that our deployment was successful, we issue an HTTP request and check the response code. To do this, we use a community step template called HTTP - Test URL (Bash).

As before, this step will run on the Tomcat instances:

The step will attempt to open the index.html page from the newly deployed application, expecting an HTTP response code of 200:

Perform the initial deployment

Let’s go ahead and perform the initial deployment. For this deployment, we’ll specifically select a previous version of the Random Quotes application. Version 0.1.6.1 in this case, is our second last artifact version:

Octopus then proceeds to download the WAR file from the Maven repository, push it to the Tomcat instances and deploy it to Tomcat via the Manager. When complete, the smoke test runs to ensure the application can be opened successfully:

Inspect a request through the stack

With the deployment done, we can access it via the load balancer.

In the previous configuration examples, we had a floating IP address of 10.0.0.30, but for these screenshots, I used Keepalived to assign a public IP address to the load balancers.

Here is a screenshot of the application opened with the Chrome developer tools:

There are three things to note in this screenshot:

The JSESSIONID cookie is set with a random session ID and the name of the Tomcat instance that responded to the request. In this example, the Tomcat instance whose jvmRoute was set to worker1 responded to the request.
We have opened version 0.1.6.1 of the application.
The Quote count is set to 1, but this will increase as we hit the Refresh button.

Let’s increase the Quote count by clicking the Refresh button. This value is saved on the Tomcat server in the session associated with the JSESSIONID cookie:

Let’s now shut down the worker1 Tomcat instance. After shutdown, we click the Refresh button again:

There are three things to note in this screenshot:

The suffix on the JSESSIONID cookie changed from worker1 to worker2.
The JSESSIONID cookie session ID remained the same.
The Quote count increased to 6.

When Tomcat is gracefully shut down, it will write out the contents of any sessions to the database. Then because the worker1 Tomcat instance was no longer available, the request was directed to the worker2 Tomcat instance. The worker2 Tomcat instance loaded the session information from the database and incremented the counter. The JvmRouteBinderValve valve rewrote the session cookie to append the current Tomcat instance name to the end, and the response was returned to the browser.

We can now see that it is important that the worker names in the load balancer /etc/libapache2-mod-jk/workers.properties file match the names assigned to the jvmRoute in the /etc/tomcat9/server.xml file because matching these names allow sticky sessions to be implemented.

Because the Quote count did not reset back to 1, we know that the session was persisted to the database and replicated to the other Tomcat instances in the cluster. We also know that the request was served by another Tomcat instance because the JSESSIONID cookie shows a new worker name.

Even if we brought worker1 back online, this browser session would continue to be handled by worker2 because the load balancers implement sticky sessions by inspecting the JSESSIONID cookie. This also means that the load balancers don’t need to share state, as they only require the cookie value to direct traffic.

We’ve now demonstrated that the Tomcat instances support session replication and failover, making them highly available.

To demonstrate failover of the load balancers, we only need to restart the instance designated as the master by Keepalived. We can then watch the events on the backup instance with the command:

journalctl -u keepalived -f

Very quickly we’ll see these messages appear on the backup as it assumes the master role:

Apr 01 03:15:00 ip-10-0-0-21 Keepalived_vrrp[32485]: VRRP_Instance(loadbalancer2) Transition to MASTER STATE
Apr 01 03:15:01 ip-10-0-0-21 Keepalived_vrrp[32485]: VRRP_Instance(loadbalancer2) Entering MASTER STATE

Having assumed the master role, the load balancer will be assigned the virtual IP address, and distribute traffic as the previous master did.

After the previous master instance restarts, it will re-assume the master role because it is configured with a higher priority, and the virtual IP address will be assigned back.

The whole process is seamless, and upstream clients never need to be aware that a failover and failback took place. So we have demonstrated that the load balancers can failover, making them highly available.

In summary:

The JSESSIONID cookie contains the session ID and the name of the Tomcat instance that processed the request.
The load balancers implement sticky sessions based on the worker name appended to the JSESSIONID cookie.
The JvmRouteBinderValve valve rewrites the JSESSIONID cookie when a Tomcat instance received traffic for a session it was not originally responsible for.
Keepalived assigns a virtual IP to the backup load balancer if the master goes offline.
The master load balancer re-assumes the virtual IP when it comes back online.
The infrastructure stack can survive the loss of one Tomcat instance and one load balancer and still maintain availability.

Zero downtime deployments

We have now successfully deployed version 0.1.6.1 of our web application to Tomcat. This version of the application uses a very simple table structure to hold the names of those credited with the quotes, placing both the first name and last name into a single column called AUTHOR.

This table structure was originally created by a Flyway database script with the following SQL:

create table AUTHOR
(
    ID INT auto_increment,
    AUTHOR VARCHAR(64) not null
);

The next version of our application will split the names into a FIRSTNAME and LASTNAME column. We add these columns with a new Flyway script that contains the following SQL:

ALTER TABLE AUTHOR
ADD FIRSTNAME varchar(64);

ALTER TABLE AUTHOR
ADD LASTNAME varchar(64);

At this point, we have to consider how these changes can be made in a backward compatible way. A cornerstone of the zero downtime deployment strategy requires that a shared database support both the current version of the application and the new version. Unfortunately, there is no silver bullet to provide this compatibility, and it is up to us as developers to ensure that our changes don’t break any existing sessions.

In this scenario, we must keep the old AUTHOR column and duplicate the data it held into the new FIRSTNAME and LASTNAME columns:

UPDATE AUTHOR SET FIRSTNAME = 'Rob', LASTNAME = 'Siltanen' WHERE ID = 1;
UPDATE AUTHOR SET FIRSTNAME = 'Albert', LASTNAME = 'Einstein' WHERE ID = 2;
UPDATE AUTHOR SET FIRSTNAME = 'Charles', LASTNAME = 'Eames' WHERE ID = 3;
UPDATE AUTHOR SET FIRSTNAME = 'Henry', LASTNAME = 'Ford' WHERE ID = 4;
UPDATE AUTHOR SET FIRSTNAME = 'Antoine', LASTNAME = 'de Saint-Exupery' WHERE ID = 5;
UPDATE AUTHOR SET FIRSTNAME = 'Salvador', LASTNAME = 'Dali' WHERE ID = 6;
UPDATE AUTHOR SET FIRSTNAME = 'M.C.', LASTNAME = 'Escher' WHERE ID = 7;
UPDATE AUTHOR SET FIRSTNAME = 'Paul', LASTNAME = 'Rand' WHERE ID = 8;
UPDATE AUTHOR SET FIRSTNAME = 'Elon', LASTNAME = 'Musk' WHERE ID = 9;
UPDATE AUTHOR SET FIRSTNAME = 'Jessica', LASTNAME = 'Hische' WHERE ID = 10;
UPDATE AUTHOR SET FIRSTNAME = 'Paul', LASTNAME = 'Rand' WHERE ID = 11;
UPDATE AUTHOR SET FIRSTNAME = 'Mark', LASTNAME = 'Weiser' WHERE ID = 12;
UPDATE AUTHOR SET FIRSTNAME = 'Pablo', LASTNAME = 'Picasso' WHERE ID = 13;
UPDATE AUTHOR SET FIRSTNAME = 'Charles', LASTNAME = 'Mingus' WHERE ID = 14;

In addition, the new JPA entity class needs to ignore the old AUTHOR column (through the @Transient annotation). The getAuthor() method then returns the combined values of the getFirstName() and getLastName() methods:

@Entity
public class Author {
    @Id
    @GeneratedValue(strategy= GenerationType.AUTO)
    private Integer id;
    @Column(name = "FIRSTNAME")
    private String firstName;
    @Column(name = "LASTNAME")
    private String lastName;

    @OneToMany(
            mappedBy = "author",
            cascade = CascadeType.ALL,
            orphanRemoval = true
    )
    private List<Quote> quotes = new ArrayList<>();

    protected Author() {

    }

    public Integer getId() {
        return id;
    }

    @Transient
    public String getAuthor() {
        return getFirstName() + " " + getLastName();
    }

    public List<Quote> getQuotes() {
        return quotes;
    }

    public String getFirstName() {
        return firstName;
    }

    public String getLastName() {
        return lastName;
    }
}

While this is a trivial example that is easy to implement thanks to the fact the AUTHOR table is read-only, it demonstrates how database changes can be implemented in a backward compatible manner. It would be possible to write entire books on the strategies for maintaining backward compatibility, but for the purposes of this post, we’ll leave this discussion here.

Before we perform the next deployment, reopen the existing application and refresh some quotes. This creates a session against the existing 0.1.6.1 version, which we’ll use to test our zero downtime deployment strategy.

With our migration scripts written in a backward compatible manner, we can deploy the new version of our application. For convenience this new version has been pushed to Maven Central as version 0.1.7:

After the deployment completes, open the Manager application at http://tomcatip:8080/manager/html. Note, that while you could access the Manager through the load balancer, you do not get to choose which Tomcat instance you will be managing as the load balancer picks a Tomcat instance for you. This means it is better to connect to the Tomcat instance directly, bypassing the load balancer:

There are two things to notice in this screenshot:

We have two applications under the path /randomquotes, each with a unique version.
The application with the earlier version has a session associated with it. This was the session we created by accessing version 0.1.6.1 before version 0.1.7 was deployed.

If we go back to the browser where we opened version 0.1.6.1 of the application, we can continue to refresh the quotes. The counter increases as we expect, and the version number displayed in the footer remains at version 0.1.6.1.

If we reopen the application in a private browsing window, we can be guaranteed that we won’t reuse an old session cookie, and we are directed to version 0.1.7 of the application:

With that, we have demonstrated zero downtime deployments. Because our database migrations were designed to be backward compatible, version 0.1.6.1 and version 0.1.7 of our application can run side by side using Tomcat parallel deployments. Best of all, sessions for old deployments can still be transferred between Tomcat instances, so we retain high availability along with the parallel deployments.

Rollback strategies

As long as database compatibility has been maintained between the last and current version of the application (versions 0.1.6.1 and 0.1.7 in this example), rolling back is as simple as creating a new deployment with the previous version of the application.

Because the Tomcat version has a timestamp calculated at deployment time, deploying version 0.1.6.1 of the application again results in it processing any new traffic as it has a later version.

Note, any existing sessions for version 0.1.7, will be left to naturally expire thanks to the Tomcat’s parallel deployments. If this version has to be taken offline (for example, if there is a critical issue and it can not be left in service), we can use the Start/stop App in Tomcat step to stop a deployed application.

We’ll create a runbook to run this step, as it is a maintenance task that may need to be applied to any environment to pull a bad release.

We start by adding a prompted variable that will be populated with the Tomcat version timestamp corresponding to the deployment we want to shutdown:

The runbook is then configured with the Start/stop App in Tomcat step. The Deployment version is set to the value of the prompted variable:

When the runbook is run, we are prompted for the timestamp version of the application to stop:

After the runbook has completed, we can verify the application was stopped by opening the Manager console. In the screenshot below, you can see version 200401140129 is not running. This version no longer responds to requests, and all future requests will then be directed to the latest version of the application:

Feature branch deployments

A common development practice is completing a feature in a separate SCM branch, known as a feature branch.

A CI server will typically watch for the presence of feature branches and create a deployable artifact from the code committed there.

These feature branch artifacts are then versioned to indicate which branch they were built from. GitVersion is a popular tool for generating versions to match commits and branches in Git, and they offer this example showing versions created as part of their GitHub flow:

As you can see from the image above, a commit to a feature branch called myfeature generates a version like 1.2.1-myfeature.1+1. This in turn, produces an artifact with a filename like myapp.1.2.1-myfeature.1+1.zip.

Despite the fact that tools like GitVersion generate SemVer version strings, the same format can be used for Maven artifacts. However, there is a catch.

SemVer will order a version with a feature branch lower than a version without any pre-release component. For example, 1.2.1 is considered a higher version number than 1.2.1-myfeature. This is the expected ordering as a feature branch will eventually be merged back into the master branch.

When a feature branch is appended to a Maven version, it is considered a qualifier. Maven allows any qualifiers but recognizes some special ones like SNAPSHOT, final, ga, etc. A complete list can be found in the blog post Maven versions explained. Maven versions with unrecognized qualifiers (and feature branch names are unrecognized qualifiers) are treated as later releases than unqualified versions.

This means Maven considers version 1.2.1-myfeature is to be a later release than 1.2.1, when clearly that is not the intention of a feature branch. You can verify this behavior with the following test in a project hosted on GitHub.

However, thanks to the functionality of channels in Octopus, we can ensure that both SemVer pre-release and Maven qualifiers are filtered in the expected way.

Here is the default channel for our application deployment. Note, the regular expression ^$ for the Pre-release tag field. This regular expression only matches empty strings, meaning the default channel will only ever deploy artifacts with no pre-release or qualifier string:

Next, we have the feature branch channel, which defines a regular expression of .+ for the Pre-release tag field. This regular expression only matches non-empty strings, meaning the feature branch channel will only deploy artifacts with a pre-release or qualifier string:

Here is the list of versions that Octopus allows a release to be created from in the default channel. Notice, the only version displayed has no qualifier, which we take to mean it is the master release:

Here is the list of versions that Octopus allows a release to be created from using the feature branch channel. All of these versions have a qualifier embedding the feature branch name:

The end result of these channels is that versions like 1.2.1-myfeature will never be compared to versions like 1.2.1, which removes the ambiguity with feature branch version numbers being considered later releases.

The final step is to deploy these feature branch packages to unique contexts so they can be accessed side by side on a single Tomcat instance. To do this we modify the Context path to:

/randomquotes#{Octopus.Action.Package.PackageVersion | Replace "^([0-9\.]+)((?:-[A-Za-z0-9]+)?)(.*)$" "$2"}

Using the regular expression above on the version 1.2.1-myfeature.1+1 will do the following:

^([0-9\.]+) groups all digits and periods at the start of the version as group 1, which matches 1.2.1.
((?:-[A-Za-z0-9]+)?) groups the leading dash and any subsequent alphanumeric characters (if any) as group 2, which matches -myfeature.
(.*)$ groups any subsequent characters (if any) as group 3, which matches .1+1.

This variable filter will result in the complete pre-release or qualifier strings being replaced by just the second group from the regular expression. This results in the following context path:

Version 1.2.1-myfeature.1+1 generates a context path of /randomquotes-myfeature.
Version 1.2.1 generates a context path of /randomquotes.

Here is a screenshot of the Tomcat deployment step with the new context path applied:

The SemVer project provides a more robust regular expression that reliably captures the groups in a SemVer version.

The regular expression with named capture groups is:

^(?P<major>0|[1-9]\d*)\.(?P<minor>0|[1-9]\d*)\.(?P<patch>0|[1-9]\d*)(?:-(?P<prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+(?P<buildmetadata>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$

The regular expression without named groups is:

^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:-((?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+([0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$

Public certificate management

To finish configuring our infrastructure, we will enable HTTPS access via our load balancers. We can edit the Apache web server virtual host configuration to enable SSL and point to the keys and certificates we have obtained for our domain.

Obtain a certificate

For this post, I have obtained a Let’s Encrypt certificate, generated through our DNS provider. The exact process for generating HTTPS certificates is not something we’ll look at here, but you can refer to your DNS or certificate provider for specific instructions.

In the screenshot below, you can see the various options available for downloading the certificate. Note, while instructions and downloads are provided for Apache, we are downloading the PFX file provided under the instructions for IIS. PFX files contain the public certificate, private key, and certificate chain in a single file. We need this single file to upload the certificate to Octopus. Here we download the PFX file for the octopus.tech domain:

Create the certificate in Octopus

Deploying certificates is an ongoing operation. In particular, certificates provided by Let’s Encrypt expire every three months, and so need to be frequently refreshed.

This makes deploying certificates a great use case for runbooks. Unlike a regular deployment, runbooks don’t need to progress through environments, and you don’t need to create a deployment. We’ll create a runbook to deploy the certificate to the Apache load balancers.

First, we need to upload the PFX certificate that was generated by the DNS provider. In the screenshot below you can see the Let’s Encrypt certificate uploaded to the Octopus certificate store:

Deploy the certificate

Create a new project in Octopus, and add the certificate we just uploaded as a variable:

Next, create a runbook with a single Run a Script step.

The first step in the script is to enable mod_ssl with the command:

a2enmod ssl

Then create some directories to hold the certificate, certificate chain, and private key:

[ ! -d "/etc/apache2/ssl" ] && mkdir /etc/apache2/ssl
[ ! -d "/etc/apache2/ssl/private" ] && mkdir /etc/apache2/ssl/private

The contents of the certificate variable are then saved as files into the directories above. Certificates are special variables that expose the individual components that make up a certificate with expanded properties. We need to access three properties:

Certificate.CertificatePem, which is the public certificate.
Certificate.PrivateKeyPem, which is the private key.
Certificate.ChainPem, which is the certificate chain.

We print the contents of these variables into three files:

get_octopusvariable "Certificate.CertificatePem" > /etc/apache2/ssl/octopus_tech.crt
get_octopusvariable "Certificate.PrivateKeyPem" > /etc/apache2/ssl/private/octopus_tech.key
get_octopusvariable "Certificate.ChainPem" > /etc/apache2/ssl/octopus_tech_bundle.pem

If you recall, earlier in the post we created the file /etc/apache2/sites-enabled/000-default.conf with the following contents:

<VirtualHost *:80>
  ErrorLog ${APACHE_LOG_DIR}/error.log
  CustomLog ${APACHE_LOG_DIR}/access.log combined
  JkMount /* loadbalancer
</VirtualHost>

We want to modify this file, like so:

<VirtualHost *:443>
  ErrorLog ${APACHE_LOG_DIR}/error.log
  CustomLog ${APACHE_LOG_DIR}/access.log combined
  JkMount /* loadbalancer
  SSLEngine on
  SSLCertificateFile /etc/apache2/ssl/octopus_tech.crt
  SSLCertificateKeyFile /etc/apache2/ssl/private/octopus_tech.key
  SSLCertificateChainFile /etc/apache2/ssl/octopus_tech_bundle.pem
</VirtualHost>

This is done by echoing the desired text into the file /etc/apache2/sites-enabled/000-default.conf:

{
cat << EOF
<VirtualHost *:443>
  ErrorLog ${APACHE_LOG_DIR}/error.log
  CustomLog ${APACHE_LOG_DIR}/access.log combined
  JkMount /* loadbalancer
  SSLEngine on
  SSLCertificateFile /etc/apache2/ssl/octopus_tech.crt
  SSLCertificateKeyFile /etc/apache2/ssl/private/octopus_tech.key
  SSLCertificateChainFile /etc/apache2/ssl/octopus_tech_bundle.pem
</VirtualHost>
EOF
} > /etc/apache2/sites-enabled/000-default.conf

The final step is to restart the apache2 service to load the changes:

systemctl restart apache2

Here is the complete script for reference:

a2enmod ssl

[ ! -d "/etc/apache2/ssl" ] && mkdir /etc/apache2/ssl
[ ! -d "/etc/apache2/ssl/private" ] && mkdir /etc/apache2/ssl/private
get_octopusvariable "Certificate.CertificatePem" > /etc/apache2/ssl/octopus_tech.crt
get_octopusvariable "Certificate.PrivateKeyPem" > /etc/apache2/ssl/private/octopus_tech.key
get_octopusvariable "Certificate.ChainPem" > /etc/apache2/ssl/octopus_tech_bundle.pem

{
cat << EOF
<VirtualHost *:443>
  ErrorLog ${APACHE_LOG_DIR}/error.log
  CustomLog ${APACHE_LOG_DIR}/access.log combined
  JkMount /* loadbalancer
  SSLEngine on
  SSLCertificateFile /etc/apache2/ssl/octopus_tech.crt
  SSLCertificateKeyFile /etc/apache2/ssl/private/octopus_tech.key
  SSLCertificateChainFile /etc/apache2/ssl/octopus_tech_bundle.pem
</VirtualHost>
EOF
} > /etc/apache2/sites-enabled/000-default.conf
systemctl restart apache2

After the runbook has completed, we can verify the application is exposed via HTTPS:

Internal certificate management

As we’ve seen, it is useful to connect directly to the Tomcat instances when using the Manager console. This connection transfers credentials and should be done across a secure connection. To support this, we configure Tomcat with self-signed certificates.

Create self-signed certificates

Because our Tomcat instances are not exposed via a hostname, we don’t have the option of getting a certificate for them. To enable HTTPS, we need to create self-signed certificates, which can be done with OpenSSL:

openssl genrsa 2048 > private.pem
openssl req -x509 -new -key private.pem -out public.pem -days 3650
openssl pkcs12 -export -in public.pem -inkey private.pem -out mycert.pfx

Add the certificate to Tomcat

The certificate is configured in Tomcat using the Deploy a certificate to Tomcat step.

The Tomcat CATALINA_HOME path is set to /usr/share/tomcat9 and the Tomcat CATALINA_BASE path is set to /var/lib/tomcat9.

We reference a certificate variable for the Select certificate variable field. The default value of Catalina is fine for the Tomcat service name.

We have a few choices for how the certificate is handled by Tomcat. Generally speaking, the Blocking IO, Non-Blocking IO, Non-Blocking IO 2, and Apache Portable Runtime options have an increasing level of performance. The Apache Portable Runtime is an additional library that Tomcat can take advantage of, and it is provided by the Tomcat packages we installed with apt-get, so it makes sense to use that option.

To allow Tomcat to use the new configuration, we need to restart the service with a script step using the command:

systemctl restart tomcat9

We can now load the manager console from https://tomcatip:8443/manager/html.

Scale up to multiple environments

The infrastructure we have created thus far can now be used as a template for other testing or production environments. Nothing we have presented here is environment specific, meaning all the processes and infrastructure can be scaled out to as many environments as needed.

By associating the Tentacles assigned to the new Tomcat and load balancer instanced to additional environments in Octopus, we gain the ability to push deployments through to production:

Conclusion

If you have reached this point, congratulations! Setting up a highly available Tomcat cluster with zero downtime deployments, feature branches, rollback, and HTTPS is not for the fainthearted. It is still up to the end user to combine multiple technologies to achieve this result, but I hope the instructions laid out in this blog post expose some of the magic that goes into real world Java deployments.

To summarize, in this post we:

Configured Tomcat session replication with a PostgreSQL database and session cookie rewriting with the JvmRouteBinderValve valve.
Configured Apache web servers acting as load balancers with the mod_jk plugin.
Implemented high availability amongst the load balancers with Keepalived.
Performed zero downtime deployments with Tomcat’s parallel deployment feature and Flyway performing backward compatible database migrations.
Smoke tested the deployments with community steps in Octopus.
Implemented feature branch deployments, taking into account the limitations of the Maven versioning strategy with Octopus channels.
Looked at how applications can be rolled back or pulled from service.
Added HTTPS certificates to Apache.
Repeated the process for multiple environments.

Creating a Kubernetes Operator with Kotlin

Matthew Casperson — Thu, 27 Feb 2020 23:47:26 +0000

Most environments will initially treat their Kubernetes cluster as a tool to orchestrate containers and configure traffic between them. Kubernetes supports this use case very well by providing declarative descriptions of the desired container state and their connections.

When used in this way, developers and operations staff sit outside of the cluster, looking in. The cluster is managed with calls to kubectl that are made in an ad-hoc fashion or from a CI/CD pipeline. This means Kubernetes itself is quite naïve; it understands how to reconfigure itself to match the desired state, but it has no understanding of what that state represents.

For example, a common Kubernetes deployment might see three pods created: a front end web application, a backend web service, and a database. The relationship between these pods is well understood by the developers deploying them as a classic three-tier architecture, but Kubernetes literally sees nothing more than three pods to be deployed, monitored, and exposed to network traffic.

The operator pattern has evolved as a way of encapsulating business knowledge and operational workflows in the Kubernetes cluster itself, allowing a cluster to implement high level, domain-specific concepts with the common, low-level resources like pods, services, and deployments, etc.

The term was originally coined by Brandon Philips in the blog post Introducing Operators: Putting Operational Knowledge into Software and offers this definition:

It builds upon the basic Kubernetes resource and controller concepts but includes domain or application-specific knowledge to automate common tasks.

The three key components identified in this definition are:

Resource
Controller
Domain or application-specific knowledge

In practice, a resource means a Custom Resource Definition (CRD), a controller means an application integrated into and responding to the Kubernetes API, and the application-specific knowledge is the logic implemented in the controller to reify high-level concepts from standard Kubernetes resources.

To understand the operator pattern, let’s look at a simple example written in Kotlin. The code for this operator is available from GitHub, and it is based on the code from this RedHat blog. The operator will extend the Kubernetes cluster with the concept of a web server with a WebServer CRD and a controller that builds pods with an image known to expose a sample web server.

The CRD meets the resource requirement, the code we’ll write to interact with the Kubernetes API meets the controller requirement, and the knowledge that a particular Docker image is used to expose a sample web server is the application-specific knowledge.

The pom.xml file

We start with the Maven pom.xml file. This file defines the dependencies required for Kotlin itself and the fabric8 Kubernetes client library. The complete pom.xml file is shown below:

<?xml version="1.0" encoding="UTF-8"?>
<project>
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.octopus</groupId>
    <artifactId>kotlink8soperator</artifactId>
    <version>1.0</version>

    <properties>
        <kotlin.version>1.3.61</kotlin.version>
        <version.fabric8.client>4.7.0</version.fabric8.client>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.jetbrains.kotlin</groupId>
            <artifactId>kotlin-stdlib</artifactId>
            <version>${kotlin.version}</version>
        </dependency>
        <dependency>
            <groupId>io.fabric8</groupId>
            <artifactId>kubernetes-client</artifactId>
            <version>${version.fabric8.client}</version>
        </dependency>
    </dependencies>

    <build>
        <sourceDirectory>${project.basedir}/src/main/kotlin</sourceDirectory>
        <testSourceDirectory>${project.basedir}/src/test/kotlin</testSourceDirectory>

        <plugins>
            <plugin>
                <groupId>org.jetbrains.kotlin</groupId>
                <artifactId>kotlin-maven-plugin</artifactId>
                <version>${kotlin.version}</version>

                <executions>
                    <execution>
                        <id>compile</id>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>

                    <execution>
                        <id>test-compile</id>
                        <goals>
                            <goal>test-compile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

Anatomy of a Kubernetes resource

Before we dive into the Kotlin code, we need to understand the common structure of all Kubernetes resources. Here is the YAML definition of a deployment resource that we’ll use as an example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80
status:
  availableReplicas: 2
  observedGeneration: 1
  readyReplicas: 2
  replicas: 2
  updatedReplicas: 2

This resource can be broken down into four components.

The first component is the Group, Version, and Kind (GVK). The deployment resource has a group of apps, a version of v1 and a kind of Deployment:

apiVersion: apps/v1
kind: Deployment

The second component is the metadata. This is where labels, annotations, names, and namespaces are defined:

metadata:
  name: nginx-deployment
  labels:
    app: nginx

The third component is the spec, which defines the properties of the specific resource:

spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

The fourth component is the status. The details in this component are generated by Kubernetes to reflect the current state of the resource:

status:
  availableReplicas: 2
  observedGeneration: 1
  readyReplicas: 2
  replicas: 2
  updatedReplicas: 2

The CRD classes

Now that we know the components that make up a Kubernetes resource, we can look at the code that reflects the CRD implemented by the operator.

We are creating a new CRD called WebServer, which is represented by a class also called WebServer. This class has two properties defining the spec and the status:

package com.octopus.webserver.operator.crd

import io.fabric8.kubernetes.client.CustomResource

data class WebServer(var spec: WebServerSpec = WebServerSpec(),
                     var status: WebServerStatus = WebServerStatus()) : CustomResource()

The spec for our CRD is represented in the WebServerSpec class. This has a single field called replicas indicating how many web server pods this CRD is responsible for creating:

package com.octopus.webserver.operator.crd

import com.fasterxml.jackson.databind.annotation.JsonDeserialize
import io.fabric8.kubernetes.api.model.KubernetesResource

@JsonDeserialize
data class WebServerSpec(val replicas: Int = 0) : KubernetesResource

The status of our CRD is represented in the WebServerStatus class. It contains a single field called count that reports how many pods have been created:

package com.octopus.webserver.operator.crd

import com.fasterxml.jackson.databind.annotation.JsonDeserialize
import io.fabric8.kubernetes.api.model.KubernetesResource

@JsonDeserialize
data class WebServerStatus(var count: Int = 0) : KubernetesResource

The final two classes, called WebServerList and DoneableWebServer, contain no custom properties or logic, and are boilerplate code required by the fabric8 library:

package com.octopus.webserver.operator.crd

import io.fabric8.kubernetes.client.CustomResourceList

class WebServerList : CustomResourceList<WebServer>()

package com.octopus.webserver.operator.crd

import io.fabric8.kubernetes.client.CustomResourceDoneable
import io.fabric8.kubernetes.api.builder.Function

class DoneableWebServer(resource: WebServer, function: Function<WebServer,WebServer>) :
        CustomResourceDoneable<WebServer>(resource, function)

The main function

The main() function is the entry point to our controller. Here is the complete code:

package com.octopus.webserver.operator

import com.octopus.webserver.operator.controller.WebServerController
import com.octopus.webserver.operator.crd.WebServer
import com.octopus.webserver.operator.crd.WebServerList
import io.fabric8.kubernetes.api.model.Pod
import io.fabric8.kubernetes.api.model.PodList
import io.fabric8.kubernetes.api.model.apiextensions.CustomResourceDefinitionBuilder
import io.fabric8.kubernetes.client.DefaultKubernetesClient
import io.fabric8.kubernetes.client.dsl.base.CustomResourceDefinitionContext


fun main(args: Array<String>) {
    val client = DefaultKubernetesClient()
    client.use {
        val namespace = client.namespace ?: "default"
        val podSetCustomResourceDefinition = CustomResourceDefinitionBuilder()
                .withNewMetadata().withName("webservers.demo.k8s.io").endMetadata()
                .withNewSpec()
                .withGroup("demo.k8s.io")
                .withVersion("v1alpha1")
                .withNewNames().withKind("WebServer").withPlural("webservers").endNames()
                .withScope("Namespaced")
                .endSpec()
                .build()
        val webServerCustomResourceDefinitionContext = CustomResourceDefinitionContext.Builder()
                .withVersion("v1alpha1")
                .withScope("Namespaced")
                .withGroup("demo.k8s.io")
                .withPlural("webservers")
                .build()
        val informerFactory = client.informers()
        val podSharedIndexInformer = informerFactory.sharedIndexInformerFor(
                Pod::class.java,
                PodList::class.java,
                10 * 60 * 1000.toLong())
        val webServerSharedIndexInformer = informerFactory.sharedIndexInformerForCustomResource(
                webServerCustomResourceDefinitionContext,
                WebServer::class.java,
                WebServerList::class.java,
                10 * 60 * 1000.toLong())
        val webServerController = WebServerController(
                client,
                podSharedIndexInformer,
                webServerSharedIndexInformer,
                podSetCustomResourceDefinition,
                namespace)

        webServerController.create()
        informerFactory.startAllRegisteredInformers()

        webServerController.run()
    }
}

We create a DefaultKubernetesClient, which gives us access to the Kubernetes API:

val client = DefaultKubernetesClient()

The client knows how to configure itself based on the environment it is executed in. We’ll run this code locally when testing, meaning the client will access the details of the Kubernetes cluster from the ~/.kube/config file. The namespace is then extracted from the client’s configuration, or set to default if no namespace setting was found:

val namespace = client.namespace ?: "default"

The CustomResourceDefinitionBuilder defines the WebServer CRD that this controller manages. This is used when working with the client to update resources in the cluster.

val podSetCustomResourceDefinition = CustomResourceDefinitionBuilder()
        .withNewMetadata().withName("webservers.demo.k8s.io").endMetadata()
        .withNewSpec()
        .withGroup("demo.k8s.io")
        .withVersion("v1alpha1")
        .withNewNames().withKind("WebServer").withPlural("webservers").endNames()
        .withScope("Namespaced")
        .endSpec()
        .build()

The controller works by listening to events that indicate the resources it should be managing have changed. To listen to events relating to the WebServer CRD, we create a CustomResourceDefinitionContext:

val webServerCustomResourceDefinitionContext = CustomResourceDefinitionContext.Builder()
        .withVersion("v1alpha1")
        .withScope("Namespaced")
        .withGroup("demo.k8s.io")
        .withPlural("webservers")
        .build()

We are notified of events through informers, and the informers are created from a factory provided by the client:

val informerFactory = client.informers()

Here we create an informer that will notify us of events relating to pods. Because pods are a standard resource in Kubernetes, creating this informer did not require a CustomResourceDefinitionContext:

val podSharedIndexInformer = informerFactory.sharedIndexInformerFor(
        Pod::class.java,
        PodList::class.java,
        10 * 60 * 1000.toLong())

Here we create an informer that will notify us of events relating to our CRD. This required the CustomResourceDefinitionContext created previously:

val webServerSharedIndexInformer = informerFactory.sharedIndexInformerForCustomResource(
        webServerCustomResourceDefinitionContext,
        WebServer::class.java,
        WebServerList::class.java,
        10 * 60 * 1000.toLong())

The logic of the operator is contained in the controller. In this project, the WebServerController class fulfills the role of the controller:

val webServerController = WebServerController(
        client,
        podSharedIndexInformer,
        webServerSharedIndexInformer,
        podSetCustomResourceDefinition,
        namespace)

The controller links up event handlers in the create() method, we start listening for events, and then enter the reconcile loop by calling the run() method:

webServerController.create()
informerFactory.startAllRegisteredInformers()

webServerController.run()

The controller

The WebServerController class implements the controller in our operator. Its job is to listen for changes to Kubernetes resources and reconcile the current state with the desired state. The complete code for the class is shown below:

package com.octopus.webserver.operator.controller

import com.octopus.webserver.operator.crd.DoneableWebServer
import com.octopus.webserver.operator.crd.WebServer
import com.octopus.webserver.operator.crd.WebServerList
import io.fabric8.kubernetes.api.model.OwnerReference
import io.fabric8.kubernetes.api.model.Pod
import io.fabric8.kubernetes.api.model.PodBuilder
import io.fabric8.kubernetes.api.model.apiextensions.CustomResourceDefinition
import io.fabric8.kubernetes.client.KubernetesClient
import io.fabric8.kubernetes.client.informers.ResourceEventHandler
import io.fabric8.kubernetes.client.informers.SharedIndexInformer
import io.fabric8.kubernetes.client.informers.cache.Cache
import io.fabric8.kubernetes.client.informers.cache.Lister
import java.util.*
import java.util.AbstractMap.SimpleEntry
import java.util.concurrent.ArrayBlockingQueue


class WebServerController(private val kubernetesClient: KubernetesClient,
                          private val podInformer: SharedIndexInformer<Pod>,
                          private val webServerInformer: SharedIndexInformer<WebServer>,
                          private val webServerResourceDefinition: CustomResourceDefinition,
                          private val namespace: String) {
    private val APP_LABEL = "app"
    private val webServerLister = Lister<WebServer>(webServerInformer.indexer, namespace)
    private val podLister = Lister<Pod>(podInformer.indexer, namespace)
    private val workQueue = ArrayBlockingQueue<String>(1024)

    fun create() {
        webServerInformer.addEventHandler(object : ResourceEventHandler<WebServer> {
            override fun onAdd(webServer: WebServer) {
                enqueueWebServer(webServer)
            }

            override fun onUpdate(webServer: WebServer, newWebServer: WebServer) {
                enqueueWebServer(newWebServer)
            }

            override fun onDelete(webServer: WebServer, b: Boolean) {}
        })

        podInformer.addEventHandler(object : ResourceEventHandler<Pod> {
            override fun onAdd(pod: Pod) {
                handlePodObject(pod)
            }

            override fun onUpdate(oldPod: Pod, newPod: Pod) {
                if (oldPod.metadata.resourceVersion == newPod.metadata.resourceVersion) {
                    return
                }
                handlePodObject(newPod)
            }

            override fun onDelete(pod: Pod, b: Boolean) {}
        })
    }

    private fun enqueueWebServer(webServer: WebServer) {
        val key: String = Cache.metaNamespaceKeyFunc(webServer)
        if (key.isNotEmpty()) {
            workQueue.add(key)
        }
    }

    private fun handlePodObject(pod: Pod) {
        val ownerReference = getControllerOf(pod)

        if (ownerReference?.kind?.equals("WebServer", ignoreCase = true) != true) {
            return
        }

        webServerLister
                .get(ownerReference.name)
                ?.also { enqueueWebServer(it) }
    }

    private fun getControllerOf(pod: Pod): OwnerReference? =
            pod.metadata.ownerReferences.firstOrNull { it.controller }

    private fun reconcile(webServer: WebServer) {
        val pods = podCountByLabel(APP_LABEL, webServer.metadata.name)
        val existingPods = pods.size

        webServer.status.count = existingPods
        updateStatus(webServer)

        if (existingPods < webServer.spec.replicas) {
            createPod(webServer)
        } else if (existingPods > webServer.spec.replicas) {
            kubernetesClient
                    .pods()
                    .inNamespace(webServer.metadata.namespace)
                    .withName(pods[0])
                    .delete()
        }
    }

    private fun updateStatus(webServer: WebServer) =
            kubernetesClient.customResources(webServerResourceDefinition, WebServer::class.java, WebServerList::class.java, DoneableWebServer::class.java)
                    .inNamespace(webServer.metadata.namespace)
                    .withName(webServer.metadata.name)
                    .updateStatus(webServer)

    private fun podCountByLabel(label: String, webServerName: String): List<String> =
            podLister.list()
                    .filter { it.metadata.labels.entries.contains(SimpleEntry(label, webServerName)) }
                    .filter { it.status.phase == "Running" || it.status.phase == "Pending" }
                    .map { it.metadata.name }

    private fun createPod(webServer: WebServer) =
            createNewPod(webServer).let { pod ->
                kubernetesClient.pods().inNamespace(webServer.metadata.namespace).create(pod)
            }

    private fun createNewPod(webServer: WebServer): Pod =
            PodBuilder()
                    .withNewMetadata()
                    .withGenerateName(webServer.metadata.name.toString() + "-pod")
                    .withNamespace(webServer.metadata.namespace)
                    .withLabels(Collections.singletonMap(APP_LABEL, webServer.metadata.name))
                    .addNewOwnerReference()
                    .withController(true)
                    .withKind("WebServer")
                    .withApiVersion("demo.k8s.io/v1alpha1")
                    .withName(webServer.metadata.name)
                    .withNewUid(webServer.metadata.uid)
                    .endOwnerReference()
                    .endMetadata()
                    .withNewSpec()
                    .addNewContainer().withName("nginx").withImage("nginxdemos/hello").endContainer()
                    .endSpec()
                    .build()

    fun run() {
        blockUntilSynced()
        while (true) {
            try {
                workQueue
                        .take()
                        .split("/")
                        .toTypedArray()[1]
                        .let { webServerLister.get(it) }
                        ?.also { reconcile(it) }
            } catch (interruptedException: InterruptedException) {
                // ignored
            }
        }
    }

    private fun blockUntilSynced() {
        while (!podInformer.hasSynced() || !webServerInformer.hasSynced()) {}
    }
}

The create() method assigns anonymous classes as informer event handlers. The event handlers identify instances of the WebServer CRDs that need to be processed by calling either enqueueWebServer() or handlePodObject():

fun create() {
        webServerInformer.addEventHandler(object : ResourceEventHandler<WebServer> {
            override fun onAdd(webServer: WebServer) {
                enqueueWebServer(webServer)
            }

            override fun onUpdate(webServer: WebServer, newWebServer: WebServer) {
                enqueueWebServer(newWebServer)
            }

            override fun onDelete(webServer: WebServer, b: Boolean) {}
        })

        podInformer.addEventHandler(object : ResourceEventHandler<Pod> {
            override fun onAdd(pod: Pod) {
                handlePodObject(pod)
            }

            override fun onUpdate(oldPod: Pod, newPod: Pod) {
                if (oldPod.metadata.resourceVersion == newPod.metadata.resourceVersion) {
                    return
                }
                handlePodObject(newPod)
            }

            override fun onDelete(pod: Pod, b: Boolean) {}
        })
    }

enqueueWebServer() creates a key identifying the WebServer CRD and adds it to the workQueue:

private fun enqueueWebServer(webServer: WebServer) {
    val key: String = Cache.metaNamespaceKeyFunc(webServer)
    if (key.isNotEmpty()) {
        workQueue.add(key)
    }
}

handlePodObject() first determines if the pod is managed by a WebServer through the ownerReference. If it is, the owning WebServer is added to the workQueue by calling enqueueWebServer():

private fun handlePodObject(pod: Pod) {
    val ownerReference = getControllerOf(pod)

    if (ownerReference?.kind?.equals("WebServer", ignoreCase = true) != true) {
        return
    }

    webServerLister
            .get(ownerReference.name)
            ?.also { enqueueWebServer(it) }
}

private fun getControllerOf(pod: Pod): OwnerReference? =
        pod.metadata.ownerReferences.firstOrNull { it.controller }

reconcile() provides the logic that ensures the cluster has as many pods as required by the WebServer CRD. It calls podCountByLabel() to find out how many pods exist, and updates the status of the CRD with a call to updateStatus(). If there are too few pods to meet the requirements, createPod() is called. If there are too many pods, one is deleted.

By continually creating or deleting pods to push the cluster towards the desired state, we will eventually satisfy the requirements of the WebServer CRD:

private fun reconcile(webServer: WebServer) {
    val pods = podCountByLabel(APP_LABEL, webServer.metadata.name)
    val existingPods = pods.size

    webServer.status.count = existingPods
    updateStatus(webServer)

    if (existingPods < webServer.spec.replicas) {
        createPod(webServer)
    } else if (existingPods > webServer.spec.replicas) {
        kubernetesClient
                .pods()
                .inNamespace(webServer.metadata.namespace)
                .withName(pods[0])
                .delete()
    }
}

updateStatus() uses the client to update the status component of our custom resource. The status component is unique because updating it does not trigger an update event in our code. Only a controller can update the status component of a resource, and Kubernetes has been designed to prevent status updates from triggering an infinite event loop:

private fun updateStatus(webServer: WebServer) =
        kubernetesClient.customResources(webServerResourceDefinition, WebServer::class.java, WebServerList::class.java, DoneableWebServer::class.java)
                .inNamespace(webServer.metadata.namespace)
                .withName(webServer.metadata.name)
                .updateStatus(webServer)

podCountByLabel() returns the names of pods that are managed by the CRD that are either running or in the process of being created:

private fun podCountByLabel(label: String, webServerName: String): List<String> =
        podLister.list()
                .filter { it.metadata.labels.entries.contains(SimpleEntry(label, webServerName)) }
                .filter { it.status.phase == "Running" || it.status.phase == "Pending" }
                .map { it.metadata.name }

createPod() and createNewPod() create a new pod. It is here that our business logic has been codified with the use of the nginxdemos/hello Docker image as our test web server:

private fun createPod(webServer: WebServer) =
        createNewPod(webServer).let { pod ->
            kubernetesClient.pods().inNamespace(webServer.metadata.namespace).create(pod)
        }

private fun createNewPod(webServer: WebServer): Pod =
        PodBuilder()
                .withNewMetadata()
                .withGenerateName(webServer.metadata.name.toString() + "-pod")
                .withNamespace(webServer.metadata.namespace)
                .withLabels(Collections.singletonMap(APP_LABEL, webServer.metadata.name))
                .addNewOwnerReference()
                .withController(true)
                .withKind("WebServer")
                .withApiVersion("demo.k8s.io/v1alpha1")
                .withName(webServer.metadata.name)
                .withNewUid(webServer.metadata.uid)
                .endOwnerReference()
                .endMetadata()
                .withNewSpec()
                .addNewContainer().withName("nginx").withImage("nginxdemos/hello").endContainer()
                .endSpec()
                .build()

The run() method is an infinite loop continually consuming a web server resource ID added to the workQueue by the event listeners and passing it to the reconcile() method:

fun run() {
    blockUntilSynced()
    while (true) {
        try {
            workQueue
                    .take()
                    .split("/")
                    .toTypedArray()[1]
                    .let { webServerLister.get(it) }
                    ?.also { reconcile(it) }
        } catch (interruptedException: InterruptedException) {
            // ignored
        }
    }
}

private fun blockUntilSynced() {
    while (!podInformer.hasSynced() || !webServerInformer.hasSynced()) {}
}

The CRD YAML

The final piece of the operator is the CRD itself. A CRD is simply another Kubernetes resource, and we define it in the following YAML:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: webservers.demo.k8s.io
spec:
  group: demo.k8s.io
  version: v1alpha1
  names:
    kind: WebServer
    plural: webservers
  scope: Namespaced
  subresources:
    status: {}

Putting it all together

To run the operator, we first need to apply the CRD YAML:

kubectl apply -f crd.yml

Then, we create an instance of our CRD with the YAML:

apiVersion: demo.k8s.io/v1alpha1
kind: WebServer
metadata:
  name: example-webserver
spec:
  replicas: 5

The controller can then run locally. Because the client we used in our code knows how to configure itself based on where it is run, executing our code locally, means that the client configures itself from the ~/.kube/config file. In the screenshot below you can see the controller run directly from my IDE:

The controller responds to the new web server CRD and creates the required pods:

$ kubectl get pods
NAME                         READY   STATUS    RESTARTS   AGE
example-webserver-pod92ht9   1/1     Running   0          54s
example-webserver-podgbz86   1/1     Running   0          54s
example-webserver-podk58gz   1/1     Running   0          54s
example-webserver-podkftmp   1/1     Running   0          54s
example-webserver-podpwzrt   1/1     Running   0          54s

The status of the web server resource is updated with the count of the pods it has successfully created:

$ kubectl get webservers -n default -o yaml
apiVersion: v1
items:
- apiVersion: demo.k8s.io/v1alpha1
  kind: WebServer
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"demo.k8s.io/v1alpha1","kind":"WebServer","metadata":{"annotations":{},"name":"example-webserver","namespace":"default"},"spec":{"replicas":5}}
    creationTimestamp: "2020-01-16T20:19:23Z"
    generation: 1
    name: example-webserver
    namespace: default
    resourceVersion: "112308"
    selfLink: /apis/demo.k8s.io/v1alpha1/namespaces/default/webservers/example-webserver
    uid: 9eb08575-8fa1-4bc9-bb2b-6f11b7285b68
  spec:
    replicas: 5
  status:
    count: 5
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

The power of operators

Without an operator, the concept of a test web server lived outside of the cluster. Developers may have emailed around the YAML they use to create test pods with, but more likely, everyone had their own opinion of what a test web server was.

The operator we created extends our Kubernetes cluster with a specific implementation of a test web server. Encapsulating this business knowledge allows the cluster to create and manage high-level concepts specific to our environment.

Creating and managing new resources is just one example of what an operator can do. Automating tasks like security scans, reporting, and load testing are all valid use cases for operators. A list of popular operators is available here.

Conclusion

Operators are a much hyped but often poorly understood pattern. With the definition from the original blog post describing operators, we saw the three simple parts to an operator: a resource to define them, a controller to act on the Kubernetes resources, and logic to implement application-specific knowledge. We then implemented a simple operator in Kotlin to create test web servers.

Kubernetes deployment strategies visualized

Matthew Casperson — Mon, 17 Feb 2020 21:24:42 +0000

One of the benefits Kubernetes provides administrators and developers is the ability to intelligently manage deployments of new software or configuration.

Kubernetes includes two built-in deployment strategies called recreate and rolling update, which are configured directly on the deployment resources. Octopus offers a third Kubernetes deployment strategy called blue green, which is managed through the Deploy Kubernetes containers step.

But what do these strategies actually do? In this blog post, we’ll visualize these deployment strategies to highlight their differences and note why you would choose one strategy over another.

The test deployment

For the videos below, we will watch a deployment as it is updated on a multi-node Kubernetes cluster. Each node in the cluster has a unique label, and the deployment is updated to place the new pods on a specific node.

The end result is that the new deployment shifts pods from one node to the other. We can then watch how the pods are moved between nodes to see the effect of the different deployment strategies.

Kubernetes deployment recreate strategy

The Kubernetes deployment recreate strategy is the simplest of the three. When a deployment configured with the recreate strategy is updated, Kubernetes will first delete the pods from the existing deployment, and once those pods are removed, the new pods are created.

In the above video, you can see all the pods on node 1 are deleted, and only after they are removed are the new pods on the second node created.

The recreate strategy ensures that the old and new pods do not run concurrently. This can be beneficial when synchronizing changes to a backend datastore that does not support access from two different client versions. However, there is a period of downtime before the new pods start accepting traffic.

Kubernetes rolling update

As its name suggests, the Kubernetes rolling update strategy incrementally deploys new pods as the old pods are removed. You can see this in the above video, where a number of the pods on the first node are deleted at the same time as new pods are created on the second node. Eventually, the pods on the first node are all removed, and all the new pods are created on the second node.

The rolling update strategy ensures there are some pods available to continue serving traffic during the update, so there is no downtime. However, both the old and new pods run side by side while the update is taking place, meaning any datastores or clients must be able to interact with both versions.

Kubernetes blue green deployment

Unlike the other deployment strategies, the Kubernetes blue green deployment strategy is not something natively implemented by Kubernetes. It involves creating an entirely new deployment resource (i.e., a deployment resource with a new name), waiting for the new deployment to become ready, switching traffic from the old deployment to the new deployment, and finally deleting the old deployment. This process is implemented in Octopus via the Deploy Kubernetes containers step.

In the above video, you can see during a blue/green deployment, the pods on the second node are deployed and initialized, and when they are ready, the pods on the first node are deleted.

The blue/green strategy ensures the new deployment is fully initialized and healthy before any traffic is sent to it. Should the new deployment fail, the old deployment continues to serve traffic.

Like the rolling update strategy, the blue/green strategy deploys two versions side by side for a period of time, so any backing datastores need to support two different clients. However, by cutting all traffic over to the new deployment when it’s ready, only one version of the deployment will be accessible at any time.

Conclusion

Selecting the correct deployment strategy is crucial to ensuring that your Kubernetes updates are reliable and remove or minimize downtime. Visualizing the available update strategies is helpful in understanding the differences between them, and in this post, we saw how pods were created and destroyed with the Kubernetes native strategies of recreate and rolling updates, and then with the blue/green strategy implemented by Octopus.

This post was originally published at octopus.com.