Forem: Pascal Gillet

Restricting Access by Geographical Location using NGINX with Helm

Pascal Gillet — Wed, 29 May 2024 08:35:00 +0000

This article explains how you can restrict content distribution to a particular country from services in your Kubernetes cluster, using the GeoIP2 dynamic module.

Prerequisites

Install NGINX Ingress Controller in your Kubernetes cluster using Helm.

Getting the GeoLite2 databases from MaxMind

The MaxMind company provides the GeoLite2 free IP geolocation databases. You need to create an account on the MaxMind website and generate a license key.

Configuring the NGINX Ingress Controller

Override the NGINX Helm chart with the following values:

controller:
  # Maxmind license key to download GeoLite2 Databases
  maxmindLicenseKey: ""
  extraArgs:
    # GeoLite2 Databases to download (default "GeoLite2-City,GeoLite2-ASN")
    maxmind-edition-ids: GeoLite2-Country
  service:
    # Preserve source IP...
    externalTrafficPolicy: Local
    # ...Which is only supported if we enable the v2 proxy protocol for the OVH load balancer (specific to OVH Cloud provider)
    annotations:
      service.beta.kubernetes.io/ovh-loadbalancer-proxy-protocol: "v2"
  config:
    use-proxy-protocol: "true"
    # Enable Ingress to parse and add -snippet annotations/directives
    allow-snippet-annotations: "true"
    # Enable geoip2 module
    use-geoip: "false"
    use-geoip2: "true"
    # Configure access by geographical location.
    # Here, we create a variable $allowed_country whose values 
    # depend on values of GeoIP2 variable $geoip2_country_code,
    # which lists all ISO 3166 country codes.
    # Map directives are only allowed at Ingress Controller level.
    http-snippet: |
      map $geoip2_country_code $allowed_country {
        default no;
        FR yes;
        US yes;
      }

Example Ingress

A minimal Ingress resource example:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: minimal-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    # Restrict access by geographical location
    nginx.ingress.kubernetes.io/server-snippet: |
      if ($allowed_country = no) {
        return 451;
      }
spec:
  ingressClassName: nginx-example
  rules:
  - http:
      paths:
      - path: /testpath
        pathType: Prefix
        backend:
          service:
            name: test
            port:
              number: 80

Note: The HTTP status code 451 was chosen as a reference to the novel
"Fahrenheit 451".

References

KubeCon 2022: nouveautés dans Kubernetes v1.24

Pascal Gillet — Thu, 19 May 2022 21:14:13 +0000

Au deuxième jour de la KubeCon 2022, la keynote démarre par l'annonce des nouveautés du projet Kubernetes v1.24.
En voici les principales:

Suppression de Dockershim

Rappelez-vous, Kubernetes v1.20 avait annoncé que Dockershim serait bientôt supprimé, c’est chose faite dans Kubernetes v1.24!

Pour rappel, les premières versions de Kubernetes ne fonctionnaient qu'avec un seul runtime de conteneurs: Docker Engine. Pourtant il en existe plein d'autres: containerd, CRI-O, Mirantis Container Runtime, etc.
Plus tard, Kubernetes a ajouté la prise en charge de l'utilisation d'autres runtimes de conteneurs, au travers de la norme CRI. La norme CRI a été créée pour permettre l'interopérabilité entre des orchestrateurs (comme Kubernetes) et de nombreux runtimes de conteneurs différents.
Or, Docker Engine n'implémente pas cette interface (CRI), donc le projet Kubernetes a dû créer un code spécifique pour continuer à supporter Docker Engine le temps de la transition et a intégré ce code dockershim à Kubernetes lui-même.

Le code dockershim a toujours été destiné à être une solution temporaire (d'où le nom : shim). En fait, la maintenance de dockershim était même devenue un lourd fardeau pour les mainteneurs de Kubernetes: des fonctionnalités qui sont maintenant implémentées dans les nouveaux runtimes CRI étaient largement incompatibles avec le dockershim, telles que cgroups v2 et les espaces de noms utilisateurs. La suppression du dockershim de Kubernetes permet de poursuivre le développement dans ces domaines.

Pour en savoir plus sur ce que cela signifie vraiment, consultez le billet de blog "Don't Panic: Kubernetes and Docker".

Pour déterminer l'impact que la suppression de dockershim peut avoir sur vos projets, vous pouvez lire "Check whether dockershim removal affects you".

Kubernetes Cluster API

Cluster API v1.0 est sortie!

Cluster API est un sous-projet Kubernetes axé sur la fourniture d'API déclaratives et d'outils pour simplifier le provisionnement, la mise à niveau et l'exploitation de clusters Kubernetes.

Lancé par le Special Interest Group (SIG) Kubernetes "Cluster Lifecycle", le projet Cluster API utilise des APIs et des modèles de style Kubernetes pour automatiser la gestion du cycle de vie des clusters pour les opérateurs de plates-formes. Cela signifie que l'infrastructure de votre cluster Kubernetes, c'est-à-dire les machines virtuelles, les réseaux, les load balancers et les VPCs, ainsi que la configuration du cluster lui-même sont définis de la même manière que les charges de travail qui y sont déployées: au travers d'objets Kubernetes définis dans des fichiers de configuration au format yaml. Cela permet des déploiements de cluster cohérents et reproductibles dans différents environnements d'infrastructure.

Storage

Volume Expansion

L'"expansion de volume" a été introduite en tant que fonctionnalité alpha dans Kubernetes 1.8, passée en bêta dans 1.11, et finalement GA avec Kubernetes 1.24.

Cette fonctionnalité permet de spécifier simplement une nouvelle taille dans la Spec des objets PersistentVolumeClaim et Kubernetes étendra automatiquement le volume à l'aide du backend de stockage et étendra également à chaud (quand cela est possible) le système de fichiers sous-jacent utilisé par le pod.

Storage Capacity Tracking

Kubernetes v1.24 inclut la prise en charge de l'API (cluster-level) pour le suivi de la capacité de stockage.
Pour l'utiliser, vous devez également utiliser un driver CSI prenant en charge le suivi de la capacité.

Kubernetes garde une trace de la capacité de stockage pour permettre au scheduler de placer des pods sur des noeuds avec une capacité de stockage suffisante pour les volumes manquants restants. Sans suivi de la capacité de stockage, le scheduler peut choisir un noeud qui n'a pas assez de capacité pour provisionner un volume et plusieurs tentatives de scheduling seront alors nécessaires.

Security - Pod Security Admission

Dans Kubernetes v1.24, la fonctionnalité en bêta PodSecurity est activée par défaut.

Pour rappel, la fonctionnalité "Pod Security Admission" permet de définir différents niveaux d'isolation pour les pods, pour restreindre le comportement des pods de manière claire et cohérente. Plus d'infos ici.

Performances

Les performances du scheduler sont accrues dans Kubernetes 1.24: notamment la latence est réduite et le débit de placement atteint maintenant 300 pods/sec (avez-vous besoin d'autant dans votre projet?)

D'autre part, le SIG "Batch Working Group" a été créé pour discuter et améliorer la prise en charge des workloads de type batch (HPC, AI/ML, analyse de données, CI, etc.) dans Kubernetes. Le but est d'unifier la façon dont les utilisateurs déploient ce type de charges de travail pour améliorer la portabilité et simplifier la prise en charge pour les providers Kubernetes.

Le travail se poursuit également au sein du SIG Scheduling avec le support d'une dizaine de plugins, pour notamment faire du co-scheduling avec les PodGroups (analogue au gang scheduling de Volcano?).

My Journey With Spark On Kubernetes... In Python (3/3)

Pascal Gillet — Mon, 12 Apr 2021 08:11:03 +0000

We need to operate Kubernetes as part of a Python client application. So, we need to interact with the Kubernetes REST API. Luckily we do not need to implement the API calls and manage HTTP requests/responses ourselves: we can rely on the Kubernetes Python client, among other officially-supported Kubernetes client libraries for other languages such as Go, Java, .NET, JavaScript and Haskell (there are also a lot of community-maintained client libraries for many languages).

Kubernetes Python Client

The Kubernetes Python Client is compatible with Python 2.7 and 3.4+. See the compatibility matrix for the supported versions of Kubernetes.

When using the client library, we must first load authentication and cluster information.

Load Authentication And Cluster Information

First, you need to setup the required service account and roles.

k8s/python-client-sa-rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: python-client-sa
  namespace: spark-jobs
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: spark-jobs
  name: python-client-role
rules:
- apiGroups: [""]
  resources: ["configmaps", "pods", "pods/log", "pods/status", "services"]
  verbs: ["*"]
- apiGroups: ["networking.k8s.io"]
  resources: ["ingresses", "ingresses/status"]
  verbs: ["*"]
- apiGroups: ["sparkoperator.k8s.io"]
  resources: [sparkapplications]
  verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: python-client-role-binding
  namespace: spark-jobs
subjects:
- kind: ServiceAccount
  name: python-client-sa
  namespace: spark-jobs
roleRef:
  kind: Role
  name: python-client-role
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-reader
rules:
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: python-client-cluster-role-binding
subjects:
- kind: ServiceAccount
  name: python-client-sa
  namespace: spark-jobs
roleRef:
  kind: ClusterRole
  name: node-reader
  apiGroup: rbac.authorization.k8s.io

kubectl create -f k8s/python-client-sa-rbac.yaml

This command creates a new service account named python-client-sa, a new role with the needed permissions in the spark-jobs namespace and then binds the new role to the newly created service account.

WARNING: The python-client-sa is the service account that will provide the identity for the Kubernetes Python Client in our application. Do not confuse this service account with the driver-sa service account for driver pods.

The Easy Way

In this method, we can use an helper utility to load authentication and cluster information from a kubeconfig file and store them in kubernetes.client.configuration.

from kubernetes import config, client

config.load_kube_config("path/to/kubeconfig_file")

v1 = client.CoreV1Api()

print("Listing pods with their IPs:")

ret = v1.list_namespaced_pod(namespace="spark-jobs")
for i in ret.items:
    print("%s\t%s\t%s" % (i.status.pod_ip, i.metadata.namespace, i.metadata.name))

But we DO NOT want to rely on the default kubeconfig file, denoted by the environment variable KUBECONFIG or, failing that, in ~/.kube/config. This kubeconfig file is yours, as user of the kubectl command. Concretely, with this kubeconfig file, you have the right to do almost everything in the Kubernetes cluster, and in all namespaces. Instead, we're going to generate one especially for the service account created above, with the help of the script kubeconfig-gen.sh:

#!/usr/bin/env bash

# set -eux

# Reads the API server name from the default `kubeconfig` file.
# Here we suppose that the kubectl command-line tool is already configured to communicate with our cluster.
APISERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')

SERVICE_ACCOUNT_NAME=${1:-python-client-sa}
NAMESPACE=${2:-spark-jobs}
SECRET_NAME=$(kubectl get serviceaccount ${SERVICE_ACCOUNT_NAME} -n ${NAMESPACE} -o jsonpath='{.secrets[0].name}')
TOKEN=$(kubectl get secret ${SECRET_NAME} -n ${NAMESPACE} -o jsonpath='{.data.token}' | base64 --decode)
CACERT=$(kubectl get secret ${SECRET_NAME} -n ${NAMESPACE} -o jsonpath="{['data']['ca\.crt']}")


cat > kubeconfig-sa << EOF
apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority-data: ${CACERT}
    server: ${APISERVER}
  name: default-cluster
contexts:
- context:
    cluster: default-cluster
    namespace: ${NAMESPACE}
    user: ${SERVICE_ACCOUNT_NAME}
  name: default-context
current-context: default-context
users:
- user:
    token: ${TOKEN}
  name: ${SERVICE_ACCOUNT_NAME}
EOF

The kubeconfig file thus created configures access to the cluster for the python-client-sa service account, with only the rights needed for our client application and in the single namespace spark-jobs ("principle of least privilege").

The Hard Way

Fetch credentials

Here, we're going to configure the Python client in the most programmatic way possible.

First, we need to fetch the credentials to access the Kubernetes cluster. We’ll store these in Python environmental variables.

export APISERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
SECRET_NAME=$(kubectl get serviceaccount python-client-sa -o jsonpath='{.secrets[0].name}')
export TOKEN=$(kubectl get secret ${SECRET_NAME} -o jsonpath='{.data.token}' | base64 --decode)
export CACERT=$(kubectl get secret ${SECRET_NAME} -o jsonpath="{['data']['ca\.crt']}")

Note that environment variables are captured the first time the os module is imported, typically during IDE/Python startup. Changes to the environment made after this time are not reflected in os.environ (except for changes made by modifying os.environ directly).

Python sample usage

import base64
import os
from tempfile import NamedTemporaryFile

from kubernetes import client

api_server = os.environ["APISERVER"]
cacert = os.environ["CACERT"]
token = os.environ["TOKEN"]

# Set the configuration
configuration = client.Configuration()
with NamedTemporaryFile(delete=False) as cert:
    cert.write(base64.b64decode(cacert))
    configuration.ssl_ca_cert = cert.name
configuration.host = api_server
configuration.verify_ssl = True
configuration.debug = False
configuration.api_key = {"authorization": "Bearer " + token}
client.Configuration.set_default(configuration)

v1 = client.CoreV1Api()

print("Listing pods with their IPs:")

ret = v1.list_namespaced_pod(namespace="spark-jobs")
for i in ret.items:
    print("%s\t%s\t%s" % (i.status.pod_ip, i.metadata.namespace, i.metadata.name))

Getting Started

Kubernetes Object Management

With the Kubernetes Python Client, you can create and manage Kubernetes objects programmatically.

In the following example (provided in the GitHub repository), we create, update then delete a Deployment using AppsV1Api:

"""
Creates, updates, and deletes a deployment using AppsV1Api.
"""

from kubernetes import client, config

DEPLOYMENT_NAME = "nginx-deployment"


def create_deployment_object():
    # Configureate Pod template container
    container = client.V1Container(
        name="nginx",
        image="nginx:1.15.4",
        ports=[client.V1ContainerPort(container_port=80)],
        resources=client.V1ResourceRequirements(
            requests={"cpu": "100m", "memory": "200Mi"},
            limits={"cpu": "500m", "memory": "500Mi"}
        )
    )
    # Create and configurate a spec section
    template = client.V1PodTemplateSpec(
        metadata=client.V1ObjectMeta(labels={"app": "nginx"}),
        spec=client.V1PodSpec(containers=[container]))
    # Create the specification of deployment
    spec = client.V1DeploymentSpec(
        replicas=3,
        template=template,
        selector={'matchLabels': {'app': 'nginx'}})
    # Instantiate the deployment object
    deployment = client.V1Deployment(
        api_version="apps/v1",
        kind="Deployment",
        metadata=client.V1ObjectMeta(name=DEPLOYMENT_NAME),
        spec=spec)

    return deployment


def create_deployment(api_instance, deployment):
    # Create deployement
    api_response = api_instance.create_namespaced_deployment(
        body=deployment,
        namespace="default")
    print("Deployment created. status='%s'" % str(api_response.status))


def update_deployment(api_instance, deployment):
    # Update container image
    deployment.spec.template.spec.containers[0].image = "nginx:1.16.0"
    # Update the deployment
    api_response = api_instance.patch_namespaced_deployment(
        name=DEPLOYMENT_NAME,
        namespace="default",
        body=deployment)
    print("Deployment updated. status='%s'" % str(api_response.status))


def delete_deployment(api_instance):
    # Delete deployment
    api_response = api_instance.delete_namespaced_deployment(
        name=DEPLOYMENT_NAME,
        namespace="default",
        body=client.V1DeleteOptions(
            propagation_policy='Foreground',
            grace_period_seconds=5))
    print("Deployment deleted. status='%s'" % str(api_response.status))


def main():
    # Configs can be set in Configuration class directly or using helper
    # utility. If no argument provided, the config will be loaded from
    # default location.
    config.load_kube_config("path/to/kubeconfig_file")
    apps_v1 = client.AppsV1Api()

    deployment = create_deployment_object()

    create_deployment(apps_v1, deployment)

    update_deployment(apps_v1, deployment)

    delete_deployment(apps_v1)


if __name__ == '__main__':
    main()

This is great, but this involves mastering the client's API, and above all we must configure our objects imperatively: we specify the desired operation (create, replace, etc.) on Python objects that represent Kubernetes objects. Here, we prefer to manage our objects in a declarative way and operate on object configuration files (stored locally along the Python code source) like we usually do with the kubectl command. Indeed, Python code should only be a simple execution backend to trigger Kubernetes operations, and business logic, so to speak, should be concentrated in manifest files.

The deployment we created above is the same as in the nginx-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.15.4
        ports:
        - containerPort: 80

We can directly load the manifest as follows:

from os import path
import yaml
from kubernetes import client, config


config.load_kube_config("path/to/kubeconfig_file")

with open(path.join(path.dirname(__file__), "nginx-deployment.yaml")) as f:
    dep = yaml.safe_load(f)
    k8s_apps_v1 = client.AppsV1Api()
    resp = k8s_apps_v1.create_namespaced_deployment(
        body=dep, namespace="default")
    print("Deployment created. status='%s'" % resp.metadata.name)

This is the equivalent in Python of kubectl create -f nginx-deployment.yaml.

As you can see, you must call create_namespaced_deployment to create a Deployment. In the same way, you would call create_namespaced_pod to create a Pod, and so on. This is because the Python client is automatically generated following the OpenAPI specifications of the Kubernetes API.

It's a shame to have to call a specific method to create a particular type of object, even though the type of object itself is already specified in the manifest that we load through this method. Luckily, the Kubernetes Python Client provides a utility method that acts as an input hub for any kind of object.

import os
import yaml
from kubernetes import client, config, utils


config.load_kube_config("path/to/kubeconfig_file")

with open(os.path.join(os.path.dirname(__file__), "nginx-deployment.yaml")) as f:
    dep = yaml.safe_load(f)
    k8s_client = client.ApiClient()
    resp = utils.create_from_dict(k8s_client, dep)
    print("Deployment created. status='%s'" % resp[0].metadata.name)

utils.create_from_dict is the magic method here. It only takes a Dict holding valid kubernetes objects. It is a blessing to have found it, because it is well hidden in the client and not documented at all.

So, to launch a Spark job with spark-submit, you could just call the code snippet above with a single YAML file which groups all the needed resources (separated by --- in YAML).

But what about the Spark Operator? utils.create_from_dict does not support custom resources, that means object types that are not part of the core Kubernetes API, namely SparkApplication from the Spark Operator. To run a Spark job with the Spark Operator, you have no other choice than calling the create_namespaced_custom_object function of CustomObjectsApi:

import os
import yaml
from kubernetes import client, config, utils


config.load_kube_config("path/to/kubeconfig_file")

with open(os.path.join(os.path.dirname(__file__), "k8s/spark-operator/pyspark-pi.yaml")) as f:
    dep = yaml.safe_load(f)
    custom_object_api = client.CustomObjectsApi()

    custom_object_api.create_namespaced_custom_object(
        group="sparkoperator.k8s.io",
        version="v1beta2",
        namespace="spark-jobs",
        plural="sparkapplications",
        body=dep,
    )
    print("SparkApplication created")

Regarding spark-submit, there is an even more direct method utils.create_from_yaml, which reads Kubernetes objects from a YAML file. But we cannot use it, as we need to "parameterize" our YAML files before submitting them to the Kubernetes Python client.

Templating

As you know, when you apply a manifest file to Kubernetes - the YAML-formatted resource descriptions that Kubernetes can understand - you must specify the resource name which must be unique for that type of resource (and within the same namespace), otherwise Kubernetes will complain that the resource already exists.

For example, you can only have one Pod named myapp-1234 within the same namespace, but you can have one Pod and one Deployment that are each named myapp-1234.

As we want to run multiple Spark jobs simultaneously, and as these Spark jobs are mostly identical except for a few runtime parameters, we need to parameterize, or templatize, our Kubernetes YAML files.

Normally, you don't do that, at least that's not in Kubernetes' philosophy: Kubernetes files should be template-free and should only be patched, by the means of Kustomize for instance. Helm has also its own templating system.

You cannot use such a tool with the Kubernetes Python client. Instead, we are going to substitute references to variables of the form $VAR or ${VAR} with the corresponding values, exactly like envsubst, but programmatically.

Let's get back to Spark (native). We saw earlier the YAML file that defines a driver pod to run the Pi example program. As you can see, we have placeholders to specify the namespace, the priorityClassName, the
serviceAccountName, the nodeAffinity and a NAME_SUFFIX to make the pod's name unique.

Now, in the Python code, we replace at runtime these variables with the desired values before creating the pod:

import binascii
import os
from os import listdir
from pprint import pprint

import yaml
from kubernetes import client, config, utils


def create_k8s_object(yaml_file=None, env_subst=None):
    with open(yaml_file) as f:
        str = f.read()
        if env_subst:
            for env, value in env_subst.items():
                str = str.replace(env, value)
        return yaml.safe_load(str)


def main():
    # Configs can be set in Configuration class directly or using helper utility
    config.load_kube_config("path/to/kubeconfig_file")

    name_suffix = "-" + binascii.b2a_hex(os.urandom(8))
    priority_class_name = "routine"
    env_subst = {"${NAMESPACE}": "spark-jobs",
                 "${SERVICE_ACCOUNT_NAME}": "driver-sa",
                 "${DRIVER_NODE_AFFINITIES}": "driver",
                 "${EXECUTOR_NODE_AFFINITIES}": "compute",
                 "${NAME_SUFFIX}": name_suffix,
                 "${PRIORITY_CLASS_NAME}": priority_class_name}

    k8s_client = client.ApiClient()
    verbose = True

    # Create driver pod
    k8s_dir = os.path.join(os.path.dirname(__file__), "k8s/spark-native")
    k8s_object_dict = create_k8s_object(os.path.join(k8s_dir, "pyspark-pi-driver-pod.yaml"), env_subst)
    pprint(k8s_object_dict)
    k8s_objects = utils.create_from_dict(k8s_client, k8s_object_dict, verbose=verbose)

    # TODO: create the other resources

    print("Submitted %s" % (k8s_objects[0].metadata.labels["app-name"]))

if __name__ == "__main__":
    main()

Putting the pieces together

We have now the general mechanics and we can create all the resources needed.
Remember, the driver pod consumes a ConfigMap to define environment variables and to mount configuration files in the Spark container (including the template for the executor pods). We also have a Service that allows executors to communicate back with the driver. And finally, we have another Service, backed by an Ingress, to expose the Spark UI.

We just iterate over the YAML files that define these resources and just call the same method utils.create_from_dict:

    # List all YAML files in k8s/spark-native directory, except the driver pod definition file
    other_resources = listdir(k8s_dir)
    other_resources.remove("pyspark-pi-driver-pod.yaml")
    for f in other_resources:
        k8s_object_dict = create_k8s_object(os.path.join(k8s_dir, f), env_subst)
        pprint(k8s_object_dict)
        utils.create_from_dict(k8s_client, k8s_object_dict, verbose=verbose)

    print("Submitted %s" % (k8s_objects[0].metadata.labels["app-name"]))

Now that we've launched a full Spark application, let's see what happens when we kill it 😈 or when the application completes normally.

Garbage collection

Killed applications

The role of the Kubernetes garbage collector is to delete certain objects that once had an owner, but no longer have one. The goal is to make sure that the garbage collector properly deletes resources that are no longer needed when killing a Spark application.
It is important to free up the resources of the Kubernetes cluster when you are going to run tens / hundreds of Spark applications in parallel.

For this, certain Kubernetes objects can be declared owners of other objects. "Owned" objects are called dependent on the owner object. Each dependent object has a metadata.ownerReferences field that points to the owning object. When deleting an owner object, all dependent objects are also automatically deleted (cascading deletion) by default.

Example of an executor pod owned by its driver pod:

apiVersion: v1
kind: Pod
metadata:
  labels:
    spark-role: executor
  ownerReferences:
    - apiVersion: v1
      controller: true
      kind: Pod
      name: pyspark-pi-routine-0245dc3d340cd533-driver
      uid: 3b10fa97-c847-4fce-b3e1-71f779cffbef
...

The Spark Operator automatically sets the value of ownerReference, at different levels: the custom SparkApplication resource owns the driver pod which owns its executors.

For applications that are submitted natively (without the Spark Operator), the highest level owner object is the driver pod: the executor pods automatically set the ownerReference field, pointing to the driver pod. But we must manage the ownership relationship ourselves for the other ConfigMap, Service and Ingress resources.
For this, we must retrieve the auto-generated uid of the newly created driver pod and inject it into the dependent objects: it is impossible to manually set the uid in the YAML definition files, this can only be done at runtime through code (and that's why we cannot put all the resources in a single YAML file).

import binascii
import os
from os import listdir
from pprint import pprint

import yaml
from kubernetes import config, utils
from kubernetes.client import ApiClient


def create_k8s_object(yaml_file=None, env_subst=None):
    with open(yaml_file) as f:
        str = f.read()
        if env_subst:
            for env in env_subst:
                str = str.replace(env, env_subst[env])
        return yaml.safe_load(str)


def main():
    # Configs can be set in Configuration class directly or using helper utility
    config.load_kube_config("path/to/kubeconfig_file")

    name_suffix = "-" + binascii.b2a_hex(os.urandom(8))
    priority_class_name = "routine"
    env_subst = {"${NAMESPACE}": "spark-jobs",
                 "${SERVICE_ACCOUNT_NAME}": "driver-sa",
                 "${DRIVER_NODE_AFFINITIES}": "driver",
                 "${EXECUTOR_NODE_AFFINITIES}": "compute",
                 "${NAME_SUFFIX}": name_suffix,
                 "${PRIORITY_CLASS_NAME}": priority_class_name}

    k8s_client = ApiClient()
    verbose = True

    # Create driver pod
    k8s_dir = os.path.join(os.path.dirname(__file__), "k8s/spark-native")
    k8s_object_dict = create_k8s_object(os.path.join(k8s_dir, "pyspark-pi-driver-pod.yaml"), env_subst)
    pprint(k8s_object_dict)
    k8s_objects = utils.create_from_dict(k8s_client, k8s_object_dict, verbose=verbose)

    # Prepare ownership on dependent objects
    owner_refs = [{"apiVersion": "v1",
                   "controller": True,
                   "kind": "Pod",
                   "name": k8s_objects[0].metadata.name,
                   "uid": k8s_objects[0].metadata.uid}]

    # List all YAML files in k8s/spark-native directory, except the driver pod definition file
    other_resources = listdir(k8s_dir)
    other_resources.remove("pyspark-pi-driver-pod.yaml")
    for f in other_resources:
        k8s_object_dict = create_k8s_object(os.path.join(k8s_dir, f), env_subst)
        # Set ownership
        k8s_object_dict["metadata"]["ownerReferences"] = owner_refs
        pprint(k8s_object_dict)
        utils.create_from_dict(k8s_client, k8s_object_dict, verbose=verbose)

    print("Submitted %s" % (k8s_objects[0].metadata.labels["app-name"]))

if __name__ == "__main__":
    main()

Applications normally completed

When an application completes normally, the executor pods terminate and are cleaned up, but the driver pod persists logs and remains in "completed" state in the Kubernetes API "until it's eventually garbage collected or cleaned up manually".

Note that in the completed state, the driver pod does not use any compute or memory resources.

The Spark Operator has TTL support for SparkApplications through the optional field named .spec.timeToLiveSeconds, which if set, defines the Time-To-Live (TTL) duration in seconds for a SparkAplication after its termination. The SparkApplication object will be garbage collected if the current time is more than the .spec.timeToLiveSeconds since its termination. The example below illustrates how to use the field:

spec:
  timeToLiveSeconds: 86400

On the native Spark side, there is nothing in the doc that specifies how driver pods are ultimately deleted. We could set up a simple Kubernetes CronJob that would run periodically to delete them automatically.

At the time of writing this article, there are pending requests in Kubernetes to support TTL in Pods like in Jobs: "TTL controller only handles Jobs for now, and may be expanded to handle other resources that will finish execution, such as Pods and custom resources."

Background cascading deletion

When killing an application from the Python code, we delete the owner object using the Background cascading deletion policy.
In background cascading deletion, Kubernetes deletes the owner object immediately and the garbage collector then deletes the dependents in the background. This is useful so as not to delay the main execution thread.

To delete a Spark job launched by spark-submit:

from kubernetes import client


core_v1_api = client.CoreV1Api()
core_v1_api.delete_namespaced_pod("driver-pod-name", "spark-jobs", propagation_policy="Background")

To delete a Spark job launched with the Spark Operator, we must delete the enclosing SparkApplication resource:

from kubernetes import client, config

custom_object_api = client.CustomObjectsApi()
custom_object_api.delete_namespaced_custom_object(
    group="sparkoperator.k8s.io",
    version="v1beta2",
    namespace="spark-jobs",
    plural="sparkapplications",
    name="app_name",
    propagation_policy="Background")

Pod priority & preemption

Whether it is Volcano or the default kube-scheduler, job preemption relies on job priorities. For two jobs, the scheduler decides whose priority is higher by comparing .spec.priorityClassName (then createTime).

The priority is propagated to driver and executor pods, whether with native spark-submit or with Spark Operator, and regardless of the node affinities.

How to know which pods have been preempted

You can retrieve high-level information on what is happening in the cluster. To list all events in the namespace spark-jobs you can use:

# List Events sorted by timestamp
kubectl get events --sort-by=.metadata.creationTimestamp --namespace=spark-jobs

LAST SEEN   TYPE      REASON             OBJECT                                                   MESSAGE
69s         Normal    Scheduled          podgroup/podgroup-6083b4f9-f240-4a0e-95f2-06882aec2942   pod group is ready
93s         Normal    Scheduled          pod/pyspark-pi-driver-routine-bf20cae50b6a8253           Successfully assigned spark-jobs/pyspark-pi-driver-routine-bf20cae50b6a8253 to gke-yippee-spark-k8s-clus-default-pool-0b72dd1d-jpfp
92s         Normal    Started            pod/pyspark-pi-driver-routine-bf20cae50b6a8253           Started container pyspark-pi
92s         Normal    Pulled             pod/pyspark-pi-driver-routine-bf20cae50b6a8253           Container image "eu.gcr.io/yippee-spark-k8s/spark-py:3.0.1" already present on machine
92s         Normal    Created            pod/pyspark-pi-driver-routine-bf20cae50b6a8253           Created container pyspark-pi
82s         Normal    Scheduled          pod/pythonpi-34b3597593d246a1-exec-2                     Successfully assigned spark-jobs/pythonpi-34b3597593d246a1-exec-2 to gke-yippee-spark-k8s-clus-default-pool-0b72dd1d-pvdl
82s         Normal    Scheduled          pod/pythonpi-34b3597593d246a1-exec-1                     Successfully assigned spark-jobs/pythonpi-34b3597593d246a1-exec-1 to gke-yippee-spark-k8s-clus-default-pool-0b72dd1d-wxck
82s         Normal    Created            pod/pythonpi-34b3597593d246a1-exec-1                     Created container spark-kubernetes-executor
82s         Normal    Started            pod/pythonpi-34b3597593d246a1-exec-1                     Started container spark-kubernetes-executor
82s         Normal    Pulled             pod/pythonpi-34b3597593d246a1-exec-1                     Container image "eu.gcr.io/yippee-spark-k8s/spark-py:3.0.1" already present on machine
81s         Normal    Created            pod/pythonpi-34b3597593d246a1-exec-2                     Created container spark-kubernetes-executor
81s         Normal    Pulled             pod/pythonpi-34b3597593d246a1-exec-2                     Container image "eu.gcr.io/yippee-spark-k8s/spark-py:3.0.1" already present on machine
80s         Normal    Started            pod/pythonpi-34b3597593d246a1-exec-2                     Started container spark-kubernetes-executor
42s         Normal    Killing            pod/pythonpi-34b3597593d246a1-exec-1                     Stopping container spark-kubernetes-executor
42s         Normal    Killing            pod/pythonpi-34b3597593d246a1-exec-2                     Stopping container spark-kubernetes-executor
42s         Warning   FailedScheduling   pod/pyspark-pi-driver-rush-bb25fc0b8efe7c4d              all nodes are unavailable: 3 node(s) resource fit failed.
42s         Normal    Killing            pod/pyspark-pi-driver-routine-bf20cae50b6a8253           Stopping container pyspark-pi
42s         Warning   Evict              pod/pyspark-pi-driver-routine-bf20cae50b6a8253           Pod is evicted, because of preempt
34s         Warning   Unschedulable      podgroup/podgroup-ee4b9210-35c2-4d68-841a-2daf7712a816   0/1 tasks in gang unschedulable: pod group is not ready, 1 Pipelined, 1 minAvailable.
41s         Warning   FailedScheduling   pod/pyspark-pi-driver-rush-bb25fc0b8efe7c4d              1/1 tasks in gang unschedulable: pod group is not ready, 1 Pipelined, 1 minAvailable.
18s         Normal    Scheduled          podgroup/podgroup-ee4b9210-35c2-4d68-841a-2daf7712a816   pod group is ready
33s         Normal    Scheduled          pod/pyspark-pi-driver-rush-bb25fc0b8efe7c4d              Successfully assigned spark-jobs/pyspark-pi-driver-rush-bb25fc0b8efe7c4d to gke-yippee-spark-k8s-clus-default-pool-0b72dd1d-jpfp
32s         Normal    Started            pod/pyspark-pi-driver-rush-bb25fc0b8efe7c4d              Started container pyspark-pi
32s         Normal    Created            pod/pyspark-pi-driver-rush-bb25fc0b8efe7c4d              Created container pyspark-pi
32s         Normal    Pulled             pod/pyspark-pi-driver-rush-bb25fc0b8efe7c4d              Container image "eu.gcr.io/yippee-spark-k8s/spark-py:3.0.1" already present on machine
22s         Normal    Scheduled          pod/pythonpi-f36cce7593d332f1-exec-1                     Successfully assigned spark-jobs/pythonpi-f36cce7593d332f1-exec-1 to gke-yippee-spark-k8s-clus-default-pool-0b72dd1d-wxck
22s         Normal    Scheduled          pod/pythonpi-f36cce7593d332f1-exec-2                     Successfully assigned spark-jobs/pythonpi-f36cce7593d332f1-exec-2 to gke-yippee-spark-k8s-clus-default-pool-0b72dd1d-pvdl
22s         Normal    Pulled             pod/pythonpi-f36cce7593d332f1-exec-1                     Container image "eu.gcr.io/yippee-spark-k8s/spark-py:3.0.1" already present on machine
21s         Normal    Started            pod/pythonpi-f36cce7593d332f1-exec-1                     Started container spark-kubernetes-executor
21s         Normal    Created            pod/pythonpi-f36cce7593d332f1-exec-1                     Created container spark-kubernetes-executor
21s         Normal    Pulled             pod/pythonpi-f36cce7593d332f1-exec-2                     Container image "eu.gcr.io/yippee-spark-k8s/spark-py:3.0.1" already present on machine
21s         Normal    Created            pod/pythonpi-f36cce7593d332f1-exec-2                     Created container spark-kubernetes-executor
21s         Normal    Started            pod/pythonpi-f36cce7593d332f1-exec-2                     Started container spark-kubernetes-executor

In the output above, we can see that the pod pyspark-pi-driver-routine-bf20cae50b6a8253 has been "evicted because of preempt" by another job with the "rush" priority.

Future work

Kubernetes provides containers with lifecycle hooks. The hooks enable containers to be aware of events in their management lifecycle and run code implemented in a handler when the corresponding lifecycle hook is executed.

In particular, the PreStop hook can be called immediately before a container is terminated due to preemption (among other events).
Thus, we can consider an action, whatever it is, to be triggered in case of preemption. All you need to do is implement and register a handler for this hook.

See Container Lifecycle Hooks.

We are soon coming to the end of our journey. Before wishing you good night, let's take a look at how we can monitor a Spark application from Python code.

Monitoring

Getting the status of a Spark application

Many operations in the Kubernetes Python client can be watched. This allows our Python program to watch for changes in a specific resource until you get the desired result or the watch expires.

Here, we want to monitor the lifecycle of the pod driver, starting in the Pending phase, moving through Running if the Spark container starts OK, and then through either the Succeeded or Failed phases:

from kubernetes import client, watch

app_name = 'pyspark-pi-routine-bf20cae50b6a8253'
v1 = client.CoreV1Api()

count = 2
w = watch.Watch()
label_selector = 'app-name=%s,spark-role=driver' % app_name
for event in w.stream(v1.list_namespaced_pod, namespace='spark-jobs', label_selector=label_selector, timeout_seconds=60):
    print('Event: %s' % event['object'].status.phase)
    count -= 1
    if not count:
        w.stop()

Here, the driver pod (hence, the Spark application) is expected to complete, successfully or not, within 60 seconds or less. Its status should only change twice during this period: ideally Pending > Running > Succeeded.

Getting the logs

It is possible to retrieve the logs of the driver pod and mix them into those of the host application.
Getting the logs is just as easy:

from kubernetes import client, watch
from threading import Thread

v1 = client.CoreV1Api()
pod_name = 'pyspark-pi-routine-bf20cae50b6a8253-driver'

def logs(pod_name):
    w = watch.Watch()
    for event in w.stream(v1.read_namespaced_pod_log, pod_name=pod_name, namespace='spark-jobs', _request_timeout=300):
        yield event

# We surely don't want to block the main thread while reading the logs
def consumer():
    for log in logs(pod_name):
        print(log)

t = Thread(target=consumer)
t.start()

Getting the Ingress URI

Once a Spark application has started, the ingress (at least, its public IP address) that exposes the Spark UI may take a while before it becomes available. Here too, we can monitor the Ingress resource as follows:

from kubernetes import client, watch

app_name = 'pyspark-pi-routine-bf20cae50b6a8253'

networking_v1_beta1_api = client.NetworkingV1beta1Api()
w = watch.Watch()
label_selector = 'app-name=%s' % app_name
for event in w.stream(networking_v1_beta1_api.list_namespaced_ingress, namespace=namespace,
                      label_selector=label_selector,
                      timeout_seconds=30):
    ingress = event['object'].status.load_balancer.ingress
    if ingress:
        external_ip = ingress[0].ip
        print('Event: The Spark Web UI is available at http://%s/%s' % (external_ip, app_name))
        w.stop()
    else:
        print('Event: Ingress not yet available')

Conclusion

What a hell of a journey!

We have seen that the Python code for launching or deleting Spark applications is slightly different depending on whether we are using the Spark Operator or spark-submit. But since we name and label the Kubernetes objects consistently between the two, and as we set the ownership relationships properly, we can monitor our Spark applications and manage their lifecycle equally.

Would you like to know more? The Python scripts explained in this last article are available in this GitHub repository. Serve yourself.

My Journey With Spark On Kubernetes... In Python (2/3)

Pascal Gillet — Mon, 12 Apr 2021 08:10:36 +0000

In the previous article, we saw how to launch Spark applications with the Spark Operator. In this article, we'll see how to do the same thing, but natively with spark-submit. Let's first explain the differences between the two ways of deploying your driver on the worker nodes.

Client vs Cluster Mode

Spark-submit in cluster mode

In cluster mode, your application is submitted from a machine far from the worker machines (e.g. locally on your laptop). You need a Spark distribution installed on this machine to be able to actually run the spark-submit script.
In this mode, the script exits normally as soon as the application has been submitted. The driver is then detached and can run on its own in the kubernetes cluster. You can print the logs of the driver pod with the kubectl logs command to see the output of the application.

It is possible to use the authenticating kubectl proxy to communicate to the Kubernetes API.

The local proxy can be started by:

kubectl proxy &

If the local proxy is running at localhost:8001, the remote Kubernetes cluster can be reached by spark-submit by specifying --master k8s://http://127.0.0.1:8001 as an argument to spark-submit.

Spark-submit in client mode

In client mode, the spark-submit command is directly passed with its arguments to the Spark container in the driver pod. With the deploy-mode option set to client, the driver is launched directly within the spark-submit process which acts as a client to the cluster. The input and output of the application are attached to the logs from the pod.

Who Does What?

With "native" Spark, we will execute Spark applications in client mode, so as not to depend on a local Spark distribution. Specifically, the user creates a driver pod resource with kubectl, and the driver pod will then run spark-submit in client mode internally to run the driver program.

With Spark Operator, a SparkApplication should set .spec.deployMode to cluster, as client is not currently implemented. Behind the scenes, the behavior is exactly the same as with native Spark: the operator's controller thus embeds a Spark distribution which plays the role of Spark scheduler; driver pods are spawned from the controller... and then run spark-submit in client mode internally to run the driver program. But this is globally transparent for the end user.

Additional details of how SparkApplications are run can be found in the design documentation.

Driver Pod

With native Spark, the main resource is the driver pod.
To run the Pi example program like with the Spark Operator, the driver pod must be created using the data in the following YAML file:

pyspark-pi-driver-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  labels:
    app-name: pyspark-pi-${PRIORITY_CLASS_NAME}${NAME_SUFFIX}
    spark-role: driver
  name: pyspark-pi-${PRIORITY_CLASS_NAME}${NAME_SUFFIX}-driver
  namespace: ${NAMESPACE}
spec:
  containers:
  - name: pyspark-pi
    image: eu.gcr.io/yippee-spark-k8s/spark-py:3.0.1
    imagePullPolicy: IfNotPresent
    ports:
    - containerPort: 5678
      name: headless-svc
    - containerPort: 4040
      name: web-ui
    resources:
      requests:
        memory: 512Mi
        cpu: 1
      limits:
        cpu: 1200m
    env:
    # Overriding configuration directory
    - name: SPARK_CONF_DIR
      value: /spark-conf
    - name: SPARK_HOME
      value: /opt/spark
    # Configure all key-value pairs in ConfigMap as container environment variables
    envFrom:
      - configMapRef:
          name: pyspark-pi-${PRIORITY_CLASS_NAME}${NAME_SUFFIX}-cm
    args:
    - $(SPARK_HOME)/bin/spark-submit
    - /opt/spark/examples/src/main/python/pi.py
    - "10"
    volumeMounts:
      - name: spark-config
        mountPath: /spark-conf
        readOnly: true
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
            - key: type
              operator: In
              values: [${DRIVER_NODE_AFFINITIES}]
  priorityClassName: ${PRIORITY_CLASS_NAME}
  restartPolicy: OnFailure
  schedulerName: volcano
  serviceAccountName: ${SERVICE_ACCOUNT_NAME}
  volumes:
    # Add the executor pod template in read-only volume, for the driver to read
    - name: spark-config
      configMap:
        name: pyspark-pi-${PRIORITY_CLASS_NAME}${NAME_SUFFIX}-cm
        items:
        - key: spark-defaults.conf
          path: spark-defaults.conf
        - key: spark-env.sh
          path: spark-env.sh
        - key: executor-pod-template.yaml
          path: executor-pod-template.yaml

Don't pay attention to the variable placeholders for now, even though you can imagine what they can be used for.

As you can see, the driver pod depends on other Kubernetes resources. We will detail each of them.

Spark Configuration

We use a ConfigMap for setting Spark configuration data separately from the driver pod definition.

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app-name: spark-${PRIORITY_CLASS_NAME}${NAME_SUFFIX}
  name: spark-${PRIORITY_CLASS_NAME}${NAME_SUFFIX}-cm
  namespace: ${NAMESPACE}
data:
  # Comma-separated list of .zip, .egg, or .py files dependencies for Python apps.
  # spark.submit.pyFiles can be used instead in spark-defaults.conf below.
  # PYTHONPATH: ...
  spark-env.sh: |
    #!/usr/bin/env bash

    export DUMMY=dummy
    echo "Here we are, ${DUMMY}!"
  spark-defaults.conf: |
    spark.master                                k8s://https://kubernetes.default
    spark.submit.deployMode                     client

    spark.executor.instances                    2
    spark.executor.cores                        1
    spark.executor.memory                       512m

    spark.kubernetes.executor.container.image   eu.gcr.io/yippee-spark-k8s/spark-py:3.0.1
    spark.kubernetes.container.image.pullPolicy IfNotPresent
    spark.kubernetes.namespace                  spark-jobs
    # Must match the mount path of the ConfigMap volume in driver pod
    spark.kubernetes.executor.podTemplateFile   /spark-conf/executor-pod-template.yaml
    spark.kubernetes.pyspark.pythonVersion      2
    spark.kubernetes.driver.pod.name            spark-${PRIORITY_CLASS_NAME}${NAME_SUFFIX}-driver

    spark.driver.host                           spark-${PRIORITY_CLASS_NAME}${NAME_SUFFIX}-driver-svc
    spark.driver.port                           5678

    # Config params for use with an ingress to expose the Web UI
    spark.ui.proxyBase                          /spark-${PRIORITY_CLASS_NAME}${NAME_SUFFIX}
    # spark.ui.proxyRedirectUri                   http://<load balancer static IP address>
  executor-pod-template.yaml: |
    apiVersion: v1
    kind: Pod
    metadata:
      labels:
        app-name: spark-${PRIORITY_CLASS_NAME}${NAME_SUFFIX}
        spark-role: executor
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: type
                operator: In
                values: [${EXECUTOR_NODE_AFFINITIES}]
      priorityClassName: ${PRIORITY_CLASS_NAME}
      schedulerName: volcano

Executor Pod Template

We use a template file in the ConfigMap to define the executor pod configuration. Templates are typically used for configuring Spark pods that cannot be configured otherwise, via Spark properties or environment variables. Thus, template files mostly contain fine-grained configuration related to deployment at the Kubernetes level: here, the node affinity and the priority class name.

To make the pod template file accessible to the spark-submit process, we must set the Spark property spark.kubernetes.executor.podTemplateFile with its local pathname in the driver pod. To do so, the file will be automatically mounted onto a volume in the driver pod when it’s created.

Executor Pod Garbage Collection

We must also set spark.kubernetes.driver.pod.name for the executors to the name of the driver pod. When this property is set, the Spark scheduler will deploy the executor pods with an ownerReference, which in turn will ensure that once the driver pod is deleted from the cluster, all of the application’s executor pods will also be deleted.

Client Mode Networking

In client mode, the driver runs inside a pod. Spark executors must be able to connect to the Spark driver by the means of Kubernetes networking. To do that, we use a headless service to allow the driver pod to be routable from the executors by a stable hostname. When deploying the headless service, we ensure that the service will only match the driver pod and no other pods by assigning the driver pod a (sufficiently) unique label and by using that label in the label selector of the headless service. We can then pass to the executors the driver’s hostname via spark.driver.host with the service name and the spark driver’s port to spark.driver.port.

spark-driver-svc.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    app-name: spark-${PRIORITY_CLASS_NAME}${NAME_SUFFIX}
  name: spark-${PRIORITY_CLASS_NAME}${NAME_SUFFIX}-driver-svc
  namespace: spark-jobs
spec:
  clusterIP: None
  ports:
  - name: 5678-5678
    port: 5678
    protocol: TCP
    targetPort: 5678
  selector:
    app-name: spark-${PRIORITY_CLASS_NAME}${NAME_SUFFIX}
    spark-role: driver
  type: ClusterIP

Spark UI

Ingress

The Spark UI is accessible by creating a service of type ClusterIP which exposes the UI from the driver pod:

spark-ui-svc.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    app-name: spark-${PRIORITY_CLASS_NAME}${NAME_SUFFIX}
  name: spark-${PRIORITY_CLASS_NAME}${NAME_SUFFIX}-ui-svc
  namespace: spark-jobs
spec:
  ports:
  - name: spark-driver-ui-port
    port: 4040
    protocol: TCP
    targetPort: 4040
  selector:
    app-name: spark-${PRIORITY_CLASS_NAME}${NAME_SUFFIX}
    spark-role: driver
  type: ClusterIP

With this service alone, the UI is only accessible from inside the cluster. We must then create an Ingress to expose the UI outside the cluster. In order for the Ingress resource to work, the cluster must have an ingress controller
running. You can choose the ingress controller implementation that best fits your cluster.
Traefik and Nginx are very popular choices. The ingress below is configured for Nginx:

spark-ui-ingress.yaml

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  labels:
    app-name: spark-${PRIORITY_CLASS_NAME}${NAME_SUFFIX}
  name: spark-${PRIORITY_CLASS_NAME}${NAME_SUFFIX}-ui-ingress
  namespace: spark-jobs
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/proxy-redirect-from: "http://$host/"
    nginx.ingress.kubernetes.io/proxy-redirect-to: "http://$host/spark-${PRIORITY_CLASS_NAME}${NAME_SUFFIX}/"
spec:
  rules:
  - http:
      paths:
      - path: /spark-${PRIORITY_CLASS_NAME}${NAME_SUFFIX}(/|$)(.*)
        backend:
          serviceName: spark-${PRIORITY_CLASS_NAME}${NAME_SUFFIX}-ui-svc
          servicePort: 4040

The Ingress is backed by as many services as driver pods that run concurrently in the cluster. Each UI must then be addressed with a unique path in the Ingress. The path in question is directly derived from the name of the Spark application, with its unique ID.

The interface is always available at http://<driver-pod-ip>:4040 (from inside the cluster), as each driver pod is assigned a unique IP and there is only one SparkContext running in the pod host (otherwise, the UI would be bound to successive ports beginning with 4040: 4041, 4042, etc).

So, natively, the different UIs that might be available at a given time on a same host are only differentiated by their assigned port number. The UI has been designed following that principle, and all HTTP operations are performed relatively to the same root path in the URL, which is.../.

So for the UI to be effectively accessible through the Ingress, we must set up HTTP redirection with an alternative root path (the one configured in the Ingress). To work smoothly, the UI itself must be aware of this redirection by setting spark.ui.proxyBase to this root path...and that's it!

Spark Operator

The operator supports creating an optional Ingress for the Spark Web UI. This can be turned on by setting the ingress-url-format command-line flag. The ingress-url-format should be a template like {{$appName}}.{ingress_suffix}/{{$appNamespace}}/{{$appName}}. The {ingress_suffix} should be replaced by the user to indicate the cluster's Ingress url and the operator will replace the {{$appName}} and {{$appNamespace}} with the appropriate value. Please note that Ingress support requires that cluster's Ingress url routing is correctly setup. For e.g. if the ingress-url-format is {{$appName}}.ingress.cluster.com, it requires that anything *.ingress.cluster.com should be routed to the Ingress controller on the K8s cluster.

Example of Spark Operator install with the ingress-url-format command-line flag:

./helm install spark-operator incubator/sparkoperator --namespace spark-operator --set enableWebhook=true --set enableBatchScheduler=true --set ingressUrlFormat="\{\{\$appName\}\}.ingress.cluster.com"

Note that curly braces must be escaped.

Future Work

I could not enable the Spark Operator Ingress during my experiments, simply because a DNS name is required. Instead, the same Ingress as for native Spark is "grafted" to the SparkApplication, with path-based routing.

The operation proposed by the Spark Operator, with routing based on hostname wildcards (for example *.ingress.cluster.com), is nevertheless interesting as it would overcome the problem of HTTP redirect described above.

With hostname wildcards, and therefore without the HTTP redirect, the UI service could be switched to NodePort type (a NodePort service exposes the Service on each Node's IP at a static port) and still be compatible with the Ingress.
The UI would thus be accessible outside the cluster through both the Ingress with its external URL configured, and the NodePort service at http://<node-ip>:<service-port>. A service of type NodePort is still relevant in a private cloud.

Object Names and Labels

We use the app-name label to semantically group all the Kubernetes resources related to a single Spark application.
In our case, this label provides uniqueness, and we do not expect multiple Spark applications to carry the same value for this label (at least in the namespace-time considered).

For a given Spark application, all the object names are then derived from the app-name. We simply add a suffix which also qualifies the type of the object: -driver for the driver pod, -driver-svc for the driver service, -ui-svc for the Spark UI service, -ui-ingress for the Spark UI ingress, and -cm for the ConfigMap.

We also set the label spark-role at the pod level to differentiate the drivers from their executors.

This naming and labeling is consistent with the Spark Operator. As a result, Spark applications can be treated and filtered in the same way, whether they are launched with the Spark Operator or with spark-submit. 👍

Deployment

We have now managed to mimic the same behavior we got with the Spark Operator, here is what we have deployed in Kubernetes:

Now that we've defined all of our Kubernetes resources for spark-submit, we're going to get our hands on some Python code to orchestrate all of this. See you in the third and final article in this series.

My Journey With Spark On Kubernetes... In Python (1/3)

Pascal Gillet — Mon, 12 Apr 2021 08:09:38 +0000

Je vous parle d'un temps
Que les moins de vingt ans
Ne peuvent pas connaître 🎶

Until not long ago, the way to go to run Spark on a cluster was either with Spark's own standalone cluster manager, Mesos or YARN. In the meantime, the Kingdom of Kubernetes has risen and spread widely.

And when it comes to run Spark on Kubernetes, you now have two choices:

Use "native" Spark's Kubernetes capabilities: Spark can run on clusters managed by Kubernetes since Spark 2.3. Kubernetes support was still flagged as experimental until very recently, but as per SPARK-33005 Kubernetes GA Preparation, Spark on Kubernetes is now fully supported and production ready! 🎊
Use the Spark Operator, proposed and maintained by Google, which is still in beta version (and always will be).

This series of 3 articles tells the story of my experiments with both methods, and how I launch Spark applications from Python code.

"Cabin crew, arm doors and cross check". Let's go! ✈️

Prerequisites

Service Account for Driver Pods

Remember, Spark applications run as independent sets of processes on a cluster, coordinated by the SparkContext object in your main program, called the driver. Once connected, the SparkContext acquires executors on nodes in the cluster, which are the processes that run computations and store data for your application.

Thus, Spark driver pods need a Kubernetes service account in the pod's namespace that has permissions to create, get, list, and delete executor pods. Below an example RBAC setup that creates a driver service account named driver-sa in the namespace spark-jobs, with a RBAC role binding giving the service account the needed permissions.

k8s/driver-sa-rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: driver-sa
  namespace: spark-jobs
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: spark-jobs
  name: spark-role
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["*"]
- apiGroups: [""]
  resources: ["services"]
  verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: spark-role-binding
  namespace: spark-jobs
subjects:
- kind: ServiceAccount
  name: driver-sa
  namespace: spark-jobs
roleRef:
  kind: Role
  name: spark-role
  apiGroup: rbac.authorization.k8s.io

kubectl create namespace spark-jobs
kubectl create -f k8s/driver-sa-rbac.yaml

Node Affinity

By default, the scheduler automatically places pods on nodes by ensuring nodes have sufficient free resources, distributing pods evenly across nodes, etc.
But there are circumstances where you may want more control on a node where a pod lands, for example to ensure that a pod ends up on a memory or compute-optimized machine, or with an SSD attached to it.
In our case, Spark executors need more resources than drivers. We thus need to constrain driver pods and executor pods to only be able to run on particular node(s). We will use Node Affinities with label selectors to make the selection.

Execute the following command for the node(s) intended to execute driver pods:

kubectl label nodes <node-name> type=driver

For executor pods:

kubectl label nodes <node-name> type=compute

Pod Priority and Preemption

In my project, we aim to run multiple Spark jobs simultaneously in parallel. But some workloads have higher priority than others. If a job cannot be scheduled, the scheduler (here, Volcano) tries to preempt (evict) lower priority Pods to make scheduling of the pending Pod possible.
To use priority and preemption capabilities, we must first create the necessary PriorityClasses:

k8s/priorities.yaml

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: routine
value: 2
preemptionPolicy: Never
globalDefault: false
description: "Routine priority"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: urgent
value: 10
preemptionPolicy: Never
globalDefault: false
description: "Urgent priority"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: exceptional
value: 50
preemptionPolicy: Never
globalDefault: false
description: "Exceptional priority"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: rush
value: 100
preemptionPolicy: PreemptLowerPriority
globalDefault: false
description: "Rush priority"

kubectl create -f k8s/priorities.yaml

Here, only the PriorityClass "rush" is allowed to preempt lower-priority pods. Pods with other priorities will be placed in the scheduling queue ahead of lower-priority pods, but they cannot preempt other pods. They'll just have to wait until sufficient resources are free to be scheduled.

Volcano Scheduler

For our experiments, we will use Volcano which is a batch scheduler for Kubernetes, well-suited for scheduling Spark applications pods with a better efficiency than the default kube-scheduler.
The main reason is that Volcano allows "group scheduling" or "gang scheduling": while the default scheduler of Kubernetes schedules containers one by one, Volcano ensures that a gang of related containers (here, the Spark driver and its executors) can be scheduled at the same time. If for any reason it is not possible to deploy all the containers in a gang, Volcano will not schedule that gang.
This article explains in more detail the reasons for using Volcano.

Install Volcano

Install Volcano on the cluster:

kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/master/installer/volcano-development.yaml

Enable job preemption

The preempt action is responsible for preemptive scheduling of high priority tasks in the same queue according to priority rules.

This action is disabled by default in Volcano. To enable job preemption, edit the Volcano configuration as follows:

kubectl edit configmap volcano-scheduler-configmap --namespace volcano-system

And add preempt to the list of actions:

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  volcano-scheduler.conf: |
    actions: "enqueue, allocate, preempt, backfill"
    tiers:
    - plugins:
      - name: priority
      - name: gang
      - name: conformance
    - plugins:
      - name: drf
      - name: predicates
      - name: proportion
      - name: nodeorder
      - name: binpack
kind: ConfigMap
metadata:
  ...

Note that job preemption in Volcano relies on the priority plugin that compares the priorities of two jobs or tasks. For two jobs, it decides whose priority is higher by comparing job.spec.priorityClassName. For two tasks, it decides whose priority is higher by comparing task.priorityClassName, task.createTime, and task.id in order.

Enable Volcano scheduling in your workload

For your workload to be scheduled by Volcano, you just need to set schedulerName: volcano in your pod's spec (or batchScheduler: volcano in the SparkApplication's spec if you use the Spark Operator). By default, the workload is scheduled with the default kube-scheduler.

To be consistent, we will ensure that the same scheduler is used for driver and executor pods.

Spark In Docker

To run Spark applications on Kubernetes ... ehh ... You need a Docker image that embeds a Spark distribution.
This section explains how to build an "official" Spark Docker image and how to run a basic Spark application with it.

Spark Docker image

Spark (starting with version 2.3) ships with Dockerfiles that can be used to build different Spark Docker images (and customize them to match an individual application’s needs) to use with a Kubernetes backend. They can be found in the kubernetes/dockerfiles/ directory.

Spark also ships with a bin/docker-image-tool.sh script that eases the building (and the publishing) of the Docker images.

Example usage to build an image with the Python binding (PySpark):

./bin/docker-image-tool.sh -t <tag> -p ./kubernetes/dockerfiles/spark/bindings/python/Dockerfile build

This will create a local Docker image named spark-py:<tag>. You can set the <tag> with the Spark's actual version.

Running the Examples

Once the image is bundled, you can launch a Spark application using the bin/spark-submit script.

# Run Spark locally with two worker threads
docker run --rm spark-py:3.0.1 /opt/spark/bin/spark-submit --master local[2] /opt/spark/examples/src/main/python/pi.py

Recommendation

It is strongly recommended starting from an official base image to create any custom Spark image.

Spark Operator

Configuring and installing the Kubernetes Operator for Apache Spark

In this section, you use Helm to deploy the Kubernetes Operator for Apache Spark from the incubator Chart repository. Helm is a package manager you can use to configure and deploy Kubernetes apps.

Install Helm

Download and install the Helm binary:

wget https://get.helm.sh/helm-v3.3.4-linux-amd64.tar.gz

Unzip the file to your local system:

tar zxfv helm-v2.12.3-linux-amd64.tar.gz
cp linux-amd64/helm .

Ensure that Helm is properly installed by running the following command:

./helm version

If Helm is correctly installed, you should see the following output:

version.BuildInfo{Version:"v3.3.4", GitCommit:"a61ce5633af99708171414353ed49547cf05013d", GitTreeState:"clean", GoVersion:"go1.14.9"}

Install the chart

helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator
kubectl create namespace spark-operator
helm install spark-operator incubator/sparkoperator --namespace spark-operator --set enableWebhook=true --set enableBatchScheduler=true

The flag enableBatchScheduler=true enables Volcano. To install the operator with Vocano enabled, you must also install the mutating admission webhook with the flag enableWebhook=true.

Now you should see the operator running in the cluster by checking the status of the Helm release:

./helm status spark-operator --namespace spark-operator

About the Spark Job Namespace and the Service Account for Driver Pods

We did not set a specific value for the Helm chart property sparkJobNamespace when installing the operator, that means the Spark Operator supports deploying SparkApplications to all namespaces.
As a consequence, the Spark Operator did not automatically create the service account for driver pods, and we must set up the RBAC for driver pods of our SparkApplications to be able to manipulate executor pods in a specific namespace.

See About the Spark Job Namespace and About the Service Account for Driver Pods sections for more details.

Running the Examples

To run the Spark Pi example provided within the operator, run the following command:

kubectl apply -f examples/spark-py-pi.yaml

Spark-submit vs Spark Operator

Let's take a closer look at the Pi example from the Spark Operator. A single YAML file is needed, adapted to our configuration: .metadata.namespace must be set to "spark-jobs" and .spec.driver.serviceAccount is set to the name of the service account "driver-sa" previously created.

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: pyspark-pi
  namespace: spark-jobs
spec:
  batchScheduler: volcano
  batchSchedulerOptions:
    priorityClassName: routine
  type: Python
  pythonVersion: "2"
  mode: cluster
  image: "gcr.io/spark-operator/spark-py:v3.0.0"
  imagePullPolicy: Always
  mainApplicationFile: local:///opt/spark/examples/src/main/python/pi.py
  arguments:
  - "10"
  sparkVersion: "3.0.0"
  restartPolicy:
    type: OnFailure
    onFailureRetries: 3
    onFailureRetryInterval: 10
    onSubmissionFailureRetries: 5
    onSubmissionFailureRetryInterval: 20
  timeToLiveSeconds: 86400
  driver:
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: type
              operator: In
              values: [driver]
    cores: 1
    coreLimit: "1200m"
    memory: "512m"
    labels:
      version: 3.0.0
    serviceAccount: driver-sa
  executor:
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: type
              operator: In
              values: [compute]
    cores: 1
    instances: 2
    memory: "512m"
    labels:
      version: 3.0.0

Pretty simple, right!?

The Spark Operator aims to make specifying and running Spark applications in a cloud-native way and as easy and idiomatic as running other workloads on Kubernetes. It uses Kubernetes custom resources for specifying, running, and monitoring Spark applications.

With the high-level resource SparkApplication, the operator greatly reduces the boilerplate YAML configuration files and takes care of all the needed plumbing for you: networking between the driver and its executors, garbage collection, pod configuration, access to the driver UI.

The following diagram shows what is actually deployed in Kubernetes under the hood:

In use, the operator is way much easier than spark-submit. But spark-submit is definitely not going away and is still the Spark native way of launching applications. "In the long term, for application submission, the operator will not semantically nor functionally diverge from spark-submit and will always use it under the hood". More importantly, "the spark-submit script use all of Spark’s supported cluster managers through a uniform interface so you don’t have to configure your application especially for each one" (see here). Still, that shouldn't prevent the Apache Spark project from developing its own operator in my opinion.

Eventually, choosing between the Spark Operator and spark-submit is a matter of if you are more Kubernetes-centric and you run Spark workloads among other types of workloads, or you do Spark first, and Kubernetes is just a mean to allocate resources on a cluster.

In the following article, we will see how the magic of the Spark Operator operates, by reproducing all of its internals with spark-submit.

Asguard, a security solution for bringing sensitive code into the Cloud

Pascal Gillet — Tue, 02 Jun 2020 09:31:17 +0000

This article presents a security solution implemented as part of a project led by Stack Labs on behalf of one of our customers in the space sector.
The purpose of this project was to migrate a legacy application to Google Cloud (Lift & Shift). It is a sensitive application, as its source code is the fruit of a real savoir-faire in satellite imagery analysis, with algorithms developed in-house over many years.

The application is protected by a legal framework which aims to protect access to the knowledge and technologies of the organization that requests it. Conversely, and as far as I understand, the said organization can be prosecuted if it does not put in place the necessary measures to protect their assets, like the access control, physical or digital, to sensitive information.

Context

Our customer brings a lot of its activities into public cloud platforms and plans to do the same with this application, but in this particular case, he needed more security guarantees.

Yes, the cloud can be an important source of security risks. The cloud's responsibility for security essentially extends to the services provided, but not to the hosted applications. Thus, cloud providers already offer integrated security solutions at storage, infrastructure and network levels, but these may not be sufficient due to intellectual property or local regulations. The OWASP, in its "Cloud Top 10 Security Risks", reveals that data ownership and regulatory compliance are respectively the 1st and 3rd highest risks when migrating into the cloud. Businesses need additional guarantees and the ability to extend, or even replace, these security solutions to the applications themselves.

Here, our application is originally a desktop application. The long-term goal is to servicize it and use it as a main “shared library” among the satellite imagery services delivered to our customer's customers. The application is also a plain old good monolith, and will be broken later into microservices. The first project milestone was to take this monolith to the cloud and attach a REST API facade to it, and above all to put the security solution in place, and to test it.

Security requirements

The security requirements are the following:

Execution control: the application must be “enclaved”, that means the application can only be loaded and executed inside a trusted execution environment, here in this case a GCP project. The containerized application is basically encrypted at build time, then decrypted at runtime, with the help of Vault from Hashicorp. It should not be possible to decrypt the application outside of this execution context. This article is mainly about how we fulfilled this requirement.
Code obfuscation: protection against reverse engineering. As a last bastion of security, and in the event that an attacker manages to grab the image, decrypt it and inspect it, the application source code must be obfuscated, so as to make it difficult for humans to understand.
Access control: this is more classic. The application will expose a REST API. Access control is provided by an in-house API gateway - a server program which is the only possible entry point and acts as an API front-end, receives API requests, passes requests to the back-end service(s) and then passes the response back to the requester - that supports authentication, authorization, and security audit. The API gateway is used by many projects within our customer's organization and is an internal project by itself.

"The application can only be loaded and executed inside a trusted execution environment. A user should not be able to inspect the contents of the application outside of this execution context."

A last requirement, but not least, is to use only open source tools to implement this security solution, and to be cloud agnostic (you know, to change the provider if necessary).

Secured CI/CD

We set up a CI/CD pipeline for building, securing and deploying the application into the targeted GCP project. Vault from Hashicorp is the corner stone of this workflow. Vault provides secrets management, and encryption of application data (without storing it). In particular, you can generate, use, rotate, and destroy AES256, RSA 2048, RSA 4096, EC P256, EC P384 and EC P521 cryptographic keys. Vault provides a Web interface, as well as a CLI and a HTTP API, to interact with it. It can be viewed as "encryption as a service".

The CI/CD pipeline, implemented with Gitlab CI/CD, is as follows:

The steps are numbered and detailed below:

The source code is checked out, built and obfuscated.
We create (first time) or rotate (successive times) a "key encryption key" (KEK) in Vault. Here, we authenticate to Vault using a simple token, stored in a Gitlab protected environment variable.
Envelope encryption: we implement a key hierarchy with a local "data encryption key" (DEK), protected by the KEK.
We make an archive out of the application, and we use the DEK to encrypt it using OpenSSL or GnuPG locally.
We store the encrypted application in a Docker container image, along with the encrypted DEK and a bootstrap bash script that will be used at runtime to decrypt and start running the application.
We push the image in Google Container Registry.
Self-starting process. The bootstrap script is the entrypoint of the Docker image. The workloads are orchestrated with Google Kubernetes Engine.
Here, we authenticate to Vault with a method specific to Google Cloud, detailed below.
We decrypt the DEK, by giving to Vault the ciphertext and the name of the KEK to decrypt against.
We decrypt the archive locally with OpenSSL or GnuPG, and we unpack the application.
The application can finally do the job for which it is intended.

You may ask why we create the "intermediate" DEK (step 3) to encrypt the application (step 4) when we could directly encrypt the application with the KEK in the first place. This would require to send the whole application to Vault over the network. We would therefore take the risk of exposing the application in transit, even if TLS is enabled, and above all, the application can be heavy.

So let's remember these two important definitions:

Data encryption key (DEK): is an encryption key whose function is to encrypt and decrypt data. The DEK is meant to be downloaded to encrypt the application locally.
Encryption key (KEK): is an encryption key whose function is to encrypt and decrypt the DEK. The KEK never leaves Vault.

Vault Auth methods

Authentication in Vault is the process of verifying that the system that logs in is which it says it is, and within a particular context. Vault supports multiple auth methods including Github, LDAP, AWS, Azure, Google Cloud, and more.

Before a client can interact with Vault, it must authenticate against a chosen auth method. Once authenticated, a token is granted to the client, like a session ID, for subsequent exchanges with Vault.

In steps 2 and 8 above, we use two different methods to authenticate to Vault:

At built time, we authenticate using the built-in token auth method (a simple password generated by Vault). This is a "shortcut" method, as if we were authenticating directly with the session ID mentioned above.
At runtime, we authenticate with a method specific to Google Cloud, and this is precisely what will allow us to ensure that we are executing our application in our GCP project, and not elsewhere.

Vault GCE Auth workflow

The gcp auth method allows Google Cloud Platform entities to authenticate to Vault. Vault treats Google Cloud as a trusted third party and verifies authenticating entities against the Google Cloud APIs. This Vault backend allows for authentication of Google Cloud IAM service accounts, but here we are interested by authenticating Google Compute Engine (GCE) instances that make up our Kubernetes cluster.

The auth method must be configured in advance before machines can authenticate. In particular, we must declare that only the machines in the targeted GCP project, in certain zones and with certain labels will be able to authenticate.

Also, the GCE instances that are authenticating against Vault must have the following role:
roles/iam.serviceAccountTokenCreator

In order to check the authentication process, Vault itself must authenticate to the Google Cloud Platform. For that, we specify the credentials of a service account with the minimum scope https://www.googleapis.com/auth/cloud-platform
and the role roles/compute.viewer or a custom role with the following exact permissions:

iam.serviceAccounts.get
iam.serviceAccountKeys.get
compute.instances.get
compute.instanceGroups.list
compute.instanceGroups.listInstances

These allow Vault to:

Verify the service account, either directly authenticating or associated with authenticating GCE instance, exists
Get the corresponding public keys for verifying JWTs signed by service account private keys.
Verify authenticating GCE instances exist
Compare bound fields for GCE roles (zone/region, labels, or membership in given instance groups)

The following diagram shows how Vault communicates with Google Cloud to authenticate and authorize JWT tokens:

The client obtains an instance identity metadata token on a GCE instance.
The client sends this JWT to Vault along with a role name.
Vault extracts the kid header value, which contains the ID of the key-pair used to generate the JWT, to find the OAuth2 public cert to verify this JWT.
Vault authorizes the confirmed instance against the given role, ensuring the instance matches the bound zones, regions, or instance groups. If that is successful, a Vault token with the proper policies is returned.

See https://www.vaultproject.io/docs/auth/gcp/ for even more details.

Putting the pieces together

Now that we have the theory, it is time to assemble things. As an example, our target application is a very basic web server in Python, serving files relative to its current directory:

import SimpleHTTPServer
import SocketServer

PORT = 8000

Handler = SimpleHTTPServer.SimpleHTTPRequestHandler

httpd = SocketServer.TCPServer(("", PORT), Handler)

print "serving at port", PORT
httpd.serve_forever()

In particular, it will serve the following index.html page:

The goal of our demo will be to modify this index page, commit our changes, and create a git tag to see the CI automatically triggered and our new version deployed, and see in the meantime what are the security elements that have been applied.

Code obfuscation

Code obfuscation is done by using a command line tool. A variety of tools exist to perform or assist with code obfuscation, and there is no such thing as a universal tool for obfuscation, as it depends of the architecture and the characteristics of the language.

We use PyArmor to obfuscate our Python script above, and this will be typically the result:

from pytransform import pyarmor_runtime
pyarmor_runtime()
__pyarmor__(__name__, __file__, b'\x50\x59\x41\x52\x4d\x4f\x52\x00\x00\x03\x05\x00\x17\x0d\x0d\x0a\x02\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x40\x00\x00\x00\x97\x01\x00\x00\x00\x00\x00\x18\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xe0\x50\x8c\x64\x26\x42\xd6\x01\xf3\x0d\x68\x2c\x25\x9f\x19\xaa\x05\x4c\x7b\x3c\x45\xce\x28\xdf\x84\xef\xd3\x8e\xed\xac\x78\x4b\x2c\xc5\x39\x30\xcf\xc1\x8b\x68\x89\x12\x6a\xe2\x4a\x89\x4c\x3a\xf8\xbd\x5d\xb2\xc9\xea\xf8\x6e\x38\xa4\x87\x32\xa3\x44\xe1\x0c\x7c\x1c\x75\x30\xc0\xe7\x58\xe5\xad\x5b\xe6\xaa\x02\x76\x03\x59\x08\xc1\x4f\x69\x0f\xee\xf7\x42\xc3\x9c\x6f\xb0\x82\x0f\x2a\x55\x34\x6b\x4d\x3a\xd6\xb6\xae\xdf\x0b\x3c\xf9\x6f\xde\xef\x0f\x90\xd6\x75\x3b\xe8\xc1\xc8\x42\x2b\xb0\x5f\x54\x1d\xd9\xee\xcd\x5d\xa2\x09\xcb\x7b\x5c\xd5\x26\xc8\xf7\x00\x4e\xd3\x3b\xc7\x44\xca\x94\xa6\xdf\x00\x83\x59\x2d\x7b\x75\xf0\x03\x9e\xae\xe2\x79\x8d\xaa\xa0\xbf\x3b\xc0\xc1\x7b\xaa\x84\x7a\xc7\xcb\x33\x82\x54\x5a\x6a\x25\xea\x30\xc4\x53\x0b\x01\xd6\x36\x67\x28\xdb\x80\x19\xe6\x1a\x81\x76\x4a\xb4\x98\xef\xff\xaa\x25\xaa\xe2\x81\x93\x05\xf2\x78\x80\xc6\xc7\x76\x66\xf0\x44\xe0\x15\xec\xe4\xe2\x40\x19\x80\x72\xc2\x0b\x56\x8e\x5e\xe5\x8c\x6a\x14\x2c\x55\x2a\xc8\xdc\x7a\x3b\x69\x3c\x56\x12\xd0\x8a\xe5\x0d\x66\x8e\x51\x4a\xbd\xe7\x8b\x91\xfb\x68\x2d\x05\x9f\x29\x1d\xd5\xd7\x2f\x20\xd3\x89\x76\x14\x69\xe6\xc7\x1a\x54\xa2\xf4\x10\x5e\x6d\xcf\xb9\x54\xd1\x9f\x92\x78\xa5\xb9\xcf\x80\xee\x40\xb7\x04\xd6\xbf\xa9\xca\xc4\x3e\x17\x16\xd6\x9a\x47\xa3\x73\x35\xf6\xd7\xfc\xcc\x3c\x74\x12\x60\x68\x3b\x67\x97\x9f\x12\xc3\x0a\x59\x3e\xe5\xe9\x2d\xf5\xcf\x24\x06\x7d\xdf\xd3\x1a\xec\x3c\x3b\xe3\x8a\xed\x61\x6f\xfc\x49\x0a\xcf\x46\x53\xcb\x55\xdf\x1c\x37\x5b\x1e\xe8\x58\xeb\x94\x15\xdc\xb5\x65\xd1\x77\xaa\xa5\x3c\x24\xf5\x49\x10\x00\x69\x01\x2f\x8f\x51\x48\x49\xa5\xe7\x33\x8e\x04\xd0\x95\x59\xec\x17\x14\x01\x39\xf9\xb4\xbf\x68\xee\xa3\x8e\x73', 1)

The obfuscated script is a normal Python script, with an extra module pytransform.py and a few extra runtime files. The plain Python script can be replaced with the obfuscated one seamlessly.

Code obfuscation does not alter the output of a program, which remains fully functional. Just remember that code obfuscation is not encryption. It is just a way to delay human understanding of the code, and to make reverse-engineering of a program difficult and economically unfeasible, as this would need too much time and effort.

Vault configuration

Vault must be accessible from both build time and runtime environments. It can be deployed either on-premises or in the cloud, as long as this environment is trusted. This guide provides guidance for a production hardened deployment of Vault.

For the purposes of a demo, you can use the following script which automates the deployment and configuration of a Vault server in development mode:

#!/bin/bash

# Download Vault
VAULT_VERSION=1.2.3
curl https://releases.hashicorp.com/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_linux_amd64.zip -o vault.zip
unzip vault.zip
chmod u+x vault

export VAULT_ADDR=http://127.0.0.1:8200
export VAULT_ROOT_TOKEN_ID=root

# Start Vault in dev mode
nohup vault server -dev -dev-listen-address=0.0.0.0:8200 -dev-root-token-id=${VAULT_ROOT_TOKEN_ID} &

# Wait until the Vault server has actually started
sleep 5

# Login to vault
echo $VAULT_ROOT_TOKEN_ID | vault login --no-print=true -

# Enable the Transit secrets engine which generates or encrypts data in-transit
vault secrets enable transit

# Create dev policy
vault policy write asguard-policy policy-dev.hcl

# Enable the Google Cloud auth method
vault auth enable gcp

# Give Vault server the JSON key of a service account with the required GCP permissions:
# roles/iam.serviceAccountKeyAdmin
# roles/compute.viewer

# These allow Vault to:
# - Verify that the service account associated with authenticating GCE instance exists
# - Get the corresponding public keys for verifying JWTs signed by service account private keys.
# - Verify authenticating GCE instances exist
# - Compare bound fields for GCE roles (zone/region, labels, or membership in given instance groups)

vault write auth/gcp/config credentials=@credentials.json

# The GCE instances that are authenticating against Vault must have the following role: roles/iam.serviceAccountTokenCreator

ROLE=asguard-gce-role

# Create role for GCE instance authentication
vault write auth/gcp/role/${ROLE} \
    type="gce" \
    policies="asguard-policy" \
    bound_projects="asguard" \
    bound_zones="europe-west1-b" \
    bound_labels="foo:bar,zip:zap,gruik:grok"

with policy-dev.hcl:

# This section grants all access on "transit/*".
path "transit/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

Vault provide many "secrets engines" to store, generate, or encrypt data.
Here, we use the transit secrets engine to handle cryptographic functions on data in-transit only. With the exception of the main key (KEK), Vault doesn't store the data sent to the secrets engine.

The last section in the bash script states that only GCE instances in the GCP project named "asguard", in the zone europe-west1-b and with the labels foo:bar, zip:zap, and gruik:grok will be able to authenticate to Vault.

The Docker images

We distinguish 2 containers for our app: the app container itself, and one init container, which is a specialized container that will run before the app container in a Kubernetes pod. The init container contains the bash script that will decrypt then start the application.

Here is our pod configuration:

apiVersion: v1
kind: Pod
metadata:
  name: asguard-demo
  labels:
    app: asguard-demo
spec:
  containers:
  - name: asguard-demo
    image: eu.gcr.io/$GCLOUD_PROJECT_ID/$CI_PROJECT_NAME:$CI_COMMIT_TAG
    imagePullPolicy: Always
    ports:
    - containerPort: 8000
    env:
      - name: BOOTSTRAP_DIR
        value: /asguard
      - name: CACHE_DIR
        value: /cache
      - name: VAULT_ADDR
        valueFrom:
          secretKeyRef:
            name: vault-secret
            key: vault-addr
      - name: VAULT_KEY_NAME
        valueFrom:
          secretKeyRef:
            name: vault-secret
            key: vault-key-name
      - name: VAULT_ROLE
        valueFrom:
          secretKeyRef:
            name: vault-secret
            key: vault-role
    volumeMounts:
      - name: workdir
        mountPath: /asguard
      - name: cache-volume
        mountPath: /cache
  # These containers are run during pod initialization
  initContainers:
  - name: install-bootstrap
    image: eu.gcr.io/$GCLOUD_PROJECT_ID/init-$CI_PROJECT_NAME:$CI_COMMIT_TAG
    command:
      - cp
      - "-R"
      - "."
      - "/work-dir"
    volumeMounts:
      - name: workdir
        mountPath: "/work-dir"
  dnsPolicy: Default
  volumes:
  # Volume shared by init and app containers
  - name: workdir
    emptyDir:
      medium: Memory
  # Volume to decrypt the application in memory
  - name: cache-volume
    emptyDir:
      medium: Memory

In the configuration file, you can see that the pod has a volume that the init container and the application container share.

The init container mounts the shared Volume at /work-dir, and the application container mounts it at /asguard. The init container runs the following command and then terminates:

cp -R . /work-dir

The command copies the bootstrap script and other utilities from the init container's working directory to the shared volume. This done, the application container has now access to the bootstrap script, which is its actual entrypoint.

The app container has another tmpfs mount at /cache. A tmpfs volume is a RAM-backed filesystem, that means a tmpfs mount is temporary, and only persisted in the host memory and outside the container's writable layer. This is particularly useful to decrypt our application in this directory: when the container stops, the tmpfs mount is removed, and files written there won't be persisted.

Here is the Dockerfile for our application image:

# Start by obfuscating the application
FROM python:3.5.3-slim AS obfuscate

RUN pip install pyarmor
COPY app/sensitive-app.py app/
# Python version must be the same as the target version in the final image
RUN pyarmor obfuscate app/sensitive-app.py
# Creates a folder dist with the obfuscated Python script


# Then encrypting the application
FROM gcr.io/distroless/base:debug AS build-env

SHELL ["/busybox/sh", "-c"]

# Define build arg
ARG VAULT_KEY_NAME

# Define env variable
ENV VAULT_KEY_NAME ${VAULT_KEY_NAME}

ARG VAULT_ADDR
ARG VAULT_ROOT_TOKEN_ID

WORKDIR /dist
ENV PATH /dist:${PATH}

# Install jq
ARG JQ_VERSION=1.6
RUN wget https://github.com/stedolan/jq/releases/download/jq-${JQ_VERSION}/jq-linux64 -O jq
RUN chmod u+x jq

# Embed the application. DO NOT USE ADD HERE!
COPY app/ app/
# With only the obfuscated Python script
RUN rm app/sensitive-app.py
COPY --from=obfuscate /dist/ app/

# Package the application
RUN tar cvf app.tar.gz app/

# Create a named encryption key
# If the key already exists, this has no effect
RUN wget \
    --header "X-Vault-Token: ${VAULT_ROOT_TOKEN_ID}" \
    --post-data "" \
    ${VAULT_ADDR}/v1/transit/keys/${VAULT_KEY_NAME}

# Rotates the version of the named key
RUN wget \
    --header "X-Vault-Token: ${VAULT_ROOT_TOKEN_ID}" \
    --post-data "" \
    ${VAULT_ADDR}/v1/transit/keys/${VAULT_KEY_NAME}/rotate

# Generate data key
# The name specified as part of the URL is the name of the encryption key
# created in the init container and used to encrypt the datakey
RUN wget \
    --header "X-Vault-Token: ${VAULT_ROOT_TOKEN_ID}" \
    --post-data "" \
    ${VAULT_ADDR}/v1/transit/datakey/plaintext/${VAULT_KEY_NAME} -O datakey.json

RUN cat datakey.json | jq -r ".data.ciphertext" > passphrase.enc

# Encrypt the binary with the plaintext key 
RUN cat datakey.json | jq -r ".data.plaintext" | openssl aes-256-cbc -salt -in app.tar.gz -out app.enc -pass stdin

# Delete unsecured files (delete all files that do not match *.enc)
RUN find /dist/* ! -name '*.enc' -exec rm -rf '{}' +

RUN chmod -R 500 /dist


# App container
FROM gcr.io/distroless/python3

USER nobody:nobody
COPY --chown=nobody:nobody --from=build-env /dist/ /dist/
CMD ["python app/sensitive-app.py"]
# The entrypoint will be mounted from the init container
ENTRYPOINT ["/asguard/bootstrap.sh"]

And here is the Dockerfile for the init container:

FROM gcr.io/distroless/base:debug AS build-env

SHELL ["/busybox/sh", "-c"]

WORKDIR /asguard
ENV PATH /asguard:${PATH}

# Install jq
ARG JQ_VERSION=1.6
RUN wget https://github.com/stedolan/jq/releases/download/jq-${JQ_VERSION}/jq-linux64 -O jq

# Install cat, tar, wget and cp for bootstrap.sh later execution
ARG BUSYBOX_VERSION=1.30.0-i686
RUN wget https://busybox.net/downloads/binaries/${BUSYBOX_VERSION}/busybox_CAT -O cat
RUN wget https://busybox.net/downloads/binaries/${BUSYBOX_VERSION}/busybox_TAR -O tar
RUN wget https://busybox.net/downloads/binaries/${BUSYBOX_VERSION}/busybox_WGET -O wget
RUN wget https://busybox.net/downloads/binaries/${BUSYBOX_VERSION}/busybox_CP -O cp

COPY bootstrap.sh ./

RUN chmod -R 500 /asguard


# Init container
FROM gcr.io/distroless/base

USER nobody:nobody
COPY --chown=nobody:nobody --from=build-env /asguard /asguard
WORKDIR /asguard
ENV PATH /asguard:${PATH}

The two Dockerfiles alone will perform steps 1 to 5 of the CI pipeline. Nothing more than docker build is needed to create the automated build.

We can make some comments on these Dockerfiles:

We use Docker multi-stage builds to separate the main steps of image build and to keep the final images as light as possible.
We use "distroless" images to harden the images as much as possible.
Distroless images contain only the application and its runtime dependencies. They do not contain package managers, shells (you cannot even do a ls or a cd) or any other programs you would expect to find in a standard Linux distribution. You can find different distroless base images depending on your application stack: Java, Python, Golang, Node.js, dotnet, Rust.
For intermediate build stages, we use distroless debug images which provide a busybox shell. This allows to run all the necessary commands to actually build the image, and for the init image, to download and install all the busybox commands (cat, tar, wget, in a unitary way) needed for bootstrap.sh later execution in the non-debug final image. We also install cp that will be used to copy the bootstrap script in the shared volume.
We use the HTTP API to interact with Vault. We could have added the CLI into the image, but it is more secure and lighter not to embed a tool that offers far more services that the ones we only need.
As a consequence, we must install jq to process the JSON output from HTTP calls.
We can use OpenSSL or GnuPG to encrypt the application locally in the container. Here, we use OpenSSL as it is already shipped in distroless images.
We run the final images with a non-root user.

The build commands are the following:

docker build -t init-asguard-demo -f Dockerfile-init .

docker build -t asguard-demo \
    --network=host \
    --build-arg VAULT_ADDR=https://vaulthost:8200 \
    --build-arg VAULT_ROOT_TOKEN_ID=s3cr3tt0k3n \
    --build-arg VAULT_KEY_NAME=aname \
    -f Dockerfile-app .

The final images are expected to run in Kubernetes (due to the init container and the mounted volumes), and the bootstrap script will attempt to authenticate to Vault with the GCP method. So, you won't be able to run the final images locally with the default entrypoint (and that is our goal).

If you edit the Dockerfile to change the final image to :debug, you can nevertheless get a busybox shell to enter and see that the application is indeed encrypted:

docker run -it --rm --entrypoint=sh asguard-demo:latest

Gitlab CI/CD

GitLab CI/CD is configured by a file called .gitlab-ci.yml placed at the
repository’s root:

stages:
  - build
  - deploy

build:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - export GOOGLE_APPLICATION_CREDENTIALS=/kaniko/kaniko-secret.json
    - echo $GOOGLE_APPLICATION_CREDENTIALS_BASE64 | base64 -d > $GOOGLE_APPLICATION_CREDENTIALS
    - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile Dockerfile-init --build-arg VAULT_ADDR=$VAULT_ADDR --build-arg VAULT_ROOT_TOKEN_ID=$VAULT_ROOT_TOKEN_ID --build-arg VAULT_KEY_NAME=$VAULT_KEY_NAME --destination eu.gcr.io/$GCLOUD_PROJECT_ID/init-$CI_PROJECT_NAME:$CI_COMMIT_TAG
    - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile Dockerfile-app --build-arg VAULT_ADDR=$VAULT_ADDR --build-arg VAULT_ROOT_TOKEN_ID=$VAULT_ROOT_TOKEN_ID --build-arg VAULT_KEY_NAME=$VAULT_KEY_NAME --destination eu.gcr.io/$GCLOUD_PROJECT_ID/$CI_PROJECT_NAME:$CI_COMMIT_TAG
  only:
    - tags


deploy:
  stage: deploy
  image:
    name: google/cloud-sdk:latest
  script:
    # Authenticate with GKE
    - echo $GOOGLE_APPLICATION_CREDENTIALS_BASE64 | base64 -di > key.json
    - gcloud auth activate-service-account --key-file=key.json
    - gcloud config set project $GCLOUD_PROJECT_ID
    - gcloud container clusters get-credentials $CLUSTER_NAME --zone europe-west1-b
    # Create Kubernetes secret with Vault info
    - kubectl delete secret vault-secret || true
    - kubectl create secret generic vault-secret --from-literal=vault-addr=$VAULT_ADDR --from-literal=vault-key-name=$VAULT_KEY_NAME --from-literal=vault-role=$VAULT_ROLE
    # Install envsubst
    - apt-get install -y gettext-base
    # Deploy
    - cat k8s/deployment.yml | envsubst | kubectl apply -f -

  only:
    - tags

The pipeline of scripts builds the images (steps 1 to 5), pushes them into the Google Container Registry (step 6), then deploys the application into GKE (step 7) at every tag in the master branch of the repository.

Some observations that are worth noting:

Each stage in the file is executed by the Gitlab Runner in a container image.
Instead of docker build, we use Kaniko to build the application images from the Dockerfiles. Kaniko is meant to be run inside a container or Kubernetes cluster. Kaniko does not depend on a Docker daemon and it is therefore an alternative to Docker-in-Docker, which is subject to potential security issues.
We use protected environment variables in Gitlab to configure the pipeline (Vault host, Vault's root token, Google credentials and so on). Protected variables are securely passed to the pipeline running.
We use envsubst to substitute the values of environment variables into Kubernetes YAML files. I can already hear the haters saying that Kubernetes files should be template-free and should only be patched (see this other article on kubectl kustomize), but hey, this is a demo...😱

Gitlab itself must be a trusted part of the chain: it can be either self-hosted on-premises or online (SaaS). The Gitlab project must have at least restricted access to the only developers needed for the project. If online, the project repository should be private so as not to publish the source code publicly. Gitlab protects the source code with encryption in transit and at rest, and AFAIK you can manage your own encryption keys. Regardless of the project repository, the Gitlab Runner can also be installed on-premises.

Let's run

Here we are! We committed and pushed our changes, we created a git tag, the images have been pulled, and the Kubernetes pod has started. The bootstrap script can finally proceed:

#!/bin/sh

set -eux

VAULT_ADDR=${VAULT_ADDR}
VAULT_KEY_NAME=${VAULT_KEY_NAME}
VAULT_ROLE=${VAULT_ROLE}

WORK_DIR=/dist
CACHE_DIR=${CACHE_DIR:-/cache}

BOOTSTRAP_DIR=${BOOTSTRAP_DIR:-/asguard}
cd ${BOOTSTRAP_DIR}
export PATH=${BOOTSTRAP_DIR}:${PATH}

# Get the JWT token for the GCE instance that runs this container
INST_ID_TOKEN=$(wget \
  --header "Metadata-Flavor: Google" \
  "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/identity?audience=vault/${VAULT_ROLE}&format=full" -qO -)

AUTH_PAYLOAD="{\"role\": \"${VAULT_ROLE}\", \"jwt\": \"${INST_ID_TOKEN}\"}"

# GCE login
CLIENT_TOKEN=$(wget \
    --post-data "${AUTH_PAYLOAD}" \
    ${VAULT_ADDR}/v1/auth/gcp/login -O - | \
    jq -r '.auth.client_token')

PAYLOAD="{\"ciphertext\": \"$(cat ${WORK_DIR}/passphrase.enc)\"}"

# Decrypt the application
wget \
    --header "X-Vault-Token: ${CLIENT_TOKEN}" \
    --post-data "${PAYLOAD}" \
    ${VAULT_ADDR}/v1/transit/decrypt/${VAULT_KEY_NAME} -O - | jq -r '.data.plaintext' | openssl aes-256-cbc -d -salt -in ${WORK_DIR}/app.enc -out ${CACHE_DIR}/app.tar.gz -pass stdin

# Unpack
tar xvf ${CACHE_DIR}/app.tar.gz -C ${CACHE_DIR}

# Execute arg command
cd ${CACHE_DIR}
/bin/sh -c "$1"

The script performs final steps 7 to 11 of the CI pipeline. The last instruction executes the command python app/sensitive-app.py passed as an argument to the image's entrypoint. At this point, the application is now decrypted and unpacked: the Python web server can start and Grumpy the cat is now happy!

If you visit the Kubernetes ingress's endpoint, you will see the following web page:

Note that the self-starting process here can be implemented in whatever language that suits you: you can pick the right distroless image for the desired stack, and Vault provides many client libraries to consume the API more conveniently.

The "need to know", and the principle of the least privilege

We cannot emphasize this enough, all these security measures are only relevant if all parts of the pipeline are trusted themselves. Most attacks are indeed initiated by users with regular access to the system or communication channel (man-at-the-end attack). That means Gitlab, Vault, the GCP project, and all communication media between them, must limit their access to the smallest number of people (developers, administrators). And within these systems, users must be able to access only the information and resources that are necessary for their job (ex: all users who end up to be administrators of all GCP projects in an organization, that is not a good thing).

Decrypting in memory

We saw that the application is decrypted in memory in a tmpfs mount. If such a mount is temporary, the filesystem is still visible from the host. If an attacker has access to the Kubernetes node where the pod is running, he can find our sensitive source code:

# find / -name sensitive-app.py
/home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/cb74781d-7fc7-11ea-adc6-42010a8400f8/volumes/kubernetes.io~empty-dir/cache-volume/app/sensitive-app.py
/var/lib/kubelet/pods/cb74781d-7fc7-11ea-adc6-42010a8400f8/volumes/kubernetes.io~empty-dir/cache-volume/app/sensitive-app.py

Of course the source code is obfuscated, but still.

What about trust?

When transferring container images over internet, trust is a central concern. It would be good to ensure their integrity and that the CI pipeline is the source of these images.

Docker Content Trust (DCT) provides this ability to use digital signatures for data sent to and received from remote Docker registries. These signatures allow client-side or runtime verification of the integrity and publisher of specific image tags.

Through DCT, the CI pipeline could sign the images and Kubernetes could ensure that the images it pulls are signed. But a prerequisite for signing an image with DCT is a Docker Registry with a "notary" server attached (such as the Docker Hub). This is not currently supported in Google Container Registry (see here).

Google Cloud now implements the concept of Binary Authorization based off of open source Kritis. However, this is specific to Google Cloud and I don't know if it's possible to set up a vanilla Kritis there.

With Vault, we could easily sign our application and verify this data from the bootstrap script. But then, an attacker could modify the script to not verify the data, and thus break the trust chain. Consequently, the script (or the init image) should also be signed. But it cannot verify itself, and we end up with a chicken-or-egg problem...

Criticism

An alternative to the images' build would be putting the encrypted datakey next to the bootstrap script in the init image, instead of placing it next to the application in the app image. In that way, we would separate the encrypted application from the key needed to decrypt it in two separate images. This may be viewed as an additional security. But in that case, the key must be created (or rotated) outside of Docker - as it must be both placed in the init image and used in the app image to actually encrypt it - and docker build alone does not suffice anymore to build the images. The first solution is therefore an assumed choice.

Also, the solution we presented here relies on mechanisms that are internal to the image: this is security by the contents, which is a form of security through obscurity. Like docker trust, it would be nice if Docker could encrypt an image with a docker encrypt command line. The result image would be an envelope with clear metadata and encrypted contents.

Links

We named this security solution for bringing sensitive apps into the cloud "Asguard". It is named after the location of Ásgard in the Norse religion (not Marvel, please 😠). Ásgard is an impregnable divine fortress surrounded by an impassable wall that shelters the gods from all invasions.

We made a demo which includes all the elements explained in this article. You will find the source code here.

We also presented this subject at the "Capitole du Libre" conference at Toulouse in November 2019. Here is the YouTube video (in French):