Forem: Otomato

Why You Need an Internal Developer Portal NOW!!!

otomato — Sun, 21 May 2023 08:51:52 +0000

IDPs are all the buzz now. But they aren't just buzz - they are an actual blessing for the organizations that get them right. Our CEO Anton describes why in his new post

DevOpsCon Munich 2022 - Human Interactions that Matter

Ant(on) Weiss — Fri, 09 Dec 2022 08:50:00 +0000

I'm at the Munich airport waiting for my flight home.
This is a cold, gloomy morning - a perfect time to introspect and retrospect.

The conference had the level of quality I've learned to expect from S&S Media. They organize a few international lines of events - DevOpsCon, Jax, Serverless Conference, etc. and they all rock. The Munich conference was 4 days total, but I only arrived on the 3rd day - to give my talk called "Resilience in Engineering... and Life" and then do my part in the advanced CI/CD workshop.

The talk was an experiment and a challenge I've imposed on myself. During the COVID-19 era I got so fed up with virtual talks and so hungry for true human interaction that I found myself unable to attend and enjoy even the regular conference talks. You know - those where the speaker tries to put on a show on stage while half the audience are on their phones and laptops.

So instead of just writing and rehearsing a talk as I would usually do - I decided to have an open conversation with the audience about resilience - technical and psychological. After all - resilience engineering is about what we do to continue operating in expected as well as unexpected conditions. So I've opened the tap of unexpected events and drank from it. And - from the get-go I encouraged my audience to get up close and personal and engage. And it all clicked! We talked about burnout and observability, about HA and emotional support, about entropy and growth. And how it's all components of one big socio-technical system. Probably my most important conference talk until now.

That night Sebastian Meyen - the chief content officer at S&S Media took the speakers out and I had a great, deep conversation with Zbynek Roubalik - one of the maintainers of both Knative and KEDA. He got me all excited about the GPTchat. I even tried to play with it that same night when I came back to the hotel. But I got bored after 20 minutes. It still feels like talking to a machine... I don't see the threat that everybody seems to talk about. I also still don't see the value it provides. Which for me would be the reduction of toil that goes into coding and content creation. But I'm optimistic - there's field for improvement!

And then came the workshop day. Thanks to Nir Koren for throwing this together. My part was about Progressive Delivery with Argo Rollouts. I believe I did a great job presenting the concept and the technology - complete with Traefik Ingress and Prometheus integration. Folks came up to me to say they enjoyed it and learned from it - what more do I need?

And again - the best part was the panel discussion on CI/CD topics that we did in the end. So again - there's nothing like open human conversation for making work better. If you're organizing a conference - take note. It's the human interactions that will make your event memorable and enjoyable.

Next week I'm teaching 2 Kubernetes classes. More human interactions for me. Life is great and full of meaning when you're focused on humans. That's how I always try to work. I don't always succeed, but I continue learning.

And I would like to finish this post with a great quote from Joseph Campbell which I recently found in Brene Brown's book:

“If you can see your path laid out in front of you step by step, you know it's not your path. Your own path you make with every step you take. That's why it's your path.”

Liveness Probes: Feel the Pulse of the App

Roman Belshevitz — Mon, 28 Nov 2022 13:30:16 +0000

This article will provide some helpful examples as the author examines probes in Kubernetes. A correct probe definition can increase pod availability and resilience!

A Kubernetes Liveness Probe: What Is It?

Based on a given test, the Liveness probe makes sure that an application inside a container is active and working.

⚙️ Liveness probes

They are used by the kubelet to determine when to restart a container. Applications that crash or enter broken states are detected and, in many cases, can be rectified by restarting them.

A successful configuration of the liveness probe results in no action being taken and no logs being kept. If it fails, the event is recorded, and the container is killed by the kubelet in accordance with the restartPolicy settings.

When a pod might seem to be running, but the application might not be working properly, a liveness probe should be utilized. During a standstill, as an illustration. The pod might be operational, but it is ineffective since it cannot handle traffic.

🖼️ Pic source: K21Academy

Since the kubelet will check the restartPolicy and restart the container automatically if it is set to Always or OnFailure, they are not required when the application is configured to crash the container on failure. The NGINX application, for example, launches rapidly and shuts down if it encounters a problem that prevents it from serving pages. You are not in need of a liveness inquiry in this instance.

There are common adjustable fields for every type of probe:

initialDelaySeconds: Probes start running after initialDelaySeconds after container is started (default: 0)
periodSeconds: How often probe should run (default: 10)
timeoutSeconds: Probe timeout (default: 1)
successThreshold: Required number of successful probes to mark container healthy/ready (default: 1)
failureThreshold: When a probe fails, it will try failureThreshold times before deeming unhealthy/not ready (default: 3)

The periodSeconds field in each of the examples below says that the kubelet should run a liveness probe every 5 seconds. The initialDelaySeconds field instructs the kubelet to delay the first probe for 5 seconds.

The timeoutSeconds option (Time to wait for the reply), successThreshold (Number of successful probe executions to mark the container healthy), and failiureThreshold (Number of failed probe executions to mark the container unhealthy), among other options, can also be customized, if desired.

All different liveness probes can use these five parameters.

What other Kubernetes probes are available?

Although the use of Liveness probes will be the main emphasis of this article, you should be aware that Kubernetes also supports the following other types of probes:

⚙️ Startup probes

The kubelet uses startup probes to help it determine when a container application has begun. When enabled, these make sure startup probes don't obstruct the application startup by disabling liveness and readiness checks until they are successful.

These are especially helpful for slow-starting containers since they prevent the kubelet from killing them before they have even started when a liveness probe fails. Set the startup probe's failureThreshold greater if liveness probes are used on the same endpoint in order to enable lengthy startup periods.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-api-deployment
spec:
  template:
    metadata:
      labels:
        app: test-api
    spec:
      containers:
      - name: test-api
        image: myrepo/test-api:0.1
        startupProbe:
          httpGet:
            path: /health/startup
            port: 80
          failureThreshold: 30
          periodSeconds: 10

When a Pod starts and the probe fails, Kubernetes will try failureThreshold times before giving up. Giving up in case of liveness probe means restarting the Pod. In case of readiness probe, the Pod will be marked Unready. Defaults to 3. The minimum value is 1.

Some startup probe's math: why it is important?

0 - 10 s: the container has been spun up but the kubelet doesn't do anything waiting for the initalDelaySeconds to pass
10 - 20 s: the first probe request is sent but no response is sent back, this is because the app hasn’t stood up the APIs yet, this is either a failure due to 2 seconds timeout or an immediate TCP connection error
20 - 30 s: the app has got up but has only started fetching credentials, configurations and so on, so the response to the probe request is 5xx
30 - 210 s: the kubelet has been probing but the success response didn’t come and is reaching the limit set by the failureThreshold. In this case, as per the deployment configuration for the startup probe, the pod will be restarted after roughly 212 seconds.

🖼️ Pic source: Wojciech Sierakowski (HMH Engineering)

It might be a little excessive to wait more than 3 minutes for the app to launch locally with faked dependencies!

🎯 It might be also better to shorten this interval if you are absolutely certain that, for example, reading secrets, credentials, and establishing connections with DBs and other data sources shouldn't take so long. Doing so will slow down the deployment speed.

Maybe it’s important to figure out if you even need more nodes. You don’t want to waste your money on resources you don’t need. Take a look at kubectl top nodes to see if you even need to scale the nodes.

🚧 If probe fails, the event is recorded, and the container is killed by the kubelet in accordance with the restartPolicy settings.

When a container gets restarted you usually want to check the logs why the application went unhealthy. You can do this with the following command:

kubectl logs <pod-name> --previous

⚙️ Readiness probes

Readiness probes keep track of the application's availability. No traffic will be forwarded to the pod if it fails. These are employed when an application requires configuration before it is usable. Additionally, an application may experience traffic congestion and cause the probe to malfunction, stopping further traffic from being routed to it and allowing it to recover. The endpoints controller takes the pod out if it fails.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-api-deployment
spec:
  template:
    metadata:
      labels:
        app: test-api
    spec:
      containers:
      - name: test-api
        image: myrepo/test-api:0.1
        readinessProbe:
          httpGet:
            path: /ready
            port: 80
          successThreshold: 3

The kubelet finds that the container is not yet prepared to receive network traffic, but is making progress in that direction if the readiness probe fails but the liveness probe succeeds.

The operation of Kubernetes probes

The kubelet controls the probes. The main "node agent" that executes on each node is the kubelet.

🖼️ Pic source: Andrew Lock (Datadog). SVG is here

The application needs to support one of the following handlers in order to use a K8S probe effectively:

ExecAction handler: Executes a command inside the container. If the command returns a status code of 0, the diagnosis is successful.
TCPSocketAction handler tries to establish a TCP connection to the pod's IP address on a particular port. If the port is discovered to be open, the diagnostic is successful.

Using the IP address of the pod, a particular port, and a predetermined destination, the HTTPGetAction handler sends an HTTP GET request. If the response code given falls between 200 and 399, the diagnostic is successful.

Before version 1.24 Kubernetes did not support gRPC health checks natively. This left the gRPC developers with the following three approaches when they deploy to Kubernetes:

🖼️ Pic source: Ahmet Alp Balkan (Twitter, ex-Google)

As of Kubernetes version 1.24, gRPC handler can be configured to be used by kubelet for application liveness checks if your application implements the gRPC Health Checking Protocol. To configure checks that use gRPC, you must enable the GRPCContainerProbe feature gate.

When the kubelet conducts a probe on a container, it answers with Success, Failure, or Unknown, depending on whether the diagnostic was successful, unsuccessful, or incomplete for some other reason.

So, how rushy to track the pulse?

You should examine the system behavior and typical starting timings of the pod and its containers before defining a probe so that you can choose the appropriate thresholds. Additionally, as the infrastructure or application changes, the probe choices should be changed. For instance, a pod's configuration to use more system resources can have an impact on the values that need to be configured for the probes.

Handlers in action: some examples

`ExecAction` handler: how can it be useful in practice?

🎯 It allows you to use commands inside containers to control the status of life of a counter in pods. With the help of this option, you may examine several aspects of container's operation, such as the existence of files, their contents, and other choices (accessible at the command level).

ExecAction is executed in pod’s shell context and is deemed failed if the execution returns any result code different from 0 (zero).

The example below demonstrates how to use the exec command with the cat command to see if a file exists at the path /usr/share/liveness/html/index.html.

apiVersion: v1
kind: Pod
metadata:
  labels:
    test: liveness
  name: liveness-exec
spec:
  containers:
  - name: liveness
    image: registry.k8s.io/liveness:0.1
    ports:
    - containerPort: 8080
    livenessProbe:
      exec:
        command:
        - cat
        - /usr/share/liveness/html/index.html
      initialDelaySeconds: 5
      periodSeconds: 5

🚧 The container will be restarted if there is no file and the liveness probe will fail.

`TCPSocketAction` handler: how can it be useful in practice?

In this use case, the liveness probe makes use of the TCP handler to determine whether port 8080 is active and open. With this configuration, your container will try to connect to the kubelet by opening a socket on the designated port.

apiVersion: v1
kind: Pod
metadata:
  name: liveness
  labels:
    app: liveness-tcp
spec:
  containers:
  - name: liveness
    image: registry.k8s.io/liveness:0.1
    ports:
    - containerPort: 8080
    livenessProbe:
      tcpSocket:
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 5

🚧 The container will restart if the socket is dead and liveness probe fails.

`HTTPGetAction` handler: how can it be useful in practice?

This case demonstrates the HTTP handler that will send an HTTP GET request to the /health path on port 8080. A value between 200 and 400 indicates that the probe was successful.

apiVersion: v1
kind: Pod
metadata:
  labels:
    test: liveness
  name: liveness-http
spec:
  containers:
  - name: liveness
    image: registry.k8s.io/liveness:0.1
    livenessProbe:
      httpGet:
        path: /health
        port: 8080
        httpHeaders:
        - name: Custom-Header
          value: ItsAlive
      initialDelaySeconds: 5
      periodSeconds: 5

🚧 The probe fails, and the container is restarted if a code outside this range is received. Any custom headers you want to transmit can be defined using the httpHeaders option.

gRPC handler: how can it be useful in practice?

gRPC protocol is on its way to becoming the lingua franca for communication between cloud-native microservices. If you are deploying gRPC applications to Kubernetes today, you may be wondering about the best way to configure health checks.

This example demonstrates how to check port 2379 responsiveness using the gRPC health checking protocol. A port must be specified in order to use a gRPC probe. You must also specify the service if the health endpoint is set up on a non-default service.

apiVersion: v1
kind: Pod
metadata:
  labels:
    test: liveness
  name: liveness-gRPC
spec:
  containers:
  - name: liveness
    image: registry.k8s.io/liveness:0.1
    ports:
    - containerPort: 2379
    livenessProbe:
      grpc:
        port: 2379
      initialDelaySeconds: 5
      periodSeconds: 5

🚧 The container will restart if the gRPC socket is dead and liveness probe fails.

Since there are no error codes for gRPC built-in probes, all errors are regarded as probe failures.

Using liveness probes in the wrong way can lead to disaster

Please remember that the container will be restarted if the liveness probe fails. It is not conventional to examine dependencies in a liveness probe, unlike a readiness probe.

To determine whether the container itself has stopped responding, a liveness probe should be utilized.

A liveness probe has the drawback of maybe not verifying the service's responsiveness. For instance, if a service maintains two web servers, one for service routes and the other for status routes, such as readiness and liveness probes or metrics gathering, the service may be delayed or inaccessible while the liveness probe route responds without any issues. The liveness probe must use the service in a comparable way to dependent services for it to be effective.

Like the readiness probe, it's crucial to take into account dynamics that change over time. A slight increase in response time, possibly brought on by a brief rise in load, could force the container to restart if the liveness-probe timeout is too short. The restart might put even more strain on the other pods supporting the service, leading to a further cascade of liveness probe failures and worsening the service's overall availability.

🖼️ Pic source: Wojciech Sierakowski (HMH Engineering)

These cascade failures can be prevented by configuring liveness probe timeouts on the order of client timeouts and employing a forgiving failureThreshold count.

Liveness probes may have a small issue with the container startup latency varying over time (see above about the math). Changes in resource allocation, network topology changes, or just rising load as your service grows could all contribute to this.

If the initialDelaySeconds option is insufficient and a container is restarted as a result of a Kubernetes node failure or a liveness probe failure, the application may never start or may start partially before being repeatedly destroyed and restarted. The container's maximum initialization time should be greater than the initialDelaySeconds option.

Some notable suggestions are:

Keep dependencies out of liveness probes. Liveness probes should be reasonably priced and have consistent response times.
So that system dynamics can alter temporarily or permanently without causing an excessive number of liveness probe failures, liveness probe timeouts should be conservatively set. Consider setting client timeouts and liveness-probe timeouts to the same value.
To ensure that containers can be restarted with reliability even if starting dynamics vary over time, the initialDelaySeconds option should be set conservatively.

The inevitable summary

By causing an automatic restart of a container after a failure of a particular test is discovered, the proper integration of liveness probes with readiness and startup probes can increase pod resilience and availability. It is necessary to comprehend the application in order to specify the appropriate alternatives for them.

The author is thankful to Guy Menachem from Komodor for inspiration! Stable applications in the clouds to you all, folks!

Kubernetes TLS, Demystified

Roman Belshevitz — Tue, 11 Oct 2022 18:16:13 +0000

This is the anniversary 10th article in this series.

🛡️ It is more than obvious a secured connection to any exposed service running in Kubernetes cluster is important.

The supposition for this article is that you wish to set up TLS (Transport Layer Security) realm for your ingress resource and that you already have a functioning ingress controller established in your cluster.

The SSL replacement technology is called Transport Layer Security (TLS) today. An enhanced version of SSL is TLS.

Similar to how SSL operates, it uses encryption to safeguard the transmission of data and information. Although SSL is still commonly utilized in the industry, the two names are frequently used interchangeably.

Getting a certificate: what paths can be taken?

A TLS/SSL certificate is the fundamental prerequisite for ingress TLS. These certificates are available to you in the following ways.

Path one. Self-signed certificates: Our own Certificate Authority (root CA) created and signed the TLS certificate. It is a well-known, stunt choice for testing scenarios where you can "collaborate" on the root CA so that browsers will accept the certificate.

🖼️ Pic source: Bizagi

Path two. Get an SSL certificate: for production use cases, you must purchase an SSL certificate from a reputable certificate authority that operating systems and browsers trust. But you must bear in mind that a so-called wildcard certificate suitable to protect all subdomains in a domain can cost $300+/year from major commercial issuers.
Path three. Use a Let's Encrypt certificate: Let's Encrypt is a reputable certificate authority that issues free TLS certificates.

A few words about Let's Encrypt

It is a non-profit organization founded by enthusiasts in the field of struggle for privacy and security in 2014.

The challenge–response protocol used to automate enrolling with the certificate authority is called Automated Certificate Management Environment (ACME). It can query either Web servers or DNS servers controlled by the domain covered by the certificate to be issued.

🖼️ Pic source: Let's Encrypt

If you are interested in the implementation of the protocol, read what Adrian Colyer from SpringSource writes about them.

Each SSL certificate has an expiration date. So, before the certificate expires, you need to rotate it. For instance, Let's Ecrypt certificates have a three-month expiration date (and here they tell why).

This way, below in this article series, the author will dwell on the third path further in detail. Why? Well, since this path is interesting for its relative self-sufficiency and [relative] independence from commercial / state-owned certificate issuers. In general, the motto of this path is: "If made something with your hands, you know how it works - you're more adapted to survival!"

Of course, Let's Encrypt approach does not always fit needs, but for academic purposes and for startups it works.

But let's look at the situation "in manual mode" - just try to associate a certificate with a protected application. So,

Chicken or egg?

The ingress controller, not the ingress resource, is in charge of SSL. In other words, the ingress controller accesses the TLS certificates you provide to the ingress resource as a Kubernetes secret and incorporates them into its configuration.

Setup TLS/SSL certificates for ingress

Let's examine the procedures for setting up TLS for ingress. We'll start by launching a test application on the cluster. This application will be used to test our TLS-secured ingress.

Establish the new namespace, trial.

kubectl create namespace trial

Keep this as hello-app.yaml. The Deployment and Service objects are present.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-app
  namespace: trial
spec:
  selector:
    matchLabels:
      app: hello
  replicas: 1
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
      - name: hello
        image: "rbalashevich/hello-app:2.0"
---
apiVersion: v1
kind: Service
metadata:
  name: hello-service
  namespace: trial
  labels:
    app: hello
spec:
  type: ClusterIP
  selector:
    app: hello
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP

Deploy the application with a command:

kubectl apply -f hello-app.yaml -n trial

Create a Kubernetes TLS Secret

It is necessary to make the SSL certificate a Kubernetes secret. It will subsequently be directed to the tls block for Ingress resources.

The server.crt (CA trust chain) and server.key (private key) SSL files are assumed to be available from a Certificate Authority, your company, or self-signed, as a last resort.

⚠️ A private key is created by you (the certificate owner) when you request your certificate with a Certificate Signing Request (CSR). Saying other words, you receive a private key when generate a CSR. You submit the CSR code to the certificate authority and keep private key in a safe place.

As for the big three public cloud providers, they have instructions for exporting certificates: AWS CM, GCP CAS, Azure KV.

It is necessary to make the SSL certificate a Kubernetes secret. It will subsequently be directed to the tls block for Ingress resources.

And yes, keep the private key (server.key)!

Let's use the server.crt and server.key files to construct a Kubernetes secret of tls type (SSL certificates). In the trial namespace, where the hello-app deployment is located, we are creating the secret.

Run the kubectl command listed below from the directory where your server is located. Supply the absolute path to the files or the .crt and .key files. The name hello-app-tls is made up.

kubectl create secret tls hello-app-tls \
    --namespace trial \
    --key server.key \
    --cert server.crt

The comparable YAML file, where you must include the contents of the .crt and .key files, is provided below.

apiVersion: v1
kind: Secret
metadata:
  name: hello-app-tls
  namespace: trial
type: kubernetes.io/tls
data:
  server.crt: |
       <crt contents here>
  server.key: |
       <private key contents here>

A Kubernetes ingress is a set of rules that can be configured to give services externally reachable URLs. Based on this understanding, to turn on secure connection, we should add tls block to Ingress object. So, in the trial namespace, we create the sample ingress TLS-capable resource:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-app-ingress
  namespace: trial
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - app.hosting.cloudprovider.com
    secretName: hello-app-tls
  rules:
  - host: "app.hosting.cloudprovider.com"
    http:
      paths:
        - pathType: Prefix
          path: "/"
          backend:
            service:
              name: hello-service
              port:
                number: 80

⚠️ Replace app.hosting.cloudprovider.com to your actual hostname. The host(s) should be the same in both the rules and tls blocks in the Ingress manifest. In other words, they must match.

In case of using NGINX ingress, you can add the supported annotation by the ingress controller you are using if you want a strict SSL. For instance, you can use the annotation nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" in the Nginx ingress controller to permit SSL traffic up until the application.

The way to make sure

Let's check with curl https://app.hosting.cloudprovider.com -kv, is the connection to the app secure now:

* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=app.hosting.cloudprovider.com
*  start date: Oct 6 15:35:07 2022 GMT
*  expire date: Oct 6 15:35:07 2023 GMT
*  issuer: CN=Go Daddy Secure Certificate Authority - G2,
              OU=http://certs.godaddy.com/repository/,
              O="GoDaddy.com, Inc.",L=Scottsdale,ST=Arizona,C=US
*  SSL certificate verify ok.

🔒 If the certificate is valid, then the browser will not swear and there will be no frightening warnings either. Voilà, the connection to our app is secure!

Okay, we've covered the situations for the first and second paths. The next step is pathfinding the third path involving Let's Encrypt certificate.

Estne vita vere brevis?

Sed vita est cum dignitate vivendum. As the author noted above, the life of a certificate from a let's encrypt is short. You have to pay for insolence. Accordingly, some solution is required that would automate the re-issuance of short-lived certificates, right? And such a solution exists, it is cert-manager! It streamlines the process of getting, renewing, and using certificates by adding certificates and certificate issuers as resource types in Kubernetes clusters.

It can generate certificates from a number of supported sources, including Let's Encrypt.

Furthermore, it will check that certificates are current and valid and make an attempt to renew them for a specified period before they expire.

Install cert-manager on Kubernetes

According to the official cert-manager documentation, you can install it by using kubectl or by the provided helm chart.

# Create a dedicated Kubernetes namespace for cert-manager
kubectl create namespace cert-manager

# Add official cert-manager repository to helm CLI
helm repo add jetstack https://charts.jetstack.io

# Update Helm repository cache (think of apt update)
helm repo update

# Install cert-manager on Kubernetes
## cert-manager relies on several Custom Resource Definitions (CRDs)
helm install certmgr jetstack/cert-manager \
    --set installCRDs=true \
    --version v1.9.1 \
    --namespace cert-manager

The Issuer is responsible for issuing certificates. It is the signing authority and based on its configuration. The issuer knows how certificate requests are handled.

Cert-manager also creates several objects using different specifications such as CertificateRequest.

A Certificate resource is a readable representation of a certificate request. Certificate resources are linked to an Issuer who is responsible for requesting and renewing the certificate.

To determine if a certificate needs to be re-issued, cert-manager looks at the the spec of Certificate resource and latest CertificateRequests as well as the data in Secret containing the certificate.

Let's Encrypt: staging or production server?

An Issuer is a custom resource (CRD) which tells cert-manager how to sign a Certificate. Following this howto (section 7) the Issuer will be configured to connect to the Let's Encrypt staging server, which allows us to test everything without using up your Let's Encrypt certificate quota for the domain name.

After debugging, you can safely issue a certificate by using LE's production server.

A video describing cert-manager YAML syntax and recommended by the author of this article is 📽️ Anton Putra's good one.

Conclusion

SSL certificate acquisition was made simple by Let's Encrypt's reputation as a reliable certificate authority. Together with cert-manager tool, ops can quickly and easily assure correct transport encryption and interoperability with already-existing parts like NGINX Ingress. In addition to the example mentioned above, cert-manager can help with trickier situations like those involving wildcard SSL certificates.

If you're interested in using letsencrypt outside of a kubernetes cluster, take a look at Caddy, a 43k ⭐ open source web server, and also at Certbot, a 29k ⭐ ACME client which is open source, too.

Ever tried using wireshark to monitor web traffic? Follow Aaron Phillips from Comparitech to learn how.

Safe connections to you!

Getting Started with Buffalo

Ant(on) Weiss — Tue, 13 Sep 2022 15:02:23 +0000

Intro - Rapid Software Development in the Modern World

(Feel free to skip to the tutorial)

This post is the first in a planned series about rapid software development in the modern world.

Rapid developer onboarding is something we've been dealing a lot with at Otomato software. But what really got us started on diving deeper into this - was my conversation with Elad Meidar a few months back.

Elad was talking to me about how non-trivial it has become to choose a stack in today's world. And let's say you've chosen your tools - only the most experienced developers really know how to set up CI, testing, deployment, instrumentation, monitoring, security, etc. correctly. A lot of Ops knowledge is involved in even getting things initially running.

Our current goal is to take that knowledge that we've accumulated and make it available as a service. And while we're doing that - we're exploring the tooling that's currently available.

Enter Buffalo

Buffalo is a framework, (or a tool) for rapid web development in Go. Cloud Native DevOps folks (and that's what we are at Otomato) have a soft spot for Golang and that's why I'm starting this series with Buffalo.

The official getting started section of Buffalo documentation is great but as I ran through it I noticed it lacks some operational details that I'm planning on exposing. Again there's Ops knowledge lurking in the dark!

Installations

Quite naturally one would need to install Go.
On a Mac:



brew install golang

On Ubuntu/Debian:



sudo apt update && sudo apt install golang

Note: on older systems (such as Ubuntu 20.04) you'll get a very old version of Go (1.13) by default when installing with apt. So instead - choose the download-n-extract option here.

For additional installation options go here

Do you do frontend?

Buffalo can generate both pure backend API services and fully-fledged webapps with frontend matter included. The frontend is in Javascript - so if you want that - you'll also need Node and either yarn or npm (the default).
On a Mac:



brew install nodejs

On Ubuntu/Debian:



sudo apt update
sudo apt install nodejs npm

Do you want containers?

Buffalo makes quite a few educated assumptions when generating your project. One of them is that you'll want to wrap your app in a container. You can, of course opt out, but why? So if you're going with the flow and enjoying the benefits of containerization - you probably already have Docker installed. If not - please do so now - we'll need it further along the tutorial.

Finally - bring in the Buffalo

On a Mac:



brew install gobuffalo/tap/buffalo

On Linux;



wget https://github.com/gobuffalo/cli/releases/download/v0.18.8/buffalo_0.18.8_Linux_x86_64.tar.gz
tar -xvzf buffalo_0.18.8_Linux_x86_64.tar.gz
sudo mv buffalo /usr/local/bin/buffalo

Create a project

Buffalo has a project scaffolding feature that allows us to generate a new app complete with:

a local git repository
a backend api
a db integration
a frontend
a Dockerfile
a CI pipeline.

Let's create a webapp called testr. It will be used to manage test assignments for new and existing trainees. (Did I mention we do technical training at Otomato too?)

The command to create a new project is buffalo new.
The default DB backend used by Buffalo is PostgreSQL.
We will be using Github for SCM, so we'll choose Github Actions as our CI provider.



buffalo new testr --ci-provider github

After buffalo shows us what it's bringing in and generating (quite a bunch of stuff really) it will say:



Initialized empty Git repository in /Users/username/git/testr/.git/
DEBU[2022-08-28T23:37:54+03:00] Exec: git add .
DEBU[2022-08-28T23:37:54+03:00] Exec: git commit -q -m Initial Commit
INFO[2022-08-28T23:37:54+03:00] Congratulations! Your application, testr, has been successfully generated!
INFO[2022-08-28T23:37:54+03:00] You can find your new application at: /Users/antweiss/git/testr
INFO[2022-08-28T23:37:54+03:00] Please read the README.md file in your new application for next steps on running your application.

So we 'll do just what it tells us to:



cd testr
git add .
git commit -q -m "Initial Commit"

Set up the DB

Before we can actually start developing our code there's
a need to spin up a database. We could, of course, use a managed DB but it would probably cost us a few bucks. So for local development it makes much more sense to run the DB in a container.

Let's run PostgreSQL (the default Buffalo DB backend):



docker run --name buffalo-postgres -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres

Note: we're running PostgreSQL with a very naive password here , which is fine for local development but not fit for anything production-like.
We're also exposing it on localhost:5432 - which is where a Buffalo app is configured to look for it by default.

These configurations are defined in a buffalo-generated file database.yml which we'll use shortly.

Running the DB container isn't enough. We also need to create a database for our app.

This can be done by entering the container and running good old SQL commands. But Buffalo creators recommend the use of Soda - a small and useful CLI utility that makes managing DBs easier.

Install soda:



go install github.com/gobuffalo/pop/v6/soda@latest

And create a DB:



soda create -a

Soda creates all the databases configured in the file database.yaml that Buffalo has generated for us.

Note: By default Buffalo uses its own ORM library called pop for DB integrations. Pop provides a wrapper for soda - so we can also run soda commands through buffalo aliases: buffalo pop create -a or buffalo db create -a.

Buffalo delivers even more useful DB stuff with the help of pop - like model generation. But we'll cover that in the follow-up post.

Start development

Buffalo provides us with a buffalo dev command which allows running our app with live reloading - i.e restarting the application server each time we change the code.

Let's run!



buffalo dev

Now we can visit http://localhost:3000 in browser and see our app running!

And - we're live!

The web UI we see is generated from testr/templates/home/index.plush.html using the plush templating engine. Also to be covered in a separate post.

Let's Have Some CI

As the final step of this walkthrough - let's push our code to Github and verify the generated CI pipeline works.

Generate a new Github repo

I heartily recommend using Github's gh cli tool:
From the testr directory run:



gh repo create --public <your-user-or-org>/buffalo-testr --push  --source .

This will create the repo and immediately push the code to it, which in turn starts the workflow defined in .github/workflows/test.yml:



name: test

on:
  push:
  pull_request:

permissions:
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:9.6-alpine
        env:
          POSTGRES_DB: testr_test
          POSTGRES_PASSWORD: postgres
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          - 5432:5432
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-go@v3
        with:
          go-version: ~1.18
          cache: true
      - name: setup
        run: |
          go install github.com/gobuffalo/cli/cmd/buffalo@latest
      - name: test
        env:
          TEST_DATABASE_URL: postgres://postgres:postgres@localhost:5432/testr_test?sslmode=disable
        run: |
          buffalo test

We can see that this workflow:

spins up a PostgreSQL service container
installs buffalo
runs buffalo test - which in turn creates the DB in the container and runs some tests:



[POP] 2022/09/12 15:24:04 info - dropped database testr_test
[POP] 2022/09/12 15:24:05 info - created database testr_test
pg_dump: error: connection to server at "127.0.0.1", port 5432 failed: FATAL:  database "testr_development" does not exist
[POP] 2022/09/12 15:24:05 info - Migrations already up to date, nothing to apply
[POP] 2022/09/12 15:24:05 info - 0.0102 seconds
[POP] 2022/09/12 15:24:05 warn - Migrator: unable to dump schema: open migrations/schema.sql: no such file or directory
time="2022-09-12T15:24:06Z" level=info msg="go test -p 1 -tags development testr/actions testr/cmd/app testr/grifts testr/locales testr/models testr/public testr/templates"
go: downloading github.com/gobuffalo/suite/v4 v4.0.3
go: downloading github.com/gobuffalo/httptest v1.5.1
go: downloading github.com/stretchr/testify v1.8.0
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading github.com/pmezard/go-difflib v1.0.0
ok      testr/actions   0.019s
?       testr/cmd/app   [no test files]
?       testr/grifts    [no test files]
?       testr/locales   [no test files]
ok      testr/models    0.012s
?       testr/public    [no test files]
?       testr/templates [no test files]

Voila! The workflow works. It doesn't create a Docker image for us (so no artifacts) - but it does run some basic integration testing.

Tests for Buffalo apps is another topic for yet another post.

To Sum Things Up

Buffalo is a well thought-out rapid development framework for full-stack apps or standalone backend APIs. It does pack quite a lot to get us started, but it still leaves the developer in the playground - without any clear guidelines regarding where and how to deploy their code for production.

And what are you using for rapid bootstrapping of new services?
What other rapid development frameworks would you like us to cover?

Let us know in comments - our research is only starting!

Ops Deliver Confidence

Ant(on) Weiss — Mon, 12 Sep 2022 08:37:40 +0000

What is confidence? It's a belief that whatever happens, whatever comes our way - we'll be able to handle it. Either by ourselves or with someone's help.

The world is far from predictable. We never really know what will happen the next moment. Believing otherwise would mean lying to ourselves. Because we don't have control of the outside world. The only thing a person can control is their body and mind. And even that - only to some extent.

The best service providers always sell confidence. Confidence that the room will be clean, confidence that the luggage will get picked up, confidence that the mail will be delivered.

It's the same with Ops service:

We cannot really prevent incidents from happening. Rarely can we promise 5 nines of uptime (it can be done but it's awfully expensive). What we can offer our customers is confidence.

Confidence that whatever incident occurs - it will be taken care of. Confidence that if we can automate a process - we'll put it on our backlog and prioritize it correctly.
That whenever we discover system's new failure mode - we'll create an alert and a remediation for it.

And we need to always remember to convey that confidence. In the way we document our processes. In how we publish our post-mortem analyses. In how we treat customers' tickets and requests. In the way we communicate our plans and yes - even our doubts.

Because when you convey confidence - you share it. You give it to your customer - so they can check one more worry off their list. And like us all - they have a lo-o-o-ng list.

That's how we strive to do business at Otomato - with maximum transparency, integrity and confidence. So our customers can rest assured their software will get delivered and keep running even when nobody's watching. And if anything goes down - we'll bring it back up.

We deliver confidence.

Admission Controllers in Action: Datree's Approach

Roman Belshevitz — Sun, 11 Sep 2022 09:19:45 +0000

In the eighth part, the author talked about admission controllers. In this, the ninth, we will see how ACs can be used for practical purposes.

At the same time, it may be considered that this is the second part of the review, so both of the parts will be marked with the appropriate #datree tag.

The originality of Datree's approach

In brief, Datree's integration enables you to check your resources against the defined policy a moment before you put them into a cluster... by leveraging an admission webhook! 😎

The webhook implemented with ValidatingWebhookConfiguration will detect operations such as CREATE, UPDATE or DELETE, and it will start a policy check against the configs related to each operation.

If any configuration errors are discovered, the webhook will refuse the action and show a thorough output with guidance on how to fix each error.

Every cluster operation that is tied up once the webhook is installed will cause a Datree policy check. If there are no configuration errors, the resource will get a green light🚦 to be applied or updated.

🔬 Datree is functioning well in a full-scale cluster and also in a k3s/k3d-baked one! It make a debugging process more suitable even for local development.

Let's go step by step

Following the Software-as-a-Service paradigm, Datree provides their users access to the misconfigurations' database and to the personal workspace at their website, where all the checks initiated by the user become aggregated.

This cheeky astronaut design will brighten up your day.

🎯 Step 1. Access token

They [want to] know everything about you! Well, relax, it is a joke. Sign up or log in, then grab your token to access datree programmatically. API access tokens are widespread in 2022, aren't they? 🔏

💰 There is only one token available when using the so-called Free Plan, enough for evaluation purposes (up to 4 Kubernetes nodes are supported; service access logs will be stored for two weeks).

🎯 Step 2. Set up your CLI environment

The following binaries must be installed on the machine: kubectl, openssl (required for creating a certificate authority, CA) and curl.

Assume everything is in place. Let's run:

$ DATREE_TOKEN=[your-token] bash <(curl https://get.datree.io/admission-webhook)

What you should see and what should happen to your cluster (yes, API requests are additionally encrypted):

🔑 Generating TLS keys...
Generating a RSA private key
Signature ok
subject=CN = webhook-server.datree.svc
Getting CA Private Key
/home/roman
🔗 Creating webhook secret tls...
secret "webhook-server-tls" deleted
secret/webhook-server-tls created
🔗 Creating core resources...
serviceaccount/webhook-server-datree created
clusterrolebinding.rbac.authorization.k8s.io/rolebinding:webhook-server-datree created
clusterrole.rbac.authorization.k8s.io/webhook-server-datree created
deployment.apps/webhook-server configured
service/webhook-server created
deployment "webhook-server" successfully rolled out
🔗 Creating validation webhook resource...
validatingwebhookconfiguration.admissionregistration.k8s.io/webhook-datree configured
🎉 DONE! The webhook server is now deployed and configured

🎯 Step 3. Protect your access token

Because your token is private and you don't want to store it in your repository, we advise setting or changing it using a different kubectl patch command:

$ kubectl patch deployment webhook-server -n datree -p '
spec:
  template:
    spec:
      containers:
        - name: server
          env:
            - name: DATREE_TOKEN
              value: "<your-token>"'

🎯 Step 4. Deploy something: magic will work for you

The author does not want to reinvent the wheel:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

Let's focus and look at the kubectl apply -f nginx-deployment.yaml routine's result (the deployment has been denied by AC):

$ kubectl apply -f nginx-deployment.yaml
Error from server: error when creating "nginx-deployment.yaml": admission webhook "webhook-server.datree.svc" denied the request: 
webhook-nginx-deployment-Deployment.tmp.yaml

[V] YAML validation
[V] Kubernetes schema validation

[X] Policy check

❌  Ensure each container has a configured CPU limit  [1 occurrence]
    - metadata.name: nginx-deployment (kind: Deployment)
💡  Missing property object `limits.cpu` - value should be within the accepted boundaries recommended by the organization

❌  Ensure each container has a configured CPU request  [1 occurrence]
    - metadata.name: nginx-deployment (kind: Deployment)
💡  Missing property object `requests.cpu` - value should be within the accepted boundaries recommended by the organization

❌  Ensure each container has a configured liveness probe  [1 occurrence]
    - metadata.name: nginx-deployment (kind: Deployment)
💡  Missing property object `livenessProbe` - add a properly configured livenessProbe to catch possible deadlocks

❌  Ensure each container has a configured memory limit  [1 occurrence]
    - metadata.name: nginx-deployment (kind: Deployment)
💡  Missing property object `limits.memory` - value should be within the accepted boundaries recommended by the organization

❌  Ensure each container has a configured memory request  [1 occurrence]
    - metadata.name: nginx-deployment (kind: Deployment)
💡  Missing property object `requests.memory` - value should be within the accepted boundaries recommended by the organization

❌  Ensure each container has a configured readiness probe  [1 occurrence]
    - metadata.name: nginx-deployment (kind: Deployment)
💡  Missing property object `readinessProbe` - add a properly configured readinessProbe to notify kubelet your Pods are ready for traffic

❌  Prevent workload from using the default namespace  [1 occurrence]
    - metadata.name: nginx-deployment (kind: Deployment)
💡  Incorrect value for key `namespace` - use an explicit namespace instead of the default one (`default`)


(Summary)

- Passing YAML validation: 1/1

- Passing Kubernetes (v1.21.5) schema validation: 1/1

- Passing policy check: 0/1

+-----------------------------------+-----------------------+
| Enabled rules in policy "Default" | 21                    |
| Configs tested against policy     | 1                     |
| Total rules evaluated             | 21                    |
| Total rules skipped               | 0                     |
| Total rules failed                | 7                     |
| Total rules passed                | 14                    |
| See all rules in policy           | https://app.datree.io |
+-----------------------------------+-----------------------+

Opening the link, you'll be redirected to your personal workspace.

🎯 Step 5. Is such rigor necessary?

You can audit reactive policies and review invocation history. If checks are too strict, unset some of the policies.

For example, not in every deployment you really need pre-configured container readiness probes [or CPU & memory limits].

Well, another tryout will be on edited YAML (Octopus may be your fellow):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
       app: nginx
  replicas: 2
  strategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.14.2
          ports:
            - containerPort: 80
          resources:
            requests:
              memory: 100Mi
              cpu: 100m
            limits:
              memory: 200Mi
              cpu: '1'

As we really want to use the default namespace and have no fears with it, let's disable Prevent workload from using the default namespace policy in Datree web UI.

Also we may want to liberate us from Ensure each container has a configured readiness probe and Ensure each container has a configured liveness probe policies.

Et voilà! 🎭

$ kubectl apply -f nginx-deployment-advanced.yaml
deployment.apps/nginx-deployment created

Now all checks are passed successfully and the admission controller have got us a green light🚦 and allowed the deployment!

🎯 Step 6. Hey, do not climb where it is not necessary

If you want datree to disregard a namespace, add the label admission.datree/validate=skip to its configuration.

$ kubectl label namespaces default "admission.datree/validate=skip"

How to wipe traces

To delete the label and resume running the datree webhook on the namespace again:

$ kubectl label namespaces default "admission.datree/validate-"

Uninstall the webhook

Copy the following command and run it in your terminal to remove the webhook:

$ bash <(curl https://get.datree.io/admission-webhook-uninstall)
validatingwebhookconfiguration.admissionregistration.k8s.io "webhook-datree" deleted
service "webhook-server" deleted
deployment.apps "webhook-server" deleted
secret "webhook-server-tls" deleted
clusterrolebinding.rbac.authorization.k8s.io "rolebinding:webhook-server-datree" deleted
serviceaccount "webhook-server-datree" deleted
clusterrole.rbac.authorization.k8s.io "webhook-server-datree" deleted
namespace/kube-system unlabeled
namespace "datree" deleted

Summing up what was said

As you could understand, the possibilities of Kubernetes API are quite wide. The author hopes that he not only prepared an overview of a useful solution, but also explained the theoretical aspects of its functionality.

Responsible Approach to Communicating With the API Server: Admission Controllers

Roman Belshevitz — Fri, 02 Sep 2022 12:25:12 +0000

A bit of theory

RBAC and Network policies are two fundamental security elements of Kubernetes that you probably already know about if you work with it. These mechanisms are helpful for enforcing fundamental guidelines regarding what operations different users or services within your cluster are permitted to carry out.

However, there are situations when you require more policy features or granularity than RBAC or network policies can provide. Alternatively, you might want to run additional checks to verify a resource before allowing it to join your cluster.

Admission Controllers (ACs) allow you to add additional options to the work of Kubernetes to change or validate objects when making requests to the Kubernetes API.

🖼️ Pic source: Giant Swarm

The image shows the various parts that make up the API component. The request initiates communication between the API and the admission controller. The authorization module determines whether the request issuer is permitted to carry out the operation after the request has been authenticated. The admittance magic kicks in once the request have been duly approved.

If the controller rejects the request, then the entire request to the API server is rejected and an error is returned to the end user.

To activate controllers discussed, you must specify the names of the controllers in the form of a list when creating or updating a cluster. After that, kube-apiserver will be started or restarted with the --enable-admission-plugins option and access controllers set.

Passing a controller that is not available for the current version of Kubernetes will return an appropriate error.

What exactly can ACs be?

🔬 In a scope of implementation

Admission controllers that are built into and made available by Kubernetes itself are known as static admission controllers. Not every one of them is turned on by default. The cloud companies also grab some of them or restrict some of them for their own usage. You can enable and utilize them if you are the owner of your Kubernetes deployment. Some examples:

LimitRanger makes sure that any of the restrictions listed in the LimitRange object in a namespace are not broken by incoming requests. Use this admission controller to impose those restrictions if you are utilizing LimitRange objects in your Kubernetes setup. Applying default resource requests to pods without any specifications is also possible with this AC.

AlwaysPullImages changes the image pull policy for every new Pod. This is useful, for example, in multi-tenant clusters to ensure that only those with the credentials to fetch private images can access them. Without this admission controller, after an image has been pulled to a node, any pod from any user can use it just by knowing the image's name without any authorization checks. This feature must be enabled in the cluster.

NamespaceLifecycle enforces that a namespace that is undergoing termination cannot have new objects created in it, and ensures that requests in a non-existent namespace are rejected.

And there are dynamic ones. See details below.

🔬 In a scope of request processing

There are two types of dynamic admission controllers in Kubernetes. They work slightly differently. Saying shortly, one just validates the requests, and the other modifies it if it isn’t up to spec.

⚙️ The first type is the validating admission controller ValidatingAdmissionWebhook, which proxies the requests to the subscribed webhooks. The Kubernetes API registers the webhooks based on the resource type and the request method. Every webhook runs some logic to validate the incoming resource, and it replies with a verdict to the API.

In case the validation webhook rejects the request, the Kubernetes API returns a failed HTTP response to the user. Otherwise, it continues with the next admission.

⚙️ The second type is a mutating admission controller MutatingAdmissionWebhook, which alters the resource that the user has submitted so that default values can be set, or the schema can be verified. The API can have mutation webhooks attached by cluster administrators so that they can execute them similarly to validation.

Hooks! Hooks are everywhere!

Any resource type, including those that are pre-built like pods, jobs, or services, may be the primary resource type for a controller. The issue is that most built-in resources, if not all of them, already come with associated built-in controllers. In order to prevent having many controllers update the status of a shared object, custom controllers are frequently built for special resources.

If resources are merely Kubernetes API endpoints, writing a controller for a resource is just a fancy way to bind a request handler to an API endpoint!

Conditional resource modification can be implemented using a so-called webhook, which is essentially an API endpoint.

It is possible to configure dynamically what resources are subject to what admission webhooks via ValidatingWebhookConfiguration or MutatingWebhookConfiguration kinds.

Both are available in admissionregistration.k8s.io/v1 API version.

$ kubectl api-versions | grep admiss
admissionregistration.k8s.io/v1
admissionregistration.k8s.io/v1beta1

How would I activate admission controllers?

To use before changing cluster objects, the Kubernetes API server flag --enable-admission-plugins accepts a comma-delimited list of AC plugins. For instance, the following command line activates the LimitRanger and NamespaceLifecycle admission control plugins:

kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger

⚠️ Note: You may need to apply the parameters in different ways depending on how your Kubernetes cluster is installed and how the API server is launched. For instance, if Kubernetes is deployed using self-hosted Kubernetes, you may need to alter the manifest file for the API server and/or the systemd Unit file if the API server is installed as a systemd service.

⚠️ Note: API kind admissionregistration.k8s.io/v1beta1 became deprecated in 1.22+

Public cloud providers' implementation

In this case, everything is already set up for you.

To learn more about using dynamic admission controllers with Amazon EKS, see the Amazon EKS documentation.

Azure AKS Policy, Microsoft's implementation of OPA Gatekeeper, is another interesting thing. Involving AC webhooks, if there are problems in the admission control pipeline, it can block numerous requests to the API server.

The VMware Tanzu team followed a similar path in their Tanzu Kubernetes Grid (TKG).

Of course, OPA Gatekeeper itself is a separate and extensive topic, so more on that another time.

In the ninth article of the series, the author will talk about how smart people were able to translate the theory described above into a useful solution.

Be careful and stay tuned!

Many thanks to Leonid Sandler, Douglas Makey @douglasmakey Mendez Molero, Luca 🐦 @LucaDiMaio11 Di Maio, Kristijan Mitevski and W.T. Chang!

Virtual Kubernetes Clusters: What Are They Needed For?

Roman Belshevitz — Mon, 29 Aug 2022 18:58:50 +0000

Developer Wishlist Never Ends

Imagine you can

have many virtual clusters within a single cluster, and
they are much cheaper than the traditional Kubernetes clusters, and
they require lower management and maintenance efforts.

Sounds intriguing, eh? This makes v/clusters ideal for running experiments, continuous integration, and setting up sandbox 🧪 environments.

So, Loft Labs created such a solution written natively in Golang and made it an ~2k⭐ open source.

What's under the hood?

On top of other Kubernetes clusters, virtual clusters are fully functional Kubernetes clusters. Virtual clusters utilize the worker nodes and networking of the host cluster, as opposed to completely distinct "real" clusters. They schedule all workloads into a single namespace of the host cluster and have their own control plane. Virtual clusters divide a single physical cluster into several distinct ones, much like virtual machines do.

🖼️ Right click, don't even think too long.

Only the essential Kubernetes components - the API server, controller manager, storage backend (such as etcd, sqlite, mysql, etc.), and - optionally - a scheduler—make up the virtual cluster itself. In order to minimize virtual cluster overhead, vcluster builds by default on k3s, a fully functional, certified, lightweight Kubernetes distribution that compiles the Kubernetes components into a single binary and disables by default all unnecessary Kubernetes features, such as the pod scheduler or specific controllers.

Other Kubernetes distributions, such k0s and vanilla k8s, are supported in addition to k3s. In addition to the control plane, the virtual cluster also includes a Kubernetes hypervisor that simulates networking and worker nodes. Between the virtual and host clusters, this component syncs a few key resources that are crucial for cluster functionality:

Pods: All the virtual cluster's started pods are rewritten before being launched in the virtual cluster's namespace in the host cluster. Environment variables, DNS, service account tokens, and other configurations are updated to point to the virtual cluster rather than the host cluster. In the pod, it appears that the virtual cluster rather than the host cluster is where the pod is started.
Services: On the namespace of the virtual cluster in the host cluster, all services and endpoints are rewritten and generated. The service cluster IPs are shared by the host cluster and virtual cluster. This implies that there are no performance consequences when a service in the host cluster is accessed from within the virtual cluster.
PersistentVolumeClaims: In the event that persistent volume claims are generated in the virtual cluster, they will be modified and generated in the host cluster's namespace. The relevant persistent volume data will be synchronized back to the virtual cluster if they are bound in the host cluster.
ConfigMaps & Secrets: Only ConfigMaps and secrets mounted to pods within the virtual cluster will be synced to the host cluster; all other ConfigMaps and secrets will only be retained within the virtual cluster.
Other Resources: Deployments, StatefulSets, CRDs, service accounts, etc. do not sync with the host cluster; instead, they only reside in the virtual cluster.

Who lost the magic mirror?

For each pod with the spec.nodeName value it encounters inside the virtual cluster, vcluster by default creates a false node. Because vcluster does not by default have RBAC permissions to access the real nodes in the host cluster because doing so would require a cluster role and cluster role binding, those false nodes are produced. Additionally, each node will get a false kubelet endpoint that will either send requests to the real node or rewrite them to keep virtual cluster names intact.

Vcluster supports multiple modes to customize node syncing behavior. For a detailed list of the resources that may have been synced, see details here in the docs.

The hypervisor also proxies some Kubernetes API calls, including pod port forwarding or container command execution, to the host cluster in addition to synchronizing virtual and host cluster resources. It essentially performs the function of the virtual cluster's reverse proxy.

To ensure proper network operation for the virtual cluster, resources like Service and Ingress are synced by default from the virtual cluster [down] to the host cluster.

There are never too many levels of abstraction

Certain resources (such as CRDs or RBAC policies) reside cluster-wide, and you can’t isolate them using namespaces. For instance, it is not feasible to install an operator simultaneously in multiple versions inside a same cluster.

$ kubectl api-resources --namespaced=false|true

Although Kubernetes itself already offers namespaces for various settings, their use of cluster-scoped resources and the control plane is constrained.

In many circumstances, virtual clusters are also more stable than namespaces. In its own data store, the virtual cluster produces its own Kubernetes resource objects. These resources are unknown to the host cluster.

This kind of isolation is good for resilience. The necessity for access to cluster-scoped resources like cluster roles, shared CRDs, or persistent volumes still exists for engineers who adopt namespace-based isolation. Each team that depends on one of these shared resources will probably experience failure if an engineer destroys something in it.

Finally, virtual cluster configuration is independent of physical cluster configuration. This is excellent for multi-tenancy because it allows you to easily create a fresh environment or amazing demo applications. 😎

How it looks in your CLI

Create file vcluster.yaml:

vcluster:
  image: rancher/k3s:v1.23.5-k3s1   # Choose k3s version

Then, install helm chart using vcluster.yaml for chart values:

helm upgrade --install my-vcluster vcluster \
  --values vcluster.yaml \
  --repo https://charts.loft.sh \
  --namespace host-namespace-1 \
  --repository-config=''

Access:

Get the admin tool

curl -s -L "https://github.com/loft-sh/vcluster/releases/latest" | sed -nE 's!.*"([^"]*vcluster-linux-amd64)".*!https://github.com\1!p' | xargs -n 1 curl -L -o vcluster && chmod +x vcluster;
sudo mv vcluster /usr/local/bin;

Then, connect:

# Connect and switch the current context to the vcluster
vcluster connect my-vcluster -n my-vcluster

# Switch back context
vcluster disconnect

You have an option to create a separate kubeconfig to use instead of changing the current context:

vcluster connect my-vcluster --update-current=false

Or you may execute a command directly with vcluster context without changing the current context:

vcluster connect my-vcluster -- kubectl get namespaces
vcluster connect my-vcluster -- bash

Usage:

# Run any kubectl, helm, etc. command in your vcluster
kubectl get namespace
kubectl get pods -n kube-system
kubectl create namespace demo-nginx
kubectl create deployment nginx-deployment -n demo-nginx --image=nginx
kubectl get pods -n demo-nginx

Cleanup:

helm delete my-vcluster -n vcluster-my-vcluster --repository-config=''

What if you're planning some serious thing?

Well, the stock K8s distribution is compatible with high availability in vcluster. What is meant by high availability? Well, one of the entities is to make etcd database more robust. The second one is to boost syncer's performance. As mentioned above, vcluster uses a so-called syncer which copies the pods that are created within the virtual cluster to the underlying host cluster.

🪲TL;DR #1: etcd uses a leader-based consensus protocol for consistent data replication and log execution. Etcd cluster members elect a single leader, all other members become followers. The cluster elects a new leader automatically when one falls out of favor. Once the incumbent fails, the election does not take place immediately. Since the failure detection methodology is timeout based, electing a new leader takes roughly an election timeout.

🪲TL;DR #2: Why it is recommended to have a minimum of three instances in an etcd cluster, is well described here, a first hand info.

Currently, vcluster's high availability setup does not allow single binary distributions like k0s and k3s.

Create a values.yaml with the following structure in order to operate vcluster in high availability mode:

# Enable HA mode
enableHA: true

# Scale up syncer replicas
syncer:
  replicas: 3

# Scale up etcd
etcd:
  replicas: 3

# Scale up controller manager
controller:
  replicas: 3

# Scale up api server
api:
  replicas: 3

# Scale up DNS server
coredns:
  replicas: 3

To summarize

A virtual Kubernetes cluster that is fully functioning can be built using vcluster! The underlying K8s cluster's namespace is where each vcluster operates. It provides better multi-tenancy and isolation than conventional namespaces, and it is less expensive than building independent, fully-fledged clusters.

Virtual clusters can be a good option to running numerous instances of k3s, or k0s side by side, but they cannot exist on their own without a host cluster.

Compared to fully independent Kubernetes clusters, they are faster, lighter, and simpler to reach. Therefore, give using a virtual cluster a shot if you're tired of having to reset your local or CI/CD Kubernetes clusters all the time. However, this is a topic for a completely different story, much sadder than what you just read.

Be in good & non-ghost shape! 👻

Many thanks to Viktor 🐦@vfarcic Farcic and Mauricio 🐦@salaboy Salatino for inspiration!

How to Stop Rampant Kubernetes Cluster Growth

Roman Belshevitz — Thu, 25 Aug 2022 18:20:00 +0000

Some lyrics as an introduction

The Edvard Munch's famous painting "Scream" was first presented to the public at the Berlin exhibition in December 1893. It was conceived as part of the "Frieze of Life" - a program cycle of paintings about the spiritual life of a person. Munch wrote about him:

“The Frieze of Life” is conceived as a series of paintings connected with each other, which together should give a description of a whole life. A winding line of the coast passes through the picture, behind it is the sea, it is always in motion, and under the crowns of trees there is a diverse life with its sorrows and joys. Frieze is conceived as a poem about life, love and death.

The author of this brief, of course, will not talk about the spiritual life, but about practical approaches that prevent thoughts about the terrible and otherworldly and save the nerves of engineers.

The essence of the Ops' problem

Kubernetes was originally designed to support the consolidation of workloads on a single cluster. However, there are many problematic scenarios that require a multi-cluster approach to optimize performance. These may include workloads across regions, fault propagation radius limits, compliance issues, harsh multi-user environments, security, and custom software solutions.

Unfortunately, this multi-cluster approach poses management challenges, as the complexity of managing a Kubernetes cluster only increases as the size of the cluster increases. The end result is a phenomenon called cluster sprawl, which occurs when the number of clusters and workloads grows and is not managed coherently.

The solution to this problem lies in the early and rapid identification and implementation of the best management practices in order to avoid serious work in the future.

What is Kubernetes governance?

In order to ensure accountability, transparency, and responsibility, a well-defined collection of rules, policies, and procedures is referred to as governance.

Governance is also about synchronizing clusters and providing centralized policy management. Kubernetes' governance is defined as a set of rules created with policies that need to be enforced across all clusters. This is a critical component for large enterprises running Kubernetes.

Typically, this process means applying matching rules across Kubernetes multi-clusters, as well as applications running in those clusters. And while managing Kubernetes may seem insignificant, it pays off in the long run, especially if implemented in a large organization.

Assume that the enterprise continues to increase the number of clusters in use and does not apply management. These clusters will exist under different rules, which will create a huge amount of extra work for the teams in the near future.

Fortunately, there are only a few very important components to building a successful Kubernetes governance.

Creating successful Kubernetes governance

When considering a successful Kubernetes governance strategy, the first component is to ensure good multi-cluster management and monitoring. You must maintain control over how and where clusters are created and configured, as well as which software versions can be used.

🔧 Well-built observabilty

Application development and operations teams should be able to centrally view and manage clusters to better optimize resources and troubleshoot. Solutions in this area are developed, for example, by Red Hat, Platform9, Fairwinds and even Rancher Labs. Improved management practices and greater transparency can also save a company from the headaches of a range of security risks and performance issues down the road.

🔧 RBAC strategies

Next, enterprises must have an authentication and access control system in place. Having centralized authentication and authorization will help an organization streamline the login process and help keep track of user activity. This will allow application development and operations teams to ensure that the right people are doing important tasks in real time.

🔧 Policy management

Finally, to govern Kubernetes, enterprises must optimize policy management. Companies need to think about how Kubernetes will impact their development culture and work on finding the right balance of business agility and development. Ultimately, governance (with the appropriate level of flexibility) ensures that businesses can meet customer needs and deploy mission-critical services in a consistent and consistent manner.

In Kubernetes, Admission Controllers enforce policies on objects during create, update, and delete operations. Admission control is fundamental to policy enforcement in Kubernetes.

Admission controllers allow you to enforce the adherence to certain practices such as having good labels, annotations, resource limits, or other settings.

Being the CNCF project, Open Policy Agent (OPA) is a great tool to develop and implement such policies at scale throughout an organization. Every request will go through the OPA, as illustrated below, and will be decided depending on the policies established for the Kubernetes cluster. The request will be carried out if it complies with the policy. The OPA will reject the request if it violates the established policies.

As a good practice, by deploying OPA as an admission controller, you can:

Require specific labels on all resources.
Require container images come from the corporate image registry.
Require all pods specify resource requests and limits.
Prevent conflicting Ingress objects from being created.

Goals to achieve

But what should be the goals of governance? Where should it be enforced and tested? The four most effective management objectives are security policy, network management, access control, and image management. Let's look at each of these goals one by one:

🎯 Security policy

In security policies for governing Kubernetes, it is important to restrict user access to pods in clusters. Cluster users should have well-defined access based on their role.

To do this, enterprises must implement a security policy that will have rules and conditions related to access and privileges. In this policy, they must specify that containers have read-only access to the file system and that containers and child processes cannot be subject to privilege changes.

🎯 Network management

Network policy plays a very important role in determining which services can communicate with each other. Here, companies must determine which modules and services can interact with each other and which should be isolated. This also applies to module security in Kubernetes management.

The right approach is aimed at controlling traffic within Kubernetes clusters. This approach can be based on modules, namespaces, or IPs, depending on management requirements.

Each popular CNI plugin uses a different type of configuration for the network setup. For example, Calico uses layer 3 networking paired with the BGP routing protocol to connect pods.

Cilium configures an overlay network with eBPF on layers 3 to 7. Along with Calico, Cilium supports setting up network policies to restrict traffic.

🎯 Administration and access control

In access control, when configuring role-based access control (RBAC) policy, administrators need to restrict access to cluster resources. Using Kubernetes objects such as Role, ClusterRole, RoleBinding, and ClusterRoleBinding, they need to fine-tune access to cluster resources appropriately.

Because permissions granted by a ClusterRole apply across the entire cluster, you can use ClusterRoles to control access to different kinds of resources than you can with Roles. These include:

Cluster-scoped resources such as nodes
Non-resource REST Endpoints such as /healthz
Namespaced resources across all Namespaces (for example, all Pods across the entire cluster, regardless of Namespace).

After creating a Role or ClusterRole, you have to assign it to a user or group of users by creating a RoleBinding or ClusterRoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: testadminclusterbinding
subjects:
  - kind: ServiceAccount
    name: myaccount
    namespace: test
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

🎯 Image management

Using public Docker images can increase the speed and flexibility of application development, but there are many vulnerable Docker images, and using them in a production cluster can be very risky.

Image management is also part of Kubernetes governance. All images that will be used in the cluster must be pre-scanned for vulnerabilities. There are several approaches to finding vulnerabilities. How and where an organization checks for vulnerabilities depends on its preferred workflows. However, it is recommended that you test your images before deploying them to a cluster.

Hacker activity has increased exponentially in recent years, and loopholes in systems continue to be discovered. Therefore, it is very important for companies to be vigilant when implementing practices to ensure that they only use official, clean, and verified Docker images on a cluster.

Threat actors can mount sophisticated assaults employing previously dependable third-party artifacts as an attack vector by using malicious scripts or malware concealed in a container image. Static, pattern-based, or signature-based scanners are not effective against this kind of attack because it only appears during runtime.

By evaluating the attack kill chain and running images in a secure hosted sandbox environment, several security solutions can reduce this risk.

To examine images in a running state both before and after the image is checked into a registry, these tools, i.e. trivy by Aqua Security, are frequently incorporated into CI/CD processes. Malicious behavior and unfulfilled policy requirements can mark an image for registry deletion or prevent check-in entirely.

Instead of conclusion

Thus, the author has given in brief the directions needed to better govern Kubernetes and ensure the security of important enterprise systems and data, as well as to limit cluster growth and possible disorder. Stay strong and focused!

The author is thankful to Arthur Chiao, Oleg Chunikhin (CNCF), Tomas Fernandez (Rendered Text / Semaphore), Mike Jordan (Coredge), Kristijan Mitevski and Steven Zimmerman (Aqua Security) for their contribution to comunity.

How to Shut Down Kubernetes Pod Gracefully

Roman Belshevitz — Wed, 17 Aug 2022 21:33:00 +0000

Essence of the question

As the operating processes on your cluster are represented by pods, Kubernetes offers graceful termination when pods are no longer required. By imposing a default grace period of 30 seconds after you submit a termination request, Kubernetes offers graceful termination.

The steps listed below make up a typical Kubernetes Pod termination:

To end the Pod, you send a command or make an API call.
The duration of time after which a Pod is to be regarded as dead is reflected in Kubernetes changes the Pod status (the time of the termination request plus the grace period).
When a pod enters the Terminating state, Kubernetes stops transmitting traffic to it.
The SIGTERM signal from Kubernetes instructs the Pod to stop operating.

Pods can be terminated for a variety of reasons over the lifecycle of an application. In Kubernetes, these reasons include user input via kubectl delete or system upgrades, among others.

🖼️ A larger picture is here. Also, you may open it in a new browser's tab to zoom in.

On the other hand, a resource problem could lead to its termination.

Misunderstood action

With some configuration, Kubernetes in this scenario enables gentle termination of the operating containers in the Pod. Let's first understand how the delete / termination process proceeds before moving on to the setup.

Once the user issues the kubectl delete command, the command is transmitted to the API server, where it is removed from the endpoints object. As we saw when creating the pod, the endpoint is crucial to receive updates when providing any services.

In this action, the endpoint will be immediately removed from the control plane while readiness probes are disregarded. This will start events on the DNS, ingress controller, and kube-proxy.

As a result, all of those components update their references and stop forwarding traffic to the IP address. Please be aware that while this procedure may be speedy, the component may occasionally be preoccupied with other tasks. As a result, a delay might be anticipated, and the reference won't be updated right once.

At the same time, the Pod's status in etcd is changed to Terminating.

The polling alerts Kubelet, which then delegates the activity to components like pod creation.

Here

By using the Container Storage Interface to unmount all volumes from the container (CSI).
Relinquishing the IP address to the Container Network Interface and disconnecting the container from the network (CNI).
To the Container Runtime Interface, destroy the container (CRI).

Note: Kubernetes updates the endpoints after waiting for the kubelet update to provide the IP data during the Pod creation. However, when the Pod terminates, it simultaneously updates the kubelet and removes the endpoint.

Premature termination?

How is this a problem? The hitch, however, is that sometimes it takes some time for components to update endpoints. In this situation, if the pod is killed before endpoints have been propagated, we would experience downtime. Yet why?

As previously indicated, ingress and other high-level services are still not changed, thus traffic is still forwarded to the removed pod. However, we might believe that Kubernetes should update the modifications throughout the cluster and prevent such a problem.

But it unquestionably is not.

Kubernetes does not validate that the changes on the components are current because it distributes the endpoints using endpoint objects and sophisticated abstractions like Endpoint Slices.

We cannot guarantee a 100% application uptime due to this possibility of downtime. The only way to accomplish this is to wait until the Pod is destroyed before updating the endpoint. We made assumptions based solely on what we observed, but is that really possible? Let's investigate.

API magic?

For that, we must have a thorough understanding of what transpires in containers when the delete command is sent.

Pod receives the SIGTERM signal after getting kubectl delete. The SIGTERM signal is sent by default by Kubernetes, which then waits 30 seconds before forcibly ending the process. Therefore, we can enable a setting that requires us to wait before acting, such as:

Prior to leaving, wait a while.
For another 10 to 20 seconds, the traffic will still be processed.
Close all backend connections, including those to databases and WebSockets.
At last, finish the procedure.

You can add or modify terminationGracePeriodSeconds in your pod definition if your application needs longer time (more than 30 seconds) to terminate.

You can add a script that will wait for a while before exiting. In this instance, Kubernetes exposes a pre-stop hook in the pod before the SIGTERM was executed. You can implement this as the following:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
    - name: nginx
      image: nginx
      ports:
        - name: nginx
          containerPort: 80
      lifecycle:
        preStop:
          exec:
            command: ["sleep", "10"]

This option would make the kubelet wait for 30 seconds before advancing the SIGTERM, although it should be noted that this might not be enough since your application might still be handling some older requests. How do you avoid them? This can be done by including terminationGracePeriodSeconds, which will cause the container to wait longer before being terminated. The final manifest will appear:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
    - name: nginx
      image: nginx
      ports:
        - name: nginx
          containerPort: 80
      lifecycle:
        preStop:
          exec:
            command: ["sleep", "10"]
        terminationGracePeriodSeconds: 45

Command line hacks

This setting ought to make it easier for the app to handle all requests and shut down connections. This will prevent a shutdown that is forced.

When manually deleting a resource, you may also modify the default grace period by providing the --grace-period=SECONDS argument to the kubectl delete command. For instance:

# kubectl delete deployment test --grace-period=60

What about the Rolling updates?

Pods are removed for yet another reason when we upgrade or deploy a new version. What happens if you upgrade your app from, let's say, v1.1 to v1.2 while operating a v1.1 version with 3 replicas?

A Pod is created using the new container image.
Eliminates a current Pod.
Awaits the completion of the Pod.
Until every pod has been transferred to the new version, repeat.

Ok, this ensures the deployment of the new version. But what about old pods? Does Kubernetes wait until all the pods have been deleted?

The answer is no.

The old version pods will be gracefully terminated and eliminated as it moves forward. However, as old ones are continuously being removed, there may occasionally be twice as many Pods.

Putting an end to ongoing processes

Even though we have taken all necessary precautions, some apps or WebSockets may require prolonged service, or we may be unable to halt if any lengthy operations are in progress or requests are being made. Rolling updates will be at danger throughout that period. How can we overcome?

There are two options.

The terminationGracePeriodSeconds can be increased to a few hours. Or modifying a current deployment, or starting a new one.

Option 1

If you choose to do so, the pod's endpoint will be out of reach. Also take note that you must manually monitor those pods and cannot use any monitoring software to track them. All monitoring programs gather data from endpoints, and once that data is withdrawn, all monitoring tools will behave similarly.

Option 2

Your old deployment will still be there when you establish the new one. Therefore, all the lengthy processes will continue to run until they are finished. You manually eliminate the previous processes after you can see that they have finished.

You can configure an autoscaler to scale your deployment to zero replicas (third-party tools like KEDA can simplify this) when they run out of jobs if you want to eliminate them automatically. Furthermore, you can keep earlier pods running for longer than the grace period by using this every time.

A less obvious but superior option is to start a fresh Deployment for each update. While the most recent Deployment serves the new users, current users can keep using updates. You can gradually reduce the replication and retire old Deployments as users disconnect from old Pods.

The author hopes this is a helpful article. Consider your options and choose the option that best satisfies your needs!

Pictures courtesy 🐦 @motoskia & 🐦 @foxutech

Some great schematic diagrams can be found here. Thanks, Daniele Polencic.

More to read: How Kubernetes Reinvented Virtual Machines (in a good sense), a great article by Ivan Velichko.

Datree, a Tool Which Really Shifts Your Cluster Security Even More Left

Roman Belshevitz — Fri, 05 Aug 2022 11:45:00 +0000

The idea of writing this article was born by the author after getting acquainted with two entities: this and this repository. 😎

So, noone wants to throw poop in the pot

As part of the GitOps approach, increasingly used today, for the majority of the time, you will need to update your Helm charts (Deployment, Service, or Pod) or YAML manifests for Kubernetes and then either immediately apply the changes to the Production, Staging, or Test environments, depending on the situation.

Worrying about whether the new Kubernetes manifest YAML modification which you're deploying in the production environment? Will it actually work during a release or deployment?

Well, a slightly nervous situation, illustrated (the author just remembered this) about six years ago.

No matter how skilled or experienced you are as an engineer, this will be a scary experience if you are unsure about your changes.

So, is there a way to validate your YAML manifests and Helm charts before they are used in production? Fortunately, the answer is “yes”. A tool named Datree can be used to validate your Kubernetes manifest before making any mods!

What makes Datree special

While Kubescape is a utility that improves Kubernetes security by scanning clusters and detecting already deployed YAML files that are not compliant with security standards, or subject to vulnerabilities (that the engineer might not have known about), Datree was made 💡to prevent Kubernetes misconfigurations from reaching production with Datree’s automated policy checks for your pipeline.

Datree, the 5.8k⭐ open-source CLI tool empowers engineers to write more stable configurations, so they can actually sleep at night. It allows you to integrate it into any CI flow and trigger it whenever you want, such as each time team make a change or submit a pull request, for example.

🎯 Is it safe in terms of data leaks?

Creators' statements can be interpreted in such a way that the appraisal of Datree's policies is purely local.

🖼️ A larger picture is here. Also, you may open picture in a new browser's tab to zoom in.

Since the CLI runs the policy check on your system, your files and their contents are not transferred to their backend. To their backend, which is used to show your policy check history on your dashboard, the tool only sends metadata.

What is not less important: to do checks, Datree does not need to be connected to the cluster. It has an offline mode as well.

🎯 What to verify?

The following three assertions are verified by Datree:

Is the file a well-composed YAML file?
Schema for Kubernetes: is this a valid Kubernetes file?
Is the file compliant with your Kubernetes policy?

🎯 Policy as code

Policy-as-a-code, similar to Infrastructure-as-a-Code, is the concept of using declarative code to replace actions that require using a user interface. Proven software development best practices, such as version control, collaboration, and automation, can be applied by encoding policies in code.

🎯 Centralized policy

This idea refers to managing distributed policy execution from a single point. This gives the policy owner simple control over the rules that Datree evaluates during each run without adding more work to the operation. You can control the central policy by logging into the dashboard.

🎯 Rules' flexity

To fit your preferences, you can toggle any of the 50+ built-in rules "ON" or "OFF" in the dashboard. When a rule is turned on or off, any policy checks run against that policy will automatically be updated (via account token). This eliminates the need for the policy owner to manually update every device (cluster host) linked to the policy.

You can pick from dozens of tried-and-true rules that address various Kubernetes resources and use cases, which are tied to: containers, cron jobs, workloads (running apps), networking, security, API deprecation, ArgoCD rollouts, CVEs described by the NSA and to other Kubernetes syntax nuances.

🎯 Custom rules

You can write any tests you like and run them against your Kubernetes setups to check for rule violations in addition to the tool's built-in rules. The built-in rule engine supports both YAML and JSON declarative syntax because it is based on JSON Schema.

Installation and general usage

All you need to do is run the command below:

curl https://get.datree.io | /bin/bash

Once installed, you can easily use Datree to check the security of Kubernetes manifests.

The syntax is as follows:

datree test [k8s-manifest-file]

When a check is started, it goes through 3 main stages:

YAML validation;
checking Kubernetes charts;
checking Kubernetes policies.

For example, when checking demo manifest, the command would look like this:

datree test ~/.datree/k8s-demo.yaml

Let's see the output:

$ datree test ~/.datree/k8s-demo.yaml
>>  File: .datree/k8s-demo.yaml

[V] YAML validation
[V] Kubernetes schema validation

[X] Policy check

❌  Ensure each container image has a pinned (tag) version  [1 occurrence]
    - metadata.name: rss-site (kind: Deployment)
💡  Incorrect value for key `image` - specify an image version to avoid unpleasant "version surprises" in the future

❌  Ensure each container has a configured liveness probe  [1 occurrence]
    - metadata.name: rss-site (kind: Deployment)
💡  Missing property object `livenessProbe` - add a properly configured livenessProbe to catch possible deadlocks

❌  Ensure each container has a configured memory limit  [1 occurrence]
    - metadata.name: rss-site (kind: Deployment)
💡  Missing property object `limits.memory` - value should be within the accepted boundaries recommended by the organization

❌  Ensure workload has valid label values  [1 occurrence]
    - metadata.name: rss-site (kind: Deployment)
💡  Incorrect value for key(s) under `labels` - the vales syntax is not valid so the Kubernetes engine will not accept it


(Summary)

- Passing YAML validation: 1/1

- Passing Kubernetes (1.20.0) schema validation: 1/1

- Passing policy check: 0/1

+-----------------------------------+-----------------------+
| Enabled rules in policy "Default" | 21                    |
| Configs tested against policy     | 1                     |
| Total rules evaluated             | 21                    |
| Total rules skipped               | 0                     |
| Total rules failed                | 4                     |
| Total rules passed                | 17                    |
| See all rules in policy           | https://app.datree.io |
+-----------------------------------+-----------------------+

From the output, you can see detailed information about the violations present in the manifest. This gives engineers the necessary guidance to resolve them.

Each Datree policy check is performed using the default policy, which includes 50+ built-in rules.

To configure the policy, you must return to the terminal and register by clicking on the link provided at the end of the output.

kubectl plugin

This kubectl plugin extends the Datree CLI's capabilities to allow scanning resources within your cluster for misconfigurations.

The Kubectl plugin can be installed using the command:

kubectl krew install datree

What you should see:

$ kubectl krew install datree
Updated the local copy of plugin index.
Installing plugin: datree
Installed plugin: datree
\
 | Use this plugin:
 |  kubectl datree
 | Documentation:
 |  https://github.com/datreeio/kubectl-datree
 | Caveats:
 | \
 |  | Before using this plugin, the Datree CLI needs to be installed.
 |  | See https://hub.datree.io/ for quick and easy installation.
 | /
/

Let's try to check our cluster's namespace (having this release by Bitnami deployed in it).

Right now, the default K8s schema version that is checked when you run a policy check is 1.2. To check for deprecated APIs before deploying you K8s manifest, change the default K8s version in your dashboard to match your culster server version.

Author's cluster playground is k3s/k3d, so API version is

$ k3d version
k3s version v1.22.7-k3s1 (default)

Another (and more common) approach:

$ kubectl version --short
Server Version: v1.22.7+k3s1

🥁 Let's perform a scan routine with Datree now.

$ kubectl datree test -s "1.22.7" -- service my-release-drupal

(Summary)

- Passing YAML validation: 1/1

- Passing Kubernetes (1.22.7) schema validation: 1/1

- Passing policy check: 1/1

+-----------------------------------+-----------------------+
| Enabled rules in policy "Default" | 21                    |
| Configs tested against policy     | 1                     |
| Total rules evaluated             | 21                    |
| Total rules skipped               | 0                     |
| Total rules failed                | 0                     |
| Total rules passed                | 21                    |
| See all rules in policy           | https://app.datree.io |
+-----------------------------------+-----------------------+

The following cluster resources in namespace 'default' were checked:

service/my-release-drupal

You may see a very similar report on Datree's SaaS dashboard:

If there are no rules failed, then you get a green mark! Well done.

helm plugin

This plugin is used to check charts against Datree policy. The mentioned plugin can be installed using the command:

helm plugin install https://github.com/datreeio/helm-datree

To run a Datree policy check on Helm charts, run a command with the following syntax.

helm datree test [CHART_DIRECTORY]

If you need to pass arguments to your template, add -- in front of them, as shown below:

helm datree test [CHART_DIRECTORY] -- --values values.yaml --set name=prod

CI integration

To integrate Datree into CI/CD, you can follow the example below. You need to follow these steps:

Get your account token (you can find it in the dashboard's Settings).
Set DATREE_TOKEN as secret/environment variable
Add Datree to your CI flow via token as shown (i.e., for GitHub).

🖼️ A larger picture is here. Here is the example of involving action:

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

env:
  DATREE_TOKEN: ${{ secrets.DATREE_TOKEN }} 

jobs:
  k8sPolicyCheck:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Run Datree Policy Check
        uses: datreeio/action-datree@main
        with:
          path: '**/*.yaml'
          cliArguments: '--only-k8s-files'

Instead of a conclusion

This brief article on using Datree to perform security checks on Helm charts and Kubernetes manifests is now complete. The author hopes readers can all agree that Datree can be used to prevent configuration problems in Kubernetes that can result in production cluster failure.

Wishing you to have clean YAMLs and green marks, folks! ✅

The author expresses his sincere gratitude to Noaa 🐦@BarkiNoaa Barki, Eyar 🐦@eyarzilb Zilberman, Anais 🐦@urlichsanais Urlich and Scott 🐦@chiefmartec Brinker.