Forem: Innovation Process Technology AG (ipt)

Beyond the Pod: Why wasmCloud and WebAssembly Might Be the Next Evolution of the Platform

Jakob Beckmann — Tue, 21 Oct 2025 05:08:44 +0000

Over the past few months I have invested some time to contribute to an open source project I find fascinating: wasmCloud. As a platform engineer and architect, I am very familiar with how software platforms are typically built in practice. However, with the ubiquity of Kubernetes, you run the risk to being stuck in the "doing it the Kubernetes way" line of thinking. But then again, are there any better ways? This is where wasmCloud caught my attention. A modern platform building on proven concepts from Kubernetes, but with some significant differences. In this article I want to introduce wasmCloud, how it compares to Kubernetes, what its internal architecture looks like, and what ideas are, in my humble opinion, a step up from "the Kubernetes way of things".

Before getting started, I need to get some things out of the way. This article will make quite a few comparisons to Kubernetes and bytecode interpreters like the JVM. If you are unfamiliar with these technologies, it might make sense to have a short look at what these are. Considering you clicked on this article, I am however guessing that you are familiar with them and have some experience in platform engineering practices, either as a poweruser of a platform, or as a designer and developer of one.

Moreover, I want to thank the company I work for, ipt, for allowing me to invest time to learn about new technologies such as wasmCloud. Not only is contributing to open source a great way to pay back a community powering the modern world, it is also a huge passion of mine. Being able to help the development of such projects during paid worktime enables me to learn so much on emerging technologies, and maybe help build the revolutionary tools of tomorrow.

So... wasmCloud!? I have been interested in WebAssembly ever since it promised to replace JavaScript, a language I personally consider as extremely poorly designed (someone once told me it was designed in three days, so no wonder there). While WebAssembly is very far from doing anything close to replacing JavaScript in the browser, it has evolved into something else: an application runtime and a potential replacement for containers.

WebAssembly as a Platform Foundation

Modern platforms nearly all build on top of containers as their foundational element to run executable code. This is a logical evolution from Docker's meteoric growth, and the ecosystem that grew around its open standards (such as the OCI - Open Container Initiative). While containers provide a huge step in terms of ease of use, standardization, and security compared to shipping raw artefacts to virtual machines, as was the case before them, they do have some shortcomings.

First and foremost, containers are not composable. In part due to their flexibility, they do not offer standard ways of expressing how the world should interact with them at runtime, or what they rely on to perform their functionality. This means that containers are typically deployed as REST-based microservices, where containers communicate with one another over a network using APIs agreed upon outside of the container standards. This lack of standardization makes building reusable components more challenging than it has to be. Moreover, each container essentially needs a server, authentication, authorization, and more to run. This results in quite some waste in the compute density of the platform, with lots of compute wasted on boilerplate.

Moreover, while containers are a huge step in the right direction in terms of security, they are not quite as secure as most people are led to believe. Containers are "allow by default" constructs, which take quite some work to properly harden.

Finally, due to how containers are typically built, their startup times are not that great. It is not abnormal to see container start times in the dozens of seconds. This does not bother people very much because containers are mostly used to run long running processes (since we need these REST APIs everywhere). However, a large part of containers are mostly idle, waiting for some API request to come in. If one considers that workloads could be called (and thus the process started) only when needed, startup times over 100ms is considered slow.

This is where WebAssembly comes it. WebAssembly addresses these challenges. Composability is addressed by the component model.

WebAssembly: The Component Model

The component model is a way that WebAssembly modules can be built with metadata attached to them which describe their imports and exports based on a rich type system. Moreover, they are composable such that a new component can be built from existing components as long as the imports of one are satisfied by the exports of another. This means that components can interact with one another via direct method/function calls, whose specification is fully standardized. This interface specification is declared in a language known as the WebAssembly Interface Types (WIT) language. An example of a WIT specification of a component relying on a system clock can be seen below:

package wasi-example:clocks;

world mycomponent {
    import wall-clock;
}

interface wall-clock {
    record datetime {
        seconds: u64,
        nanoseconds: u32,
    }

    now: func() -> datetime;

    resolution: func() -> datetime;
}

WIT can be compared to the Interface Definition Language (IDL) from gRPC but for wasm components.

This declaration says that the component relies on an interface wall-clock (it imports the interface) which defines two functions: now and resolution. Both take no arguments and return a datetime object consisting of a seconds and nanoseconds field. This component could then be composed with any other component which exports this wall-clock interface.

If this were a container which would rely on accessing some API, we would need to read a non-standardized documentation of the container image, and then read up on other containers to ensure they provide APIs that match the ones called by the first container.

The WebAssembly component model can essentially be seen as a form of contract-based programming to formalize interfaces between WebAssembly core modules.

WebAssembly: Secure by Default

Whereas containers provide some form of security by namespacing processes and filesystems, WebAssembly actually sandboxes modules such that they cannot affect one another, or the host they run on. By default a WebAssembly module cannot perform any privileged action and needs to be granted explicit permission. I will not dive deeper into the details of this or I might loose myself in a rant on how software security in the modern day and age is abysmal.

WebAssembly: Performance

WebAssembly's main goal is performance. This means that WebAssembly modules run fast, but also that loading modules and starting them is much faster than containers. This has proven to be very useful already, for instance in use cases such as serverless computing, where hyperscalers heavily rely on WebAssembly as a runtime to reduce cold start times, and reduce the delay in function calls.

Considering the idea to avoid having long running servers providing REST APIs and move to raw function calls on short running modules, having extremely short start times is imperative.

Alright, so we can see that WebAssembly can be a great choice for the foundation runtime of a platform. So where are platforms leveraging this? Well, actually, quite some "platforms" leverage this idea already. For instance, SpinKube does exactly this, enabling to run WebAssembly functions on Kubernetes. However, you still interact with these functions via a REST call. Another example is Kubewarden, leveraging WebAssembly modules to evaluate policies. While some might argue that this is not a platform, Kubewarden provides a runtime for arbitrary programs, including their scheduling and deployment. Sounds like a platform to me.

Finally: wasmCloud! wasmCloud is probably what people would consider the closest to a full blown platform to run WebAssembly modules. In other words, what Kubernetes is to containers, wasmCloud is to WebAssembly components. It provides a way to deploy, schedule, link, and lifecycle WebAssembly components on a distributed platform.

wasmCloud Architecture

Let us look at the wasmCloud architecture a little.

This section will contain quite a few comparisons to Kubernetes concepts.

Generally, the wasmCloud architecture can be seen as quite similar to the Kubernetes architecture, with the difference being that wasmCloud does not provide as much flexibility in swapping out building blocks as Kubernetes does. This makes sense as it is a more nascent technology and is currently more opinionated.

As a reference, here is the diagram wasmCloud uses to provide an overview of the platform:

As one can see, the architecture is essentially a set of hosts connected via a so called "lattice". Thus, the architecture distributes the runtime over a set of compute instances in order to achieve resilience against hardware/compute failures. The principle is identical to the one from Kubernetes, providing a cluster in order to be able to quickly shift payloads on the platform to different nodes in case of node failures.

Hosts

wasmCloud hosts are the foundation of the compute platform that is provided. They are the equivalent of Kubernetes nodes and provide a WebAssembly runtime for components to run on. Just as with Kubernetes nodes, application developers will rarely need to worry about the hosts other than for deployment affinities and the like.

In practice, hosts can be anything from a virtual machine, an IoT device, or even a pod running on Kubernetes. In fact, hosting wasmCloud on Kubernetes is a relatively straight forward way to get started with the technology, providing wasmCloud as an application runtime, while providing services via Kubernetes.

Lattice

The wasmCloud lattice is its networking layer. This can seem a bit strange when considering that this a NATS instance.

For those unfamiliar with NATS: it is an event streaming component similar to Kafka, but provides additional features such as a key values store, an object store, and publish-subscribe capabilities.

Having a NATS instance as the "networking layer" confused me quite a lot at first. However, one has to remember that thanks to the component model, we no longer require HTTP/TCP network calls for our components to interact with one another. Thus we don't necessarily need an IP to address a component we want to reach. Of course NATS itself will require a physical network to run on in order to distribute events to its different instances, but wasmCloud then only needs to use NATS.

Essentially, every component exposing a function becomes a subscriber to a queue for this function on NATS. Other components can then call this function via wRPC (gRPC for WebAssembly) by publishing a call to some subject. This is quite different from Kubernetes networking, where calls need to know the location of the callee in the network. Using a subject-based addressing model simplifies deployment and improves scaling and resilience.

As a user of wasmCloud, you do not need to worry about this though. How function calls are preformed under the hood is abstracted away from the user.

This distributed networking aspect is one of the superpowers of wasmCloud, as one does not need to worry about how to address a component on the platform. However, it can also introduce strange behaviour in some cases. For instance, on Kubernetes, it's common sense that a HTTP call to a different pod running on the cluster can fail. On wasmCloud however, if the interface we are calling from a different component returns some type, we use the component like a raw function call in our components code. What if that call fails, not because of the called component but due to a networking issue? In the current implementation of wasmCloud this will lead to a panic in the caller. As this is typically not the desired outcome, efforts are underway to design an adapted way how the interfaces need to be designed to handle failures in the transport layer. On top of that, function calls might change such that might avoid using NATS as a transport layer if the component being called in on the same host and the caller.

Capabilities

This is where Kubernetes and wasmCloud start differing in their philosophy. Thanks to the standardized way interfaces can be declared in the component model, one can describe an abstract interface which provides some functionality, without providing an implementation. This is what capabilities are. They are abstract interfaces that describe some useful functionality, such as reading and writing to a key value store, or retrieving some sensitive information from a secured environment. These capabilities are published on wasmCloud for applications to use.

An application developer can then write a component that makes use of that interface if he/she needs that functionality. He/she does not need to worry about how this capability is implemented. He relies on the "contract" provided by the capability.

In my opinion, while this is quite challenging to grasp initially, this is what makes wasmCloud so promising. Having worked on many platforms in the past, the main challenge is always how additional services can be provided on top of raw platforms such as Kubernetes in a way that makes then highly standardized while easily consumable. In the current state of platform engineering, this quickly becomes a question of good product management. Unfortunately, doing this correctly is surprisingly difficult. Capabilities provide a technical solution to this, with the only limitation being complete incompatibility with existing software.

Providers

A provider is a specific implementation of a capability. For instance, taking the example of the capability enabling the reading and writing to a key value store, a provider might implement this by having a ValKey instance backing the capability. Another provider might implement the very same capability using NATS, Redis, or even an in-memory key-value store.

Abstracting the provider away from the consumer via a capability enables the platform to swap providers based on needs. Of course performing such a swap might be quite complex, for instance involving a data migration from NATS to ValKey. However, the beauty is that the applications do not require any changes as would be the case in traditional platforms.

It should be noted that the provider might run completely outside of wasmCloud itself. However, wasmCloud also provides internal providers that are backed into the hosts themselves, providing functionality such as logging or randomness.

Components

Components refer to the WebAssembly payload that contain your business logic. In the traditional sense, this is your application. However, in wasmCloud lingo, an application is a set of interlinked components including all information about what capabilities they require.

Applications

Applications are an abstraction enabling to declaratively define a combination of components, capabilities, and providers together into a deployable unit. Applications are based on the open application model (OAM) and should thus look quite familiar to people working with Kubernetes. In terms of definition, they are similar to a Kubernetes Deployment, describing not only the deployment unit (component or pod in the Kubernetes context), but also its replication, affinities, links to capabilities, etc.

It should be noted that in wasmCloud v2, applications are re-worked to be much more closely modelled after Kubernetes Deployments and ReplicaSets. Version 2 drops the idea of Applications alltogether and uses Workload, WorkloadReplicaSets, and WorkloadDeployments objects. These are also no longer linked to the OAM. In all likelihood we will write another blog post showcasing the capabilities of composition provided by version 2 in the future.

wadm

The wasmCloud Application Deployment Manager (wadm) manages Applications. It can be seen as the deployment controller from Kubernetes for wasmCloud Applications. It essentially orchestrates the deployment of components, capabilities, their links, etc. on the platform. This construct will also be dropped with wasmCloud version 2.

Verdict

With a decent understanding of the architecture we can now get an idea of the uses of wasmCloud in the real world. While I have not yet run anything productive on wasmCloud, I have played with the platform a lot over the past few months, and have come to really appreciate some of its innovative ideas.

Thus, to summarise my experience: wasmCloud is a relatively new platform and provides interesting new approaches to how inter-component communication can be modeled. On top of that, it does it while building on open standards such as WebAssembly and the component model, such that the business logic of your application remains portable. While these new concepts are very promising, wasmCloud still suffers from a couple drawbacks:

For people unfamiliar with WebAssembly, it has a quite steep learning curve. This is highly accentuated for people unfamiliar with existing platforms such as Kubernetes.
The set of supported providers and capabilities is extremely small to date. This will of course grow as adoption increases, but currently early adopters will have to write their own providers most of the time and will not be able to rely on third-party components.
As wasmCloud shifts more responsibility to the platform level, it will require a strong platform team to operate this with low developer friction. This can be an issue as finding highly skilled platform engineers is quite difficult at the moment. However, the team behind wasmCloud is focused on making application delivery as frictionless as possible.
Finally, I am not sure I currently understand the security model wasmCloud uses to authenticate and authorize calls between components. While I am not sure this is a drawback, it does not yet feel as intuitive as Kubernetes simple yet relatively powerful RBAC. I will have to dive deeper into this to have a final opinion on it though (another blog post might follow on this).

That Time We Found a Service Account Token in my Log Files

Vincent von Büren — Thu, 04 Sep 2025 07:59:04 +0000

⚠️ Disclaimer
This article assumes you're already somewhat familiar with Kubernetes concepts (Pods, ServiceAccounts) and the basics of JSON Web Tokens (JWTs).

It was a Tuesday.

Nothing special - just your average day as a platform engineer. My team's notifications were mercifully quiet, and I thought, "Perfect, I can finally clean up that old Helm chart that's been bothering me."

I opened the repo of the underlying image written in Go to double-check the config before merging.

Before I even got far, a colleague — Martin Odermatt — pinged me:

“You might want to take a look at this…”

He had spotted something concerning in the code:

log.Println("SA Token:", token)

Wait. What?

A debug statement. Still in production code. Logging an actual Kubernetes ServiceAccount token. Not cool...

I paused. My heart rate didn’t. Curious but mostly horrified, I took the token Martin had found and decoded the payload in my shell:

{
"iss": "https://kubernetes.default.svc.cluster.local",
"kubernetes.io/serviceaccount/namespace": "payments",
"kubernetes.io/serviceaccount/secret.name": "payments-token-6gh49",
"kubernetes.io/serviceaccount/service-account.name": "payments-sa",
"kubernetes.io/serviceaccount/service-account.uid": "f9a2c144-11b3-4eb0-9f30-3c2a5063e2e7",
"aud": "https://kubernetes.default.svc.cluster.local",
"sub": "system:serviceaccount:payments:payments-sa",
"exp": 1788201600,   // Sat, 01 Aug 2026 00:00:00 GMT
"iat": 1756665600    // Fri, 01 Aug 2025 00:00:00 GMT
}

Default audience claim. A 1-year expiry.

As we dug deeper, Martin Odermatt pointed out the underlying issue: this was a legacy ServiceAccount token, and we should be using projected tokens instead.

This "bad boy" wasn't just a dev leftover - it was a high-privilege token with zero constraints floating around in plaintext logs!

What This Article Covers

In this post, I'll guide you through:

The inner workings of Vault authentication with JWT and Kubernetes methods
What Kubernetes ServiceAccounts and their tokens are, and how they’re (mis)used
How projected ServiceAccount tokens fix many of the hidden dangers of older token behavior
Why you should start adopting token projection and Vault integration today

We'll cover real-world use cases, implementation tips, and common pitfalls - so you don't end up like I did, staring at a:

log.Println("SA token:", token)

...and wondering how close you just came to a security incident.

Why This Matters

To really understand why that log statement gave me chills, we need to unpack a few core concepts:

What is a JWT?
How do Kubernetes ServiceAccounts and their tokens work?
And what role do these tokens play in authenticating to systems like Vault?

Let's start with the fundamentals.

What Is a JWT?

If you've been around authentication systems long enough, you've probably seen one of these beasts:

eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9...

This is a JSON Web Token (short: JWT).

It's a compact, URL-safe format for representing claims between two parties. They're used everywhere: web apps, APIs, and yes — inside your Kubernetes cluster.

A JWT consists of three parts:

Header – declares the algorithm used to sign the token (e.g. RS256)
Payload – contains the claims (who you are, what you're allowed to do, etc.)
Signature – a cryptographic seal that verifies the payload hasn't been tampered with

Claims are the heart of a JWT — key-value pairs that describe who the token refers to and what it can be used for.

They can be:

Standard claims defined by the spec (e.g., iss, sub, exp, aud)
Custom claims added by the issuer for domain-specific needs

Closer Look at `aud`

The audience (aud) claim tells who the token is meant for. Think of it as the intended recipient.

Example: Imagine a Coldplay concert ticket. It says valid for Stadium X on 01-09-2025. You can't take the same ticket and use it at Stadium Y — they'll reject it (...trust me, I tried).

A JWT works the same way:

If the token has "aud": "https://kubernetes.default.svc", then only the Kubernetes API server should accept it.
If some other service receives that token, the aud won't match and the token must be rejected.

Without this check, a token could be misused anywhere that trusts the signing key. With aud, it's scoped to the right system.

Kubernetes and ServiceAccounts

Kubernetes is an open-source platform that orchestrates containers at scale. At its heart is the Pod — the smallest deployable unit.

But every pod needs an identity. That's where ServiceAccounts come in.

ServiceAccounts 101

Every Pod references a ServiceAccount (default if none is set), but a token is only mounted if enabled
Kubernetes mounts the identity at:

/var/run/secrets/kubernetes.io/serviceaccount/token

That token is a JWT, signed by the Kubernetes control plane
It lets the pod authenticate with the API server — and sometimes even external systems like Vault

The Catch

Until recently, these tokens came with dangerous defaults:

Long-lived (often valid for a year)
Previous to Kubernetes v1.24, there was no default audience set (https://kubernetes.default.svc)
Automatically mounted into every pod, even if unused

Enter Vault: The Gatekeeper of Secrets

HashiCorp Vault is your cluster’s paranoid librarian:
it stores API keys, certs, passwords — and only hands them out when it's sure you should have them.

How? Authentication methods.

Vault Authentication Methods

Username & password
AppRole
LDAP
Kubernetes
JWT

Let's zoom into the last two.

Kubernetes Auth Method

Pod sends its mounted ServiceAccount token to Vault
Vault validates it against the Kubernetes API
If valid, Vault maps it to a policy

This is simple and works well when Vault runs inside the cluster.

JWT Auth Method

Vault verifies the JWT itself (signature, claims, expiration)
No need for Kubernetes API access
More portable

Rule of thumb:

Use Kubernetes if Vault runs inside your cluster and simplicity matters
Use JWT if you want portability, stronger boundaries, and flexibility

Projected Tokens: Because It's 2025

Old tokens were static and long-lived — exactly what we were looking at here. As Martin pointed out during the investigation, projected tokens are designed to fix this entire class of problems.

Instead of mounting a one-year token into every pod, Kubernetes can now generate short-lived, audience-bound tokens on demand.

What You Get

Short TTL (e.g. 10 minutes)
Audience restrictions (aud: vault)
Automatic rotation by kubelet
No automatic mounting into pods

Example Pod with Projected Token

apiVersion: v1
kind: Pod
metadata:
  name: projected-token-test-pod
  namespace: demo
spec:
  serviceAccountName: projected-auth-sa
  containers:
    - name: projected-auth-test
      image: demo/vault-curl:latest
      command: ["sleep", "3600"]
      volumeMounts:
        - name: token
          mountPath: /var/run/secrets/projected
          readOnly: true
  volumes:
    - name: token
      projected:
        sources:
          - serviceAccountToken:
              path: token
              expirationSeconds: 600
              audience: vault

Why Vault Loves This

Vault's JWT auth method is tailor-made for projected tokens:

It parses and verifies the JWT signature (via a configured PEM key or JWKS endpoint)
Validates all claims (aud, sub, exp, iss) locally
Issues secrets only if every check passes

Minimal dependencies. Strong claim validation. Secure, verifiable checks.

Back to the Log

Imagine you stumble upon this in a Go app:

log.Println("Auth Token:", token)

Old world: a one-year, cluster-wide token with no audience. A time bomb.
New world: a 10-minute token, scoped to Vault, rotating automatically.

It's still bad to log tokens — but at least it's not catastrophic.

Try It Yourself: Vault + K8s AuthN Lab

I've built a hands-on demo repo where you can test this locally with KIND (Kubernetes in Docker) and Vault Helm charts:

👉 GitHub: VincentvonBueren/erfa-projected-sa-token

What's Inside

KIND cluster with Vault
Both Kubernetes and JWT auth methods enabled
Vault policies and roles
Four demo pods:
- Kubernetes auth method
- JWT with static token
- JWT with projected token
- JWT with wrong audience (failure demo)

Final Drop 🎤

If your pods still run with default, long-lived tokens:
you’re one debug log away from giving away the keys to your cluster.

Projected tokens aren't optional. They're essential.
Adopt them today — and stop shipping security disasters.

Acknowledgment

The discovery of the exposed ServiceAccount token — and the push towards using projected tokens — came from my dear fellow engineer Martin Odermatt, whose input significantly shaped this investigation and motivated me to tell this story.

Dissecting Kubewarden: Internals, How It's Built, and Its Place Among Policy Engines

Jakob Beckmann — Mon, 07 Jul 2025 05:31:30 +0000

Kubernetes offers amazing capabilities to improve compute density compared to older runtimes such as virtual machines. However, in oder to leverage the capabilities of the platform, these tend to host applications from various tenants. This introduces a strong need for properly crafted controls and well-defined compliance to ensure the tenants use the platform correctly and do not affect one another. The RBAC capabilities provided out of the box by Kubernetes are quickly insufficient to address this need. This is where policy engines such as Kubewarden come into play. In this post we will look at how Kubewarden can be leveraged to ensure correct usage of a platform, how it compares to other policy engines, and how to best adopt it.

Policy Engines

Kubernetes provides role-based access control (RBAC) out of the box to control what actions can be performed against the Kubernetes API. Generally, RBAC works by assigning sets of roles to users or groups of users. Capabilities are attached to these roles, and users having a role obtain these capabilities. This simple mechanism is very powerful, mostly because it is quite flexible while allowing a simple overview of a user's capabilities. However, in the case of Kubernetes, the definition of capabilities is very restricted. Roles only allow or deny access to Kuberenetes API endpoints, but to not allow control based on payload content. This means that these capabilities are mostly restricted to CRUD operations on Kubernetes primitives (e.g. Deployments, Ingresses, or custom resources). Unfortunately, this is often not enough.

For instance, it is quite common to allow users to perform actions on some primitives under specific conditions. An example would be that creating Deploymentss is only allowed as long as its name follows some convention and the pods its creates are not privileged and set proper resource requests/limits. The naming convention cannot be enforced by standard RBAC controls as these have no possibility to represent more complex logic. Controlling the configuration of the pods created by a Deployment is a validation of the payload pushed to the API, and is thus not supported either.

Note: Security contexts and resources on pods can be controlled via methods such as Security Context Constraints or Pod Security Policies and ResourceQuotas. However, these do not reject the creation of the deployment, but will only block the creation of the pods themselves. It is therefore possible to apply a Deployment that is known to not allow the creation of pods. In my personal opinion this is not ideal, as it does not fail early.

These scenarios is where policy engines come into play. They utilise Kubernetes' Dynamic Access Control mechanisms to enable cluster administrators to manage permissions using more complex logic. The exact capabilities of policy engines can vary greatly as these are essentially arbitrary software that validates or mutates Kubernetes requests. However, the majority of major policy engines work similarly. They tend to implement the operator pattern, enabling the configuration of policies using Kubernetes custom resources. In this blog post we will have a look at Kubewarden in more detail, and how it compares to other engines.

Kubewarden Architecture

Kubewarden leverages WebAssembly (WASM) to enable extremely flexible policy evaluation. Essentially, Kubewarden can be seen as a WASM module orchestrator where policies are deployed as serverless functions that get called when necessary. The result of these WASM functions then determines whether an API request against Kubernetes is allowed, denied, or altered (mutated).

This similee can also help explain Kubewarden's architecture. Essentially, the Kubewarden controller (operator) manages policy servers and admission policies. Policy servers can be seen as hosts for the serverless execution of functions, whereas admissions policies are the functions themselves. Therefore, in order to perform policy validation, one needs at least one policy server running to host the policies one wants to enforce. The controller then takes care of configuring the runtime (policy server) to properly run the adequate policy executable with the appropriate inputs when a policy needs to be evaluated. The diagram below illustrates this:

As policies are WASM modules, they can themselves support configuration. This makes policy reuse a major feature of Kubewarden. Complex logic can be contained in the WASM module while exposing some tuning as configuration, allowing a policy to perform a relatively generic task. To understand this better, let us have a look at such a policy:

apiVersion: policies.kubewarden.io/v1
kind: ClusterAdmissionPolicy
metadata:
  annotations:
  name: "cel-policy-replica-example"
spec:
  module: registry://ghcr.io/kubewarden/policies/cel-policy:v1.0.0
  backgroundAudit: true
  mode: protect
  mutating: false
  policyServer: default
  rules:
    - apiGroups: ["apps"]
      apiVersions: ["v1"]
      operations: ["CREATE", "UPDATE"]
      resources: ["deployments"]
  settings:
    variables:
      - name: "replicas"
        expression: "object.spec.replicas"
      - name: maxreplicas
        expression: int(5)
    validations:
      - expression: "variables.replicas <= variables.maxreplicas"
        messageExpression: "'the number of replicas must be less than or equal to ' + string(variables.maxreplicas)"
  namespaceSelector:
    matchLabels:
      environment: test

In this example, we are using a WASM module which evaluates a Common Expression Language (CEL) expression to define our policy. Evaluating a CEL expression is not something we want to implement every time ourselves. Thankfully, Kubewarden provides this as a WASM module on their ArtefactHub. Thus we do not need to implement anything and can reuse that module. It is referenced on the module line above. Of course we also need to actually define the CEL expression that should be the heart of the policy rule. This is done within the settings block. Note how we can use object internals (such as replicas defined in a Deployment) in the validation expression. Finally, we need to define on what objects this policy should be evaluated. In order to do this, we provide rules that tell Kubewarden on what Kubernetes API endpoints to trigger the policy, and additionally provide information about which namespaces should be affected by the policy with a namespaceSelector. The remaining options configure the following:

backgroundAudit: informs Kubewarden to report on this policy for objects that are already deployed. In this case, we validate the replicas on created or updated Deployment objects. However, there might already be Deployments on the cluster that violate the policy before we start enforcing it. This option will tell Kubewarden to provide reports on such violations.
mode: Kubewarden supports enforcing policies (in protect mode), or monitoring the cluster (in monitor mode). Using the monitor mode can be interesting when investigating how people use the Kubernetes cluster or providing them with warnings before enforcing policies.
mutating: policies can also mutate (change) requests. In this case we are only performing validation to potentially reject requests. Thus we set mutating to false.
policyServer: as explained above, Kubewarden can manage many policy servers. This simply informs the controller on which policy server this specific policy should be deployed.

As one can see based on the sample policy above, while Kubewarden technically uses programs as policies, it is usually not necessary to write any code to use Kubewarden. This is thanks to its strong focus on module configuration and re-usability. The above CEL module alone already enables the configuration of a very wide range of policies. On top of that, other modules shared on ArtefactHub provide more specific validations or mutations that might incorporate more complex logic. If this is not enough, policy groups (a feature we will not cover in this post) can be utilised to combine other policies and express more complex logic as well. Finally, if one has very specific needs that cannot be addressed by any of the publicly shared modules, one can still fall back to writing code and building ones own module with fully arbitrary logic. How such policies can be written, in actual code, might follow in a separate blog post.

The above architecture of Kubewarden is what makes it stand apart from most other policy engines. Generally policy engines contain the logic fully in the controller, only exposing configuration via the custom resource. Since Kubewarden can essentially execute arbitrary WASM bytecode, it is not bound by the expressiveness of the custom resource declaration.

All this considered, is Kubewarden the best choice for a policy engine and should be used in all scenarios?

Comparison

There are many other policy engines out there, such as Kyverno, Gatekeeper, or Polaris. So why would you choose Kubewarden over any other?

As explained above, Kubewarden provides unprecedented flexibility, thanks to the way it evaluates its policies. This has the massive advantage that you will never reach a point that you have a policy that you would like to enforce but are restricted by the policy engine itself. However, it also has some drawbacks. The primary one being complexity. Writing WASM modules is not for the fainthearted, as WebAssembly is not yet incredibly mature, and most developers will not be familiar with it. The complexity issue can however be sidestepped as the vast majority of policies can be expressed using off-the-shelf WASM modules provided by Kubewarden.

Another aspect that often needs to be considered in enterprise contexts, is support. Kubewarden is an open source project that is loosely backed by SUSE (as it was originally developer for its Rancher offering). Thus enterprise support is only available via a SUSE Rancher Prime. Other tools such as Kyverno are not only more mature, but offer more flexible enterprise support (via Isovalent).

Finally, another aspect to consider is the featureset of a policy engines. Not all policy engines support mutating requests, and are thus much more restricted in their use. However, in this category Kubewarden offers all the features typically desired from policy engines. Some engines such as Kyverno support more features such as synchronizing Secret objects. While this can be useful, it is, in my humble opinion, not a feature for a policy engine.

Of course, there are also personal preference aspects to consider. As an example, Kubewarden and Kyverno handle policy exceptions very differently. Kubewarden has matchers that can be defined as part of the policy itself, which allow to exclude some resources from being validated. Kyverno on the other hand uses a separate CRD called PolicyException. Both have advantages and disadvantages.

Verdict

Kubewarden is a very interesting piece of software. Its internal architecture enables it to be incredibly flexible, at the cost of complexity. However, due to a smart concept of WebAssembly module re-use, that complexity is mostly under the hood, unless one wants or needs to dive deep. In my opinion, Kubewarden can be an absolutely great consideration when ones operates very large Kubernetes clusters what might have quite exceptional requirements. However, even in these cases, I would recommend starting very slow, and slowly building up to the complexity Kubewarden can hold in store.

If you do not operate a large Kubernetes fleet, or expect to have rather standard requirements in terms of how you want to restrict access to you cluster(s), you might be better off with more mature and simpler tools like Kyverno. Getting support for these tools is likely to also be much simpler.

A large part of the complexity of Kubewarden also comes from all that is required to even run this in an enterprise context. Unless you allow pulling WASM modules directly from the internet, you will also need a registry to host OCI packaged modules. On top of that, should you decide to write your own modules, you will need a process to do this, and build knowhow in that area. These are some of the aspects I hope to cover in a follow up post.

Your Cluster Deserves Better Traffic Management. Enter: Gateway API.

Ignacio de los Rios — Thu, 05 Jun 2025 09:07:17 +0000

A Kubernetes cluster is like a beautifully designed city—with excellent plumbing, stable buildings, and all essential services—but no roads leading in or out. No visitors, no deliveries, no exits. If you want your application or services to interact with the outside world (say, users), you’ll need to pave some highways. Kubernetes has traditionally offered a few ways to do this—some better than others. Now, there’s a new infrastructure project in town: the Gateway API—and it's here to stay.
In this post, we’ll explore Kubernetes’ latest external access mechanism and show how it integrates with Cert-Manager to handle certificates. But before diving into this new world, let’s briefly review the most common ways to expose services in Kubernetes.

Current Alternatives

NodePort: Exposing a Service on a Node’s Port

A NodePort service is the most basic way to expose a service externally. Kubernetes opens a specific port on each node, and any traffic to a node’s IP on that port is forwarded to the service inside the cluster. Simple, but with notable limitations.

LoadBalancer: External Load Balancers for Services

A LoadBalancer service goes a step further by integrating with external load balancing infrastructure. When you create a Service of type LoadBalancer, Kubernetes asks the cloud provider to provision an external load balancer (e.g., AWS ELB, GCP Load Balancer, Azure Load Balancer). The service receives an external IP or hostname that forwards traffic to the backing Pods.
This works well in managed cloud environments but scales poorly and doesn’t work out of the box for bare-metal or on-prem clusters—unless you add a software load balancer like MetalLB.

Ingress/Routes: Layer-7 Routing for HTTP/S

Ingress (or Route in OpenShift) is a separate API object used for external access at Layer 7 (HTTP/S). With Ingress, you define routing rules—for example, "send requests for api.example.com to Service A, and www.example.com to Service B."
An Ingress Controller implements these rules, typically using a proxy or cloud load balancer. However, Ingress only supports HTTP(S) and lacks native support for TCP or other protocols.

From Ingress to Gateway API: Why a New API?

The Gateway API is an evolving standard—now an official Kubernetes SIG project—designed to address the limitations of Ingress and support more advanced traffic management. If you're unfamiliar with the term “SIG” it stands for Special Interest Group: volunteer teams that manage and maintain specific areas of the Kubernetes ecosystem (see the project home here). Think of the Gateway API as a modern urban plan for our Kubernetes city — with zoning laws, dedicated roads, and clearly defined roles.

Unlike Ingress, Gateway API is more than a single traffic cop signaling HTTP cars through an intersection. It is an expressway system that speaks multiple protocols—HTTP, HTTPS, TCP, TLS, even UDP—while offering built-in traffic tricks such as path and header rewrites, query-parameter matching, and weighted routing for canary releases - all without using vendor-specific annotations. Crucially, it separates concerns: platform teams lay down GatewayClasses and Gateways (the asphalt), while application teams own their Routes (the traffic signs). Because the spec is vendor-neutral, the very same YAML can drive Istio, NGINX or Cilium, meaning your road network is portable as your underlying architecture evolves.

Three of the most important Resources introduced by GatewayAPI are:

GatewayClass – urban planners
Gateways – the roads and entry points
Routes – detailed traffic control rules

These are implemented as Kubernetes Custom Resource Definitions (CRDs), providing flexibility, scalability, and separation of concerns. Let's dive into each of them.

GatewayClass

A GatewayClass defines a category of Gateways and is managed cluster-wide. It encapsulates the controller responsible for implementing the associated gateways.
This is similar in concept to StorageClass in Kubernetes: administrators define them, and users reference them.
Example:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: istio
spec:
  controllerName: istio.io/gateway-controller

In this case, any Gateway with gatewayClassName: istio will be handled by the Istio controller. It applies similar for other controllers.

Gateway

A Gateway is an instance of ingress infrastructure—an actual network entry point. It references a GatewayClass and defines one or more listeners, each specifying a protocol (HTTP, HTTPS, TCP), port, and optional hostname.
Example:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: example-gateway
  namespace: istio-ingress  # Gateways are namespaced
spec:
  gatewayClassName: istio   # must match a GatewayClass
  listeners:
    - name: http            # an arbitrary name for the listener
      protocol: HTTP
      port: 80
      hostname: "*.example.com"
      allowedRoutes:
        namespaces:
          from: All         # allow Routes from any namespace to bind (could be restricted in production)

Routes: The Core of Gateway API’s Flexibility

Here’s where the Gateway API truly shines—dedicated, protocol-specific route types like HTTPRoute, TCPRoute, and TLSRoute. These give you granular control over traffic, such as:

Matching by host, path, headers, or query parameters
Routing to multiple services with weighted distribution (for canary or A/B testing)

Think of Routes as traffic signs guiding vehicles through the city. They keep everything flowing smoothly.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: myapp-route
  namespace: default
spec:
  hostnames:
    - "myapp.example.com"
  parentRefs:
    - name: example-gateway
      namespace: istio-ingress
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: "/"
      backendRefs:
        - name: myapp-service
          port: 80

Example highlights from an HTTPRoute:

parentRefs: Attach the route to a specific Gateway and listener
hostnames: Match specific domains (e.g., myapp.example.com)
rules: Define matching logic and forwarding behavior
matches: Conditions like path prefix or headers
backendRefs: Destination services, possibly weighted

Earlier, we talked about one of the key advantages of the Gateway API over Ingress: native support for advanced traffic management—or what we called “traffic tricks.” A concrete example of this comes in the form of on-object filters. For instance, if you need to strip /v1 from all incoming URLs, here’s how you can do it using Gateway’s built-in URL rewrite filter:

spec:
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /v1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: /
    backendRefs:
    - name: myapp-service
      port: 80

No annotations, no sidecars—just one CRD doing the job.

Gateway Controllers and Implementations

Gateway API resources (like Gateway, HTTPRoute, and GatewayClass) are declarative. To actually route traffic, you need a Gateway Controller—a component that reads these resources and configures the data plane (Envoy, NGINX, etc.).
Supported controllers include:

Service meshes like Istio and Linkerd (Istio is moving to Gateway API by default)
Ingress controllers such as Contour, NGINX, Traefik, Kong (adding Gateway support)
Cloud providers (GKE, AWS, Azure) that map Gateway resources to native load balancers
Other projects like Cilium and Gloo offering enhanced networking

Managing TLS Certificates with Gateway API

Let’s be honest: rotating and renewing TLS certificates remains one of the most thankless tasks in day-to-day ops. Exposing HTTP apps to the internet requires HTTPS, and thus, TLS certificates. In the Ingress world, cert-manager automates this via annotations and certificate blocks. Fortunately, Gateway API is supported too.
To automate TLS certificate issuance with cert-manager and Gateway API:

Ensure cert-manager and Gateway API CRDs are installed
Configure a ClusterIssuer or Issuer (e.g., Let’s Encrypt)
Annotate your Gateway to trigger cert-manager Example:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: example-gateway
  namespace: istio-ingress
  annotations:
    cert-manager.io/issuer: "letsencrypt-prod"  # Reference to a cert-manager Issuer or ClusterIssuer
spec:
  gatewayClassName: istio
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      hostname: "myapp.example.com"
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: myapp-example-com-tls  # Secret that will hold the TLS cert and key
      allowedRoutes:
        namespaces:
          from: All

Cert-manager will:

Detect the need for a certificate
Create a Certificate resource behind the scenes
Complete the ACME challenge (if using Let’s Encrypt)
Store the key and cert in the referenced Secret
Keep the certificate renewed automatically

This works much like how cert-manager integrates with Ingress—only now, it's applied to Gateway. This seamless integration with cert-manager is key to automating secrets management for external access and eliminates the need for hacky workarounds.

Conclusion

Kubernetes offers several ways to expose services—from the basic NodePort to cloud-managed LoadBalancer and flexible Ingress. The Gateway API builds on those lessons and provides a modern, scalable way to manage external access with better separation of infrastructure and application concerns.
We explored how GatewayClass, Gateway, and Routes work together, and how Gateway controllers like Istio implement them. On the secrets side, tools like cert-manager integrate seamlessly to automate HTTPS. For broader secrets management, tools like HashiCorp Vault are an excellent addition — perhaps a topic for another blog. 😊
Kubernetes has evolved from a small village into a thriving metropolis. And just like any modern city, it needs robust, scalable infrastructure to manage how traffic flows and how identities are verified at its gates. The Gateway API is that next-generation road system—built with urban planning in mind. The pavement is poured; now it’s time to open the on-ramps and let your applications cruise.

The Tortoise and the Hare: do AI Agents Really Help for Software Development?

Jakob Beckmann — Wed, 23 Apr 2025 05:37:56 +0000

Making my development workflow as fast as possible is a big passion of mine. From customizing my development setup to get the last inkling of efficiency out of it, to thinking how to manage notes and knowledge resources to access them as quickly as possible. With the sudden ubiquity of AI in development tools, I came to wonder how AI could help me write code faster. Being quite the skeptic when it comes to AI actually generating code for me (using tools such as Cursor or GitHub Copilot), I came to investigate AI agents which specialise in code reviews. In this blog post I will share my experience using such an agent on a real world case. I will explore where such agents shine and where they are severely lacking.

I am an AI Skeptic

Generally I am not fond of using AI to develop software. My background is mostly in systems software, where correctness of the software can be critical. This means that using tooling that is non-deterministic and might not produce adequate results makes me uneasy. Furthermore, even if AI were to produce amazing results, a developer relying on it could quickly lose understanding of the code. This results in skill atrophy and large risks if the AI reaches the limits of its capabilities. In other words, I am not keen on having any AI generating code for me on a large scale for anything more than a proof of concept or low risk project.

Nonetheless, one would be foolish to ignore AI's capabilities when it comes to developer tooling.

AI Support Agents

Thus starts my journey investigating AI agents that can support me in the software development lifecycle, but whose main use is not to generate code. Many such agents exist, mostly focusing on reviewing code. I am quite the fan of such a use case, as the AI essentially plays the role of another developer I might work with. It reviews my code, provides feedback, suggestions, and potentially even improvements. It however does this immediately after I have opened a pull request, rather than having to wait for days or weeks on a human review.

How is this different from using an AI that generates code you might ask? The main difference lies in the fact that I still have to think on how to solve the problem I am working on, and provide a base solution. This forces me to understand the issue at hand. Thus, I am much better prepared to accept or reject any suggestions from an AI than if the AI just generated a first solution for me. Moreover, people (myself included) tend to be slightly defensive about the code they write. Thus I will, in all likelihood, only accept AI generated code improvements if it offers a real improvement, rather than blindly incorporating them into the codebase.

All in all, it is extremely unlikely that I will lose understanding of the codebase or have my problem solving skills atrophy, but I can iterate on reviews much faster.

CodeRabbitAI

In order to gain first experiences with such an AI agent, I chose to try out CodeRabbitAI. This was not a thoroughly researched decision. The main reason I chose CodeRabbitAI is that I could try it out for free during 14 days and that it integrates well with GitHub. I am aware that performance between AI models varies greatly. However, CodeRabbitAI uses Claude under the hood, a model typically known to perform surprising well on programming tasks. I thus expect it to not perform significantly worse than any other state of the art model out there.

Starting Small

In my opinion, such agents need to be tested on real world examples. One can see demos using AI to generate a dummy web app all over the place. However, common software projects are significantly larger, contain more complex logic, and are less standardized than these demos. Unfortunately, most software I work on professionally is not publicly available, so I cannot use CodeRabbitAI on these. I therefore picked two (still very small) personal projects of mine:

A NeoVim plugin providing a colour scheme.
A command execution engine to run templated commands.

Both projects are extremely small, with under two thousand lines of code. Both projects are written in Lua, a quite uncommon language. I wanted to see how the AI fares against something it is unlikely to have seen too much during its training.

With that in mind, I wrote a first pull request implementing a fix in highlight groups for pop-up menus in NeoVim. I enabled CodeRabbitAI to summarize the PR for me.

The summary looks good, even though it somehow marks some fixes as features. This is especially intriguing as I use conventional commits and explicitly marked these changes as fixes. Additionally, CodeRabbitAI offers a "walkthrough" of the changes made in the PR. In the case of such a simple PR, I found the walkthrough to be mostly confusing. In the case of larger PRs I can however see how this may be appealing.

In reality, I initially opened the PR with only the fixes the pop-up menus. I then pushed commits introducing the support for additional plugins later on. I would have expected CodeRabbitAI to complain that the new commits introduce changes unrelated to the PR, which is not seen as best practice. It did nothing of the sort.

While the summary, walkthrough, and disregard for best practices were unsatisfying, one unexpected benefit emerged: the integration of linting feedback directly within the pull request comments. It provided nitpicks from linting tools (in this case markdownlint. On one side, it is very disappointing to see that the AI agent did nothing more than lint the code and generate a nice comment out of the output. On the other hand it is quite nice that it introduces "quality gates" such as linting without me having to write a pipeline for it. Moreover, producing easily digestible output from a linter is nothing to be underestimated. The quality of life of having this directly as a comment rather than having to go through pipeline logs to read the raw linter output is quite nice. Is it worth two dozen USD per month? No, definitely not!

On the upside, it did update the summary of the PR to reflect the other changes:

The first PR was extremely trivial. It did not introduce any code containing logic. Other than not pointing out that it should probably have been two separate PRs, CodeRabbitAI fared as I would have expected another developer to have reviewed the PR. With two small differences:

The CodeRabbitAI review was close to immediate (took around 30-60 seconds to run). This is amazing to iterate quickly.
Where I would have expected a human reviewer to point our the nitpick or simply approve, CodeRabbitAI is extremely verbose with explanations, walkthroughs, and so on. This in turn wastes time for the author, as he/she would need to read through this. The verbosity could be nicer on larger PRs, but for small concise PRs this is massive overkill and borderline annoying.

To further evaluate CodeRabbitAI's capabilities, I decided to test it on a pull request with more substantial changes...

A More Complex PR

Armed with dampened expectations from my first PR, I opened another PR in the command execution repository implementing a feature affecting multiple files. These changes also update existing logic.

In this second PR, CodeRabbitAI went above and beyond, and generated a walkthrough containing two sequence diagrams showcasing the control flow of the code that was modified! I was actually quite impressed by this. While probably not necessary for the author of a PR, this is great even only for documentation purposes. New team members with less experience may benefit from such visual aids to understand complex logic within the code. Unfortunately the diagrams didn't highlight the specific modifications introduced by the pull request.

However, the supporting text suddenly becomes more relevant when considering such PRs.

On top of that, CodeRabbitAI actually posted interesting comments. It found the odd nitpick here and there, but also found more meaningful potential issues. For instance, I modified a test configuration to use a different shell. CodeRabbitAI identified that this shell is not listed as a dependency anywhere in the repository, and that it would thus not work off-the-shelf. In this case this was only a test file used to parse the configuration and the configured shell did not affect anything, but this is a great finding generally.

I also started conversing with CodeRabbitAI about some changes. Requesting it to give me a suggestion on some configurations. It managed just fine, but did not actually provide these as code suggestions that can be applied, but rather as code blocks in comments, which was a bit disappointing.

Additionally, I decided to try to use CodeRabbitAI's commands feature. This enables ChatOps to control actions taken by CodeRabbitAI. I generated the PR title using one such command. The title turned out generic and not very informative. In CodeRabbitAI's defense, I am quite unsure how I would have named that PR.

I then tried to get it to write docstrings for new functions that were introduced in the PR. It massively misunderstood the request, and created a PR adding docstrings to all functions in the affected files, even ones that already had docstrings... This goes to show that in some cases, it cannot even do what the most junior of all engineers would be capable of doing thanks to a even so tiny dose of common sense. Moreover, it started adding commits with emojis in the title. This goes to show that these AIs are probably not trained much on professional projects.

After that first disaster, with significantly less ambition, I requested it creates a PR to change a small typo. CodeRabbitAI informed me that it created a branch with the changes included, but that it was not capable of creating pull requests. This shocked me, considering it had created its first disaster PR no 10 minutes before.

After another nudge, CodeRabbitAI however did create a PR. It targeted main instead of the branch I was initially using. I guess this is my own fault though for not being specific enough.

Finally, I also tried to get it to update the wording on a commit it did to use conventional commits. Unfortunately it seems that it only has access to the GitHub API and cannot execute any local git commands. It is therefore not able to perform some relatively common operations in the SDLC that are not part of the GitHub API. However, I am guessing this is subject to change relatively soon with the emergence of technologies such as the model context protocol, which would enable it to control external tools such as git.

All in all, I would say CodeRabbitAI did as I would have expected after the first PR. It corrected nitpicks and allowed me to perform some simple actions. Did it deliver a review of the same quality like a senior engineer familiar with the project would have? No. In fact, in order to test this I intentionally implemented a feature that was already present in the repository, while making a couple design decisions that go against most of what the rest of the repository does. CodeRabbitAI neither detected that the logic I was introducing was already present in the codebase, nor did it complain about the sub-optimal design decisions. This goes to show that such agents are still not capable replacing humans with nuanced understanding of the project's history and architectural principles, potentially leading to the introduction of redundant or suboptimal solutions.

Dashboards!

Another feature of AI agents next to the reviews is the analytics capabilities that come with them. In my personal opinion, analytics are important to measure the impact the introduction of such tooling has on the software delivery. CodeRabbitAI provides a couple nice dashboards on how much it is being used, and what kind of errors it helped uncover.

I did not try out CodeRabbitAI for long enough to have any meaningful metrics, but I am confident that the capabilities provided are enough to get a decent understanding of the quality of adoption.

Moreover, CodeRabbitAI supports reporting. This allows to generate reports based on natural language prompts that could be useful for product owners to get insights of changes made to the software over the course of a sprint.

Verdict

While this whole article might seem like a slight rant against such tools, I would in fact wish I could use such tools at work. Not as a replacement for human reviewers, but as an addition to them. For instance, the quite verbose walkthroughs CodeRabbitAI provides can be a very helpful entrypoint to a human reviewer on larger PRs. Moreover, while the quality of the review is insufficient for projects where quality matters, having near instant feedback is amazing.

Finally, as mentioned above, I believe one major selling point of such agents is in the way we humans interact with them. Even if the agent might do little more than execute linters or similar in the background, having the output of these tools in natural language directly as comments in the PRs is not to be underestimated. This is especially true in the age where more and more responsibility is being shifted to developers. With DevSecOps, developers have to understand and act upon the output of all kinds of tools. Presenting this output in a more understandable format, potentially enriched with explanations, can have a significant impact.

Therefore, as a final word, I would actually encourage people to explore such agents to augment their workflow safely, albeit with caution and a clear understanding of their limitations.

f4z3r / gruvbox-material.nvim

Material Gruvbox colorscheme for Neovim written in Lua

Gruvbox Material

A NeoVim colour scheme in pure Lua allowing for highly flexible configuration and customization.

Features | Installation | Usage and Configuration | API Reference

Note

This is a continuation of the original work from WittyJudge https://github.com/WIttyJudge/gruvbox-material.nvim

A port of gruvbox-material colorscheme for Neovim written in Lua. It does not aim to be 100% compatible with the mentioned repository, but rather focuses on keeping the existing scheme stable and to support popular plugins. This colorscheme supports both dark and light themes, based on configured background, and harder or softer contrasts.

Dark theme:

Light theme:

Different contrasts

Contrast	Dark	Light
Hard
Medium
Soft

Features

Supported Plugins:

Please feel free to open an issue if you want some features or other plugins to be included.

…

View on GitHub

f4z3r / sofa

A command execution engine powered by rofi.

Sofa

A command execution engine powered by `rofi` and `fzf`.

About

sofa is a small utility to enable easy execution of templated commands. It can be used to store snippets that you often rely on, or fully template complex commands. It is meant to be used with a shortcut manager to enable launching from anywhere, but can also inject commands into your current shell session for commands that make more sense to run there (see Integration).

Examples

For Snippets Management

You can use sofa for standard snippets management. Use the integration described below, and have configuration such as:

Configuration

namespaces
  lua:
    commands:
      install-local:
        command: luarocks --local make --deps-mode {{ deps_mode }} {{ rockspec }}
        description: Install rock locally
        tags:
        - local
        - luarocks
        interactive: true

…

View on GitHub

A Comprehensive Guide to Managing Large Scale Infrastructure with GitOps

Jakob Beckmann — Tue, 08 Apr 2025 04:31:02 +0000

GitOps is getting adopted more and more. However, there still seems to be some confusion as to what GitOps is, how it differs from regular CI/CD pipelines, and how to best adopt it. In this post we
will quickly cover what GitOps is, and the three main lessons learned from using GitOps to manage infrastructure at scale both on premise and in the cloud.

GitOps Overview

GitOps is a set of principles enabling the operation of a system via version controlled, declarative configuration. More specifically, the OpenGitOps project defines four principles which define whether a system or set of systems is managed via GitOps:

Declarative: A system managed by GitOps must have its desired state expressed declaratively.
Versioned and Immutable: Desired state is stored in a way that enforces immutability, versioning and retains a complete version history.
Pulled Automatically: Software agents automatically pull the desired state declarations from the source.
Continuously Reconciled: Software agents continuously observe actual system state and attempt to apply the desired state.

Note that git is not referenced anywhere, as GitOps is not bound to any tooling. However, in layman terms, many consider a system operated via git to be a GitOps system. This is not quite correct.

GitOps is More than CI/CD Pipelines

Taking the "layman's definition" from above, any system that has CI/CD via pipelines triggered on repository changes would be a GitOps system. This is not accurate. Consider an IaC pipeline which applies declaratively defined infrastructure (such as a standard opentofu apply in a pipeline, or a Docker build followed by a kubectl apply). While such a system adheres to the first two principles, it does not adhere to the latter two. This implies that changes made to the target system are not corrected (reconciled) until the pipeline runs the next time. Similarly, if the pipeline fails for whatever reason, the desired state does not change the pipeline: a configuration drift is not detected, even if not reconciled.

This is an important distinction when considering "standard CI/CD" and GitOps. Simply having something declared as code does not make it GitOps.

The Advantages of GitOps

GitOps has many advantages over standard ways of managing systems. The advantages of having a declarative desired state, version controlling it, and interacting with the system only via git (or whatever version control system you use) are tremendous. From improved security and higher efficiency to better change visibility. These are well known to most people and will thus not be covered here.

Drift detection and automatic reconciliation are the two other aspects that make GitOps absolutely amazing. This is especially true in the current day and age, with the proliferation of complex systems being worked on by many people concurrently. Being able to observe that the system is not in the desired state has massive advantages, such as for standard SRE operations. Continuous reconciliation ensures that manual operational tasks are kept to a minimum, and that systems cannot degrade over time as small undesired changes creep in.

Tooling

In this post we will mostly focus on using GitOps to manage resources handled via the Kubernetes API, but it should be noted that GitOps as a concept is in no way restricted to Kubernetes. In the Kubernetes space there are two major players for GitOps: ArgoCD and FluxCD. We will not go into the details as to what the advantages for each tool are, other than saying that according to our own experience, ArgoCD might be more developer focused, while FluxCD might suit platform engineers with more Kubernetes experience that want more flexibility.

The rest of this post is tool agnostic and everything we are talking about can be done with either tool (but some aspects might be easier to do with one or the other).

Infrastructure: Disambiguation

Before we dive into how to structure your GitOps configuration, it might make sense to draw a line as to where infrastructure starts and where it ends. We consider infrastructure everything that is part of the platform provided to an application team. Hence this line might vary depending on the maturity of the platform you provide your teams. If we consider a simple Kubernetes platform with little additional abstraction for its users, the infrastructure would contain the Kubernetes platform itself as well as all system components that are shared between the teams, such as a central monitoring stack, a central credential management solution, centralized policy enforcement of specific Kubernetes resources, and the like.

The lower end of the spectrum will likely not be managed by GitOps. That is simply because the GitOps tooling itself typically needs to run somewhere, and also needs to be bootstrapped somehow. Some tools such as FluxCD allow the GitOps controller to manage itself, but even in these cases the runtime for the controller needs to exist when the controller is initially installed, and is thus typically not part of the GitOps configuration.

Now that this is cleared up, let us consider how the configuration should be managed.

App-of-Apps

A very popular pattern for managing configuration via GitOps is the "app-of-apps" pattern. This was popularized by ArgoCD, but is also applicable to other tooling. We will use ArgoCD in the example below, but the same can be implemented using FluxCD Kustomizations.

Let us consider a component from our infrastructure that we want to manage via GitOps. Typically, we would need to tell the GitOps controller how to manage this component. For instance, let us assume the component is installed via raw Kubernetes manifests. Then we would tell the GitOps controller which repository contains these manifests and in which namespace to install them. Depending on the controller you are using, you might also configure additional parameters such as how often it should be reconciled, whether it depends on other components, and so on. In ArgoCD jargon this would be an "Application" (the root of "app-of-apps" naming), and would look as follows:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sealed-secrets
  namespace: argocd
spec:
  project: default
  source:
    chart: sealed-secrets
    repoURL: https://bitnami-labs.github.io/sealed-secrets
    targetRevision: 1.16.1
    helm:
      releaseName: sealed-secrets
  destination:
    server: "https://kubernetes.default.svc"
    namespace: kubeseal

You would then apply this Application resource to Kubernetes. Your component would then be managed by GitOps, as any changes you push to the manifests repository would be reflected on the Kubernetes cluster.

Then a second infrastructure component needs to be installed, and you repeat the process. The result would be a second Application which installs and manages a component. You might also want to version your deployment (such as using version 1.16.1 of the Helm chart). This implies that lifecycles require a change to this Application manifest, and thus a call against the Kubernetes API to edit it.

The end result is a set of Application resources, some of which you periodically modify when lifecycling a component. Now imagine you need to deploy your infrastructure elsewhere (for instance a second Kubernetes cluster in our example), or maybe even a couple dozen times. Then you need to manage this entire set of Application resources on every platform. A better approach is to add an abstraction layer, which itself deploys the Application resources via GitOps. Hence you put all your Application resources into a repository, and define another, "higher level" Application which deploys this repository. This means that when deploying to new platforms, you only need to deploy that one "higher level" Application, and any changes to the component Application resources can be made via Git, conforming to our GitOps approach. This "higher level" Application is only there to deploy the component Applications thus the name "app-of-apps". Visually, you thus have the following structure:

It should be noted that this also massively helps when customizing platforms. Typically, components cannot be deployed truly one-to-one in several places, but require slight configuration differences. Consider for instance hostnames for UIs of your components. Two of these components deployed in different locations cannot share the same hostname and routing. Using an "app-of-apps" approach allows you to define variables on the top level application, and inject these into the downstream applications such that they can slightly adapt the way they are installed. We will not dive deeper into how this is done as it is highly dependent on the tooling you use (ArgoCD uses ApplicationSet, FluxCD uses variable substitution), but know this is enabled by such an approach.

Consolidating your Configuration

In the organisation I first used GitOps at scale, we deployed all our components as Helm charts to a Kubernetes cluster. Each component was essentially contained within two different repositories in our version control system:

the source code repository which typically built a Docker image as an artefact
the Helm chart definition which referenced the Docker image from above

When we then introduced GitOps, we decided to add a third repository containing the exact deployment definition (in our case the Application declarations) for the component. Using the app-of-apps pattern from above, we could then reference each of these "GitOps repositories" and deploy specific overlays (customizations) of the Application to specific platforms. This worked well for quite some time. However, with time the number of components we managed increased, and so did the number of target platforms to which these components needed to be deployed. This lead to quite a few issues.

When a new target platform was introduced, all such "GitOps repositories" needed to be updated to contain a new overlay customizing the Application to the specific platform. This is very tedious when you have several dozen such repositories.

Moreover, components had dependencies to other components. This meant that we were referencing components within a repository that were defined in another repository. While not problematic in itself, this can become very tricky when one component has a dependency on a configuration value of another component. The configuration value is then duplicated in both repositories and becomes difficult to maintain. While this sounds like we did not properly separate the components, it is very common to see such cases in infrastructure configurations. Consider for instance a deployment of an ingress controller which defines a hostname suffix for its routes. All components deployed on the same Kubernetes platform that deploy a route/ingress will need to use exactly that hostname suffix in order to have valid routing.

The above issue also results in tricky situations when configurations need to be changed for components that are dependent on one another. If the deployment configuration is separated into different repositories, PRs to these repositories need to be synchronized to ensure the deployment occurs at the same time.

Finally, distributing the deployment configuration over so many repositories meant that it became increasingly difficult to have an overview of what is deployed on a target platform. One would need to navigate through dozens of repositories to check this is correctly done.

After identifying these issues we decided to move all our configuration into a single repository. This repository would then contain a templated definition of the entire set of components which would need to be deployed. A set of platform definitions within the same repository would then feed values to templates to ensure consistent configuration. This massively helped us with to address the issues mentioned above. On top of that, it allows to version the "template" and thus enables rollouts of a versioned infrastructure layer.

You can find an example repository of such a structure
designed with FluxCD here:

f4z3r / flux-demo

This repository shows an example how one can use a single mono-repository to manage multiple clusters' infrastructure in a controlled fashion.

Flux Demo

An example how one can use a mono-repo to manage large infrastructure in a controlled fashion using FluxCD.

Setup | Structure of the Repo | Application vs Infrastructure | Workflow

Setup

Generate a GitHub PAT

See the documentation: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens

For fined-grained control, grant the token Admin and content read/write permissions on the repository.

Setup a Cluster with Flux

Setup the required tooling with devbox shell, then

# install a cluster
kind create cluster -n demo
# set the token
export GITHUB_TOKEN='<redacted>'
# onboard flux
flux bootstrap github \
  --token-auth \
  --owner=f4z3r \
  --repository=flux-demo \
  --branch=main \
  --path=clusters/demo \
  --personal

Sample output from Flux installation

► connecting to github.com
► cloning branch "main" from Git repository "https://github.com/f4z3r/flux-demo.git"
✔ cloned repository
► generating component manifests
✔ generated component manifests
✔ committed component manifests to "main" ("158753158f3c760f741f22ed7f68bdee1b66e475")
► pushing component manifests to "https://github.com/f4z3r/flux-demo.git"
► installing components in

…

View on GitHub

Gitops Bridge

The last challenge we want to address in this blog post is a concept called a "GitOps bridge". In public cloud environments, there is typically a relatively strong cut between infrastructure deployed via Terraform (or any similar tool), and the infrastructure deployed via GitOps. For instance, one might deploy an Azure Kubernetes Service and some surrounding services (such as the required network, a container registry, etc) via Terraform, and them deploy components and applications within the AKS using GitOps. The issue that we face here is that the GitOps configuration very often depends on the Terraform configuration. Consider for instance the container registry. Its address is set up by Terraform, but is used in every image declaration in the GitOps configuration. One option is to duplicate such values in the respective configurations, while another option is to use a GitOps bridge.

The GitOps bridge is an abstract concept on how to pass configuration values from tooling such as Terraform as inputs to the GitOps configuration. How this is done in practice very much depends on which tools you use. For instance, if looking at Terraform and FluxCD, a common way to achieve this is to have Terraform write a ConfigMap onto the AKS where the FluxCD controller will run containing all variables (and their values) that will be required by the GitOps configuration. The FluxCD controller then supports injecting variables from a ConfigMap via variable substitution.

Using a GitOps bridge has the advantage that changes in the Terraform configurations are much less likely to break the GitOps configuration that builds on top of it. Moreover, it allows Terraform to directly bootstrap the entire GitOps setup when creating new platforms without the need to manually redefine the required variables in the GitOps repository.

Summary

So, to recap, we have looked at what GitOps really is (and isn't). Understanding these basics is critical to correctly implement GitOps in your projects. On top of that, we looked at three best practices:

Use an app-of-apps pattern to improve resiliency for when you need to recreate platforms.
Consider using a mono-repository for all your GitOps configuration as your setup grows.
Have a look at GitOps bridges to improve the automation when setting up platforms and ensuring your Terraform and GitOps configurations are consistent.

I hope this has helped you understand a bit better how to use GitOps at scale. If you have any questions or comments, feel free to let me know below.

How to Join Lists in Kafka Streams Applications

Jan Kleine — Mon, 17 Mar 2025 06:00:00 +0000

I recently joined a project where we do data processing with Kafka Streams applications and came across an interesting problem: "list joins". There was already a working solution to the problem, but I was not quite satisfied, so I dug a little deeper. Since I didn't find much on the topic online, I wanted to share my findings here.

TL;DR The final topology is discussed here and you can find the code here:

iptch / kafka-list-join-demo

A demo showing how to do a list join in a kafka streams application.

Kafka List Join Demo

This project shows two ways to perform a list join in a Kafka streams application. A list join refers to joining a record that contains a list with a KTable, such that each element in the list gets joined with the corresponding element in the KTable.

This image shows the high level idea, joining a persons address list, with the corresponding addresses:

Check out this blog post for a discussion of the approaches.

The tests should cover all relevant cases of message ordering and updates.

Building

To build and test the project, run

./gradlew clean build

View on GitHub

I assume you have some understanding of Kafka Streams, but I'll try to link to relevant documentation and resources where possible. It's worth checking out the Kafka Streams DSL Developer Guide if you are unfamiliar with Kafka Streams.

The Problem

We have a topic with records (of type Outer, e.g., a person) that contain (among other things) a list. The elements in the list reference records on another topic (of type Inner, e.g., addresses). For every element in the outer topic we want to look up the corresponding record in our inner topic, and merge them to enhance our list.

Additional Considerations

There are a few additional constraints we need to keep in mind:

Eventual Consistency: The corresponding records on the inner topic may not be available right away, that is, records may come in after our outer record.
Updates: Our outer record may be updated over time. This may include updates (additions/removals) of inner list elements, but also updates to fields other than the list.
Duplicates: In our specific use case we don't need duplicate list entries. However, all approaches discussed here can be modified to allow duplicates without too much effort.

The Existing Solution

The existing solution to the problem looks roughly like follows:

We set a timestamp on every record on our outer stream (needed later).
We flat map our outer records so we have exactly one inner list element in every flat mapped record, to differentiate the flat records we change the keys to composite keys (of the form <outer-key>$$<inner-key>).
We interpret the resulting stream as a KTable and perform a foreign key left join with the other KTable. The KTable-KTable left join (as opposed to, e.g., a KStream-KTable join) is needed to fulfill our eventual consistency constrain.
We group the the resulting records by the first part of the composite key.
Finally, we reduce the records in each group by appending the lists, but we only consider the newest timestamp we see. Whenever we see a newer timestamp we discard all older records. This ensures that we do not add stale elements to our list that may be deleted in the current list.

The code looks similar to the below, though I left some smaller details out, such as how we forward tombstones. The full code can be found here.

(outerKStream, innerKTable) -> outerKStream
    .mapValues(outerMapper)
    .flatMap(outerFlatMapper)
    .toTable(buildStore(outerSerde, "listJoinFlatStore"))
    .leftJoin(
            innerKTable,
            outer -> outer.getInnerCount() == 0 ? null : outer.getInner(0).getId(),
            outerInnerJoiner,
            buildStore(outerSerde, "listJoinJoinerStore")
    )
    .toStream()
    .groupBy(
            (key, _) -> key.split("\\$\\$")[0],
            Grouped.with(Serdes.String(), outerSerde)
    )
    .reduce(
            outerReducer,
            buildStore(outerSerde, "listJoinReducerStore")
    )
    .toStream()

Issues

While this approach works for our use case, I don't like it for two reasons:

It feels hacky: The timestamps feel a bit hacky and force us to change our data model just for this operation. While we can (and in production do) clear the timestamps after the list join is complete, we are using protobufs for our records. So the timestamp is part of the protobuf definition, even if downstream services don't need it.

It is inefficient: Whenever the original protobuf changes (be it, a change to the list, or any other field), all list elements go through the join again, followed by reducing them all (including stale old records) again. That just feels wasteful.

These issues prompted me to spend some time on the problem and see if I can come up with something better.

Improvements

We want to reduce the number of join and reduce operations and remove the need for a dedicated timestamp field.

For this we need a component that keeps track of the changes in the list of a record so that we only process relevant changes.

The List Join Pre-Processor

The only way I found to accomplish this, is to write a custom Processor using the Processor API provided by Kafka Streams.

The list join pre-processor performs the task of the flat-map operation from before, but maintains internal state¹ to remember what list elements are currently in each record. When a record gets updated, it compares the previous and new list, and (1) only issues flat mapped records for new list elements and (2) issues tombstones for elements removed from the list.

(1) ensures that we only have to join new list elements, reducing duplicate work. (2) allows us to more efficiently remove old list elements, so we don't need to rely on a hacky timestamp.

The pre-processor also always forwards a copy of the current record with an empty list. We need this later to correctly propagate changes to other fields that are not the list.

The logic of the pre-processor, which can also be found here, looks as follows:

public void process(Record<String, TOuter> record) {

    // in case of tombstone or empty list, this map is empty
    Map<String, TOuter> flatValuesMap = flatMapper.apply(record.value()).stream()
            .collect(Collectors.toMap(
                    innerIdStringExtractor,
                    Function.identity(),
                    (_, newValue) -> newValue
            ));

    Set<String> newIds = flatValuesMap.keySet();

    Set<String> oldIds = Optional.ofNullable(listStore.get(record.key()))
            .map(Set::copyOf)
            .orElse(Set.of());

    // if both new and old ids are empty we don't need to do anything and can short circuit
    if (newIds.isEmpty() && oldIds.isEmpty()) {
        return;
    }

    // forward flat mapped records for new list elements
    Sets.SetView<String> addedIds = Sets.difference(newIds, oldIds);
    for (String newId : addedIds) {
        forward(record.key(), newId, flatValuesMap.get(newId));
    }

    // send tombstones for removed list elements
    Sets.SetView<String> removedIds = Sets.difference(oldIds, newIds);
    for (String removedId : removedIds) {
        forward(record.key(), removedId, null);
    }

    // if the current list is empty delete the list from the store and send a tombstone
    // for the empty list record, otherwise save the current list and send an empty list
    // record to propagate changes to other fields
    if (newIds.isEmpty()) {
        forward(record.key(), null, null);
        listStore.put(record.key(), null);
    } else {
        forward(record.key(), null, listCleaner.apply(record.value()));
        listStore.put(record.key(), List.copyOf(newIds));
    }
}

Better Reduce Operation

Now that the flat-map and the subsequent join are improved, we can further improve the reduce operation.

Previously, the reduce operation was responsible for filtering out outdated records (based on the timestamp) and aggregating the list of inner elements. This was done on all flat-mapped values (including outdated values) and needed to happen every time anything in the original outer record changed.

Now, that we can issue tombstones for removed elements, we can transition from a KStream group-by and reduce to a KTable group-by and reduce.

The KTable reduce works a bit differently than the KStream reduce. Where the KStream reduce has a single reducer that receives the current aggregate and any new record (be it a tombstone or a normal record), the KTable reduce has an adder reducer and a remover reducer.

The adder reducer is called when ever an element is added to the KTable and receives the current aggregate and the newly added record.

The remover reducer is called when ever an element is removed from the KTable (via a tombstone) and is provided with the current aggregate and the removed record. This allows us to remove the inner element from the current aggregate, as we know exactly which one got removed.

Our adder looks as follows. Remember that our pre-processor also sends a copy of the original record with an empty list. This way we can update other fields as well.

public Outer apply(Outer currentValue, Outer newValue) {
    if (newValue.getInnerList().isEmpty()) {
        // if list is empty update all fields other than the list
        return newValue.toBuilder()
                .addAllInner(currentValue.getInnerList())
                .build();
    } else {
        // otherwise add inner item to current list
        return currentValue.toBuilder()
                .addAllInner(newValue.getInnerList())
                .build();
    }
}

The full code can also be found here.

Our remover only has to worry about removing the list elements. In our case we do it based on the inner ID.

public Outer apply(Outer currentValue, Outer oldValue) {
    // if oldValue inner list is empty, there is nothing to do
    // the adder handles empty list updates
    if (oldValue.getInnerList().isEmpty()) return currentValue;

    // remove the single inner value from the current list,
    // in this case we do it by id
    Inner innerToRemove = oldValue.getInnerList().getFirst();

    List<Inner> innerList = currentValue.getInnerList().stream()
            .filter(inner -> !inner.getId().equals(innerToRemove.getId()))
            .toList();

    return currentValue.toBuilder()
            .clearInner()
            .addAllInner(innerList)
            .build();
}

The full code can also be found here.

New Topology

Incorporating the two improvements from above, we end up with a topology which looks surprisingly similar to our initial topology.

(outerKStream, innerKtable) -> outerKStream
    .process(preProcessorSupplier)
    .toTable(buildStore(outerSerde, "listJoinFlatStore"))
    .leftJoin(
            innerKTable,
            // if the inner list is empty the foreign key extractor should
            // return null so the outer is joined with null
            outer -> outer.getInnerCount() == 0 ? null : outer.getInner(0).getId(),
            outerInnerJoiner,
            buildStore(outerSerde, "listJoinJoinerStore")
    )
    .groupBy(
            // group by first part of composite key
            (key, value) -> KeyValue.pair(key.split("\\$\\$")[0], value),
            Grouped.with(Serdes.String(), outerSerde)
    )
    .reduce(
            listAdder,
            listRemover,
            buildStore(outerSerde, "listJoinReducerStore")
    )
    .toStream()

Again, I skipped over some smaller details, like how we forward tombstones. The full topology can be found here.

Since copying this code to different topologies is a bit cumbersome, we decided to wrap this entire sub-topology in a ListJoin utility. This allows easier reusing of the code, as the developer only has to provide a couple key components and make the resulting topologies easier to understand.

// setup
ListJoin<...> listJoin = ListJoin.builder()
        // specify joiner, reducers, etc.
        .build()

// later in the topology
KStream<...> joinedKStream = listJoin.apply(myLeftKStream, myRightKTable);

Conclusion

This topology is probably still not optimal. For one, it may make sense to deduplicate the empty list records to further reduce downstream work. However, in our case this is not necessary.

In any case, this new approach is a clear improvement over the previous timestamp based approach.

Have you solved a similar, or even the same, problem before? If so, please consider leaving a comment as I'd very much like to hear what you did. Likewise, if you have found issues with the above code or have ideas for improvements, I'm keen to hear from you!

Acknowledgments

Cover image: "Close Up Photo of Blue Background" by Harrison Candlin

Jan KleineFollow

Maintaining state can be done via a state store, which is automatically backed up to a changelog topic in kafka, ensuring the processor can tolerate application scaling and recover in case of application failures. ↩

Data Governance with dbt, Terraform, and Dataplex: A Practical Guide to BigQuery Policy Tags

Diana Tahchieva — Mon, 24 Feb 2025 07:39:48 +0000

Welcome to a hands-on guide for implementing BigQuery Policy Tags, an important feature for data governance. If you're new to Google Cloud Platform (GCP) and have heard of dbt, Terraform, and Data Catalog but aren't sure how they work together, this tutorial provides a simple, practical example. We'll apply policy tags to a sample clients table in BigQuery to enforce data governance.

What Are Policy Tags?
Policy Tags are classification labels for data in BigQuery, helping manage privacy, compliance, and access control. These tags are particularly important in industries like healthcare and finance, where data sensitivity is a key concern.

Why Use dbt, Terraform, and Dataplex for Policy Tag Management?
Using dbt and Terraform to define Policy Tags as code, and Dataplex for governance, you can keep track of changes, facilitate team collaboration, audit activities, and easily roll back to previous configurations.
Additionally, you can consistently manage Policy Tags across multiple datasets and projects, reducing manual labor. Automation through these tools minimizes human error and ensures policy tags are consistently applied.
While dbt integrates seamlessly into existing data pipelines, applying Policy Tags during data transformation and modeling, dataplex unifies governance across various data stores.

Understanding the Tools

Let us have a closer look at what are dbt, terraform and dataplex.
• dbt (Data Build Tool) is an open-source tool for transforming and modeling data within your data warehouse (like BigQuery). dbt enables you to write a more maintainable SQL code and allows you to attach metadata, such as Policy Tags, to your data transformations. Additionally, as the objects in BigQuery can be referenced, dbt is able to make a Directed Acyclic Graph of the entire data platform, based on which we can observe all dependencies among the data.
• Terraform is an Infrastructure as Code (IaC) tool that lets you define and manage your cloud infrastructure using configuration files. Terraform automates the provisioning and management of resources including enabling APIs, managing permissions, and creating Policy Taxonomies and Tags.
• Dataplex is a Google Cloud service that provides unified data governance and management. It helps discover, organize, and manage data assets, ensuring consistent data handling and enforcement of Policy Tags.

How They Work Together

OK, now that we know what dbt, Terraform, and Dataplex are, let's explore how they work together.
Terraform establishes the required infrastructure and permissions for managing Policy Tags by creating taxonomies and tags within Google Cloud Data Catalog. At the same time, dbt handles data transformation in BigQuery, applying Policy Tags to specific columns within your models. The meta section in the dbt model facilitates metadata association, ensuring proper organization and governance. Meanwhile, Dataplex functions as a centralized governance layer, maintaining consistency in the application and monitoring of Policy Tags across all data assets. Together, these tools create a seamless, scalable, and automated data governance system that enhances visibility, reduces manual effort, and minimizes the risk of human error.

Implementing Policy Tags: Step-by-Step Guide

First, you need to create a GCP project and connect it to a billing account; otherwise, Terraform won’t be able to function. Don’t worry—this project is small and won’t incur any costs (my billing account is still at zero). Just remember to delete it after testing by running terraform destroy in the command line—I’ll remind you at the end of the tutorial.
For convenience, I’ve created a Git project that you can clone. It contains two sub-projects: one for Terraform (gcp-data-catalog-terraform) and one for dbt (data_catalog_dbt_project). In a real-world scenario, these sub-projects would likely be managed as separate projects.

The terraform project has the following structure:
• variables.tf: Defines the GCP project ID and region.
• iam.tf: Creates service accounts for Terraform and dbt, assigning necessary IAM roles.
• datacatalog.tf: Defines the taxonomy for organizing Policy Tags and creates tags for PII and non-PII data.
• bigquery.tf: Creates a BigQuery dataset and a table without predefined policy tags.
• output.tf: Outputs IDs for easy access to created resources.

The dbt project in this example is a significantly simplified version of a standard dbt project and includes only the essential files necessary for our use case. Here’s a brief description of each file and its purpose:
• dbt_project.yml: This is the primary configuration file for your dbt project. It includes metadata about the project, such as the project name and version, paths to model files, and configurations for materializations and other project-wide settings.
• profiles.yml: This configuration file contains the connection details and credentials required for dbt to connect to your data warehouse (BigQuery in this case). It includes information such as the project ID, dataset, and authentication method (service account key file).
• models/customers/: This directory holds the models for the project. In dbt, a model is essentially a SQL file that transforms raw data into more refined tables. This directory contains our specific model for customers.
o customers.sql: This SQL file represents the transformation logic for the customers’ data. It selects and processes the necessary columns from the raw data, applying the transformations required for our data analysis needs.
o customers.yml: This YAML file provides additional metadata about the customers model. It includes descriptions of each column, tests to ensure data quality, and policy tags to enforce data governance rules.

Step 1: Enable APIs with Terraform

First, enable the necessary Google Cloud APIs for your project using Terraform. These are:
• Identity and Access Management (IAM) API
• BigQuery
• Data Catalog API
• Dataplex API

Step 2: Grants Permissions with Terraform

Let us walk through how to create and configure a service account that Terraform and dbt can use to interact with Google Cloud resources.
A service account in Google Cloud is like a robot user—it allows Terraform and dbt to authenticate and interact with GCP without needing a human user to log in.
In our setup, Terraform will provision BigQuery datasets, tables, and policies, while dbt will query and transform the data. To ensure that the permissions for Terraform and dbt are properly separated, we will define two distinct service accounts.

We can create the new service accounts (called terraform-sa and dbt-sa) by running:
gcloud iam service-accounts create terraform-sa --display-name "Terraform Service Account" --project gcp-data-governance
gcloud iam service-accounts create dbt-sa --display-name "dbt Service Account" --project gcp-data-governance

Alternatively, you can add the service account manually in the GCP console (IAM & Admin > Service Accounts). Then, in Terraform, you only manage the IAM roles in iam.tf

The Terraform service account needs permissions to manage BigQuery, Data Catalog, and Dataplex. Add these IAM roles in your Terraform configuration:

# Give the Terraform user permissions to manage IAM, Dataplex, and BigQuery
resource "google_project_iam_member" "terraform_bigquery_admin" {
  project = var.project_id
  role    = "roles/bigquery.admin"
  member  =  "serviceAccount:terraform-sa@gcp-data-governance.iam.gserviceaccount.com" 
}

resource "google_project_iam_member" "terraform_datacatalog_admin" {
  project = var.project_id
  role    = "roles/datacatalog.admin"
  member  = "serviceAccount:terraform-sa@gcp-data-governance.iam.gserviceaccount.com"
}

resource "google_project_iam_member" "terraform_dataplex_admin" {
  project = var.project_id
  role    = "roles/dataplex.admin"
  member  = "serviceAccount:terraform-sa@gcp-data-governance.iam.gserviceaccount.com"
}

resource "google_project_iam_member" "terraform_sa_admin" {
  project = var.project_id
  role    = "roles/iam.serviceAccountAdmin"
  member  = "serviceAccount:terraform-sa@gcp-data-governance.iam.gserviceaccount.com"
}

Note: For more information on the available roles, refer to the documentation. As a best practice, always assign the minimum permissions necessary. For instance, if a user only needs to view the data, provide read-only access.

• To allow dbt to authenticate, we need to generate a JSON key file for our service account, which we can do in two ways:

Option 1: Using the Google Cloud Console

Go to IAM & Admin → Service Accounts in the Google Cloud Console.
Select the service account dbt-sa.
Navigate to the Keys tab and click "Add Key".
Choose "JSON", then click "Create".
The key.json file will be downloaded to your computer.

Option 2: Using the gcloud CLI

If you prefer using the command line, run:

gcloud iam service-accounts keys create dbt-sa-key.json  --iam-account=dbt-sa@your-project-id.iam.gserviceaccount.com

• Now that we have our dbt-sa-key.json, we need to update dbt's configuration to use the service account. Open profiles.yml and update the value of your keyfile:

data_catalog_dbt:
  target: dev
  outputs:
    dev:
      type: bigquery
      method: service-account
      project: "gcp-data-governance"  # project_id
      dataset: multiplexer_dataset
      threads: 4
      keyfile: "/path/to/dbt-sa-key.json"  # Reference your environment variable
      location: "europe-west6"  # Set this to your BigQuery region

Step 3: Create Data Policy Taxonomies and Tags with Terraform

To allow Terraform to interact with Google Data Catalog we need to specify providers. The providers allow us to manage resources on a specific cloud platform. We have created them in datacatalog.tf as they are relevant to Data Catolg API, however you can make a separate file providers.tf and define them there. The google provider is the standard Terraform provider for managing GCP resources, while the google-beta provider gives access to Google Data Catalog, which is needed for policy tags.

Why Not Just Use google-beta?

Some resources (e.g., IAM, BigQuery datasets) don’t need google-beta, so we keep google for those. On the other hand side Data Catalog resources require google-beta, so we configure both providers.

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 4.0"
    }
  }
}

provider "google" {
  project = var.project_id
  region  = var.region
}

provider "google-beta" {
  project = var.project_id
  region  = var.region
}

A taxonomy is a container containing data policy tags ( a grouping mechanism). Fine-Grained Access Control specified in the activated policy tags ensures that only authorized users can see or query certain columns.

# Create a Data Catalog Taxonomy
resource "google_data_catalog_taxonomy" "multiplexer_pii_taxonomy" {
  provider = google-beta
  display_name = "Multiplexer PII Taxonomy"
  description  = "Taxonomy for sensitive data classification"
  project      = var.project_id
  region       = var.region
  activated_policy_types = ["FINE_GRAINED_ACCESS_CONTROL"]
}

In our test-case we define only two tags: sensitive (PII) and non-sensitive data. The PII tags ensure proper access control. These tags will later be attached to BigQuery columns in dbt. More tags can be added as needed for better governance.

# Create Policy Tags for PII and Non-PII
resource "google_data_catalog_policy_tag" "pii_sensitive" {
  provider = google-beta
  taxonomy     = google_data_catalog_taxonomy.multiplexer_pii_taxonomy.id
  display_name = "PII"
  description  = "Policy tag for Personally Identifiable Information"
}

resource "google_data_catalog_policy_tag" "non_pii_sensitive" {
  provider = google-beta
  taxonomy     = google_data_catalog_taxonomy.multiplexer_pii_taxonomy.id
  display_name = "Non-PII"
  description  = "Policy tag for non-sensitive data"
}

Step 4: Apply Policy Tags with dbt

Now that the Policy Tags are defined, we will attach them to relevant columns in the dbt project:

Set up your dbt project in a directory parallel to your Terraform project. Before running dbt, ensure that your service account key is correctly set (see explanation in Step 2). Remember, dbt profile.yml should be configured to use BigQuery and your service account key. Before running dbt, make sure all dependencies are installed:

dbt deps –upgrade

Afterwards check if dbt can connect to BigQuery, by running:

dbt debug

If successful, you’ll see the message:

All checks passed!

To run all dbt models use dbt run and to run only specific models, e.g., customers use dbt run -s customers

Define columns and attach Policy Tags in your dbt models.

Navigate to your dbt project and modify your customers.yml to attach Policy Tags:

version: 2

models:
  - name: customers
    description: "Customers data with PII"
    meta:
    columns:
      - name: customer_id
        description: "Unique customer identifier"
        policy_tags:
          - "{{ var('non_pii_sensitive_policy_tag_id') }}"
      - name: email
        description: "Customer email"
        policy_tags:
          - "{{ var('pii_sensitive_policy_tag_id') }}"
      - name: phone_number
        description: "Customer phone number"
        policy_tags:
          - "{{ var('pii_sensitive_policy_tag_id') }}"

Now run dbt to apply the Policy Tags to BigQuery automatically: dbt run -s customers

Conclusion
Integrating dbt, Terraform, and Dataplex allows you to efficiently manage BigQuery Policy Tags, enforcing data governance policies in a scalable and automated way. This approach enhances security, compliance, and operational efficiency while reducing manual effort.

To avoid unnecessary charges, remember to destroy your project after testing by running terraform destroy in the command line.

Rusty Backends

Jan Kleine — Tue, 21 Jan 2025 06:30:00 +0000

This post was written together @sekael and @zkck

Rust has been a developer favorite for many years now and is consistently highly admired by developers[1]. It made a buzz when it was introduced as a complementary language to C and Assembly in the Linux kernel with version 6.8[2] and its speed and memory safety make it a popular choice for low-level and performance-critical applications. But with the whole world using the internet, we wanted to find out whether it can also do web and challenge the reigning backend champions like Java Spring or Go.

To find answers, we wanted to get our hands dirty with three popular Rust web frameworks including rocket, axum, and actix, and get a feeling for their performance, features, and most importantly the developer experience.

In this post, we will examine each of these frameworks by implementing the same example API endpoints in each one, connect to a MongoDB database, and run the server in a Docker container.

All three, rocket, axum, and actix, cover the full range of functionality you would expect from a web framework, like routes, handlers, request and response parsing, middleware, state management, database interactions, logging, and testing. Furthermore, all of these frameworks perform well and will meet and surpass the needs of most web applications, so the choice really comes down to ergonomics and how it feels to develop within each framework.

So let’s jump right in…

Rocket (rocket.rs)

rwf2 / Rocket

A web framework for Rust.

Rocket

Rocket is an async web framework for Rust with a focus on usability, security extensibility, and speed.

#[macro_use] extern crate rocket;

#[get("/<name>/<age>")]
fn hello(name: &str, age: u8) -> String {
    format!("Hello, {} year old named {}!", age, name)
}

#[launch]
fn rocket() -> _ {
    rocket::build().mount("/hello", routes![hello])
}

Visiting localhost:8000/hello/John/58, for example, will trigger the hello route resulting in the string Hello, 58 year old named John! being sent to the browser. If an <age> string was passed in that can't be parsed as a u8, the route won't get called, resulting in a 404 error.

Documentation

Rocket is extensively documented:

Overview: A brief…

View on GitHub

Rocket is a web framework for Rust that comes with the most batteries included out of the three we looked at. It positions itself in the same league as Rails (Ruby) or Flask (Python) and aims to offer functionality for anything a web backend might need, including route handling, request and response parsing, database integrations, validation, and so on. One of its main goals is to minimize the amount of boiler plate code you will have to write, and it does so through metaprogramming, i.e. heavy use of Rust macros.

This makes it easy for the developer to integrate her actual logic with the rocket framework. Let’s have a quick look on how you might define a route handler including parameter guards and validation, a database connection pool, and a JSON object.

We can define a route handler simply by adding the appropriate macro to a function.

#[get("/texts/<uuid>")]
pub async fn get_text(db: Connection<TextsDatabase>, uuid: Uuid) -> (Status, Value) {
    // ...
    (Status::Ok, json!({"data": data}))
}

If the request parameter cannot be parsed, the appropriate response code is sent instead of calling the function, so we have the usual comfort of type safety. Request bodies are handled very similarly, and of course, parsing integrates seamlessly with serde.

#[derive(Deserialize)]
pub struct Message<'m> {
    pub data: &'m str,
}

#[post("/texts", format = "application/json", data = "<msg>")]
pub async fn post_text(db: Connection<TextsDatabase>, msg: Json<Message<'_>>) -> (Status, Value) {
    // ...
}

Adding a database is as simple as annotating the appropriate struct and initializing it at startup. Of course many popular DBMS's are supported.

#[derive(Database)]
#[database("texts")]
pub struct TextsDatabase(mongodb::Client);

Rocket goes heavy on the nerdy space vocabulary - which adds to its charm. Thus, to get your web server off the ground, you simply launch it. Notice again, how annotations are used to get the rocket going.

#[launch]
fn rocket() -> _ {
    rocket::build()
        .attach(TextsDatabase::init())
        .mount("/", routes![get_text, post_text])
}

We have found rocket very enjoyable to work with, especially if you want to focus on the business logic of your server and feel safe trusting the framework to take handling of the boilerplate code off your hands. If you are rather new to Rust and maybe switching over from other frameworks that make heavy use of annotations, you will feel at home with Rocket.

Axum

tokio-rs / axum

Ergonomic and modular web framework built with Tokio, Tower, and Hyper

axum

axum is a web application framework that focuses on ergonomics and modularity.

More information about this crate can be found in the crate documentation.

High level features

Route requests to handlers with a macro free API.
Declaratively parse requests using extractors.
Simple and predictable error handling model.
Generate responses with minimal boilerplate.
Take full advantage of the tower and tower-http ecosystem of middleware, services, and utilities.

In particular the last point is what sets axum apart from other frameworks axum doesn't have its own middleware system but instead uses tower::Service. This means axum gets timeouts, tracing, compression, authorization, and more, for free. It also enables you to share middleware with applications written using hyper or tonic.

Usage example

use axum::{
    routing::{get, post},
    http::StatusCode,
    Json, Router,
};
use serde::{Deserialize,

…

View on GitHub

If you are looking for a framework that is more bare-metal Rust and comes with less batteries included, you may want to check out the axum framework developed by tokio-rs, the same team that is responsible for the tokio async library. Opting for axum means you will have to take care of some more of the boilerplate code but you are also not fighting a potentially opinionated framework to get your logic just the way you want it. Compared to rocket, axum does not depend heavily on macros or annotations, and it sets itself apart from its peers by not implementing its own middleware but rather relying on tower for this. Through the tower ecosystem axum can offer timeouts, tracing, compression, authorization, and much more, while also enabling you to share your middleware with applications written with other web libraries like hyper.

Looking at axum’s ergonomics, previous Go developers will feel right at home. The syntax looks very similar to a web server written e.g. with the gorilla/mux module in Go. So how would you go about the implementation of your first route handler including connecting to a database and starting the server? Let’s have a look.

The same two functions from above look something like this:

async fn post_text(
    State(state): State<Arc<state::MongoAppState>>,
    Json(text_payload): Json<payloads::TextPayload>,
) -> Result<
    (StatusCode, Json<payloads::InsertedResponse>),
    (StatusCode, Json<payloads::ErrorResponse>),
> {
    // ...
}

async fn get_text(
    State(state): State<Arc<state::MongoAppState>>,
    Path(text_id): Path<String>,
) -> Result<Json<payloads::TextPayload>, (StatusCode, Json<payloads::ErrorResponse>)> {
    // ...
}

No macros, but a lot more verbose. We see the same when we look at the startup code, we have to do a lot more manually, e.g., pass the DB connection along.

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let client = mongodb::Client::with_uri_str("mongodb://...").await.unwrap();

    let shared_state = std::sync::Arc::new(state::MongoAppState::new(client));

    // build our application with a single route
    let app = Router::new()
        .route("/texts", post(post_text))
        .route("/texts/:text_id", get(get_text).delete(delete_text))
        .with_state(shared_state);

    // run our app with hyper, listening globally on port 3000
    let listener = tokio::net::TcpListener::bind("0.0.0.0:3000").await.unwrap();
    axum::serve(listener, app).await.unwrap();
    Ok(())
}

Axum is a very powerful web framework that benefits from the expertise of the tokio-rs team in creating great Rust tooling. It is being very actively developed and relies on other state-of-the-art crates for middleware. You will most likely enjoy axum the most if you are already familiar with Rust, enjoy having control over a lot of detail in implementation, or maybe if you are switching from another low-level web language like Go.

Actix (actix.rs)

actix / actix-web

Actix Web is a powerful, pragmatic, and extremely fast web framework for Rust.

Actix Web

Actix Web is a powerful, pragmatic, and extremely fast web framework for Rust

Features

Supports HTTP/1.x and HTTP/2
Streaming and pipelining
Powerful request routing with optional macros
Full Tokio compatibility
Keep-alive and slow requests handling
Client/server WebSockets support
Transparent content compression/decompression (br, gzip, deflate, zstd)
Multipart streams
Static assets
SSL support using OpenSSL or Rustls
Middlewares (Logger, Session, CORS, etc)
Integrates with the awc HTTP client
Runs on stable Rust 1.72+

Documentation

Example

Dependencies:

[dependencies]
actix-web = "4"

Code:

use actix_web::{get, web, App, HttpServer, Responder};
#[get("/hello/{name}")]
async fn greet(name: web::Path<String>) -> impl Responder {
    format!("Hello {name}!")
}

#[actix_web::main

…

View on GitHub

Last but not least we want to take a look at actix-web. It combines aspects from both previous frameworks, rocket as well as axum, which is resembled in its ergonomics. Annotations are back on the menu and macros are more heavily used, especially in the definition of route handlers. When implementing middleware or database connections, however, you will not find the same level of abstraction as you might with rocket. Actix is extremely fast 3 and it aims to cater to both experienced Rust developers as well as newcomers who are just starting with Rust development. It ships with its own middleware, e.g. for logging, session management, or cross-origin resource sharing, and allows you to expand the framework with your own middleware that can hook into actix. Let’s have a look at how route handlers, database connections, and starting the server are handled in actix.

#[post("/texts")]
async fn post_text(client: web::Data<Client>, payload: web::Json<TextResponse>) -> impl Responder {
    // ...
}

#[get("/texts/{uuid}")]
async fn get_text(client: web::Data<Client>, uuid: web::Path<Uuid>) -> impl Responder {
    // ...
}

Again, we have a lot less boilerplate. Though launching the server falls in between the first two examples with regards to verbosity.

#[actix_web::main] // or #[tokio::main]
async fn main() -> std::io::Result<()> {
    let db_client = Client::with_uri_str("mongodb://....").await.expect("failed to connect");

    HttpServer::new(move || {
        App::new()
            .wrap(Logger::default())
            .app_data(web::Data::new(db_client.clone()))
            .service(post_text)
            .service(get_text)
    })
    .bind(("0.0.0.0", 8080))?
    .run()
    .await
}

The benchmarks speak for themselves, actix is indeed blazingly fast! And that does not come at the cost of developer experience, which actix manages to keep to a very high standard. It ships with more batteries included than axum, but does not abstract everything quite as much as rocket. If you already feel comfortable writing programs in Rust and other low-level web programming languages, you will enjoy developing in actix, and your user will enjoy the performance!

Acknolegements

Selim wrote the initial draft of this post and Selim, Zak, and I each implemented the sample API using one of the above frameworks.

Cover image: "Brown Chains" by Miguel Á. Padriñán

Selim KaelinFollow

I am a mountain athlete at heart, enjoy learning about and writing code, and passionate about Rust, Cloud, and maps!

Zak CookFollow

Jan KleineFollow

Two Years in the Vault: 4 Best Practices 🔒

Zak Cook — Mon, 09 Dec 2024 08:30:00 +0000

I work as an IT consultant. Over the past two years, we've been working with our client on a platform engineering project, where the use of HashiCorp Vault for secrets management was prevalent. Our Vault has enabled us to manage hundreds of credentials, increasing the security of our developer platform, and has resulted in a thorough and battle-tested configuration of our Vault which we can be proud of.

However, over the past two years, there have been quite a few learning moments. Vault is a great product with lots of flexibility, but this opens up a lot of possibilities for misconfiguration or misuse. For us, that meant a few tedious reconfigurations of our Vault. This post aims to share learnings that I wish we had known from the very beginning, in the form into digestible tips.

These tips are for everyone that needs to work with HashiCorp Vault, whether it be as a developer, or as an administrator (you may still learn something). If you don't use Vault, maybe simply bookmark this post, it may come in handy in the future.

TL;DR:

Configure with an infrastructure-as-code (IaC) tool
Policies describe permissions
KVv2: save raw data, format elsewhere
Beware of write permissions on sys/policy
Consider templated policies (upcoming)
KVv2: make secrets "connection bundles" (upcoming)
Utilize -output-policy from the CLI (upcoming)
Think about your path structure (upcoming)

This is a two-part series, I'll cover the first four tips here, and the remaining four in an upcoming post. Stay tuned.

Glossary: I overuse the words component and system a lot. This could refer to microservices, short running jobs, monolith servers, which all run together, to form some sort of workload. If you're running on Kubernetes, think of a component as a Deployment, and a system as the Kubernetes cluster.

Tip 1: Configure with an infrastructure-as-code (IaC) tool

When first starting to integrate a Vault into a large system, one naturally does explorative work locally with the Vault CLI. This is all good, but concretizing that configuration into infrastructure-as-code (IaC) is essential, and by doing it early you'd avoid the pain of migrating your configuration down the road.

Potential solutions for IaC are:

For configuring your Vault, I would generally recommend going for Terraform/OpenTofu, which takes the declarative approach as opposed to Pulumi. Pulumi is a good solution when you have lots of dependencies in a complex system, which need to be handled programmatically.

We started off on our Vault journey by simply documenting how we configured our secrets engines, auth methods and policies. So when we had to set up a new Vault, that meant going through that documentation and running the commands one-by-one. We at some point (painfully) changed to Terraform/OpenTofu, and it directly opened up many doors for us:

Writing a testing framework, where we would test the access to Vault paths by simulating the components in our system, allowing us to check for any regressions
Reusing the IaC to configure Vaults in multiple environments
Integrate with CI pipelines to automate updates to our Vaults

There's many more advantages. Point is, start with IaC directly, so you don't have to go through the same process as we did, of integrating an already configured Vault with IaC. In case you have to: Terraform Import Blocks saved our lives :)

Tip 2: Policies describe permissions

Policies are core to HashiCorp Vault, being one of the pillars in its access control mechanisms. A Vault policy is a set of permissions, each being a combination of a path and capabilities on that path. Efficient management of policies is important in order to keep your Vault lean and scalable.

Let's consider a very basic Vault, which has a KVv2 engine under secrets/, used to store secret recipes under the recipes/ subpath. The following policy could be the "head chef" policy, allowing the head chef to browse the entire secrets engine and update any of the secret recipes:

vault policy write head-chef - <<EOF
# browse secrets
path "secrets/metadata/*" {
  capabilities = ["read", "list"]
}
# manage recipes
path "secrets/data/recipes/*" {
  capabilities = ["create", "update", "read", "delete"]
}
EOF

We would then attach this policy to a role with:

vault write auth/approle/role/head-chef token_policies=head-chef

Here we decided that the name of the policy would match the role, namely head-chef. This works, but we have found that as policies become larger and more roles are added to the Vault, breaking down policies into smaller, more meaningful sets of permissions becomes increasingly important. Policy names should also describe the permissions encapsulated, and should not refer back to the role that uses it.

To help breaking down policies, it is useful to decide on a naming standard for your policies. An example would be <verb>-<subject of policy>, like read-nuclear-codes. In this example we would have to break down the policy a bit more in order for our convention to apply. Let's split the head-chef policy into two policies, browse-secrets and manage-recipes:

vault policy write browse-secrets - <<EOF
# browse secrets
path "secrets/metadata/*" {
  capabilities = ["read", "list"]
}
EOF
vault policy write manage-recipes - <<EOF
# manage recipes
path "secrets/data/recipes/*" {
  capabilities = ["create", "update", "read", "delete"]
}
EOF
vault write auth/approle/role/head-chef token_policies=browse-secrets,manage-recipes

It is now much more transparent what permissions are associated with this role. In addition, breaking down the policy made parts of the policy reusable, as we could now do something like the following, reusing the browse-secrets policy:

vault policy write manage-greek-recipes - <<EOF
# manage greek recipes
path "secrets/data/recipes/greek/*" {
  capabilities = ["create", "update", "read", "delete"]
}
EOF
vault write auth/approle/role/chef-george token_policies=browse-secrets,manage-greek-recipes

Deciding on this naming convention for our policies allowed us to scale to many more roles in our Vault. One thing to consider though, is that paths may end up overlapping among the policies assigned to a role. This requires being aware of how Vault's priority matching works.

Tip 3: KVv2: save raw data, format elsewhere

Vault's KVv2 engine allows for saving static secrets in the form of key-value pairs. Choosing the appropriate key-value pairs to save in a secret will both maximize flexibility in how your infrastructure consumes these secrets and would minimize duplicated data.

Let's say you have a component which requires credentials to a database, and needs them provided inside a config, e.g. in TOML format. Since the config holds secrets, it may be tempting to save the entire config in Vault's KVv2 engine, and fetch that config when starting your component:

vault kv put secrets/components/my-component config.toml=- <<EOF
[database]
host = "database.example.com"
username = "postgres"
password = "1234"
EOF

However, this approach is limiting:

Reusing the database credentials means copy-pasting to another config, and doesn't allow for a source of truth
Any non-secret config changes need to go over the Vault

This can be avoided: the Vault ecosystem has a large amount of helper components which support templating, among which:

Vault agent templating
External Secrets Operator templating in case you're running on Kubernetes.

As an example let's make use of an ExternalSecret for our templating. First, let's save the standalone credentials instead of our config file:

vault kv put secrets/databases/my-database username=postgres password=1234

Now we'll write an ExternalSecret making use of this raw data:

apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
  name: config
spec:
  # ... (reference to SecretStore)
  target:
    template:
      engineVersion: v2
      data:
        config.toml: |
          [database]
          host = "database.example.com"
          username = {{ .username | quote }}
          password = {{ .password | quote }}
  dataFrom:
  - extract:
      key: /secrets/databases/my-database

This would then render a Secret resource with the following content under the config.toml key, which can be mounted in our component:

[database]
host = "database.example.com"
username = "postgres"
password = "1234"

The structure of our KVv2 engine is now cleaner, as it is not aware of the components that access it, and holds sensible reusable secrets. Another option is to tweak how the component is configured. Instead of taking credentials directly, our component could take a Vault path instead:

[database]
host = "database.example.com"
vault_path = "secrets/databases/my-database"

This however means a dependency on HashiCorp Vault in the code of the application itself, as the code now uses the Vault API in order to retrieve the secret. Some applications may want to be agnostic to the type of software storing the credentials, in which case a templating solution would be more suitable.

Tip 4: Beware of write permissions on `sys/policy`

Understanding this tip is a bit more involved, but please bear with me. It is arguably the most important tip in this series, especially in environments where security is critical.

Vaults used in a complex system sometimes need to support new users being asynchronously added to the Vault: for example adding a new role/user which has access to a subdirectory of a KVv2 secrets engine. To automate this, companies may resort to building some management component which automates the creation of a role/user and policies attached to it.

The sys/policy (or sys/policies/acl) path is used to grant permissions on managing policies in Vault, and is necessary to create policies on the fly. We will see in this chapter how granting write permissions on sys/policy can lead to severe security risks, and provide some examples on how to mitigate these risks.

As an example, let's consider our Vault which is meant to manage recipes. Our company wants to support employees requesting their own subdirectory in our secrets/ KVv2 engine. This would involve the employees sending an API request to a component, let's call it the employee-manager, that would need to execute following commands on the Vault upon receiving one of these requests:

# create policy for employee
vault policy write "manage-recipes-$EMPLOYEE_NAME" - <<EOF
path "secrets/data/recipes/$EMPLOYEE_NAME/*" {
  capabilities = ["create", "read", "update", "delete"]
}
EOF
# create role for employee with attached policy
vault write auth/approle/role/$EMPLOYEE_NAME token_policies="manage-recipes-$EMPLOYEE_NAME"

This would require the employee-manager component to have the necessary permissions to create both roles and policies for our employees. Let's setup a role and policy to allow the component to execute the above commands:

# create "create-policies", "create-roles" policies
vault policy write "create-policies" - <<EOF
path "sys/policy/+" {
  capabilities = ["create", "update"]
}
EOF
vault policy write "create-roles" - <<EOF
path "auth/approle/role/+" {
  capabilities = ["create", "update"]
}
EOF
# attach policies to role
vault write auth/approle/role/employee-manager token_policies=create-policies,create-roles

The above permissions are extremely dangerous to provide to any component or user of Vault, especially in conjunction.

Imagine our employee-manager component was insecure, and is compromised by an attacker. The attacker then uses the component's credentials to log in to Vault. They can now create new policies, with any permissions, and assign those permissions to the employee-manager which they have control over:

# create malicious policy
vault policy write "read-secrets" - <<EOF
path "secrets/data/*" {
  capabilities = ["read"]
}
EOF
# update the compromised role with malicious policy
vault write auth/approle/role/employee-manager token_policies=create-policies,create-roles,read-secrets

# after logging in again, the attacker could execute:
vault read secrets/data/recipes/gordon/super-secret-meatballs-recipe

So the component must technically be considered as admin on the Vault, as it can create policies for absolutely anything and assign it to itself. To mitigate this security risk, there are the following solutions:

Avoid granting permissions on sys/policy altogether and consider using templated policies. This is often the cleanest solution, and has a big advantage of reducing the amount of configuration.
Grant permissions on a subfolder of sys/policy. This subfolder must be disjoint from the policies granted to employee-manager, and employee-manager should not be able to update its own role.
Grant only create permissions on anything under sys/policy, and employee-manager should not be able to update its own role.

There may be more mitigation options, but the point of this chapter is to consider the event of a breach and be aware of the effective permissions that you may be giving to your system components.

That's all the tips for now. Thanks for reading, I hope you learned something, and stay tuned for the second half and other posts to come.

A Very Deep Dive Into Docker Builds

Jakob Beckmann — Tue, 26 Nov 2024 06:25:41 +0000

Containers are everywhere. From Kubernetes for orchestrating deployments and simplifing operations to Dev Containers for flexible yet reproducible development environments. Yet, while they are ubiquitous, images are often built sub-optimally. In this post we will be looking at a full example of a Docker build for a Python application and what best practices to consider.

Disclaimer

This is a real world example from an very small component we built in Python for one of our clients. Very few alterations were made to the original configuration (changing URLs, and removing Email addresses mostly). We will go in depth as to why we did every single little thing. While some stuff is quite Python-centric, the same principles apply to other languages, and the text should be broad enough so that it is understandable how to transfer this example to different languages.

Also, this is a long article, so if you actually plan on reading it, grab yourself a snack and a fresh drink first.

Goal

The goal of this post is to showcase how one can setup a Docker build that is:

fully reproducible,
as fast as possible,
fails early on code issues,
isolates testing from deployed code,
is secure.

The example we will use for this implements quite a lot to ensure only quality code reaches production, and that it can do so as fast as possible. Going all the way might not be necessary for all projects using Docker. For instance, if you release code to production only once a day (or less) you might care less about release build cache optimization. This example is however meant to show the "extreme" to which you can push Docker, so that you can (in theory) push code to production fully continuously (CI/CD principles). But yeah ...

Why

Why do we have these goals? Reproducible builds are one of the most important factors for proper compliance, and for easier debugging. Debugging is simpler, since we ensure that no matter the environment, date, or location of the build, if it succeeds, the same input generates the same output. Moreover, it brings stability, as a pipeline might not suddenly fail on a nightly build (if you still do such things) because a new upstream library or program was released that is used somewhere in your supply chain.

Regarding compliance, we need to be able to tell and revert to the exact state of software that was deployed in the past. Without reproducible builds, using Git to track back to a previous state of deployed code does not help you much, because while you can validate what code you deployed, you don't know what versions of everything else you deployed with it.

Builds should be fast, and fail fast. The reason here is that no one likes to wait. You don't want to wait for 2 hours to figure out whether a tiny code change breaks tests or does not even compile.

You will want to isolate test code from deployed code, because more code equals more bugs. While testing frameworks are very good at isolating test code from code being tested, writing tests generates a risk of bugs. Moreover, the test code is unneeded bloat for your runtime application. Thus it should be isolated from it.

Finally security. While some people think that containers improve security by default, this is not the case. Container technology has the potential to indeed improve the robustness of some security measures and controls. However, in order to achieve this, one needs to correctly utilize containers and build the images with security in mind. For instance, if an image contains certain utilities that allow it to connect to the internet (such as curl or wget), it suddenly makes the container much more vulnerable to container escape attacks (where an attacker manages to move from the container to the underlying host), and hence the whole isolation benefit of the container (which can be a security control) is broken. The same is true for containers that contain interpreters and allow the runtime user to open, edit and execute arbitrary files. As our container will contain Python code, and hence the Python interpreter, this is definitely something we need to take very seriously.

Python Goals

Our example is based on Python, an interpreted language. This is not ideal, as it means that it does not require a compile step. Compilation optimization is however a very important aspect in Docker builds. In order to still address this, I will talk about this, but will not refer to the configuration examples. One could ask why I did not take a compiled language example then. The reason is very simple, I wanted a real world example such that this post is not just theoretical goodness, and most Golang image builds I am currently working on are more basic and not as educational.

Yet another question could be "why deploy Python in Docker in the first place?". This is a very legitimate question. Python requires a lot of OS bloat to just be able to run. This means that typically a VM is a good choice to host it. For all those saying that Docker is still better because of performance (due to faster startup, no hardware virtualization overhead, etc): this is not true for such cases where a large part of an OS needs to be in the Docker image. A VM of a full init-based Linux system can be launched in less than 250ms on modern technology. A full Ubuntu installation with systemd can be completely booted in around 2.5 seconds. The former is in the same order of magnitude that it might take the Python interpreter to just load the code of a large Python application.

So performance cannot be said to be better with Docker, why choose Docker then? Better reasons are that you can strip down a Docker image much easier than an OS. This is critical for us due to security requirements. While Python requires a lot of OS features, the majority of the OS is still bloat. Every piece of bloat is a potential attack vector (each of these unused components might have one or more CVEs that we need to patch, even though we don't even use that software). Another reason is that the build process of Docker is much simpler to manage. There are tools such as Packer that allow similar processes for VMs, but these are not as standardized as the open container initiative (OCI - which Docker adheres to).

Another very important point is the ease of development. Docker and other OCI compliant products provide us with a possibility to build, test, and run our build artefacts (in this case Docker images) everywhere. This makes it very simple and fast for our developers to test the build and perform a test run of an image locally on their development machine. This would not be quite the case with VMs or raw artefacts (JARs, source code archives, ...). Moreover, the OCI ecosystem does not only include specifications on how to interact with images, but also how to setup and configure critical elements such as persistence and networking. These aspects are made very simple with Docker, and would be quite a pain to manage securely with most other technologies.

Finally the main reason for us is the choice of runtime. We have very decent container runtimes (RKE, RHOS, K3s) available to deploy applications. We are very familiar with them, and they offer us a lot of functionality. These all support containers primarily.

A Tiny Bit of Background

Last before we get into the dirty details, a tiny bit of background into what we are building. The application we will be building here is a sort of a facade reverse proxy. It offers a standardized API to clients, which can connect and perform requests. Based on the content of the request, the component will trigger a routing algorithm that defines where the request needs to be routed. This routing algorithm might require several API calls in the backend to different systems to figure out where the call should go. Once done, the component will relay the call to a backend, and forward the response to the client. The client is never aware that it is talking to more than one component, and only needs to authenticate to that single system. Imagine an API Gateway, but where the routing is extremely complex and requires integration with systems such as Kubernetes, a cloud portal, and more.

The Details

Here is an overview of our Dockerfile:

FROM internal.registry/base/ca-bundle:20220405 AS cert-bundle

FROM internal.registry/base/python:3.9.2-slim AS builder

COPY --from=cert-bundle /certs/ /usr/local/share/ca-certificates/
RUN update-ca-certificates

WORKDIR /app

RUN pip install \
  --upgrade \
  --no-cache-dir \
  --ignore-installed \
  --trusted-host pypi.python.org \
  --trusted-host pypi.org \
  --trusted-host files.pythonhosted.org \
  pipenv==2024.2.0

ENV PIPENV_VENV_IN_PROJECT=1
ENV REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

COPY Pipfile Pipfile
COPY Pipfile.lock Pipfile.lock

RUN pipenv install --deploy

### Tester image
FROM builder AS test

RUN pipenv install --dev --deploy

COPY ./pyproject.toml pyproject.toml
COPY ./assets/ ./assets
COPY ./features/ ./features
COPY ./tests/ ./tests
COPY ./src/ ./

RUN --mount=type=cache,target=./.mypy_cache/ \
  --mount=type=cache,target=./.pytest_cache/ \
  pipenv run mypy . \
  && pipenv run black --check . \
  && pipenv run bandit -ll ./*.py \
  && PYTHONPATH=./ pipenv run pytest


### Runner image
FROM internal.registry/base/distroless-python:3.9.2
LABEL maintainer="Redacted <redacted-email>"

COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
ENV REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

WORKDIR /app/
USER 1000

COPY --from=builder --chown=1000 /app/.venv/lib/python3.9/site-packages ./my-app
WORKDIR /app/my-app

COPY --chown=1000 ./src/ ./

ENTRYPOINT ["python3"]
CMD ["./main.py"]

We will go through it line by line and figure out why we did what we did, and why we did not choose a different approach. Let's start!

FROM internal.registry/base/ca-bundle:20220405 AS cert-bundle

In this line we reference a container image for later use and provide it a name alias cert-bundle. This container image contains only data: our production network proxy certificates and all internal certificate authorities. We need these CAs as we will connect over TLS to backend components that have internal certificates. We also need the production network proxy certificates as we will pull dependencies straight from the internet, and all that traffic is routed over a gateway proxy. Why distribute these certificates over a Docker image instead of a compressed TAR? The main reason is that we want to have a unified way that we build artefacts and manage CI/CD pipelines. By creating and managing the certificates via Docker, we can use our entire Docker setup (such as UCD/Jenkins/Tekton pipelines for building, registry for distribution, quality gates for security, etc) and do not need to have a different system to manage the certificates. Note that we refer to the exact state of the certificate bundle (20220405), which refers to the state of the certificates per 5th of April 2022. This is very important to make the build reproducible. If we did not pin the version of the certificates, it would mean that we could build the image maybe today, but it would fail tomorrow, once the certificates change (even though we did not change the code at all). You will note that we will pin every single version in the entire build process.

FROM internal.registry/base/python:3.9.2-slim AS builder

In this line, we reference the base image we will start building from. This is the official Python image for Python version 3.9.2. We use the slim version because we don't need much more than the standard Python installation. We pull this from our own registry, as all Docker images are scanned beforehand to reduce the risk of supply chain attacks. Also here, the version is pinned. We provide this build step the builder alias. In essence this means that starting from this line we define an image stage that will contain the build process of our application. For Python, this mostly includes downloading dependencies (both software and system level), and injecting the source code, as there will be no compile step.

COPY --from=cert-bundle /certs/ /usr/local/share/ca-certificates/

This copies our certificates into our build image. We do this by referencing the build step cert-bundle (see first line of the Dockerfile again) in the --from argument of the COPY command. Note that we could have referenced the image directly in the --from argument. We choose to use build stage aliases for visibility, and reduce duplication if the certificates need to be copied into different stages. Note that this copies only the raw certificates. A OS specific bundle would still need to be generated.

RUN update-ca-certificates

Here we do exactly this, we generate a certificate bundle for the underlying OS of our builder image (Debian). This allows our subsequent build steps to use the certificate bundle to validate host certificates on TLS connections.

WORKDIR /app

We then set a working directory. The idea is to have a base directory on which we now operate. This can be nearly any working directory, and will be created if non-existent. We choose /app/ by convention. Moreover, note that we tend to reference directories with the trailing / to make it more explicit that we are referencing directories and not files. We use this convention throughout the configuration/code.

RUN pip install \
  --upgrade \
  --no-cache-dir \
  --ignore-installed \
  --trusted-host pypi.python.org \
  --trusted-host pypi.org \
  --trusted-host files.pythonhosted.org \
  pipenv==2024.2.0

We use an environment virtualization technology for Python. This is called pipenv. It allows us to have many different versions of the same dependency installed locally, without them conflicting. This is very important when you are developing many applications at the same time locally. By running this line we install version 2024.2.0 of pipenv (pinned). Other than Python itself, these are the only tools required for our Python development environment. If we were using a different language, pipenv would be substituted with your dependency management tool (such as Maven for Java). Note that we only install pipenv itself, we do not install the dependencies. Also using the flags provided we ensure a fully clean install of pipenv.

This is an example where we reach out to the internet and thus needed the network proxy certificates.

A very good question here might be "why to use pipenv at all, considering it is typically used for environment virtualization, which is already covered by Docker itself?". There are two aspects here. The first is to allow us to lock dependencies using their hash, which is not natively supported by pip (the standard Python package manager). The second is that we want to keep the build process within Docker as close to the build process outside of it. While we do not build artefacts outside of Docker per-se, the IDEs of our developers need to fall back on these technologies to support features such as library-aware code completion, type-checking, test integration, debugging, etc. This could also be achieved by connecting the IDE to an instance running directly in Docker. This however is relatively complex and requires the setup to support remote debugging. In theory, these are not really problems as long as the dev environments are uniform, but we allow each developer to work with the tools he/she desires to develop code. It then suddenly becomes very difficult to have a stable setup that works for everybody, especially considering that some of our developers do not want/know how to configure their environments to that level (client-server debugger setups, network and volume management between the IDE and Docker, ...).

ENV PIPENV_VENV_IN_PROJECT=1
ENV REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

Here we set environment variables for pipenv. Firstly we want the dependencies to be installed directly in the project repository, not centrally. This allows us to ensure that we do not accidentally copy a system Python dependency that installed by default with the base image. The second configures the certificate bundle we generated in the beginning to be used by pipenv. It does not use the system configured bundle by default, so it needs to be configured manually here.

COPY Pipfile Pipfile
COPY Pipfile.lock Pipfile.lock

Now the interesting stuff. Here we copy the dependency files into the image. The first file contains a list of dependencies that we use for our project. The second contains a hash the dependencies should have, including indirect dependencies (dependencies of dependencies), in order to ensure that we always get exactly the same dependency code for very install. The first looks as follows:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
requests = "==2.28.2"
pydantic = "==1.10.4"
# more dependencies ...

[dev-packages]
black = "==23.1"
bandit = "==1.7.4"
pytest = "==7.2.1"
pytest-mock = "==3.9.0"
pytest-bdd = "==6.1.1"
mypy = "==1.1.1"
types-Pygments = "==2.14.0.6"
# more dev dependencies ...

[requires]
python_version = "3.9"

Note that we split dependencies into normal packages we require for our application, and packages only required for testing and our quality gates ([dev-packages]). This is important later on, as we do not wish to have packages only required for testing in our production Docker image.

I will not show you an example of the lock file, as it contains mostly checksum hashes. Simply trust me that it contains the exact checksum that every package (such as the dependencies of requests) has to have to be installed. The reason this is required in the first place, is because the dependencies of requests are likely not pinned to an exact version and might thus change between installations unless locked via our Pipfile.lock. This would undesired as it would make our builds un-reproducible. The lock file itself is generated by our developers in two different scenarios. The first is when a library is added due to some new feature. In such a case the new library is added to the Pipfile, and an installation is triggered outside of Docker. This will install the new library and potentially update already installed ones (in case of conflicts). Hence new hashes will be added to the lock file. The second is on a lifecycle of the existing libraries or of our Python version. In such a case we update the pinned version in the Pipfile and trigger an installation outside of Docker. Again, pipenv would then update the direct dependencies, and potentially transitive ones, and update their hashes in the lock file.

RUN pipenv install --deploy

Here we install the dependencies for our application. The --deploy flag means that we want to install the dependencies based on the lock file. Moreover, we do not install the dev packages yet, only the ones needed for the production code.

### Tester image
FROM builder AS test

Here we generate a new Docker build stage. We have generated a stage with builder that contains the required certificates and the production dependencies, and nothing more. We now want to test our code and validate quality gates. We do not want to perform this in the builder stage, because it would pollute our production dependencies. Moreover, using a different stage allows to trigger builds more granularly with BuildKit. For instance, I would be able to configure (with --target=test) to only build the image up to the test stage, and skip any later stages (such as the runtime image in our case). This can be very useful in pipelines, for instance, where we want to run the test on every commit, but are not interested in building a real artefact unless the commit is tagged.

With this line we essentially say "start a new stage called test from the latest state of builder". We also add a comment above to make it more visible that we are starting a new stage in the Dockerfile. Stage comments are typically the only comments we have in the Dockerfiles.

RUN pipenv install --dev --deploy

In this line we now deploy the development dependencies, including tools for quality checks (mypy, bandit, black, see below for details) and for testing. Again, we use the --deploy flag to ensure we always use the same versions to make the build fully reproducible.

COPY ./pyproject.toml pyproject.toml
COPY ./assets/ ./assets
COPY ./features/ ./features
COPY ./tests/ ./tests
COPY ./src/ ./

Here is the first time we copy actual content, other than the list of dependencies, into our image. This means that up until now all layers in the build process can be fully cached if we perform code changes. Thinking about this is primordial if you want an efficient build process. Even here, we copy code in reverse order based on likelihood of change. The first file we copy in configures our tooling and quality gates. This is unlikely to change unless we introduce a new tool or change configuration of an existing one. An example of the file can be seen below.

The second line copies assets. These are used for testing, such as test configurations for configuration validation etc. These are also quite unlikely to change unless we write new tests of our configuration classes.

The third line copies in our Cucumber files for BDD testing. These change only when we either define new behavioral tests or add features.

The fourth line copies our test code, this is quite likely to change, as it contains all our unit tests, and the testing framework for behavioral tests.

Finally the last line copies in our actual code. This, along with the unit tests, is the code that is most likely to change, and thus comes last. This way on a code change, all lines up to this one (assuming we did not add/change tests) can be used from cache.

[tool.black]
line-length = 100

[tool.pytest.ini_options]
pythonpath = [
  "src",
  "tests",
]
bdd_features_base_dir = "features/"

[tool.mypy]
exclude = [
  '^tests/.*\.py$',
]
ignore_missing_imports = false
warn_unused_configs = true
warn_redundant_casts = true
# more settings ...

[[tool.mypy.overrides]]
module = [
  "kubernetes",
  "parse_types",
]
# skip libraries without stubs
ignore_missing_imports = true

RUN --mount=type=cache,target=./.mypy_cache/ \
  --mount=type=cache,target=./.pytest_cache/ \
  pipenv run mypy . \
  && pipenv run black --check . \
  && pipenv run bandit -ll ./*.py \
  && PYTHONPATH=./ pipenv run pytest

This line aggregates our quality gates and testing. For quality gates we have:

mypy: checks typing information where provided. We do not perform strict typing so that type information is required everywhere, but we validate that the provided typing is correct.
black: checks formatting of the code to ensure it is according to your guidelines.
bandit: performs basic security checks. This is a non-blocking check, meaning that the build will only fail if issues of severity MEDIUM or higher a found. LOW severity check fails are ignored.

Finally we run our testing (with pytest). We run the testing last, as it is the most time consuming of the tasks, and it does not need to be executed if the code fails to adhere to our standards. Note that you could add any other gates here, such as a code coverage baseline that needs to be adhered to, various code analysis checks, or security scans. We only perform one more security check against dubious code and supply chain attacks during the build process. This check is however done on the final Docker image and is thus executed by the pipeline itself outside of the Docker build process.

Note that all commands are executed as one RUN statement. This is best practice, as none of these commands can be cached individually. Either all have to be executed again if layer it builds upon changed, or none has to run. Putting them into the same RUN statement generates a single new layer for all four commands, which reduces the layer count and build overhead for Docker.

Finally, note the --mount options passed to RUN (introduced with BuildKit 1.2). These allow to cache content within the Docker build between builds. Here we mount two caches, one for mypy and one for pytest. These ensure that if a subsequent Docker build is triggered for code that does not affect some files, the typing checks and tests are not run again for these files, but taken from the cache. For pytest this is actually done on a "per-test" basis, ensuring tests are not run unless code they are testing is changed. Such caches can massively increase the speed of your pipelines, especially when your project grows and the test suites start to take more time to run through.

### Runner image
FROM internal.registry/base/distroless-python:3.9.2

This defines the runner image. We are done with testing and want to build the productive artefact, as all checks have passed. In a compiled setup, this would mean we would now have a release compilation stage (before building the runtime image). This is done after testing as the release binary/JAR will be compiled with optimizations, which can take quite long, and is unnecessary if the tests fail anyways. Thus in a compiled language like Java or Golang, we would now continue from the builder again, copy the code back into the layer, and compile. Here one should be careful, most languages support incremental compilation to reduce compilation times. When this is supported, one needs to mount a build cache, or the incremental compilations from previous builds will be lost every time the code changes, as the entire compilation layer will be discarded from the cache. This is done the same way as in the previous block, with --mount parameters.

Once the compilation is completed, and we have our final artefact (binary or JAR), we want to copy it into the runtime image. The idea is again to restrict bloat to reduce our attack surface. For instance, in a Java setup, we only need a working JRE to run our application, we no longer need Maven, the Java compiler, etc. Thus, after the build process, we use a new stage for the runtime image. This is what we did for Python here, since we have no compilation step. We use a different image than our initial internal.registry/base/python:3.9.2-slim image, as we no longer need pip (the Python package manager), and other bloat. Instead we use a distroless image, which is essentially a stripped down Debian image containing truly the base minimum to run Python code, but nothing to manage it, etc. Again, we use our own copy of the distroless image from our scanned registry.

LABEL maintainer="Redacted <redacted-email>"

This line adds metadata to the image. This is not necessary to have a good image, but useful when using images that are shared across huge organisations. This is the official maintainer label we use, where we reference our team, such that anyone that downloads the image and inspects it can see who built it, and how to get into contact with us in case of issues.

COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
ENV REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

Same as before, we copy certificates and configure Python to use our bundle. Note that this time we directly copy the bundle generated in the builder and not from the certificate image, as we need a bundle and cannot create it in this image (update-ca-certificates is not contained in the distroless image). We need to copy this explicitly since we started from a fresh image. The test stage had the bundle implicitly configured from the builder stage, upon which it was set up.

WORKDIR /app/
USER 1000

We set a working directory again. This is also necessary since starting from a fresh image. Also we set a non root user. This is necessary since we do not want to run our code as root for security reasons (reduce the impact of a remote code execution - RCE vulnerability). Note that any statement after the USER statement will be executed in the context of that user. Therefore I would for instance not be allowed to run update-ca-certificates (if it was present in the image) in a RUN statement from now on, as this requires root privileges.

COPY --from=builder --chown=1000 /app/.venv/lib/python3.9/site-packages ./my-app
WORKDIR /app/my-app

Here we copy the non-dev packages from the builder stage into our productive image. Note that we use a path from within the project root (/app/), since we set pipenv to install the virtual environment directly in the project (the PIPENV_VENV_IN_PROJECT variable). We copy the site-packages (the dependencies) directly into a subfolder, in which our application will live. This ensures that they are treated as if we wrote them ourselves, as individual Python modules in our code. They essentially become indistinguishable from our own code. This allows to keep consistency in our module names are resolved. Note we need to add the --chown flag, as the dependencies were installed by the root user in the builder image, and they need to be readable by our user 1000 that will run the application. The --chown flag will change the files' owner (and group) to the provided argument.

The second line simply sets the new working directory to be the new project directory into which we copied the code from the dependencies.

COPY --chown=1000 ./src/ ./

Here we copy the source code back into the production image. We did this after copying the dependencies, such that the dependency layer can be cached again. Moreover, we only copy the source code, no tests, no assets, no Cucumber features. All these latter ones are not needed to run our application. Finally note that we copy it not from the test stage, but again back from the outside build context. This is because we mock a lot during testing, changing some code behavior dynamically. Copying it back in from the outside context ensures we do copy the exact code that is in our Git repository, and not something that was accidentally modified during testing, etc.

ENTRYPOINT ["python3"]
CMD ["./main.py"]

Finally we set an entrypoint and a command. The entrypoint defines what will always be executed on a Docker run (unless explicitly overwritten), and the command provides the default arguments unless overwritten via the Docker run arguments. We always use lists instead of full strings to ensure that the arguments get passed to the Kernel as system calls instead of being executed by a shell. This is important to ensure proper signal handling (when you want to terminate containers), and since there is simply no shell in the distroless image we are building.

That's it

Holy molly... There is a lot that goes into building a simple Docker image. And that considering we did not even compile anything, which would require a decent amount of extra work, and that all our tooling can be managed directly via pipenv and do not need to be installed separately via curl or some OS package manager.

So is it worth it? To put so much thought into how a simple Docker image gets built? I would argue yes. I will not start an idiomatic discussion on the benefits of smaller images, security best practices, or having tests being run directly in the Docker build. If you want such a discussion, go to Reddit or Youtube, you find plenty of beef between people fighting about these topics like their life depends on it. All I will say is this:

I can run docker build ... after each save on a file, since the caching is optimized to a point where a full build on a code changes takes about 1-2 seconds. Being able to run this so often gives me the confidence that what I will push will actually pass in the pipeline.
Using proper caching makes me avoid having to wait 2-5 minutes each time I want to compile something. Since 2-5 minutes is typically too little for a context switch to something else, it might be time I would have just sat around thinking about how much it sucks to wait on stuff. So it has considerably improved not only my productivity, but also my mood.
Docker avoids some "it works on my machine" issues. With proper version pinning and fully reproducible builds, it really nearly eradicates the issue. Now the only time something like this can happen is when running on different Docker versions.
We all sometimes would like to fix tests by skipping them to "save time" when something needs to go to production quickly. Since testing is fully baked into the build process, changing flags on Jenkins/Tekton/whatever will not allow you to skip any testing or quality checks on the code. The only way would be to comment out the test code or update the Dockerfile, which would not pass a PR review. This gives me immense peace of mind.

Since the build process and testing is (nearly) fully defined in the Dockerfile which lies in the git repository, we nearly never need to change pipelines to add/change/remove anything, as all of this can be done in the repository of the corresponding image directly. This also has downsides, as it creates duplication. I would argue that this is beneficial though, as legacy applications might not be able to switch to newer tooling as fast as greenfield projects, which want to leverage that new tooling. Having this "configured" in each repository allows each to move at its own pace. Strict guidelines (such as we don't want to use tool X anymore) can still be enforced on pipeline level via container scanning tools (which you will need either way).

What's the major downside of this approach? Well I would argue there is one large one. Many people might not understand Docker well enough to figure out how the build process works, or might not have time to invest to learn how to do it correctly. This means that some people might not be able to make changes to the build processes by themselves and need might help. I think this would also be the case without a proper Docker setup, but maybe this problem is augmented by having a slightly more complex Docker build setup.

I hope this has given you some food for thought. Feel free to comment any questions or remarks below, or to reach out! Do you also take your Docker builds this far?

Jakob BeckmannFollow

I am passionate about cloud native software and infrastructure, security tooling, and various programming languages. I am currently a Principal Architect at ipt.ch, working on a variety of mandates.

Innovation Process Technology AG (ipt) Follow

We are a boutique IT consultancy based in Switzerland focused on building individual solutions using leading edge technology. For more information visit our website: https://ipt.ch

Forem: Innovation Process Technology AG (ipt)

Beyond the Pod: Why wasmCloud and WebAssembly Might Be the Next Evolution of the Platform

WebAssembly as a Platform Foundation

WebAssembly: The Component Model

WebAssembly: Secure by Default

WebAssembly: Performance

wasmCloud Architecture

Hosts

Lattice

Capabilities

Providers

Components

Applications

wadm

Verdict

That Time We Found a Service Account Token in my Log Files

What This Article Covers

Why This Matters

What Is a JWT?

Closer Look at aud

Kubernetes and ServiceAccounts

ServiceAccounts 101

The Catch

Enter Vault: The Gatekeeper of Secrets

Vault Authentication Methods

Kubernetes Auth Method

JWT Auth Method

Projected Tokens: Because It's 2025

What You Get

Example Pod with Projected Token

Why Vault Loves This

Back to the Log

Try It Yourself: Vault + K8s AuthN Lab

What's Inside

Final Drop 🎤

Acknowledgment

Dissecting Kubewarden: Internals, How It's Built, and Its Place Among Policy Engines

Policy Engines

Kubewarden Architecture

Comparison

Verdict

Your Cluster Deserves Better Traffic Management. Enter: Gateway API.

Current Alternatives

NodePort: Exposing a Service on a Node’s Port

LoadBalancer: External Load Balancers for Services

Ingress/Routes: Layer-7 Routing for HTTP/S

From Ingress to Gateway API: Why a New API?

GatewayClass

Gateway

Routes: The Core of Gateway API’s Flexibility

Gateway Controllers and Implementations

Managing TLS Certificates with Gateway API

Conclusion

The Tortoise and the Hare: do AI Agents Really Help for Software Development?

I am an AI Skeptic

AI Support Agents

CodeRabbitAI

Starting Small

A More Complex PR

Dashboards!

Verdict

f4z3r / gruvbox-material.nvim

Material Gruvbox colorscheme for Neovim written in Lua

Gruvbox Material

A NeoVim colour scheme in pure Lua allowing for highly flexible configuration and customization.

Features

f4z3r / sofa

A command execution engine powered by rofi.

Sofa

A command execution engine powered by rofi and fzf.

About

Examples

For Snippets Management

A Comprehensive Guide to Managing Large Scale Infrastructure with GitOps

GitOps Overview

GitOps is More than CI/CD Pipelines

The Advantages of GitOps

Tooling

Infrastructure: Disambiguation

App-of-Apps

Closer Look at `aud`

A command execution engine powered by `rofi` and `fzf`.

Tip 4: Beware of write permissions on `sys/policy`