Forem: Felix Schuster

Setting up a "confidential" GitLab🦊🔒

Felix Schuster — Sun, 20 Nov 2022 11:50:06 +0000

Hey there. If you ever programmed something "sensitive" or, like me, are managing private meeting notes with Git, you probably know this uneasy feeling when pushing to GitHub. It's simply impossible to tell who may get access to your private repos. There could be malicious insiders leaking stuff or there could be bugs allowing other users to access data etc.

This uneasy feeling is one of the primary reasons for people to use self-managed GitLab. Especially companies concerned about leakage of IP like to run GitLab on-prem. However, maintaining on-prem infrastructure is expensive, tedious, and often not even an option.

Confidential computing to the rescue!

Recent server CPUs from AMD and Intel come with so called confidential computing features. With these, sensitive workloads can be kept encrypted during runtime in memory and are shielded from the rest of a system. Not even system administrators, cloud provider employees or privileged attackers can access workloads protected this way. And this property can even be verified remotely 🤯

Confidential computing effectively lets you create your own runtime-encrypted and isolated "on-prem" on the public cloud. It's like having your cake and eating it 🍰

If you want to learn more about confidential computing, I recently published a whitepaper on it. The Confidential Computing Consortium also has some good resources.

How to make GitLab🦊 "confidential"?

Ok, now that we sort of know what confidential computing is - how do we apply it (correctly) to GitLab and the software it relies on, e.g., PostgreSQL, NGINX, and Redis?

Easy! I recently posted about our open-source project Constellation. Constellation is the first confidential Kubernetes distribution. Think Rancher Kubernetes Engine (RKE) or RedHat OpenShift for confidential computing.

Constellation encrypts and shields entire K8s clusters with the help of confidential computing. Basically, Constellation ensures that all data in a cluster is always encrypted (at runtime, at rest, and in transit) and that the cluster cannot be accessed from the infrastructure.

So making GitLab "confidential" is as simple as running it with Constellation on confidential computing enabled hardware, which is available in Azure and GCP.

Running GitLab🦊 on Constellation💫

The first step is to download and install the Constellation CLI.

Next, a Constellation cluster can be created in a few steps. In essence:

constellation config generate azure
constellation create --control-plane-nodes 1 --worker-nodes 2 -y
constellation init
export KUBECONFIG="$PWD/constellation-admin.conf"

We can now connect to the cluster with kubectl or other tools using the auto-generated constellation-admin.conf. The config ensures that the connection "confidential" and terminates inside the correct cluster. (No man-in-the-middle possible.)

Installing GitLab (manually)

We want our "confidential" GitLab to be accessible from the Internet under its own URL. For this, we need to install and configure external-dns in our freshly created cluster.

Further, we need to install Constellation's CSI driver for Azure. The driver ensures that all data written to cloud storage from the cluster gets encrypted automatically. With the upcoming v2.3 release of Constellation, it will be installed by default.

Finally, we can install GitLab via the official Helm chart, which also installs Redis, PostgreSQL, NGINX, and other components in the cluster.

Installing GitLab (automated)

We also created a custom Helm chart that installs and configures all of the above. This can be used as follows:

export GODADDY_API_KEY=<your creds here>
export GODADDY_SECRET_KEY=<your creds here>
export TARGET_DOMAIN=<your domain, e.g. gitlab.edgeless.systems>
export TLS_ISSUER_EMAIL=<your e-mail address>
helm dependency update ./gitlab
helm upgrade gitlab-infra ./gitlab --install --namespace default --set infra.enabled=true --set apiKey=$GODADDY_API_KEY --set secretKey=$GODADDY_SECRET_KEY
helm upgrade gitlab-app ./gitlab --install --timeout 600s --set app.enabled=true --set gitlab.global.hosts.domain=$TARGET_DOMAIN --set gitlab.certmanager-issuer.email=$TLS_ISSUER_EMAIL --namespace gitlab --create-namespace

Note that the above assumes you're using GoDaddy as your domain provider. It should be easy to adapt to other providers.

Live demo📺

I deployed our confidential GitLab to https://gitlab.edgeless.systems/ and synced the Constellation repo from GitHub.

What are the security properties?

Constellation ensures that the entire K8s cluster runs in runtime-encrypted and isolated Confidential VMs (CVMs) and that all data that leaves the CVMs remains encrypted. For example, if Redis or PostgreSQL write data to cloud storage that data is automatically encrypted. The corresponding cryptographic keys are all generated and managed within CVMs. We as DevOps engineers don't need to worry about these. Bottom line is that all data is always encrypted, even during runtime in memory.

Constellation also verifies the integrity and authenticity of all CVMs and that they all are running the same "good" Constellation node image. This way, once constellation init completes, you know that you are actually talking to an end-to-end confidential cluster via kubectl.

You can learn more about these concepts in the Constellation docs.

What about user connections?

As I discussed already in a previous post on confidential Rocket.Chat, users of the service should also have means to verify that the service is confidential and that it is the right instance they're talking to.

For browser-based connections, currently, the only way is to manually check the fingerprint of the TLS certificate. In the case of our demo instance of GitLab (https://gitlab.edgeless.systems/) the SHA-256 fingerprint is CB C2 F9 23 7B 46 66 FC 5B 88 70 D5 3C 73 24 98 B4 EC 1D 12 EC 38 85 8C FA A6 44 B2 43 73 34 18🧐

For Git CLI-based connections, things are simpler. By adding the following entry to our gitconfig file, we can ensure that Git always checks that the right certificate is present when talking to https://gitlab.edgeless.systems/:

[http "https://gitlab.edgeless.systems/"]
    pinnedPubkey = sha256//aZTzzCepU+Sa34+xkqyFGiWXG+/yHtF6Q5AgMMAjHhs=

With this in place, we know for sure that commands like git clone https://gitlab.edgeless.systems/edgelesssys/constellation.git or a corresponding git push will always go to the right confidential instance of GitLab and can't be diverted or intercepted by an attacker in control of the host of gitlab.edgeless.systems or the corresponding DNS entries.

Wrap up

Hope you enjoyed the quick walkthrough. Personally, I have to admit that I love the idea of having a GitLab running on Azure, which can't be accessed from the infrastructure and keeps all of the data always encrypted.

What do you think? Any suggestions for apps we should make "confidential" next?

How to set up "always encrypted" Rocket.Chat🚀 on Kubernetes

Felix Schuster — Tue, 01 Nov 2022 13:01:03 +0000

I recently posted about the open-source release of our "always encrypted" K8s distro Constellation.

This post is about running Rocket.Chat on Constellation, which gives, well, "always encrypted" Rocket.Chat 🙂

This is cool, because "always encrypted" Rocket.Chat shields the entire server-side of Rocket.Chat from the (cloud) infrastructure and ensures that all data is always encrypted - even during processing in memory.

What is Constellation💫?

The idea of Constellation is to shield entire K8s clusters from the cloud infrastructure using Confidential VMs, which are available on recent AMD EPYC server CPUs. Constellation ensures that all data in the cluster is always encrypted (even during processing) and is inaccessible from the infrastructure. Based on the CPUs' remote attestation feature, Constellation makes this verifiable for the DevOps engineer.

What is Rocket.Chat🚀?

Rocket.Chat is an open-source communications platform. In less fancy terms, it's a chat server with a web frontend. Think of it as a self-hosted alternative to Slack. It's website touts it as a "secure and compliant communications platform".

Why do we need "always encrypted"🔒 Rocket.Chat🚀?

Rocket.Chat offers E2E encryption of messages but that comes with some downsides for UX (e.g., no more search and user needs to manage the key manually) and also, conceptually, cannot protect metadata, server configurations, code etc.

Consequently, different infrastructure-based attackers, for example someone with root access to a K8s node, can still learn and manipulate a lot. In fact, such an attacker could even inject malicious JS into client sessions and leak Rocket.Chat's E2E encryption keys or plaintext messages or just disable encryption. So it's probably fair to say that Rocket.Chat's E2E encryption mostly helps against passive attackers who are able to read or intercept some server-side data.

If we want to protect against the active attackers from the picture above, we need "always encrypted" Rocket.Chat 💪

Running Rocket.Chat🚀 on Constellation💫

Running Rocket.Chat on Constellation turned out to be pretty straightforward. In the first step, I created a Constellation cluster:

constellation create --control-plane-nodes 1 --worker-nodes 2 -y
constellation init
export KUBECONFIG="$PWD/constellation-admin.conf"

Note that I created my cluster on Azure. If you don't have an Azure or GCP subscription but want to try this out nonetheless, there is also MiniConstellation, which you can run locally and doesn't require confidential-computing hardware.

Installing Rocket.Chat

My goal was to make Rocket.Chat available under its own URL with TLS. For this, I installed and configured the following components via Helm charts on my freshly created Constellation cluster:

NGINX Ingress Controller
cert-manager for managing the TLS certificate
Constellation's azuredisk-csi-driver to get encrypted persistent storage on Azure (note that this will be automatically installed with the upcoming Constellation 2.2 release)
Finally, Rocket.Chat itself, which also brings MongoDB

If you're not in the mood to do this manually, we also created a combined Helm chart. The chart details how each component needs to be configured. After you set your desired hostname in the values.yaml, the chart can be installed as follows (assuming GoDaddy as domain provider):

export GODADDY_API_KEY=<your creds here>
export GODADDY_SECRET_KEY=<your creds here>
export MONGO_USER_SECRET=$(openssl rand -base64 32)
export MONGO_ROOT_SECRET=$(openssl rand -base64 32)
helm dependency update ./rocketchat
helm upgrade rocket-infra ./rocketchat --install --namespace default --set infra.enabled=true --set apiKey=$GODADDY_API_KEY --set secretKey=$GODADDY_SECRET_KEY
helm upgrade rocket-app ./rocketchat --install --namespace default --set app.enabled=true rocketchat.mongodb.auth.password=$MONGO_USER_SECRET --set rocketchat.mongodb.auth.rootPassword=$MONGO_ROOT_SECRET

Done! 🏃‍♂️🏁

The result is publicly accessible at https://rocket.edgeless.systems/ Feel free to come over and say "hi" 🙂

What does this give us?

Running Rocket.Chat on Constellation ensures that our entire deployment, including MongoDB and all other components that we installed above, is shielded from the infrastructure and that all data is always encrypted.

What does this mean?

Constellation ensures that all K8s nodes run on AMD-based Confidential VMs (CVMs). CVMs are strongly isolated from the host and remain encrypted in memory at runtime. Constellation also ensures that all nodes run the same minimal mkosi-based node image.

How does Constellation ensure these properties?

Constellation leverages the hardware-based "remote attestation" feature of CVMs. The Constellation CLI uses remote attestation to verify the first node in the cluster and to bootstrap a secure connection to the cluster. All subsequent nodes that are created in the cluster are verified by existing ones. During the verification process, it is checked if a "good" node image is running on "good" hardware with a "good" configuration. Constellation knows what a "good" node image is, because we sign and publish the measurements (i.e., hashes) of each node image we release.

Once a node has been verified, it gets access to a Cilium/Wireguard-based VPN, which is shared by the nodes. A verified node also receives a key for storage encryption. Nodes ensure that all data they write to the network or to storage is encrypted. The K8s admin doesn't have to deal with key management. It all happens automatically.

If we put all of this together, we get a K8s cluster that

keeps all data encrypted on the network, in storage, and in memory.
is isolated from the underlying infrastructure (incl. hypervisor and host OS).
is verified based on "remote attestation".

The Constellation docs have more details on the architecture and security features.

What about user connections?

Users connect to Rocket.Chat via the browser, e.g., by browsing to https://rocket.edgeless.systems/. TLS connections to Rocket.Chat on Constellation terminate inside CVMs, so the encryption is end-to-end.

But how does the user know for sure? Unfortunately, there is no direct way for the user to tell. However, they can manually inspect the TLS certificate (by clicking on the lock symbol in the browser bar) and confirm with their peers if its the expected certificate. The SHA-256 fingerprint of the server I created is CB 2C 2D 96 E6 B2 59 9D DC FD 99 C5 7A 80 58 60 3C A1 22 99 6D 28 BF FA AD 5B F0 52 EF F0 31 C5, btw. 🙂

Ideally, a browser plugin would exist that would allow pin a certificate for a domain. Ideally, the plugin would also be able to verify hardware-based remote attestation statements. This is future work 🙂 Even without such a plugin, Rocket.Chat on Constellation protects against many types of infrastructure-based attackers and accidental data leaks.

Wrap up

Hope you enjoyed this walkthrough. Personally, I'm super-excited by the possibilities that confidential computing and Constellation create. What do you think? Any other apps that you'd like to see become 🪄 "always encrypted"?

If you're interested in learning more about confidential computing, I recently wrote a whitepaper on the core concepts.

What are your thoughts about "always encrypted" Rocket.Chat, Constellation, or confidential computing in general?

We built the first always (!) encrypted K8s 🔐

Felix Schuster — Fri, 23 Sep 2022 15:28:55 +0000

Hey there dev.to 👋

Wanted to share what my colleagues and I haven been working on for the last 1.5 years and open sourced last week. Would love to get your feedback.

The project is called Constellation and it's a K8s distribution like Rancher or OpenShift. What makes Constellation special is that it's designed to shield entire K8s clusters from the cloud infrastructure (and cloud employees) and keep all data always encrypted - even at runtime.

Foundations

A key ingredient are AMD SEV-based Confidential VMs (CVMs), which are available in Azure and GCP. CVMs protect workloads against a compromised host and malicious admins. For this, CVMs remain encrypted at runtime in memory, are strongly isolated, and have advanced remote attestation capabilities. This is also often referred to as confidential computing.

Intel and Arm have announced similar features for their server CPUs and we plan to support these once they become available.

Goals

We designed Constellation with two goals in mind:

It should provide the best possible security and essentially allow one to safely run a K8s cluster on potentially compromised infrastructure. Even privileged cloud provider employees shouldn't be able to access the data in a cluster.
From a UX-perspective it should work like normal K8s.

Architecture & features

Constellation consists of three main components: a CLI, a custom node image, and a set of containers/services.

CLI

The CLI allows for the easy set up of a Constellation cluster in Azure or GCP (we're working on supporting more clouds). For example:

$ constellation config generate azure
$ constellation create --control-plane-nodes 1 --worker-nodes 2 -y
$ constellation init
$ kubectl ...

The CLI automatically creates CVM-based nodes using Constellation's node image. The node image is based on Fedora CoreOS and is optimized for CVM remote attestation.

Cluster attestation/verification

The first node in a cluster is verified 🔎 by the CLI directly. All others are verified by existing nodes. Thus, transitively, the users knows that only "good" nodes running "good" images are part of the cluster.

How to know what a "good" node image is? We sign all node images (and also the CLI) and publish the corresponding signatures on the Sigstore's public transparency log.

Network & cloud storage encryption

After a node is verified, it receives cryptographic keys to talk securely to other nodes over the network and for encrypting data that is written to cloud storage. These keys are all managed by Constellation. For network encryption, Constellation relies on Cilium.

CVM runtime encryption, network encryption, and storage encryption together ensure that all data is always encrypted.

Check the 📄README for a more comprehensive list of features and benchmark comparisons.

Demo

Constellation Demo GCP - asciinema

asciinema.org

What is this good for?

Fully encrypting and isolating a K8s cluster helps with the following:

⬆️ Increasing data security; preventing data breaches
🚛 Moving sensitive workloads from on-prem to the cloud
⚖️ Meeting regulatory requirements

For example, in our native Germany and elsewhere, confidential computing and runtime encryption are already mandatory for certain eHealth applications.

What do you think?