Forem: Julie Hourcade

Level up your secrets management in Kubernetes using AWS Secret Manager and Helm

Julie Hourcade — Mon, 26 Dec 2022 09:04:11 +0000

Engineers' biggest struggle when writing Kubernetes resources is to keep all secrets secure. To be honest, the secrets k8s resource is not secured at all because base64 is not encryption!

Required skills:

Understanding basic concepts about Kubernetes
Cloud providers resources knowledge
Helm v3 installed on a Kubernetes cluster.

Understanding Helm

For this article and for Formance infrastructure we chose to use Helm as a templating system for our k8s resources. This software is commonly known as the package manager for Kubernetes but they also provide a good system for deploying many versions of the same Charts, like for production and staging environments.

Initialize the application

First, we have to create a new application using the Helm CLI. You could find how to install it here: https://helm.sh/docs/intro/quickstart/. Once all is installed correctly, you could run the command:

helm create "<your_application_name>"

The directory Helm creates should look like this ⬇️

In this article, we’re going to stay basic and remove some templates to keep it simple. We’re only keeping:

The deployment template
The service template
The service account template
The helpers, Notes, and Tests

Finally, we will create a file named secret.yaml for the template of our SecretProvider object.

If you’re not familiar with the Helm templating language, I recommend you take a look at the documentation: https://helm.sh/docs/chart_template_guide/

The secret provider object

Concept

What are CSI Drivers?

To perform what we want, we’re going to use a Kubernetes resource named SecretProviderClass from the API secrets-store.csi.x-k8s.io/v1alpha1. The secret store is provided by the Kubernetes Container Storage Interface which here helps us to connect basic resources like volumes or secrets to cloud providers' services.

The main problem of Secrets resources on Kubernetes is the lack of security and if I want to share some private values as database passwords to my application deployment, I need more than just a base64 encryption.

Here at Formance we use Amazon Web Services as cloud provider so I will use the service AWS Secret Manager (docs: https://docs.aws.amazon.com/secretsmanager/?id=docs_gateway). When we have created our Relational Database Service with MariaDB or PostgreSQL or whatever, using InfraAsCode tools Terraform, we have automatically created a Secret with the database login information. The problem is to retrieve those values directly on my application container automatically.

Simplified overview of the workflow

Installing the Secret Store CSI driver

That’s here where Kubernetes CSI enters in action ! It will create an interface between our Secret object on AWS and our application deployment. First of all, we have to install the Secret Store CSI driver using helm.

helm repo add secrets-store-CSI-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --namespace kube-system

You could find an alternative installation here: https://secrets-store-csi-driver.sigs.k8s.io/getting-started/installation.html

It will install things like the operator, and service account with the right RBAC permissions and most important, it will declare the Custom Resources Definition of the SecretProviderClass. Without this, the Kubernetes Scheduler will not understand the API version and the object kind.

Secret store CSI driver also supports GCP, Azure, and Vault secrets providers.

Deep dive into the secret template

The requirement is to make a template where we could inject some specific values like:

The name obviously
The secret provider (note that this example is only tested on AWS)
And the secrets to retrieve using the secret name or in AWS its Amazon Resource Name (ARN)

Keep in mind that this example is a simple template with multiple secrets but you personalize it as you wish. You can find various

Let’s take a look at the template and its values:

Once you create all templates, all you have to do is to apply the command:

helm install <application name> .

And you're all done!

Image credits: Midjourney

Building an Uber Eats Clone

Julie Hourcade — Thu, 02 Jun 2022 11:43:29 +0000

Initialy published on https://32b6.com by Clément Salaün

Templating money flows with Numscript

We explore in this article the money problem behind creating a food delivery platform, focusing on building the core money movements using the Formance ledger and Numscript as the foundation of our real-time accounting system.

The Three-sided Platform

From a money flow perspective, Eats is a classic n-sided marketplace (where N=3). It is a machine that reads your credit card details, materializes a pizza on your porch and a platform, facilitates a transaction between three parties: the customer, the restaurant and the rider.

Some payments services providers give us constructs to route an incoming payment, to a certain extent. Stripe Connect for example has destination charges, which give you the ability to provide the end destination account and your commission amount at payment. This is not super helpful for our food delivery app given that:

We have more than 1 destination (the restaurant and rider)
We don’t know, at payment time, who is going to be the rider

To these issues, Stripe answers with another turnkey solution called separate charges and transfers. It basically solves all our concerns, at one cost: we are now responsible for tracking what is owed to whom. Ledger systems are a good candidate for achieving this, as we’ll attempt to demonstrate in this article.

Templating our flow

Not bothering with unique values (for now), let’s start to encode our money flow in Numscript using hardcoded values. If you want to follow along, Numscript files can be executed against a locally running Formance ledger directly from the cli:

numary exec "eats-clone-demo" example.num

The payment

Using an intermediate payment ledger account helps us divide our flow into multiple sub-problems. Prior to splitting, our first mission is to properly track how much we collected from our customer. Let’s start with a simple case by assuming we managed to collect $59 through Stripe:

We use here the @world account to introduce money into our ledger (as per the spec). If its USD balance was zero before this transaction, it will now be negative 59. We can note as well that we use here USD/2 instead of USD, which is basically a notation for USD scaled to 10^-2 i.e. USD cents.

We conclude the payment by flushing the funds we collected to an order account, which will receive the funds without ever bothering about their origin:

Philosophically, we consider stripe as an external ledger that now owes money to our local ledger. In the real world, money hasn’t moved but we penciled in that this money is now dedicated to this order.

The split

Moving further in the flow, we now task ourselves with splitting the customer payment to the rider, restaurant and our platform. This splitting script is typically where we would encode our platform commission logic. We make the decision here of taking a 15% cut on each amount we pay the restaurant, leaving the delivery fee entirely to the rider.

The payout

We now have riders and and restaurants accounts in our ledger with a positive balance, whose purpose is to eventually be paid out to their owners’ banks.On the rider and restaurant views of our app, we’ll typically display these balances as “available for payout” funds.

Sticking to Stripe, one way to achive the processing of these balances can be to move the balance from a restaurants:001 account to a restaurants:001:payouts:001 account, initiating a transfer and a payout on Stripe of that amount, and eventually moving the balance from restaurants:001:payouts:001 back to world once we successfully processed the payout.

Automating our flow

Using hardcoded values in Numscript will only get us so far. As we start to execute our templates automatically from a backend, we will typically start to use variables, passing them along with the script on the /script endpoint

Using execution variables

Using a commission from metadata

As we iterate with our platform economics, we’ll likely end up with different commission rates per restaurant. We can inject this commission as a variable as well, fetching it the example below from the restaurant account metadata:

More complexity

Coupons

Inundating the buy side with generous coupons is a common strategy to break the chicken and egg problem platforms face. If you’re really going for an Eats clone, you may subsidize the buy side so much that you’ll end up spending more for a purchase in discount than the commission you’ll get out of it:

In the grand scheme of the above example, as a platform, we are $8 short overall. If this was the only payment flow to ever happen on our Stripe account, we wouldn’t be able to cover payouts to the rider and restaurant.

As we rinse and repeat this process for thousands of sales, it becomes critical to properly track these marketing losses to avoid putting ourselves at risk of ever defaulting payouts.

Numscript comes with multi-sources transactions which we can use to model our marketing expense. Let’s first credit $15k to a coupon account for a specific campaign:

Then in script executing at payment time, we’ll use this coupons as a source of marketing funds, from which we’ll draw up to $19.

Refunds at loss

So, the food arrived colder than it left the freezer at the restaurant. Customer files a ticket and you can’t know for sure wether the weather, rider, or restaurant is at fault here. As a platform you may decide to eat the loss, issuing a partial refund to your customer.

Properly registering this loss as a money movement in our ledger will, in addition to keeping the books in check, once again help us prevent defaulting risk as we’ll be able to create routine processes of topping-up your Stripe platform account to cover for losses.

Reconciliation

The internal flow we defined in blue sits between input and output. It tracks what is owed to whom, and only makes sense if it eventually supports actual money movements in the real world, which we achieve through payments processors.

Recon is a complex payments 101 problem. We want to make sure that assumptions we make in our system, e.g. that we received $59 from our customer, hold true in the external systems we use over time. As an example, your customer may charge back their payment after a few weeks, breaking your reconciliation. Ideally, every payment on your provider should be linked to an internal ledger transaction. In a further article, we’ll cover techniques and scripts to continuously reconcile our flows.

Notes on legal design and compliance

We mentioned defaulting risk a few times in this piece. I’ve heard objections to this concern, usually quoting perpetual growth combined with positive cash float as a mechanism to make up for platform losses and marketing expenses. This can only be true if properly measured with a ledgering strategy and to the surprise of no one, auditors are not particularily fond of this scheme, no matter how exciting the idea of a fractional-reserve marketplace sounds on paper.

In any case, your local lawyers are the best advisors on what you can and cannot do with your payments flows.

Conclusion

As a platform, having a holistic view of your money flows and being able to know at any point in time what is owed to whom is critical. Using a ledger-oriented programmation model will help us achieve both reliability and visibility, as immutable transactions become the fundamental construct of state in our system.

Docker everywhere is not a good thing.

Julie Hourcade — Fri, 10 May 2019 10:20:29 +0000

Docker is a trend and everyone want to be part of this but not really in a good way. The bad side of this trend is that everything goes on container, like web app or database even GitLab to the detriment of data persistence, security and maybe performance in in certain case.

One of a solution is, IMHO, to use a orchestrator like Swarm or Kubernetes with a good study of which kind of volumes all your containers need. You can also make security better with a good reverse proxy and using RBAC for Kubernetes.

What is your methods to keep your data safe, to make container secure and to have good performances on your conteneurized website ?

Docker Best Practices, back to basics.

Julie Hourcade — Tue, 30 Apr 2019 10:16:14 +0000

You're new to Docker ? I was too. Everyone wants to build awesome projects with Docker, build some images and start containers. That's cool ! Docker is cool. But you missed a point, there are some rules to use it.

Rule number one ! Keep it simple and small.

Docker images are meant to be SMALL. As small as possible.
Docker is a conteneurisation system not a virtual machine. Don't reproduce all your GNU/Linux distribution on it.

To have a lightweight image, you can first take a look to Alpine versions. Alpine is a Linux based on MUSL and BusyBox instead of basical GNU. If you want to use a Debian base image anyway, don't forget to clean apt after installing packets.

Choose a Official Docker image, you must.

There is an ocean of Docker images available on Docker Hub but they are not all secured. You must choose a official image and add to it what you need. For example, don't use a PHP image made by a unknown person found on the Hub but prefer to use the php official image with the adapted tag.

If you really need to use a unofficial image, take a look at the Dockerfile. Most of images provides the Dockerfile associated to the image. If the base image is based itself on an other unofficial image, search it ! Go the deeper possible. Finally, avoid closed images with no Dockerfile provided, it could be insecure.

You're saying image tag ? Don't use the latest.

When you're building a Dockerfile, everyone has already added a FROM statement with the latest tag on the image. Everyone, don't lie.
So don't do that ! For the simple reason that the latest tag is the latest not the more stable. If the latest tag change to an upper version it could broke all your system. You should consider to choose a version no matter the base image you use.

Dev environements are not the same as Prod environements.

For a use in production, there are more rules.
First of all, don't use docker-compose on production. Prefer using a orchestrator like Swarm or Kubernetes. If you have to use docker-compose on a project, you could switch to an orchestrator. It's not too much.

Secondly, containers should be stateless. For example, don't conteneurize databases.

Finally, One container, One task. A container should do only one thing like running a webserver, execute a script, run a ElasticSearch or a Redis or a RabbitMQ even a GitLab but not all together.

TL;DR

To conclude :

Make small and simple images
Use official Docker Hub images as possible
Never use the latest tag
Be careful on what you're doing on production

Useful links

https://docs.docker.com/develop/dev-best-practices/
https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
and all the Docker documentation

Feel free to add your best practices in comments !

Explain container orchestration like i'm five

Julie Hourcade — Wed, 18 Oct 2017 15:22:02 +0000

Hi everyone,

After a little more than 3 months internship, i have to do my report. I know what is container and orchestration but i'm really bad at explain it. So explain me container orchestration like i'm five !