Painless migration of existing Helm's Tiller setup to Kubernetes Secrets

Kirill Kuznetsov — Wed, 04 Jul 2018 16:49:22 +0000

Meet a command-line tool that solves a security problem with Helm installations by replacing unencrypted ConfigMaps for Tiller with industry-standard Kubernetes Secrets.

Helm and its security options

If you are a Helm user, you perhaps stumbled upon an excellent "Securing Helm installation" write-up in the official repository. If you were not aware of this document, take time to read it, that's a good starting point.

To summarize, there are four main points to consider when securing your installation:

Role-based access control, or RBAC
Tiller's gRPC endpoint and its usage by Helm
Tiller Release Information
Helm charts

Tiller Release Information is something that I want to talk about further.

What is the problem with Tiller releases?

Historically, for each "release," which is basically any configuration or version update of an application, Tiller creates a ConfigMap containing all operational data.

It is mostly OK, except that applications often need sensitive data to operate (passwords, tokens) and ConfigMaps are not the best format to store them: they are persisted in a Kubernetes runtime DB (Etcd) as is, without any encryption. Any person who has access to Etcd or even its dump can easily peek inside. That should not be allowed.

Kubernetes Secrets to the rescue

The option to encrypt stored Kubernetes Secrets was introduced in Kubernetes 1.7. Tiller has support for Secrets as backend storage starting from version 2.7.0.

If you are making a new Helm installation, use helm init --override=... option described here.

E.g.:

helm init --override 'spec.template.spec.containers[0].command'='{/tiller,--storage=secret}'

For the sake of brevity, I am omitting other helm init options that you might need

But what if you don't have the luxury to start from scratch?

Secrets are great and easy to use for new installations, but if you ever wanted to migrate an existing cluster that manages a pack of your production releases from ConfigMaps to Secrets and avoid downtime—you were out of luck. Not anymore.

I wrote a small tool that automates the otherwise daunting manual process. Meet tiller-releases-converter!

Using tiller-releases-converter

After you install the app, upgrading your Helm's Tiller setup to Secrets storage backend (with zero-downtime!) is a matter of executing three shell commands in the following order:

tiller-releases-converter convert # Will create a Secret for every ConfigMap release
tiller-releases-converter secure-tiller # Will update tiller deployment with "--storate=secret"
tiller-releases-converter cleanup # Deletes tiller's ConfigMaps

There is a 4th command called list - it just prints Tiller's current ConfigMaps:

tiller-releases-converter list

Output example:

I've found these Tiller's ConfigMap releases for you:

kube-state-metrics.v1
kube-state-metrics.v2

There are some global command line arguments for your convenience:

Flags:
      --context string      kube config context
  -h, --help                help for tiller-releases-converter
  -c, --kubeconfig string   config file (default is $HOME/.kube/config)
  -n, --namespace string    tiller namespace (default is kube-system)

Spread the word

Thank you for reading! I hope tiller-releases-converter will save you some valuable time and put your mind at ease. If you run into any trouble or have a suggestion on how to improve the tool—feel free to open the issue!

P.S.

Keep in mind that the ConfigMaps to Secrets migration will no longer be an issue once the Helm 3 arrives: next major release of the tool will not include Tiller at all. The release date, however, is not yet announced. Follow official milestones to keep yourself in the loop!

Read more developer articles by Evil Martians on dev.to and in Martian Chronicles