Forem: Daniel Albuschat

Cryptography: What is the difference between Hashing, Signing and MAC?

Daniel Albuschat — Sun, 30 Jan 2022 14:04:29 +0000

I set out to learn everything about cryptography in 2022 and share what I have learned along the way. I started the Cryptography Primer this year, where you can find more information, and which will grow each week. Following is an excerpt from this site explaining everything you need to know to get started with hashes, MACs and digital signatures. Have fun!_

-- Daniel

Hashes, MACs and digital signatures are primitives of cryptography, where hashes are also used outside of cryptography - e.g. to validate that a message has not been corrupted during transport.

Hashes, MACs and digital signatures have a few things in common:

They can be used to validate the "integrity" of a message - this means that you can be sure that the message was not corrupted if it matches the hash, signature or MAC that you compare it with.
The original message can not be extracted from them.
Hence, they don't encrypt messages and are not encryption algorithms.

Here is a table showing the differences of the possibilities for each primitive:

Feature	Hash	Message Authentication Code (MAC)	Digital Signature
Validate that data has not been tampered with or has been corrupted ("Integrity")	✅	✅	✅
Validate the sender of a message by using the Private Key ("Authentication")	❌	✅	✅
Validate the sender of a message by using the Public Key ("Authentication")	❌	❌	✅
Prove that the sender has written and published a message ("Non-Repudiation")	❌	❌	✅

What are use-cases for hashes?

A hash basically "reduces" an arbitrary large message into a fixed size digest in a non-reversible way. In particular, a hash function aims to do this in a way that possible collisions are as unlikely as possible. Nowadays, when you say "hash function", you usually mean cryptographic hash functions. There are non-cryptographic hash functions¹, too (but some even refuse to call those hash functions): Most notably CRC² (cyclic redundancy check), which is often used to verify that data has not been (unintentionally) corrupted during transport.

But even cryptographic hash functions can be used for non-cryptographic as well as cryptographic use cases:

Non-Cryptographic use-cases for hash functions

Here are some examples how hash functions are used in non-cryptographic context:

Validate that a message has not been corrupted (or modified) during transport. For example, you can often find hashes next to a download link that can be used to validate that the file has exactly the same content as it supposed to have after you have downloaded it.
"Shrink" information to a unique identifier that can be used for lookups. For example, you can look up a whole sentence or even a whole paragraph of text in a database by using it's hash, instead of comparing all characters of the paragraph in the database.

Cryptographic use-cases for hash functions

Here are some examples how hash functions ar used in cryptograhpic context:

Usually digital signatures are not applied to the whole message or data block, but on a hash digest of that message. In this scenario, the collision-resistance of the hash function is of utter importance³⁴.
Store passwords⁵.
Some MAC algorithms are based on hash functions - these are called "HMAC" (hash-based message authentication code) and basically build a hash on a mixup of the Private Key and the message.

Comparison of hashing functions

Name	Digest Sizes	Description
SHA2	224, 256, 384, 512	Recommended SHA2-family hashing functions are state-of-the-art and considered very secure. SHA2 is not less secure than SHA3. Also known as SHA-224, SHA-256, SHA-384, SHA-512, SHA-512/224 and SHA-512/256.
SHA3	224, 256, 384, 512	Recommended SHA3-family hashing functions are state-of-the-art and considered as secure as SHA2. SHA3 was inventend as an alternative to SHA2 in case that SHA2 would be broken, but is computationally more expensive than SHA2. Also known as SHA3/224, SHA3/256, SHA3/384 and SHA3/512.
SHA1	160 Bit	Not recommended There are practical known collission attacks for SHA1.
MD5	128 Bit	Not recommended There are practical known collission attacks for MD5.

What are use-cases for digital signatures?

Digital signatures also provide the integrity validation function of hashes. But additionally, digital signatures let you verify that the sender of the message is authentic, e.g. the message originates from the source that you expected.

Because digital signatures are using "asymmetric cryptography", you can use the Public Key to validate the integrity and authenticity of the message. This has the advantage that you do not need to share a common Private Key between the sender and the recipient.

Use-cases for digital signatures:

Publish a message and "sign" it so that everyone can verify that it has been written and published by you.
For example, TLS (and therefore HTTPS, which builds on TLS) uses digital signatures to authenticate the server behind the domain that you have requested data from.
The underlying building block for this are x.509 certificates that are also widely used in other systems where it is important to let anybody know that a "certificate", that provides arbitrary permissions or identifications, can be trusted.
Mobile platforms such as Apple's iOS and Google's Android use digital signatures to sign apps in they App Store/Play Store so that the system is able to trust these apps (and in turn is able to block running untrusted apps).

Comparison of digital signature algorithms

Name	Description
EdDSA	Recommended This algorithm is a variant of DSA, but uses "twisted Edwards curves", which have a few advantages⁶: High performance on a wide range of systems More resilient to side-channel attacks Does not require a unique random number for each messsage
ECDSA	Recommended This algorithm is a variant of DSA, but uses elliptic curves instead of modular arithmetic. This decreases the required key size for the same safety level drastically. Additionally, the same Public/Private Key pair could be used for encryption using ECIES, which could be an advantage.
RSA	Recommended RSA can be used for digital signatures and asymmetric encryption using the same Public/Private key pair. Compared to ECDSA and EdDSA it has much larger key sizes and is computationally more expensive. However, if your system can profit from using the same Private/Public key pair for signing and encrypting, and the somewhat rarely used ECIES is not a feasible option for you, RSA can be a good fit.
DSA	Use EdDSA, ECDSA or RSA instead (preference in this order) DSA was the first standardised digital signature algorithm and is still considered secure. However, the large key size and expensive computations makes it less pratical than it's modern successors such as ECDSA and EdDSA.

What are MACs used for?

You should usually not require to use MACs yourself, because these are often part of an "authenticated encryption" cipher such as AES-GCM or ChaCha20-Poly1305.

MACs are similar to digital signatures, but they do not have the advantage of asymmetric cryptography, because they require the same Private Key for "signing" a message and authenticating the message.

MACs are of utter importance to prevent CCA on ciphers⁷ - every cipher should include message authentication, which is usually accomplished by using a MAC.

List of hash functions on Wikipedia ↩
CRC on Wikipedia ↩
MD5 considered harmful today - Creating a rogue CA certificate from Eindhoven University of Technology ↩
Announcing the first SHA1 collision on Google Security Blog ↩
How to Securely Store Passwords in Database? ↩
What is the difference between ECDSA and EdDSA? on crypto.stackexchange.com ↩
All the crypto code you’ve ever written is probably broken as blogged by Tony Arcieri. ↩

Request for Comments: Slurping Kubernetes logs

Daniel Albuschat — Sat, 31 Aug 2019 08:33:41 +0000

I've investigated many tools and stacks to store and search logs of containers, especially from Kubernetes clusters. In my environment, I have a hefty constraint that I don't want the logs, which main contain personal information, e.g. usernames in URIs, to be stored outside the EU. GDPR and all. That excludes the ready-to-use services like datadog, dynatrace, newrelic, etc.

It seems the ELK stack consisting of ElasticSearch, Logstash and Kibana seems pretty popular and can be used on-premise. However, investigating this and other solutions I found them all to be very complex, and read comments about them being hard to set up and maintain.

Thinking about it, I wonder whether there is not a very simple approach that might yield good results. My idea is simple: Just write a small script that fetches logs from containers via kubectl logs --timestamps=true once a minute or so. Consecutive fetches use --since-time of the last received timestamp. The script pushes the results into some database to archive them and make them searchable.

I know, this is a "roll-your-own..." approach, but it seems pretty simple, but also effective. I'd like to ask the dev community for feedback on this idea, and pointers to alternatives that are not complex and can be used on-premise.

So, what are your thoughts?

Nginx: Everything about proxy_pass

Daniel Albuschat — Tue, 20 Aug 2019 18:45:20 +0000

With the advent of Microservices™, ingress routing and routing between services has been an every-increasing demand. I currently default to nginx for this - with no plausible reason or experience to back this decision, just because it seems to be the most used tool currently.

However, the often needed proxy_pass directive has driven me crazy because of it's - to me unintuitive - behavior. So I decided to take notes on how it works and what is possible with it, and how to circumvent some of it's quirks.

First, a note on https

By default proxy_pass does not verify the certificate of the endpoint if it is https (how can this be the default behavior, really?!). This can be useful internally, but usually you want to do this very explicitly. And in case that you use publicly routed endpoints, which I have done in the past, make sure to set proxy_ssl_verify to on. You can also authenticate against the upstream server that you proxy_pass to using client certificates and more, make sure to have a look at the available options at https://docs.nginx.com/nginx/admin-guide/security-controls/securing-http-traffic-upstream/.

A simple example

A proxy_pass is usually used when there is an nginx instance that handles many things, and delegates some of those requests to other servers. Some examples are ingress in a Kubernetes cluster that spreads requests among the different microservices that are responsible for the specific locations. Or you can use nginx to directly deliver static files for a frontend, while some server-side rendered content or API is delivered by a WebApp such as ASP.NET Core or flask.

Let's imagine we have a WebApp running on http://localhost:5000 and want it to be available on http://localhost:8080/webapp/, here's how we would do it in a minimal nginx.conf:

daemon off;
events {
}
http {
    server {
        listen 8080;
        location /webapp/ {
            proxy_pass http://127.0.0.1:5000/api/;
        }
    }
}

You can save this to a file, e.g. nginx.conf, and run it with

nginx -c $(pwd)/nginx.conf.

Now, you can access http://localhost:8080/webapp/ and all requests will be forwarded to http://localhost:5000/api/.
Note how the /webapp/ prefix is "cut away" by nginx. That's how locations work: They cut off the part specified in the location specification, and pass the rest on to the "upstream". "upstream" is called whatever is behind the nginx.

To slash or not to slash

Except for when you use variables in the proxy_pass upstream definition, as we will learn below, the location and upstream definition are very simply tied together. That's why you need to be aware of the slashes, because some strange things can happen when you don't get it right.

Here is a handy table that shows you how the request will be received by your WebApp, depending on how you write the location and proxy_pass declarations. Assume all requests go to http://localhost:8080:

location	proxy_pass	Request	Received by upstream
/webapp/	http://localhost:5000/api/	/webapp/foo?bar=baz	/api/foo?bar=baz
/webapp/	http://localhost:5000/api	/webapp/foo?bar=baz	/apifoo?bar=baz
/webapp	http://localhost:5000/api/	/webapp/foo?bar=baz	/api//foo?bar=baz
/webapp	http://localhost:5000/api	/webapp/foo?bar=baz	/api/foo?bar=baz
/webapp	http://localhost:5000/api	/webappfoo?bar=baz	/apifoo?bar=baz

In other words: You usually always want a trailing slash, never want to mix with and without trailing slash, and only want without trailing slash when you want to concatenate a certain path component together (which I guess is quite rarely the case). Note how query parameters are preserved!

$uri and $request_uri

You have to ways to circumvent that the location is cut off: First, you can simply repeat the location in the proxy_pass definition, which is quite easy:

location /webapp/ {
    proxy_pass http://127.0.0.1:5000/api/webapp/;
}

That way, your upstream WebApp will receive /api/webapp/foo?bar=baz in the above examples.

Another way to repeat the location is to use $uri or $request_uri. The difference is that $request_uri preserves the query parameters, while $uri discards them:

location	proxy_pass	request	received by upstream
/webapp/	http://localhost:5000/api$request_uri	/webapp/foo?bar=baz	/api/webapp/foo?bar=baz
/webapp/	http://localhost:5000/api$uri	/webapp/foo?bar=baz	/api/webapp/foo

Note how in the proxy_pass definition, there is no slash between "api" and $request_uri or $uri. This is because a full URI will always include a leading slash, which would lead to a double-slash if you wrote "api/$uri".

Capture regexes

While this is not exclusive to proxy_pass, I find it generally handy to be able to use regexes to forward parts of a request to an upstream WebApp, or to reformat it. Example: Your public URI should be http://localhost:8080/api/cart/items/123, and your upstream API handles it in the form of http://localhost:5000/cart_api?items=123. In this case, or more complicated ones, you can use regex to capture parts of the request uri and transform it in the desired format.

location ~ ^/api/cart/([a-z]*)/(.*)$ {
   proxy_pass http://127.0.0.1:5000/cart_api?$1=$2;
}

Use try_files with a WebApp as fallback

A use-case I came across was that I wanted nginx to handle all static files in a folder, and if the file is not available, forward the request to a backend. For example, this was the case for a Vue single-page-application (SPA) that is delivered through flask - because the master HTML needs some server-side tuning - and I wanted to handle nginx the static files instead of flask. (This is recommended by the official gunicorn docs.)

You might have everything for your SPA except for your index.html available at /app/wwwroot/, and http://localhost:5000/ will deliver your server-tuned index.html.

Here's how you can do this:

location /spa/ {
   root /app/wwwroot/;
   try_files $uri @backend;
}
location @backend {
   proxy_pass http://127.0.0.1:5000;
}

Note that you can not specify any paths in the proxy_pass directive in the @backend for some reason. Nginx will tell you:

nginx: [emerg] "proxy_pass" cannot have URI part in location given by regular expression, or inside named location, or inside "if" statement, or inside "limit_except" block in /home/daniel/projects/nginx_blog/nginx.conf:28

That's why your backend should receive any request and return the index.html for it, or at least for the routes that are handled by the frontend's router.

Let nginx start even when not all upstream hosts are available

One reason that I used 127.0.0.1 instead of localhost so far, is that nginx is very picky about hostname resolution. For some unexplainable reason, nginx will try to resolve all hosts defined in proxy_pass directives on startup, and fail to start when they are not reachable. However, especially in microservice environments, it is very fragile to require all upstream services to be available at the time the ingress, load balancer or some intermediate router starts.

You can circumvent nginx's requirement for all hosts to be available at startup by using variables inside the proxy_pass directives. HOWEVER, for some unfathomable reason, if you do so, you require a dedicated resolver directive to resolve these paths. For Kubernetes, you can use kube-dns.kube-system here. For other environments, you can use your internal DNS or for publicly routed upstream services you can even use a public DNS such as 1.1.1.1 or 8.8.8.8.

Additionally, using variables in proxy_pass changes completely how URIs are passed on to the upstream. When just changing

proxy_pass https://localhost:5000/api/;

set $upstream https://localhost:5000;
proxy_pass $upstream/api/;

... which you might think should result in exactly the same, you might be surprised. The former will hit your upstream server with /api/foo?bar=baz with our example request to /webapp/foo?bar=baz. The latter, however, will hit your upstream server with /api/. No foo. No bar. And no baz. :-(

We need to fix this by putting the request together from two parts: First, the path after the location prefix, and second the query parameters. The first part can be captured using the regex we learned above, and the second (query parameters) can be forwarded using the built-in variables $is_args and $args. If we put it all together, we will end up with a config like this:

daemon off;
events {
}
http {
    server {
        access_log /dev/stdout;
        error_log /dev/stdout;
        listen 8080;
        # My home router in this case:
        resolver 192.168.178.1;
        location ~ ^/webapp/(.*)$ {
            # Use a variable so that localhost:5000 might be down while nginx starts:
            set $upstream http://localhost:5000;
            # Put together the upstream request path using the captured component after the location path, and the query parameters:
            proxy_pass $upstream/api/$1$is_args$args;
        }
    }
}

While localhost is not a great example here, it works with your service's arbitrary DNS names, too. I find this very valuable in production, because having an nginx refuse to start because of a probably very unimportant service can be quite a hassle while wrangling a production issue. However, it makes the location directive much more complex. From a simple location /webapp/ with a proxy_pass http://localhost/api/ it has become this behemoth. I think it's worth it, though.

Better logging format for proxy_pass

To debug issues, or simply to have enough information at hand when investigating issues in the future, you can maximize the information about what is going on in your location that uses proxy_pass.

I found this handy log_format, which I enhanced with a custom variable $upstream, as we have defined above. If you always call your variables $upstream in all your locations that use proxy_pass, you can use this log_format and have often much needed information in your log:

log_format upstream_logging '[$time_local] $remote_addr - $remote_user - $server_name to: $upstream: $request upstream_response_time $upstream_response_time msec $msec request_time $request_time';

Here is a full example:

daemon off;
events {
}
http {
    log_format upstream_logging '[$time_local] $remote_addr - $remote_user - $server_name to: "$upstream": "$request" upstream_response_time $upstream_response_time msec $msec request_time $request_time';
    server {
        listen 8080;

        location /webapp/ {
            access_log /dev/stdout upstream_logging;
            set $upstream http://127.0.0.1:5000/api/;
            proxy_pass $upstream;
        }
    }
}

However, I have not found a way to log the actual URI that is forwarded to $upstream, which would be one of the most important things to know when debugging proxy_pass issues.

Conclusion

I hope that you have found helpful information in this article that you can put to good use in your development and production nginx configurations.

Kubernetes: Certificates, Tokens, Authentication and Service Accounts

Daniel Albuschat — Sun, 19 May 2019 19:22:53 +0000

Mostly for personal/learning experiences, I have created quite a few Kubernetes clusters, such as the one on my Raspberry Pi rack. I also created two clusters for a production and a staging environment on ultra-cheap cloud servers from Hetzner Cloud. Luckily, none of those environments where serious business.

Disclaimer: I'm not a Kubernetes expert, nor am I a security expert, so make sure that you second-source the information you find on this post before you rely on them. I just wanted to publish the experience and insights that I made during this trip - thanks!

Why was that lucky?

Because I accidentally leaked the certificates for my admin access to the staging cluster. I was trying to set up a CI/CD pipeline for an open source project using CircleCI. While I was testing out the steps one by one, I dumped the content of ${HOME}/.kube/config that has been created from a BASE64-encoded environment variable, like described on this blog post. That was fatal, though, since a) the job logs of open source projects are publicly visible and b) jobs and their logs can not be deleted manually, I had to reach out to the support for this. Ouch!

So let's dig into what happened here.

Creating a cluster

First of all, I created the cluster manually using kubeadm, following the official docs. Doing so, I created a cluster with RBAC enabled and a kube-config has been created for me that includes a user that is identified by a certificate.

Accessing the cluster

After kubeadm created the cluster successfully, it instructs you what to do to access your cluster:

"Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:"

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

The admin.conf that kubeadm creates includes a user identified by a certificate:

- name: kubernetes-admin
  user:
    client-certificate-data: <BASE64 ENCODED X509 CERTIFICATE>
    client-key-data:  <BASE64 ENCODED PRIVATE KEY FOR THE CERTIFICATE>

Following these instructions is your only way to access this cluster using kubectl as of now, so you should go ahead and do this now. After you copied the admin.conf, you have cluster-admin access. You are root, so to say.

How is that?

What kubeadm did is that it created a new CA (Certificate Authority) root certificate that is the master certificate for your cluster. It looks something like this:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 0 (0x0)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = kubernetes
        Validity
            Not Before: May 19 11:11:04 2019 GMT
            Not After : May 16 11:11:04 2029 GMT
        Subject: CN = kubernetes
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    <REDACTED>
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Certificate Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
         <REDACTED>

So... it doesn't really contain much besides the info that it is a CA and it's CN (= Common Name) is Kubernetes. That's because this cert only acts as a root for other certs that are used for different purposes on the cluster. You can have a look at /etc/kubernetes/pki to take a peek at some of the certs that are used in your cluster and have been signed by the CA:

daniel@kube-box:~# ls /etc/kubernetes/pki/ -1
apiserver.crt
apiserver-etcd-client.crt
apiserver-etcd-client.key
apiserver.key
apiserver-kubelet-client.crt
apiserver-kubelet-client.key
ca.crt
ca.key
etcd
front-proxy-ca.crt
front-proxy-ca.key
front-proxy-client.crt
front-proxy-client.key
sa.key
sa.pub

It is possible to allow access to clients that authenticate themselves using certificates that are trusted by the CA. This is enabled by passing this ca.crt to kube-controller-manager in the --client-ca-file parameter. This is what the docs have to say about it:

--client-ca-file string
If set, any request presenting a client certificate signed by one of the
authorities in the client-ca-file is authenticated with an identity 
corresponding to the CommonName of the client certificate.

Back to your kube-config: The certificate that is included in BASE64 in your admin.conf is signed by that exact CA. This is why it is trusted by the cluster. Let's have a look at the certificate:

grep 'client-certificate-data: ' ${HOME}/.kube/config | \
   sed 's/.*client-certificate-data: //' | \
   base64 -d | \
   openssl x509 --in - --text

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 3459994011761527671 (0x30045e38cc064b77)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = kubernetes
        Validity
            Not Before: May 12 10:54:39 2019 GMT
            Not After : May 11 10:54:42 2020 GMT
        Subject: O = system:masters, CN = kubernetes-admin
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    <REDACTED>
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Client Authentication
    Signature Algorithm: sha256WithRSAEncryption
         <REDACTED>

What does this cert tell us (and the cluster)?

a) It is issued and trusted by our kubernetes cluster
b) It identifies the Organisation (O) system:masters, which is interpreted as a group by kubernetes
c) It identifies the Common Name (CN) kubernetes-admin, which is interpreted as a user by kubernetes

In other words: This certificate logs in as the user kubernetes-admin with the group system:masters. This is the reason why you don't need to provide the group name in the kube-config, and why you can change the user's name at will in the kube-config, without this changing the actual user that is being logged in.

Where are the permissions defined?

In RBAC-enabled clusters, permissions are defined in Roles (per namespace) or ClusterRoles (for all namespaces). These permissions are then granted to objects using RoleBindings and ClusterRoleBindings. So what you have to look for are RoleBindings and ClusterRoleBindings that grant permissions to the group system:masters or the user kubernetes-admin. You can do this by having a look at the output of

kubectl -A=true get rolebindings && kubectl -A=true get clusterrolebindings

The default setup that kubeadm created for me yielded one hit for that search, the ClusterRoleBinding named cluster-admin, which grants permissions to a ClusterRole with the same name. Here's the definition:

kind: ClusterRole
metadata:
  name: cluster-admin
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'
- nonResourceURLs:
  - '*'
  verbs:
  - '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cluster-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:masters

So there we have it! The group system:masters, which the certificate authorizes as, grants '*' permissions to all resources using all verbs, hence full access.

Unvalidated assumption: I think that users and groups in this case are only defined in the certificates, and new users can be created by issuing a new certificate with the CommonName set to the desired username and the Organisation set to the desired group. This username and group can then, without further ado, be used in ClusterRoleBindings and RoleBindings. I did not take the time to validate this, though, which would be possible by issuing a new certificate using openssl, signed with the cluster's CA. It'd be great if someone could confirm or debunk this assumption in a comment!

The mystery

So I got this far and found out how the user and group are identified and how permissions are granted to this user and group. My guess then was that when I delete the ClusterRoleBinding, or rather remove the group system:masters from it, that the certificate should not have access to the cluster anymore. If I did that, and it had the expected result, I would lose all access to the cluster and would have successfully logged me out for good. So I first added a serviceaccount and created a kube-config that logged in using a token for that serviceaccount and verified that the access worked. We will see later how to do this. Then, after setting the safety net in place, I removed the system:masters subject from the ClusterRoleBinding. To my surprise, this did not lock the user out. I could still fully access the cluster using the old kube-config… maybe someone can explain this behaviour in a comment?

Alternative 1: Replace the CA

One sure-as-hell way to make the leaked certificate useless is to replace the CA in the cluster. This would require a restart of the cluster, though. And it would require to re-issue all the certificates that we have seen above, and maybe some more. I rated the possibility to totally fuck everything up and waste multiple hours on the trip at about 99%, so I abandoned the plan. :-)

Alternative 2: Rebuild the whole cluster

Luckily, it was a staging cluster, so I had plenty of freedom. Before starting my investigations, I powered off all nodes. Then, after not finding a proper solution to only make the leaked certificate useless, I killed the whole cluster using

kubeadm reset
rm -rf /etc/kubernetes
rm -rf /var/lib/kubelet

And recreated from scratch with kubeadm. (Which is so great, by the way!!)

Then I went ahead and made a few dozen more commits reading 'NOT printing the content of the kube-config anymore', 'Getting CI/CD to work', 'Maybe now it works', 'Uhm what?', 'That gotta work', 'fuck CI/CD', … :-)

Lessons learned: Use service-accounts with tokens

(Or other authentication methods like OpenID, as recommended in this awesome post.)

So my lesson learned is to do what I've seen at the big managed kubernetes providers: Use a service-account and it's access token for authorization. Here I'll show how to set up a super-user that uses a token instead of a cert:

kubectl -n kube-system create serviceaccount admin

To grant super-user permissions, the easiest way is to create a new ClusterRoleBinding to bind this service-account to the cluster-admin ClusterRole:

kubectl create clusterrolebinding add-on-cluster-admin \
  --clusterrole=cluster-admin \
  --serviceaccount=kube-system:admin

Use your new service-account

Your admin user is now ready and armed. Now we need to log in with this user. I assume that you have the admin.conf in ${HOME}/.kube/conf. We now want to add the new user, identified by it's token, and add a new context that uses this user:

TOKENNAME=`kubectl -n kube-system get serviceaccount/admin -o jsonpath='{.secrets[0].name}'`
TOKEN=`kubectl -n kube-system get secret $TOKENNAME -o jsonpath='{.data.token}' | base64 -d`
kubectl config set-credentials admin --token=$TOKEN
kubectl config set-context admin@kubernetes --cluster kubernetes --user admin

Now go ahead and try your new, shiny service-account:

kubectl config use-context admin@kubernetes
kubectl -n kube-system get all

If this went well, you should go ahead and delete the certificate-based user and the corresponding context:

kubectl config unset users.kubernetes-admin
kubectl config delete-context kubernetes-admin@kubernetes

Yay! Now we have a kube-conf that only includes token-based access. This is great, because it is very easy to revoke that token if this config might be leaked or published.

How to invalidate a leaked token

This is easy! Just delete the secret that corresponds to the user's token. We already saw how to find out which is the correct secret:

kubectl -n kube-system get serviceaccount/admin -o yaml

You will see a field "name" in the "secrets" array. This is a name of a secret that holds this service-account's token. Now go ahead and simply delete it:

kubectl -n kube-system delete secrets/token-admin-xyz123

Then wait a few seconds, and try to access your cluster:

dainel@kube-box:~# kubectl -n kube-system get all
error: You must be logged in to the server (Unauthorized)

Wohoo!

But how do you regain access? Well, if you're on your master node, simply copy the admin.conf back to your ${HOME}/.kube/conf and repeat the steps from "Use your new service-account". Kubernetes will have created and assigned a new token by now.

I hope that this helped, and I'd love to hear feedback, errata, etc. in the comments!

Also make sure to read the VERY comprehensible and awesome post "Kubernetes Security Best-Practices" by Peter Benjamin.

And a big thank you to Andreas Antonsson, vaizki and Alan J Castonguay, who have helped me on the official Kubernetes Slack channel to get a better understanding of what is going on.

Where is HTTPS for IoT? (Update)

Daniel Albuschat — Mon, 10 Sep 2018 21:47:44 +0000

Updated, 10/22/2018: Added proposal 1.a, with added HTTP Public Key Pinning

By now everyone should know that HTTPS is secure, and that HTTPS is important. Browsers even declare HTTP websites as "insecure" in this day and age. And yet, when you look at IoT devices, such as your Smart Home gadgets, they are commonly using HTTP. This means that they communicate without encryption.

Disclaimer: I'm not a security expert, so please bear with me if you find information that may appear to be not totally correct or incomplete. I'll be happy to receive your feedback in a comment and update the article!

The problem with IoT (transport-layer) security

IoT devices are often cheap and hence the engineering efforts need to be cheap and fast, too. Often there is just no budget or time to take security into consideration. So often even absolute security basics such as authentication are often not in place. There are more than enough gadgets out there that, when you have access to someone else's LAN, you can take complete control of.

This is not only a problem of no-name products from Asia, but even of devices from high-profile companies, such as the Google Home. There is inofficial documentation of the local Google Home API that the Google Home App uses. According to this documentation - and I hope that by now the API has been revoked and redesigned - at least at some point in time the communication was unencrypted (HTTP) and unauthenticated. Wow.

While not using authentication is something that you can - and must - blame any engineer for, you can not blame anyone for not using HTTPS on their IoT device. Why? Because it is plain impossible*. At least not when seeing HTTPS like it is designed to work. We'll see why this is the case later, but fact is that the mechanisms of secure communication via HTTPS are designed in a way that you can not easily use it for local communication. Ouch.
*) Yes I know, this is an exaggeration, but how else should I have got your attention? :-)

Mahmoud Al-Qudsi has written a nice blog-post on this.

Here's a shocking inline picture that should increase your blood pressure instantly:

Immediate security measures for all IoT developers

Before we deep-dive into HTTPS, there are a few very important security measures that every developer of IoT devices absolutely must follow:

Use authentication for all your API endpoints.
Use unique, strong passwords for each device, generated during the manufacturing process.
Close all ports that are not used by your public IP using a firewall/iptables/netfilter.
Don't expose your services that are written in whatever language with whatever HTTP server implementation directly - instead use a well-maintained, well-tested HTTP server as your ingress, such as nginx, which is available for ARM, too.
Allow firmware updates by your users, or even automatically.

Benefits of using HTTPS in IoT

But why should you even consider using HTTPS? Well, first of all, if you suppose that your home network is a sacred place that only you have access to, you can feel safe with HTTP. But if you accept the fact that it will always be possible to gain access to more or less all of your LAN, you will feel safer when you know that even with that intrusion possible, without knowing the proper credentials or authorization your IoT devices are still safe.

One way that someone can have access to your LAN is a very old, very easy attack called "DNS rebinding", which we will talk about later for another reason. DNS rebinding allows an attacker to access vital components such as your router - and routers often have a lot of security flaws. Are you convinced, yet? So how can HTTPS help?

HTTPS prevents communication sniffing, data manipulation and offers verification of your peer. This should make you feel safer, since even when your router is compromised, your bank account login is still safe. Also, HTTPS verifies the identity of the server, so that you can be sure who you are communicating with, and in case of a crime happening, you know who to sue. Yay!

Let's remember these three important promises that HTTPS guarantees, since we will come back to them later:

1) Privacy: No communication sniffing
2) Integrity: No data manipulation
3) Identification: You know who you are talking to (or: whose software you are talking to)

The core problem

The core problem with HTTPS is that a fundamental component of it is the host name. Browsers compare the host name that you entered into the address bar with the host name that the server's certificate has been issued to. If these don't match, the browser will not connect to the server, except when your user wades through hardly visible links to finally add an exception for your device. And it will tell your users that it is highly insecure what he is currently doing along the way. Frightening!

And that's not all: Since November 1st 2015, this domain must not be an IPv4 or IPv6 address and must be a FQDN with a public top level domain. No certificates for 192.168.1.20 or raspberry.local!

In other scenarios, where no such control is available, such as built-in clients in other IoT gadgets like the Homee that can trigger arbitrary https endpoints (often used for IFTTT, but can point to your local device, too), communication will be plain impossible.

So, this bites you, when:

You want to provide a secure web frontend hosted on your IoT device
You want to provide a secure, local API on your IoT device

No solution in sight?

So we seem to be shit out of luck. A versatile solution for Transport Layer Security for local HTTPS connections seems to be impossible as of now. HTTPS has been designed with the public internet in mind, and does not contain mechanisms that handle local usage properly. But what options do we have to still secure our IoT communication? Spoiler: None of them work out of the box in all environments.

Solution #1: Ignore invalid hosts in SSL certificates

If you are lucky enough to build all clients yourself - i.e. when you build a mobile or desktop app to connect to your device or your devices connect to each other - you can still establish an HTTPS connection. Nearly all HTTPS client allow you to handle certificate errors in your code. Here's an example for Java. Just please make sure that you only ignore errors regarding the host name, and still let the connection fail when the certificate has other errors, such as when it has expired.

Pro: Works in offline scenarios and not a single byte leaves your local network.
Con: Only works when you control all clients. :-(

Solution #2: Use public domain names for private IPs

This is how Plex is doing it. You create dynamic domain names for all your devices, e.g. with a GUID in the subdomain or anything, and resolve them to your device's private IP. This requires some infrastructure to programmatically add DNS entries and create certificates for those. This is easily possible today with Cloud DNS providers from Google, Azure & Co. That's rather complex, but works perfectly - as long as your user's routers do not do DNS rebinding protection.

If you happen to only have customers that use routers that don't do DNS rebinding protection, or you control the local network infrastructures and internet access to resolve the domain name, or run a local DNS server, you are cool. This is a very viable and maybe the best solution then. If you don't control the network infrastructure, especially the router, the risk is high that DNS rebinding protection becomes more popular and your solution begins to fail for more and more users.

Pro: Except for the domain lookup, all traffic is local and marked safe - even in security sensitive browsers - and works in many scenarios without a hassle.
Con: Works flawlessly in most setups, but is very troublesome to the non-tech-savvy user that happens to use secure routers. :-(

Solution #3: Roll your own crypto! Well, not really

I've actually seen this done in production. You can rely on well-tested and proved, high quality JavaScript cryptography libraries that allow you to implement things like certificate validation (except the hostname, yo!) and encryption for the payload. There's even standardization for such a library under way in the W3C.

You should totally make sure that you have the proper knowledge to tackle this, or let your solution verify by someone that knows this kind of stuff. It's very easy to make mistakes.

Pro: It's pretty much guaranteed to work and can be very safe when you do it right.
Con: Your browser will still tell your users that communication is insecure, albeit all the effort. And you can not communicate with your device using standard clients, such as curl. :-(

Solution #4: Public reverse proxy

Crazy: To gain security that is trusted by all browsers and clients, you have to open your local network to the public internet! To get HTTPS for local devices, you could set up a publicly accessible reverse proxy that forwards incoming traffic to your local device through a secure tunnel. A simple home-grown setup could look like this:

1) Create a unique and not-easily-guessable subdomain for each of your IoT devices, such as 09460e83-1759-4fbd-afed-9ee4adf8b288.iot.example.com
2) Generate a certificate via Let's Encrypt for that domain and deploy it to your IoT device.
3) Set up a ssh connection from your IoT device that tunnels incoming traffic on the HTTPS port to the local web server:

ssh user@09460e83-1759-4fbd-afed-9ee4adf8b288.iot.example.com -R *:443:localhost:443

And zap, there you go, you have HTTPS to your local device.

There are services that make this even easier, such as https://ngrok.com/ and https://webhookrelay.com/.

Pro: Will be displayed as secure in all browsers.
Con: It does not work locally without internet access and exposes your device to the public internet. :-( But at least the access point cannot be found by guessing or scraping your service. Combined with very good passwords, this is not that bad, but it's still much more exposed than it ought to be.

Solution #5: Port forwarding and dynamic DNS

This is basically the same as Solution #4, but even less secure. Your device can connect to a dynamic DNS service (either built by yourself or you use one of the popular ones) and keep up a domain that points to the router's public IP address. Then it gets a certificate from Let's Encrypt for that domain, and via UPnP it can programmatically set up a port forwarding to itself. But heck, your device is now publicly accessible from the internet, and public IP enumeration will make it detectable by hackers.

Pro: Not that complex to setup up
Con: Very insecure. First, you don't want to use port forwarding at all, because it drills a whole in your local network that is otherwise cut off from the public net. Second, this certain setup makes your device publicly accessible by enumerating ISP's IP ranges, and it will be very likely to be targeted by script kiddies and crackers. Don't do this! :-(
Additionally - and I am glad that this is the case by now - most routers don't allow UPnP by default and it must be enabled manually.

How would a proper solution look like?

Well, the problem that we have here is trust. Your browser does not trust your device, because the browser can not verify that the device contains only software from someone trustworthy. On the public web, this is done through the hostname, because domains are registered to an entity that you can track down and stick your foot up their ass if they do something malicious. But how can you verify this with devices, that can be potentially spoofed?

It seems that Apple has a solution for this in it's HomeKit echosystem. The strongest solution seems to be possible with a special Authentication Coprocessor. But as of somewhen in 2017, a software authentication solution has been offered, too. But this solution is by no means open or publicly documented (as is typical for Apple), so I don't have a clue how it works. I bet it is pretty secure, though, and would very much like to know how it works!

Short of knowing Apple's secrets, I'd come up with my own proposals of how a secure system could be designed.

Proposal #1: Simply drop the host name check in HTTPS for private IPs

This would be a no-brainer for browsers, and IMHO not that bad for security. The full certificate is checked, like expiration date, chain of trust. But the contained domain is ignored. Instead, the user is shown the company that the certificate has been issued to, and asked whether she trusts this company. Similar to how desktop applications ask you whether the setup may run as admin/root. The security would be still very high - at least much higher than the current state of the art.

How well does this solution deliver on the three promises of HTTPS?

1) Privacy: The communication is encrypted and can not be sniffed, even when someone took complete control over your local network - except for the device you are connecting to.
2) Integrity: This holds true in the same way
3) Identification: This is not as strong as it is with internet domains. By typing in the domain, or clicking on a link, you already tell the browser beforehand who you trust. When browsing to an IP, you don't make verifyable assumptions about the identity of your peer. That's why I'd consider the one-time popup telling you the identity of whoever controls the device with the IP you entered, which is, IMHO, basically as strong as typing in a domain name.

Proposal #1.a: With added Public Key Pinning

This proposal is my favorite. Browsers act like described in Proposal #1, but with added HTTP Public Key Pinning of the most immediate certificate - it must not be any arbitrary certificate from the certificate chain, but the immediate certificate that verifies the very host that you are just connecting to. The browser will "pin" this certificate to this host - which may be an IP address - and warn the user when it connects to the same host, but receives a different certificate this time. This is how ssh has been doing it for ages. This means that it's considered secure enough for a remote shell that gives you potential root access to a device, so I guess it's secure enough to connect to IoT devices that are only accessible from your local network, via a usually limited API, too.

Proposal #2: Let the manufaturer validate the device's authenticity

This proposal is more sophisticated than #1, but more complex, too. As always, for a bit more security, you'll need exponentially more effort.

I think there is no general method to verify that a device has not been tinkered with. But, for sure, the manufacturer should be the instance that knows best how to check the device for authenticity. So why not let the manufacturer do this? Browsers could implement a mechanism following this scheme:

Connect to an IoT device using HTTPS
The browser reads the certificate and validates the usual stuff, like expiration date and chain of trust - but NOT the hostname
The HTTPS response contains a randomly generated session id and a random token, and the certificate includes an URL to an online validation web service (used later)
The device is expected to connect to that web service, and send it a random token for the given session id
The manufacturer can choose to only accept this token after it did some kind of validation of the device's integrity
Before any more communication is done, let the user verify that the manufacturer from the certificate is the institution that she expected and trusts. This is much like what desktop operating systems do when installing software. Asin Proposal #1, this is the important part that establishes the trust.
When the user confirms, the browser contacts the validation web service (it must be HTTPS!), send it the session ID and read back the token. This token is compared to the one contained in the HTTPS response from the device
Only when the tokens match, the connection is considered successful -As a bonus: The device's certificate and the validation service's certificate must be issued to the same company. This would further enhance the trustworthyness.

How well does this solution deliver on the three promises of HTTPS?

1) and 2) are the same as in the first proposal, and as strong as for public domains
3) Is a bit stronger in this proposal, because you couple the trusted, well-tested public domain verification with the local device verification. Plus, the manufacturer can opt to implement additional steps to verify that the device has not been tempered with. This would be pretty strong, IMHO not a bit less secure than public domain verification, but of course it'd be quite a hassle to implement. Plus: It requires an active internet connection for the device and the browser. (But again, the I in IoT stands for "Internet".)

I think, however, that the first proposal is trustworthy enough. You just want to verify that the device you are connecting with is the one you expect to. If this is the case, because you recognize the manufacturer's name, this should be good enough™, at least for my taste. And with the added Public Key Pinning of proposal 1.a, you get quite a solid security setup.

Conclusion

Fact is that local HTTPS connections will only be possible without additional tinkering after the standards have been changed. I think there is really a need that the IETF, browser developers and whoever else is needed, have a look at this issue and provide a solution to IoT manufacturers. It has always been common to offer local web pages as front-ends for IoT devices, and it will become even more popular. Because of the problems discussed in this post, which have become even more proplematic over the last months and years, currently nearly all manufacturers fall back to HTTP, to circumvent all the trouble. This is a real pity and should be changed ASAP!

Go: Asynchronous Real-Time Broadcasting using Channels and WebSockets

Daniel Albuschat — Mon, 09 Apr 2018 21:00:14 +0000

Recently, I had the need to execute a long-running command on a server and send the result to - potentially - multiple clients, in real-time. I naturally chose to use WebSockets as a transport layer. But when implementing the backend in Go, I was looking for a solution to distribute the results that exec.Command spit out to multiple clients in a thread-safe manner. In Go, channels are the tool of choice when sending data asynchronously and thread-safe, but channels communicate between two endpoints only. There's no way to "hook into" a channel from multiple clients. So I needed a different solution. On my search, I learned that you can send channels over channels, which makes channels a much more versatile tool than they already had been for me. Using this, much of the constrains that I thought channels had, have been lifted.

Equipped with this knowledge, you can write stuff like this, which is basically a client/server infrastructure:

serverChan := make(chan chan string, 1)
clientChan := make(chan string, 1)
serverChan <- clientChan

Yes, we sent a channel into a channel! This doesn't do much, indeed. We need to receive the clientChan and send something back, to have a true client/server feeling:

go func(serverChan chan chan string) {
    client := <-serverChan
    client <- "Hello, World!"
}(serverChan)
fmt.Println(<-clientChan)
close(serverChan)
close(clientChan)

Yay, we made it! We sent a channel via a channel, just to send back a string. That in itself might be already breathtaking (right?), but it gets even better: We can use this as a building block to send multiple clients to the server, to make the server "broadcast" stuff to multiple clients, in real-time. We use select for that, so it's even very light on CPU usage.

Let's have a look at the server code:

func server(serverChan chan chan string) {
    var clients []chan string
    for {
        select {
        case client, _ := <-serverChan:
            clients = append(clients, client)
            // Broadcast the number of clients to all clients:
            for _, c := range clients {
                c <- fmt.Sprintf("%d client(s) connected.", len(clients))
            }
        }
    }
}

The server just listens endlessly for new clients, adds all of them to a list and sends a text stating the number of total clients to all clients, whenever a new client connects.

This is what the client code can look like:

func client(clientName string, clientChan chan string) {
    for {
        text, _ := <-clientChan
        fmt.Printf("%s: %s\n", clientName, text)
    }
}

And this code glues it all together:

// Start the server:
serverChan:= make(chan chan string, 4)
go server(serverChan)

// Connect the clients:
client1Chan := make(chan string, 4)
client2Chan := make(chan string, 4)
serverChan <- client1Chan
serverChan <- client2Chan

// Notice that we have to start the clients in their own goroutine,// because we would have a deadlock otherwise:
go client("Client 1", client1Chan)
go client("Client 2", client2Chan)

// Just a dirty hack to wait for everything to finish up.
// A clean and safe approach would have been too much boilerplate code
// for this blog-post
time.Sleep(time.Second)

This will output, in some semi-random order:

Client 1: 1 client(s) connected.
Client 1: 2 client(s) connected.
Client 2: 2 client(s) connected.

Voilá! In theory, this should scale "indefinitely", just as indefinitely as everything in computer science does. :-)

Now we can hook this up to our WebSocket server. I chose github.com/gorilla/websocket as my WebSocket library. In this example, the server will just count it's uptime in seconds, and will push this info to all clients. We can choose whether new clients get all output that has happened since the server started, or just the new stuff since the client connected. In my implementation, I chose to send only the new stuff for now.

First off, let's write a slightly modified server that counts it's uptime:

func uptimeServer(serverChan chan chan string) {
    var clients []chan string
    uptimeChan := make(chan int, 1)
    // This goroutine will count our uptime in the background, and write
    // updates to uptimeChan:
    go func (target chan int) {
        i := 0
        for {
            time.Sleep(time.Second)
            i++
            target <- i
        }
    }(uptimeChan)
    // And now we listen to new clients and new uptime messages:
    for {
        select {
        case client, _ := <-serverChan:
            clients = append(clients, client)
        case uptime, _ := <-uptimeChan:
            // Send the uptime to all connected clients:
            for _, c := range clients {
                c <- fmt.Sprintf("%d seconds uptime", uptime)
            }
        }
    }
}

This was rather easy. However, if you've read carefully, you will notice that clients are being registered via append(clients, client), but never removed. This will lead to orphaned clients that will cause the server to crash when it tries to write to them. We will leave it at that for the moment, in order to not clutter the code with boilerplate. So beware that your server will crash when you close browser tabs in the following sections.

Next, we write our WebSocket server around our new uptimeServer goroutine:

// This upgrader is needed for WebSocket connections later:
var upgrader = websocket.Upgrader {
    ReadBufferSize: 1024,
    WriteBufferSize: 1024,
    CheckOrigin: func(r * http.Request) bool {
        return true // Disable CORS for testing
    },
}

// Start the server and keep track of the channel that it receives
// new clients on:
serverChan := make(chan chan string, 4)
go uptimeServer(serverChan)

// Define a HTTP handler function for the /status endpoint, that can receive
// WebSocket-connections only... so note that browsing it with your browser will fail.
http.HandleFunc("/status", func(w http.ResponseWriter, r *http.Request) {
    // Upgrade this HTTP connection to a WS connection:
    ws, _ := upgrader.Upgrade(w, r, nil)
    // And register a client for this connection with the uptimeServer:
    client := make(chan string, 1)
    serverChan <- client
    // And now check for uptimes written to the client indefinitely.
    // Yes, we are lacking proper error and disconnect checking here, too:
    for {
        select {
        case text, _ := <-client:
            writer, _ := ws.NextWriter(websocket.TextMessage)
            writer.Write([]byte(text))
            writer.Close()
        }
    }
})
http.ListenAndServe(":8080", nil)

Phew, that's a harder nut to crack. As you can read from the inline comments, we do some stuff to set up the WebSocket infrastructure, start our "server" that counts the uptime and then define a HTTP handler function to receive client requests, which in turn will lead to clients registered to our uptime-server, and subsequently write all texts received from the uptime-server to the WebSocket clients.

You can compile and start this, and test it out with this minimalistic JavaScript example:
https://jsfiddle.net/L31wy1gL/13/
(BTW: That's why we disable CORS in the code; to be able to reach the WebSocket server from jsfiddle/codepen/your dummy index.html, etc.)

That's where I'll end this blog post. You have seen a lot:

• Send channels into channels
• Use this to implement a "client"-registry in a goroutine
• Send broadcast messages to these clients
• Wrap all this into a WebSocket server

But you still have much to do:

• Implement proper error handling
• Implement proper handling for disconnecting clients
• Maybe turn this into a library that you can re-use across your project(s)

I hope you enjoyed this read and would be grateful for suggestions and feedback in the comments.

Using Elm on a Raspberry Pi

Daniel Albuschat — Sun, 04 Feb 2018 22:33:09 +0000

Running Elm on your Raspberry Pi

Elm is an awesome programming language that makes you feel comfortable, empowered and safe. Since I went out on a journey to put a Kubernetes cluster on a few Raspberry Pis and built a frontend for a few experiments (see kube-alive), I had the need to compile Elm code on Raspberry Pis. Unfortunately, there are no official Elm binaries for the arm CPU architecture distributed via npm or any other way.

    daniel@kubepi-red:~ $ npm install elm
    > elm@0.18.0 install /home/daniel/node_modules/elm
    > node install.js

    Unfortunately, there are currently no Elm Platform binaries available for your operating system and architecture.

    If you would like to build Elm from source, there are instructions at https://github.com/elm-lang/elm-platform#build-from-source

    npm WARN This failure might be due to the use of legacy binary "node"
    npm WARN For further explanations, please read
    /usr/share/doc/nodejs/README.Debian

So I set out to compile Elm for arm using a Raspberry Pi on raspbian (stretch) myself. This turned out to be rather tedious, because of two factors that are a bad combo: Not all versions of Elm dependencies (especially Cabal) work as expected and are compatible. And compiling one version of Cabal takes literally days. (Well, at least a bit more than 24 hours.)

The good news is that it is pretty straight-forward and should work without hassles if you follow the steps in this tutorial and use the exact same versions I did. It will still take approximately two days to install and compile everything that you need. The even better news is that I already did this for you, and can provide a much faster way to run Elm on your Raspberry Pi via docker.

The shortcut: Use a prebuilt docker image

If you don't want to wait that long to build Elm, you can use my pre-built docker image that you can run on your raspi. Find it on Docker Hub as danielkun/elm-raspbian-arm32v7.

Just run

docker run -it danielkun/elm-raspbian-arm32v7 bash

and inside that container you can use the elm commands as you are used to. Note that everything that happens in this docker run session will be discarded once you close the shell. That's why you should mount a directory as a volume into the container - these changes are made directly to your filesystem and will be permanent.

To run the elm docker container with the current directory mounted to /code and everything set up correctly, execute

docker run -it --rm -v "$(pwd):/code" -w "/code" -e "HOME=/tmp" -u $UID:$GID -p 8000:8000 danielkun/elm-raspbian-arm32v7

If you want to stick with this for longer, you can make your local "elm" command an alias for this, and all parameters will be appended, so that you can run "elm make", "elm reactor", etc. as if it was installed on your local machine. Set up the alias like this:

alias elm='docker run -it --rm -v "$(pwd):/code" -w "/code" -e "HOME=/tmp" -u $UID:$GID -p 8000:8000 danielkun/elm-raspbian-arm32v7 elm'

(Credit goes to maport from https://github.com/maport/docker-elm for this trick)

Building Elm yourself

If you're not satisfied with using a docker image, you can still build Elm yourself, if you bring enough time.

Elm does only have a few platforms it depends on, but these are massive, and not all can be obtained as binaries:

llvm-3.5 (for ghc, can be obtained as binary)
ghc 7.10.3 (can be obtained as binary)
cabal-install 1.22.6 (must be compiled, hence slooooow to install)

Besides these larger dependencies, there are a few packages that you can install via apt, and many that will be downloaded and compiled while building Elm via cabal.

So let's start with installing the packages that we can obtain from apt, since this is the easiest part:

sudo apt-get update && sudo apt-get install -y wget curl make libgmp-dev libnss3 apt-utils libnss-lwres libnss-mdns netbase ca-certificates cmake g++ libtinfo-dev git zlib1g-dev

As part of the building process and to use the elm commands later, we need a few directories for llvm, cabal and finally elm in our PATH:

echo 'export PATH="$PATH:$HOME/elm/llvm-3.5/bin:$HOME/.cabal/bin:$HOME/elm/Elm-Platform/0.18/.cabal-sandbox/bin/"' >> ~/.bashrc && source ~/.bashrc

Now, download everything we'll need later:

mkdir ~/elm-downloads && cd ~/elm-downloadswget http://releases.llvm.org/3.5.2/clang+llvm-3.5.2-armv7a-linux-gnueabihf.tar.xz
wget https://downloads.haskell.org/~ghc/7.10.3/ghc-7.10.3-armv7-deb8-linux.tar.bz2
wget http://www.haskell.org/cabal/release/cabal-install-1.22.6.0/cabal-install-1.22.6.0.tar.gz

Then extract the packages in a dedicated directory for building Elm:

mkdir ~/elm && cd ~/elm
tar xf ~/elm-downloads/clang+llvm-3.5.2-armv7a-linux-gnueabihf.tar.xz && mv clang+llvm-3.5.2-armv7a-linux-gnueabihf llvm-3.5
tar xf ~/elm-downloads/ghc-7.10.3-armv7-deb8-linux.tar.bz2
tar xf ~/elm-downloads/cabal-install-1.22.6.0.tar.gz

LLVM is now ready to be used from where it has been extracted. GHC, however, needs to be installed:

cd ~/elm/ghc-7.10.3/
./configure
make install

That should be fairly quick, since it's not building anything, but just copies around files. Next, go for the hard (and time-consuming) task: Build cabal-install

cd ~/elm/cabal-install-1.22.6.0
./bootstrap.sh

Now grab a coffee. And your wife and kids and go outside for a few hours. Then go to sleep and wake up the next morning, go to work and then, after work, it might have finished. Mind you that I used the most recent Raspberry Pi 3 and it still took around 10 hours or so. If you have a Raspberry Pi 2, for example, it might take even longer.

But it will finish eventually. And when it did, you can download and compile Elm now. (Remember: Your PATH is already set up correctly.) Note that I had to fix some small issue with the installation script, since it sets "split-Objs: True", which seems to be not compatible with the GHC or cabal version, so I changed it to "split-Objs: False".

curl -sSL https://raw.githubusercontent.com/elm-lang/elm-platform/master/installers/BuildFromSource.hs \
 | sed "s/split-objs: True/split-objs: False/" > BuildFromSource.hs && runhaskell BuildFromSource.hs 0.18

This, again, takes about a day or so. I haven't measured exactly. It takes so long because there are quite a few cabal dependencies that will be downloaded and built before Elm is built.

Hooray! You should have working `elm' commands now! Have fun hacking away on your raspi and make some cool IoT projects.

Finally, you can clean up quite a bit by removing no longer necessary downloads and temporary files:

rm -rf ~/elm-downloads ~/elm/ghc-7.10.3 ~/elm/cabal-install-1.22.6.0

Elm for the win!

Kubernetes: It's alive!

Daniel Albuschat — Sat, 20 Jan 2018 19:29:28 +0000

I recently found an interest in Kubernetes and learned about it at night, while working at something not web-related at day. As part of my learning journey I wanted to quickly see and experience how Kubernetes actually works in action. So I decided to write a few services that can be used to trigger and observe certain behavior of Kubernetes. I started with Load Balancing, Self Healing of Services and Auto Scaling depending on CPU utilization.

In this blog post I will explain how each service works and how Kubernetes behaves in practice. Note that this was the first time I wrote Go, so I take no guarantee that the code is not shitty or against Go conventions and best practices that I do not yet know. :-)

This blog post addresses anyone who already has a Kubernetes cluster up and running. You should know what a Pod is, a Replica Set or a Deployment, and can build Docker containers and have used a Docker Registry.

If you do not run a Kubernetes cluster already, I can recommend the book "Kubernetes: Up & Running" and/or the blog post "How to Build a Kubernetes Cluster with ARM Raspberry Pi" by Scott Hanselman. If you don't know what Kubernetes is or how it works, you can read the excellent series of blog posts "Understanding Basic Kubernetes Concepts" by Puja Abbassi from giantswarm.io.

For the impatient

Before running kube-alive on your cluster, make sure that you have the following:

kubectl installed and configured to a running cluster (check that "kubectl get nodes" gives you a list of at least one node in state "Ready")
bash
Your cluster runs on Linux on amd64 or ARM CPUs

If you do not have a cluster up and running already, I recommend the already mentioned article by Scott Hanselman to start a cluster on Raspberry Pis, or you can use Minikube to run a local cluster on your PC or Mac.

If you just want to deploy kube-alive to your cluster and see it in action, you can do this with this single command:

curl -sSL https://raw.githubusercontent.com/daniel-kun/kube-alive/master/deploy.sh | bash

Using 192.168.178.79 as the exposed IP to access kube-alive.
deployment "getip-deployment" created
service "getip" created
deployment "healthcheck-deployment" created
service "healthcheck" created
deployment "cpuhog-deployment" created
service "cpuhog" created
horizontalpodautoscaler "cpuhog-hpa" created
deployment "frontend-deployment" created
service "frontend" created

FINISHED!
You should now be able to access kube-alive at http://192.168.178.79/.

Load-Balancing

The most basic capability of Kubernetes is load balancing between multiple services of the same kind. To observe whether the request was served from the same or from different instances, I decided to let the service return it's host's IP address. In order to run a service in Kubernetes, you need to a) write the service, b) build a container hosting the service, c) push the container to a registry, d) create an object in Kubernetes that runs your container and finally e) make the service accessible from outside the cluster.

Phase A: Writing the Service

So let's dig into the code. I wrote a server in Go that serves on port 8080, parses the output of the command "ip a" and returns the container's IP address.

package main
import "fmt"
import "bufio"
import "os/exec"
import "log"
import "strings"
import "net/http"

/**
getip starts an HTTP server on 8080 that returns nothing but this container's IP address (the last one outputted by "ip a").
**/
func getIP() string {
    // Left out for brevity, see 
https://raw.githubusercontent.com/daniel-kun/kube-alive/master/src/getip/main.go 
}

func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintf(w, getIP())
    })
    fmt.Printf("'getip' server starting, listening to 8080 on all interfaces.\n")
    http.ListenAndServe(":8080", nil)
}

Phase B: Building the Container

Since everything running in Kubernetes must be a container, I wrote a Dockerfile to run this service:

FROM golang
COPY main.go /go/src/getip/main.go
RUN go install getip
ENTRYPOINT /go/bin/getip

This Dockerfile is simple: It uses a golang base container that is prepared to compile and run Go code. It then copies over the only source code file, main.go and compiles and installs it to /go/bin/ using "go install".

The installed binary /go/bin/getip is set as the Entrypoint, so that when no argument is given to docker run, it executes our service.

You can build the container using:

docker build .

Note that there is a "." at the end of the command, meaning that you must have cd'ed to the getip source directory before executing docker build.

After docker build finishes, you will be able to see the new container with a new, randomly generated image id via

docker images

The container is only available locally on the machine that it has been built. Since Kubernetes will run this container on any node that it sees fit, the container must be made available on all nodes. That's where a Docker Registry steps into the game, which is basically a remote repository for Docker containers that is available from all nodes.

Phase C: Pushing the Container to a Registry

I first tried to set up a local registry, which can be done, but the setup is not
portable across clusters. That's why I decided to simply use Docker's own registry, https://hub.docker.com. To push your freshly built container, you first need to register at Docker Hub, then tag the container with the repository, desired container name and an optional tag. If no tag is given, "latest" is assumed.

docker tag <your-repository>/getip <image id> # tag the docker image with your repository name and the service name, such as "getip"
docker login # enter your username and password of http://hub.docker.com now.
docker push <your-repository>/getip # and then push your container

It is now available to be pulled (without authorization) by anyone - including your Kubernetes nodes.

Phase D: Define a Replica Set

To let Kubernetes know that it should run this container as a service, and to run multiple instances of this service, you should use a Replica Set. I wrapped the Replica Set into a Deployment to easily upgrade the service later:

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: getip-deployment
  labels:
    app: getip
spec:
  replicas: 4
  selector:
    matchLabels:
      app: getip
  template:
    metadata:
      labels:
        app: getip
    spec:
      containers:
      - name: getip
        image: <your-repository>/getip
        ports:
        - containerPort: 8080

I set the number of replicas to 4, which means that Kubernetes will do everything it can to always have exactly 4 instances running at any time. However, this does not give us a single URL to connect to these instances. We will use a Service to Load Balance between these instances:

kind: Service
apiVersion: v1
metadata:
  name: getip
spec:
  selector:
    app: getip
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080

This service provides a single, load-balanced URL to access the individual service instances. It remaps default HTTP port 80 to the service's own port 8080 in the process. The service will be available as http://getip.svc.default.cluster or, even shorter, as http://getip on any running Kubernetes Pod.

However, this service is only available from inside Kubernetes and not from "outside" the cluster.

Phase E: Publish the Service

I decided to build my own nginx container to serve the static HTML and JavaScript files that make up the frontend and publish the services from a specific IP.

events {
    # empty
}
http {
    server {
        root /www/data;
        location / {
            # for the frontend SPA
        }
        # Forward traffic to <yourip>/getip to the getip service.
        location /getip {
              proxy_pass http://getip/;
        }
        # I have left out the other services like "cpuhog" and "healthcheck" here for brevity.
        # See their code on https://github.com/daniel-kun/kube-alive/

        # Allow WebSocket connections to the Kubernetes API:
        location /api {
              proxy_pass https://kubernetes.default/api;
              proxy_http_version 1.1;
              proxy_set_header Upgrade $http_upgrade;
              proxy_set_header Connection "Upgrade";
              proxy_set_header Authorization "Bearer %%SERVICE_ACCOUNT_TOKEN%%";
        }
    }
}

So we see that nginx expects the SPA in /www/data/, which will be the target of COPY commands in our Dockerfile. The service getip is reached via Kubernetes DNS, which will automatically resolve a service's name to it's Cluster IP, which in turn load-balances requests to the service instances. The third location /api is used by the frontend to receive information about running pods. (Currently, the full API is exposed with full admin privileges, so this is highly insecure - do it in isolated environments only! I will fix this in the near future.)

Here's the Dockerfile for the frontend service:

FROM nginx
COPY nginx.conf /etc/nginx/
COPY index.html /www/data/
COPY output/main.js /www/data/output/main.js
COPY run_nginx_with_service_account.sh /kube-alive/
CMD /kube-alive/run_nginx_with_service_account.sh

The shell script run_nginx_with_service_account.sh will substitute variables in the nginx.conf to use the Kubernetes Service Account token in the authorization header to let nginx handle the authorization so that the frontend does not have to.

So now we are prepared to put the last piece of the puzzle into place: A Replica Set to run the frontend and a Service that externally publishes the frontend. Note that I wrapped the Replica Set into a Deployment again:

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: frontend-deployment
  labels:
    app: frontend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - name: frontend
        imagePullPolicy: Always
        image: <your-repository>/frontend_amd64
        ports:
        - containerPort: 80
---
kind: Service
apiVersion: v1
metadata:
  name: frontend
spec:
  selector:
    app: frontend
  externalIPs:
  - <put your external IP, e.g. of your cluster's master, here>
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80

That's it! You can kubectl apply this after you inserted a valid externalIP and everything should be up and running to execute your first experiment with Kubernete's load balancing.

Reaching to the IP that your "kubectl" is configured against, should give you this UI:

… followed by more experiments. These all follow the same concept as getip, you can have a look at their code and deployment yamls here:

Self-Healing

The code of the Self-Healing experiment is here:

https://github.com/daniel-kun/kube-alive/tree/master/src/healthcheck
https://github.com/daniel-kun/kube-alive/blob/master/deploy/healthcheck.yml

Rolling Updates

The code of the Rolling Updates experiment is here:

https://github.com/daniel-kun/kube-alive/tree/master/src/incver
https://github.com/daniel-kun/kube-alive/blob/master/deploy/incver.yml

Auto Scaling (cpu-based)

The code of the Auto Scaling experiment is here:

https://github.com/daniel-kun/kube-alive/tree/master/src/cpuhog
https://github.com/daniel-kun/kube-alive/blob/master/deploy/cpuhog.yml

I named the service "cpuhog" because it uses as much CPU as it can for 2 seconds for every request.

I plan to add more experiments in the future, such as an experiment for rolling updates using Deployments.

I hope that you found this blog post and the kube-alive services useful and would be thankful if you could leave feedback in the comments. Maybe one day kube-alive will be a starting point to see Kubernetes behaviour live in action for many starters and engineers that are evaluating Kubernetes for their own use.

Update from 01/25/2018: I removed the security warning, because security on the latest version of kube-alive has been tightened. Only a portion of the API is exposed (pods in the kube-alive namespace), and the frontend runs with a service account that has access only to the dedicated kube-alive namespace and only to read and list Pods. Hence, there is not much more information available via the API than is visible in the frontend anyways.

Update from 03/08/2018: Updated the gifs to the new, improved visuals and added the rolling updates experiment.

Hacky and Clean Programming

Daniel Albuschat — Tue, 09 May 2017 20:09:47 +0000

When I do programming, I do it in either of two "modes". Each of these modes serves a different purpose and I use them at different stages to achieve a goal. It is very important to be aware of the current stage you are in, because programming in the wrong mode can be most harmful for either productivity or code quality.

I call these two modes the "hacker mode" and the "clean mode".

Hacker mode

In "hacker mode", I try out either new technologies or architectural ideas. For example, when your current goal is to write a library to control a Bluetooth device, I would first go into "hacker mode" and try things out and figure out what would be the most efficient and most stable way to use the raw Bluetooth API.

When hacking, I do not care for code quality. I may violate my own guidelines, use ugly casts, name variables "x", "foo" or "data". The uglier, the better. Because I will definitely throw that code away. This is a very important aspect that should not be violated. You have to be aware that you will throw the code away, and you have to actually do this in the end. If what you hacked somehow does not work out, throw it away and start with a new hacking session - and throw this one away, too, once you learned the important aspects. No clean code. No docs. No comments. No unit tests.

And if you are afraid that, when re-writing what you have hacked before, the Second System Effect (as described by Fred Brooks in The Mythical Man Month) might kick in, I can assure you that with this method I have not yet ever experienced this. I think the Second System Effect applies to larger scales only, and only when putting effort into producing high quality in the first system, too.

After I hacked away and figured out which calls must be made in which order and know the parameters for best results and found a good structure for the API and the implementation, I sit back, have a good look at it and memorize the important parts. Then I stash the code aside for later reference and mentally mark it as "To Be Deleted".

Now I can switch to "clean mode".

Clean mode

I am programming in "clean mode" when I have a good mental model of what I want to create. When I know the important key parts of the implementation and have an idea of how the API and the architecture should look like. Often this information comes from a hacking session. Sometimes it is just there because I have been thinking about the problem for days, months or even years and now finally decided that everything is clear and can be put into code.

Most of the time, when programming in clean mode, I am doing DDD - Documentation Driven Development. Don't worry, this is not the new hip paradigm that you missed. It's essentially TDD, but I am writing the documentation of the code even before writing the tests.

Most important: The Docs

One major argument why TDD is cool is because you get "a feeling how your code works out when it is put into actual use". The same argument goes for writing docs. (Important note: I am talking about documentation that describes the functionality and properties of a class/function/etc. I am not talking about inline code comments, which describe details of the implementation). When writing docs, I always try to state the important details, not the obvious. When thinking about the non-obvious, you will learn whether the overall design is slick, or does have a few rough edges. With docs and tests combined, you can be pretty sure that the design is sound and that it works out well in practice. You usually even come up with new test cases this way.

There are a few, easy rules to follow when writing docs. This is especially important when writing them before the actual implementation.

First, it must be clear what the purpose of the element you are documenting is. If there is not one core function, but multiple tasks that are accomplished, you are most likely doing it wrong. Describe the core function of the element in one sentence, i.e. in a @brief. If the core function of the element can be unambigously deduced from it's name, you can skip this part.

Second, state the preconditions that must be met to use the element. The less preconditions you can name, the better. If there are no preconditions that are not enforced by the type system, you are doing it right. If the type system is not strong enough to express the constraints of the input parameters, document them in detail. The rumor has it that Haskell has an awesome type system, but unfortunately I have had not yet a chance to use it for productive work.

Third, and maybe most important for maintenance, state the side effects that may happen and which conditions they may likely happen under. If you are a really good programmer, you are writing functional code and can skip this part. You simply do not have side-effects.

Additionally to these, the standard rules obviously apply: Describing each parameter in detail (again: not the obvious!), possible exceptions thrown, how the return value has to be interpreted, etc.

While I do this, I always have two aspects in mind:

What actual use does the element have for a potential user and which goals would she target?
How will I possibly implement this element?

And it is very important not to get distracted by the implementation and create a bad API that does not resemble the tasks that a user of the class want it to accomplish, but instead just wraps the underlying technology. But it is also important to not create APIs that can not possibly be implemented in any reasonable way, or the performance will suffer considerably. You should avoid leaky abstractions for (nearly) all costs, but there is often a limit to this. (This is often worth prototyping in hacky mode.)

When I am done writing the docs, I begin writing the tests and see how my idea of how the API might be put to use actually performs. I use the docs that I have written and the side-effects and corner-cases that I have described to derive test cases and therefore get a pretty decent coverage.

I often draw some rough pseudo-UML to visualize the dependencies and relationships.

When actually implementing the functionality, I apply all the lessons that you have learned from Clean Code. I am aware of the docs and update them, when necessary. This may also lead to redesigning the API and therefore the tests. I am consciously taking the risk that implementation details that do not fit into the API are costly, because I have experienced it many times that this approach is worth it, since it results in well architectured, maintainable, clean and most importantly easy to use code.

Final notes

As stated in the beginning, it is most important to distinguish these two modes of programming and apply the correct one in each situation. Also, I would advice against using a mixture of both. Do not write hacky code in a clean code base and do not write clean code while hacking. It is just not worth it, because you either harm your code bases quality or are less productive than you could.

Only do hacking in fresh, isolated code bases that are drilled down to the minimal. This of course requires you to isolate parts and think of good, small components to build your software in, which is valuable in itself.