Forem: Bernardo

Mini guide, access your k8s service local development

Bernardo — Thu, 22 Sep 2022 19:47:33 +0000

In a kind/minikube/etc. cluster, start a deployment (a workload resource that manages a pod replicaset)

kubectl create deployment home --image=b4bz/homer

expose the deployment, i.e. make it receive comms from other containers or the outside.

kubectl expose deploy/home --port 8080

-> make sure the port exposed is actually the one the container exposes, e.g. for postgres this would 5432, for b4bz/homer it is 8080. There is no default here.

The command above creates a service of the type ClusterIP, which makes it accessible to other services only within cluster.

To make it accessible outside the cluster, the service has to be of type NodePort, like so

kubectl expose deploy/home --type=NodePort --port 8080

Now, how to you open it in a browser using ~~kind~~, k8s? Well, you can port-forward from the localhost to the container port using kubectl port-forward - checking out kubectl port-forward --help clarifies a lot.

kubectl port-forward services/home :8080
# assigns a random localhost port to the port 8080 of the container

Troubleshooting Traefik + Let's Encrypt + CloudFlare

Bernardo — Wed, 26 Jan 2022 12:58:08 +0000

Originally posted on brnrdo.hashnode.com.

Why I am writing this post

Hi. I'm a data scientist and I am currently developing a platform for MLOps using Docker Compose. This is to say, now that I am playing the role of an engineer, there are a lot of concepts that I am learning fast, and it can feel overwhelming and even scary. If you're new to deploying applications with CloudFlare and configuring a reverse-proxy with Traefik, I wrote this guide for you.

The goal of this post is also to show you how you can get that sweet green lock for SSL certificates by having Traefik fetch Let's Encrypt (LE) certificates for you and how to place your application behind a CloudFlare proxy. For sake of preview, here are some pain points that I've faced as a starting engineer.

Cloudflare as your DNS server.
CloudFlare flexible, full and strict modes.
- Watch out for https redirects in Traefik.
Exposing your server in CloudFlare: Development mode and temporarily disabling CloudFlare to bypass its proxy.
- Traefik configuration to fetch Let's Encrypt.

This post is not supposed a complete tutorial to Docker Compose, Traefik, CloudFlare and Let's Encrypt - there is already a lot of resources out there for that purpose. Rather, it is almost a personal log of the obstacles I encountered that I hope helps other people overcome them.

What is CloudFlare, Traefik and Let's Encrypt?

CloudFlare (CF) is mainly a DNS server with extra features - these extra features are attributed to CloudFlare's (reverse-)proxy functions, which you can enable and disable whenever you want.

For example, you set your DNS records to point your domain and subdomains to the IP of the server where your application is running. When you set these records to 'proxied' (i.e. orange cloud) you benefit from the proxy functions that CloudFlare carries out, such as concealing the real IP address of your server and DDoS protection.

Traefik is a reverse-proxy that sits at the edge of your application on the server. This reverse-proxy is Wait, what? Another "proxy"?! Well yes, but this Traefik is configured in your docker compose application and has no overlap in functionality with CloudFlare. Traefik will intercept requests to a given route, say a-route.your-domain.com and match with any existing rules that you have set to a service running in Compose.

Let's Encrypt (LE) is a Certificate Authority (CA) that signs and ensures that your certificates are genuine to encrypt the connection between the clients and your server. The best part is that by using LE, you are taking advantage of the ACME protocol that provides you with autorenewals of your certificates. You could use a self-signed certificate, but there are disadvantages to that. You can use Let's Encrypt from Traefik to minimize set up effort.

Setting up Traefik

Setting up proxy entrypoints

It's best to test things locally as much as one can before deploying it to a live server. Your setup should have two entrypoints, which I have chosen to name web and websecure for the sake of having distinct names than http and https, so that we know which fields are arbitrary names given by us.



services:
  traefik:
    command:
      - '--entrypoints.web.address=:80'
      - '--entrypoints.websecure.address=:443'

The entrypoints are part of the dynamic configuration of Traefik. This setup is less than ideal as it allows for unecrypted http connections. In Traefik, a redirection to https can be done in the following way.



services:
  traefik:
    command:
      - '--entrypoints.web.address=:80'
      - '--entrypoints.web.http.redirections.entryPoint.to=websecure'
      - '--entrypoints.web.http.redirections.entryPoint.scheme=https'
      - '--entrypoints.websecure.address=:443'

One of the options available from CloudFlare is none other that HTTPS redirection, so the headers are already rewritten at CloudFlare's proxy. The redirection configured in Traefik (~'origin server' as per CloudFlare's terminology) acts as a failsafe should you disable CF's proxy.

Setting up Let's Encrypt (from Traefik)

This step is entirely optional if you're just developing on your machine. TLS can be enabled without LE, in which case, Traefik issues its own certificates.

Of course, what is desirable in production is to have CA certificates. Under the hood, Traefik uses lego, an LE CLI client, to connect to LE servers and fetch certificates. You only need to supply Traefik with an email and some options and Traefik will handle the rest for you. Here's an example of a setup.



services:
  traefik:
    command:
      - '--certificatesresolvers.le.acme.email=${ACME_EMAIL}'
      - '--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json'
      # TLS challenge serves the certificates back to the CA in order to renew them
      - '--certificatesresolvers.le.acme.tlschallenge=true'
      # Optionally use the staging server to prevent exhausting rate limits
      - '--certificatesresolvers.le.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory'

The bare minimum is comprised of the first two lines: the email and the TLS challenge. This challenge is the simplest one to setup, as the only thing to do is to enable a boolean flag. However, taking into account CloudFlare, CF does not work with the TLS challenge, and either the DNS challenge or the HTTP challenge must be configured in order to be able to have the edge proxy enabled.

As a note, the default method used for ACME authentication by the Let's Encrypt client utilizes the DVSNI method. This will fail for a domain which has Cloudflare enabled as we terminate SSL (TLS) at our edge and the ACME server will never see the certificate the client presents at the origin. Using alternate ACME validation methods, such as DNS or HTTP will complete successfully when Cloudflare is enabled.

Note that challenges are mutually exclusive per certificate resolver, and each router can only use one certificate resolver at a time. Effectively, a router can only use one type of ACME challenge at a time. The snippets below are illustrative of choosing just one ACME challenge for the le certificate provider that is being configured.

Setting up Let's Encrypt with HTTP challenge



services:
  traefik:
    command:
      - '--certificatesresolvers.le.acme.httpchallenge=true'
      - '--certificatesresolvers.le.acme.httpchallenge.entrypoint=web'

Setting up Let's Encrypt with DNS challenge



services:
  traefik:
    command:
      - '--certificatesresolvers.le.acme.dnschallenge=true'
      - '--certificatesresolvers.le.acme.dnschallenge.provider=cloudflare'
    environment:
      # you may choose to use secrets instead of environment variables like this
      - CF_API_EMAIL=${CF_API_EMAIL}
      - CF_API_KEY=${CF_API_KEY}

In this example, the cloudflare provider is being used because that's where the DNS records are set up - i.e. the nameservers of the domain are pointing to CloudFlare. If you are using another DNS server, then you must set the environment variables specific to your provider.

Enable the use of Let's Encrypt in a router

Refer to the section Using the certificate resolver, this blog post will be more clear than me on how to use Traefik :)

Setting up CloudFlare

DNS records and subdomains

You only need one A record for the root of the platform you're developing. For any subdomain, you can set CNAMEs that point to that A record. The A record should point at the IP where Traefik is being served. Traefik will handle routing to the correct machines itself i.e. load balancing - should you have your application running in Swarm mode.

For example, DNS records on CloudFlare could look like this:

As a general rule, you only need to set A records (@ and www) that point to the real IP of your server. If the services you defined in Traefik follow the template a-service.yourdomain.com, then you only need to set CNAME records with the Name a-service pointing to the Content yourdomain.com.

Note the orange clouds: they indicate that that requests to the specificied (sub)domain are being proxied by CloudFlare's edge proxy. They can be disabled one by one, but that is not of interest to what I write in this post.

CloudFlare options

In general, and from my experience, the default configurations work well. I only call your attention to the CF's encryption modes (which I mention in the Troubleshooting section) and the HSTS option, that could be problematic for your case.

Tips

A very useful tip is to enable CF "Development Mode" so that web requests bypass any cache that CF may have created and allows you to check for changes without interference of a cache.

If you have set up LE only with the TLS challenge: use the option "Pause CloudFlare on Site" before doing docker-compose up in your server. This will allow Let's Encrypt to find the real IP of your server without CF hiding it. Related with this is the use of a DNS client (or dnschecker.org) to check where the nameservers are pointing your names to. Using dog (a CLI DNS client), you should get a record pointing to the real IP of your live server:



$ dog some-domain.com
A some-domain.com. 38s   xx.yyy.xx.yyy

Once the TLS challenge is complete and the certificates are issued, you should be able to check that

the certificates are stored in the acme.json file
in the browser window, at the page of the service being served at the specified route rule, you should click on the lock icon in the address bar, open "certificates" and verify that they are in fact issued by Let's Encrypt. This means you set up Traefik + LE correctly!

Once everything is working as it should, you can click on the option "Enable CloudFlare on site" and let CF hide your IP again. The output of the DNS check should be something like this



$ dog auth.some-domain.com
A auth.some-domain.com. 5m00s   104.21.28.234
A auth.some-domain.com. 5m00s   172.67.147.200

Those IP's belong to CF proxy and the output confirms that CF is enabled again.

Some troubleshooting

The list below describes issues that I have experienced with CloudFlare and Traefik. It may not be exactly your case, but they are noted down in case you face them.

Browser throws ERR_TOO_MANY_REDIRECTS when accessing a service: If you deployed using CloudFlare's DNS servers and set CF to 'Flexible mode' then you will incurr in this error. This is because you have https from the client to the CF proxy, and forced http between the CF proxy and the server (forced by CF). As Traefik was configured with an http-to-https redirection itself, this will cause an infinite loop of redirects. In order to solve this issue, you need to set CF to Full encryption mode, and even without LE (using Traefik's self signed certificate) your website/application is now accessible.
Browser throws ERR_SSL_VERSION_OR_CIPHER_MISMATCH when accessing a service: I ran into this error because the "edge certificates" (the certificates sitting at CloudFlare's proxy) did not have the same coverage for the routes I was setting up with Traefik.
- I was setting up routes with the structure service.platform.your-domain.com. This is called a 4th level subdomain.
- The "edge" certificates provided by CloudFlare (the "edge" being CF's proxy) only provide SSL up to 3rd level subdomains, specifically, to your-domain.com and *.your-domain.com.
- Thus the LE certificates fetch by Traefik did not match the names "made available" by CloudFlare, and the error above was produced.
- The only solution is to upgrade to a Enterprise subscription to CloudFlare 💸, as this allows you to issue edge certificates for any level of subdomains you want.

Closing words

I hope that you have found at least one solution to your problems if you're working with this particular set of tools (Docker Compose, Traefik, CloudFlare and Let's Encrypt). I may update this post as I add more functionalities to the platform I am developing.

A docker appreciation post and use case with tensorflow

Bernardo — Tue, 02 Jan 2018 21:12:16 +0000

Recently I had to work with a code base that used tensorflow, not the current version 1.4, but version 1.2.1.

An open-source software library for Machine Intelligence
tensorflow.org

So, tensorflow is a library that allows you to implement machine learning algorithms, namely neural network architectures. It allows you to train a neural network using the CPU (less cores with more clock speed) or using the GPU (more cores with less clock speed). Because neural networks are highly parallelizable in the GPU, it was in my interest to run tensorflow using the GPU: less training time, more iteration for parameter tuning.

However, tensorflow has some specific requirements with regards to some nvidia libraries, in order to run on the GPU:

cuda 8.0
cudnn 6.0

which are legacy and some other software I use for work requires cuda 9.0... I've had my share of installing and uninstalling stuff on my system at work and didn't want to have so much hassle just for a smaller project assignment.

So I thought to myself "if only there was, like, a tiny environment I could run tensorflow on the GPU... like, a docker image or something". My thoughts were heard and I could not believe my eyes when I found out that tensorflow could be run from a docker container and on the GPU :)

If you don't know what docker is or how to use it, I recommend you watch the video below or the written tutorial by the same guy Jonny L. I will try my best to summarize it here though.

Docker is a containerization system. It allows you to take slimmed down versions of operating systems (such as ubuntu with 111mb) and to build a specific image just for a specific application. It is like running someone else's computer system & configuration except you run it inside your own computer because the container is light enough for such.

To clarify, allow me to introduce quickly some concepts:

An image is like a snapshot of an operating system. It is not something that runs.
A container is an instance of an image. You start a container from an image, and you can leave it running or stop it. A container has its own filesystem, just like your machine does.
A volume, for the purposes of this post, is a directory that you specify to be shared between the host system (i.e. your machine) and the container that is running.

In my case, I was trying to solve sort of a "runs not on my machine" problem.^1 So, tensorflow (the organization) provides images on docker hub. For my case, it was a matter of finding the legacy image tag 1.2.1-gpu-py3. So to start with, I did

docker pull tensorflow/tensorflow:1.2.1-gpu-py3

So now, when you run docker images you can see this specific image listed. To start a container from this image I ran

nvidia-docker run -it --rm -p 8888:8888 --name followme -v $(pwd):/notebooks tensorflow/tensorflow:1.2.1-gpu-py3

Let's dissect this command:

nvidia-docker is just a special case for this particular problem to provide GPU access to the docker container, as is instructed by tensorflow. Otherwise, most use cases will simply use docker.
run is the docker subcommand to start a container
-it is a flag enabling an interactive tty, or simply put, a terminal of the container.
--rm is a flag instructing to automatically remove the container after exiting it.
-p 8888:8888 is a flag instructing to connect port 8888 of the container to port 8888 of the host. I tried to look up which one corresponds to which but I could not find this one out, sorry. The port 8888 is the default port of jupyter notebook server, which is included in the image.
--name followme, names the container "followme", which looking back is completely stupid given that the container is immediately removed. Well, the more you know, the better, though.
-v $(pwd):/notebooks connects the /notebooks directory of the container to the current working directory $(pwd) of the host system. Whatever you do in the container that writes out, is written to the host system.
finally tensorflow/tensorflow:1.2.1-gpu-py3 is the image from which this instance is started.
in this particular example, this command kicks off a jupyter notebook server by default. But if you need more control over the container "session", you could add bash to the end of the above command to start a bash session on the terminal.

And with that, I was able to train neural networks faster on the GPU using somebody else's legacy code. I find it to be a great example of what docker can do. And I can't wait to start building specific images, controlling and separating my development environments with docker, and living a happier life as a developer more focused on code :)

^1 Another use case may include one or similar or the entirety of the following issues: older os versions using an older linux kernel; camera drivers that are only compatible with older kernel versions; custom kernel patches that will break your beloved machine. In such case, it is desirable to test all this installation mess within a docker container and leave your system intact :)

Conda & Dealing with Conflicting Python(s) in your system

Bernardo — Thu, 09 Nov 2017 15:23:34 +0000

(Disclaimer: talking about a unix (me on linux) system here)

The problem

I took up a course on robotics using Robot Operating System (ROS). However, I always keep a miniconda3 installation to handle my data science adventures and respective dependencies. To clarify, it installs conda, a general purpose package manager - although really I only use it to install python libraries - and python 3.6.

What happens is that ROS uses the system python - i.e. python 2.7. The problem arises if you set your "python path" to default to that of miniconda3, a procedure which is automated when you accept that the miniconda3 installer adds the following line to your .bashrc file.

export PATH="/home/user/miniconda3/bin:$PATH"

In other words, everytime you open a shell, your system sets the python interpreter to that of miniconda3, which is version 3.6. So everytime I ran something from ROS, it would crash running python scripts that were written in version 2.7, namely due to syntax differences on the print command between the two versions.

The solution - meet symlinks

After reading conda's documentation on the subject, I decided to test my ability to implement their suggested solution, which is to:

remove miniconda3 from the PATH
create symlinks to three components of miniconda3
- conda
- activate
- deactivate
add these symlinks to the PATH

To remove miniconda3 from the PATH, one simply has to comment out or delete the aforementioned line in the .bashrc file, using some text editor like nano for example.

Now that that is done, let us create symlinks. A symlink, for the purpose of this post, is like a pointer to some other executable file and will function as a command. I created a specific folder (.symlinks) just for that.

mkdir .symlinks
cd .symlinks

Now that we're in the .symlinks folder, let's actually create symlinks.

ln -s /home/user/miniconda3/bin/conda conda
ln -s /home/user/miniconda3/bin/activate activate
ln -s /home/user/miniconda3/bin/deactivate deactivate

To verify that they were created, run ls -l (list files) in the .symlinks folder, you'll be able to see the pointers.

lrwxrwxrwx 1 user user 34 Nov  9 14:08 activate -> /home/user/miniconda3/bin/activate
lrwxrwxrwx 1 user user 31 Nov  9 14:10 conda -> /home/user/miniconda3/bin/conda
lrwxrwxrwx 1 user user 36 Nov  9 14:08 deactivate -> /home/user/miniconda3/bin/deactivate

At last, we add the .symlinks folder to the path. But we want to have these symlink commands automatically available every time we open a new shell. The answer is to "add" them to the script which is run everytime a shell is opened: the .bashrc file. So, add this line to the said file.

export PATH="/home/user/.symlinks:$PATH"

For the changes to take effect, you may close and reopen the terminal, but to feel like a pro without doing that move, run source .bashrc.

Now, if you run python --version you will get the system python back.

user@ryzen:~$ python --version
Python 2.7.12

If you want to use python 3.6 from the root conda environment, do

user@ryzen:~$ source activate root
(root) user@ryzen:~$ python --version
Python 3.6.3 :: Anaconda, Inc.

As you can see from the python --version bit, we're now using the 3.6 interpreter. If I want to activate the environment where I installed data science libraries¹ I'd run source activate ds.

Now that the newer interpreter is activated, you can run scripts that were written for that interpreter. Anyways, if you need to revert back to the system interpreter (2.7) on the current shell, then run

(root) user@ryzen:~$ source deactivate
user@ryzen:~$ python --version
Python 2.7.12

As you can see, we're back to 2.7. So you can run software that runs on this version with no problems, and switch back to your 3.6 antics whenever you want. ðŸ˜

Have a good one. ðŸ˜

1: I think conda does not allow to install packages to root anymore and forces users to create their own environments so as to incentivise good practices. Not that I checked, just personal experience.