Forem: Thiago Pinto

Raspberry Pi K3s Cluster

Thiago Pinto — Sat, 01 Jan 2022 22:46:36 +0000

The last time I played with the Raspberry Pi, I used Ansible to configure docker and some services. However, I still wasn't happy with the solution: my RPi 2 was slow to run the Ansible configuration, I couldn't run much on it, and the setup couldn't recover from any errors without manual intervention.

Now I've upgraded my hardware and certainly over complicated the software for a small home server, but now I got metrics =).

Hardware

Since I use Kubernetes daily, I decided to upgrade my home setup to a k3s cluster. I've started with three nodes:

RPi 2 model B
RPi 3
RPi 4 4GB ram

I added "so much" hardware because my objective is to monitor the cluster using Prometheus and Grafana.

The oldest node, RPi 2 model B, got decommissioned in the process. It wasn't consistent and had issues joining the cluster. So I replaced it with an RPi4 to run the cluster with HA.

OS bootstrapping

The first problem I didn't want to deal with manually was configuring the ssh on the RPis. In the previous process, I had one playbook to configure the ssh and user on Raspberian and another to configure everything else. This time, though, the node will boot with password login disabled and with my public keys automatically added to the host. To achieve this, I used Raspberian Firstboot with a custom script to set up the ssh configuration and hostname.

The second step is putting the Rapberian Firstboot image on the SD card with the firstboot.sh and bootstrap.env in the boot dir. After the node boots up, we can proceed directly to the Ansible configuration.

Ansible

Ansible is responsible for the initial setup of the nodes. After that, the playbook optimizes the RPi to run headless, configures k3s, and sets my local kubectl context with the new cluster. Since ArgoCD manages all k3s resources, Ansible also installs it in the cluster. Argo manages itself from here on.

ArgoCD

Why ArgoCD for such a small setup?

In the previous setup, I had to expose the ssh service on the internet to play with the cluster outside the LAN. I also had to run Ansible every time I modified the services. Because my RPi was slow, it was boring to wait several minutes for simple modifications. With GitOps on ArgoCD, I don't need direct access to the nodes to configure the services.

There was a risk that I would exhaust the cluster resources with only the maintenance services, but there are still around 60% free resources after adding:

ArgoCD
Prometheus
Grafana
External DNS
Cert-Manager
Kong

ArgoCD project and application bootstrapping

To bootstrap the ArgoCD configuration, I used the app of apps pattern. This configuration is Ansible's responsibility: it applies to k3s the initial set up. Service and infrastructure applications are separated to give fewer permissions to the non-infrastructure service configuration.

Secrets

My first approach to keeping secrets versioned was using Sealed Secrets. However, I gave up on the sealed secrets because I rebuilt the cluster frequently and always lost the encryption keys since I didn't care to make backups. I found it a lot easier to keep an encrypted secrets.yaml file in the repository. Every time I spin up a new cluster in this workflow, I have to decrypt the file and apply it to the new cluster.
Although it is encrypted, I don't recommend keeping the file in a public Github repository. I'm doing this to avoid losing the secrets while I work out a better flow for personal projects.

My Raspberry Pi Server Setup

Thiago Pinto — Sat, 01 Jan 2022 15:05:26 +0000

I've had a raspberry pi mainly sitting in a drawer for a while now. Haphazardly I dust off the SD card to set up the box with dynamic DNS and play with it as a personal server. It always went back to the drawer when I broke the SO with some crazy experiment.

As an SRE, I spend most of my time thinking about what flows are more burdensome to the team and how to automate the process to gain more time to do other important stuff. But with my RBP it was different. I'd never versioned the scripts I executed and never thought about creating proper automation for the box.

To avoid giving up on the Pi again, I started my journey differently this time. Every configuration running on the Pi must be replayable to fix it just by burning a new SO in the SD card and running the automation to set it up. This is what I have up to now.

Provisioning Raspbian

I started with the Rasbian Buster Lite image and burned it on the SD card. Assure you're pointing to the correct disk before running the commands below!

unzip -p 2020-02-13-raspbian-buster-lite.zip | sudo dd of=/dev/disk2 bs=4m
touch /Volumes/boot/ssh # to enable ssh

Ansible automation

Ansible Ansible is an excellent tool for automating systems configurations. This setup starts with the default user and password.

Initialize

The initialize playbook adds my user and public keys to the RBP and disables password ssh logins.

ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook initialize.yaml -i inventories/thspinto

Notice that inventories/thspinto/all.yaml is unreadable. That's because its encrypted with Ansible Vault. I don't recommend versioning secrets in public repositories for most use cases, even if they are encrypted. In my case, I versioned it because I'll certainly lose the file otherwise. Also the only sensitive information I have is a token to alter my DNS registry. Since there is virtually no traffic in my domain, it's no big deal.

Basic setup

In the second phase, I did a little hardening and added docker. This is done by the setup playbook:

ansible-galaxy install -r requirements.yml
ansible-playbook --vault-password-file /path/to/my/vault-password-file setup.yaml -i inventories/thspinto

Services

The last playbook spins up the services I want to run using docker-compose. I separated it from the rest because it is frequently changed and deployed.

ansible-playbook --vault-password-file /path/to/my/vault-password-file services.yaml -i inventories/thspinto

Currently I run:

Cloudflare DDNS to register the Pi's id the my domain.
Traefik 2.1 as proxy with docker service discovery
ForwardAuth to have OpenConnectID login using Auth0

My Pi

Everything runs fine on model B with one 700MHz core and 512MB of ram. Checkout the raspberry-config repo for more details and updates. However I can't really do much with this hardware. That's why as next steps I'm considering buying more RBPs and setting up a clustered environment with k3s.

ESP8266 Adventures

Thiago Pinto — Fri, 31 Dec 2021 23:02:42 +0000

How can my knowledge of technology help in global warming impacted future? My main area of study is in Computer Science and I've mostly had contact with intangible stuff. If, as a consequence of climate change, we enter a new era of scarcity, how can this intangible knowledge help?

With this in mind and with curiosity about IoT. Our most basic need is food. Vegetal food needs light, healthy soil and the right amount of water to grow. The most basic gadget I could build that can help food to grow and to learn with my experimentation is a weather station.

All that introduction is just to say that my goal is collecting the temperature, air humidity, soil humidity and sending it to my cluster for processing. Since we are in the future, no cables are allowed therefore of wireless connection is required. So I went online and bought the most simple-looking and cheap module I could find to couple it to an Arduino I've already owned. I got the ESP-12e and that is where a whole new world opened beneath my feet and thing got a little more complicated.

Programming the ESP8266

What I didn't know is that the ESP8266 is not just a Wifi module, it can be programmed and even run a web server inside. Of course there was a community leveraging this for IOT, and they led me to OpenMQTTGateway. This project collects various signals and sensor from IOT devices and sends it to a queue to be used by HomeAutomation services such as Home Assistant. Given my astonishment of the capacity of this module of course I didn't have any tools that would make my life easier in programming it.

Since I had an Arduino I thought it would be easy to use it to program the ESP. My first blocker is that I saw on a tutorial that I needed an external 3,3V power source because Arduino's 3,3V output didn't have enough current for the module. I found 3 used 1,5V batteries that connected in series gave me 3,6V. The datasheet says that 3,6V is within the operational voltage, so achievement unlocked for recycling used batteries. (I do have to notice that I found out later that used batteries are really bad at keeping the voltage I measured without any usage.)

Now I just had to flash the OpenMQTTGateway project into the module. Well, to put it into flash mode you have to reboot the module with the GPIO0. I found this nice schematics in instructables:

https://www.instructables.com/ESP-12E-ESP8266-With-Arduino-Uno-Getting-Connected/

My execution of it was a disaster because I bought the wrong pin bar (the ESP-12E has a smaller distance between the roles in comparison to the default protoboard) so I just attached the wires directly to the module. Of course that doing so made it very unstable and I couldn't breath near it without triggering a reset. But it I got it be be stable enough to flash it. The next issue I had was not flowing correctly the instructions: I was trying to use Arduino IDE to program the module and it wasn't able to connect. First of all it is very important to connect the ground to reset in the Arduino board. Apparently when you open the usb-to-serial converter Arduino automatically resets and keeping reset pin to ground avoids that.

Pin info

⚠️ To flash software to the module no other sensors can be attached.

I ended buying a fiberglass protoboard to weld everything together and make the setup more stable (I admit I do have practice welding to get more motor coordination). Finally, I also added a USB cable and a tension regulator to 3,3V for a constant power supply.

You might ask me why didn't you just buy the nodemcu? Well, I basically didn't know of its existence and I never thought that "wifi module" could do so much. It was fun to learn, but it is certainly better to buy the nodemcu because it is pretty cheap (even in Brazil), and flashing it and connecting stuff to it is a lot simpler.

OpenMQTTGateway Configuration

To manually configure the network and MQTT I added the following to User_config.h:

#define ESPWifiManualSetup true
#define MQTT_SERVER "your.server"
#define MQTT_USER "user"
#define MQTT_PASS "pass"
#define wifi_ssid "ssid"
#define wifi_password "pass"
#define SECURE_CONNECTION
#define MQTT_PORT "9443"
const char* certificate CERT_ATTRIBUTE = R"EOF("
-----BEGIN CERTIFICATE-----
... // your root certificate: see here for let`s encypt https://letsencrypt.org/certificates/
-----END CERTIFICATE-----
")EOF";

Collecting the metrics

To collect and view the sensor data I'm using Mosquitto, Prometheus, and Grafana. For Prometheus to be able to collect the data I'm running an exporter that reads the data from the queue and provides an HTTP endpoint in the format Prometheus expects.

References

🔗 OpenMQTTGateway

🔗 AT instruction set