Forem: Get DevOps Pro

Building a K3s Cluster with Armbian on Orange Pi 5 Plus: A Step-by-Step Guide

Job Céspedes Ortiz — Mon, 13 Nov 2023 15:47:24 +0000

In this blog post, I'll be sharing my hands-on experience in building a K3s Cluster on the Orange Pi 5 Plus (opi5+) using Armbian as the Operating System (OS) and Ansible. The post is meant as a straightforward guide for anyone looking to replicate the process. I hope it proves helpful for your own setup.

Please consider that in some sections, there are references to external guides containing the respective steps.

Why Armbian?

After my failed attempt using Fedora CoreOS, Armbian + K3s was one of the options I considered. Among the reasons behind selecting Armbian as the OS, there are:

Wide Hardware Support: Armbian provides support for a wide range of ARM-based single-board computers (SBCs), giving you flexibility in choosing hardware for your K3s cluster.
Stability and Reliability: Armbian is known for its stability and reliability, making it a suitable choice for building robust and resilient K3s clusters.
Optimized for ARM Architecture: Armbian is optimized specifically for ARM architecture, ensuring efficient performance on ARM-based devices commonly used for SBCs.
Regular Updates: Armbian releases regular updates and security patches, helping you keep your system up-to-date and secure.
Low Resource Footprint: Armbian is designed to be resource-efficient, making it suitable for resource-constrained environments common in SBCs, which is ideal for setting up cost-effective K3s clusters.
Community Support: Being an open-source project, Armbian has an active community that provides support, documentation, and troubleshooting assistance, which can be valuable when setting up and maintaining a K3s cluster.

Warning

There are a couple of things you should be aware of:

This is a laboratory environment.
Some steps involve commands that will erase data.

Assumptions

The following assumptions have been made:

Device is Orange Pi 5+ version.
There is another laptop/PC for preparations.

Preparations

Items

The items used for this task are:

1x SD Card with at least 4GB
3x Orange Pi5 Plus 16G + 256G EMMC Module with US 5V4A Type C Power
3x Geekworm Orange Pi 5 Plus Cluster Acrylic Case with Fan and Heatsink Kit. Please note:
- The included heatsink may not fit perfectly and might require some force to install.
- One of the screws on the fan may touch a clip on the heatsink and lift the top cover on one side. To address this, you can remove that specific fan screw, leaving only 3 of the 4, to prevent any unevenness in the top stack.
- There is no step-by-step manual or video guide available.
1x HORACO 8 Port 2.5G Ethernet Switch

Other items

Two additional items were acquired, specifically for a Linux storage server and not intended for the K3s cluster. Below are the details of these items in case they serve as a reference. Kindly note that they are not meant or required for the K3s Cluster:

1x 25G SFP28 SFP+ DAC Cable - 25GBASE-CR SFP28 to SFP28 Passive Direct Attach Copper.
1x ConnectX-4 Lx EN CX4121A-ACAT 2-Port 10/25GbE SFP28 PCI-E

Install Armbian

To install Armbian, follow Armbian Quickstart. The steps include:

Downloading Armbian image for opi5+
Installing Armbian image in the SD Card
Booting from SD card, do basic config, and install image in opi5+ eMMC with: armbian-install
Powering off
Removing SD card
Booting into eMMC and Armbian

Other Basic Config

Remove the SD card

Boot into eMMC and configure basic settings before proceeding further. Access using SSH keys without asking for a password is required for installing K3s using ansible. The other steps are optional:

Get root access
Change root password
Change user password
Upgrade OS and packages
Allow sudo without password
Allow auth using SSH keys
Customize /etc/hosts
Create network teaming/bonding

Below are the commands to perform the steps listed above. Change variable values according to your context. Replicate them in each of the other devices. Remember you need root access:

## as root.
# sudo bash

## change passwords
# passwd root
# passwd ${opi_hostname}

## upgrade pkgs
apt update
DEBIAN_FRONTEND=noninteractive apt upgrade -y

## vars /CHANGE_THEM
opi_username=changeme
opi_hostname=changeme-01
opi_domain=change.me
opi_fqdn=${opi_hostname}.${opi_domain}
opi_cidr=192.168.1.10/24
opi_gw=192.168.1.1
opi_dns='8.8.8.8 8.8.4.4'
opi_ssh_authorized_key='ssh-rsa CHANGEME'

## sudo
cat << _EOF > /etc/sudoers.d/10-opi5-user
# User rules for ${opi_username}
${opi_username} ALL=(ALL) NOPASSWD:ALL
_EOF

## ssh
mkdir -p /home/${opi_username}/.ssh
cat << _EOF > /home/${opi_username}/.ssh/authorized_keys
ssh-rsa ${opi_ssh_authorized_key}
_EOF
chmod 0700 -R /home/${opi_username}/.ssh
chown ${opi_username}:${opi_username} -R /home/${opi_username}/.ssh
sed -i "s@#PasswordAuthentication.*@PasswordAuthentication no@" \
  /etc/ssh/sshd_config
systemctl restart ssh

## /etc/hosts
cat << _EOF > /etc/hosts
127.0.0.1   localhost
127.0.1.1   ${opi_fqdn} ${opi_hostname}
::1         localhost ip6-localhost ip6-loopback ${opi_fqdn} ${opi_hostname}
fe00::0     ip6-localnet
ff00::0     ip6-mcastprefix
ff02::1     ip6-allnodes
ff02::2     ip6-allrouters
_EOF

## hostname
hostnamectl set-hostname ${opi_hostname}

## lock root password
passwd -l root

## network bonding
# workaround: https://github.com/coreos/fedora-coreos-tracker/issues/919
cat << _EOF > /etc/systemd/network/98-bond-inherit-mac.link
[Match]
Type=bond

[Link]
MACAddressPolicy=none
_EOF

# add bond interface
# this could cause disconection
nmcli connection add type bond con-name bond0 ifname bond0 \
  bond.options "mode=balance-alb,miimon=1000" \
  ipv4.addresses "${opi_cidr}" ipv4.gateway "${opi_gw}" \
  ipv4.dns "${opi_dns}" ipv4.dns-search "${opi_domain}" \
  ipv4.method manual bond.options "mode=balance-alb,miimon=1000" \
  mtu 9000
# add current devices
nmcli connection add type ethernet slave-type bond \
  con-name bond0-port1 ifname enP3p49s0 master bond0 mtu 9000
nmcli connection add type ethernet slave-type bond \
  con-name bond0-port2 ifname enP4p65s0 master bond0 mtu 9000
# delete old connections
nmcli connection delete "Wired connection 1"
nmcli connection delete "Wired connection 2"

Install K3s

Now you are ready to install K3s, using Ansible and a K3s role. Perform the following steps on a system other than your Opi5+ devices:

Install Ansible and the K3s role.
- For installing Ansible, you could run pip install ansible
- For installing the K3s role, you could run ansible-galaxy role install xanmanning.k3s
Install a Highly Available (HA) K3s Cluster with embedded etcd, following these steps. They include:
- Creating an inventory with your opi5+ devices
- Creating a playbook, using that inventory and the K3s role
- Running the playbook

Enjoy!

To wrap it up, building the K3s Cluster on the Orange Pi 5 Plus was a hands-on experience. Now I will use it to develop and test Moodle™ instances managed by Krestomatio operators for Kubernetes. I hope this guide makes it easy for you to do the same. Good luck with your K3s Cluster setup on the Orange Pi 5 Plus, and I hope you too have fun!

Building a K3s Cluster with Fedora CoreOS on Orange Pi 5 Plus: Lessons from My Unsuccessful Attempt

Job Céspedes Ortiz — Mon, 13 Nov 2023 15:43:17 +0000

This blog post summarizes my unsuccessful attempt to build a K3s Cluster on the Orange Pi 5 Plus (opi5+) using Fedora CoreOS (FCOS). It also outlines the steps taken before arriving at that conclusion and presents a list of possible alternatives to achieve a similar result.

Why Fedora CoreOS?

The main reasons behind selecting FCOS as the Operating System (OS) were:

Automatic Updates: Fedora CoreOS follows an automated update model, and reboots can also be orchestrated using Zincati.
Immutable OS: The operating system is designed to be immutable, meaning that changes are made by replacing the entire OS image rather than modifying the existing system. This helps improve stability and reproducibility.
Container-Optimized: It is optimized for running containers and container orchestration platforms like Kubernetes. This makes it well-suited for modern cloud-native applications.
RPM-OSTree: Fedora CoreOS uses RPM-OSTree, which allows for atomic updates and rollbacks, making it easier to maintain system consistency and recover from issues.
Security: Frequent updates and immutability contribute to a more secure system. Security patches are quickly applied to minimize vulnerabilities.
Community, Ecosystem, and Open Source: Being part of the Fedora Project and Open Source, Fedora CoreOS benefits from a vibrant community and a rich ecosystem of tools and applications.
Don't start from scratch: As part of Krestomatio tools for automating infrastructure, a terraform module was developed to help the deployment of a K3s Cluster using FCOS. The module outputs ignitions (ign) files required, just by setting some variables.

Warning

There are a couple of things you should be aware of:

This is a laboratory environment.
Ensure that you use the correct official images and URL for your device model, where aplicable. For instance, the opi5+ images and urls differs from the opi5 ones.
Some steps involve commands that will erase data.

Assumptions

The following assumptions have been made:

Device is Orange Pi 5+ version.
There is another Fedora-based laptop/PC for preparations.

Preparations

Items

The items used for this task are:

1x SD Card with at least 4GB
1x USB Drive with at least 4GB
1x Orange Pi5 Plus 16G + 256G EMMC

SD Card

For the SD card:

Check SD card requirements in opi5+ manual
Install OS image, following the opi5 official docs for burning it to the SD.

USB Drive

In the USB drive;

Copy EDK2 latest release, naming it edk2-firmware.img
Copy FCOS image, generate it with the step below, naming it fcos.img

FCOS image

To prepare the FCOS image:

Create an ignition file and store it in /tmp/config.ign
Create FCOS image

# Create FCOS image
STREAM=next # CHANGEME: `stable` or `testing` or `next`
FCOS_IMG=fcos.img
IGN_FILE=/tmp/config.ign # CHANGEME
CONTAINER_BUILDER=podman # CHANGEME: `podman` or `docker`
sudo losetup -D
rm -f "${PWD}/${FCOS_IMG}"
truncate -s 3GB "${PWD}/${FCOS_IMG}"
sudo losetup -f -P "${PWD}/${FCOS_IMG}"
sudo $CONTAINER_BUILDER run --pull=always --privileged --rm \
  -v /dev:/dev -v /run/udev:/run/udev -v $IGN_FILE:$IGN_FILE \
  quay.io/coreos/coreos-installer:release \
  install --firstboot-args=console=tty0 \
  -p "metal" -a aarch64 -s $STREAM -i $IGN_FILE /dev/loop0
sudo sync
sudo losetup -D

First Boot

Once preparations are done, continue:

Insert and boot from SD Card
Insert USB drive and mount it to /mnt
Copy EDK2 to SPI, following steps below
Copy FCOS image to eMMC, following steps below
Power off
Remove SD Card
Boot from eMMC

EDK2 to SPI

To install EDK2 to SPI:

Erase SPI
Copy EDK2 to SPI

# erase SPI
sudo dd if=/dev/zero of=/dev/mtdblock0
sudo sync
# copy EDK2 to SPI
sudo dd if=edk2-firmware.img of=/dev/mtdblock0
sudo sync

FCOS image to eMMC

Identify eMMC device
Erase eMMC device
Copy FCOS image to eMMC

# identify emmc device. Ex: /dev/mmcblk1
ls /dev/mmcblk*boot0 | cut -c1-12
# erase
sudo dd bs=1M if=/dev/zero of=/dev/mmcblk1 count=1000 status=progress
sudo sync
# copy
sudo dd bs=1M if=fcos.img of=/dev/mmcblk1 count=1000 status=progress
sudo sync

Result

Remember to power off and remove the SD card. Then test booting from FCOS image in eMMC on the opi5+.

The generic FCOS image does not work out of the box. It throws the error Timed out wating for device for boot and root devices and then enters emergency mode. Attempts to run using F39 and F40 were made, with the same result though. I suspect that the eMMC is not being recognized. I did not proceed beyond this point. Instead, I began considering alternatives to achieve my goal of building a K3s cluster.

Alternatives

After this result, there are at least four (4) alternatives I am considering to build a K3s Cluster using opi5+ devices. Those are:

Use official or third party images + K3s
Use Ubuntu + Libvirt + Fedora CoreOS VM + K3s
Build a custom Fedora CoreOS, with Orange Pi linux Kernel + K3s
Wait for full RK3588 Mainline Kernel support

Running Virtual Machines on Orange Pi 5 with Libvirt and Ubuntu

Job Céspedes Ortiz — Thu, 09 Nov 2023 21:49:11 +0000

In this guide, we'll walk through the steps to install libvirt on the Single Board Computer (SBC) Orange Pi 5 (opi5) for running virtual machines (VMs). I've compiled these steps after dealing with the opi5 instructions, searching on the internet and communities, and my own experience. I hope this guide helps someone accomplish this task more quickly than I did initially.

Installing Orange Pi 5

The first step is to install an Operating System (OS) on the opi5. You have several options here: you can use official images, third-party images, or build your own. In this case, I used a third-party image, Ubuntu. Additionally, I used a PCIe NVMe SSD for storage. Here are the installation steps:

Install Ubuntu on the NVMe, following the opi5 official docs for burning it to the SSD.
Apply basic configurations, including:
- Network
- Locale
- Keyboard
Perform a package update/upgrade.

Installing Libvirt

To install libvirt, you will need to install some other related packages and add a user to the libvirt group. You may also consider applying a workaround for running UEFI VMs. Afterward, you can easily test with a Cirros VM. Follow these steps to accomplish this.

## `--no-install-recommends` for avoiding recommended pkgs in a server
## alternative or additional pkgs
## `qemu-system`  for all qemu architecture or
## `qemu-efi-arm` for arm 32bits efi
## `u-boot-qemu` if planning to use uboot
# install pkgs in ubuntu 22.04
sudo apt install --no-install-recommends libvirt-daemon \
  libvirt-daemon-system libvirt-clients qemu-kvm qemu-system-arm \
  qemu-utils qemu-efi-aarch64 qemu-efi-arm arm-trusted-firmware \
  seabios bridge-utils virtinst dnsmasq-base ipxe-qemu

# add user to libvirt group
sudo adduser $USER libvirt
newgrp libvirt
export LIBVIRT_DEFAULT_URI=qemu:///system

# workaround
# set `60-edk2-aarch64.json` as the default uefi configuration
# using a symlink to place the descritor file first
# https://bugzilla.redhat.com/show_bug.cgi?id=1564270
sudo ln -s  /usr/share/qemu/firmware/60-edk2-aarch64.json \
  /usr/share/qemu/firmware/00-edk2-aarch64.json

# test cirros VM
## download cirros image
sudo wget http://download.cirros-cloud.net/0.5.2/cirros-0.5.2-aarch64-disk.img \
  -P /var/lib/libvirt/images
## create root qcow2 from image
sudo qemu-img create -b /var/lib/libvirt/images/cirros-0.5.2-aarch64-disk.img \
  -F qcow2 -f qcow2 /var/lib/libvirt/images/test.qcow2
## autostart default net
virsh net-autostart --network default
virsh net-start default
## install test VM in ubuntu host
virt-install -n test --memory 1024 --arch aarch64 --vcpus 1 \
  --disk /var/lib/libvirt/images/test.qcow2,device=disk,bus=virtio \
  --os-variant=cirros0.5.2 \
  --nographic \
  --boot loader=/usr/share/AAVMF/AAVMF_CODE.fd,loader.readonly=yes,loader.type=pflash

On Debian?

I attempted to install libvirt on a Debian image on the opi5, but I was not successful. I subsequently switched to Ubuntu. Here are the steps that I used to install libvirt on Debian, in case anyone wants to explore that route.

sudo apt install --no-install-recommends qemu-system-arm libvirt-clients \
  libvirt-daemon-system bridge-utils virtinst libvirt-daemon qemu-utils \
  qemu-efi-aarch64

Have Fun

Originally, my goal with a Single Board Computer (SBC) was to unwind and take a break from the usual work routine. I opted for an opi5 in the way. However, lo and behold, here I am, enabling virtualization, almost like it's a rarity in my line of work over at Krestomatio! It's been a fun experience, nonetheless.