Forem: Hung Vu

Guide: Update NVM firmware on Intel i226 and Intel i225 Ethernet controllers

Hung Vu — Sun, 01 Feb 2026 17:20:35 +0000

The Intel i225 and i226 Ethernet controllers and their variants are widely known for their unreliable performance, namely random network disconnections, packet loss, and power saving incompatibility. Since the release back in Q4 of 2019, Intel unfortunately has not been able to fully resolve these various issues as far as I know. Some say the hardware itself is flawed, some says it is the firmware (non volatile memory - NVM) that needs to be updated, and some says it is the driver issue. In reality, it is probably a combination of all these factors.

In this article, I want to talk about the NVM firmware part. As of early 2026, the Intel i225 and i226 controllers are still widely used in many consumer devices, but Intel opted to not release the Intel i226 NVM firmware update tool for public download on their official website, while the Intel i225 counterpart is available. Instead, they only provide the firmware update tool to OEMs and system integrators. With that said, most users are stuck with the outdated NVM firmware that came with their devices. Still, there is a way to update the NVM firmware for these controllers, and I will show you how.

DISCLAIMER: Performing NVM firmware update carries inherent risks, including the possibility of bricking your device. Proceed with caution and at your own risk. Ensure you have a backup of important data and configurations before proceeding.

Prerequisites

Before we begin, make sure you have the following prerequisites:

A computer with an Intel i225-V or i226-V Lan on Motherboard (LOM) or PCIe Network Interface Card (NIC). Note, only the V variants are discussed in this article. Performing the NVM firmware update on non-V variants may brick your device.
A Windows operatiing system installed on the computer. Note, other operating systems should work too, but this guide focuses on Windows.
Administrative privileges on the Windows operating system.

Steps to update NVM firmware

Step 1: Download NVM binary files

Download and extract the NVM firmware binary from here (forked from BillyCurtis repository). As always, be cautious when downloading firmware files from third-party sources. For the purpose of this article, I store the extracted files in ~ (your home location).

Step 2: Download Intel driver and NVM update Tool

Download Intel Driver and NVM Update Tool from here. As of this writing, the package version is 30.6.

This package contains both the Intel Ethernet drivers and the NVM Update Tool. I recommend updating the Ethernet drivers to the latest version before proceeding with the NVM firmware update, unless you have a reason not to. This is to ensure you are not running an outdated driver or OEM customized driver that may interfere with the NVM update process.
The driver for i225-V and i226-V locates in ~\Release_30.6\PRO2500\Winx64\W11 folder inside the downloaded package.
There is no setup.exe file in the package, so you will need to manually update the driver via Device Manager.
To do this, open Device Manager, locate your Intel i225-V or i226-V Ethernet controller under "Network adapters", right-click on it, and select "Update driver". Then, choose "Browse my computer for drivers" and navigate to the folder mentioned above. Follow the prompts to complete the driver update.
If you have multiple Network Adapters showing up, make sure to update all of them.

Step 3: Understand NVM update tool structure

Extract the NVM Update Tool from the downloaded package. The tool is located in ~\Release_30.6\NVMUpdatePackage folder.

If you use Intel i225-V, navigate to ~\I225\I225_NVMUpdatePackage_v1_00_Windows folder. Intel provide a convenient exe file to perform the update. Run the file as Administrator and follow the prompts to update the NVM firmware.
If you use Intel i226-V, instead of running an exe file, you will extract it to find packaged binaries and a command line tool to perform the update. I personally use 7-Zip to extract the exe file. Continue to the next step.

Step 4: Confirm Ethernet controller details

Open Command Prompt as Administrator. To do this, search for "cmd" in the Start menu, right-click on "Command Prompt", and select "Run as administrator". Then, navigate to the ~\Release_30.6\NVMUpdatePackage\I225\I225_NVMUpdatePackage_v1_00_Windows\I225_NVMUpdatePackage_v1_00_Windows\I225\Winx64 folder. Copy the binaries extracted from step 4 to this folder. I choose I226-V\2.32 binaries because they are the latest as of this writing.

Let's pause here for a moment and ensure the following before proceeding:

You have administrative privileges in the Command Prompt.
You have navigated to the correct folder containing the NVM Update Tool. Either via cd command or by openning Command Prompt directly in that folder.
You have copied the correct NVM firmware binaries to this folder.
If all is good, continue to the next step.

Run the following command nvmupdatew64e.exe -i -l inventory.txt to list specification and status of all Intel Ethernet controllers on your system. This step is to ensure that the NVM Update Tool can detect your Intel i226-V controllers properly. Afterward, you will see an inventory.txt file generated in the same folder. Open the file to verify the information, which should look similar to the following in structure:

# Detail of each line may vary from device to device, but the structure remains the same.
[00:008:00:00]: Intel(R) Ethernet Controller I226-V
    Vendor                 : 8086
    Device                 : 125C  ===> Intel(R) Ethernet Controller I226-V
    Subvendor              : 8086
    Subdevice              : 0000
    Revision               : 4
    LAN MAC                : FFFFFFFFFFFF
    Alt MAC                : 000000000000
    SAN MAC                : FFFFFFFFFFFF
    ETrackId               : 8000028D ===> ETrack ID for 2MB NVM variant
    SerialNumber           : FFFFFFFFFFFFFFFF
    NVM Version            : 2.20(2.14) ===> 2.14 is the binary version
    PBA                    : FFFFFF-FFF
    VPD status             : Not set
    VPD size               : 0
    NVM update             : No config file entry
      checksum             : Valid
    OROM update            : No config file entry
      CIVD                 : 0.0.0
      EFI                  : 0.1.1, checksum None

This helps me confirm the NVM Update Tool is functioning correctly and can communicate with the Ethernet controllers. Also, my NIC is using an outdated version 2.14 firmware, and it is a 2MB variant as indicated by the ETrackId.

Step 5: Configure NVM update settings

Now, open nvmupdate.cfg file in a text editor (e.g., Notepad) to configure the NVM update settings. You will need to specify the path to the NVM firmware binary file you extracted in step 4. The configuration file should look similar to the following:

BEGIN DEVICE
DEVICENAME: FoxPond1_I225_15F2_2MB_1p94_800003BB
VENDOR: 8086
DEVICE: 15F2 ===> Intel(R) Ethernet Controller I225-LM
SUBVENDOR: 8086
SUBDEVICE: 0001
NVM IMAGE: FoxPond1_I225_15F2_2MB_1p94_800003BB.bin ===> Binary within the same directory
EEPID: 800003BB ===> ETrackID
EEPROM MAP: Foxpond_Map_File_v01.txt
RESET TYPE: REBOOT
REPLACES: 80000180 800003BB => Original ETrackIDs being replaced
END DEVICE

We can ignore what is originally included in the file because based on the language, it is unique to Intel i225-LM 2MB variant, and create a new entry at the bottom of the file as follows (remember to remove my comments after the ===> symbols and save the file):

BEGIN DEVICE
DEVICENAME: Intel(R) Ethernet Controller I226-V
VENDOR: 8086
DEVICE: 125C ===> Intel(R) Ethernet Controller I225-LM
SUBVENDOR: 8086
SUBDEVICE: 0000
NVM IMAGE: FXVL_125C_V_2MB_2.32.bin ===> Binary within the same directory
EEPID: 80000422 ===> ETrackId, but for version 2.32, 2MB variant
RESET TYPE: REBOOT
REPLACES: 8000028D ===> Original ETrackID being replaced
END DEVICE

Step 6: Perform NVM firmware update and verify result

Finally, run the following command to perform the NVM firmware update: nvmupdatew64e.exe -u -f nvmupdate.cfg -l update_log.txt. This command will initiate the update process using the configuration file you just edited. An update_log.txt file will be generated to log the update process. Monitor the Command Prompt for any prompts or messages during the update process. If there is no message, check the update_log.txt file for details. Once the update is complete, you may need to reboot your computer for the changes to take effect. Below is an example of a successful update log:

... (truncated for brevity) ...

[00:013:00:00]: Intel(R) Ethernet Controller I226-V
Flash update started.
NVM verification started.
Shadow RAM verification started.
Shadow RAM verification finished.
Flash verification started.
Flash verification finished.
NVM verification finished.
Flash update successful.
Device update successful.

... (truncated for brevity) ...

A reboot is required to complete the update process.

If an error occurs during the update process, refer to the update_log.txt file for troubleshooting information. Common errors may include issues with the NVM firmware binary, incorrect configuration settings, or communication problems with the Ethernet controller. Below are some examples of error messages you may encounter:

Error: Config file line 11: File does not exist: 'FXVL_125C_V_2MB_2.32.bin'
Error: Cannot parse CONFIG VERSION parameter due to the mistakes in configuration file.
Error: Flash update failed.

After successfully updating the NVM firmware, you can verify the update by running the inventory command again: nvmupdatew64e.exe -i -l inventory_post_update.txt. Check the inventory_post_update.txt file to confirm that the NVM Version reflects the updated firmware version (e.g., 2.32).

That's it! You have successfully updated the NVM firmware for your Intel i225-V or i226-V Ethernet controller. Remember to monitor your network performance to see if the update resolves any issues you were experiencing. If problems persist, consider exploring other factors such as driver updates or hardware compatibility.

How was my experience after updating NVM firmware?

Well, hard to tell, but in a sense it got worse. Back in 2024, I was testing the Intel i226-V quad port NIC on my OpenWRT x86 setup. This means my NIC was using NVM version 2.14 at the time. Below was my observation at the time (copied from my homelab README).

It appears Intel I226-V is buggy with ASPM.

With an Intel I226-V NIC, the system boots really slow, and when ASPM is enforced via powertop, the system completely freezes because the CPU utilization gradually increase up to 100%. Probably, this is an upstream bug of igc. There seems to be many report on this matter. Meanwhile, Realtek RTL 8125BG works perfectly.

Currently, there is no available NVM update for Intel I226-V (only Intel I225 series), so not sure if an NVM update can resolve the problem or not.

Unrelated note: Normally, ethX is based on whichever NIC is electrically closet to the CPU. For example, eth0 is an integrated NIC, eth1 is a nic at PCIE slot 1. However, the I226-V nic messes up the order, and making itself the first set of interface, from eth0 to eth3, other NIC order is also changed too.

How about now after updating to NVM version 2.32? Installing the NIC makes my system experience a kernel panic and boot loop. I attached part of the kernel panic log below for reference. Looking at it, it seems the igc driver crashes. Is it a firmware issue? Driver issue (null pointer reference)? Hardware issue (PCIe Bus)? Not sure yet, but possibly a combination of all these factors again.

... (truncated for brevity) ...

[   22.283426] pcieport 0000:00:1c.4: AER: Uncorrectable (Non-Fatal) error message received from 0000:0a:00.0
[   22.2937641] igc 0000:0a:00.0: PCIe Bus Error: severity=Uncorrectable (Non-Fatal), type=Transaction Layer, (Requester ID)
[   22.3051211] igc 0000:0a:00.0: device [8086:125c] error status/mask=00004000/00000000
[   22.421718] genirq: Flags mismatch irq 145. 00200000 (eth2) us. 00200000 (eth2)
[   22.4297171] ------------[ cut here ]------------
[   22.434546] remove_proc_entry: removing non-empty directory 'irq/145', leaking at least 'eth2'
[   22.443341] WARNING: CPU: PID: 128 at

... (truncated for brevity) ...

[   27.347954] CR2: 0000000000000080
[   27.351617] ---[ end trace 0000000000000000 ]---
[   27.356573] RIP: 0010:0xffffffffa0286787 [igc@00000000ae720c49+0x14000]
[   27.363497] Code: 89 f7 c6 07 00 0f 1f 00 8b b3 b8 fe ff ff 31 c9 31 ff 85 f6 7e 5a 49 8b 84 24 a8 00 00 00 48 8b 94 cb c0 fe ff ff a8 04 74 35 <0f> b7 82 80 00 00 00 44 0f b7 82 82 00 00 00 41 89 f9 0f b7 72 44
[   27.382861] RSP: 0018:ffffc90001b23e20 EFLAGS: 00010202
[   27.388426] RAX: 0000000000000007 RBX: ffff888102044b68 RCX: 0000000000000000
[   27.395875] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
[   27.403316] RBP: ffffc90001b23e50 R08: ffff88810235aec0 R09: ffff88810004cac0
[   27.410757] R10: 0000000000000007 R11: 0000000000000000 R12: ffff888102044000
[   27.418208] R13: ffff8881020449c0 R14: ffff888102044bf8 R15: ffff888102044b70
[   27.425640] FS:  0000000000000000(0000) GS:ffff88846f600000(0000) knlGS:0000000000000000
[   27.434021] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   27.440075] CR2: 0000000000000080 CR3: 0000000002230004 CR4: 0000000000370ef0
[   27.447508] Kernel panic - not syncing: Fatal exception
[   27.453116] Kernel Offset: disabled
[   27.456937] Rebooting in 3 seconds..

I'm using OpenWRT 25.12.0-rc1 at the moment, and my environment has changed a lot since an intital test back in 2024. I have not investigated further nor attempting to reproduce the issue using my 2024 configuration. So for now, I would say the NVM firmware update did not help much in my case. Your mileage may vary.

Wrap up

Updating the NVM firmware for Intel i225 and i226 Ethernet controllers can potentially improve performance and resolve certain issues. However, it is essential to proceed with caution due to the inherent risks involved in firmware updates. Always ensure you have a backup of important data and configurations before proceeding with the update process. If you encounter any issues during or after the update, refer to the logs for troubleshooting information and consider exploring other factors such as driver updates or hardware compatibility.

If you find this article to be helpful, I have some more for you!

And, let's connect!

Eliminate IPv4 tax on AWS, is it that easy?

Hung Vu — Fri, 03 May 2024 15:39:49 +0000

Starting from February 1st, 2024, Amazon Web Services (AWS) starts charging public IPv4 fees at the rate of $0.005 per IP per hour, or $3.6 per IP per month. Not a lot to me, but considering my AWS usage only has been around $6 per month, that is a 60% increase in price. How to cut costs? That was when I looked into public IPv6, which is free on AWS.

This article is about my experience migrating hungvu.tech to IPv6 on AWS. Although this is not a tutorial, it can give you some insight into the findings during the migration. For the infrastructure as code, you can visit my GitHub repository.

Which parts of my technology stack are affected by the IPv6 migration?

All of them.

It was just switching a public IP, how hard could it be? Or so I thought. Unlike IPv4 which is universally supported, IPv6 support is still limited at many levels even though the protocol itself was introduced and standardized years ago. It is not just a software problem but is also an infrastructure issue. I like to loosely conceptualize my findings based on the OSI model, which includes 7 different layers of abstractions in a network stack. I simplified the abstraction quite a bit though, from 7 layers to just "hardware, network" and "software". The below is how I associate my technology stack.

AWS (hardware, network, software)
My homelab (hardware, network)
Next.js, Payload CMS (Express), Node.js (software)
Caddy via Caddy Docker Proxy (network).
Mongo DB Atlas, ghcr.io, and GitHub (network).
Cloudflare (network, and software).

Is my homelab network IPv6 compatible?

Because my AWS infrastructure will soon utilize a public IPv6 address, I must be able to connect to AWS from my homelab. Technically, everything is accessible via web GUI, but that is only useful for an administration purpose. I also need to test and monitor my website from the lab, so without proper IPv6 support, that will not be possible. Luckily, my ISP (CenturyLink) does support IPv6, and with the help of an x86 OpenWRT router/firewall, my homelab infrastructure is dual-stack.

Does AWS support IPv6 across the services that I utilize?

I needed to check whether the AWS services I utilized supported IPv6 or not. It was funny to think about this I must say, like, have I ever needed to check whether a service supports IPv4 before?

EC2: I looked into AWS specifications. My instance is t4g.micro and it supports IPv6. In any case, it seems all instance types support IPv6 at the time of writing, so it should not be a problem.
VPC: It supports IPv6 in my region (US West - Oregon). AWS also offers a /56 CIDR block for IPv6, which will be helpful down the road because a smaller subnet can complicate the setup. I heard some other cloud providers offer a CIDR smaller than /100, which becomes an issue down the road when segmenting network and routing traffic.
EC2 Instance Connect: There are 2 different options, either an SSH connection via public IP, or an internal VPC network. At the time of writing, Instance Connect via public IP only supports IPv4. I think that is also the case for internal VPC network connections. However, my VPC still offers a private IPv4 address, so Instance Connect via the internal VPC network is usable.
API for IaaC: At least for Pulumi, it appears not all IPv6 settings are configurable. For example, I could not retrieve IPv6 automatically in the same way as IPv4 via Elastic IP, or I could not configure ICMPv6 ACL (a similar issue is reported in the Terraform repository).

If you are interested in how I implemented AWS IaaC via Pulumi, check out the hungvu.tech's repository.

Do Next.js, Payload CMS (Express.js), and Node.js offer IPv6 support?

I host both Next.js and Payload CMS (with Express.js underneath) on the same EC2 instance. They both support IPv6 out of the box. This means their server will listen and bind to an IPv6 address and its port. I am not too familiar with low-level networking programming, but I guess it is thanks to Node.js, which has supported IPv6 since the very beginning. Starting from Node.js v17 (released in 2021), IPv6 is even preferred in a dual-stack host, so I can say IPv6 is a first-class citizen here.

Can Caddy via Caddy Docker Proxy route IPv6 traffic?

Caddy as a reverse proxy works with IPv6 because the service simply binds to all interfaces by default, nothing to complain about here, but here comes the unexpected. Out of the box, Caddy automatically provisions TLS certificates from either Zero SSL (primary public ACME CA), or Let's Encrypt (fall-back public ACME CA). However, Zero SSL does not support IPv6 so all requests will fail. Technically, this is not an issue, because Let's Encrypt supports IPv6. However, somehow my deployment bugged out so it did not fall back to Let's Encrypt. It took a while for me to figure out what went wrong.

## AAAA records 
https://acme.zerossl.com/v2/DV90: Not found
https://acme-v02.api.letsencrypt.org/: 2606:4700:60:0:f53d:5624:85c7:3a2c

TLS provisioning issue aside, it appears Caddy Docker Proxy does not support IPv6 upstream out of the box. With the use of upstream, the Caddy Docker Proxy container will automatically detect the IP of containers in the same network (masquerade or not). However, I was not able to make it detect my services' IPv6 addresses. Eventually, I decided to just assign static IPv6 to each of my services and set up reverse_proxy accordingly. The docker-compose.yml implementation is available at hungvu.tech's GitHub repository.

How about MongoDB Atlas, ghcr.io, and GitHub?

In case you do not know, MongoDB Atlas is a managed MongoDB service, while ghcr.io is GitHub's container registry service. Guess what?

"Atlas does not support connectivity via IPv6 addresses at this time. "- MongoDB Atlas support's response to my question

They all do not support IPv6. For my MongoDB cluster, I chose AWS us-west (the same as my other services), so clearly, the infrastructure is IPv6 compatible. This means the lack of IPv6 support is on the MongoDB Atlas side. Meanwhile, GitHub is the largest code collaboration platform as of 2024, and IPv6 is still not there yet... But hey, GitHub is working on IPv6 enablement now, so maybe it will be ready by the end of this year.

I do not want to move away from MongoDB Atlas right now, so I self-hosted the database solution instead. For the container registry though, I simply modified my GitHub Actions workflow and pointed the publish to Docker Hub, which also just generally supported IPv6 in August 2023 (better late than never).

Cloudflare is not a bulletproof solution to migrate from IPv4 to IPv6

My journey started with this article from Cloudflare: Amazon’s $2bn IPv4 tax — and how you can avoid paying it. It is a great article to have me explore the IPv6 side of Cloudflare, but it does not give a full picture of how the IPv4 to IPv6 migration happens from start to finish, and how complicated the process can be.

Putting their article aside, Cloudflare has offered IPv6 support since 2011, so their IPv6 service and portfolio are way more than enough for my use case. Without their IPv6 compatibility service, it would potentially take more time for me to configure an equivalent NAT64 and further complicate the network stack. Most end users are still on IPv4 after all, and without proper IPv6 configuration on their end, they will not be able to access my website. Considering the IPv6 on the end-user side is out of my control, so it is better to leave Cloudflare, the perimeter, to do the hard labor here.

Conclusion: Did the IPv6 migration work out?

In the end, I was able to fully migrate hungvu.tech from IPv4 to IPv6. It took me more or less 2 days though, so do remember to carefully plan for your production migration. Unfortunately, I was not able to bear the fruit of this migration, because my EC2 instance does not have enough resources to self-host MongoDB. At the time of writing, my website is hosted on the homelab. Technically speaking, utilizing homelab is a win-win situation for me, but I digress, that is a story for another day.

Spending time learning more about IPv6 infrastructure and migration was a worthwhile experience for me, and I would suggest you try that out too!

If you find this article to be helpful, I have some more for you!

And, let's connect!

Expand Promox storage with external enclosures in a homelab environment

Hung Vu — Wed, 03 Apr 2024 04:00:11 +0000

This article was originally published at hungvu.tech - Can I use external SSD enclosures for Proxmox disks?

While experimenting with the storage configuration of my Proxmox node in my homelab, I wondered what the experience would be if I utilized my spare SSD-to-USB enclosures to expand storage. Normally, hypervisor products are appliances so the underlying OS is relatively locked down. Using external USB storage might not be possible with these appliances at all. However, Proxmox allows free root access to its Debian distro underneath, so I expect the drives to be mountable without any issues.

Part list

For internal storage, drives connect to the CPU through a SATA/SAS controller, motherboard chipset, or direct PCIe lanes. However, for external SSD-to-USB enclosures, the drives connect to the SATA/NVMe-to-USB controller within the SSD enclosure, and then to the USB hub controller of the server. Regardless of which SSD enclosure you purchase, its SATA/NVMe-to-USB controller likely comes from either ASMedia or Realtek. Therefore, I decided it would be interesting to test out both. For this experiment, I used:

Alxum AX-S207A: An NVMe SSD enclosure with an ASMedia ASM2362 NVMe-to-USB controller (10 Gbps).
Orico AM2-C3-2N: An NVMe SSD enclosure with a Realtek RTL9210 NVMe-to-USB controller (10 Gbps). There is no suffix, indicating that it is neither RTL9210A nor RTL9210B on paper; the controller is simply RTL9210.

These SSD enclosures were directly attached to the back of my MSI B660M Mortar Wifi DDR4 motherboard (with an Intel I7 13700k CPU installed). The MSI motherboard features 5 Gbps, 10 Gbps, and 20 Gbps USB ports, but I only tested the 10 Gbps and 20 Gbps ports (or USB 3.2 Gen 2 and USB 3.2 Gen 2x2, according to these USB-IF naming conventions).

The three 10 Gbps USB ports are powered by the Genesys GL3590 USB hub controller and all share a single 10 Gbps uplink.
The one 20 Gbps port is powered by the B660 chipset, offering a full 20 Gbps uplink.

Last but not least, the SSDs I utilized were four Intel Optane P1600x 118 GB. Why? There was no specific reason; they were simply spare parts I had available.

Hardware Configuration

In my opinion, the controller (or the internals) is more important than the enclosure model, unless the vendor has somehow compromised the implementation. For technical clarity, I will refer to the enclosures by their associated controllers. Among the four Intel Optane SSDs, two were installed directly on the motherboard: one had access to the direct-to-CPU PCIe lane, and the other was routed via the B660 chipset. The remaining two SSDs were installed in SSD enclosures, providing a variety of configurations to test.

Configuration 1: Two Intel Optane P1600x 118 GB drives in ASM2362 enclosures, both using 10 Gbps USB ports.
Configuration 2: Similar to Configuration 1, but one enclosure connects to a 10 Gbps port and the other to a 20 Gbps port.
Configuration 3: One Intel Optane drive in an ASM2362 enclosure and the other in an RTL9210 enclosure, both using 10 Gbps ports.
Configuration 4: Similar to Configuration 3, but the ASM2362 enclosure connects to a 10 Gbps port and the RTL9210 to a 20 Gbps port.
Configuration 5: Similar to Configuration 3, but the ASM2362 enclosure connects to a 20 Gbps port and the RTL9210 to a 10 Gbps port.
Configuration 6: Two Intel Optane P1600x 118 GB drives in RTL9210 enclosures, both using 10 Gbps ports.
Configuration 7: Similar to Configuration 6, but one enclosure connects to a 10 Gbps port and the other to a 20 Gbps port.

Why so many test cases? Hardware can exhibit unique behaviors when interfaced with specific components. While protocols are standardized, eliminating most issues, there are anecdotes of hardware, like GPUs, exhibiting bugs when plugged into certain PCIe slots. Hence, I explored every possible combination in this experiment.

Software Configuration

The experiment was conducted on Proxmox v8.1.3. With four drives (two internal and two external), I opted to create a ZFS pool with one VDEV in RaidZ1. I chose RaidZ1, the least safe configuration, to observe behavior in potentially the worst-case scenario.

The point of failure would have been the external drives. These could fail at any moment, necessitating a time-consuming reconfiguration. Had I selected RaidZ2, which tolerates up to two failed drives within a VDEV, even if the external drives all failed, my pool would remain intact. Although this redundancy is theoretically advantageous, it could obscure minor errors that I aimed to explore in this experiment.

Previously, on a different system, I utilized two ASM2362 enclosures as a mirrored metadata VDEV in a TrueNAS Scale experiment, which proceeded without noticeable trivial bugs or performance degradation. Given that TrueNAS Scale is also Debian-based, like Proxmox, I surmised that Proxmox could similarly manage the external enclosures. Well, to some extent...

The results

Let's go over my observations.

ASMedia ASM2362 enclosures always error out and were not usable even after ASMedia firmware changes

This outcome was surprising to me, especially since these specific enclosures worked fine on a TrueNAS Scale system I had mentioned earlier. However, this discrepancy is understandable given that everything about the two experiments was different. My TrueNAS Scale experiment was conducted on a bare metal AMD system the same one discussed in virtualizing TrueNAS system, while the Proxmox experiment was carried out on an Intel system. Regardless, the error rendered the Disks page of Proxmox inaccessible, displaying a permanent spinning circle, and sometimes it even caused the PVE dashboard to crash or made the node fail to shut down, reboot, or boot (particularly at the initramfs stage).

If relevant, I also have a handy Windows To Go installation, which is only functional with the RTL9210 enclosure on the Intel system. Although the ASM236 enclosure is operational, the latency is extremely high—for example, it takes more than 10 minutes to boot and similarly over 10 minutes to reach the login screen. This behavior may be related to the USB errors observed in Proxmox, though I'm not certain.

USB enumeration, power cycle, address acceptance, and timeout errors (code -62, -71).

Unable to read config index 0 descriptor/start: -61 error.

Realtek RTL9210 enclosures were buggy but worked fine after a firmware change

Initially, my RTL9210 enclosures were operating on firmware version 4.30.23, built on 2022-07-19. Unfortunately, they did not perform well with Proxmox. Sometimes, the devices would disappear after a reboot, and at other times, they would generate numerous errors in dmesg. After conducting some research and coming across the Station-Driver forum discussion, I concluded that a firmware upgrade might resolve these issues. Consequently, I upgraded to version 1.32.87 (build date: 2023-08-29), specifically for the Realtek RTL9210B – note the suffix 'B'. This upgrade eliminated all the previously encountered buggy behaviors.

In the future, I may write an article about the firmware upgrade process, as it took me an hour or two to sift through Station-Driver forum discussions to grasp the procedure.

Error 5, type 2 with RTL9210 controller pre-firmware upgrade.

RTL 9210 enclosures experienced an over-current condition error on firmware v4.30.23.

Genesys GL3590 USB hub controller (10 Gbps port) cannot handle any two external enclosures at once.

When using two RTL9210 enclosures, having one plugged into a 10 Gbps port and the other into a 20 Gbps port allowed my ZFS pool to remain operational. This setup enabled my Windows 11 VM to achieve more than 30 days of uptime. However, attempting to connect both enclosures to two 10 Gbps ports led to issues. It appeared that the Genesys GL3590 USB hub controller struggled with this configuration, resulting in connection errors.

Regarding the Windows To Go installation previously mentioned, performance was significantly better when using the 20 Gbps port. This port benefits from a dedicated PCIe lane that facilitates direct communication with the CPU. In contrast, disk performance is severely degraded when utilizing the 10 Gbps port. While it's unclear whether this is a hardware or software issue, I think it is a hardware limitation, correlating with my observation in the Proxmox environment. The following are some of the errors observed when connecting two RTL9210 enclosures to the 10 Gbps ports.

Connect-debounced failed error on the Genesys GL3590 USB hub controller.

Various types of device, I/O, buffer, timeout errors when using two 10 Gbps ports at the same time.

If one external enclosure fails, it is best to have the failure happen on the 10 Gbps port

ZFS RaidZ1 can tolerate up to one drive failure. If a failure occurs, it is most likely to be in the external enclosures. However, more common than hardware issues are connection issues, such as loose USB connections. Therefore, I simulated a failure by simply unplugging one external enclosure. As a reminder, only the scenario in which one RTL9210 enclosure is plugged into a 10 Gbps port and the other into a 20 Gbps port worked. With that in mind, I noticed that if the failure occurs on a 10 Gbps USB port, my ZFS pool continues to perform as expected. Conversely, if the failure occurs on a 20 Gbps USB port, then random issues may arise.

The issues varied, but they resembled what was shown in other sections of this article. Interestingly, the likelihood of experiencing random issues when the enclosure is not plugged into the 20 Gbps port varies. If the enclosure is unplugged before booting, or during the shutdown/reboot process, the pool is likely to fail, leading to a system stuck in the shutdown/reboot/boot sequence. However, if the system is running and the enclosure is unplugged, sometimes my ZFS pool continues to operate, and sometimes it degrades immediately.

Failed to start and import ZFS pool due to drive failure.

Performance-wise, there was an inconsistency

Delving into storage performance is beyond my expertise, so drawing any definitive conclusions was challenging. From what I understand, virtualized storage performance can be influenced by multiple factors due to the various layers of abstraction and configuration inherent in virtualization. One observation I made was that performance fluctuated when the OS drive was under heavy use.

My OS drive offers around of usable space. When conducting tests with CrystalDiskMark at smaller test sizes (below 16 GB per run), performance appeared relatively stable. This stability is likely because Proxmox and ZFS were able to cache all the data, resulting in speeds of around 20 GB/s. However, when I increased the test size to 32 GB with 9 runs, performance inconsistencies began to emerge, occasionally causing the Windows 11 VM to freeze until the tests concluded. I believe these inconsistencies were due to a combination of factors, including hardware limitations (like the data access patterns to USB and bottlenecks) and software configurations (like storage tuning). Below are some images highlighting the observed inconsistencies. These runs were not consecutive, so will see differences in usable storage.

High performance with small test size.

Storage performance inconsistencies starts showing up at 32gb, 9 runs.

Low measured performance, possibly due to cache saturation and external SSD enclosures access, leading to OS freeze during the run.

Conclusion

Following this experiment, my conclusion is that utilizing external storage for a ZFS pool in Proxmox is indeed feasible. Would I recommend it? Not particularly for valuable data or in a production environment. However, in the context of a homelab—where the expectation is that things might break down eventually—it was an enjoyable experience to explore the possibility of expanding Proxmox storage with external SSD enclosures. I did not delve into and troubleshoot the underlying causes of errors, as my goal was merely to observe behaviors in this experiment. Perhaps I'll conduct a more thorough investigation in a future article. Until next time!

If you find this article to be helpful, I have some more for you!

And, let's connect!

My checklist for a production-ready website

Hung Vu — Sun, 31 Dec 2023 18:21:29 +0000

This article was originally published at: My checklist for bringing a website to production - hungvu.tech.

The article below is my personal note on what to consider before putting a website into production. It is certainly applicable to this website, hungvu.tech, but your mileage may vary depending on the requirements of your application.

Read the "production checklist" of the chosen technology stack documentation

Most likely, the documentation of the technology you use has a "production checklist" somewhere. Reading documentation is a very good starting point. Let's have a real example, the core technologies behind my website are Next.js, Payload CMS, MongoDB, Cloudflare, AWS, Docker, and Caddy. There are specific points to look out for.

What is the interaction between technologies if I combine them? For example,
What do I need to consider when hosting Next.js using a Docker container?
- What happens when I put the Payload CMS backend behind a layer 7 reverse proxy like Caddy?
- Do I need to connect Payload CMS to MongoDB in a specific way?
- What happens when running Caddy on an EC2 instance, with traffic proxied through Cloudflare?
What is the security consideration, considering that the application is facing the public Internet now?
What is the performance consideration? For example, hardware specification, traffic behavior, and more.

As you can see, these points are very high level, and as I think more about them, a lot more questions showed up and led me to some corners on GitHub discussion, community-specific platforms (like Payload CMS Discord server), and so on.

Now, let's review my personal checklist, in no particular order.

Metadata

Thinking about the word "front end", I mean it for both "front of the front end", and "front of the back end".

This blog is built with Next.js, which is capable of both delivering HTML/CSS/JS content and operating its own API routes. In this context, I say the HTML/CSS/JS part is the front end of API "front-end" routes, hence "front of the front end". Meanwhile, my back end, which Payload CMS powers, has its admin dashboard. I call this admin dashboard "front of the back end".

Every front end has its metadata information. Overall, they are used to support crawlers, and clients in understanding what the website is about. For a blog site, I need 4 types of metadata.

The first is represented by meta tags.

Charset
Viewport
Theme color
Description
Google site verification
Open graph (og)
Twitter

The second is other important tags inside the head element.

Title
Canonical link
Favicon

The third one is Schema.org JSON-LD. I actually question the usefulness of this feature, but if it is relatively easy to implement without a glaring downside, just give it a shot I guess? In Next.js, there is next-seo package that greatly simplifies the implementation of Schema.org JSON-LD.

The last one is not an HTML element, instead, it consists of special files that affect how the web crawlers see your page.

Robots.txt
Sitemap.xml
RSS.xml (my website currently does not have this, but may in the future)

Caching

In Next.js

The modern version of Next.js supports caching fetch requests, so that raises a few questions.

Which request should be cached?
What should be the cache duration?
Should the cache be ephemeral or persistent?

Keep in mind, as of Next.js v14.0.4, the cached response size limit per request is 2 MB. Exceeding the limit will error out the fetch request, hence making this feature useless (or at least, not intended for) large responses. Caching larger responses, in combination with Next.js images, is resource-intensive. At the moment, I cannot say if this is an error from my implementation or not, but I can say the caching mechanism did hammer down my CPU, RAM, and bottleneck IO throughput of my EC2 instance.

Besides, I make my Next.js cache ephemeral so that every time I update the web service image, the stale cache is purged when creating a new container, ensuring my website is always up to date.

In Cloudflare

There are many layers of caching in web development. Aside from the Next.js system, hungvu.tech uses Cloudflare (a content delivery network) as a second cache layer. This brings up certain questions.

Is Cloudflare caching at the moment? This can be confirmed by monitoring Cloudflare statistics or observing the cache header of the response. I recently noticed the cache hit rate of my blog suddenly dropped and am investigating the issue.
If an issue is observed in production, is it due to the Cloudflare cache? The issues are variable, from cross-origin resource sharing, and no response request, to broken layout (due to Cloudflare "optimization" like Rocket loader, auto minify), and more. In my experience, purging the cache does not always make the website up to date, it is better to enable Development Mode to completely bypass the Cloudflare cache layer to troubleshoot the issue in real-time.

Security

I'm not a security expert by any means, so I do not want to pretend I know the ins and outs of the items mentioned below. There are plenty of deep-dive articles out there such as those from Mozilla documentation, so this list is, well, just a checklist. It is best to search these keywords below in your technology stacks documentation, I bet they do have a section regarding reference implementation somewhere.

Access control list (ACL): This goes without saying. ACL exists at multiple layers.
- Software-wise, some notable ones are navigation route protection, API rules, identity-aware proxy, and identity access management.
- Hardware-wise, there are physical, network (North-South and East-West traffic), and host protection. Really, defense in depth is a very specialized topic and I am still learning more about it over time.
Content Security Policy (CSP): This helps the front end mitigate certain types of security issues.
Cross-origin resource request (CORS), same-origin policy (SOP), and cross-site request forgery (CSRF): Similar to CSP, these are presented to mitigate common attack vectors. These concepts are not the same though, but in my experience, they often belong to the same section in a documentation.
Rate limit, IP blocking, and CAPTCHA: These exist to mitigate bot traffic and API abuse issues. There are good bots like search engine bots, but the majority are bad bots that exist to exploit websites. Rate limit, IP blocking, and CAPTCHA are prominent ways to fight against bad bots. There are some things to consider though.
- Rate limit and IP block can be applied at different layers. In my case, Cloudflare, Caddy, and Payload CMS (with Express underneath) do have their own implementation for rate limit and IP blocking.
- IP block may not behave as expected when traffic is proxied through multiple layers such as Cloudflare and Caddy at hungvu.tech.
- CAPTCHA may affect real user experience and their privacy, so use it at your judgment. At the time of writing, my blog does not have CAPTCHA implemented.
Trust proxy: This in Express.js, and equivalent features in other frameworks, is used to specify configuration when the web server is behind a reverse proxy. The setting itself can have security implications.
Secure Socket Layer (SSL) and Transport Layer Security (TLS) for Hypertext Transfer Protocol Secure (HTTPS): SSL was deprecated about 2 decades ago, and its successor is TLS. That said, people still use both terms interchangeably nowadays, even in the Cloudflare dashboard, hence the confusion. Just keep in mind that even though they all use the keyword "SSL", products on the market right now are pretty much TLS only. Anyway, simple TLS and HTTPS are the main reasons why I use Caddy in the first place. There is one thing to consider.
- What does the encryption tunnel look like in a full round trip? In my case, a high-level view of traffic order is as follows: Client browser, then Cloudflare, then origin server (Caddy on port 443), then Next.js server, then Payload CMS server, and back. With my website, the priority is to have traffic between the client browser, Cloudflare, and the origin server fully encrypted. Internal traffic on the server is a different story though.

Performance

Caching itself should belong under this performance section, but I consider it important enough to be in a dedicated section. Now, performance is a measurable metric, and there are various aspects of an application that can be optimized using the said benchmark. The list below is just agnostic items that help improve performance on paper. Are they really useful? It is unknown until there is a benchmark.

HTTP/3 and QUIC: The third major version of the HTTP protocol, and is said to greatly improve performance along with many other improvements compared to HTTP/2. This is another reason why I use Caddy as a reverse proxy, as it supports HTTP/3 and QUIC protocols with just a little configuration. But, like SSL/TLS/HTTPS, there are a few things to consider.
- What does the request look like in a full round trip?
- What happens when one part of the chain is not HTTP/3 and QUIC supported? For encryption, I can say if a chain is broken, then that specific part is unencrypted and comes with security implications. For network protocol though, a benchmark will be helpful here.

My website 7-day netowkr statistics, monitored in Cloudflare dashboard.

Compression: I use the term compression loosely here, and below is what I mean
- Request/response compression: This category includes gzip, ztsd, and Brottli. They have reduced the size of the response, but I also need to consider how the compression is applied when there are many layers in-between (as discussed with HTTPS, and HTTP3). Besides, compression means increasing server load, so if not configured properly, the compression can slow down response time.
- Image optimization: Broadly said, this is static asset optimization, but when coming to images specifically, the two widely used compressed formats are AVIF and WebP.
Request and response size: This is an issue when the application overfetches. It may not be a glaring issue in a local development environment, but in production where applications are deployed to cloud services (or on-prem servers), it is a real deal.
- Bandwidth costs money, and latency can heavily affect performance.
- By default, applications and anywhere that requests and responses go through, have a limitation on request and response sizes. For example, Next.js fetch cache limit, and Payload CMS max allowed JSON body size are default to 2 MB. Without proper configuration to lift the limits, it can cause unexpected behaviors or outright crash the applications.
Frequency of requests and responses: In addition to the size mentioned above, frequency is also a factor.
- Frequent requests and responses mean increasing bandwidth usage.
- This can create an increased load on the host, leading to unexpected side effects. For example, the Payload CMS editor has an autosave feature and defaults at 800 milliseconds. The default is fine in the local development environment but breaks down in production. I changed the frequency to 15 seconds and issues are resolved.
Host machine architecture and specification: This is less of an issue for containerized applications as they can run on different CPU architectures. From my research, AWS's general purpose offer on ARM is more performant than x86 alternatives. Anyway, hungvu.tech is containerized on an EC2 t4g.micro instance (as nano lacks memory), which is an ARM platform powered by AWS Graviton2 processors. There are some required steps to create an ARM image though, but it is greatly simplified using Docker GitHub Actions.

Potential bugs in compiler/transpiler and bundler

Here is a thing, to make the application production-ready, the source code must go through a compilation process of some sort. In Next.js with TypeScript, the bundling and transpiling process are required. Going through the process means the source code is transformed and optimized by the framework bundler and transpiller. If you can fully control the process, great. In reality, these processes are abstracted away to improve developer experience, and that by itself is a double-edged sword.

What if the bundler and compiler/transpiler are "stealthily" broken?
What if the bundler and compiler/transpiler conflict with your code base?

Considering the criticality of these components, they are mostly stable, but there is always an exception. For example, the Next.js production build can potentially mess up the front-end layout due to unexpected CSS ordering. This has been an ongoing issue since 2020 and is not fixed as of the latest version (Next.js v14.0.4). References are as below.

Wrap up

So, the list above is what I consider before bringing the application to production. The items, I suppose are all over the place, from software layer to infrastructure consideration. That's why I created the article to consolidate my thoughts for future reference. Is it the best and most accurate list? Certainly not, because each project has its own goal and looks at matters differently, so the points above are mostly personal to me. I hope it is useful to you, but remember to treat the list with a grain of salt.

If you find this article to be helpful, I have some more for you!

And, let's connect!

TrueNAS virtualization using Harvester and Kubevirt

Hung Vu — Wed, 05 Jul 2023 15:42:02 +0000

Virtualizing TrueNAS has been a common theme in the homelab world for a long time. However, there are not many discussions on this topic with Harvester as a hypervisor, so I would like to go over my experience doing so in this article.

Virtual machine configuration

Planning

Harvester uses Kubevirt for its virtualization management, so the virtual machines are configured using a Kubevirt YAML template. Before creating a virtual machine, let's sketch out the configuration beforehand.

CPU

A general recommendation is to start with 2 vCPUs and scale them according to your workload and CPU model. In my case, I only use TrueNAS for archiving records, so 2 vCPUs of Ryzen 5 5600G are enough for me.
RAM

TrueNAS recommends 16 GB at minimum, and besides, ZFS loves RAM. I am not a ZFS expert by any means, but on experimenting with an 8 GB to 128 GB setup, I see no visible difference to my use case after 16 GB. I can technically run at 16 GB, but I decided to use 24 GB instead to give the machine some leeway.
Boot drives

Modern versions of TrueNAS (both Scale and Core) require more than 8 GB for a boot drive, 16 GB is a good starting point. For my installation, I use a 32 GB boot drive to give the OS some overhead, and also, create a second virtual disk to later form a mirror boot drive during TrueNAS installation.
Installer drive

TrueNAS ISO ranges about 2 GB or less depending on the version, 5 GB is a good starting point for an installer drive. Make this one as bootOrder: 1

so the virtual machine can boot into that right away. It actually does not need to be the first boot, but from my experience, sometimes this makes the virtual not able to detect the installer drive.
Network

For my use case, I bridge the virtual machine with a L2VlanNetwork so it can communicate with my infrastructure. TrueNAS works well with VirtIO, so use it to maximize the performance (usually 3 to 5 times better in throughput compared to emulation).
Bind the virtual machine to a specific node

TrueNAS, or more likely ZFS, does not like virtual IO (if at all). Therefore, PCI passthrough is required for all data drives. Enabling PCI passthrough in Harvester means your machine cannot perform a live migration, so you have to schedule the VM on a specific Harvester node.

Side note: There might be a strategy for falling back or migration between multiple virtual machines if you have identical nodes with PCI passthrough enabled, I am not certain how that work at the moment.
Boot loader mode

TrueNAS installation supports both Legacy BIOS and UEFI boot modes. By default, Harvester creates a virtual machine with Legacy BIOS with Secure Boot disabled. UEFI boot is an option but from what in my use case, it makes no difference, so use the Legacy BIOS option. Make sure to choose the correct installation type, or else, you will be greeted with Booting to hard disk stuck indefinitely.

YAML configuration

Configure the virtual machine via the Harvester web interface based on this specification, and the final result will resemble the following template.

Side note: This is from Edit YAML, not Edit Config > Edit as YAML. I am not certain about the relationship between those two but changes in one might not reflect in the other, even if they look almost identical.

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: network.harvesterhci.io/mgmt
                operator: In
                values:
                  - 'true'
  domain:
    cpu:
      cores: 2
      sockets: 1
      threads: 1
    devices:
      disks:
        - bootOrder: 1
          disk:
            bus: virtio
          name: disk-2
          # Note: serial number is defined
          serial: dae05b93-c3c4-4a3e-9ec0-5403f9e70494
        - bootOrder: 2
          disk:
            bus: virtio
          name: disk-1
          serial: ec0082c0-b2ec-4104-96d9-dec3562a6d06
        # Note: cloudinit is missing here
      hostDevices:
        # Note: PCI devices are passed through
        - deviceName: amd.com/500_SERIES_CHIPSET_USB_3_1_XHCI_CONTROLLER
          name: htpc-000016000
        - deviceName: amd.com/500_SERIES_CHIPSET_SATA_CONTROLLER
          name: htpc-000016001
        - deviceName: asmedia.com/1064
          name: htpc-000026000
      inputs:
        - bus: usb
          name: tablet
          type: tablet
      interfaces:
        - bridge: {}
          macAddress: f6:28:ac:32:c2:3a
          model: virtio
          name: br-vlan1-lan
    features:
      acpi:
        enabled: true
    machine:
      type: q35
    memory:
      guest: 24476Mi
    resources:
      limits:
        cpu: '2'
        memory: 24Gi
      requests:
        cpu: 125m
        memory: 16Gi
  evictionStrategy: LiveMigrate
  hostname: truenas-core
  networks:
    - multus:
        networkName: default/vlan1-lan
      name: br-vlan1-lan
  nodeSelector:
    kubernetes.io/hostname: htpc
  volumes:
    - name: disk-2
      persistentVolumeClaim:
        claimName: zfs-truenas-core-2
    - name: disk-1
      persistentVolumeClaim:
        claimName: zfs-truenas-core-1

Actually, this template is after I finish setting up TrueNAS, so it is a bit different than your brand new YAML template. Let's talk about the differences.

Missing installer , and cloudinit disks

When the TrueNAS installer runs, you will see 3 different disks: vtbd1, vtbd2, vtbd3. Aside from the installer, there should only be 2 drives for boot, which are vtbd1: zfs-truenas-core-1, and vtbd2: zfs-truenas-core-2. Meanwhile, the vtbd3: cloudinit, is hidden from Harvester web GUI but is visible in the YAML config. It is responsible for managing and applying a configuration in the first boot of the virtual machine.

After using vtbd1, and vtbd2 as boot drives, installer and cloudinit becomes redundant and can cause certain issues, so they are removed just in cases.

Drives have serial numbers.

The boot drives are virtual, so by default, they do not have associated serial numbers. TrueNAS uses serial numbers as a way to keep track of drives, so without the identifier, certain operations such as creating a new pool are not possible (at least via the web GUI), even if your pool does not consist of the unidentified drive.

For example, vtbd2 here is the cloudinit (number changes due to different boot environments). Before it is unmounted, a warning is shown in TrueNAS and that prevents pool creation. The same applies to the boot drives. Even though they do not appear in the warning if they do not have a unique serial number, no operation can be completed.

Harvester does not expose a mechanism to set a serial number for drives via web GUI, so I manually adjust the Kubevirt YAML config and add the serial property (with UUID). TrueNAS is perfectly happy with that.

disks:
  - bootOrder: 1
    disk:
      bus: virtio
    name: disk-2
    # Note: serial number is defined
    serial: dae05b93-c3c4-4a3e-9ec0-5403f9e70494
  - bootOrder: 2
    disk:
      bus: virtio
    name: disk-1
    serial: ec0082c0-b2ec-4104-96d9-dec3562a6d06

PCI device devices are passed through.

Depending on your hardware, the PCI passthrough experience might differ. I did go over this a bit in my previous article on deploying an OpenWRT firewall in Harvester. For my motherboard, which is MSI PRO B550M-VC WIFI, let's have a look at the block diagram below (from the motherboard manual).

I have to pass through the section circled in green so TrueNAS can see the drives. There are some rules when passing through PCI devices.

All devices in an IOMMU group must be passed through. Their group can be determined by looking at the device address. Generally, devices in the same domain:bus are in a group. However, it is best to look at /sys/kernel/iommu_groups/ (F12 to access Harvester's node console), to see which device belongs to which group. Many times, devices with different domain:bus addresses can be in the same group too.
It is not necessary to make all passed-through devices available to the virtual machine. Instead, some will even prevent the virtual machine to boot or causes random errors or behaviors. In extreme cases, a node failure may happen. I guess the extreme cases happen because an errored device overwhelms the logging and I/O of the node, hence node failure. Just make what is needed available by cherry-picking them.
All devices in the same domain:bus must be made available to the virtual machine, or else it cannot boot.

So with that, I choose the following devices (all in IOMMU group 12).

But only make 3 of them available to the virtual machine, because the other two PCI devices can make my node fail. The failure causes all sorts of issues like management address becoming unavailable, node not responding, and having to reboot, etc.

hostDevices:
  - deviceName: amd.com/500_SERIES_CHIPSET_USB_3_1_XHCI_CONTROLLER
    name: htpc-000016000
  - deviceName: amd.com/500_SERIES_CHIPSET_SATA_CONTROLLER
    name: htpc-000016001
  - deviceName: asmedia.com/1064
    name: htpc-000026000

It may have been easier if I use my IBM M5110/LSI 9207-8i (SAS 2308, IT mode), but it makes no difference in the end for me. My disks eventually look like this, and the performance at default is very reasonable considering the overhead of virtualization. I guess it can still be further improved.

Side note*: The test runs only a few seconds, hence the skewness in its statistics. That said, the max throughput can reach close to 9 Gbps at one point on a 5 x 2 TB SSD pool, RAID-Z2.*

Miscellaneous details about TrueNAS installation

Standard first-boot procedure

When I first log in to TrueNAS installation, there is a standard procedure I use to set the machine up.

Set static IP, DNS, and default gateway (required for Internet access).
Install the latest update.
Create a client account to access a network share.
Move system dataset to boot pool.
Enable 2FA.
For the SMART tests, schedule a weekly LONG test, and a daily SHORT test.
For snapshots, schedule a daily task.
For scrubbing, schedule a weekly task.
Create a pool, data set, set user permission (OPEN template), enable SMB service, and create a share.

The below are optional, because they are not critical in a homelab environment, but are generally a must in production.

Disable the root account and use an admin account instead.
If using SSH, harden SSH authentication by disabling password login and enabling the public key requirement.
Set up an encrypted email notification.
Granularly configure alert settings.
Configured a verified SSL connection.
Schedule a replication task for a different system.
Schedule a cloud sync task for an off-site backup (3-2-1 strategy at minimum).
Modify ACL for granular access permission.

"This system does not support virtualization" error

At Harvester v1.1.2, it seems virtual machines on some AMD platforms cannot detect nested virtualization. This might relate to Harvester issue #3900, but that issue is categorized as a user interface bug only. Besides, I can run virtual machines on my Ryzen 5600G node normally (with SVM, IOMMU, and SR-IOV enabled, while /sys/module/kvm_amd/parameters/nested returns 1).

Up to this point in the article, I have been generalizing the TrueNAS version because both Core and Scale should be relatively the same. In fact, my first try is to migrate my bare metal TrueNAS Scale instance. It has some virtual machines and apps, but the migration to Harvester virtual machine is not without an issue. After the migration, TrueNAS Scale virtual machines are now nested virtualization, and they simply break the web GUI operations.

Essentially, TrueNAS Scale virtual machines cannot work in my nested environment. When I want to do any operation via the web GUI, such as creating a new pool, or new data set, this error shows up and interrupts the operation. However, operations via shell are not affected, nor are any other underlying features. For example, I import the pool via shell, and network sharing performs without a hassle. Apps also start automatically and are accessible via LAN connection, but are not modifiable via TrueNAS Scale web GUI.

As now I use Harvester as a hypervisor type 1, I figure TrueNAS Scale is not the most optimal choice considering its bleeding edge nature. Therefore, I go back to TrueNAS Core, which has been battle-tested in the last decade and is widely considered a feature completed by this point. The TrueNAS Scale is only my playground, so there is no issue for me to completely wipe it off and move to the Core. I suppose a similar issue might raise again if I decide to enable virtualization on the current TrueNAS Core installation, so I leave it untouched.

Wrap up

Originally, I dedicate the whole Ryzen 5600G node to a TrueNAS Scale bare metal installation. It has no issue but is certainly not a good utilization of the hardware. The current configuration makes no difference, and besides, it helps me learn a lot more about configuring virtual environments.

This labwork also makes me recognize one important thing: "Searching for Harvester information is not always the way to go". This is because Harvester consists of many different components, such as the underlying Linux OS, Kubernetes, Kubevirt, Longhorn, and so on. When I face issues with virtual machines, it is better to search for the answer in Kubervirt context, like: "XYZ error in Kubevirt", not "XYZ error in Harvester".

That's all for now. I wonder what would be a fun topic to learn about next!

This article was originally published at Virtualize TrueNAS with Harvester (and Kubevirt for VM management).

If you find this article to be helpful, I have some more for you!

And, let's connect!

I virtualize OpenWRT in Harvester HCI, and it was a long story...

Hung Vu — Mon, 19 Jun 2023 12:00:00 +0000

In my previous article, Is Harvester a good hypervisor for a beginner? My hands-on experience, I tried out Harvester as a virtualization platform for my multi-node homelab. The experience was not flawless, but that is to be expected in a homelab environment. In this article, I will go over my experience setting up an OpenWRT firewall for my 1-gigabit fiber connection in a virtual machine (VM) in Harvester. There are some guides for configuring OpenWRT, but they are not full-on tutorials (many details were omitted). In a sense, this is more about my findings on working with Harvester.

OpenWRT, on an x86 machine?

Talking about firewalls with routing capabilities in the homelab scene, pfSense, and OPNSense are the usual recommendations. They are great products, and I also use pfSense in a production environment. That said, networking is not my specialty, nor the focus of my homelab. I only need a few features such as VLAN and QoS, pfSense, and OPNSense are practically overkill for me.

I had been using off-the-shelf OpenWRT-flashed routers for years. If the solution can put up a decent performance while maintaining up-to-date security guards, that is more than enough for me. OpenWRT fits the bill, and it is Linux-based, which I am more familiar with. Therefore, I looked into the idea of using OpenWRT on an x86 platform, and surprisingly, it is possible. The OpenWRT team also has a dedicated page for the x86 platform, which for some reason, I have not been aware of up till now. For the VM, I used openwrt-22.03.5-x86-64-generic-ext4-combined-efi.img, the latest version at the time of writing.

Configuring a VM for the firewall, a long story...

The YAML snippet below is my configs. Unnecessary information is removed to make it more concise

apiVersion: kubevirt.io/v1
kind: VirtualMachine
...  
spec:
    ...
    spec:
      affinity: {}
      domain:
        cpu:
          cores: 2
          sockets: 1
          threads: 1
        devices:
          disks:
            - bootOrder: 1
              disk:
                bus: virtio
              name: disk-0
            - disk:
                bus: virtio
              name: cloudinitdisk
          hostDevices:
            - deviceName: intel.com/ETHERNET_CONNECTION_5_I219LM
              name: harvester-ha2-0000001f6
            - deviceName: intel.com/200_SERIES_Z370_CHIPSET_FAMILY_SMBUS_CONTROLLER
              name: harvester-ha2-0000001f4
            - deviceName: intel.com/200_SERIES_PCH_LPC_CONTROLLER_Q270
              name: harvester-ha2-0000001f0
            - deviceName: realtek.com/RTL8125_2_5GBE_CONTROLLER
              name: harvester-ha2-000001000
          ...
          interfaces:
            - macAddress: 52:54:00:de:03:c1
              masquerade: {}
              model: virtio
              name: default
          ...
        machine:
          type: q35
        memory:
          guest: 924Mi
        resources:
          limits:
            cpu: '2'
            memory: 1Gi
          requests:
            cpu: 125m
            memory: 682Mi
      evictionStrategy: LiveMigrate
      hostname: openwrt
      networks:
        - name: default
          pod: {}
      nodeSelector:
        kubernetes.io/hostname: harvester-ha2
      volumes:
        - name: disk-0
          persistentVolumeClaim:
            claimName: openwrt-disk-0-e9h90
        - cloudInitNoCloud:
            networkDataSecretRef:
              name: openwrt-hilki
            secretRef:
              name: openwrt-hilki
          name: cloudinitdisk
status:
  conditions:
    ...
    - lastProbeTime: null
      lastTransitionTime: null
      message: VMI uses a PCI host devices
      reason: HostDeviceNotLiveMigratable
      status: 'False'
      type: LiveMigratable
...

Virtual hardware specification

Which node to run on?

My Harvester cluster consists of 3 nodes, in which the second node is a medium form factor computer with 3 network interface controllers (NIC), while the other nodes only have 1 NIC each. Technically, it is possible to set up OpenWRT with only 1 virtual NIC, shared between host and guest OS, and utilize the high availability (fallback) feature of a Harvester cluster, but I decided not to do so for now. Because this VM is for a firewall, it sorts of creates a chicken-and-egg issue, so using only 1 NIC will complicate the setup quite a bit since it involves configuring VLAN and virtual NIC for the VM. I may try that out down the road, but for now, running on the second node with PCIe passthrough is my choice.

CPU

For this OpenWRT VM, I decided to go with 2 virtual cores. Actually, I tried out with only 1 virtual core, but that made the LAN to WAN performance inconsistent. With 1 core, fresh installation, and no additional service enabled (e.g., AdBlock, QoS), the LAN to WAN throughput ranged from 400 Mbps to 900 Mbps inbound and outbound. With 2 cores enabled, the performance has been consistent at around 800 - 900 Mbps with many additional services enabled. One thing to keep in mind, my second node has an Intel i5-6500 CPU, which is not a powerhouse by any means, so the performance is strictly applied to this CPU only.

Memory

OpenWRT has a very small memory footprint and only consumes under 120 MB at fresh installation. That is understandable because OpenWRT is designed for consumer all-in-one devices with minimal hardware, it just so happens that the OS is compilable to x86 so here we are. Therefore, 1 GB of memory is more than enough for my use case. I probably will never consume over half of that though

Disk size

Depending on your use case, you may require more drive space than me (e.g., for logging). Generally, the same story as memory applies here. OpenWRT installation consumes around 128 MB, so 1 GB for the installation is overkill. By default, OpenWRT does not even use all available drive space, and users must expand the root partition manually. Not that I need all available space though, so I tried to create an under-1 GB volume, but Harvester does not allow that.

# 1 GB is the minimum disk size you can choose.
# Chossing a smaller value (e.g., 0.256 GB) causes the following error.
disk-0: Exceed maximum size 999999999 GiB!

NIC

The second node has 3 NICs: Intel I219-LM, Realtek RTL8125 (only 8125, neither rev. A nor B), and Intel X710 DA-2. The X710 is a management NIC, so I passed through the I219-LM and RTL8125. The x86 image supports both of these ethernet controllers out-of-the-box, separate driver installation is not necessary. It is also possible to utilize VLANs to set up WAN and LAN on 1 NIC, but that is a story for another day.

# RTL8125 uses kmod-r8169, and I219-LM uses kmod-e1000.
# The below are drivers bundled in the OpenWRT x86 image.
# Source: https://github.com/openwrt/openwrt/blob/3556455f040341099a6563968a6e9e8c22c0847e/target/linux/x86/image/64.mk#L6

DEVICE_PACKAGES += \
    kmod-amazon-ena kmod-amd-xgbe kmod-bnx2 kmod-e1000e kmod-e1000 \
    kmod-forcedeth kmod-fs-vfat kmod-igb kmod-igc kmod-ixgbe kmod-r8169 \
    kmod-tg3

Although there are stories of people having terrible experiences using these NICs in OpenWRT (e.g., packet drop, kernel panic, etc.), they had been working fine on bare metal for me, so virtualization should be the same, right? Well, I did hope for that..., but I eventually decided to swap the roles of X710 and RTL8125. Let's talk more about it below.

PCIe passthrough is straightforward but with caveats

Harvester offers PCIe passthrough out-of-the-box, but at the time of writing, this feature is still experimental. PCIe passthrough is accessible via Virtual machine config > PCI Devices web GUI, and the feature just takes a few clicks to enable (assuming your hardware supports that, e.g., VT-d, IOMMU, etc.) There is a technical difference between PCI and PCIe terminologies, but I will use the term PCIe devices in the following sections.

So, I went ahead and passed through my NICs without any issues, or at least, not yet...

RTL 8125 does not like being passed through in Harvester

At first sight, the RTL8125 NIC could be passed through easily, and indeed, it was. I could fully utilize it in my OpenWRT VM without any performance drop, hangs, or equivalent issues. Well, until I decided to reboot my VM.

In virtual machine events

Reason: Sync Failed

Resource: VirtualMachineInstance openwrt - Server error.command SyncVMI failed: "LibvirtError(Code=1, Domain=0, Message="internal error: Unknown PCI header type '127' for device '0000:01:00.0")"

A very cryptic error message, but in short, the VM was not bootable. This usually happens whenever I reboot the VM by any means (e.g., via OpenWRT GUI, power off and start in Harvester GUI, etc.) Alongside, there were all sorts of random problems when this error message appeared such as:

VM became unresponsive to actions (via web GUI) like power off, restart, delete, edit-config, and so on.
VM was indefinitely stuck at Starting state.
Node became unresponsive.

I was not able to reliably reproduce the issue, and still do not know the reasons behind that. From the research I made, all I can say is it has something to do with the virtual machine type (q35), its virtual BIOS, and the NIC itself is stuck in a certain state.

# dmesg on the second node.
vfio-pci 0000:01:00.0: Refused to change power state, currently in D3

Rebooting and unplugging the node (to discharge) a few times can resolve this. Technically, it is not a big deal if the VM stays online for a long period. However, I was in the tinkering phase, so the issue got on my nerves due to its random nature. That is why, I eventually turned RTL8125 into a management NIC, while dedicating the X710 to my VM. I had to reinstall Harvester, but at least, I have not faced the same problem since then.

But ... it did not stop there

I think if you have extensive knowledge of K8s, the following error messages tell something. To a newcomer like me, they are more or less comparable to reading an AES-256 encrypted message. I did spend a good amount of time searching for these error messages but yielded no results. I guess that is because these messages are somewhat specific to Harvester's usage of K8s. Considering there are not many online resources about the platform at the moment (relative to equivalent solutions like VM Ware or Proxmox), probably I need to dig even deeper to find out the answers.

Here are some generic errors that I faced when enabling PCIe passthrough. The same as the RTL8125 story mentioned above, the following errors randomly appear. Essentially, they prevent the VM from completing an action, such as starting, or stopping. They may go away after some time, or get stuck there indefinitely and require a node reboot. The error messages are notified via several ways, such as pop up, VM events, statuses, etc. I do not use a coding textbox below because they are very long messages, and I am not sure if it consists of many different ones. Harvester does not give any new lines though.

0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector.preemtion: 0/3 nodes are available: 3 Preemtion is not helpful for scheduling

Error: Stop virtual machine default/openwrt failed, Operation cannot be fulfilled on virtualmachine.kubevirt.io "openwrt": Halted does not support manual stop requests

FailedAttachVolume: Pod virt-launcher-openwrt AttachVolume.Attach failed for volume "...": rpc error: code = Internal desc = Bad response statusCode[500]. Status [500 Internal Server Error]. Body: [code=Server Error, detail=, message=unable to attach volume ... to harvester-ha2: node harvester-ha2 is not ready, couldn't attach volume ... from [http://longhorn-backend:9500/v1/volumes/...?action=attach\]

0/3 nodes are available: 1 Insufficient devices.kubevirt.io/kvm, 1 Insufficient devices.kubevirt.io/tun, 1 Insufficient devices.kubevirt.io/vhost-net, 1 insufficient intel.com/200_SERIES_PCH_LPC_CONTROLLER_Q270, 1 Insufficient intel.com/200_SERIES_Z370_CHIPSET_FAMILY_SMBUS_CONTROLLER, 1 Insufficient intel.com/ETHERNET_CONNECTION_5_I219LM, 1 Insufficient realtek.com/RTL8125_2_5GBE_CONTROLLER, 2 node(s) didn't match Pod's node affinity/selector.preemption: 0/3 nodes are available: 1 No preemption victims found for incoming pod, 2 Preemption is not helpful for scheduling.

0/3 nodes are available: 1 Insufficient CPU

Unable to retrieve container logs for containerd://...

All devices in the same IOMMU group must be passed through

This is not an issue with Harvester but is more about how the platform handles it. When passing through a PCIe device, all in the same IOMMU group must be passed through unless Access Control Service is enabled (which I do not have). It is not necessary to activate all passed-through devices though. In Harvester, there is no clear indicator for which IOMMU group a PCIe device belongs, but it can be determined using the Address column on the PCI Devices page.

The addresses follow a Bus, Device, Function (BDF notation) for PCI Domain, which consists of 3 parts: Domain (0000), Bus (00), Device (02), and Function (0). All devices that have the same Domain:Bus:Device sections, they are in the same IOMMU group. Another way to check is to look at a device name because Harvester by default uses a format of node name-address as a convention. For example, in the picture below, I passed through all devices in the group0000:00:1f .

Here is a thing, when a device is not passed through probably, there is no indicator. I faced three different scenarios that are related to this:

Devices stuck in In Progress state.
Devices are shown as Enabled, but in fact, are not visible in the VM.
And I simply forgot to pass through one device.

In these cases, the VM could not boot up, or bootable but stuck in Starting state and became unresponsive. There were no indicators, no logs, nothing. Admittedly, I only used web GUI, so maybe something could be seen if using the terminal. This is not a good user experience in my opinion.

On a side note, although I passed through 5 devices in this IOMMU group as shown in the picture, I must not activate the Power Management Controller (0000:00:1f.2). Activating that device always makes my VM not bootable and throws the following error message.

1. 0/4 nodes are available: 1 Insufficient intel.com/200_SERIES_Z370_CHIPSET_FAMILY_POWER_MANAGEMENT_CONTROLLER, 3 node(s) didn't match Pod's node affinity/selector.preemption: 0/4 nodes are available: 1 No preemption victims found for incoming pod, 3 Preemption is not helpful for scheduling

Setting up OpenWRT for the gigabit network

Alright, I have been focusing on Harvester up to this point, let's briefly shift the focus to my experience on configuring the OpenWRT for my 1-gigabit line. To do so, I accessed the VM via Harvester serial console and modify the network config file, then ran the command service network restart to apply changes. Note, it was restart in my case, as reload did not work.

# vim /etc/config/network

# These are default OpenWRT config, no need to modify
config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd6c:3930:3b2c::/48'

# Add your customize configs down below
config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'br-vlan.1'

# I could add eth3 to use as a backup NIC
# but I just did not do so
config device
        option type 'bridge'
        option name 'br-vlan'
        list ports 'eth2'
        option bridge_empty '1'

config bridge-vlan
        option device 'br-vlan'
        option vlan '1'
        list ports 'eth2:u*'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option ipaddr '10.10.10.1'
        option netmask '255.255.255.0'
        option ip6assign '60'

config interface 'wan'
        option type 'bridge'
        option proto 'pppoe'
        option username 'your-username'
        option password 'your-password'
        option device 'eth1.201'
        option ipv6 'auto'

So, what led me to perform these changes in the first place?

OpenWRT uses a classic unpredictable naming scheme (eth0, 1, 2, etc.) From my understanding, whichever is closer to the CPU goes first. With an educated guess, eth0 is a virtual NIC from Harvester, eth1 is an onboard NIC Intel I219-LM and eth2, eth3 is for X710-DA2. I used the I219-LM for WAN and one port of X710 for LAN.
You may wonder why I did not use two NICs of X710-DA2 for WAN and LAN. It is because I do not have a compatible SFP+ to multi-gig RJ45 NIC for the X710.

# X710-DA2 driver and firmware (NVM) blocks unbranded SFP modules.
# You will not see this issue before installing X710 driver though.
kern.err kernel: [  415.070583] i40e 0000:02:00.0: Rx/Tx is disabled on this device because an unsupported SFP module type was detected.
kern.err kernel: [  415.070831] i40e 0000:02:00.0: Refer to the Intel(R) Ethernet Adapters and Devices User Guide for a list of supported modules.

After this, my WAN was up and the OpenWRT instance had access to the Internet, but that was just the start. I ran the following commands right after.

# Retrieve package list from OpenWRT.
#
# The less packages (and size), the better
# because that reduce attack surface.
opkg update

# Install driver for x710-DA2.
# This allows eth2 and eth3 to appear.
opkg install kmod-i40e

# This package contains several security patches.
# E.g., Meltdown, Spectre, etc.
#
# There is an equivalent package for different platforms too.
opkg install intel-microcode

# This is for DNS logging, and can be substitue with tcpdump-mini.
opkg install tcpdump

# Network-wide adblocker, this utilize tcpdump.
#
# If a package name is prepended with luci-app
# it provides a web GUI.
opkg install luci-app-adblock

# OpenWRT on x86 does not have a firmware flash feature.
# In other words, an upgrade is a complete reinstallation
# of the OpenWRT image.
#
# The package below allows an in-place upgrade for OpenWRT
# while preserving packages and settings.
opkg install luci-app-attendedsysupgrade

# Network-wide DNS enforcer.
# Test config at: https://www.dnsleaktest.com/
opkg install luci-app-https-dns-proxy

# Mitigate bufferbloat.
# Test config at: https://www.waveform.com/tools/bufferbloat (or DSL report)
opkg install luci-app-sqm

These packages can be configured either via serial console, or via the web GUI, but I will skip over the tutorials for that. To be frank, I am more of a power user with basic domain-specific knowledge in this regard, meaning I am aware of which toggle to turn on, and which number to put in. However, I do not have a deep networking background to explain those options, hence my explanation might not be the best. If you want to learn more, the OpenWRT documentation is a good starting point.

The result

There is a saying, a picture says a thousand words. Here are some tests I performed on the OpenWRT VM.

Web GUI, notice the system information?

Waveform bufferbloat test result.

Cloudflare network test result

iperf3 from client to OpenWRT result

iperf3 loopback result

Conclusion

It was a wild ride but at last, I have been able to use this VM firewall full-time. When it runs, the performance is miles ahead compared to off-the-shelf routers, the only hiccups that I still occasionally face are random errors when rebooting the machine.

Again, this is not a huge deal breaker to me as I do not play around with networking that much in my homelab. Considering there is no measurable difference between a virtualized firewall with bare metal installation, it is certainly worth a try. Theoretically, there are security implications when virtualizing the firewall though, but I suppose I am not the best person to discuss that.

I am currently working on virtualizing TrueNAS and deploying websites on my Harvester cluster, they might be good ones to talk about in my future articles.

This article was originally published at: Deploy a virtualized OpenWRT firewall in Harvester, how did it go?.

If you find this article to be helpful, I have some more for you!

And, let's connect!

🦕 How to build a simple back end PoC with user notification using Novu, Amplication, and Discord?

Hung Vu — Sun, 11 Jun 2023 18:37:17 +0000

TL;DR

In this project, I’ll be showing you some unique ways to integrate Novu with Amplification, a BaaS platform. With it, I’ll show you some interesting ways to leverage Novu’s open-source notification platform.

Amping It Up to Eleven

Building the complex software of today’s world is no easy task. A recent trend to help developers create these new applications is low-code and no-code, tools that can generate code automatically.

Not only can these tools help non-developers create applications, they can also massively boost the productivity of a developer. These tools can help rapidly set up and create applications without needing to hand-code every aspect of an application.

In this article, I’ll be showing you how to leverage one of these tools, Amplification, alongside a powerful open-source notification platform in Novu.

This article was originally published at: Bootstrap a simple back end with a user notification feature using Novu and Amplication.

The Tech

If you already know what Novu and Amplication are, that's great! For those who don't, let me introduce you to the tip of the low-code solution iceberg.

What is Novu?

Novu is an MIT-licensed open-source notification infrastructure that empowers developers to send and manage notifications seamlessly across various channels and platforms. Its unified interface simplifies the process of delivering notifications through email, SMS, push notifications, and other channels.

Developed using cutting-edge technologies like Node.js, MongoDB, and Redis, Novu boasts exceptional scalability and reliability. With features such as notification templates and scheduled notifications that can be defined via the graphical user interface, Novu provides an exceptional developer experience even in a fast pace environment.

Using Novu can save developers considerable time and effort that would otherwise be spent building and managing a notification infrastructure. When your project scales and contains dozens of SDKs only for the messaging module, it can create unnecessary complexity in management. Without the use of Novu, you have to develop integration and interaction between different channels on your own.

On the theme of trying to speed up the development process while still maintaining the quality of your final product, backend-as-a-service (BaaS) is also a concept you must know about.

What is backend-as-a-service (BaaS)?

Backend-as-a-Service (BaaS) is a model that provides developers with a pre-built backend infrastructure to create multi-platform applications. Pre-built infrastructure is one thing, but you can also define your architecture on several aspects such as databases, and API access control lists via a friendly graphical user interface. By abstracting away the complexities of managing server-side infrastructure, BaaS platforms empower developers to focus on crafting exceptional user experiences on the front end.

BaaS providers offer an array of backend services such as user authentication, database management, file storage, ... and the list goes on. Whatever tasks are usually done on the backend, then a BaaS can achieve that in a low-code manner. These services can be easily integrated into an application using APIs or software development kits (SDKs).

BaaS is particularly valuable for startups and small businesses with limited resources that seek to develop applications quickly and efficiently. It eliminates the need for building and maintaining their backend infrastructure, which can be both costly and time-consuming.

Nevertheless, BaaS generally has some drawbacks, mainly flexibility and control of your codebase (e.g., for performance optimization). BaaS is an abstraction for barebone back-end development, so it is a trade-off. Also, using a BaaS platform could increase dependency on third-party services as most are cloud-based, meaning availability and low-level security control are not in your hand.

However, no need to be disappointed, because Amplication is here to resolve those drawbacks.

What is Amplication?

Amplication is a revolutionary Apache 2.0-licensed open-source platform that simplifies the development process by automating the creation of functional, human-readable, and editable services based on TypeScript and Node.js. By generating the boilerplate code, Amplication allows developers to focus on building their core product, rather than worrying about the underlying infrastructure. Besides, its efficient automation not only saves developers a considerable amount of time and effort but also allows them to make changes quickly and easily. This ensures that developers can respond to evolving needs and requirements promptly and efficiently.

Like many other BaaS products, Amplication offers features like data modeling, user authentication, and access control, Amplication offers a comprehensive solution for building web, mobile, and desktop applications and APIs. However, what makes Amplication truly exceptional and eliminate common BaaS drawback is its ability to produce code that is not only fully functional but also easily understandable and customizable, making it an excellent tool for developers of all skill levels.

Typically, most BaaS products are cloud-based, so when you develop software using their services, you are effectively locked in their wall-garden ecosystem and have little control of the underlying mechanism. In a sense, this is to be expected because BaaS is supposed to abstract away the technical parts. However, while Amplication is doing the same, the generated codebase and self-hosted options allow you to keep the whole infrastructure in your hand.

Integrate Amplication with Novu

Now, why is Novu needed, you may ask? Amplication is there to generate all of your backend infrastructure. With notification as a common feature in modern applications, it should be a part generated by Amplication, or is it?

What is a sample use case of Amplication?

Imagine a startup that specializes in selling sustainable fashion products online. As an e-commerce business, they need a reliable and scalable backend infrastructure to manage their product catalog, customer data, and orders. However, the startup's small team lacks the resources and expertise to build a custom backend infrastructure from scratch.

That's where Amplication comes in. Using Amplication's visual editor, the startup's development team can quickly generate a custom backend infrastructure that meets their specific needs. They can easily create services for product management, order processing, payment processing, and customer management. The codebase for this service is then generated and can be version controlled and synced to GitHub so the team has a formal commit tree to follow through, thus enabling rolling back the development when breaking bugs happen.

With Amplication, the startup can focus on building its storefront or social media outreach effort, knowing that the backend infrastructure is secure, scalable, and customizable. One thing to note, Amplication can generate most of the core codebase, that said, its scope (at default) is limited to:

Database (e.g., PostgreSQL, MongoDB)
Access control
API (REST, GraphQL)
Server (NestJS)
Admin dashboard (front-end for content managers)

Theoretically, the startup can leverage existing APIs (e.g., via plugins) to seamlessly integrate with popular shipping carriers, payment gateways, and inventory management systems. This integration saves the startup's development team time and resources while providing a seamless shopping experience for their customers. The thing is, not many plugins are available at the time of writing.

Considering the open-source nature of Amplication, more community plugins along with first-party solutions will become available over time. The startup needs a notification feature to let their customers know when new items arrive, but there is no notification plugin at the moment. Does it mean the startup has to wait for that specific plugin to come out? Not at all, for now, the startup can take advantage of the generated codebase and build a custom business logic using Novu, an open-source notification infrastructure.

What is custom code (and business logic) in Amplication?

In Amplication, there are two main layers of abstraction to have a separation of concerns.

One layer is dedicated to code generated by Amplication.
Another layer is dedicated to custom code so developers can customize apps and minimize merge conflicts. Technically said, the conflicts should not happen in the first place as two layers are isolated. However, as Amplication gets updated and breaking changes arrive, merging the code generated from different Amplication versions may require manual intervention.

When your application is generated, the folder structure looks like the below:

// Root
(config files)
prisma
scripts
src
- (NestJS project files/folders)
- (Entity folders)
  - base
  - <entity-name>.controller.ts
  - <entity-name>.module.ts
  - <entity-name>.resolver.ts
  - <entity-name>.service.ts

One layer of abstraction is in the base folder, and this is for files that are generated by Amplication. These files are not for customized code and will be overwritten by subsequent commits from Amplication. Meanwhile, the other layer of abstraction consists of files on the same level as the base folder. This means controller, module, resolver, and service are customizable and you can add more files or folders to the same directory as needed.

Being customizable, many categories of code can be implemented, such as:

An action when a request hits an API.
A query in a GraphQL resolver.
Business logic in service and responses.

Business logic in this context is an operation performed when a database query is invoked (e.g., send a notification when a new item is added to the database). The following is a generic sample of implementing custom business logic in Amplication, but in the next section, you will dive into the practical usage of this feature.

// Author: Hung Vu

// <entity-name>.service.ts
// Example of adding business logic to a service
import { Injectable } from "@nestjs/common";
import { PrismaService } from "../prisma/prisma.service";
import { SneakerServiceBase } from "./base/sneaker.service.base";

@Injectable()
export class SneakerService extends SneakerServiceBase {
  constructor(protected readonly prisma: PrismaService) {
    super(prisma);
  }
  async create(data: any) {
    // Add your custom business logic here
    notifyCustomersOnNewArrivals()

    return this.prisma.myModel.create({
      data,
    });
  }
}

// <entity-name>.controller.ts
// Example of adding business logic to a REST API response
import * as common from "@nestjs/common";
import * as swagger from "@nestjs/swagger";
import * as nestAccessControl from "nest-access-control";
import { ProductService } from "./product.service";
import { ProductControllerBase } from "./base/product.controller.base";

@swagger.ApiTags("products")
@common.Controller("products")
export class ProductController extends ProductControllerBase {
  constructor(
    protected readonly service: ProductService,
    @nestAccessControl.InjectRolesBuilder()
    protected readonly rolesBuilder: nestAccessControl.RolesBuilder,
  ) {
    super(service, rolesBuilder);
  }
  @Post()
  @HttpCode(200)
  async yourResponse() {
    // Add your custom business logic here
  }
}

Example: How to implement a Discord notification in Amplication using Novu?

In this section, you will explore the integration between Amplication and Novu via Discord webhooks. This tutorial is not intended to cover the entire process from start to finish, and it assumes a certain knowledge of the initial steps like installation and user interface navigation. Considering the introductory nature of the tutorial, you do not need to be an expert in Amplication, Novu, and Discord. If you have not tried them out, go ahead and spend like 30 minutes reading over the documentation. That should be more than sufficient before getting to this tutorial.

Define user story, a hypothetical scenario

Coming back to the mentioned example, let's say you are a fashion store owner. Your store has grown at a very decent rate in the past few years and now, you have a dedicated community on Discord. To increase customer experience, you want to have multi-channel notification features so customers know when new products arrive. Your website is powered by Amplication, and you have just learned about the unified API capabilities of Novu, an open-source notification infrastructure. This seems to be a good combination and with that in mind, your Discord community channel is the first group of customers to receive this update.

Create a Discord channel webhook

There are two main ways to integrate service with Discord: Webhooks and Discord Bots. At the time of writing, Novu only supports a webhook approach. To create a webhook:

Go to your Discord server and choose your targeted announcement channel.
Hover your mouse over that channel, then go to the Edit channel dashboard.
Navigate to Integrations, then choose Create Webhooks.
Your first webhook is automatically created, click on that webhook and Copy Webhook URL. Remember, Discord webhooks do not require authentication, so it is sensitive. Do not share the URL publicly.

Configure Discord message channel in Novu

Novu offers a variety of integrations in their store, and Discord is one of them.

Navigate to the Integration Store and activate Discord in the Chat section.
Navigate to the Notifications tab, and create a new template.
In the Edit Template dashboard, go to the Workflow editor.
After Trigger, add the Chat step. In the Edit Chat Template, add the following:A new product has just arrived, check it out at store.example.com!

Register a service user with Novu

Normally, Novu is integrated using the flow below.

Create a user in your user database (e.g., Amplication database).
Upon user creation, create a Novu subscriber using your user id and contact information. This subscriber acts as a service user for the Discord notification functionality.
Upon receiving a specific event on your backend, retrieve your targeted user(s) from the database.
Using the user id, trigger notification templates via Novu unified API. This whole step can be inside a loop.
Upon receiving a Trigger, Novu will use the provided user information, go over each workflow then send out a message dedicated to that user across different channels.

Although Discord is more of a group, non-personalized channel, there still needs an associated user in each trigger request. This user does not need to be in your store user database, it can be a pre-configured service user that you create in Novu at the beginning. To do so, fire up a temporary Node.js application and call Novu API.

// Author: Hung Vu
import { Novu } from '@novu/node';

const novu = new Novu(process.env.NOVU_API_KEY);

// The second parameter can be ignored as this
// service user is only there to trigger Discord notification.
await novu.subscribers.identify("discord");

Implement a notification business logic in Amplication backend

For demonstration purposes, Amplication provides a sample service for the e-commerce backend and it will be used in the tutorial. To start, navigate to the server folder and install Novu's SDK using:

npm install @novu/node

Then, navigate to src, create a global folder and put the following two files in there: notification.service.ts and notification.module.ts. In Nest.js, that is a way to define reusable modules which can then be imported across different services. In this case, we want to define a notification feature once and reuse it in different business logic.

/**
 * notification.service.ts
 *
 * Author: Hung Vu
 *
 * Discord notification service powered by Novu.
 */

import { Injectable } from "@nestjs/common";
import { ChatProviderIdEnum, Novu } from "@novu/node";

@Injectable()
export class NotificationService {
  novu = new Novu(process.env.NOVU_API_KEY!);
  notifyDiscordCommunity = async () => {
    try {
      // You may notice there is no credential configuration for Discord webhook in Novu dashboard, so this step is to assign a URL to your service user.
      await this.novu.subscribers.setCredentials(process.env.NOVU_SERVICE_USER_ID!, ChatProviderIdEnum.Discord, {
        // This is a Discord webhook URL retrieved in a very first step.
        webhookUrl: process.env.DISCORD_WEBHOOK!,
      });

      // Send Discord notification via a service user.
      await this.novu.trigger("discord", {
        to: {
          // This is a user ID created in a previous step.
          subscriberId: process.env.NOVU_SERVICE_USER_ID!,
        },
        payload: {},
      });
    } catch (err) {
      console.error(err);
    }
  };
}

/**
 * notification.module.ts
 *
 * Author: Hung Vu
 *
 * Export the service to use in other modules.
 */

import { Module } from "@nestjs/common";
import { NotificationService } from "./notification.service";

@Module({
  providers: [NotificationService],
  exports: [NotificationService],
})
export class NotificationModule {}

Now, let's revisit the project requirement one more time.

You want to have multi-channel notification features so customers know when new products arrive.

There are two parts to this statement:

Multi-channel notification: This is out of the scope of our tutorial, but with the unified API of Novu, you can continue to scale horizontally with the provided introductory codebase.
Let customers know when new products arrive: This means whenever you add a new product to the database, Novu is triggered and sends out a Discord notification. To implement this, you need to modify product.module.ts and product.controller.ts. The controller is only responsible for REST API requests. GraphQL-wise, product.resolver.ts needs to be modified, but the general concept is the same.

/**
 * product.module.ts
 *
 * Author: Hung Vu
 *
 * A central location to configure modules to be used in product endpoint.
 */
import { Module } from "@nestjs/common";
import { ProductModuleBase } from "./base/product.module.base";
import { ProductService } from "./product.service";
import { ProductController } from "./product.controller";
import { ProductResolver } from "./product.resolver";
import { NotificationModule } from "../global/notification.module";

@Module({
  // Import the Notification module, so it can be used
  // in the controller.
  imports: [ProductModuleBase, NotificationModule],
  controllers: [ProductController],
  providers: [ProductService, ProductResolver],
  exports: [ProductService],
})
export class ProductModule {}

/**
 * product.controller.ts
 *
 * Author: Hung Vu
 *
 * Controller for product endpoint, that invoke Novu notification workflow
 * when a new product is successfully added to the database.
 */
import * as common from "@nestjs/common";
import * as swagger from "@nestjs/swagger";
import * as nestAccessControl from "nest-access-control";
import { ProductService } from "./product.service";
import { ProductControllerBase } from "./base/product.controller.base";
import { NotificationService } from "../global/notification.service";
import { HttpCode, Post } from "@nestjs/common";

@swagger.ApiTags("products")
@common.Controller("products")
export class ProductController extends ProductControllerBase {
  constructor(
    protected readonly service: ProductService,
    @nestAccessControl.InjectRolesBuilder()
    protected readonly rolesBuilder: nestAccessControl.RolesBuilder,
    // Gain access to NotificationService
    protected notificationService: NotificationService
  ) {
    super(service, rolesBuilder);
  }

  // When a new product is successfully added to the database
  // via REST API POST request, HTTP status code 200 is returned
  // and fire a Discord notification workflow in Novu.
  @Post()
  @HttpCode(200)
  async notifyDiscordCommunity() {
    this.notificationService.notifyDiscordCommunity();
  }
}

And that is it, you have successfully integrated Amplication with Novu. With this, whenever a new item arrives, your beloved Discord community will be the first one to learn about it.

Wrap up

Depending on the scope of what you are trying to achieve, developing a production-ready application is not always a straight line to think of. That said, with the introduction of Amplication and Novu, both platforms excel in their own job and provide great solutions to speed up your development process while still maintaining a high-quality and human-readable codebase. All comes at no cost if self-hosted is your choice, but if you prefer delegating the hassle of managing bare metal infrastructure, both Amplication and Novu also provide cloud-based options.

This article is just the tip of the iceberg, it is highly recommended to take a look at the product documentation and see what they can do in the real world.

The repository for the codebase of this tutorial is available on GitHub, and Novu's team is always available in Discord to answer any of your questions, so check them out!

If you find this article to be helpful, I have some more for you!

And, let's connect!

🔧I tried out Harvester HCI for the homelab, and it is an interesting solution

Hung Vu — Thu, 11 May 2023 18:33:19 +0000

When I notice there are old computers collecting dust in my place, I ask about the possibility of repurposing them. With that in mind, I start building a homelab as a hobby. Each device can host one service, but I decide to virtualize my infrastructure to efficiently utilize every bit of hardware I have. There are popular solutions such as Proxmox, ESXI, and XCP-ng, but to make it more fun, I want to try out a more obscure solution. After looking around, Harvester, an open-source Hyperconverged Infrastructure, is the choice for my setup.

In this article, I will go over my very first experience with Harvester on a cluster of 3 management nodes. Keep in mind though, this article is more about documenting my experience, rather than being a proper tutorial, so please treat it with a grain of salt.

The hardware

When using a hypervisor type 1 such as Harvester, users should pay attention to the hardware of their infrastructure, because there are a plethora of compatibility issues that are lurking around, so what I experience may very well be isolated to my setup. Also, these are repurposed computers, so the specifications are nowhere near the recommendation from the Harvester team.

Node 1, an old laptop:

CPU: Intel I7-7700HQ
RAM: 16GB
NIC: Realtek RTL8111 (management NIC)
Storage: 1 TB NVME SSD (used for both Harvester and VM data)

Node 2, a medium form factor computer:

CPU: Intel I5-6500
RAM: 8GB
NIC 1: Intel I219-LM
NIC 2: Realtek RTL8125 (not rev. A or B, only 8125)
NIC 3: Intel X710 DA-2 (management NIC)
Storage: 128 GB SATA SSD (used for both Harvester and VM data)

Node 3, a mini computer:

CPU: Intel I5-6500T
RAM: 16GB
NIC: Intel I219-LM
Storage: 256 GN NVME SSD (used for both Harvester and VM data)

Harvester installation experience

The basic steps

The Harvester team has excellent documentation on initial installation, but being a geek, I decided to initially skip that to have a true first-hand experience. Overall, the installation process is fairly standard, but with some catches, more below.

Download the ISO: This confused me a little bit, because the very first "Get Started" on the Harvester website did not lead to an installation, but rather, it is a download button at the bottom of the page. Also, the switch leads to the Harvester GitHub repository, not a direct download.
Create a bootable USB drive using Rufus: There is not much about this step, because Rufus can burn the image without any issues.
Install Harvester in the other Node 1 > Node 2 > Node 3: Each node takes about 10 minutes to install. Perhaps there is a way to install them all at once (IaC?), if so, it will scale much better.

The weird first-login issue?

Here is a thing, although the image above shows the name, Rancher, I did not notice that, but bear with me and keep that in mind for a bit. When first installing Harvester, one step was to define a static VIP (Virtual IP for management). Well, it only shows an acronym and not knowing what that is, I essentially left it at default (DHCP). When the installation finished, I logged into the first node using its static node IP and was greeted with a login screen ... from Rancher?

I did not know about Rancher at that point, so here came another confusion. Later on, I also noticed that the branding of that page actually randomly changes between Harvester and Rancher branding when refreshed (F5). Also, it asks for a Bootstrap password and that is not the one I chose during the Harvester setup.

With this being nowhere documented, I dived into GitHub issues and found #1479. I learned the bootstrap password is admin, which then allowed me to log in.

And ... here is the Rancher dashboard. The best part is, I was not supposed to be here in the first place. This happened because I did not set up VIP for my cluster, so the Rancher dashboard and this whole weird first login behavior manifested. That said, there is no trivial way to modify management IP after Harvester installation, so I needed to reinstall the whole node. At the time of writing, I have an idea of Rancher's role already (albeit limited), but still, I could not find an explanation for the login behavior.

After setting up the VIP, I was able to create the first account and sign in to the Harvester dashboard (located at VIP) without an issue.

Harvester platform experience

Aside from the log-in issue that I mentioned, there were several pain points that I faced when trying out Harvester. Were they really pain points though? I'm not certain, and there is a chance they are intended behaviors. This is my first time trying out the platform after all. Anyway, here are some of them.

The failover functionality is clunky

With my setup, I have 3 nodes. Initially, only Node 1 is a management node, but after the other two nodes join the cluster, they were automatically promoted to a management one. Three nodes are required for the promotion to form a high-availability cluster. In a Harvester cluster of three, the fault tolerance is one. Well, the theory is that, but in practice, it is not as "high-availability" as I expected.

If node 1 is down, my cluster is effectively down (VIP - not ready; VMs, dashboard, and remaining nodes are not accessible).
From a power-off state, if only nodes 1 and 3 are turned on, then node 2 is cordoned. Nothing wrong there, but if I turn on node 2 afterward, then node 3 is cordoned. I have to reset node 3 so all are ready again. The same behavior applies when only nodes 1 and 2 are turned on.
From the power-off state, the nodes often cannot properly form a cluster after starting up (always not ready). This can be fixed with a few tries of rebooting the whole cluster (via console, F12).
When the cluster is working, but node 2 or 3 is cordoned, the whole cluster often becomes unstable. It takes ages to access the Harvester dashboard, VMs become unresponsive, and actions are stuck in in-progress. Resources (capacity) spike sky-high and are more or less stuck there. I guess this has something to do with live migration. If the node is online again, the resources will gradually decrease and the cluster is usable again. To emphasize, this behavior often happens, but not always. The picture below was when one node just came back from being cordoned.

And compared to after it got settled down.

Some essential processes are not trivial in Web UI, for example, the update process

There does not seem to be a Check for Update button. Only when a new release arrives, a small button appears in the top right corner of the Dashboard tab that provides the update functionality. The update is also very slow even with my 1 gigabit connection and SSD in the nodes. It took me about 90 minutes to update nodes 1 and 2, but node 3 never got updated. My cluster has been working normally under this condition though (not considering the issues mentioned in this article).

This issue can be generalized as a lack of essential UX components like an official way to power off or reboot the cluster, view SMART data for hosts, and more. They certainly can be done in a headless manner or directly using the node console, but those might not be user-friendliest depending on your preference.

System-wide logging capability is rather limited out of the box

When facing problems, my first thought was to look at the logs to troubleshoot. That's when I noticed it does not even exist in the admin dashboard. I mean something equivalent to syslog or dmesg to be specific, because Harvester does have monitoring and logging capabilities via Events, Metrics, or Grafana. The first two are rather limited, while Grafana has been hit-or-miss for me. I'm often not able to view data in Grafana for some reasons (as errors shown in the picture below, plus some other random ones).

# Some errors that prenvent me from using Grafana. They randomly happens.
Templating [job]
Error updating options: no elements in sequence

Templating
Templating init failed Couldn't find variable with id:instance

502 Bad Gateway

Harvester consumes a lot of resources

This by itself is not necessarily an issue, but when compared to other solutions like Proxmox or XCP-ng, Harvester does require a beefy device to run on. Considering it reserves about 6 cores on a 4-core 8-thread node, I suppose an 8-core 16-thread node is a better starting point, which is in line with the Harvester team recommendation.

Almost all my nodes are over-committed upon fresh installation (no VMs, or configuration changes), but the cluster is still fairly responsive under this condition. The downside is, it forces my devices to run at full power and inherently increases power consumption and heat dissipation. They are not deal breakers by any means though.

The OS is read-only, but still modifiable to a certain extent

Harvester is openSUSE-based, so I thought additional packages can be installed directly on nodes via SSH or direct console. However, Harvester enforces immutability to most locations by default.

rancher@harvester:~> sudo zypper install smartmontools
The target filesystem is mounted as read-only. Please make sure the target filesystem is writable.

However, there are a few possibilities to overcome this immutability:

The installation image can be customized.
Using cloud-config to apply boot-time changes.
Temporarily enable the read-write mode for the system.

I have not tried any of them, however, I needed to modify the system for one reason. My node 1 is a laptop, so when the lid is closed, the system will be suspended. I don't want to keep the lid open all day so I had to disable this suspension feature. Luckily, this was not restricted by the immutability of Harvester.

# Harvester only has Vim by default.
# This was done via direct console access on the laptop (F12).

# Step 1: Access logind.conf.
sudo vim /etc/systemd/logind.conf

# Step 2: Make sure the following rows use the "ignore" option, and remove the "#" sign.
HandleLidSwitch=ignore
HandleLidSwitchExternalPower=ignore
HandleLidSwitchDocked=ignore
LidSwitchIgnoreInhibited=no

# Step 3: Mask some system options.
sudo systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target

# Side note: There are many online discussions regarding on making the laptop not suspended when its lid is close. It seems the answer varies based on your device. On my laptop, the above sequence works, but may not be applicable to yours.

Wrap up

In this article, I went over multiple issues I face when trying out Harvester.

Were they annoying? Kind of.
Were they fun to go through? Yes!
Is Harvester a good option for a beginner? I think so.

At default, Harvester is fairly straightforward, but it certainly can become complicated on an established infrastructure. Getting my hands dirty is how I learn things over time, so for homelab purposes, it is a good choice. In the next article, I will go over my experience virtualizing the firewall on Harvester. Spoiler, it can handle my Gigabit connection easily, but with some catches. Stay tuned for that!

This article was first published at hungvu.tech: Is Harvester a good hypervisor for a beginner? My hands-on experience.

If you find this article to be helpful, I have some more for you!

And, let's connect!

🍂Simplify the back end with low code solutions: Amplication, Supabase, and Appwirte overview

Hung Vu — Wed, 19 Apr 2023 15:36:10 +0000

The modern web space is built up on time and resources of uncountable numbers of contributors over decades. There is a high chance that the tasks you are trying to achieve have been done by someone, somewhere.

When talking about websites, people think about the front end and the back end. Each has its own ecosystem and associate frameworks to build up the product, but you, as a developer, still need to tie everything together. This leads to a question, what if there is an existing product that helps you bind everything together out of the box? In the case of the back end, management systems such as headless content management systems, and learning management systems come into play, and back-end-as-a-service comes into play.

This article focuses on a comparison between back-end-as-a-service and headless content management systems. It is a continuation of the previous two that dived into headless CMS, so if you are not aware of them, go ahead and check them out!

What is a headless content management system (headless CMS)?

Imagine you are a creator, and you create content on a daily basis (e.g., similar to the article you are reading), at some points, you are going to have dozens if not hundreds, and more pieces in your pocket. To centrally manage them, you need a management system specifically designed for content in this case, and it is a headless CMS. In this case, the front-end team is responsible for the presentation layer, and they can retrieve information from the headless CMS to display, be it videos, images, text, and more.

Unlike traditional CMS systems that combine content creation, storage, and delivery into a single software, a headless CMS separates content management from content delivery. In other words, it provides an API for developers to access the content, which can then be delivered through any platform or device, such as websites, mobile apps, or IoT devices. With a single source of truth, developers have no need to manually modify content in their code, and effectively, create a separation of concerns.

This decoupling of the content management and delivery layers provides more flexibility and freedom to developers and content creators, allowing them to create and manage content independently of any specific presentation layer. It also enables developers to use their preferred front-end technologies and frameworks, giving them more control over the user interface and user experience.

What is Back-end-as-a-Service (BaaS)?

BaaS is a type of service that provides developers with a complete server-side infrastructure for building and deploying applications across multiple platforms (e.g., desktop apps, websites, and mobile apps). It allows developers to focus on the client-side development of their applications, while the BaaS takes care of the back-end infrastructure. Usually, BaaS is a cloud service, but as time goes on, more self-hosted solutions have appeared on the market.

A BaaS typically offers a set of pre-built services and features, such as user authentication, access control list, workflow, and most fundamentally, database management and creation. These services can be accessed through a web-based dashboard or an API, which developers can integrate into their applications. If you are familiar with the object-relational mapper, then in a sense, BaaS is a GUI version of that, plus a lot more functionalities to build a full-featured back-end.

Using BaaS developers to accelerate the development process by reducing the amount of time and resources needed to build and maintain the back-end infrastructure. This can result in faster time-to-market for applications and lower development costs. BaaS also handle the scalability and availability of the back-end infrastructure, ensuring that the applications can handle high traffic and maintain high uptime.

What are the differences between headless CMS and BaaS?

Both headless CMS and BaaS provide a battle-tested and full-featured back end out of the box. In fact, many of their functionalities overlap, so what makes the difference here?

Focus

Headless CMS focuses on providing a flexible content management system that can be used to manage content for different channels and platforms. This means their data usually are unstructured like articles, images, and videos. Besides, there is almost always a user-friendly dashboard (e.g., for creators) so it greatly reduces the learning curve for non-technical users.

BaaS, meanwhile, focuses on providing a GUI approach (little to no code) to build a back end. The data can be unstructured, but more than often, BaaS is used to create a complex structured application data model for tasks such as transactions, user profiles, and so on. These data may not even be used on the front end, instead, the other back end is its user. BaaS targets developers, so end-users usually do not know about BaaS's existence.
Customization

Both headless CMS and BaaS are very customizable. However, headless CMS focuses more on content creation-relation functionalities out of the box such as a WYSIWYG editor and content modeling, so other functionalities such as payment processing might need to be manually integrated. That said, many headless CMS has an existing method to integrate, via plugins for example. In contrast, BaaS as a full-featured package usually includes everything out of the box.
Cost

Both can provide cloud and self-hosted solutions. Usually, a self-hosted solution is only available when the product is open-source. When talking about cloud cost though, CMS, for the most part, follows a subscription per person model (e.g., $5/user/month). Meanwhile, the BaaS pricing model can vary a lot, from subscription, to pay per uptime, per request, and so on. Essentially, it is the same as traditional infrastructure-as-a-service cloud provides.

What are some open-source BaaS solutions?

There are various BaaS providers on the market, but not many are open-source, so let's focus on them today.

Amplication

Amplication is a relatively new open-source BaaS provider that aims to simplify the development process by providing a low-code platform for building and deploying applications. In fact, it just reached version 1.0 (production ready) in September 2022.

Amplication supports PostgreSQL, MySQL, MongoDB, and a variety of integrations, but most notably, it generates the whole back end with a human-readable NestJS codebase for you. You do not run Amplication as a service (or use it as a dependency), instead, you use Amplication to generate codes for you. You own the code, and it can be hosted wherever you prefer. This feature is the one and only in an open-source market (so not counting proprietary products like AWS Amplify).

Amplication provides both cloud and self-hosted solutions. There are a few points.

The price is fairly steep. Aside from a customized enterprise plan, only two plans Free and Pro exist. Free is slightly feature-limited, and Pro costs $320/month (billed annually) at the time of writing. This indicates that small business may not be their targetted segment.
It is unclear whether the self-hosted option is feature-limited.

Also, if you are a business customer in a regulated sector, there is no public Compliance statement on their website, so you may need to reach out to their customer support first to confirm the clearance.

However, as a relatively new platform, Amplication is still evolving and may not have all the features and integrations of more established BaaS providers. Still, it has the potential to be a leader in a low-code solution segment.

Appwrite

Appwrite is another open-source BaaS provider that offers a range of back-end services for building web and mobile applications. Coincidentally, its first production-ready version, v1.0, was also released back in September 2022. Appwrite provides features such as user authentication, database storage, file storage, serverless functions, and email sending.

Unlike Amplication, you integrate Appwrite to your back end with SDK. This may sound confusing because BaaS should be responsible for the whole back end in the first place, isn't it? Yes, although it handles all business logic for your application, Appwrite is a dependency of your application that you have to manually initialize and configure. This is another form of BaaS.

When talking about Appwrite, the product has three strengths. They might not be unique, but many other solutions do not have them all at the same time.

Support for multiple platforms and programming languages.
Serverless functions.
Real-time data subscription.

That said, Appwrite abstracts the database, meaning it is handled internally and only the NoSQL-like interface is exposed, so you have less flexibility in this regard.

Appwrite also provides both cloud and on-premise options. However, their cloud service is still in private beta, and there is no official pricing at the moment. The on-premise, or self-hosted, is free and full-featured, but certainly, it costs in the overhead maintenance.

And the same Amplication, Appwrite does not have any public statement about Compliance, so you may have to reach out to their support for clarification on this matter.

Supabase

Founded in 2020, in roughly the same period as Amplication and Appwrite, however, Supabase is arguably the most established open-source solution among the three. In a sense, Supabase is a direct equivalent and close to feature-parity with Appwrite, offering a self-hosted option, plus a mature cloud platform that offers enterprise support and an even greater amount of integrations. However, Supabase only supports PostgreSQL, the opposite of the NoSQL approach in Appwrite.

Regarding the cloud platform price, Supabase has a Free plan, a Pro plan, and a customized Enterprise plan. There are a few points to mention.

Unlike Amplication, which generates code and you are responsible for the hosting solution, Supabase provides integration and hosting services for you. The Pro plan costs $25 per month per project. That said, this is a different pricing model compared to Amplication, so they are not comparable.
They also have add-ons for the subscription, and the compute instance can be customized to your need. These will change the subscription fee though.
Supabase does not seem to have usage caps on their Pro plan, so if you accidentally exceed the limit, prepare your pocket.

On the topic of Compliance, Supabase has official support for SOC2 Type 2. There is no mention of other compliances, but they certainly do not support HIPAA and PCI (learned from their ToS and confirmed by a support agent).

What are some open-source headless CMS?

The articles below dived deep into 12 different solutions on the market. Do feel free to check them out if you are interested in this topic!

Wrap up

The article gives you an introduction to the concept of headless CMS, BaaS, the differences between them, and some open-source BaaS solutions on the market. That said, this article is just an introduction to the topic, the tip of the iceberg you can say, so there is a lot more to learn about. If you notice, the three mentioned BaaS all have their trade-off, so make sure to do deeper research before you decide to commit to one solution. Hopefully, this article was able to help you be aware of these fascinating products, and successfully in improving your developer experience and productivity.

This article was originally published at: Amplication vs Supabase vs Appwrite, for an effortless back end.

If you find this article to be helpful, I have some more for you!

And, let's connect!

🤯Twitter Recommendation Algorithm is now open sourced

Hung Vu — Sat, 01 Apr 2023 13:51:51 +0000

This requires a recommendation algorithm to distill the roughly 500 million Tweets posted daily down to a handful of top Tweets that ultimately show up on your device’s For You timeline.

The pipeline above runs approximately 5 billion times per day and completes in under 1.5 seconds on average. A single pipeline execution requires 220 seconds of CPU time, nearly 150x the latency you perceive on the app.

Along side with OpenAI, I personally think this is one of an important moment in the computing community as no one would ever guess a global-scale algorithm such as Twitter's Recommendation becomes open-sourced. Based on their engineer blog post, it is not out of reach to say the code base literally costs hundred thousands if not millions a day to run. How do you feel about this moment?

Twitter's engineer blog

Twitter's Recommendation Algorithm

Twitter Apache Thrift is an open-source, standalone, lightweight, data encoding library. In this blog post, we share the library we built so iOS developers outside Twitter can start using Thrift data.

blog.twitter.com

GitHub repository

twitter / the-algorithm

Source code for Twitter's Recommendation Algorithm

Twitter Recommendation Algorithm

The Twitter Recommendation Algorithm is a set of services and jobs that are responsible for constructing and serving the Home Timeline. For an introduction to how the algorithm works, please refer to our engineering blog. The diagram below illustrates how major services and jobs interconnect.

These are the main components of the Recommendation Algorithm included in this repository:

Type	Component	Description
Feature	SimClusters	Community detection and sparse embeddings into those communities.
TwHIN	Dense knowledge graph embeddings for Users and Tweets.
trust-and-safety-models	Models for detecting NSFW or abusive content.
real-graph	Model to predict likelihood of a Twitter User interacting with another User.
tweepcred	Page-Rank algorithm for calculating Twitter User reputation.
recos-injector	Streaming event processor for building input streams for GraphJet based services.
graph-feature-service	Serves graph features for a directed pair of Users (e.g. how many of User A's following liked Tweets from User B).
Candidate Source	search-index

…

View on GitHub

Moving from classic Linux distro to ChromeOS Flex? Here was what I found <🪲>

Hung Vu — Sun, 12 Mar 2023 21:07:14 +0000

I have never been a Linux guru, but as a casual user, I love playing around with the operating system from time to time. At one point, I decided that Pop!_OS (or PopOS) is a stopping point after a long distro-hopping journey. It's a great all-in-one package with several core features ready to go out of the box: Desktop customization, dedicated GPU supports, Bluetooth codec supports, and whatnot, you name it! I'm satisfied with PopOS, and I still do, but a question eventually arose.

When coming to laptop usage, 95% of my time spends on the browser. I boot up the laptop, open Firefox, search the web, consume media, and write articles like what you are reading now. For other more intensive tasks like gaming, coding, or so, my PC is responsible for them. Do I need a full-fledged highly customizable operating system for the laptop? Probably not, so ChromeOS came to my mind. In July 2022, Google announced a general availability for ChromeOS Flex (formerly CloudReady), a version that can be installed on non-Chromebook devices (any computer, in other words). In a sense, ChromeOS Flex is bleeding edge so it is expected to be somewhat unstable at the beginning, and I decided to wait until early 2023 to try it out, and here I am.

Why did ChromeOS come to my mind?

Minimalism

ChromeOS sacrifices lots of customizability but provides a decent set of Google applications out of the box. Chrome is in the name of the OS itself, so that alone covers 95% of my usage. Considering I have been working with different IT systems daily, I suppose having to tweak some Linux configurations outside of work hours is not that desirable. Don't get me wrong, playing around with technology has always been my hobby, but there is a line for when I just need to get things done, and ChromeOS provides that simple and minimalistic experience. That said, Linux on ChromeOS (or Crostini) is there, so I can always fire up the Linux environment if needed.
Lightweight

The minimalistic approach inherently means no bloatware comes with ChromeOS. It consumes about 20 GB of storage and 1.7 GB of RAM out-of-the-box. Nothing spectacular though, as PopOS is relatively the same. Both ChromeOS and Linux blow Windows 10/11 out of water because Microsoft somehow makes their OS consumes more than 60 GB and storage and around 6 GB of ram with a fresh installation. Technically, this resource consumption is negligible to my laptop specification (AMD Ryzen 7 5825U CPU, Solidigm P41 Plus 2TB SSD, and 64GB of RAM), but it is always better to remove those data-harvesting programs when possible.
Security

When considering a fresh installation, ChromeOS is arguably more secure than PopOS or other Linux distros. Minimalism greatly reduces the attack surface and ChromeOS was designed with security in mind using sandboxing and limited user write access. I'm by no means an expert on cybersecurity matters, but there are lots of resources online regarding this topic, and Security in Chrome OS is an example.

And my first try? Well, it was worse than I expected

Living on the edge is nothing new to me, eventually, things will work out. That is, assuming the fundamentals are satisfied, at least to a certain degree. The stable, beta, and canary versions are 110, 111, and 112 respectively at the time of writing. In my case, I tried out ChromeOS Flex using all those versions on three different laptop models (from Dell, HP, and Asus), and all exhibited breaking bugs in one way or another. What were they then?

Network chipset compatibility

At the moment, ChromeOS Flex does not support Intel AX210 and Mediatek MT7921 wireless modules. Unfortunately, two of my laptops have these so I could not even start the OS installation process because ChromeOS Flex requires Internet to set up. That said, these are documented in ChromeOS Flex Help - Known Issues, so it is acceptable to me. However, being a PC hobbyist, I have a plethora of computer components lying around, so I decided to try other modules which yielded the following results.

Intel AC3168 (unknown revision): Recognized
Intel AC7265 (unknown revision): Bluetooth was recognized, but no Wi-Fi
Intel AX200 (unknown revision): Recognized
Realtek - wireless module (missing label, unknown model): Unrecognized
Realtek RTL8118 - USB to 1 Gbe ethernet: Rev 3000 was recognized, while rev3100 was a hit or miss (might be due to USB implementation)

These are widely used wireless chipset and has long been supported in the Linux mainline kernel. Clearly, ChromeOS Flex needs to address these somehow.

Other miscellaneous hardware compatibilities

Unlike network chipset which has only a few big players on the market such as Intel, Mediatek, Qualcomm, and Broadcom, other components of the laptops are from more obscure providers and have a lot of variations. Using the PopOS's latest version (kernel v6.1 at the moment), there are still certain compatibility issues, so I suppose it is understandable for ChromeOS Flex to stay behind in this aspect. For example.

The touch function of a laptop screen did not work during an initial OS installation but is fine after getting to the desktop.
Touchpad, meanwhile, worked in the OS installation phase but became unusable after getting to the desktop. There were also times the touchpad worked fine from start to finish, but got disabled in a lock screen. Generally said, it was a hit or miss.
Speakers had the same issue as the touchpad.
The fingerprint was completely not supported on the three laptops I tried.
No tablet mode detection when the screen passed a 180 degrees angle (e.g., no rotation, no auto on-screen keyboard).

Somewhat lackluster in user experience

Although using a minimal OS is my goal, ChromeOS Flex is missing certain features that reduce my overall experience. They are not breaking deals though, but worth to be mentioned. For example.

Chrome tabs do not close after the browser exits, and there is no option to decide whether to keep tabs between sessions. Apparently, ChromeOS Flex simply does not support this, and the user must right-click on a browser tab, then choose to close other tabs before exiting the browser. It is an oversight in my opinion.
Website scroll speed is very inconsistent, as in, the acceleration kept jumping around. It seems this was once reported in Chromium Issue 836258, but it is not viewable anymore.
ChromeOS Flex has worse settings than PopOS, such as no main monitor resolution, no time to sleep, no sleep button, and no Bluetooth codec adjustment.
No trivial way to enable the tablet mode, which is commonly used when a laptop is a 2-in-1 type. The workaround I found was to enable the ASH shortcut flag in Chrome then press CTRL+ALT+SHIFT+T to enter and exit the tablet mode.
There is a disassociation between mouse action and touch gestures in tablet mode. For example, swiping right means return but it is only doable with a touch gesture, not with a mouse cursor.
Even inside the tablet mode, the on-screen keyboard does not automatically open when typing and must be manually turned on (and off) in Settings. This creates a blockage when a user switches between desktop and tablet mode frequently.
Non-standard way to enable an extra keyboard for special characters. Keyboards in other operating systems tend to reserve SHIFT for capitalization or expand options of one key, and an extra keyboard can be accessed using a special button at the corner. Meanwhile, SHIFT is used to switch keyboard types in ChromeOS Flex.

The list goes on, but again, not that big of a deal. Then, I found a serious issue with ChromeOS Flex which can potentially harm my laptops in the long run, and that was...

Completely broken sleep system

Using a laptop, I rarely shut down the machine, rather, sleep is my usual choice. Why? It is simply about convenience, what is better than just opening the lid and all your application is there in their previous state? And yet, Chrome OS completely broke my laptop's sleep functionality. There are three types of breaking bugs.

The laptop cannot wake up from sleep: I'm away from the keyboard or close the lid, and the laptop goes to sleep (screen off, keyboard-led off, fan spin down, no battery drain). However, when trying to turn it on, the fan spins, and the keyboard led lights up, but no screen. In this case, the screen is completely off (no backlight), and HDMI output does not provide a display signal. Only forcing the shutdown by pressing the power button can exit this state.
The laptop cannot go to the sleep state: I'm closing the lid, and the laptop does not go to sleep (screen off, keyboard-led off, fan either spins down or keeps rotating, battery drains even faster than usual).
The laptop is unresponsive after waking up: On one of my laptops that uses HDD, when the laptop goes to sleep (as expected) and wakes up. It is normal in the lock screen, but after singing, the user inputs (keystroke, mouse movement) are delayed for more than 1 minute, which effectively renders the laptop unusable. I have to force shutdown with the power button to exit this state. This persists through HDD replacement, but the problem disappears when using an SSD.

Here is the best part, these problems randomly happen but at a very high frequency, likely more than 80% of the time. I consider them harmful to my system because repetitive behaviors like force shutting down (unsafe shutdown) and battery drain can potentially corrupt OS or even degrade the hardware over time.

Are these isolated issues that only happen to my laptops? Maybe, but I could not identify the root cause. Here was what I found in the log for type one (wall of text, with my educated guess on their meaning). In this scenario, I opened the laptop and sign in, then let it stay there (not open any program or do anything) until the sleep kicks in.

# Starting to go sleep.
2023-03-02T05:21:13.976131Z INFO powerd: [suspend_delay_controller.cc(99)] Got notification that delay 67305475 (shill) is ready for suspend request 67305473 from :1.8
2023-03-02T05:21:13.977835Z INFO powerd: [suspend_delay_controller.cc(99)] Got notification that delay 67305476 (chrome) is ready for suspend request 67305473 from :1.37
2023-03-02T05:21:13.977901Z INFO powerd: [suspend_delay_controller.cc(254)] Notifying observers that suspend is ready
2023-03-02T05:21:13.978472Z INFO powerd: [suspender.cc(739)] Starting suspend
# Is this error relevant? Or just a vague message?
2023-03-02T05:21:13.978682Z ERROR powerd: [suspend_configurator.cc(75)] Failed to write deep to /sys/power/mem_sleep: Invalid argument (22)
2023-03-02T05:21:14.054817Z INFO powerd: [main.cc(408)] Running "/usr/bin/powerd_setuid_helper --action=suspend --suspend_wakeup_count_valid --suspend_wakeup_count=72"
# Sleep for about 8 minutes, then wake up.
2023-03-02T05:29:56.150638Z INFO powerd: [daemon.cc(960)] powerd_suspend returned 0
2023-03-02T05:29:56.152115Z INFO powerd: [wakeup_device.cc(67)] Device /sys/devices/platform/i8042/serio0 had event_count 65 before suspend and 66 after resume
2023-03-02T05:29:56.152227Z INFO powerd: [dark_resume.cc(35)] User triggered wake
2023-03-02T05:29:56.152250Z INFO powerd: [suspender.cc(627)] Finishing request 67305473 successfully after 8m42s
2023-03-02T05:29:56.153198Z INFO powerd: [state_controller.cc(1157)] Updated settings: dim=5m quick_dim=1m screen_off=5m30s lock=5m40s quick_lock=0s idle_warn=0s idle=6m30s (suspend) lid_closed=suspend use_audio=1 use_video=1 wake_locks=
# The laptop is on now, and screen is going to light up.
2023-03-02T05:29:56.153237Z INFO powerd: [state_controller.cc(133)] Undimming screen
2023-03-02T05:29:56.153253Z INFO powerd: [state_controller.cc(133)] Turning screen on
2023-03-02T05:29:56.153319Z INFO powerd: [display_power_setter.cc(75)] Asking DisplayService to turn all displays on
# ERROR!!! The screen cannot turn on!
2023-03-02T05:29:56.480391Z ERROR powerd: [object_proxy.cc(622)] Failed to call method: org.chromium.DisplayServiceInterface.SetPower: object_path= /org/chromium/DisplayService: org.freedesktop.DBus.Error.NoReply: Message recipient disconnected from message bus without replying
2023-03-02T05:29:56.480475Z INFO powerd: [internal_backlight_controller.cc(781)] Setting brightness to 95 (63%) over 0 ms
2023-03-02T05:29:56.488015Z INFO powerd: [bluetooth_controller.cc(97)] Writing "auto" to /sys/devices/pci0000:00/0000:00:08.1/0000:04:00.4/usb3/3-3/power/control succeeded
# Another crypted error.
2023-03-02T05:29:56.488073Z ERROR powerd: [cros_ec_device_event.cc(29)] Failed to open /dev/cros_ec: No such file or directory (2)
2023-03-02T05:29:56.488473Z INFO powerd: [daemon.cc(1662)] Received updated external policy: ac_dim=30s ac_screen_off=40s ac_lock=50s ac_idle_warn=0s ac_idle=8m30s battery_dim=30s battery_screen_off=40s battery_lock=50s battery_idle_warn=0s battery_idle=6m30s ac_idle=suspend battery_idle=suspend lid_closed=suspend use_audio=1 use_video=1 presentation_factor=1 user_activity_factor=1 wait_for_initial_user_activity=0 force_nonzero_brightness_for_user_activity=1 (Prefs)
2023-03-02T05:29:56.488573Z INFO powerd: [state_controller.cc(1157)] Updated settings: dim=30s quick_dim=0s screen_off=40s lock=50s quick_lock=0s idle_warn=0s idle=6m30s (suspend) lid_closed=suspend use_audio=1 use_video=1 wake_locks=
2023-03-02T05:29:56.489603Z INFO powerd: [suspend_delay_controller.cc(134)] Unregistering suspend delay 67305476 (chrome) due to D-Bus client :1.37 going away
2023-03-02T05:29:56.489659Z INFO powerd: [suspend_delay_controller.cc(134)] Unregistering dark suspend delay 67338241 (chrome) due to D-Bus client :1.37 going away
# Battery is at 79% (note, this is a new laptop battery, and performs well in PopOS).
2023-03-02T05:29:56.490001Z INFO powerd: [daemon.cc(1213)] On battery at 79% (displayed as 81%), 2.826/3.600Ah at 0.456A, 0s until empty (calculating)
2023-03-02T05:29:56.490444Z INFO powerd: [udev.cc(500)] Removing device /sys/devices/platform/i8042/serio1/input/input5/event11
2023-03-02T05:29:56.491477Z INFO powerd: [daemon.cc(1753)] Session state changed to stopped
2023-03-02T05:29:56.491627Z INFO powerd: [state_controller.cc(1157)] Updated settings: dim=30s quick_dim=0s screen_off=40s lock=50s quick_lock=0s idle_warn=0s idle=6m30s (suspend) lid_closed=suspend use_audio=1 use_video=1 wake_locks=
2023-03-02T05:29:56.493120Z INFO powerd: [wakeup_source_identifier.cc(137)] Stopped monitoring wakeup path /sys/devices/platform/i8042/serio1 for wake events
2023-03-02T05:29:56.493151Z INFO powerd: [udev.cc(500)] Removing device /sys/devices/platform/i8042/serio1/input/input5
2023-03-02T05:29:56.511177Z INFO powerd: [suspend_delay_controller.cc(88)] Unregistering suspend delay 67305477 (Pause VMs while suspended) on behalf of :1.134
2023-03-02T05:29:56.945571Z INFO powerd: [daemon.cc(1197)] D-Bus org.chromium.SessionManager ownership changed to :1.170
2023-03-02T05:29:57.057971Z INFO powerd: [suspend_delay_controller.cc(73)] Registering suspend delay 67305478 (chrome) of 5000 ms on behalf of :1.171
2023-03-02T05:29:57.058236Z INFO powerd: [suspend_delay_controller.cc(73)] Registering dark suspend delay 67338243 (chrome) of 5000 ms on behalf of :1.171
# Pressing the power button again, in an attempt to wake up screen. This parts seems to indicate it. Pay attention to the timestamp.
2023-03-02T05:29:57.113615Z INFO powerd: [daemon.cc(1643)] Chrome is using normal display mode
2023-03-02T05:29:57.137056Z INFO powerd: [daemon.cc(1200)] D-Bus org.chromium.DisplayService ownership changed to :1.171
2023-03-02T05:29:57.137088Z INFO powerd: [display_power_setter.cc(75)] Asking DisplayService to turn all displays on
# And the display is still off, while the whole computer is (supposedly running)
2023-03-02T05:30:02.139349Z WARNING powerd: [bus.cc(644)] Bus::SendWithReplyAndBlock took 5002ms to process message: type=method_call, path=/org/chromium/DisplayService, interface=org.chromium.DisplayServiceInterface, member=SetPower
2023-03-02T05:30:02.139379Z ERROR powerd: [object_proxy.cc(622)] Failed to call method: org.chromium.DisplayServiceInterface.SetPower: object_path= /org/chromium/DisplayService: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
2023-03-02T05:30:02.139443Z INFO powerd: [daemon.cc(1204)] D-Bus org.chromium.PrivacyScreenService ownership changed to :1.171
# Battery drops after 5 second.
2023-03-02T05:30:02.148549Z INFO powerd: [daemon.cc(1213)] On battery at 78% (displayed as 81%), 2.822/3.600Ah at 2.408A, 3h48m44s until empty (3h45m44s until shutdown)
# These screen-related errors may indicate something.
2023-03-02T05:30:05.142901Z ERROR powerd: [object_proxy.cc(622)] Failed to call method: org.chromium.PrivacyScreenService.GetPrivacyScreenSetting: object_path= /org/chromium/PrivacyScreenService: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
2023-03-02T05:30:05.142930Z ERROR powerd: [daemon.cc(1435)] D-Bus method call to GetPrivacyScreenSetting failed
2023-03-02T05:30:21.493067Z ERROR powerd: [object_proxy.cc(622)] Failed to call method: org.chromium.MlDecisionService.ShouldDeferScreenDim: object_path= /org/chromium/MlDecisionService: org.freedesktop.DBus.Error.ServiceUnknown: The name org.chromium.MlDecisionService was not provided by any .service files
2023-03-02T05:30:21.493097Z ERROR powerd: [dim_advisor.cc(181)] D-Bus method call to org.chromium.MlDecisionService.ShouldDeferScreenDim failed
2023-03-02T05:30:26.492176Z INFO powerd: [state_controller.cc(126)] Dimming screen after 30s
2023-03-02T05:30:26.492201Z INFO powerd: [internal_backlight_controller.cc(781)] Setting brightness to 26 (35.1246%) over 200 ms
2023-03-02T05:30:32.157441Z INFO powerd: [daemon.cc(1213)] On battery at 78% (displayed as 80%), 2.804/3.600Ah at 2.194A, 2h32m9s until empty (2h29m9s until shutdown)
2023-03-02T05:30:36.495411Z INFO powerd: [state_controller.cc(126)] Turning screen off after 40s
2023-03-02T05:30:36.495438Z INFO powerd: [internal_backlight_controller.cc(781)] Setting brightness to 0 (0%) over 200 ms
# A screen off request?
2023-03-02T05:30:36.695850Z INFO powerd: [display_power_setter.cc(75)] Asking DisplayService to turn all displays off
2023-03-02T05:30:41.700939Z WARNING powerd: [bus.cc(644)] Bus::SendWithReplyAndBlock took 5005ms to process message: type=method_call, path=/org/chromium/DisplayService, interface=org.chromium.DisplayServiceInterface, member=SetPower
# It seems the request somehow fails. Note, the screen been off till now while the laptop has been running (fan spins, keyboard lights up).
2023-03-02T05:30:41.700965Z ERROR powerd: [object_proxy.cc(622)] Failed to call method: org.chromium.DisplayServiceInterface.SetPower: object_path= /org/chromium/DisplayService: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
2023-03-02T05:30:46.496482Z INFO powerd: [state_controller.cc(126)] Locking screen after 50s
2023-03-02T05:30:46.497000Z ERROR powerd: [object_proxy.cc(622)] Failed to call method: org.chromium.SessionManagerInterface.LockScreen: object_path= /org/chromium/SessionManager: org.chromium.SessionManagerInterface.SessionDoesNotExist: Attempt to lock screen outside of user session.
# Battery drops again just after 1 minute, this is unusual.
2023-03-02T05:31:02.166440Z INFO powerd: [daemon.cc(1213)] On battery at 77% (displayed as 80%), 2.786/3.600Ah at 2.069A, 1h54m48s until empty (1h51m48s until shutdown)
2023-03-02T05:31:32.176555Z INFO powerd: [daemon.cc(1213)] On battery at 77% (displayed as 79%), 2.768/3.600Ah at 2.198A, 1h30m41s until empty (1h27m41s until shutdown)
2023-03-02T05:31:57.360460Z INFO powerd: [suspend_delay_controller.cc(134)] Unregistering suspend delay 67305478 (chrome) due to D-Bus client :1.171 going away
2023-03-02T05:31:57.360489Z INFO powerd: [suspend_delay_controller.cc(134)] Unregistering dark suspend delay 67338243 (chrome) due to D-Bus client :1.171 going away
# The battery drops seems really fast even with screen off, isn't it? This may indicate certain hardware components are underload and drains the battery.
2023-03-02T05:32:02.188245Z INFO powerd: [daemon.cc(1213)] On battery at 76% (displayed as 79%), 2.751/3.600Ah at 1.086A, 1h21m1s until empty (1h18m1s until shutdown)
2023-03-02T05:32:32.200221Z INFO powerd: [daemon.cc(1213)] On battery at 76% (displayed as 79%), 2.742/3.600Ah at 1.111A, 1h32m39s until empty (1h29m39s until shutdown)
2023-03-02T05:33:02.212217Z INFO powerd: [daemon.cc(1213)] On battery at 76% (displayed as 78%), 2.733/3.600Ah at 1.116A, 1h45m29s until empty (1h42m29s until shutdown)
2023-03-02T05:33:32.224203Z INFO powerd: [daemon.cc(1213)] On battery at 76% (displayed as 78%), 2.724/3.600Ah at 1.097A, 2h35s until empty (1h57m35s until shutdown)
2023-03-02T05:34:02.236210Z INFO powerd: [daemon.cc(1213)] On battery at 75% (displayed as 78%), 2.715/3.600Ah at 1.109A, 2h23m57s until empty (2h20m57s until shutdown)
2023-03-02T05:34:32.248213Z INFO powerd: [daemon.cc(1213)] On battery at 75% (displayed as 77%), 2.706/3.600Ah at 0.953A, 2h26m37s until empty (2h23m37s until shutdown)
2023-03-02T05:35:02.260236Z INFO powerd: [daemon.cc(1213)] On battery at 75% (displayed as 77%), 2.698/3.600Ah at 1.095A, 2h27m8s until empty (2h24m8s until shutdown)
2023-03-02T05:35:32.272206Z INFO powerd: [daemon.cc(1213)] On battery at 75% (displayed as 77%), 2.689/3.600Ah at 1.088A, 2h27m26s until empty (2h24m26s until shutdown)
2023-03-02T05:36:02.284024Z INFO powerd: [daemon.cc(1213)] On battery at 74% (displayed as 77%), 2.680/3.600Ah at 0.955A, 2h30m40s until empty (2h27m40s until shutdown)
2023-03-02T05:36:26.496102Z INFO powerd: [state_controller.cc(1347)] Ready to perform idle action (suspend) after 6m30s
2023-03-02T05:36:26.496195Z INFO powerd: [suspender.cc(575)] Starting request 67305474
2023-03-02T05:36:26.598662Z INFO powerd: [bluetooth_controller.cc(84)] Writing "on" to /sys/devices/pci0000:00/0000:00:08.1/0000:04:00.4/usb3/3-3/power/control succeeded
2023-03-02T05:36:26.598754Z INFO powerd: [daemon.cc(837)] Reading wakeup count from /sys/power/wakeup_count
2023-03-02T05:36:26.598856Z INFO powerd: [daemon.cc(841)] Read wakeup count 92
2023-03-02T05:36:26.598880Z INFO powerd: [suspend_delay_controller.cc(151)] Announcing suspend request 67305474 with 3 pending delay(s) and 0 outstanding delay(s) from previous request
2023-03-02T05:36:26.599324Z ERROR powerd: [cros_ec_device_event.cc(29)] Failed to open /dev/cros_ec: No such file or directory (2)
2023-03-02T05:36:26.600370Z INFO powerd: [suspend_delay_controller.cc(99)] Got notification that delay 67305473 (Bluetooth Manager) is ready for suspend request 67305474 from :1.4
2023-03-02T05:36:26.600946Z INFO powerd: [suspend_delay_controller.cc(99)] Got notification that delay 67305474 (trunksd) is ready for suspend request 67305474 from :1.6
2023-03-02T05:36:26.601620Z INFO powerd: [suspend_delay_controller.cc(99)] Got notification that delay 67305475 (shill) is ready for suspend request 67305474 from :1.8
2023-03-02T05:36:26.601700Z INFO powerd: [suspend_delay_controller.cc(254)] Notifying observers that suspend is ready
# Request to sleep again, which it should not be at this moment as I keep pressing the keyboards. That said, aside from special keys like power, keyboard backlight adjust, etc. I could not confirm whether my keystroke are registered.
2023-03-02T05:36:26.602362Z INFO powerd: [suspender.cc(739)] Starting suspend
# And it fails?
2023-03-02T05:36:26.602558Z ERROR powerd: [suspend_configurator.cc(75)] Failed to write deep to /sys/power/mem_sleep: Invalid argument (22)
2023-03-02T05:36:26.630702Z INFO powerd: [main.cc(408)] Running "/usr/bin/powerd_setuid_helper --action=suspend --suspend_wakeup_count_valid --suspend_wakeup_count=92"

Wrap up

ChromeOS Flex, in my opinion, is still very worth a try and I like what it offers at the very early phase like now. Lightweight, security, and minimalism, these aspects can bring your device a new life, be it a powerful machine or one that has been in storage for years. Feature-wise, it is expected to not be comparable to a full fledge operating system such as PopOS, and that is completely fine to me. Without the very broken sleep functionality, I may have continued my journey with ChromeOS Flex and its ecosystem, but for now, I have to postpone the plan. I wonder, a year from now, what will Chrome OS Flex become?

This article was originally published at 🪱ChromeOS Flex, have I just opened a can of worms?

If you find my article to be helpful, I have some more for you!

And, let's connect!

🌱Using Forem as an intranet website, yay or nay?

Hung Vu — Tue, 07 Mar 2023 23:16:33 +0000

I'm looking into building an intranet website and is researching on available options. There are two solutions that I have at the moment:

DIY the website. This offers customizability but come at the cost of overhead such as development and maintenance time.
Use an existing open-source forum solutions like Forem, Discourse, NodeBB, etc. This cut down the overhead, but also reduces customizability (no tailored UX?) and might not scale well when requirements evolve.

Which options would you suggest? And what else should I consider? The draft requirements can be seen below:

There are 2 main classes of users: Authors, and readers. The authors have right to create contents (articles, videos), while the readers can only consume content. A user can only be one or another.
More advanced access control list: E.g., only authorized users can see the content.
Good user experience (navigation, search, organization, etc.) Cloud file-directory based options such as Google Workspace has become an obstacle for non-technical users in this specific environment.
Targeted audience: Certain type of content can be promoted or pin to dashboard of targeted audience.
User profile.
SSO.
Optional: Scheduling (calendar), and location (map) features.