Forem: Luca Barbato

Bringing up an AmpereOne system

Luca Barbato — Sat, 28 Feb 2026 17:16:52 +0000

I decided to buy something quite large since my workstations are aging and I wanted to build stuff quickly.

I decided to buy a AMPONED8-2T/BCM with a nice 128-cores CPU and a bit of RAM.

It took a bit for the people at https://www.mifcom.de/ to source all the components, but eventually I got it.

Initial Setup

The board comes with a very pretty BMC so all you need is to connect the management one (or any since they are bonded by default) and let the dhcp get an ip and then feed it to your browser.

Since I'm using Gentoo the boot media just worked, you may feed it directly from the Remote KVM

And just boot it from the browser

Then the process is the usual one

Partition and untar

Partition	Mount point	Filesystem	Recommended size	label
p1	/boot/efi	vfat	1 GiB	EF00
p2	/	btrfs	500+ GiB	8300
p3	/home	btrfs	500+ GiB	8302

mkfs.vfat -n EFI /dev/nvme0n1p1
mkdir /mnt/btrfs
mkfs.btrfs -L root --checksum xxhash /dev/nvme0n1p2
mount /dev/nvme0n1p2 /mnt/btrfs
btrfs subvolume create /mnt/btrfs/@
btrfs subvolume create /mnt/btrfs/@repos
btrfs subvolume create /mnt/btrfs/@snapshots
btrfs subvolume create /mnt/btrfs/@var_log
btrfs subvolume create /mnt/btrfs/@containers
umount /mnt/btrfs

mount -o subvol=@ /dev/nvme0n1p2 /mnt/gentoo
mkdir -p /mnt/gentoo/{.snapshots,var/{db/repos,log},var/lib/containers}
mount -o subvol=@snapshots /dev/nvme0n1p2 /mnt/gentoo/.snapshots
mount -o subvol=@repos     /dev/nvme0n1p2 /mnt/gentoo/var/db/repos
mount -o subvol=@var_logs  /dev/nvme0n1p2 /mnt/gentoo/var/log
mount -o subvol=@containers /dev/nvme0n1p2 /mnt/gentoo/var/lib/containers

cd /mnt/gentoo
wget {stage3}
tar -xpf {stage3}

Enter the stage

Usual chroot

# Mount special filesystems
mount --types proc /proc /mnt/gentoo/proc
mount --rbind /sys /mnt/gentoo/sys
mount --make-rslave /mnt/gentoo/sys
mount --rbind /dev /mnt/gentoo/dev
mount --make-rslave /mnt/gentoo/dev
mount --bind /run /mnt/gentoo/run
mount --make-slave /mnt/gentoo/run

# copy the nameservers already present
cp --dereference /etc/resolv.conf /mnt/gentoo/etc/

# Enter the chroot
env -i \
    HOME=/root \
    TERM="$TERM" \
    PS1="(chroot) $PS1" \
    chroot /mnt/gentoo /bin/bash --login

# Inside chroot now:
source /etc/profile
export PS1="(chroot) ${PS1}"

Initial bits to set up

emerge --sync
getuto
# Install your favourite editor, even from the binhost
emerge -G vim
# do your changes
vim /etc/portage/make.conf
# minimal and simple setup to to boot
emerge gentoo-sources grub dracut
eselect kernel set 1
cd /usr/src/linux
# the default config works well enough
zcat /proc/config.gz > .config
make olddefconfig
# add what you need for e.g. docker or lxd
make menuconfig
make -j 128 && make -j128 modules && make modules_install && make install
# set up your user
emerge supeadduser
superadduser
# set up ssh
emerge openssh metalog
rc-update add sshd
rc-update add metalog

All considered the whole process was quite uneventful and took literally minutes to then build additional toolchains and start doing some tests.

qlop -c | egrep 'llvm|rust'
app-eselect/eselect-rust: 2′38″ average for 1 merge
dev-lang/rust: 15′19″ average for 1 merge
dev-lang/rust-bin: 13′21″ average for 2 merges
dev-lang/rust-common: 42s average for 1 merge
dev-util/rustup: 1′42″ average for 1 merge
llvm-core/clang: 4′21″ average for 2 merges
llvm-core/clang-common: 12s average for 2 merges
llvm-core/clang-linker-config: 5′53″ average for 2 merges
llvm-core/clang-toolchain-symlinks: 5s average for 2 merges
llvm-core/llvm: 9′05″ average for 2 merges
llvm-core/llvm-common: 2′41″ average for 1 merge
llvm-core/llvm-toolchain-symlinks: 11s average for 2 merges
llvm-core/llvmgold: 7s average for 2 merges
llvm-runtimes/clang-rtlib-config: 5′54″ average for 2 merges
llvm-runtimes/clang-runtime: 6s average for 2 merges
llvm-runtimes/clang-stdlib-config: 5′56″ average for 2 merges
llvm-runtimes/clang-unwindlib-config: 5′58″ average for 2 merges
llvm-runtimes/compiler-rt: 26s average for 2 merges
llvm-runtimes/compiler-rt-sanitizers: 37s average for 2 merges
llvm-runtimes/openmp: 6′10″ average for 2 merges

cargo-c common questions

Luca Barbato — Mon, 02 Sep 2024 09:44:38 +0000

I wrote and maintain cargo-c a cargo ~~applet~~ custom subcommand that let you build a rust crate sporting a C-API as a proper C library and install it along with C headers and pkg-config files.

When I notice some rust software sporting a C-API that I'd consider packaging in Gentoo I tend to provide a patch to add the few metadata entries needed to have cargo-c do all the work, some other people politely ask if cargo-c is supported and sometimes it is an uphill battle usually because the project maintainer doesn't know the required effort is minimum or is not aware setting crate-type=cdylib is not enough.

I'm writing this to explain some of the problems cargo-c solves and hopefully give few pointers since the README containing all the documentation maybe grew to be fairly big.

In closing

I hope this article helps people convince maintainers of crates trying to replace a C library to adopt it since it will spare quite a bit of headaches to distributors and make much simpler to foster adoption of memory-safer alternatives.

Is cargo/rustc that slow?

Luca Barbato — Fri, 30 Aug 2024 15:12:40 +0000

As a side-discussion from the ongoing mess I read this about Waiting 15 minutes for a small program, e.g. like Yazi to compile is wild.

I tried and on my laptop took about 2 minutes to build in release mode, not too bad.

❯ cargo build
...
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 27.51s
❯ cargo clean
❯ cargo build --release
...
    Finished `release` profile [optimized] target(s) in 2m 01s

yazi isn't exactly a small program:

❯ cargo tree | grep -v yazi | wc -l
     685
❯ tokei
===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 JSON                    1            1            1            0            0
 Lua                    34         1973         1639           31          303
 Markdown                2          207            0          149           58
 Nix                     4          270          235            6           29
 Shell                   3           56           41            6            9
 TOML                   18         1819         1484          128          207
 YAML                    1           33           29            1            3
-------------------------------------------------------------------------------
 Rust                  463        27288        22926          280         4082
 |- Markdown             6           56            0           51            5
 (Total)                          27344        22926          331         4087
===============================================================================
 Total                 526        31647        26355          601         4691
===============================================================================

Then I looked a bit further and I saw:

[profile.release]
codegen-units = 1
lto           = true
panic         = "abort"
strip         = true

Definitely not friendly if you want a fast release build

codegen-units = 1 may generate a bit more optimal code but then you can use only 1 core and if you have many...
lto = true tends to take lots of time for some optimality, but lto = "thin" exists and the tradeoff tends to be very good, for rav1e on x86_64 for a long while managed to produce even a better binary.

So what happens if I do those changes?

❯ cargo build --release
...
    Finished `release` profile [optimized] target(s) in 47.05s

I guess sometimes it is better to not stop at the first program to evaluate if a toolchain is fast or slow :)

P.S.

If you are on Linux, at least right now the quickest combination seems to be clang as linker with mold doing the actual work.
gcc + mold seems sensibly slower and clang + ld.bfd is curiously a tad slower than gcc + ld.bfd.
Setting

[target.{yourarch}]
linker = "clang"
rustflags = ["-C", "link-arg=-fuse-ld=mold"]

in ~/.cargo/config.toml may be optimal at least right now.

cross-stages experiments

Luca Barbato — Fri, 02 Aug 2024 09:15:40 +0000

I'm still playing with RISC-V and the bpi-f3.

Support status for BPI-F3 upstream

The upstreaming effort at least for the kernel is ongoing.

The bsp seems actively maintained with changes landing quite often in their fork of opensbi, u-boot and linux. Sadly being bsp and since it covers multiple boards they tend of lump lots of changes together so it can get annoying to follow what they are fixing or adding.

Given enough opensource developers are getting the board or similar hardware, I hope we'll have enough eyes and hands to clean up the code in a shape it can also be upstreamed to u-boot and opensbi.

My experiments

I wanted to see how different would feel a whole system built to leverage at least a bit RVV 1.0 through the compiler, sadly until this bug won't be addressed it would be hard to have a fully working system compiled with the autovectorizer on and I found it out the hard way.

The Gentoo default way to build stages and images via catalyst relies on qemu-user when no fast native system is available to avoid pitfalls of packages failing at cross-compilation. It is a bit slow but fairly reliable.

Since things should improve over time, I tried to leverage crossdev and see if:

[x] At least the stage1 packages could be cross-compiled and installed via crossdev (perl seems the only package misbehaving)
[x] Freshening a stage3 works. As seen with cross-boss it feasible, I wanted to see it can be made even more straightforward.
[x] Add packages and setup demons

All without using qemu-user so it can be made quickly, so far it mostly works.

On top of it I just cross-compiled the bsp packages and copied the approach they used to assemble the final image using genimage, trying to make it as simple as possible by using a single configuration file that does everything in a single invocation.

The crossdev-stages set of scripts should let anybody with a recent Gentoo with the dependencies listed to build their images in reasonable time.

Usage

I prepared 3 scripts:

chroot-stage.sh that make quite easy to fetch a stage3, unpack it and and optionally enter it using bubblewrap (thanks chewi for introducing me to it).

# If you already have Gentoo you can just fetch the stage
$ bash chroot-stage.sh setup stage3/ riscv64
# If you do not have Gentoo you can use the script to have a host chroot
$ bash chroot-stage.sh setup my-chroot
$ bash chroot-stage.sh enter my-chroot

cross-stage.sh that takes care of all the building, you can go from-scratch or start from a pre-existing stage3

# This will rebuild all the stage3 using the newer compiler and the CFLAGS you set in the script
$ bash cross-stage.sh update stage3/
# Install the bare minimum needed to have the bootloader working
$ bash cross-stage.sh install_boot stage3/
# Install additional packages you may find useful
$ bash cross-stage.sh install_more stage3/

make-image.sh that puts everything together and build the bsp packages.

# It will produce a gentoo-linux-k1_dev-sdcard.img.xz in the build-directory
$ bash make-image.sh /tmp/build-directory stage3/

So far it works for me you are welcome to try it, help improving it or just use the pre-made image.

Enjoy!

Bringing up BPI-F3 - Part 3

Luca Barbato — Tue, 04 Jun 2024 17:45:32 +0000

Initramfs

Initially I was hoping that it would not be needed, but since the SoC has a remote processor and the defconfig for it enables it, I guess it is simpler to use an initramfs.

Remoteproc firmware

As seen here the remoteproc needs a firmware bit and if you happen to forget about it you'd be welcomed by:

[    4.205609] remoteproc remoteproc0: rcpu_rproc is available
[    4.211421] remoteproc remoteproc0: Direct firmware load for esos.elf failed with error -2
[    4.214379] riscv-pmu-sbi: SBI PMU extension is available
[    4.219790] remoteproc remoteproc0: powering up rcpu_rproc
[    4.225306] riscv-pmu-sbi: 16 firmware and 18 hardware counters
[    4.230776] remoteproc remoteproc0: Direct firmware load for esos.elf failed with error -2
[    4.245106] remoteproc remoteproc0: request_firmware failed: -2
[    4.246235] es8326 2-0019: assuming static mclk
[    4.256170] enter spacemit_snd_sspa_pdev_probe
[    4.301833] usb 2-1: new high-speed USB device number 2 using xhci-hcd

If you like to use dracut all you need is to add to your /etc/dracut.conf.d/firmware.conf is:

install_items+=" /lib/firmware/esos.elf "

If you use Genkernel, set in /etc/genkernel.conf:

# Add firmware(s) to initramfs
FIRMWARE="yes"

# Specify directory to pull from
FIRMWARE_DIR="/lib/firmware"

# Specify a comma-separated list of firmware files or directories to include,
# relative to FIRMWARE_DIR.  If empty or unset, the full contents of 
# FIRMWARE_DIR will be included (if FIRMWARE option above is set to YES).
FIRMWARE_FILES="esos.elf"

as explained here.

Coming next

Now the remaining bits I'd like to have done are having a nicer u-boot configuration and hopefully wrap everything up so we can have a Gentoo image that can be simply flashed to the SD/eMMC/NVMe.

Bringing up BPI-F3 - Part 2.5

Luca Barbato — Fri, 24 May 2024 07:31:29 +0000

this is a sort of intermission

Getting perf to work up to a point

Apparently the opensbi-mediated access to the performance counter does not map so using the usual cycles and instructions event works in perf record. I got this board mainly to help with dav1d development efforts, so not having perf support would make harder to reason about performance.

The best workaround after a discussion in the forums, is to build the pmu-events to include custom ones and then rely on the overly precise cpu-specific events instead:

$ perf list | grep cycle
  bus-cycles                                         [Hardware event]
  cpu-cycles OR cycles                               [Hardware event]
  ref-cycles                                         [Hardware event]
  stalled-cycles-backend OR idle-cycles-backend      [Hardware event]
  stalled-cycles-frontend OR idle-cycles-frontend    [Hardware event]
  m_mode_cycle
       [M-mode cycles]
  rtu_flush_cycle
  s_mode_cycle
       [S-mode cycles]
  stalled_cycle_backend
       [Stalled cycles backend]
  stalled_cycle_frontend
       [Stalled cycles frontend]
  u_mode_cycle
       [U-mode cycles]
  vidu_total_cycle
  vidu_vec0_cycle
  vidu_vec1_cycle
...

$ perf list | grep inst
  branch-instructions OR branches                    [Hardware event]
  instructions                                       [Hardware event]
  br_inst
       [Branch instructions]
  cond_br_inst
       [Conditional branch instructions]
  indirect_br_inst
       [Indirect branch instructions]
  taken_cond_br_inst
       [Taken conditional branch instructions]
  uncond_br_inst
       [Unconditional branch instructions]
instruction:
  alu_inst
       [ALU (integer) instructions]
  amo_inst
       [AMO instructions]
  atomic_inst
       [Atomic instructions]
  bus_fence_inst
       [Bus FENCE instructions]
  csr_inst
       [CSR instructions]
  div_inst
       [Division instructions]
  ecall_inst
       [ECALL instructions]
  failed_sc_inst
       [Failed SC instructions]
  fence_inst
       [FENCE instructions]
  fp_div_inst
       [Floating-point division instructions]
  fp_inst
       [Floating-point instructions]
  fp_load_inst
       [Floating-point load instructions]
  fp_store_inst
       [Floating-point store instructions]
  load_inst
       [Load instructions]
  lr_inst
       [LR instructions]
  mult_inst
       [Multiplication instructions]
  sc_inst
       [SC instructions]
  store_inst
       [Store instructions]
  unaligned_load_inst
       [Unaligned load instructions]
  unaligned_store_inst
       [Unaligned store instructions]
  vector_div_inst
       [Vector division instructions]
  vector_inst
       [Vector instructions]
  vector_load_inst
       [Vector load instructions]
  vector_store_inst
       [Vector store instructions]
  id_inst_pipedown
       [ID instruction pipedowns]
  id_one_inst_pipedown
       [ID one instruction pipedowns]
  issued_inst
       [Issued instructions]
  rf_inst_pipedown
       [RF instruction pipedowns]
  rf_one_inst_pipedown
       [RF one instruction pipedowns]

Building perf

Perf way to deal with cpu-specific events is through some machinery called jevents.

It lives in tools/perf/pmu-events and you can manually trigger it with.

./jevents.py riscv arch pmu-events.c

And produce C code from a bunch of JSON and a CSV map file.

When I tried build the sources the first time I tried to cut it by setting most NO_{} make variables and left NO_JEVENTS=1, luckily I fixed it after noticing the different output in the forum.

## I assume you have here the custom linux sources
cd /usr/src/pi-linux/tools/perf
## being lazy I disabled about everything instead of installing dependencies, one time I disabled too much.
make -j 8 V=1 VF=1 HOSTCC=riscv64-unknown-linux-gnu-gcc HOSTLD=riscv64-unknown-linux-gnu-ld CC=riscv64-unknown-linux-gnu-gcc CXX=riscv64-unknown-linux-gnu-g++ AR=riscv64-unknown-linux-gnu-ar LD=riscv64-unknown-linux-gnu-ld NM=riscv64-unknown-linux-gnu-nm PKG_CONFIG=riscv64-unknown-linux-gnu-pkg-config prefix=/usr bindir_relative=bin tipdir=share/doc/perf-6.8 'EXTRA_CFLAGS=-O2 -pipe' 'EXTRA_LDFLAGS=-Wl,-O1 -Wl,--as-needed' ARCH=riscv BUILD_BPF_SKEL= BUILD_NONDISTRO=1 JDIR= CORESIGHT= GTK2= feature-gtk2-infobar= NO_AUXTRACE= NO_BACKTRACE= NO_DEMANGLE= NO_JEVENTS=0 NO_JVMTI=1 NO_LIBAUDIT=1 NO_LIBBABELTRACE=1 NO_LIBBIONIC=1 NO_LIBBPF=1 NO_LIBCAP=1 NO_LIBCRYPTO= NO_LIBDW_DWARF_UNWIND= NO_LIBELF= NO_LIBNUMA=1 NO_LIBPERL=1 NO_LIBPFM4=1 NO_LIBPYTHON=1 NO_LIBTRACEEVENT= NO_LIBUNWIND=1 NO_LIBZSTD=1 NO_SDT=1 NO_SLANG=1 NO_LZMA=1 NO_ZLIB= TCMALLOC= WERROR=0 LIBDIR=/usr/libexec/perf-core libdir=/usr/lib64 plugindir=/usr/lib64/perf/plugins -f Makefile.perf install

Now I have a perf with still cycles and instructions not working with perf record, I wonder if there is a way at opensbi or kernel level to aggregate events to make it work properly, but I never had to look into perf internals so probably I poke it way later if nobody address it otherwise, anyway

perf record --group -e u_mode_cycle,m_mode_cycle,s_mode_cycle

produces something close enough for cycles, well u_mode_cycle is enough.

While for instructions the situation is a bit more annoying

perf record --group -e alu_inst,amo_inst,atomic_inst,fp_div_inst,fp_inst,fp_load_inst,fp_store_inst,load_inst,lr_inst,mult_inst,sc_inst,store_inst,unaligned_load_inst,unaligned_store_inst

is close to count all the scalar instructions, but trying to add vector_div_inst,vector_inst,vector_load_inst,vector_store_inst somehow makes perf record stop collecting samples silently, adding just 3 more events works though, so I guess I can be happy with u_mode_cycle,alu_inst,atomic_inst,fp_inst,vector_inst at least.

Bringing up BPI-F3 - Part 2

Luca Barbato — Mon, 20 May 2024 19:38:42 +0000

This is part2, in part1 I shown how to get a Gentoo system on the nvme taking some shortcuts. Here are the notes from the second day with some improvements.

Setting up the eMMC

stintel already documented how to build uboot and install it, I'm summarizing it as well here.

Dependencies

We need dtc to compile the device-tree and mkimage and git if we want to clone the trees.

emerge dtc u-boot-tools dev-vcs/git

Bootloader

This board boot process relies on opensbi + u-boot

First build opensbi

git clone https://github.com/BPI-SINOVOIP/pi-opensbi -b v1.3-k1
cd pi-opensbi
make PLATFORM=generic menuconfig
make -j 8

As stintel already reported, the default configuration is wrong, so we have to make sure that the build/platform/generic/kconfig/.config has the support for K1x:

# CONFIG_PLATFORM_SPACEMIT_K1PRO is not set
CONFIG_PLATFORM_SPACEMIT_K1X=y

If everything goes well build/platform/generic/firmware/fw_dynamic.bin will exist.

Then build u-boot

export OPENSBI=../pi-opensbi/build/platform/generic/firmware/fw_dynamic.bin
git clone https://github.com/BPI-SINOVOIP/pi-u-boot -b v2022.10-k1
cd pi-u-boot
make k1_defconfig
make -j8

If everything goes well it will produce the following files

FSBL.bin  
bootinfo_emmc.bin  
bootinfo_sd.bin  
bootinfo_spinand.bin  
bootinfo_spinor.bin 
u-boot.bin

Partitions

The upstream instructions suggest to copy over the layout, so the uboot, bootfs, rootfs partitions stay the same, but now the eMMC fsbl and opensbi are to live in their dedicated /dev/mmcblk2boot0. So it is up to you to do what you prefer with them.
Surely you have to setup /dev/mmcblk2boot0:

echo 0 > /sys/block/mmcblk2boot0/force_ro
dd if=bootinfo_emmc.bin of=/dev/mmcblk2boot0
dd if=FSBL.bin of=/dev/mmcblk2boot0 bs=512 seek=1

As stintel tested, all that's need is that a partition labeled uboot exists and contains u-boot.bin and a bootfs partition that contains env_k1-x.txt. It is a good idea to have env and rootfs as well.

Kernel

Building the kernel is straightforward enough:

git clone https://github.com/BPI-SINOVOIP/pi-linux -b linux-6.1.15-k1
ln -s pi-linux/ linux
cd linux
zcat /proc/config.gz > .config
make oldconfig # (or menuconfig)
make -j 10 && make modules_install
mount /boot # make sure it picks the right partition
make install

Make sure to enable HWMON and HWMON_NVME so you'll have the thermal sensors available.

Edit boot/env_k1-x.txt accordingly and your new kernel is ready.

NOTE: The initial kernel has at least a known annoying bug and it is missing HWMON support for its thermal sensors.
If you use Rust, you'll notice immediately some strange issue with futex().

here a patch to address the bug backported to the 6.1.15 vendored kernel.

here my tiny patch to get something going regarding HWMON.

Coming next

Do not use at all an initrd (right now if you are following along you'd be using the initrd from the starting distro)
Other u-boot tweaks

Bringing up BPI-F3 - Part 1

Luca Barbato — Sun, 19 May 2024 15:05:09 +0000

I like a lot non-x86 architectures, I still enjoy writing software for PowerPC even nowadays and I'm looking forward to see how SVE will be once there is hardware in the right price range.
Today it is the turn of RISC-V since there is eventually a, hopefully, nice board sporting a cpu with the RISC-V Vector Extension 1.0 that is now part of the RISC-V.

Where to start

The official documentation is available here.

The board comes with an empty eMMC, so either you start with a SD, or you need to upload a valid image using dfu.

Sadly the usb-c does not multiplex also the serial, so you'll need to connect the usual 3 pins with a 3.3V usbserial.

ERROR:   CMD8
ERROR:   sd f! l:76
bm:0
ERROR:   emmc: invalid bootinfo magic code:0x0
ERROR:   invalid bootinfo image.
ERROR:   entering download mode.
ROM: usb download handler
usb2d_initialize : enter
Force usb FS mode!
Controller Run
usb rst int
SETUP: 0x0 0x5 0x1
SETUP: 0x80 0x6 0x100
SETUP: 0x80 0x6 0x100
SETUP: 0x80 0x6 0x302
SETUP: 0x80 0x6 0x302
SETUP: 0x80 0x6 0x301
SETUP: 0x80 0x6 0x301
SETUP: 0x80 0x6 0x30a
SETUP: 0x80 0x6 0x30a
SETUP: 0x80 0x6 0x200
SETUP: 0x80 0x6 0x200
SETUP: 0x80 0x6 0x300

Their images are Debian derivatives, I prefer Gentoo and since we have stage3 for rv64_lp64d-openrc, I guess I can give it a try.

Vendor-sources for kernel and bootloaders are available and I can piggy back over that image if I stumble over opensbi or u-boot or I run out of time sorting out the patches for linux-6.1.15

Prerequisites

wezterm or screen to see what's going on
u-boot-tools are needed since the kernel image will would come in a FIT image.
The armbian tree or any of the pre-made builds provided.
A recent stage3

Lazy Setup

The base images share this layout:

Number  Start (sector)    End (sector)  Size       Code  Name
   1             256             767   256.0 KiB   8300  fsbl
   2             768             895   64.0 KiB    8300  env
   3            2048            4095   1024.0 KiB  8300  opensbi
   4            4096            8191   2.0 MiB     8300  uboot
   5            8192          532479   256.0 MiB   8300  bootfs
   6          532480              <>   <> GiB     8300  rootfs

With the rootfs taking the space it needs.

The easiest way is to copy the first 5 partitions to the eMMC

# dd if=/dev/mmcblk0 of=/dev/mmcblk2 bs=1M count=259

And then recreate the rootfs partition.

[!note] the initramfs does not support btrfs, nor the u-boot configuration. Until kernel and initramfs are replaced it is safer to stick to ext4 (I made this mistake).

Then the usual manual unpack of a stage3 and chroot described in Gentoo Handbook works.

Pitfalls known so far

Make sure to enable the serial ports in /etc/inittab

s0:12345:respawn:/sbin/agetty -L 115200 ttyS0 vt100
s1:12345:respawn:/sbin/agetty -L 115200 ttyS1 vt100

The default u-boot command script is incomplete, if you want to use the nvme, you have to enter the u-boot console (press s)

=> nvme scan
=> run nor_boot

Coming next

I hadn't tried yet to:

update u-boot and change a bit its configuration
build a new kernel
put everything together to have btrfs on nvme

Quick notes on using insta

Luca Barbato — Fri, 16 Dec 2022 14:29:35 +0000

Insta is a nice snapshot testing tool for Rust.

Snapshot testing

Snapshot tests are a more general extension of the normal tests we have in Rust.

Usually we use assert_eq! and similar macros. Insta let you deal with the reference values in a richer way and store them inside or outside your codebase.

Pitfalls to keep in mind

I spent way too much time stumbling upon a bunch of annoyances while converting some tests in rust-code-analysis so it is better to list them so you will avoid to suffer from similar mistakes:

cargo insta is git-aware, make sure to pass --no-ignore if your snapshots directory is populated by different means even better if you can use git-submodules.
If you deal with floating points, round them to the precision you care about. Different arches/libc may have different precisions
If you save paths keep in mind that Windows is special in this regard and you may have to manually serialize it.

Nice things to keep in mind

cargo test -- --nocapture provides enough information to see what went wrong.
cargo insta test --review makes adding more tests and make sure you do not break anything by mistake.

That's all for today

Templating inline asm in Rust

Luca Barbato — Mon, 05 Sep 2022 11:51:23 +0000

Kostya wrote about it and he asked me to figure out if what's he is missing isn't already available one way or another.

Before we start

For those not used to Multimedia, Rust or Assembly

Multimedia is all about processing data and serving/rendering it to the user at the right moment.
That requires to have a good control over latency and use the least amount of cpu.
This leads to use architecture specific extensions such as x86_64 AVX2 or ARM NEON.
Higher level languages such as C (or Rust) offer access to those extension via intrinsics, but quite often they are cumbersome enough that writing assembly as-is ends up being more pleasant.
You may look dav1d and rav1e for examples.

Rust and assembly

Rust had a fairly weak point in supporting assembly:

rustc does not compile .s or .S as gcc and clang do, and that makes you rely on cc-rs or nasm-rs
rustc until recently did not have a stable support for inline-assembly, and right now it has some useful parts still in nightly.

How far are we?

Kostya tried the current stable and he managed to write some assembly for his h264 decoder and get a 20% gain, but he had 3 issues:

He couldn't figure out how to manage sub-registers and the compiler warning wasn't that helpful for him since it assumes you know the format! terminology and you know asm! relates to it. this is being addressed
asm! operands in stable do not include sym and const.
Kostya couldn't figure out how to deal with templating the assembly using macro_rules!() as he is used to do with gcc.

I asked on zulip since I couldn't think of better ways than munching tokens and usually this means I'm missing something much simpler and obvious.
Luckily Amanieu helped us and this blog post is more or less about keeping notes.

Use cases for inline asm templating

In multimedia software you often write tiny kernels that operate over blocks of pixels, 4x4, 8x8, 16x16 and so on and usually the same inner logic is shared across and you ideally would like to not repeat yourself, even more if the very same logic can be shared across the many many extensions x86 has.

Kostya used this as example

fn avg_4(dst: &mut [u8], dstride: usize, src: &[u8], sstride: usize, bh: usize) {
    unsafe {
        asm!(
            "2:",
            "movd   xmm1, [{src}]",
            "movd   xmm3, [{src} + {sstride}]",
            "movd   xmm0, [{dst}]",
            "movd   xmm2, [{dst} + {dstride}]",
            "lea    {src}, [{src} + {sstride} * 2]",
            "pavgb  xmm0, xmm1",
            "pavgb  xmm2, xmm3",
            "movd   [{dst}], xmm0",
            "movd   [{dst} + {dstride}], xmm2",
            "lea    {dst}, [{dst} + {dstride} * 2]",
            "sub    {h}, 2",
            "jnz    2b",
            src = inout(reg) src.as_ptr() => _,
            sstride = in(reg) sstride,
            dst = inout(reg) dst.as_mut_ptr() => _,
            dstride = in(reg) dstride,
            h = inout(reg) bh => _,
            out("xmm0") _,
            out("xmm1") _,
            out("xmm2") _,
            out("xmm3") _,
        );
    }
}

fn avg_8(dst: &mut [u8], dstride: usize, src: &[u8], sstride: usize, bh: usize) {
    unsafe {
        asm!(
            "2:",
            "movq   xmm0, [{src}]",
            "movq   xmm1, [{src} + {sstride}]",
            "movq   xmm2, [{dst}]",
            "movq   xmm3, [{dst} + {dstride}]",
            "lea    {src}, [{src} + {sstride} * 2]",
            "pavgb  xmm0, xmm2",
            "pavgb  xmm1, xmm3",
            "movq   [{dst}], xmm0",
            "movq   [{dst} + {dstride}], xmm1",
            "lea    {dst}, [{dst} + {dstride} * 2]",
            "sub    {h}, 2",
            "jnz    2b",
            src = inout(reg) src.as_ptr() => _,
            sstride = in(reg) sstride,
            dst = inout(reg) dst.as_mut_ptr() => _,
            dstride = in(reg) dstride,
            h = inout(reg) bh => _,
            out("xmm0") _,
            out("xmm1") _,
            out("xmm2") _,
            out("xmm3") _,
        );
    }
}

fn avg_16(dst: &mut [u8], dstride: usize, src: &[u8], sstride: usize, bh: usize) {
    unsafe {
        asm!(
            "2:",
            "movaps xmm0, [{src}]",
            "movaps xmm1, [{src} + {sstride}]",
            "pavgb  xmm0, [{dst}]",
            "pavgb  xmm1, [{dst} + {dstride}]",
            "lea    {src}, [{src} + {sstride} * 2]",
            "movq   [{dst}], xmm0",
            "movq   [{dst} + {dstride}], xmm1",
            "lea    {dst}, [{dst} + {dstride} * 2]",
            "sub    {h}, 2",
            "jnz    2b",
            src = inout(reg) src.as_ptr() => _,
            sstride = in(reg) sstride,
            dst = inout(reg) dst.as_mut_ptr() => _,
            dstride = in(reg) dstride,
            h = inout(reg) bh => _,
            out("xmm0") _,
            out("xmm1") _,
        );
    }
}

Between avg_4 and avg_8 there is just a movd vs movq and for this Amalieu suggested to use the concat! pattern he uses in corosensei.

avg_4 and avg_8 would end up being

macro_rules! avg {
    ($name: ident, $mov:literal) => {
        fn $name(dst: &mut [u8], dstride: usize, src: &[u8], sstride: usize, bh: usize) {
            unsafe {
                asm!(
                    "2:",
                    concat!($mov, " xmm1, [{src}]"),
                    concat!($mov, " xmm3, [{src} + {sstride}]"),
                    concat!($mov, " xmm0, [{dst}]"),
                    concat!($mov, " xmm2, [{dst} + {dstride}]"),
                    "lea    {src}, [{src} + {sstride} * 2]",
                    "pavgb  xmm0, xmm1",
                    "pavgb  xmm2, xmm3",
                    concat!($mov, " [{dst}], xmm0"),
                    concat!($mov, " [{dst} + {dstride}], xmm2"),
                    "lea    {dst}, [{dst} + {dstride} * 2]",
                    "sub    {h}, 2",
                    "jnz    2b",
                    src = inout(reg) src.as_ptr() => _,
                    sstride = in(reg) sstride,
                    dst = inout(reg) dst.as_mut_ptr() => _,
                    dstride = in(reg) dstride,
                    h = inout(reg) bh => _,
                    out("xmm0") _,
                    out("xmm1") _,
                    out("xmm2") _,
                    out("xmm3") _,
                );
            }
        }
    }
}

avg!{avg_4};
avg!{avg_8};

To factorize away avg_8 and avg_16, you need a way to deal with the operands and while the asm statements are literals, the operands can be only expressed as tt in a macro_rules!.

The normal way to deal with this use-case is to rely on the preprocessor #if directives, using macro_rules you have to be a bit more creative.

Amalieu gave me another example and I ended up crafting:

macro_rules! avg_common {
    ($name:ident { $($load:literal),* } { $($store:literal),* } $($out:tt)*) => {
        fn $name(dst: &mut [u8], dstride: usize, src: &[u8], sstride: usize, bh: usize) {
            unsafe {
                asm!(
                    "2:",
                    $($load),*,
                    "lea    {src}, [{src} + {sstride} * 2]",
                    "pavgb  xmm0, xmm2",
                    "pavgb  xmm1, xmm3",
                    $($store),*,
                    "lea    {dst}, [{dst} + {dstride} * 2]",
                    "sub    {h}, 2",
                    "jnz    2b",
                    src = inout(reg) src.as_ptr() => _,
                    sstride = in(reg) sstride,
                    dst = inout(reg) dst.as_mut_ptr() => _,
                    dstride = in(reg) dstride,
                    h = inout(reg) bh => _,
                    $($out)*
                )
            }
        }
    }
}

macro_rules! avg {
    (avg_8) => {
        avg_common!{avg_8 {
                "movq   xmm0, [{src}]",
                "movq   xmm1, [{src} + {sstride}]",
                "movq   xmm2, [{dst}]",
                "movq   xmm3, [{dst} + {dstride}]"
            }
            {
                "movq   [{dst}], xmm0",
                "movq   [{dst} + {dstride}], xmm1"
            }
            out("xmm0") _,
            out("xmm1") _,
            out("xmm2") _,
            out("xmm3") _,
        }
    };
    (avg_16) => {
        avg_common!{avg_16 {
                "movaps xmm0, [{src}]",
                "movaps xmm1, [{src} + {sstride}]",
                "pavgb  xmm0, [{dst}]",
                "pavgb  xmm1, [{dst} + {dstride}]"
            }
            {
                "movq   [{dst}], xmm0",
                "movq   [{dst} + {dstride}], xmm1"
            }
            out("xmm0") _,
            out("xmm1") _,
        }
    };
}

More or less solving Kostya problem even if supporting multiple blocks of operands would require extra care.

Coming next

I had been quite busy with the SIFIS-Home project in particular writing a new implementation of WebOfThings in Rust, soon we'll release the first version supporting wot-1.1 and probably I'll write a bit about it.

A month in rav1e - February

Luca Barbato — Wed, 10 Mar 2021 10:33:06 +0000

rav1e is an AV1 encoder written in Rust

Here a quick summary of what happened last month. I will try to write a recap every month.

February Summary

We poured lots of work on improving the encoding speed, you may read some details of the journey:

Analyze our memory access patterns and improve the layout and the update strategy of a structure accessed a lot in our hottest code-path.
parallelize one of the remaining bottleneck so we improve the average thread usage and improve both speed and latency.
add the temporal rdo lookahead to our speed levels, measure its quality-vs-speed impact and retune them accordingly.

The benchmarks are prepared using speed-levels-rs.

The encoder is using the following settings:
--threads 16 --tiles 16 -l 100 <file> -o <encoded> -s <level>
The source file is Bosphorus from the ultravideo test sequences, the 1080p 10bit version is the 4k 10bit version scaled down, since it is not available on the website.

Overall our aarch64 support is getting fairly good, but there is still a lot of room for improvement on 8bit.

On the other hand there are 10bit optimizations it that aren't yet available for x86_64. Help in improving our SIMD coverage is very welcome :)

Digging deeper

x86_64

As expected the memory layout optimization that happened between p20210209 and p20210216 had the largest impact on the speed 0 and 1, while optimizing and tuning the temporal rdo lookahead computation has the largest impact on speed level 9 and 10.

Speed Level | p20210209 | p20210216 | p20210223 -- | -- | -- | -- 0 | **x1.23** | **x1.29** | x1.30 1 | **x1.20** | **x1.24** | x1.33 2 | x1.08 | x1.11 | x1.22 3 | x1.04 | x1.07 | x1.25 4 | x1.04 | x1.06 | x1.24 5 | x1.05 | x1.07 | x1.27 6 | x1.04 | x1.05 | x1.37 7 | x1.03 | x1.06 | x1.36 8 | x1.04 | x1.06 | x1.39 9 | x1.00 | x1.02 | **x1.52** 10| x1.00 | x1.01 | **x1.94**

The x86_64 10bit encoding is behaving similarly. Our SIMD support for it received a large boost in January and there is an ongoing effort to improve it even further in March.

Speed Level | p20210209 | p20210216 | p20210223 -- | -- | -- | -- 0 | **x1.12** | **x1.12** | x1.17 1 | **x1.10** | **x1.11** | x1.26 2 | x1.04 | x1.04 | x1.23 3 | x1.00 | x1.02 | x1.28 4 | x1.01 | x1.02 | x1.27 5 | x1.02 | x1.02 | x1.29 6 | x1.00 | x1.01 | x1.37 7 | x1.01 | x1.01 | x1.37 8 | x1.00 | x1.01 | x1.38 9 | x0.99 | x1.00 | **x1.50** 10 | x0.99 | x1.00 | **x1.95**

Aarch64

The impact of the optimizations on aarch64 had been more radical with a fairly large relative improvement on speed 10.

Speed Level |p20210209 | p20210216 | p20210223 -- | -- | -- | -- 0 | **x1.14** | **x1.15** | x1.31 1 | **x1.11** | **x1.10** | x1.59 2 | x1.03 | x1.03 | x1.63 3 | x1.03 | x1.01 | x1.76 4 | x1.01 | x1.01 | x1.77 5 | x1.02 | x1.01 | x1.88 6 | x1.02 | x1.00 | x2.07 7 | x1.01 | x1.00 | x2.07 8 | x1.02 | x1.00 | x2.10 9 | x1.00 | x0.99 | x2.45 10 | x1.01 | x0.98 | **x4.75**

The 10bit boost is not as extreme, but still substantial.

Speed Level | p20210209 | p20210216 | p20210223 -- | -- | -- | -- 0 | **x1.13** | **x1.17** | x1.30 1 | x1.08 | x1.11 | x1.54 2 | x1.02 | x1.05 | x1.57 3 | x1.00 | x1.03 | x1.66 4 | x1.00 | x1.02 | x1.67 5 | x1.00 | x1.03 | x1.74 6 | x1.00 | x1.02 | x1.87 7 | x1.00 | x1.03 | x1.87 8 | x1.00 | x1.02 | x1.89 9 | x0.99 | x1.01 | **x2.12** 10 | x0.98 | x1.02 | **x2.96**

I tested on some different aarch64 systems to see if there is a large difference in its behavior.

The Apple M1 is fairly different, but that's something I would expect. I will talk a bit more about it in other blogposts probably.

Coming next

We already landed additional SIMD for both x86_64 and aarch64, David Barr started working on improving the segment selection and I have eventually came up with the internals architecture that would give us a better thread pool usage while not impacting a lot the overall latency.

March is going to be exciting.

Temporal RDO update optimization

Luca Barbato — Sun, 28 Feb 2021 15:26:55 +0000

This is the second story of a specific optimization that gave a significant speed-up in rav1e. For the first one go here. For the next one go here

For readers unfamiliar with AV1 and rav1e, we will start with some background. If you already read the first article you may skip ahead to the implementation details.

The Alliance for Open Media is an effort founded by Google, Cisco and Mozilla, with many other companies joining them over the past few years. The Alliance for Open Media AV1 video codec is the first to be released as a result of this effort. We will refer to it as AV1.

rav1e is an AV1 encoder -- written primarily in the Rust programming language, its goal is to be safe and fast.

Architecture of an AV1 encoder

AV1 has a conventional block transform video coding architecture with motion compensation. A video is a sequence of frames which are divided into one or more tiles. Tiles are divided into large squares called superblocks -- either 64 or 128 pixels in width and height for AV1. Superblocks are partitioned recursively into smaller square or rectangular blocks.

Each block may be predicted by neighboring blocks in the same tile or by a motion projection of regions in other frames. There are many modes to choose from for both of these cases. The core of the video encoder is deciding how to partition the blocks, which prediction mode and reference to use for each block and how to transform the difference. The following diagram illustrates the range of techniques available in AV1.

Rate-distortion optimization (RDO)

The process for guiding encoding choices is called rate-distortion optimization, commonly abbreviated as RDO. For each set of choices there is a trade between the rate of bits that describe the decisions and how much the decoded frame is distorted compared to the original. The ratio of this trade-off can be estimated for a given quality range.

Temporal RDO

The RDO logic is normally applied over a small set of frames within a larger group. We call the smaller group sub-GOP and the larger group GOP.

The temporal RDO analyzes a larger set of frames to find areas that would recur outside the single sub-GOP and compute a bias that would guide the normal RDO decisions to invest more bits in the blocks that would be more important in the future.

The idea of temporal RDO is to compress objects well-predicted in future frames with higher quality. For example, a car moving across the screen can be encoded finely on the first frame, which will improve its visual quality on all subsequent frames.
On the other hand, an object that soon disappears can be encoded coarsely to save bits. This can also apply to multi-view images: spend more bits on objects visible in multiple views and less bits on objects visible in only a single view.
-- Ivan Molodetskikh

The temporal RDO needs to see moderately far in the future in order to be effective.

Every time a decision is made for a block, it has to propagate -- both within the frame and over all the past ones.

This caused its implementation to be one of the largest bottlenecks in the encoder:

As per every lookahead, it adds latency, potentially a lot of it.
The update process is inherently serial over the frames.

Temporal RDO implementation

In rav1e the bulk of the code implementing it lives in compute_block_importances().

Being it completely serial, once the encoding process is run on a machine with enough threads it takes a fairly large share of the overall encoding time.

In the case of encoding using 16 threads and 16 tiles, it takes a little over the 30% of the time spent on receive_packet for the speed level 10.

The rest of the encoding workload is spread nearly evenly per-tile, the computation of the block importance bias is not.

Making it faster

The update process is fairly serial if we consider frames and the per-block updates. But the analysis itself has good parallelism opportunities since it happens per block.

Enter rayon

rayon is a data-parallelism crate that makes it incredibly easy to take an iterator and execute it in parallel using a pool of workers. We use it extensively in rav1e.

After a refactor¹ to split the update loop in one that computes the per-block costs and one that propagates the importance biases to its neighbors, all it took was an apparently small change:

diff --git a/src/api/internal.rs b/src/api/internal.rs
index 395a66e4..b2ab16cd 100644
--- a/src/api/internal.rs
+++ b/src/api/internal.rs
@@ -21,6 +21,7 @@ use crate::rate::{
   RCState, FRAME_NSUBTYPES, FRAME_SUBTYPE_I, FRAME_SUBTYPE_P,
   FRAME_SUBTYPE_SEF,
 };
+use crate::rayon::prelude::*;
 use crate::scenechange::SceneChangeDetector;
 use crate::stats::EncoderStats;
 use crate::tiling::Area;
@@ -809,14 +810,14 @@ impl<T: Pixel> ContextInner<T> {
     let plane_org = &frame.planes[0];
     let plane_ref = &reference_frame.planes[0];
     let lookahead_intra_costs_lines =
-      fi.lookahead_intra_costs.chunks_exact(fi.w_in_imp_b);
+      fi.lookahead_intra_costs.par_chunks_exact(fi.w_in_imp_b);
     let block_importances_lines =
-      fi.block_importances.chunks_exact(fi.w_in_imp_b);
+      fi.block_importances.par_chunks_exact(fi.w_in_imp_b);

     let costs: Vec<_> = lookahead_intra_costs_lines
       .zip(block_importances_lines)
       .enumerate()
-      .flat_map(|(y, (lookahead_intra_costs, block_importances))| {
+      .flat_map_iter(|(y, (lookahead_intra_costs, block_importances))| {
         lookahead_intra_costs
           .iter()
           .zip(block_importances.iter())

On speed level 10, the time spent on compute_block_importances() is reduced to nearly 1/3.

Across the speed levels the impact is the following

As expected, making the temporal-rdo update parallel made our quicker speed levels fairly faster.

Given that the RDO rollback optimization had a larger impact on our slower speed levels, it compounds nicely.

After this optimization we retuned a little the speed levels, changing the default rdo-lookahead depth.

https://github.com/xiph/rav1e/commit/705fb22f191c8468313a9b32ac9796887c7f93b6 ↩

Forem: Luca Barbato

Bringing up an AmpereOne system

Initial Setup

Partition and untar

Enter the stage

Initial bits to set up

cargo-c common questions

Top 6 questions, and their answers

Why this exists, why cargo isn't doing all of this on its own?

I prefer doing it on my own, why should I use cargo-c

I'm fine with just giving a static library, the user can copy it as they want

Does cargo-c require lots to set up?

How secure is cargo-c?

Since cargo-c is a cargo extension, why not move it in cargo?

In closing

Is cargo/rustc that slow?

P.S.

cross-stages experiments

Support status for BPI-F3 upstream

My experiments

Usage

Bringing up BPI-F3 - Part 3

Initramfs

Remoteproc firmware

Coming next

Bringing up BPI-F3 - Part 2.5

Getting perf to work up to a point

Building perf

Bringing up BPI-F3 - Part 2

Setting up the eMMC

Dependencies

Bootloader

Partitions

Kernel

Coming next

Bringing up BPI-F3 - Part 1

Where to start

Prerequisites

Lazy Setup

Pitfalls known so far

Coming next

Quick notes on using insta

Snapshot testing

Pitfalls to keep in mind

Nice things to keep in mind

Templating inline asm in Rust

Before we start

Rust and assembly

How far are we?

Use cases for inline asm templating

Coming next

A month in rav1e - February

February Summary

Digging deeper

x86_64

Aarch64

Coming next

Temporal RDO update optimization

Architecture of an AV1 encoder

Rate-distortion optimization (RDO)

Temporal RDO

Temporal RDO implementation

Making it faster

Enter rayon