Rpi 4 meets Flutter and Rust

AdaShoelace — Fri, 06 Nov 2020 16:05:05 +0000

At Charlie foxtrot we aim to stay relevant and keep up with new and exciting tech. So how do we do that? We experiment! This time we wanted to capitalize on our in-house competence in Flutter and Rust. Our Flutter expert, Jonathan, had read about Darts foreign function interface (ffi) capabilities and wanted to try it in conjunction with Rust. He asked me if I thought if we could build something that would run on a Raspberry pi (rpi). Without having a clue, I said “YES!” and got to work. The resulting code can be found here.

After some debating, we landed in an idea where we would have a Flutter interface that let the user take a photo via the rpi camera module and then display the image in the app. This is a simple app but considering that Flutter don’t have official support for the rpi, this could be quite the challenge. We also decided to build a small (read: very small) library in Rust to communicate with the camera.

This project has quite a few requirements:

Rust
Flutter
flutter-pi
rpi 4 4gb ram
an rpi camera module
a touch screen, we went with the Hyperpixel 4.0.

Setup

First thing to do is to set up the Raspberry Pi. In the README of the flutter-pi repo it's mentioned that flutter-pi should run without Xserver. Therefore I went with a headless version of RaspberryOS. Once you've flashed the os to a ssd card, follow the instructions in the flutter-pi README and install drivers for your touch screen of choice. Be sure to clone the engine-binaries branch of flutter-pi and place the files in the correct directories aswell.

On the software side we first want to set up Flutter on our laptop. If you are using a linux distro you may have the option to install and run Flutter via snap. I found that this leads to several issues. Because of that I cloned the Flutter repo from Github and installed it manually. Where you install Flutter doesn’t really matter as long as you put /flutter/bin in your $PATH.

Next you want to make sure you are using a Flutter version that’s compatible with flutter-pi. At the time of writing this the compatible version is 1.22.2. You can change Flutter-version by checking out the corresponding branch. You can now test your installation by running flutter create <name> followed by flutter build bundle. If it builds without errors, everything should be correct.

We will also need to install Rust so that we can build the small library that lets us talk to the camera. Do this by going over to the Rust website and follow the instructions. Once Rust is installed we need to add the armv7 target for the rust compiler in order for us to cross compile the code. Do this with: rustup component add rust-std-armv7-unknown-linux-gnueabihf. Now create a Rust project using cargo new –-lib ffi-test. The last part (ffi-test) is the name of the project and doesn’t matter, choose a name to your liking. As I mentioned earlier we need to cross compile the code since it’s going to run on the rpi and not the machine we are writing the code on. Create a directory called .cargo and place a file in there named config.toml. In that file place the following:

[build] 
target = "armv7-unknown-linux-gnueabihf"  

[target.armv7-unknown-linux-gnueabihf] 
linker = "arm-linux-gnueabihf-gcc"

The last part after “linker =” is the gcc linker for arm. Install this if you don’t already have it on your machine. Google the correct name of the package for your distro. The name displayed here is the package name on Manjaro.
We also need to declare, inside the Cargo.toml, that we want to build a dynamically linked library (.so) like this:

[lib]
name = "ffi_test"
crate-type = ["cdylib"] #dynamic library

The next logical step would be to build a small "Hello world" library in Rust and consume it from Dart/Flutter. Lets create a function that returns a hard coded string when called.

#[no_mangle]
pub extern fn string_from_rust() -> *const c_char {
    let s = CString::new("Hello from Rust!").unwrap(); // Create a CString from "Hello world"
    let p = s.as_ptr(); // Get a pointer to the underlaying memory for s
    std::mem::forget(s); // Give up the responsibility of cleaning up/freeing s
    p
}

Since Dart's ffi mechanisms utilizes the C ABI, we need to annotate the Rust function with #[no_mangle] in order to preserve the function signatures symbol upon compilation. Since we are compiling this into a .so we need to mark the function as extern so that it can be linked as any other library. The functions return type is *const c_char, which is a raw pointer, basically a raw memory adress. We have to do this since Rust's types are not available in Dart. When we send data over to Dart, Rust can no longer take responsibility for cleaning up that data. Rust no longer have the required information regarding that data's usage and could therefore free the data at an inappropriate time. To build this just run cargo b

Now we want to consume the Rust function from the Dart side of the project. To do that we need the following dependencies:

import 'dart:ffi' as ffi;
import 'dart:ffi';
import 'dart:typed_data';
import 'package:flutter/material.dart';
import 'package:ffi/ffi.dart';

Now we can load and use the Rust library:

typedef NativeRustStringFromRustFunction = ffi.Pointer<Utf8> Function();

class _MyHomePageState extends State<MyHomePage> {
  _MyHomePageState() {
      this._dl =
          ffi.DynamicLibrary.open("libffi_test.so");
      this._string_from_rust_ffi =
          _dl.lookupFunction<NativeRustStringFromRustFunction, NativeRustStringFromRustFunction>(
          "string_from_rust");
  }
  int _counter = 0;
  String _message = "Photos taken: 0";
  ffi.DynamicLibrary _dl;
    NativeRustStringFromRustFunction _string_from_rust_ffi;
  void _updateMessage {
    setState(() {
      _counter++;
      var foreignMessage = ffi.Utf8.fromUtf8(_string_from_rust_ffi());
      _message = "'$foreignMessage': '$_counter'";
    });
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text(widget.title),
      ),
      body: Center(
        child: Column(
          mainAxisAlignment: MainAxisAlignment.center,
          children: <Widget>[
            Text(_message),
            Text(
              '$_counter',
              style: Theme.of(context).textTheme.headline4,
            ),
          ],
        ),
      ),
      floatingActionButton: FloatingActionButton(
        onPressed: _updateMessage,
        tooltip: 'Increment',
        child: Icon(Icons.add),
      ),
    );
  }
}

We first load the .so library we compiled from Rust, we look up the function and store it in the state. We then write a small helper function that will call the _string_from_rust_ffi-function. Then we have to encode the string returned from Rust to UTF8 using Utf8.fromUtf8().

In order to test this enable desktop support with:

$ flutter channel dev
$ flutter upgrade
$ flutter config --enable-linux-desktop

Check the config with flutter devices and see that your os shows up. Then to run the app run flutter run -d linux. You should something like this:

Moving over to the Rpi

The main goal in this post is to get Flutter running on the rpi while consuming a Rust library. So far we haven't run anything on the rpi. Lets fix that!

Flutter does not have any official support for rpi, that's why we need to use the aforementioned flutter-pi. If you have followed the instructions in the repo for flutter-pi you should have it installed on the rpi. It would be ideal if we could just take the code we built earlier and throw it on to the rpi and run it. Unfortunately that's not the case. The engine-binariesbranch from the flutter-pi repo needs to be cloned to the "host" (the computer you are coding on) in order for us to build and compile the Flutter app. Once you've done that go through the build steps in the README of flutter-pi. To save myself time I've saved these as bash scripts. These are listed below.

build_snapshot.sh

#!/bin/bash
~/flutter/bin/cache/dart-sdk/bin/dart \
    ~/flutter/bin/cache/dart-sdk/bin/snapshots/frontend_server.dart.snapshot \
    --sdk-root ~/flutter/bin/cache/artifacts/engine/common/flutter_patched_sdk_product \
    --target=flutter \
    --aot \
    --tfa \
    -Ddart.vm.product=true \
    --packages .packages \
    --output-dill build/kernel_snapshot.dill \
    --verbose \
    --depfile build/kernel_snapshot.d \
    package:hello_world/main.dart

build_so.sh

#!/bin/sh
~/engine-binaries/gen_snapshot_linux_x64 \
  --causal_async_stacks \
  --deterministic \
  --snapshot_kind=app-aot-elf \
  --elf=build/app.so \
  --strip \
  --sim_use_hardfp \
  --no-use-integer-division \
  build/kernel_snapshot.dill

build.sh

#!/bin/bash
flutter clean && \
    flutter build bundle --release && \
    ./build_snapshot.sh && \
    ./build_so.sh \

To build for flutter-pi now run the build.shscript.
Upload the files to the rpi using the upload.sh script:

#!/bin/bash
rsync -a --info=progress2 ./build/flutter_assets/ pi@raspberrypi:/home/pi/flutter_hello_world_assets && \
    scp ./build/app.so pi@raspberrypi:/home/pi/flutter_hello_world_assets/app.so

To run the application ssh into the rpi and run flutter-pi flutter_hello_world_assets. If all goes well you should see the "Hello world" app from Flutter and once you press the button the text changes to "Hello from rust.

Incorporating the camera

Now that we know how to compile and run Rust and Flutter in conjunction on the rpi we want to be able to control the camera from our app. There might be libraries for controlling the camera written in Dart but since we want to use Rust together with Dart, we're going to do it from Rust.

I've chosen a library called rascam. It's a small library built on top of mmal-sys. We can add rascam to our Rust project by adding this to the Cargo.toml file:

[dependencies]
rascam = "*"

Now lets take a look at the code for taking an image.

// ffi-test/src/lib.rs
use std::ffi::CString;
use std::os::raw::c_char;
use rascam::*;

// ...

#[repr(C)]
pub struct ImageBuffer {
    img_ptr: *mut u8,
    len: u32,
}

#[no_mangle]
pub extern fn take_photo() -> *mut ImageBuffer {
    let info = info().unwrap();
    if info.cameras.len() < 1 {
        println!("Found 0 cameras. Exiting");
        // note that this does not run destructors
        ::std::process::exit(1);
    }
    println!("{}", info);
    let mut camera = SimpleCamera::new(info.cameras[0].clone()).unwrap();
    camera.activate().unwrap();
    let mut bytes = camera.take_one().unwrap();
    let len = bytes.len() as u32;
    let ret = bytes.as_mut_ptr();
    std::mem::forget(bytes);
    let mut ib = Box::new(ImageBuffer { img_ptr: ret, len});
    let ret = Box::into_raw(ib);
    std::mem::forget(ret);
    ret
}

There is a lot going on here. If youve never ventures in to the unforgiving territory of low level Rust, this might look a little daunting. But fear not! I will explain it all, piece by piece.

On a high level this code solves two problems, take a photo and send it to whoever is calling the function. We will start with the second problem, sending the photo to the caller. Since it's just two values we we want to pass on, one might be tempted to utilize a tuple like (*mut u8, u32). This is however not a viable option since the memory layout of a tuple in Rust is not guaranteed. So we went with a simple struct. The #[repr(C)] tells the compiler to treat the struct as a C struct. This means that the fields of the struct will be aligned the same way in memory as we declared them in the code. We need this because the C ABI being the common grounds for communication between Rust and Dart.

We start with getting the necessary info for the camera. We use this to check if there even is a camera to begin with. If there isn't we just close the process.

  let info = info().unwrap();
  if info.cameras.len() < 1 {
      println!("Found 0 cameras. Exiting");
      // note that this does not run destructors
      ::std::process::exit(1);
  }
  println!("{}", info);

We then instatiate the camera using the info and activate it.

 let mut camera = SimpleCamera::new(info.cameras[0].clone()).unwrap();
 camera.activate().unwrap();

We take a photo, get the number of bytes, get a pointer to the byte array and tell the compiler to forget about the byte array. Just as with the string earlier this array will be cleaned up by Dart.

  let mut bytes = camera.take_one().unwrap();
  let len = bytes.len() as u32;
  let ret = bytes.as_mut_ptr();
  std::mem::forget(bytes);

Next we create an ImageBuffer and place it on the heap by wrapping it in a Box. If we didn't place it on the heap and still returned a pointer to it, the ImageBuffer would be popped of the stack once the function call returns. That would make it possible for the memory to get overwritten while Dart uses the pointer. We don't want that!

  let mut ib = Box::new(ImageBuffer { img_ptr: ret, len});
  let ret = Box::into_raw(ib);
  std::mem::forget(ret);
  ret

With that everything is well and done on the Rust side but how do we actually use this in Dart?
First will need to declare a type to represent the ImageBuffer in Dart.

class ImageBuffer extends Struct {
  Pointer<Uint8> img_ptr; 

  @Uint32()
  int len;
}

then we need a type for the actual function providing us the ImageBuffer and we need to look it up in the library:

  typedef NativeRustTakePhotoFunction = ffi.Pointer<ImageBuffer> Function();
  // inside MyHomePageState
  _MyHomePageState() {
      this._dl =
          ffi.DynamicLibrary.open("/home/pi/target/debug/libffi_test.so");
      this._take_photo_ffi =
          _dl.lookupFunction<NativeRustTakePhotoFunction, NativeRustTakePhotoFunction>(
          "take_photo");
  }

We then declare a function to be called on button press:

  void _takePhoto() {
    setState(() {
      _imageBuffer = _take_photo_ffi().ref; // _imageBuffer is a member of our state
      _counter++;
      _message = "Photos taken: '$_counter'";
    });
  }

Once all that is done we can register the function with the button:

  // Inside a widget array in a scaffold
  if (_imageBuffer != null) 
    Image.memory(_imageBuffer.img_ptr.asTypedList(_imageBuffer.len)),

Unfortunately I couldn't build the Rust library on my Linux machine because of mmal-sys. There might be a solution for this. But to build this I:

pushed the Rust library code to github
built the flutter bundle on the host, using the scripts
uploaded it to the rpi and ran it. It then gave me this:

Now you should hopefully have a Flutter app running on a rpi!

Takeaways

For me the main takeaway is that our approach here might actually be a viable option for resource scarce devices. The possibility of writing a beautiful, interactive application in Flutter and have run in a car infotainment system for example is quite exciting. The possibility to handle the low level stuff with languages such as Rust and C, really opens up this domain for Flutter.
I at least know that I am waiting for official rpi support for Flutter so that I can build something more robust!

Forem: AdaShoelace

Rpi 4 meets Flutter and Rust

Setup

Moving over to the Rpi

Incorporating the camera

Takeaways