rs4j: Building a JNI Framework

RedstoneWizard08 — Tue, 01 Oct 2024 19:04:53 +0000

Hey! This is my second post, technically, but it's my first real one. (Just gonna ignore the t3d post.)

~~I also haven't used this account in forever but whatever~~

In this post, I'm going to cover what rs4j is, how to use it, and how I built it.

What is it?

rs4j is a Rust library I created to ease the creation of Java libraries that use native code written in Rust. It generates JNI (Java Native Interface) code to accomplish this.

Okay, that's cool, but why should I care?

rs4j allows you to offload high-computation work to a much faster runtime (looks at you, garbage collector) instead of running it all in the JVM and ruining performance. Minecraft mods like Create Aeronautics (or Create Simulated, to be more correct) use this technique to do some of their physics calculations that would otherwise be very laggy with Java.

rs4j allows you to easily create native interfaces like this with minimal code and easily port entire libraries for use with Java with minimal code.

Okay, now I'm interested - but how can I use it?

Using it is easy! Just follow these steps:

Edit: The step numbers are broken, I don't know why. :(

Set up your library type:

# Cargo.toml

[lib]
crate-type = ["cdylib"]

Add rs4j to your dependencies:

cargo add rs4j

Add rs4j to your build dependencies:

cargo add rs4j --build -F build # Enable the `build` feature

# Also add anyhow for error handling
cargo add anyhow --build

Set up your buildscript:

// build.rs

use rs4j::build::BindgenConfig;
use anyhow::Result;

fn main() -> Result<()> {
    // Make a new config
    BindgenConfig::new()

        // Set the package for export
        .package("your.package.here")

        // Where to save the Rust bindings
        .bindings(format!("{}/src/bindings.rs", env!("CARGO_MANIFEST_DIR")))

        // Where the input files are
        .glob(format!("{}/bindings/**/*.rs4j", env!("CARGO_MANIFEST_DIR")))?

        // Where to save java classes (is a directory)
        .output(format!("{}/java", env!("CARGO_MANIFEST_DIR")))

        // Enable JetBrains annotations (this is a TODO on my end)
        .annotations(true)

        // Go!
        .generate()?;

    Ok(())
}

Set up your post-build script (optional):

rs4j uses a post-build script to accomplish actions after building.
This is technically optional, but recommended.

# Cargo.toml

[features]
default = []
post-build = ["rs4j/build", "anyhow"]

[[bin]]
name = "post-build"
path = "post-build.rs"
required-features = ["post-build"]

[dependencies]
anyhow = { version = "[...]", optional = true } # Set the version to whatever you want
rs4j = "[...]" # Whatever you had before

// post-build.rs

use anyhow::Result;
use rs4j::build::BindgenConfig;

fn main() -> Result<()> {
    let out_path = format!("{}/generated", env!("CARGO_MANIFEST_DIR"));
    let src_path = format!("{}/java/src/generated", env!("CARGO_MANIFEST_DIR"));

    BindgenConfig::new()
        // This should be the same as the normal buildscript
        .package("com.example")
        .bindings(format!("{}/src/bindings.rs", env!("CARGO_MANIFEST_DIR")))
        .glob(format!("{}/bindings/**/*.rs4j", env!("CARGO_MANIFEST_DIR")))?
        .output(&out_path)
        .annotations(false)

        // Run post-build actions
        .post_build()?

        // Copy it to your Java project
        .copy_to(src_path)?;

    Ok(())
}

Install rs4j's CLI (optional)

This is optional if you don't want to use the post-build script.

cargo install rs4j --features cli

Build!

Modify any scripts like so:

- cargo build
+ rs4j build # `rs4j build` supports all of `cargo build`'s arguments after a `--`.

Syntax

Here's a basic run-down of the syntax:

// This class, Thing, takes in one type parameter, `A`.
// You can omit this if it doesn't take any type parameters.
class Thing<A> {
    // This makes it so that Rust knows that the type for `A`
    // will have `Clone + Copy`. This doesn't change anything
    // on the Java side, it's just so that Rust will compile.
    bound A: Clone + Copy;

    // This will generate getters and setters for the field `some`.
    field some: i32;

    // Here, the Rust function's name is `new`, and Java will treat
    // it as a constructor.
    static init fn new(value: A) -> Thing;

    // This gets the value. Since this is in snake_case, rs4j will
    // automatically convert it into camelCase, renaming this to
    // `getValue` on the Java side.
    fn get_value() -> A;

    // This marks this function as mutable, meaning in Rust it will
    // mutate the struct, as if it took a `&mut self` as an argument.
    mut fn set_value(value: A);

    // You can even include trait methods, as long as Rust can find the
    // trait it belongs to!
    fn clone() -> A;
};

How was it made?

rs4j uses a peg parser to process its language. This parser directly turns the parsed structure into an abstract syntax tree, that is turned into code.

rs4j is strongly-typed. I have a Type struct and a TypeKind enum to accomplish this.

`TypeKind`s

These are parsed using this code:

parser! {
    /// The rs4j parser.
    pub grammar rs4j_parser() for str {
        ...

        // Type kinds

        rule _u8_k() -> TypeKind = "u8" { TypeKind::U8 }
        rule _u16_k() -> TypeKind = "u16" { TypeKind::U16 }
        rule _u32_k() -> TypeKind = "u32" { TypeKind::U32 }
        rule _u64_k() -> TypeKind = "u64" { TypeKind::U64 }
        rule _i8_k() -> TypeKind = "i8" { TypeKind::I8 }
        rule _i16_k() -> TypeKind = "i16" { TypeKind::I16 }
        rule _i32_k() -> TypeKind = "i32" { TypeKind::I32 }
        rule _i64_k() -> TypeKind = "i64" { TypeKind::I64 }
        rule _f32_k() -> TypeKind = "f32" { TypeKind::F32 }
        rule _f64_k() -> TypeKind = "f64" { TypeKind::F64 }
        rule _bool_k() -> TypeKind = "bool" { TypeKind::Bool }
        rule _char_k() -> TypeKind = "char" { TypeKind::Char }
        rule _str_k() -> TypeKind = "String" { TypeKind::String }
        rule _void_k() -> TypeKind = "()" { TypeKind::Void }
        rule _other_k() -> TypeKind = id: _ident() { TypeKind::Other(id) }
        rule _uint_k() -> TypeKind = _u8_k() / _u16_k() / _u32_k() / _u64_k()
        rule _int_k() -> TypeKind = _i8_k() / _i16_k() / _i32_k() / _i64_k()
        rule _float_k() -> TypeKind = _f32_k() / _f64_k()
        rule _extra_k() -> TypeKind = _bool_k() / _char_k() / _str_k() / _void_k()

        ...
    }
}

As you can see, there's a different rule for each primitive type, and then a catch-all. This allows me to verify and output the correct code easily.

You can see more of the parser here.

Codegen

rs4j uses a custom codegen system that heavily uses format!() to create the code. While this isn't the most correct or safe, it creates correct code in almost all of my tests (the only issue is generics, which I'm working on).

The codegen is done with each AST node having its own functions to turn it into Java and Rust code.

Native Implementation

In your lib.rs, you have to include!() your bindings.rs file, which contains the native implementations.

Each struct you generate bindings for will be wrapped with JNI. Here's an example of what this looks like:

class MyOtherStruct {
    field a: String;
    field b: MyStruct;

    static init fn new() -> Self;

    fn say_only(message: String);
    fn say(p2: String);
    fn say_with(p1: MyStruct, p2: String);
};

// lib.rs

...

#[derive(Debug)]
pub struct MyOtherStruct {
    pub a: String,
    pub b: MyStruct,
}

impl MyOtherStruct {
    pub fn new() -> Self {
        Self {
            a: String::new(),
            b: MyStruct::new(),
        }
    }

    pub fn say_only(&self, message: String) {
        println!("{}", message);
    }

    pub fn say(&self, p2: String) {
        println!("{}{}", self.b.a, p2);
    }

    pub fn say_with(&self, p1: MyStruct, p2: String) {
        println!("{}{}", p1.a, p2);
    }
}

include!("bindings.rs");

// bindings.rs
// #[allow(...)] statements have been removed for brevity.

#[allow(non_camel_case_types)]
pub struct __JNI_MyOtherStruct {
    pub a: String,
    pub b: *mut MyStruct,
}

impl __JNI_MyOtherStruct {
    pub unsafe fn of(base: MyOtherStruct) -> Self {
        Self {
            a: base.a.clone(),
            // yes, this is an intentional memory leak.
            b: Box::leak(Box::new(base.b)) as *mut MyStruct,
        }
    }

    pub unsafe fn to_rust(&self) -> MyOtherStruct {
        MyOtherStruct {
            a: self.a.clone(),
            b: (&mut *self.b).clone(),
        }
    }

    pub unsafe fn __wrapped_new() -> Self {
        let base = MyOtherStruct::new();

        Self::of(base)
    }

    pub unsafe fn __wrapped_say_only(&self, message: String) -> () {
        MyOtherStruct::say_only(&self.to_rust(), message).clone()
    }

    pub unsafe fn __wrapped_say(&self, p2: String) -> () {
        MyOtherStruct::say(&self.to_rust(), p2).clone()
    }

    pub unsafe fn __wrapped_say_with(&self, p1: MyStruct, p2: String) -> () {
        MyOtherStruct::say_with(&self.to_rust(), p1, p2).clone()
    }
}

When an object is constructed, it calls the wrapped method which intentionally leaks every nested object to get its pointer. This allows me to access the object whenever I need to, in any context.

All methods are wrapped to allow JNI to call them much more easily.

JNI Code

Speaking of which, the JNI code looks like this:

// This is a field, here's the getter and setter.
// #[allow(...)] statements have been removed for brevity.

#[no_mangle]
pub unsafe extern "system" fn Java_com_example_MyOtherStruct_jni_1set_1a<'local>(
    mut env: JNIEnv<'local>,
    class: JClass<'local>,
    ptr: jlong,
    val: JString<'local>,
) -> jlong {
    let it = &mut *(ptr as *mut __JNI_MyOtherStruct);
    let val = env.get_string(&val).unwrap().to_str().unwrap().to_string();

    it.a = val;

    ptr as jlong
}

#[no_mangle]
pub unsafe extern "system" fn Java_com_example_MyOtherStruct_jni_1get_1a<'local>(
    mut env: JNIEnv<'local>,
    class: JClass<'local>,
    ptr: jlong,
) -> jstring {
    let it = &*(ptr as *mut __JNI_MyOtherStruct);
    env.new_string(it.a.clone()).unwrap().as_raw()
}

This is pretty standard stuff for the jni crate, except for accessing the object. That &*(ptr as *mut __JNI_MyOtherStruct) might look unsafe, and that's because it is. This is intentional, however, as the pointer should always be valid if done correctly.

Notice that at the end of the setter, it returns the pointer of the object. This is intended. This allows Java to reset its internal pointer, keeping track of the latest valid pointer.

Freeing Memory

Freeing memory essentially reclaims the pointer and then drops it. It frees all non-primitive fields too.

// #[allow(...)] statements have been removed for brevity.

#[no_mangle]
pub unsafe extern "system" fn Java_com_example_MyOtherStruct_jni_1free<'local, >(_env: JNIEnv<'local>, _class: JClass<'local>, ptr: jlong) {
    // Reclaim the pointer
    let it = Box::from_raw(ptr as *mut __JNI_MyOtherStruct);

    // Reclaim the other field
    let _ = Box::from_raw(it.b);
}

There is a known bug with this method, however, which is that the method will always end up leaking memory if there is a nested object more than one level deep. I have some ideas on how to fix this, but I've been focused on other things.

The Java Side

Every Java class that rs4j generates will inherit from two other interfaces, ParentClass, and NativeClass.

Here's the definition of both:

// NativeClass.java

package org.stardustmodding.rs4j.util;

public interface NativeClass {
    long getPointer();
}

// ParentClass.java

package org.stardustmodding.rs4j.util;

public interface ParentClass {
    void updateField(String field, long pointer);
}

Each class is made up of a few parts, including:

JNI Methods

// Notice how all of these functions take a `long ptr` as an argument. This is the pointer to the underlying struct in Rust.

// This is a constructor - it takes no pointer but returns one.
private native long jni_init_new();

// Methods
private static native void jni_say_only(long ptr, String message);
private static native void jni_say(long ptr, String p2);
private static native void jni_say_with(long ptr, long p1, String p2);

// Getters & Setters
private static native long jni_set_a(long ptr, String value);
private static native String jni_get_a(long ptr);
// Notice how this field isn't primitive, so it uses the pointer instead.
private static native long jni_set_b(long ptr, long value);
private static native long jni_get_b(long ptr);

// Freeing memory
private static native void jni_free(long ptr);

Fields

// The pointer to the Rust object
private long __ptr = -1;
// If this is a field in another class, it keeps track of it for updating purposes
private ParentClass __parent = null;
// The name of the field in the other class
private String __parentField = null;

Constructors

public MyOtherStruct() {
    // Sets the pointer using the constructor
    __ptr = jni_init_new();
}

Methods

// Notice how these all just call the JNI method, providing the pointer.

public void sayOnly(String message) {
    jni_say_only(__ptr, message);
}

public void say(String p2) {
    jni_say(__ptr, p2);
}

public void sayWith(MyStruct p1, String p2) {
    jni_say_with(__ptr, p1.getPointer(), p2);
}

Fields

// Notice how the setters all update the field in the parent. This allows the user to have Java-like behavior, where modifying a class that is a property of another will update that reference.

public void setA(String value) {
    __ptr = jni_set_a(__ptr, value);

    if (__parent != null) {
        __parent.updateField(__parentField, __ptr);
    }
}

public String getA() {
    return jni_get_a(__ptr);
}

public void setB(MyStruct value) {
    // .getPointer() gets the underlying pointer, this is from the NativeClass interface.
    __ptr = jni_set_b(__ptr, value.getPointer());

    if (__parent != null) {
        __parent.updateField(__parentField, __ptr);
    }
}

public MyStruct getB() {
    // Essentially this is a glorified cast.
    return MyStruct.from(jni_get_b(__ptr), this, "b");
}

Default constructors

// Just creates an instance from a pointer.
private MyOtherStruct(long ptr) {
    __ptr = ptr;
}

// Creates an instance from a pointer, with a parent
private MyOtherStruct(long ptr, ParentClass parent, String parentField) {
    __ptr = ptr;
    __parent = parent;
    __parentField = parentField;
}

// These are for other classes to "cast" to this class.

public static MyOtherStruct from(long ptr) {
    return new MyOtherStruct(ptr);
}

public static MyOtherStruct from(long ptr, ParentClass parent, String parentField) {
    return new MyOtherStruct(ptr, parent, parentField);
}

And finally, default methods.

// I'M FREE!!!!
// This is ESSENTIAL for memory management, as Rust will otherwise never know when to free the memory that was leaked.
public void free() {
    jni_free(__ptr);
}

// Override from NativeClass.
@Override
public long getPointer() {
    return __ptr;
}

// Override from ParentClass.
@Override
public void updateField(String field, long pointer) {
    // `b` is non-primitive, so when it's updated it also has to be updated here.
    if (field == "b") {
        __ptr = jni_set_b(__ptr, pointer);
    }
}

Wrapping Up

This project is probably one of my proudest projects right now, as it's taken so much work and is proving to be pretty useful for me. I hope you'll check it out and play around with it, too!

Anyway, see you in the next one! I'll try to post more often if I can!

Special Thanks

Thanks to @RyanHCode for giving me a few tips on this!

Terrible 3D: A new (and horrible) 3D game engine.

RedstoneWizard08 — Fri, 28 Oct 2022 13:49:51 +0000

Terrible 3D (also known as t3d)

So, for my coding club (which I run after school days), I've had to try to develop a 3D game engine from scratch, because I love pain and I want to overcomplicate things.

Jokes aside, the real reason is that I find Babylon.js and THREE.js a bit hard to work with, even for someone like me with six years of JavaScript under my belt, so I wanted to create a more simple engine for my club members to use, since I'm teaching them the first things they know about coding in JavaScript, and I don't think using THREE.js or Babylon.js would be very good with absolute beginners, even with a math teacher supervising the club.

Choosing a tech stack

For my project, I decided to go with TypeScript for the main language, since it has to be made in JavaScript (for the most part) to support usage in JavaScript, and I don't want to torture myself because of type errors.

I chose to use Vite to package and test my project, since it has the library mode feature to act as a blazingly fast compiler.

For styling elements (like the size of the canvas, and, in the future, HUD/GUI elements that are a second canvas), I chose to use SASS (via .scss files), since it allows me to not suffer whilst coding css styles.

To debug, I'm using Puppeteer with a little script I made to redirect console error output to a linux shell. (My school blocked the inspector on our Chromebooks, where I do most of my coding.)

Also, I'm working on rewriting the core engine module in Rust and compiling that to WASM so I can achieve better performance. Even with minified and optimized JavaScript code it was still slow in WebGL and WebGL2.

How do I install it?

If someone hasn't already taken it (it's not published yet), here are some example commands:

# With NPM:
npm install @t3d/core

# With Yarn:
yarn add @t3d/core

# With pnpm:
pnpm add @t3d/core
# -- or --
pnpm install @t3d/core

Where's the code?

Here's the code: https://github.com/ISB-Coding-Club/t3d
Contributions are welcome, and so are feature requests, but I can't guarantee I'll have the time to look at them too quickly because of the mountain of homework I'll always have.

Forem: RedstoneWizard08

rs4j: Building a JNI Framework

What is it?

Okay, that's cool, but why should I care?

Okay, now I'm interested - but how can I use it?

Syntax

How was it made?

`TypeKind`s

Codegen

Native Implementation

JNI Code

Freeing Memory

The Java Side

Wrapping Up

Special Thanks

Links

Terrible 3D: A new (and horrible) 3D game engine.

Terrible 3D (also known as t3d)

Choosing a tech stack

How do I install it?

Where's the code?

Forem: RedstoneWizard08

rs4j: Building a JNI Framework

What is it?

Okay, that's cool, but why should I care?

Okay, now I'm interested - but how can I use it?

Syntax

How was it made?

TypeKinds

Codegen

Native Implementation

JNI Code

Freeing Memory

The Java Side

Wrapping Up

Special Thanks

Links

Terrible 3D: A new (and horrible) 3D game engine.

Terrible 3D (also known as t3d)

Choosing a tech stack

How do I install it?

Where's the code?

`TypeKind`s