Forem: j1ngzoue

Bridging 8th Wall AR and React Three Fiber: How Pose Data Flows into Three.js

j1ngzoue — Tue, 17 Mar 2026 04:09:14 +0000

What I Built

I created a React Three Fiber (R3F) wrapper library for the 8th Wall open-source AR engine, called @j1ngzoue/8thwall-react-three-fiber.

It lets you add image-tracking AR to a React app with minimal boilerplate:

<EighthwallCanvas xrSrc="/xr.js">
  <EighthwallCamera />
  <ImageTracker targetImage="/targets/marker.json">
    <mesh>
      <boxGeometry />
      <meshStandardMaterial color="hotpink" />
    </mesh>
  </ImageTracker>
</EighthwallCanvas>

Point your phone camera at the target image, and the 3D object appears anchored to it.

Two Canvases, One Screen

The trickiest part of the architecture is layering XR8's camera feed with R3F's 3D scene. They run on separate canvases stacked on top of each other:

┌─────────────────────────────┐
│  R3F Canvas (alpha=true)    │  ← 3D objects, transparent background
├─────────────────────────────┤
│  XR8 Canvas                 │  ← camera feed
└─────────────────────────────┘

Both canvases are position: absolute and fill the container. R3F renders with alpha: true so the camera feed shows through.

<div style={{ position: 'relative', width: '100%', height: '100%' }}>
  {/* XR8 renders camera feed here */}
  <canvas ref={xrCanvasRef} style={fillStyle} />

  {/* R3F renders 3D scene on top, transparent */}
  <Canvas style={fillStyle} gl={{ alpha: true }}>
    {children}
  </Canvas>
</div>

XR8 is initialized with the back canvas:

XR8.run({ canvas: xrCanvasRef.current })

How XR8 Pose Data Flows into Three.js

XR8 uses a camera pipeline module system. You register a module with named hooks, and XR8 calls them each frame.

Step 1: Read pose data from XR8

Every frame, XR8 calls onUpdate with detection results. We extract the pose for our target image:

XR8.addCameraPipelineModule({
  name: 'image-tracker-marker',
  onUpdate: ({ processCpuResult }) => {
    const detectedImages = processCpuResult?.reality?.detectedImages
    // [{ name, position: {x,y,z}, rotation: {x,y,z,w}, scale }]

    const pose = detectedImages?.find((img) => img.name === 'marker')
    latestPoseRef.current = pose ?? null
  },
})

We store the latest pose in a ref — not state — because we don't want a re-render every frame.

Step 2: Apply pose to a Three.js group in useFrame

R3F's useFrame runs once per render frame. We read the latest pose and apply it directly to the <group> that wraps the AR content:

useFrame(() => {
  const pose = latestPoseRef.current
  if (!pose || !groupRef.current) return

  groupRef.current.position.set(
    pose.position.x,
    pose.position.y,
    pose.position.z,
  )
  groupRef.current.quaternion.set(
    pose.rotation.x,
    pose.rotation.y,
    pose.rotation.z,
    pose.rotation.w,
  )
  groupRef.current.scale.setScalar(pose.scale)
})

Step 3: Show/hide on detection events

XR8 fires events when a target is found or lost. We use these to toggle visibility:

listeners: [
  {
    event: 'reality.imagefound',
    process: ({ detail }) => {
      if (detail.name !== targetName) return
      setVisible(true)
    },
  },
  {
    event: 'reality.imagelost',
    process: ({ detail }) => {
      if (detail.name !== targetName) return
      setVisible(false)
      latestPoseRef.current = null
    },
  },
],

The final JSX renders a <group> whose visibility and transform are driven entirely by XR8:

return (
  <group ref={groupRef} visible={visible}>
    {children}
  </group>
)

Syncing the Camera Matrix

The 3D scene also needs to match the physical camera's field of view. XR8 provides videoWidth and videoHeight from the device camera, which we use to estimate the FOV and update the Three.js camera matrix each frame:

XR8.addCameraPipelineModule({
  name: 'camera-sync',
  onStart: ({ videoWidth, videoHeight }) => {
    // Estimate FOV from video aspect ratio
    activeFov = estimateFovFromVideo(videoWidth, videoHeight)
  },
  onUpdate: ({ processCpuResult }) => {
    const cameraProjectionMatrix =
      processCpuResult?.reality?.cameraProjectionMatrix
    if (cameraProjectionMatrix) {
      // Store for use in useFrame
      latestMatrixRef.current = cameraProjectionMatrix
    }
  },
})

useFrame(({ camera }) => {
  if (latestMatrixRef.current) {
    camera.projectionMatrix.fromArray(latestMatrixRef.current)
    camera.projectionMatrixInverse
      .copy(camera.projectionMatrix)
      .invert()
    camera.matrixAutoUpdate = false
  }
})

Without this, 3D objects would appear at the wrong depth and scale relative to the real world.

Key Takeaways

Use two stacked canvases to combine XR8's camera feed with R3F's transparent 3D scene
XR8 pose data flows through a camera pipeline module → ref → useFrame → Three.js group transform
Store pose in a ref, not state, to avoid unnecessary re-renders every frame

- Sync the Three.js camera projection matrix with XR8's data so depth and scale match the real world

The library is on npm:

npm install @j1ngzoue/8thwall-react-three-fiber

GitHub: https://github.com/activeguild/8thwall-react-three-fiber

Notes on using the Sass Package Importer with Vite

j1ngzoue — Sun, 22 Sep 2024 13:15:41 +0000

With the introduction of Sass's Package Importer feature, loading Sass files has become simpler and more flexible. This article explains how to use Vite in combination with Sass's Package Importer and how to deal with errors you may encounter when doing so.

What is Sass's Package Importer?

The Package Importer feature in Sass, released in 2024, is a revolutionary feature that allows you to import Node.js packages directly. It is more convenient than the traditional @import statement and is suitable for modern Sass projects. This feature simplifies project dependency management and eliminates the need to manually specify the Sass file for a package.

More details are explained in an announcement on the official Sass blog. It seems that most CSS frameworks can use it without any changes at the publisher.
https://sass-lang.com/blog/announcing-pkg-importers/

Also, in line with Package Importer support,

Vite and Sass Integration

Vite is a module bundler popular for its speed and ease of use. It is particularly well suited for front-end development and integrates smoothly with Sass.

Starting with Vite:5.4.0, support for the Sass moder api has been added. If you want to use it, you need to upgrade to version 5.4.0 or higher.
https://github.com/vitejs/vite/blob/v5.4.6/packages/vite/CHANGELOG.md#540-beta0-2024-07-30

You also need to set importers in vite.config.ts as follows
https://sass-lang.com/documentation/js-api/classes/nodepackageimporter/

import { defineConfig } from “vite”;.
import { NodePackageImporter } from “sass-embedded”;

export default defineConfig({
  css: {
    preprocessorOptions: {
      scss: {
        api: “modern”, { scss: { preprocessorOptions: { scss: { preprocessorOptions
        importers: [new NodePackageImporter()], }
      }, }
    }, }
  }, }
});

Internally at build time, the compile function is now called.

Build errors that occur with Vite and their causes

When using the Package Importer with Vite using the sass package, I encountered the following error at build time.

[vite:css] [sass] Error: TypeError: importer.findFileUrl is not a function

This is due to a conflict when trying to use the Vite dependency sass-embedded and the traditional Sass package, as they are recognized as different instances of NodePackageImporter. This error is due to the fact that the instances used inside Sass and Vite do not match.
It is hard to tell from the error, but the error is actually in the following section
https://github.com/sass/embedded-host-node/blob/59dde59163a58a33aa8180dbf8e23d2216913712/lib/src/importer-registry.ts#L84-L109

Solution to the error

To work around this error, you need to change your Vite settings appropriately. Specifically, it is best to proceed with development assuming that Vite does not import Sass packages separately, but uses sass-embedded provided by Vite. official.

This issue is also raised in GitHub Issue #16438. Instead of using the traditional Sass package, you should set css.preprocessorOptions to assume sass-embedded. Note that it is or sass but it doesn't work.

I thought I didn't get an error not too long ago?

I checked and sass-embedded was included in peerDependencies at this time, so it must have been an error since then.
https://github.com/vitejs/vite/commit/1025bb6d8f21c0cb9fe72405d42e0f91bb3f1d8e

Also, it seems that sass-embedded is preferred, so you need to use sass-embedded when using modern in any case.
https://github.com/vitejs/vite/blob/f9691767ad763720065cc7c5c7f369f97b4e7ea8/packages/vite/src/node/plugins/css.ts#L2030-L2046

Since I was using npm at the time of verification, I think the result would have been different if I had resolved peerDependencies and used pnpm for the installation.
Later, I also confirmed that if I actually used pnpm and installed only sass, the build finished successfully with the following code.

import { defineConfig } from “vite”;.
import { NodePackageImporter } from “sass”;

export default defineConfig({
  css: {
    preprocessorOptions: {
      scss: {
        api: “modern”, { importers:[new NodePackageImporter } from “sass”; export default
        importers: [new NodePackageImporter()], }
      }, }
    }, }
  }, }
});

Efficient CSS loading

This should go without saying, but when using the Sass package importer, if you load the entire package like @use “pkg:bulma”; all CSS will be included in the build. This may result in a large CSS file after the build, as many unnecessary styles are included.

In such cases, it is recommended to specify and load only the necessary components. For example, if you want to use only some components of Bulma, specify as follows

@use “pkg:bulma/sass/components/breadcrumb”;.

This way, only the necessary CSS is included in the build, thus compressing the post-build stylesheet and optimizing file size. Even so, unnecessary css will probably be included. In that case, you may need to write your own purge process.

If you want to use only common functions or utilities, it is better to set them to additionalData and load them as before.

import { defineConfig } from “vite”;.
import react from “@vitejs/plugin-react”;
import { NodePackageImporter } from “sass-embedded”;

export default defineConfig({
  css: {
    preprocessorOptions: {
      scss: {
        api: “modern”, importers: [new NodePackageImporter()], {
        importers: [new NodePackageImporter()], { additionalData: `@use “pkg:bulma/sass/utilities” as
        additionalData: `@use “pkg:bulma/sass/utilities” as utilities;;`,
      }, }
    }, }
  }, }
  plugins: [react()], `@use “pkg:bulma/sass/utilities” as utilities;;`, }, }
});

Build USD and bring USD up to date in usdz tool

j1ngzoue — Fri, 01 Mar 2024 10:25:02 +0000

We use the latest versions of OpenUSD and usdPython as of 2024/03/01.

Introduction

Apple's usdPython python version is stuck at 3.7.9, so build and use USD with the latest python version available for OpenUSD.
- USDZ tool
- OpenUSD

Available Python versions can be found here.

The version as of 2024/03/01 was 3.9.13.

Step 1 Preparation before build

Since we are using 3.7.9 python, install the latest version with pyenv
The latest was 3.12.2

pyenv install 3.12.2

pyenv local 3.12.2

In python, we also want to be able to run python3, so run the following

alias python="python3"

In USD, the usdView command uses pyside6, so install it.
You can also use pyside2.

python -m pip install pyside6

Step 2 Build

Peel off unnecessary things with option (be patient as it takes time to build)

python USD/build_scripts/build_usd.py /path/to/my_usd_install_dir --no-examples --no-tests --no-docs --no-ptex --no-openvdb --no-draco --no --materialx --no-examples

Dealing with build errors

Install pyOpenGL to deal with the error.

PyOpenGL is not installed; if pip is installed, run "pip install PyOpenGL" to install it, then re-run this script.
If PyOpenGL is already installed, you may need to update your PYTHONPATH to indicate where PyOpenGL is located.

pip install PyOpenGL

Step 3 Replace

The U.S. dollar/ distribution is replaced. After replacing, see here.

Step 4 Run usdzconvert

usdzconvert . /cube.glb ~/cube.usdz

I get an error.

Input file: /Users/xxxxxx/Documents/cube.glb
Traceback (most recent call last):
  File "/Users/xxxxxx/projects/USDPython/./usdzconvert/usdzconvert", line 860, in <module>
    errorValue = main()
  File "/Users/xxxxxx/projects/USDPython/./usdzconvert/usdzconvert", line 855, in main
    return tryProcess(sys.argv[1:])
  File "/Users/xxxxxx/projects/USDPython/./usdzconvert/usdzconvert", line 810, in tryProcess
    ret = process(argumentList)
  File "/Users/xxxxxx/projects/USDPython/./usdzconvert/usdzconvert", line 650, in process
    usdStage = usdStageWithGlTF_module.usdStageWithGlTF(srcPath, tmpPath, legacyModifier, openParameters)
  File "/Users/xxxxxx/projects/USDPython/usdzconvert/usdStageWithGlTF.py", line 1529, in usdStageWithGlTF
    return converter.makeUsdStage()
  File "/Users/xxxxxx/projects/USDPython/usdzconvert/usdStageWithGlTF.py", line 1512, in makeUsdStage
    self.processNodeChildren(self.gltf['scenes'][0]['nodes'], self.asset.getGeomPath(), None)
  File "/Users/xxxxxx/projects/USDPython/usdzconvert/usdStageWithGlTF.py", line 1399, in processNodeChildren
    self.processNode(nodeIdx, path, underSkeleton, indent)
  File "/Users/xxxxxx/projects/USDPython/usdzconvert/usdStageWithGlTF.py", line 1366, in processNode
    usdGeom = self.processMesh(nodeIdx, newPath, underSkeleton)
  File "/Users/xxxxxx/projects/USDPython/usdzconvert/usdStageWithGlTF.py", line 1316, in processMesh
    usdGeom = self.processPrimitive(nodeIdx, gltfPrimitives[0], path, skinIdx, underSkeleton)
  File "/Users/xxxxxx/projects/USDPython/usdzconvert/usdStageWithGlTF.py", line 1142, in processPrimitive
    normalPrimvar = usdGeom.CreatePrimvar('normals', Sdf.ValueTypeNames.Normal3fArray, UsdGeom.Tokens.vertex)
AttributeError: 'Mesh' object has no attribute 'CreatePrimvar'

Check the issue and fix usdzconvert/usdStageWithGlTF.py.
https://github.com/PixarAnimationStudios/OpenUSD/issues/2628
https://github.com/activeguild/USDPython/commit/482afe50eba109c7d68b082edef003af4ec2e279

Confirmation that usdzconverter terminates normally with the above modification.

Animated texture library for react-three-fiber available in react's hooks base

j1ngzoue — Thu, 14 Dec 2023 07:21:04 +0000

Introduction

We have created a simple library that will load animation textures by specifying an image file (gif/png/apng) file.
Repository

Frames are parsed using gifuct-js and UPNG.js. The frame parsing process has a large impact on the main thread, so it is executed on the web worker.

It can be used without preparing a sprited image.

Install

npm i animation-texture

Usage

import React, { useRef, useEffect } from "react";
import * as THREE from "three";
import { useAnimationTexture } from "animation-texture";

interface Props {
  url: string;
}

export function Model({ url }: Props) {
  const { animationTexture } = useAnimationTexture({ url });
  const meshRef = useRef();

  useEffect(() => {
    if (meshRef.current && animationTexture) {
      meshRef.current.material.map = animationTexture;
      meshRef.current.material.needsUpdate = true;
    }
  }, [animationTexture]);

  return (
    <mesh ref={meshRef} position={new THREE.Vector3(0, 0, 0)}>
      <planeGeometry args={[1, 1]} />
      <meshBasicMaterial transparent side={THREE.FrontSide} />
    </mesh>
  );
}

Pre-load if necessary.

import React from "react";
import * as THREE from "three";
import { preLoad } from "animation-texture";

export default function App() {
  preLoad('/sample.png');
  return ...
}

Demo

I believe there is still significant room for performance improvement.

Type-safe use of css custom-property in Vite

j1ngzoue — Fri, 30 Dec 2022 12:45:27 +0000

Introduce

When staring with css modules in react, I often encounter situations where custom properties are used.
When staring with css modules in react, I often encounter situations where custom properties are used.
We have released a tool that solves those problems, and we would like to introduce it to you.
If you are using Vite, you can read and use css-preprocessoroptions.

Repo

https://github.com/activeguild/css-custom-property-extractor

Usage

Install the package.

npm i -D css-custom-property-extractor

For example. Parsing bootstrap.scss outputs the following typescript file.

npx ccpe -i ./samples/scss/bootstrap.scss

output.

/**
 *  #0d6efd;
 */
export const bsBlue = "var(--bs-blue)"
/**
 *  #6610f2;
 */
export const bsIndigo = "var(--bs-indigo)"
/**
 *  #6f42c1;
 */
export const bsPurple = "var(--bs-purple)"
...

Import and use the output typescript file.

At the end

We have shown how to bring custom properties into the TS world and use them in type safety.
In addition to this, please read on to find out how to load styles and use classes in a type-safe manner.
https://dev.to/activeguild/i-made-a-typescript-plugin-for-css-modules-that-can-be-used-with-vite-3ofg
https://dev.to/activeguild/type-safe-development-with-vite-x-react-x-css-modules-x-typescript-1ebc

I made a plugin to remember todo comments on the code in vite

j1ngzoue — Wed, 01 Jun 2022 02:27:29 +0000

Introduce

Often we forget the todo left on the code.
IDE extensions and other features are added, but because of differences in development environments among members and the fact that they are not always visible, they are often noticed just before the release of a product.

We believe that incorporating it as part of the build may prevent it.
Recently, we have been using vite a lot, so we created a plugin that can be used with vite.
see plugin.

Usage

Install

npm i -D vite-plugin-hit-word

Add it to vite.config.ts

A deadline can be set on the code by assigning a date. If the deadline is exceeded, it will be output in red on the terminal.

import { defineConfig } from 'vite'
import hitWord from 'vite-plugin-hit-word'

// https://vitejs.dev/config/
export default defineConfig({
  plugins: [
    hitWord({
      include: ['src/*'],
      exclude: ['node_modules'],
      word: { value: '[todo]:', hasDeadlineDate: true },
    }),
  ],
})

e.g.

The following files are available.

import styles from './App.module.css'

function App() {
  // [todo]: [2022/05/22] issue #765

  return (
    <div className="App">
      <header className={styles.header}></header>
    </div>
  )
}

export default App

At build time, output to the terminal as follows.
You can jump to the corresponding code from the path.

/Users/XXXXXX/projects/Sample/src/App.tsx(4) :>>   // [todo]: [2022/05/22] issue #765

I have shown an example with todo, but it can be changed to suit your project.

Feedback is welcome.

Migration CSS modules to vanilla-extract

j1ngzoue — Fri, 13 May 2022 04:48:34 +0000

Introduction

We have created a tool to generate typescript files for vanilla-extract, a Zero-runtime Stylesheets, from css (scss/sass) style definitions.
https://github.com/activeguild/css-to-vanilla-extract

The tools described here can be tried in the playground.

The following APIs are covered.

Motivation

Convert style definitions received from designers without any errors.
Cost-effective migration of existing projects using css modules.

e.g.

input

.foo {
  background-color: blue;
}
@media (min-width: 1200px) {
  .foo {
    font-size: 5rem;
    color: red;
  }
}

output

import { style } from "@vanilla-extract/css";
export const foo = style({
  backgroundColor: "blue",
  "@media": {
    "(min-width: 1200px)": {
      color: "red",
      fontSize: "5rem",
    },
  },
});

This tool was created on a trial basis.I think css modules and other css can be converted with little or no problem.

It looks like using picocolors is better than chalk.

j1ngzoue — Mon, 24 Jan 2022 03:10:55 +0000

I think it is common to use the chalk library to add color to the text output to the console.It is also large and dependent on other libraries.By using a library called picocolors, we can expect to improve the size and speed of the package.The usability is almost the same, so it will be available as usual.

Taken from the library READEME

I made a prismjs plugin for esbuild.

j1ngzoue — Thu, 20 Jan 2022 01:29:51 +0000

We have created a plugin that allows you to use the same settings when migrating from prismjs + webpack + babel to prismjs + esbuild.
It's easy to use because you just need to migrate the language and plugin information set in .babelrc to the optional arguments of the new plugin.
https://github.com/activeguild/esbuild-plugin-prismjs

🔍Visualizing the Calling Hierarchy of React Components in the Browser😊

j1ngzoue — Tue, 04 Jan 2022 16:46:00 +0000

Introduce

It is very difficult to grasp the whole picture of a component just by looking at the code such as vscode. We often use tools such as Figma to create diagrams and sequences, but this is a very costly process. In order to solve this problem, we have created and released a library to visualize react components as diagrams in a browser. It is still a work in progress, so many considerations may be missing.

Demo

react-component-analyzer is a library that is useful when exchanging PR or design discussions with friends, as it allows you to visualize the component tree as a diagram and discuss it based on a common understanding. It will also help you to understand the scope of impact when you modify the implementation.
I hope it helps you to design better components.

Point

Using the popular library, @typescript-eslint/typescript-estree AST to do the analysis.
By using VSCode's schema in the Browser, you can quickly check the implementation of components you are interested in.
If you are using vite without setting it in the config file, you can resolve the alias from vite.config.ts.
It uses prismjs, which allows for code reading that is more like an IDE.

Demo

Install

npm i -D visualize-react-component

Usage

By specifying the React root file and the component files, it will parse them and output the stats.html file.

npm run rca ./src/main.tsx

I'm waiting for your feedback.

Operation on the browser of Github is fast with k

j1ngzoue — Wed, 24 Nov 2021 04:27:36 +0000

Once you get used to operating Github on your browser, it will be faster to use commands.
You can also search for files in the repository, so it seems that you can reduce the frequency of switching to the mouse.

https://docs.github.com/ja/get-started/using-github/github-command-palette

Resolve types using the typescript plugin for CSS modules available in vite

j1ngzoue — Thu, 11 Nov 2021 01:23:56 +0000

You can resolve the read type of preprocessorOptions set in thevite.config.ts file.
Click here for the package repository

Install

npm i -D ts-css-modules-vite-plugin

Add it to the `tsconfig.json`

{
  "compilerOptions": {
    ...
    "plugins": [{"name": "ts-css-modules-vite-plugin"}]
  },
}

Demo

Resolve the `vite.config.ts`

Resolve the preprocessorOptions setting within the plugin.

import path from "path";
import { defineConfig } from "vite";

export default defineConfig({
  css: {
    preprocessorOptions: {
      scss: {
        additionalData: `@use "@/styles" as common;`,
        importer(...args) {
          if (args[0] !== "@/styles") {
            return;
          }

          return {
            file: `${path.resolve(__dirname, "./src/assets/styles")}`,
          };
        },

    },
  },
});

Forem: j1ngzoue

Bridging 8th Wall AR and React Three Fiber: How Pose Data Flows into Three.js

What I Built

Two Canvases, One Screen

How XR8 Pose Data Flows into Three.js

Step 1: Read pose data from XR8

Step 2: Apply pose to a Three.js group in useFrame

Step 3: Show/hide on detection events

Syncing the Camera Matrix

Key Takeaways

- Sync the Three.js camera projection matrix with XR8's data so depth and scale match the real world

Notes on using the Sass Package Importer with Vite

What is Sass's Package Importer?

Vite and Sass Integration

Build errors that occur with Vite and their causes

Solution to the error

I thought I didn't get an error not too long ago?

Efficient CSS loading

Build USD and bring USD up to date in usdz tool

Introduction

Step 1 Preparation before build

Step 2 Build

Dealing with build errors

Step 3 Replace

Step 4 Run usdzconvert

Animated texture library for react-three-fiber available in react's hooks base

Introduction

Install

Usage

Pre-load if necessary.

Demo

Type-safe use of css custom-property in Vite

Introduce

Repo

Usage

At the end

I made a plugin to remember todo comments on the code in vite

Introduce

Usage

Install

Add it to vite.config.ts

e.g.

Migration CSS modules to vanilla-extract

Introduction

Motivation

e.g.

input

output

It looks like using picocolors is better than chalk.

I made a prismjs plugin for esbuild.

🔍Visualizing the Calling Hierarchy of React Components in the Browser😊

Introduce

Point

Demo

Install

Usage

Operation on the browser of Github is fast with k

Resolve types using the typescript plugin for CSS modules available in vite

Install

Add it to the tsconfig.json

Demo

Resolve the vite.config.ts

Add it to the `tsconfig.json`

Resolve the `vite.config.ts`