Forem: Niklas Lepistö

Building a realtime eye tracking experience with Supabase and WebGazer.js

Niklas Lepistö — Tue, 17 Dec 2024 19:25:39 +0000

TL;DR:

Built with Supabase, React, WebGazer.js, Motion One, anime.js, Stable Audio

Leverages Supabase Realtime Presence & Broadcast (no database tables used at all!)

GitHub repo

Website

Demo video

Yet another Supabase Launch Week Hackathon and yet another experimental project, called Gaze into the Abyss. This ended up being both one of the simplest and complex projects at the same time. Luckily I've been enjoying Cursor quite a bit recently, so I had some helping hands to make it through! I also wanted to validate a question in my mind: is it possible to use just the realtime features from Supabase without any database tables? The (maybe somewhat obvious) answer is: yes, yes it is (love you, Realtime team ♥️). So let's dive a bit deeper into the implementation.

The idea

I was just one day randomly just thinking about Nietzsche's quote about the abyss and that it would be nice (and cool) to actually visualize it somehow: you stare into a dark screen and something stares back at you. Nothing much more to it!

Building the project

Initially I had the idea that I would use Three.js to make this project, however I realized it'd mean that I'd need to create or find some free assets for the 3D eye(s). I decided that it's a bit too much, especially since I didn't have too much time to work on the project itself, and decided to do it in 2D with SVGs instead.

I also did not want it to be only visual: it would be a better experience with some audio too. So I had an idea that it would be awesome if the participants could talk to a microphone and others could hear it as ineligible whispers or winds passing by. This, however, turned out very challenging and decided to drop it completely as I wasn't able to hook up WebAudio and WebRTC together well. I do have a leftover component in the codebase which listens to the local microphone and triggers "wind sounds" for the current user if you want to take a look. Maybe something to add in the future?

Realtime rooms

Before working on any visual stuff, I wanted to test out the realtime setup I had in mind. Since there are some limitations in the realtime feature I wanted it work so that:

There are max. 10 participants in one channel at a time
- meaning you'd need to join a new channel if one is full
You should only see other participants' eyes

For this I came up with an useEffect setup where it recursively joins to a realtime channel like so:

This joinRoom lives inside the useEffect hook and is called when the room component is mounted. One caveat I found out when working on this feature was that the currentPresences param does not contain any values in the join event even though it is available. I'm not sure if it's a bug in the implementation or working as intended. Hence needing to do a manual room.presenceState fetch to get the number of participants in the room whenever the user joins.

We check the participant count and either unsubscribe from the current room and try to join another room, or then proceed with the current room. We do this in the join event as sync would be too late (it triggers after join or leave events).

I tested this implementation by opening a whole lot of tabs in my browser and all seemed swell!

After that I wanted to debug the solution with mouse position updates and quickly ran into some issues of sending too many messages in the channel! The solution: throttle the calls.

/**
 * Creates a throttled version of a function that can only be called at most once 
 * in the specified time period.
 */
function createThrottledFunction<T extends (...args: unknown[]) => unknown>(
  functionToThrottle: T,
  waitTimeMs: number
): (...args: Parameters<T>) => void {
  let isWaitingToExecute = false

  return function throttledFunction(...args: Parameters<T>) {
    if (!isWaitingToExecute) {
      functionToThrottle.apply(this, args)
      isWaitingToExecute = true
      setTimeout(() => {
        isWaitingToExecute = false
      }, waitTimeMs)
    }
  }
}

Cursor came up with this little throttle function creator and I used it with the eye tracking broadcasts like this:


const throttledBroadcast = createThrottledFunction((data: EyeTrackingData) => {
  if (currentChannel) {
    currentChannel.send({
      type: 'broadcast',
      event: 'eye_tracking',
      payload: data
    })
  }
}, THROTTLE_MS)

throttledBroadcast({
 userId: userId.current,
 isBlinking: isCurrentlyBlinking,
 gazeX,
 gazeY
})

This helped a lot! Also, in the initial versions I had the eye tracking messages sent with presence however broadcast allows more messages per second, so I switched the implementation to that instead. It's especially crucial in eye tracking since the camera will record everything all the time.

Eye tracking

I had ran into WebGazer.js a while back when I first had the idea for this project. It's a very interesting project and works suprisingly well!

The whole eye tracking capabilities are done in one function in a useEffect hook:

    window.webgazer
      .setGazeListener(async (data: any) => {
        if (data == null || !currentChannel || !ctxRef.current) return

        try {
          // Get normalized gaze coordinates
          const gazeX = data.x / windowSize.width
          const gazeY = data.y / windowSize.height

          // Get video element
          const videoElement = document.getElementById('webgazerVideoFeed') as HTMLVideoElement
          if (!videoElement) {
            console.error('WebGazer video element not found')
            return
          }

          // Set canvas size to match video
          imageCanvasRef.current.width = videoElement.videoWidth
          imageCanvasRef.current.height = videoElement.videoHeight

          // Draw current frame to canvas
          ctxRef.current?.drawImage(videoElement, 0, 0)

          // Get eye patches
          const tracker = window.webgazer.getTracker()
          const patches = await tracker.getEyePatches(
            videoElement,
            imageCanvasRef.current,
            videoElement.videoWidth,
            videoElement.videoHeight
          )

          if (!patches?.right?.patch?.data || !patches?.left?.patch?.data) {
            console.error('No eye patches detected')
            return
          }

          // Calculate brightness for each eye
          const calculateBrightness = (imageData: ImageData) => {
            let total = 0

            for (let i = 0; i < imageData.data.length; i += 16) {
              // Convert RGB to grayscale
              const r = imageData.data[i]
              const g = imageData.data[i + 1]
              const b = imageData.data[i + 2]
              total += (r + g + b) / 3
            }
            return total / (imageData.width * imageData.height / 4)
          }

          const rightEyeBrightness = calculateBrightness(patches.right.patch)
          const leftEyeBrightness = calculateBrightness(patches.left.patch)
          const avgBrightness = (rightEyeBrightness + leftEyeBrightness) / 2

          // Update rolling average
          if (brightnessSamples.current.length >= SAMPLES_SIZE) {
            brightnessSamples.current.shift() // Remove oldest sample
          }
          brightnessSamples.current.push(avgBrightness)

          // Calculate dynamic threshold from rolling average
          const rollingAverage = brightnessSamples.current.reduce((a, b) => a + b, 0) / brightnessSamples.current.length
          const dynamicThreshold = rollingAverage * THRESHOLD_MULTIPLIER
          // Detect blink using dynamic threshold
          const blinkDetected = avgBrightness > dynamicThreshold

          // Debounce blink detection to avoid rapid changes
          if (blinkDetected !== isCurrentlyBlinking) {
            const now = Date.now()
            if (now - lastBlinkTime > 100) { // Minimum time between blink state changes
              isCurrentlyBlinking = blinkDetected
              lastBlinkTime = now
            }
          }

          // Use throttled broadcast instead of direct send
          throttledBroadcast({
            userId: userId.current,
            isBlinking: isCurrentlyBlinking,
            gazeX,
            gazeY
          })

        } catch (error) {
          console.error('Error processing gaze data:', error)
        }
      })

Getting the information where the user is looking at is simple and works like getting mouse positions on the screen. However, I also wanted to add blink detection as (a cool) feature, which required jumping through some hoops.

When you google information about WebGazer and blink detection, you can see some remanents of an initial implementation. Like there are commented out code in the source even. Unfortunately these kind of capabilities do not exit in the library. You'll need to do it manually.

After a lot of trial and error, Cursor and I were able to come up with a solution that calculates pixels & brightness levels from the eye patch data to determine when user is blinking. It also has some dynamic lighting adjustments as I noticed that (at least for me) the web cam doesn't always recognize when you are blinking depending on your lighting. For me it worked worse the lighter my picture/room was, and better in darker lighting (go figure).

While debugging the eye tracking capabilities (WebGazer has a very nice .setPredictionPoints call that displays a red dot on the screen to visualize where you are looking at), I noticed that the tracking is not very accurate unless you calibrate it. Which is what the project asks you to do before joining any rooms.

const startCalibration = useCallback(() => {
    const points: CalibrationPoint[] = [
      { x: 0.1, y: 0.1 },
      { x: 0.9, y: 0.1 },
      { x: 0.5, y: 0.5 },
      { x: 0.1, y: 0.9 },
      { x: 0.9, y: 0.9 },
    ]
    setCalibrationPoints(points)
    setCurrentPoint(0)
    setIsCalibrating(true)

    window.webgazer.clearData()
  }, [])

  const handleCalibrationClick = useCallback((event: React.MouseEvent) => {
    if (!isCalibrating) return

    // Record click location for calibration
    const x = event.clientX
    const y = event.clientY
    window.webgazer.recordScreenPosition(x, y, 'click')

    if (currentPoint < calibrationPoints.length - 1) {
      setCurrentPoint(prev => prev + 1)
    } else {
      setIsCalibrating(false)
      setHasCalibrated(true)
    }
  }, [isCalibrating, currentPoint, calibrationPoints.length])

          <div 
            className="fixed inset-0 bg-black/50 flex items-center justify-center z-50"
          >
            {calibrationPoints.map((point, index) => (
              <div
                key={index}
                onClick={handleCalibrationClick}
                className={`absolute w-6 h-6 rounded-full transform -translate-x-1/2 -translate-y-1/2 ${
                  index === currentPoint ? 'bg-red-500 cursor-pointer pulsate' : 'bg-gray-500'
                } z-50`}
                style={{
                  left: `${point.x * 100}%`,
                  top: `${point.y * 100}%`,
                }}
              >
              </div>
            ))}
            <div className="text-white text-center z-50 font-serif text-md mb-24">
              Click the red dot to calibrate ({currentPoint + 1}/{calibrationPoints.length})
            </div>
          </div>

Basically we're rendering 5 dots on the screen: one in each corner and one in the center. Clicking them will record the screen position in WebGazer so it can adjust the model a bit better to predict where you are looking. You might wonder what does this clicking actually do and the funny part is that you are actually looking where you are clicking, right? And as you do that, WebGazer is able to process your eye movements a bit better and provide more accurate results. Very cool!

The Eye

I had already added a simple SVG implementation for the eye & hooked it up to the tracking, however it needed to be stylized a bit more. Below is a bit how it ended up looking. The inspiration was Alucard Eyes by MIKELopez.

This is an earlier version of the eye, however it is pretty much 95% there. I sent the video to couple my friends and they thought it was very cool, especially when knowing the fact that it was actually following your eye movements! You can also see the WebGazer's prediction dot moving on the screen, too.

The eye component itself is an SVG with some path animations via Motion.

      <svg
        className={`w-full h-full self-${alignment} max-w-[350px] max-h-[235px]`}
        viewBox="-50 0 350 235"
        preserveAspectRatio="xMidYMid meet"
      >
        {/* Definitions for gradients and filters */}
        <defs>
          <filter id="pupil-blur">
            <feGaussianBlur stdDeviation="0.75" />
          </filter>
          <radialGradient id="eyeball-gradient">
            <stop offset="60%" stopColor="#dcdae0" />
            <stop offset="100%" stopColor="#a8a7ad" />
          </radialGradient>
          <radialGradient 
            id="pupil-gradient"
            cx="0.35"
            cy="0.35"
            r="0.65"
          >
            <stop offset="0%" stopColor="#444" />
            <stop offset="75%" stopColor="#000" />
            <stop offset="100%" stopColor="#000" />
          </radialGradient>
          <radialGradient 
            id="corner-gradient-left"
            cx="0.3"
            cy="0.5"
            r="0.25"
            gradientUnits="objectBoundingBox"
          >
            <stop offset="0%" stopColor="rgba(0,0,0,0.75)" />
            <stop offset="100%" stopColor="rgba(0,0,0,0)" />
          </radialGradient>

          <radialGradient 
            id="corner-gradient-right"
            cx="0.7"
            cy="0.5"
            r="0.25"
            gradientUnits="objectBoundingBox"
          >
            <stop offset="0%" stopColor="rgba(0,0,0,0.75)" />
            <stop offset="100%" stopColor="rgba(0,0,0,0)" />
          </radialGradient>

          <filter id="filter0_f_302_14" x="-25" y="0" width="320" height="150" filterUnits="userSpaceOnUse" colorInterpolationFilters="sRGB">
            <feGaussianBlur stdDeviation="4.1" result="effect1_foregroundBlur_302_14"/>
          </filter>
          <filter id="filter1_f_302_14" x="-25" y="85" width="320" height="150" filterUnits="userSpaceOnUse" colorInterpolationFilters="sRGB">
            <feGaussianBlur stdDeviation="4.1" result="effect1_foregroundBlur_302_14"/>
          </filter>
          <filter id="filter2_f_302_14" x="-50" y="-30" width="400" height="170" filterUnits="userSpaceOnUse" colorInterpolationFilters="sRGB">
            <feGaussianBlur stdDeviation="7.6" result="effect1_foregroundBlur_302_14"/>
          </filter>
          <filter id="filter3_f_302_14" x="-50" y="95" width="400" height="170" filterUnits="userSpaceOnUse" colorInterpolationFilters="sRGB">
            <feGaussianBlur stdDeviation="7.6" result="effect1_foregroundBlur_302_14"/>
          </filter>
          <filter id="filter4_f_302_14" x="0" y="-20" width="260" height="150" filterUnits="userSpaceOnUse" colorInterpolationFilters="sRGB">
            <feGaussianBlur stdDeviation="3.35" result="effect1_foregroundBlur_302_14"/>
          </filter>
          <filter id="filter5_f_302_14" x="0" y="105" width="260" height="150" filterUnits="userSpaceOnUse" colorInterpolationFilters="sRGB">
            <feGaussianBlur stdDeviation="3.35" result="effect1_foregroundBlur_302_14"/>
          </filter>
        </defs>

        {/* Eyeball */}
        <ellipse
          cx="131"
          cy="117.5"
          rx="100"
          ry="65"
          fill="url(#eyeball-gradient)"
        />

        {/* After the main eyeball ellipse but before the eyelids, add the corner shadows */}
        <ellipse
          cx="50"
          cy="117.5"
          rx="50"
          ry="90"
          fill="url(#corner-gradient-left)"
        />

        <ellipse
          cx="205"
          cy="117.5"
          rx="50"
          ry="90"
          fill="url(#corner-gradient-right)"
        />

        {/* Corner reflections - repositioned diagonally */}
        <circle
          cx={45}
          cy={135}
          r="1.5"
          fill="white"
          className="opacity-60"
        />
        <circle
          cx={215}
          cy={100}
          r="2"
          fill="white"
          className="opacity-60"
        />

        {/* Smaller companion reflections - repositioned diagonally */}
        <circle
          cx={35}
          cy={120}
          r="1"
          fill="white"
          className="opacity-40"
        />
        <circle
          cx={222}
          cy={110}
          r="1.5"
          fill="white"
          className="opacity-40"
        />

        {/* Pupil group with animations */}
        <motion.g
          variants={pupilVariants}
          animate={isBlinking ? "hidden" : "visible"}
        >
          {/* Pupil */}
          <motion.ellipse
            cx={131}
            cy={117.5}
            rx="50"
            ry="50"
            fill="url(#pupil-gradient)"
            filter="url(#pupil-blur)"
            animate={{
              cx: 131 + pupilOffsetX,
              cy: 117.5 + pupilOffsetY
            }}
            transition={{
              type: "spring",
              stiffness: 400,
              damping: 30
            }}
          />

          {/* Light reflections */}
          <motion.circle
            cx={111}
            cy={102.5}
            r="5"
            fill="white"
            animate={{
              cx: 111 + pupilOffsetX,
              cy: 102.5 + pupilOffsetY
            }}
            transition={{
              type: "spring",
              stiffness: 400,
              damping: 30
            }}
          />
          <motion.circle
            cx={124}
            cy={102.5}
            r="3"
            fill="white"
            animate={{
              cx: 124 + pupilOffsetX,
              cy: 102.5 + pupilOffsetY
            }}
            transition={{
              type: "spring",
              stiffness: 400,
              damping: 30
            }}
          />
        </motion.g>

        {/* Upper eyelid */}
        <motion.path 
          custom={true}
          variants={eyelidVariants}
          animate={isBlinking ? "closed" : "open"}
          fill="#000"
        />

        {/* Lower eyelid */}
        <motion.path 
          custom={false}
          variants={eyelidVariants}
          animate={isBlinking ? "closed" : "open"}
          fill="#000"
        />

        {/* Top blurred lines */}
        <g filter="url(#filter0_f_302_14)">
          <motion.path
            custom={true}
            variants={blurredLineVariants}
            animate={isBlinking ? "closed" : "open"}
            stroke="#2A2A2A"
            strokeWidth="5"
            strokeLinecap="round"
          />
        </g>
        <g filter="url(#filter2_f_302_14)">
          <motion.path
            custom={true}
            variants={outerBlurredLineVariants}
            animate={isBlinking ? "closed" : "open"}
            stroke="#777777"
            strokeWidth="5"
            strokeLinecap="round"
          />
        </g>
        <g filter="url(#filter4_f_302_14)">
          <motion.path
            custom={true}
            variants={arcLineVariants}
            animate={isBlinking ? "closed" : "open"}
            stroke="#838383"
            strokeWidth="5"
            strokeLinecap="round"
          />
        </g>

        {/* Bottom blurred lines */}
        <g filter="url(#filter1_f_302_14)">
          <motion.path
            variants={bottomBlurredLineVariants}
            animate={isBlinking ? "closed" : "open"}
            stroke="#2A2A2A"
            strokeWidth="5"
            strokeLinecap="round"
          />
        </g>
        <g filter="url(#filter3_f_302_14)">
          <motion.path
            variants={bottomOuterBlurredLineVariants}
            animate={isBlinking ? "closed" : "open"}
            stroke="#777777"
            strokeWidth="5"
            strokeLinecap="round"
          />
        </g>
        <g filter="url(#filter5_f_302_14)">
          <motion.path
            variants={bottomArcLineVariants}
            animate={isBlinking ? "closed" : "open"}
            stroke="#838383"
            strokeWidth="5"
            strokeLinecap="round"
          />
        </g>
      </svg>

Cursor works surprisingly well with SVG paths. For example, the eye lid closing animation is basically done by straightening out a curved path. I had just highlighted the path in the editor, pasted it in the Composer and asked to add an animation which straightens out the points so that the eye looks like it's closing/blinking.

  // Define the open and closed states for both eyelids
  const upperLidOpen = "M128.5 53.5C59.3 55.5 33 99.6667 28.5 121.5H0V0L261.5 0V121.5H227.5C214.7 65.1 156.167 52.6667 128.5 53.5Z"
  const upperLidClosed = "M128.5 117.5C59.3 117.5 33 117.5 28.5 117.5H0V0L261.5 0V117.5H227.5C214.7 117.5 156.167 117.5 128.5 117.5Z"

  const lowerLidOpen = "M128.5 181C59.3 179 33 134.833 28.5 113H0V234.5H261.5V113H227.5C214.7 169.4 156.167 181.833 128.5 181Z"
  const lowerLidClosed = "M128.5 117.5C59.3 117.5 33 117.5 28.5 117.5H0V234.5H261.5V117.5H227.5C214.7 117.5 156.167 117.5 128.5 117.5Z"

  // Animation variants for the eyelids
  const eyelidVariants = {
    open: (isUpper: boolean) => ({
      d: isUpper ? upperLidOpen : lowerLidOpen,
      transition: {
        duration: 0.4,
        ease: "easeOut"
      }
    }),
    closed: (isUpper: boolean) => ({
      d: isUpper ? upperLidClosed : lowerLidClosed,
      transition: {
        duration: 0.15,
        ease: "easeIn"
      }
    })
  }

It was a very cool experience to see this in action! I applied the same approach to the surrounding lines and instructed Cursor to "collapse" them towards the center: which it did pretty much with one go!

The eyes would then be rendered inside a simple CSS grid with cells aligned so that a full room looks like a big eye.

<div className="fixed inset-0 grid grid-cols-3 grid-rows-3 gap-4 p-8 md:gap-2 md:p-4 lg:max-w-6xl lg:mx-auto">
          {Object.entries(roomState.participants).map(([key, presences]) => {
            const participant = presences[0]
            const eyeData = eyeTrackingState[key]
            if (key === userId.current) return null

            return (
              <div 
                key={key}
                className={`flex items-center justify-center ${getGridClass(participant.position)}`}
              >
                <Eyes
                  isBlinking={eyeData?.isBlinking ?? false}
                  gazeX={eyeData?.gazeX ?? 0.5}
                  gazeY={eyeData?.gazeY ?? 0.5}
                  alignment={getEyeAlignment(participant.position)}
                />
              </div>
            )
          })}
        </div>

// Helper function to convert position to Tailwind grid classes
function getGridClass(position: string): string {
  switch (position) {
    case 'center': return 'col-start-2 row-start-2'
    case 'middleLeft': return 'col-start-1 row-start-2'
    case 'middleRight': return 'col-start-3 row-start-2'
    case 'topCenter': return 'col-start-2 row-start-1'
    case 'bottomCenter': return 'col-start-2 row-start-3'
    case 'topLeft': return 'col-start-1 row-start-1'
    case 'topRight': return 'col-start-3 row-start-1'
    case 'bottomLeft': return 'col-start-1 row-start-3'
    case 'bottomRight': return 'col-start-3 row-start-3'
    default: return 'col-start-2 row-start-2'
  }
}

function getEyeAlignment(position: string): 'start' | 'center' | 'end' {
  switch (position) {
    case 'topLeft':
    case 'topRight':
      return 'end'
    case 'bottomLeft':
    case 'bottomRight':
      return 'start'
    default:
      return 'center'
  }
}

Final touches

Then slap some nice intro screen and background music and the project is good to go!

Audio always improves the experience when you are working on things like this, so I used Stable Audio to generate a background music when the user "enters the abyss." The prompt I used for the music was the following:

Ambient, creepy, background music, whispering sounds, winds, slow tempo, eerie, abyss

I also thought that just plain black screen is a bit boring, so I added some animated SVG filter stuff on the background. Additionally I added a dark, blurred circle in the center of the screen to have some nice fading effect. I probably could've done this with SVG filters, however I didn't want to spend too much time on this. Then to have some more movement, I made the background rotate on its axis. Sometimes doing animations with the SVG filters is a bit wonky, so I decided to do it this way instead.

    <div style={{ width: '100vw', height: '100vh' }}>
      {/* Background Elements */}
      <svg className="fixed inset-0 w-full h-full -z-10">
        <defs>
          <filter id="noise">
            <feTurbulence 
              id="turbFreq"
              type="fractalNoise" 
              baseFrequency="0.01"
              seed="5"
              numOctaves="1"
            >
            </feTurbulence>
            <feGaussianBlur stdDeviation="10">
              <animate
                attributeName="stdDeviation"
                values="10;50;10"
                dur="20s"
                repeatCount="indefinite"
              />
            </feGaussianBlur>
            <feColorMatrix
              type="matrix"
              values="1 0 0 0 1
                      0 1 0 0 1
                      0 0 1 0 1
                      0 0 0 25 -13"
            />
          </filter>
        </defs>
        <rect width="200%" height="200%" filter="url(#noise)" className="rotation-animation" />
      </svg>
      <div className="fixed inset-0 w-[95vw] h-[95vh] bg-black rounded-full blur-[128px] m-auto" />

Conclusion

So there you have it: fairly straight-forward look into how to implement a stylized eye tracking with Supabase's realtime capabilities. Personally I found this a very interesting experiment and didn't have too many hiccups while working on it. And surprisingly I did not have to do an all-nighter for the last night before submitting the project!

Feel free to check out the project or the demo video how it turned out. There might be some issues if a bunch of people are using it at the same time (very hard to test as it requires multiple devices & web cams to do it properly), but I guess that's in the fashion of hackathon projects? And if you do test it out, remember that if you see an eye, it's someone else watching you somewhere over the internet!

Building a self-creating website with Supabase and AI

Niklas Lepistö — Tue, 23 Apr 2024 11:36:30 +0000

TL;DR:

Built with Supabase, Astro, Unreal Speech, Stable Diffusion, Replicate, Metropolitan Museum of Art

Leverages Supabase Edge Functions, Storage, Webhooks, Database Triggers and pg_cron

Scheduled job every hour triggers a new insert which then triggers Webhook calls to Edge Functions to start generating new assets

GitHub repo

Back again writing about another Supabase hackathon project! This is about our most recent one called Echoes of Creation and focuses on the technical details rather than going explaining how it was built in general.

The idea

My friend and I have had this idea of building a website that would "destroy" and re-create itself automatically every hour for over a year now. Idea was to kind of portray how AI has made an impact and especially how it is affecting the creative fields. There would be a countdown on the page to show when a new version is generated, and we would detoriate the content using various effects little by little as the time would tick closer to zero. It's basically an art project which we think would've been cool to do at some point, and this Supabase hackathon turned out to be the time to do it! Although, due to the time restrictions, we had to change up the idea a bit.

Initially we had thought of creating a completely new layout each time the page was created, alongside with the new content. This was, however, a very technically challenging to do especially if you wanted to do using just CSS and HTML: you don't want things to just be boxed and layout out vertically like every other web page out there. So the option would've been to use WebGL to leverage 3D space if we wanted to do our very first idea. This might be a future improvement for the project, however since I'm not as experienced in WebGL as I'd like to be, it was a no-go if we wanted to make it in time of the deadline.

We talked on Discord and my friend came up with a great idea: what if instead of just AI detoriating or destroying the site we'd tell a story of an artist who is struggling in the current world filled with AI. And we'd do a scroll-based website instead, ideally just animating the content so that it'd feel that you are in a 3D space. And each website is the a new "thought/creative flow process" inside the artist's mind. We both were really excited about this idea, and began to work.

Building the project

When we started we had a clear vision of the whole scroll-based story and animation, however we ditched most of it as we moved on. Shortly: we thought it would be cool if the "camera" would change angles and spin the content in different directions on each site variant to make it feel like you're kind of navigating inside the artist's brain. We implemented this and it worked well (albeit sometimes a bit nauseating), however as we thought about the story and structure more, this kind of navigation didn't make sense anymore. We decided to focus more on the narrative via sounds and other content.

But let's focus on the actual gist of the project now: Supabase Edge Functions.

Before starting few things:

there are three database tables: thoughts, artwork, thought_texts

thought is the whole story narrative: the creative process is "one big thought"

thought_texts contains all texts in the storty and links to their audio files

artwork contains, well, all the artwork; both generated and fetched from Met Museum API

Generating artwork

In the project we ended up using only two Edge Functions: one creating artwork called generate and one creating texts and speech called create-speech. Let's start with the artwork one first which looks something like this:

/* eslint-disable @typescript-eslint/ban-ts-comment */
// Follow this setup guide to integrate the Deno language server with your editor:
// https://deno.land/manual/getting_started/setup_your_environment
// This enables autocomplete, go to definition, etc.

import { corsHeaders } from "../_shared/cors.ts";
import { createClient } from "https://esm.sh/@supabase/supabase-js@2.42.3";
import randomSample from "https://esm.sh/@stdlib/random-sample@0.2.1";
import Replicate from "https://esm.sh/replicate@0.29.1";
import { base64 } from "https://cdn.jsdelivr.net/gh/hexagon/base64@1/src/base64.js";

const supabaseClient = createClient(
  Deno.env.get("SUPABASE_URL") ?? "",
  Deno.env.get("SUPABASE_SERVICE_ROLE_KEY") ?? ""
);

const replicate = new Replicate({
  auth: Deno.env.get("REPLICATE_API_TOKEN") ?? "",
});

// @ts-expect-error
Deno.serve(async (req) => {
  if (req.method === "OPTIONS") {
    return new Response("ok", { headers: corsHeaders });
  }

  const { record } = await req.json();
  const thought_id = record.id;

  if (!thought_id) {
    return new Response("Missing thought_id", {
      status: 400,
      headers: { "Content-Type": "application/json", ...corsHeaders },
    });
  }

  const allObjectIDsResponse = await fetch(
    "https://collectionapi.metmuseum.org/public/collection/v1/objects?departmentIds=11",
    {
      method: "GET",
      headers: {
        "Content-Type": "application/json",
        Accept: "application/json",
      },
    }
  );

  const { objectIDs } = await allObjectIDsResponse.json();
  const listOfArtworks = [];
  const addedIDs: number[] = [];

  while (listOfArtworks.length < 80) {
    const randomObjectID = randomSample(objectIDs, { size: 1 })[0];

    if (addedIDs.includes(randomObjectID)) continue;

    const res = await fetch(
      `https://collectionapi.metmuseum.org/public/collection/v1/objects/${randomObjectID}`,
      {
        method: "GET",
        headers: {
          "Content-Type": "application/json",
          Accept: "application/json",
        },
      }
    );

    const artwork = await res.json();

    if (!artwork.primaryImageSmall) continue;

    addedIDs.push(artwork.objectID);

    listOfArtworks.push({
      image_url: artwork.primaryImageSmall,
      artist_name: artwork.artistDisplayName,
      title: artwork.title,
      is_main: listOfArtworks.length === 0,
      is_variant: false,
      thought_id,
    });
  }

  const mainImage = listOfArtworks[0];

  const output = await replicate.run(
    "yorickvp/llava-13b:b5f6212d032508382d61ff00469ddda3e32fd8a0e75dc39d8a4191bb742157fb",
    {
      input: {
        image: mainImage.image_url,
        top_p: 1,
        prompt: "Describe this painting by " + mainImage.artist_name,
        max_tokens: 1024,
        temperature: 0.2,
      },
    }
  );

  const file = await fetch(mainImage.image_url).then((res) => res.blob());
  const promises = [];

  for (let i = 0; i < 8; i++) {
    const body = new FormData();
    body.append(
      "prompt",
      output.join("") + `, a painting by ${mainImage.artist_name}`
    );
    body.append("output_format", "jpeg");
    body.append("mode", "image-to-image");
    body.append("image", file);
    body.append("strength", clamp(Math.random(), 0.4, 0.7));

    const request = fetch(
      `${Deno.env.get(
        "STABLE_DIFFUSION_HOST"
      )}/v2beta/stable-image/generate/sd3`,
      {
        method: "POST",
        headers: {
          Accept: "application/json",
          Authorization: `Bearer ${Deno.env.get("STABLE_DIFFUSION_API_KEY")}`,
        },
        body,
      }
    );

    promises.push(request);
  }

  const results = await Promise.all(promises);
  const variants = await Promise.all(results.map((res) => res.json()));

  await supabaseClient.from("artworks").insert(listOfArtworks);

  for (let i = 0; i < variants.length; i++) {
    const variant = variants[i];
    const randomId = Math.random();

    await supabaseClient.storage
      .from("variants")
      .upload(
        `${thought_id}/${randomId}.jpeg`,
        base64.toArrayBuffer(variant.image),
        {
          contentType: "image/jpeg",
        }
      );

    await supabaseClient.from("artworks").insert({
      image_url: `${Deno.env.get(
        "SUPABASE_URL"
      )}/storage/v1/object/public/variants/${thought_id}/${randomId}.jpeg`,
      artist_name: mainImage.artist_name,
      is_main: false,
      is_variant: true,
      thought_id,
    });
  }

  await supabaseClient
    .from("thoughts")
    .update({ generating: false })
    .eq("id", thought_id);

  return new Response(JSON.stringify({ main: mainImage }), {
    headers: { "Content-Type": "application/json", ...corsHeaders },
  });
});

function clamp(value: number, min: number, max: number) {
  return Math.min(Math.max(value, min), max);
}

Breaking it down, we'll first check if there is a new thought ID that needs artwork to be generated from before we fetch any data. I added the 400 response there just so that I'd be able to debug if Webhooks would fail for some reason later. The record data structure is passed automatically in the Webhook triggers, however we'll touch those later!

  const { record } = await req.json();
  const thought_id = record.id;

  if (!thought_id) {
    return new Response("Missing thought_id", {
      status: 400,
      headers: { "Content-Type": "application/json", ...corsHeaders },
    });
  }

Then we'll hook to Met Museum's API to fetch publicly available artwork in their collections. Isn't it amazing that a museum has a free and public API! These artworks will serve as "ideas" the artist has during the creative flow, in which one of them will be the main piece of art they create.

const allObjectIDsResponse = await fetch(
    "https://collectionapi.metmuseum.org/public/collection/v1/objects?departmentIds=11",
    {
      method: "GET",
      headers: {
        "Content-Type": "application/json",
        Accept: "application/json",
      },
    }
  );

Using a department ID as a URL param will only return pieces that are categorized under a specific department, such as "Drawings and Prints", "Modern Art", "European Paintings." In our case, we use the ID 11 which corresponds to "European Paintings" which includes over 2500 pieces of art! The endpoint we used returns a list of IDs attached the pieces which we'll use to fetch the actual data associated to them.

You can fetch pieces from multiple departments by splitting the list with the pipe | character, so ?departmentIds=11|9 would return you "European Paintings" and "Drawings and Prints." You can see the available departments in the API documentation.

We'll loop the list of 80 IDs, picking a random piece of art from the original ID list. I'm using stdlib's random-sample which uses Fisher-Yates as its algorithm to randomize entries as I didn't want to bloat my own code too much with these kind of helpers. Fetching the artwork data I unfortunately noticed that some of the artwork do not have images available. This meant that we'll need to loop the list until we have 80 pieces with images. For the main piece of the story we keep things simple and just pick the first one in the list. We store some metadata along the way, too, which we can/will use later.

  const { objectIDs } = await allObjectIDsResponse.json();
  const listOfArtworks = [];
  const addedIDs: number[] = [];

  while (listOfArtworks.length < 80) {
    const randomObjectID = randomSample(objectIDs, { size: 1 })[0];

    if (addedIDs.includes(randomObjectID)) continue;

    const res = await fetch(
      `https://collectionapi.metmuseum.org/public/collection/v1/objects/${randomObjectID}`,
      {
        method: "GET",
        headers: {
          "Content-Type": "application/json",
          Accept: "application/json",
        },
      }
    );

    const artwork = await res.json();

    if (!artwork.primaryImageSmall) continue;

    addedIDs.push(artwork.objectID);

    listOfArtworks.push({
      image_url: artwork.primaryImageSmall,
      artist_name: artwork.artistDisplayName,
      title: artwork.title,
      is_main: listOfArtworks.length === 0,
      is_variant: false,
      thought_id,
    });
  }

  const mainImage = listOfArtworks[0];

Note: The primaryImageSmall attribute has a link to a more web-suitable format and size. There is a primaryImage attribute, too, however those links can contain images of multiple megabytes!

Now we get to the part where we use AI the first time: generating a prompt from the main image which will be used to generate variants.

We're using LLaVa model to get a description of the main painting. Then we fetch the image file from the URL we got from the previous step and use it as the base for Stable Diffusion 3 to generate variants from. This allows us to keep the original image structure & colors and just modify some parts of it to make it look a copy. Depending on the strength param, the end result is either identical or very different from the original image. If you passed 1, SD would consider that you didn't pass an image at all, so we pick a number between 0.4 and 0.7, which would give a strength of 30-60% for the original image in the mix.

const output = await replicate.run(
    "yorickvp/llava-13b:b5f6212d032508382d61ff00469ddda3e32fd8a0e75dc39d8a4191bb742157fb",
    {
      input: {
        image: mainImage.image_url,
        top_p: 1,
        prompt: "Describe this painting by " + mainImage.artist_name,
        max_tokens: 1024,
        temperature: 0.2,
      },
    }
  );

  const file = await fetch(mainImage.image_url).then((res) => res.blob());
  const promises = [];

  for (let i = 0; i < 8; i++) {
    const body = new FormData();
    body.append(
      "prompt",
      output.join("") + `, a painting by ${mainImage.artist_name}`
    );
    body.append("output_format", "jpeg");
    body.append("mode", "image-to-image");
    body.append("image", file);
    body.append("strength", clamp(Math.random(), 0.4, 0.7));

    const request = fetch(
      `${Deno.env.get(
        "STABLE_DIFFUSION_HOST"
      )}/v2beta/stable-image/generate/sd3`,
      {
        method: "POST",
        headers: {
          Accept: "application/json",
          Authorization: `Bearer ${Deno.env.get("STABLE_DIFFUSION_API_KEY")}`,
        },
        body,
      }
    );

    promises.push(request);
  }

The new Stable Diffusion 3 endpoints use FormData to take in params, changing from the previous JSON payloads. A welcome improvement is that it can now return JPEG or WebP formats instead of PNG which means we can save on storage space!

Lastly we'll handle storing the variants into Supabase Storage and updating the thought record to not be in generating state anymore. This is pretty straight-forward: first we bulk insert all the artworks from the Met Museum API which is just links to images, and then we upload each generated variant to the Storage. I'm using Math.random to generate a "random" ID as the file name for the images as I don't care what they are called (semantically): they're just going be placed under a specific directory with the thought ID in the path which helps us to keep track which images are part of which thought. We're also creating records in the "artwork" table with links to the images, so we're able to fetch these images along with the other content without any extra hassle.

const results = await Promise.all(promises);
  const variants = await Promise.all(results.map((res) => res.json()));

  await supabaseClient.from("artworks").insert(listOfArtworks);

  for (let i = 0; i < variants.length; i++) {
    const variant = variants[i];
    const randomId = Math.random();

    await supabaseClient.storage
      .from("variants")
      .upload(
        `${thought_id}/${randomId}.jpeg`,
        base64.toArrayBuffer(variant.image),
        {
          contentType: "image/jpeg",
        }
      );

    await supabaseClient.from("artworks").insert({
      image_url: `${Deno.env.get(
        "SUPABASE_URL"
      )}/storage/v1/object/public/variants/${thought_id}/${randomId}.jpeg`,
      artist_name: mainImage.artist_name,
      is_main: false,
      is_variant: true,
      thought_id,
    });
  }

  await supabaseClient
    .from("thoughts")
    .update({ generating: false })
    .eq("id", thought_id);

Updating the thought record at the end will trigger a Webhook call to trigger a deployment in Cloudflare pages. This is because the site doesn't use Server-Side Rendering to access data dynamically, so we'll need to rebuild it each time there is a new thought (which is done generating content). We'll be adding SSR in the future so that you can navigate back and forth the thoughts to see how it has progressed from the original one!

Generating speech

The second function called, which I think is the coolest part of this project, is used to generate AI speech over some texts in the site. It looks like the following and we'll break it down below.

import { corsHeaders } from "../_shared/cors.ts";
import { createClient } from "https://esm.sh/@supabase/supabase-js@2.42.3";
import Replicate from "https://esm.sh/replicate@0.29.1";
import randomSample from "https://esm.sh/@stdlib/random-sample@0.2.1";
// Follow this setup guide to integrate the Deno language server with your editor:
// https://deno.land/manual/getting_started/setup_your_environment
// This enables autocomplete, go to definition, etc.

// Setup type definitions for built-in Supabase Runtime APIs
/// <reference types="https://esm.sh/@supabase/functions-js/src/edge-runtime.d.ts" />

const supabaseClient = createClient(
  Deno.env.get("SUPABASE_URL") ?? "",
  Deno.env.get("SUPABASE_SERVICE_ROLE_KEY") ?? ""
);

const replicate = new Replicate({
  auth: Deno.env.get("REPLICATE_API_TOKEN") ?? "",
});

Deno.serve(async (req) => {
  if (req.method === "OPTIONS") {
    return new Response("ok", { headers: corsHeaders });
  }
  const voice = randomSample(["Scarlett", "Dan", "Liv", "Will", "Amy"], {
    size: 1,
  })[0];

  const { record } = await req.json();
  const thought_id = record.id;
  const thoughtIdToUse = thought_id === 1 ? thought_id : thought_id - 1;

  const { data: thought } = await supabaseClient
    .from("thoughts")
    .select(
      `
      id,
      thought_texts(
        index,
        text
      )
    `
    )
    .eq("id", thoughtIdToUse);

  const thoughtTexts = thought[0].thought_texts.sort(
    (a, b) => a.index - b.index
  );

  for (let i = 0; i < thoughtTexts.length; i++) {
    let newText = thoughtTexts[i].text;

    if (thought_id !== 1) {
      const input = {
        prompt: `Write the following differently but keep the style of it being your thought: ${thoughtTexts[i].text}`,
        temperature: 1.0,
        frequency_penalty: 1.0,
        prompt_template:
          "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nReturn info wrapped as text snippet wrapped in \`\`\`. If your suggestion is longer than 150 characters, make it shorter.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
      };

      const textResponse = await replicate.run(
        "meta/meta-llama-3-70b-instruct",
        {
          input,
        }
      );

      newText = textResponse.join("").replace(/\`\`\`/g, "");
    }

    const options = {
      method: "POST",
      headers: {
        accept: "application/json",
        "content-type": "application/json",
        Authorization: `Bearer ${Deno.env.get("UNREAL_SPEECH_API_KEY")}`,
      },
      body: JSON.stringify({
        Text: newText,
        VoiceId: voice,
        Bitrate: "192k",
        Speed: "0",
        Pitch: voice === "Dan" || voice === "Will" ? "0.96" : "1.0",
        TimestampType: "word",
      }),
    };

    const voiceResponse = await fetch(
      Deno.env.get("UNREAL_SPEECH_API_URL"),
      options
    );

    const voiceResponseJson = await voiceResponse.json();
    const blob = await fetch(voiceResponseJson.OutputUri).then((r) => r.blob());
    const jsonBlob = await fetch(voiceResponseJson.TimestampsUri).then((r) =>
      r.blob()
    );

    await supabaseClient.storage
      .from("speeches")
      .upload(`${thought_id}/${i}.json`, jsonBlob, {
        contentType: "application/json",
      });

    await supabaseClient.storage
      .from("speeches")
      .upload(`${thought_id}/${i}.mp3`, blob, {
        contentType: "audio/mpeg",
      });

    await supabaseClient.from("thought_texts").insert({
      thought_id,
      text: newText,
      audio_url: `${Deno.env.get(
        "SUPABASE_URL"
      )}/storage/v1/object/public/speeches/${thought_id}/${i}.mp3`,
      index: i,
    });
  }

  await supabaseClient
    .from("thoughts")
    .update({ generating: false })
    .eq("id", thought_id);

  return new Response(JSON.stringify({ check: "database" }), {
    headers: { "Content-Type": "application/json" },
  });
});

Creating AI speech isn't that difficult thing to do, however it is a bit challenging to find an API that doesn't cost you your kidneys. Luckily I ran into Unreal Speech which has some pretty generous free monthly tier. However it will not suffice for us to run the site generations for a over month, so I'll need to do some tricks here and there to keep things running (or just open up my wallet).

With Unreal Speech, you have five voices to choose from: "Scarlett", "Dan", "Liv", "Will", "Amy." We'll only need one as the narrator (i.e. artist) should always be a singular "person" so we'll use the same random-sample helper to pick one of these voices.

  const voice = randomSample(["Scarlett", "Dan", "Liv", "Will", "Amy"], {
    size: 1,
  })[0];

Then instead of always having the same texts read out loud in different voices, we generate new texts for each new thought record. And in order to do this, we'll need the previous thought texts to base these new texts on so that we keep the original meaning throughout each iteration. Here we're using Meta's LLama 3 model to generate the texts as it's cheap and fast. The promp template is using a special syntax to direct the model to reply in specific way. It works in similar fashion as GPT4, however I do prefer the JSON array syntax over this speciality. There are guides how to use this template syntax in the model's Replicate page. The model returns the text as a stream, however since we just want the whole string at once, we'll need to join each word together in the end.

const { data: thought } = await supabaseClient
    .from("thoughts")
    .select(
      `
      id,
      thought_texts(
        index,
        text
      )
    `
    )
    .eq("id", thoughtIdToUse);

  const thoughtTexts = thought[0].thought_texts.sort(
    (a, b) => a.index - b.index
  );

  for (let i = 0; i < thoughtTexts.length; i++) {
    let newText = thoughtTexts[i].text;

    if (thought_id !== 1) {
      const input = {
        prompt: `Write the following differently but keep the style of it being your thought: ${thoughtTexts[i].text}`,
        temperature: 1.0,
        frequency_penalty: 1.0,
        prompt_template:
          "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nReturn info wrapped as text snippet wrapped in \`\`\`. If your suggestion is longer than 150 characters, make it shorter.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
      };

      const textResponse = await replicate.run(
        "meta/meta-llama-3-70b-instruct",
        {
          input,
        }
      );

      newText = textResponse.join("").replace(/\`\`\`/g, "");
    }
    // More code here...
  }

For the very first thought we wanted to keep the original story texts and just generate speech for them.

Next we'll start generating the audio. We'll do a regular POST call to the Unreal Speech API with some params:

VoiceId is the voice we pick
Bitrate is the sound quality and we'll keep it as 192k to keep the files relatively small
Speed is 0 so we let the AI talk at a "normal" pace instead of slowing it down or speeding it up
"Pitch" we adjust a bit based on if the voice is a male voice to give it a bit deeper sound
"TimestampType" allows us to define if we want a transcript where the file is timestamped by sentence or by word. We go with "word" as we planned to do word highlighting in the app whenever the audio is playing (didn't have time to implement this for the hackathon).

Note that we are using the /speech endpoint from Unreal Speech. It temporarily uploads the audio and transcript on their servers and provides an URL to these files. There is also a /stream endpoint which allows you to stream back audio in 0.3s, and /synthesisTasks which allows you to create scheduled audio generation.

After this audio generation is done, we fetch the audio and transcript files from the URLs provided, and upload them to Supabase Storage. Lastly, we insert the new text and reference to the audio file in the thought_texts table.

  const options = {
      method: "POST",
      headers: {
        accept: "application/json",
        "content-type": "application/json",
        Authorization: `Bearer ${Deno.env.get("UNREAL_SPEECH_API_KEY")}`,
      },
      body: JSON.stringify({
        Text: newText,
        VoiceId: voice,
        Bitrate: "192k",
        Speed: "0",
        Pitch: voice === "Dan" || voice === "Will" ? "0.96" : "1.0",
        TimestampType: "word",
      }),
    };

    const voiceResponse = await fetch(
      Deno.env.get("UNREAL_SPEECH_API_URL"),
      options
    );

    const voiceResponseJson = await voiceResponse.json();
    const blob = await fetch(voiceResponseJson.OutputUri).then((r) => r.blob());
    const jsonBlob = await fetch(voiceResponseJson.TimestampsUri).then((r) =>
      r.blob()
    );

    await supabaseClient.storage
      .from("speeches")
      .upload(`${thought_id}/${i}.json`, jsonBlob, {
        contentType: "application/json",
      });

    await supabaseClient.storage
      .from("speeches")
      .upload(`${thought_id}/${i}.mp3`, blob, {
        contentType: "audio/mpeg",
      });

    await supabaseClient.from("thought_texts").insert({
      thought_id,
      text: newText,
      audio_url: `${Deno.env.get(
        "SUPABASE_URL"
      )}/storage/v1/object/public/speeches/${thought_id}/${i}.mp3`,
      index: i,
    });

Finally, we update the thought record to not be in generating state again. This is just to make sure we trigger the Webhook to build the site always when all the newly generated data is there.

  await supabaseClient
    .from("thoughts")
    .update({ generating: false })
    .eq("id", thought_id);

Scheduled job and Webhook

Now that we have the main functionality done, we can do some simple scheduled jobs with triggers. All we need to do is enable pg_cron extension and Webhook functionality in the Database Settings.

After this is done, the following SQL snippet allows us to invalidate previous thoughts (i.e. make them not be the active thought), and then create a new one each hour. Very simple stuff!


create function create_thought () returns void as $$
BEGIN
     UPDATE thoughts
     SET current = false
     WHERE current = true;

     INSERT INTO thoughts (created_at)
     VALUES (NOW());
END;
$$ language plpgsql;

select cron.schedule ('hourly-thoughts', '0 * * * *', 'SELECT create_thought()');

And since we need to only trigger the generation processes whenever there is a new thought, we just need to create two Webhooks to listen to these database events and trigger a call to our Edge Functions: one for generate and one for create-speech

I hadn't used the Webhook functionality earlier other than just checking the UI a bit, so I was very pleasantly surprised to see (and remind myself since I think I've read about this in some of the blog posts..) that you actually can just select a Supabase Edge Function from the settings! Supa cool.

Remember to add Authorization header with Bearer token as the value so that the Function triggers work with these Webhooks! Otherwise you'll run into unauthorized errors (which can be caught in the Function logs). This is something which would be nice to have there by default.

Now remember earlier when we did the updates to the thought record to change the generating state to false? And that we need to trigger new deployments each time this happens? We use same Webhook functionality to do that! So in similar fashion as above with the other Webhooks, only difference this time is to select the UPDATE event instead, keep the HTTP Request option as the type, and then just inputting your provider's deployment build webhook URL as the URL value.

And that's it: now we have a self generating website! On the website I'm just fetching the current thought and displaying all the data I have for that thought. This fetching is only done once when the site builds which is pretty fast. It also reduces the amount of traffic flowing to your Supabase instance which is always nice: it's scalable solution (to some extent).

// The only Supabase call in client code
const { data: thought } = await supabase
    .from('thoughts')
    .select(`
        id,
        created_at,
        current,
        generating,
        thought_texts(
            index,
            audio_url,
            text
        ),
        artworks(
            is_main,
            is_variant,
            image_url,
            artist_name,
            title
        )
    `)
    .eq('current', true)
    .single()

It was (and still is) a very ambitious project in terms of styling and the things we visioned to do, and we both just love the concept behind it. It's truly an art project for us.

And while I'm happy that we got the project ready during the hackathon, I'm a bit disappointed that we weren't able to do all the things we wanted. We had a lot of problems in the end regarding the scrolling (still learning GSAP) and other styling issues. In the end a lot of curners were cut just push it out. 'Tis what it is, though, and we did have fun during this whole process: these hackathons are great and will keep on participating in them in the future.

Anyways, keep going back to the site every now and then as we keep on building until we're satisfied with the end result. For now we'll take a small break from it, though, and focus on working on some of our other projects in the meanwhile (I don't think I've already mentioned we're making a game!).

Hopefully this insight to the project gives you some inspiration how to approach your own solutions. Remember to not give up on your dreams: everything is possible, and in the world of AI the possibilities are endless!

Building an AI powered WebGL experience with Supabase and React Three Fiber

Niklas Lepistö — Thu, 21 Dec 2023 19:59:59 +0000

TL;DR:

Baldur's Gate 3 inspired WebGL experience called Mirror of Loss

Demo video

GitHub repo React Three Fiber with Drei makes things simple Tech stack used: React, React Three Fiber, Vite, Supabase, OpenAI, Stable Diffusion, Stable Audio

I recently participated in the Supabase Launch Week X Hackathon where I ended up dabbling in WebGL and created a project called Mirror of Loss. I had a lot of fun working on it, and figured it would be nice to share a bit about it here, too.

I've participated in three different Launch Week Hackathon's before and I've always tried to do something a bit outside my regular web dev work. Usually they turn out to be more of an experience than an app. The hackathons run for a week, so it's a good time to focus on and learn something new and cool!

Note: this article will not go into every detail how to build a similar kind of WebGL app with React Three Fiber and Supabase. For example, installation instructions can be found on the libraries own websites instead of being mentioned here.

This article just provides the bigger picture of how you can build WebGL apps & experiences by sharing my experience with it during the Supabase hackathon.

Not all the code will be displayed here, as the article would grow as long. It is, however, open sourced, so you can find all the little details in the GitHub repo.

The idea

Recently I've been indulging on Baldur's Gate 3, and what a better way to show your appreciation as a fan than to create something (close) from it! In the game they introduced a Mirror of Loss (spoiler warning), and me being very intrigued by the aesthetics (and the whole story/concept) of Shar in the game, I thought it would be nice to do a representation of it myself. And in 3D/WebGL! However, I wasn't planning on doing this during the hackathon: I just wanted to make it a new art project for myself. In the end I decided to try and create this within a week, since it seemed like a good time to do it.

Preparation

So before the hackathon started, around one or two weeks prior, I started wondering how to build this thing in WebGL. I was aware of Three.js, which I had dabbled a bit in the past, however it seemed a bit intimidating. No way I'd have time to learn the vanilla way of creating WebGL experiences. Luckily I had head about React Three Fiber before, although hadn't payed a lot of attention to it. And boy, I was really happy with what I read in their documentation! Everything seemed to abstract the tedious bits in Three.js to an easy to use React components, and I can do React for sure. They provide a lot of additional helper libraries, such as Drei, Ecctrl, to make developing a lot easier.

Drei is a collection of various, ready-made abstractions of React Three Fiber, and includes things like trails, making something face camera always, called Billboard, and animated distort material for example. Ecctrl on the other hand allows you to set up a character controller very quickly. I recommend checking out both of these if you are planning to do any React Three Fiber work.

Tip: Easiest way to get started is to setup up your project with Vite.

Other than that, my plan was to do two different scenes: one with a mirror for which the user can give a memory to, and one "inside" the mirror where you can see every single memory the mirror holds. Thought this would be a pretty cool concept, and with this in mind, I started experimenting a bit.

I started playing around and see if I could create some elements I'd like to use in the scene. I was mostly obsessed about creating a brazier, because you gotta have braziers! Below is what I come up with. It's very simple, however has a nice vibe to it. I didn't have time to study how to create a real looking flame via shaders, so I had to be quite creative in this. Basically it is just two stacked Cone geometries with some transparency.

It came nicely together in React Three Fiber with Lathe, Cone, and MeshDistortMaterial components.

const lathePoints = [];

for (let i = 0; i < 11; i++) {
  lathePoints.push(new Vector2(Math.sin(i * 0.2) * 2, (i - 5) * 0.2));
}

return (
  <>
    <Lathe args={[lathePoints]} rotation={[0, 0, 0]} position={[0, 1.2, 0]}>
      <meshLambertMaterial color="#777" />
    </Lathe>

    <pointLight color="#fff" intensity={10} position={[0, 5, 0]} />

    <Cone args={[1.5, 2, undefined, 50]} position={[0, 2.75, 0]}>
      <MeshDistortMaterial
        distort={0.5}
        speed={10}
        color="#baa8ff"
        transparent
        roughness={0.01}
        transmission={4.25}
        ior={1} />
    </Cone>
    <Cone args={[1, 1.5, undefined, 50]} position={[0, 2.25, 0]}>
      <MeshDistortMaterial
        distort={0.75}
        speed={10}
        color="#fff"
        roughness={0.8}
        ior={7} >
    </Cone>
</>
)

At this point I really had no idea what I was doing, however felt surprisingly confident that I could do some cool stuff for the hackathon.

Starting the project, and the Mirror scene

First off when the hackathon kicked off, I thought that creating a 3D model of the mirror will play an important part in this as I could quickly generate textures for other, simple objects (e.g. walls, pillars, etc.) via an AI. So I fired up Spline and got to work.

After spending several hours on the model, I now just needed to import it to the project and adding some materials to it. Before you can use GLTF files in your project, you'll need to tell Vite to include them in the assets like so:

// https://vitejs.dev/config/
export default defineConfig({
  assetsInclude: ["**/*.gltf"],
  plugins: [react()],
});

It's a bit tedious process to create meshes for the GLTF model by hand, and luckily the Poimandres collective have also created GLTFJSX library to help in that regard. They even have a website to test it out, which I just ended up using directly. It prints out a nicely grouped meshes, which can be altered individually.

At this point, after adding some initial materials to the model as you can see in the latter commit link, I realized that this is going to look very bad if I can't nail the materials perfectly. The alternative route would be creating something with a more oldschool vibe, and mixing 2D with 3D. Basically using sprites in a 3D environment like what they did in old games such as Wolfenstein, Duke Nukem, and Hexen. I especially have always liked the aesthetics of the last game mentioned, so I decided that I could try it out quickly to see if I could make it work.

Here is where DreamStudio by Stability AI, or any current AI solution really, helps out a lot. With just a few prompts, you are able to generate pixel art textures that you can use anywhere. Below are few examples what I ended up generating and using.

You of course need to edit the images a bit, since you want to use transparent images when using sprites for objects. For some repeating textures, however, you might get away with just using the generated entries directly. I myself used Piskel to remove the backgrounds, and later to create animations.

For example, the braziers needed to be animated as it would be a bit boring to have them just sit around with static flame on them. Basically, in order to create an animated sprite, this just means having each state (frame) of the animation lined up next to each other in one big, transparent file. Below is how the brazier file looks like.

Drei was super helpful again, as it comes with a SpriteAnimator component to handle the animation without needing to do it manually. You just give it some props where to start the animation from, how many frames there are, and what texture to use.

export default function Brazier({ position }: BrazierProps) {
  return (
    <Billboard position={position}>
      <pointLight intensity={50} position={[0, 1, 0.1]} castShadow />
      <SpriteAnimator
        autoPlay
        startFrame={0}
        loop
        numberOfFrames={5}
        scale={3}
        textureImageURL={brazier}
      />
    </Billboard>
  );
}

The mirror itself is just a Circle component with a double sided mesh material. How Circle differs from a Sphere is that Circle is two-dimensional, and Sphere three-dimensional. Same goes with Plane and Box, former which I used for bunch of other elements in the scenes. And since I'm working with sprites, I don't want to use textures on a 3D objects as that would wrap it around the object instead of displaying it as intended in the image.

<Circle args={[5, 100]}>
  <MeshReflectorMaterial
   map={texture}
   mirror={0.1}
   alphaTest={0.1}
   side={DoubleSide}
  />
</Circle>

I didn't wrap it in an earlier mentioned Billboard component as, even though working with sprites is more in the 2D realm, you can get a 3D feel by keeping some 2D objects stationary. For this it's important to have a backside for the 2D object, too. If you didn't, and the the camera moved behind the object, it would disappear because there is nothing to render in that direction of the 3D space: the item is facing forward, and since it is 2D object, it does not have any points to draw in the opposite direction. Using side prop and e.g. DoubleSide as the value on the mesh renders the given texture on both sides of the 2D object, making it visible from all angles. Note that this DoubleSide makes object look exactly the same from the front and the back. If you want to have a different looking backside, you'll need to create a separate 2D object with a backside texture, and BackSide as the side prop value.

Tip: Use alphaTest prop to make the mesh material transparent! If you don't set it, and you are using a texture with transparent background, it will get a black background instead.

With some initial objects created, I set up my scene a bit further to see how it would look like. And it was looking pretty decent!

In the GIF above you can see that I initially used 3D pillars (Box component with the pillar texture slapped on it), however I later on decided that they didn't fit in when I added more elements to the scene.

Next up I wanted to work on the "memories" you give to the mirror: how do they appear, what do they look like, how are they passed to the mirror.

First I started with the styling. I wanted to make them something like they were "extracted" from you, something like blobs that would just float around, and be a bit blurry, but just seeing some imagery inside them. And luckily for us, Drei supports these with some premade components!

There is a Float component which allows you to wrap any geometry and make them float: no manual calculations needed! You can adjust the speed, rotation, and floating intensities. Then there is also a MeshTransmissionMaterial component that allows you to create see-through materials. These see-through materials can also warp or adjust the objects/imagery behind/inside based on lighting, etc. This allows you to create some pretty good looking things! With a lot of trial and error I ended up with something like in the picture a below. Don't mind the different looking scene, we'll get to that in a bit.

function Blob({ imageUrl, position, visible }: BlobProps) {
  const texture = useTexture(imageUrl);

  return (
    <group visible={visible}>
      <Float speed={5} rotationIntensity={0.05} floatIntensity={1}>
        <group position={position}>
          <Sphere args={[0.33, 48, 48]} castShadow>
            <MeshTransmissionMaterial
              distortionScale={1}
              temporalDistortion={0.1}
              transmission={0.95}
              color={"#fbd9ff"}
              roughness={0}
              thickness={0.2}
              chromaticAberration={0.2}
              anisotropicBlur={0.5}
              distortion={1.2}
            />
          </Sphere>

          <Sphere args={[0.2, 48, 48]}>
            <meshPhysicalMaterial map={texture} roughness={0.1} />
          </Sphere>
        </group>
      </Float>
    </group>
  );
}

So my blobs are basically just two Sphere geometries on top of each other. One is using the see-through material, and one is a sphere with the image texture. The see-through material on top of the image gives it a nice look that it's actually "living" inside a sphere, and it allows me to achieve that "memory-like" feel.

Tip: Layering order of the materials and geometries matter! > Play around to see what gets you the best results.

At this point we're just moving our camera with basic controls (mainly just zooming and rotating with the mouse), however to make it more immersive, we'll need to add a character. Here we'll be using the previously mentioned Ecctrl library. In addition to installing Ecctrl, it needs Rapier, a physics engine. You'll also need to setup KeyboardControls, which can be exported from Drei.

Setting up keyboard controls is easy: you'll just add the KeyboardControls inside your Canvas component, and make it wrap your scene. Then give it a map of keys to use, and you're good to go! Mine looks something like this:


const keyboardMap = [
  { name: "forward", keys: ["ArrowUp", "w", "W"] },
  { name: "backward", keys: ["ArrowDown", "s", "S"] },
  { name: "leftward", keys: ["ArrowLeft", "a", "A"] },
  { name: "rightward", keys: ["ArrowRight", "d", "D"] },
  { name: "jump", keys: ["Space"] },
  { name: "run", keys: ["Shift"] },
  { name: "crouch", keys: ["c", "C"] },
];

<Canvas>
  <KeyboardControls map={keyboardMap}>
    <ambientLight color={"#fff"} />
    <Suspense fallback={null}>
      <SceneContextProvider>
        <MainScene />
      </SceneContextProvider>
     </Suspense>
  </KeyboardControls>
</Canvas>

Then you'll be needing to add physics so that your character can move properly in the environment. The Controller requires to wrap some sort of a character model in order it to work, so a simple Sphere can suffice in this case. I set it to be invisible, so you don't see it in any material reflections: you just float around. I adjusted the movement speed from the defaults a bit since my scene ended up being pretty big, and without sprinting it would take a long while to get to the mirror.

import { CuboidCollider, Physics, RigidBody } from "@react-three/rapier";
import Controller from "ecctrl";

<Physics gravity={[0, -30, 0]}>
{/* @ts-expect-error the export is slightly broken in TypeScript so just disabling the TS check here */}
  <Controller
    characterInitDir={9.5}
    camInitDir={{ x: 0, y: 9.5, z: 0 }}
    camInitDis={-0.01}
    camMinDis={-0.01}
    camFollowMult={100}
    autoBalance={false}
    camMaxDis={-0.01}
    sprintMult={2}
    maxVelLimit={askForMemories ? 0 : 15}
    jumpVel={askForMemories ? 0 : undefined}>
    <Sphere>
      <meshStandardMaterial transparent opacity={0} />
    </Sphere>
  </Controller>

  {/* other stuff.. */}

  {/* floor */}
  <RigidBody type="fixed" colliders={false}>
    <mesh receiveShadow position={[0, 0, 0]} rotation-x={-Math.PI / 2}>
      <planeGeometry args={[22, 100]} />
      <meshStandardMaterial map={flooring} side={DoubleSide} />
    </mesh>
    <CuboidCollider args={[1000, 2, 50]} position={[0, -2, 0]} />
  </RigidBody>

 {/* other stuff.. */}
</Physics>

Tip: To get the first-person view, you'll need the following props.

<Ecctrl
  camInitDis={-0.01} // camera intial position
  camMinDis={-0.01} // camera zoom in closest position
  camFollowMult={100} // give any big number here, so the camera follows the character instantly
  autoBalance={false} // turn off auto balance since it's not useful for the first-person view

The cool thing about these physics and colliders, is that you can use them as sensors, too. Say you want to trigger some event when the player enters a specific area. You can just define a RigidBody element with a geometry, and a collider (e.g. CuboidCollider) which you mark as a sensor. Then for that collider, you also give a onIntersectionEnter prop a function that will be triggered once the player is inside that collider.

For example, in my case I wanted to make the player unable to move, and focus on an input field so that they can just type without moving the character. I ended up with this simple thing:

<RigidBody
  type={"fixed"}
  position={[0, -3.5, -85.5]}
  rotation={[-Math.PI / 2, 0, 0]}
>
  <Plane args={[4, 3]}>
    <meshStandardMaterial transparent opacity={0} />
  </Plane>

  <CuboidCollider
    args={[2, 2, 1]}
    sensor
    onIntersectionEnter={() => setAskForMemories(true)}
  />
</RigidBody>

So whenever the player enters that 2x2x1 collider in, it will update the state and freeze the character.

Tip: If you don't set type={"fixed"} for the RigidBody,
the collider will collide with other physics and cause some weird behavior on mount.

For example, my collider is slightly inside the mirror stand, and would just fly high up in the sky when the scene loaded. Setting it as fixed would keep it static and fixed in its place.

Next up would be generating images via an Edge Function (as I wanted it this to be without a login and so that not everyone can insert stuff to the database with the anon key), and then displaying the generated images as blobs in realtime. Here I of course used the wonderful Supabase Realtime feature. Below is an example of the wrapping component in order to display the blobs as they appear.

export default function MemoryBlobs() {
  const [memories, setMemories] = useState([]);

  useEffect(() => {
    const channel = supabase
      .channel("memories")
      .on(
        "postgres_changes",
        {
          event: "INSERT",
          schema: "public",
          table: "memories",
          filter: `player_id=eq.${localStorage.getItem("uuid")}`,
        },
        (payload) => {
          setMemories((prev) => prev.concat([payload.new]));
        }
      )
      .subscribe();

    return () => {
      channel.unsubscribe();
    };
  }, []);

  return memories.map((memory: { id: number; image: string }) => (
    <Blob
      key={memory.id}
      position={[0, -2, -84]}
      imageUrl={memory.image}
      visible
    />
  ));
}

Very simple setup, as you can see. I will not show the whole Edge Function code here as it's a bit long, however you can check the source on GitHub to see how these images are generated. Shortly: we take in user input, pass it to OpenAI, pass its response to Stable Diffusion API, upload images generated by SD to Supabase Storage, and then insert them to the database. The database insertion then triggers the realtime updates, and the images appear as blobs on the screen.

If you take a look at the Edge Function code, you notice that I'm using the Fetch API to call OpenAI instead of the OpenAI SDK. This is for a reason: the SDK does not seem to work in these Edge Functions. It all works locally, however when you deploy the function to production and invoke it, it will crash to a FilesAPI undefined error. I'm not sure if my setup is a bit outdated, or if this is something that can be fixed by Supabase (or Deno?).

In order to take in user input, we need to have an input field, of course. It would be a task of its own to build something like that in WebGL, so it's easier to throw HTML in the mix. This can be done easily with the HTML component from Drei. With it, I could add a HTML Form element to my scene and start taking in user input to generate images for the memories.

<Html center position={[0, -1.95, -88]}>
  <form
    onSubmit={async (e) => {
      e.preventDefault();

      setGenerating(true);

      const data = new FormData(e.target);
      const input = data.get("memory");

      const { data: responseData } = await supabase.functions.invoke("generate-memories",
        {
          body: {
            input,
            playerId: localStorage.getItem("uuid"),
          },
        }
      );

      setGenerated(true);

      if (responseData) {
        localStorage.setItem(
          "memoryGroupId",
          JSON.stringify(responseData.memoryGroupId)
        );

        setTimeout(() => {
          setTransitionToVoid(true);
          }, 8000);
       }
     }}
   >
     <input
       className="memory-input"
       name="memory"
       style={{
         display: !askForMemories || generating ? "none" : undefined,
       }}
       ref={inputRef}
       type="text"
       placeholder="Think of a memory..."
       />
  </form>
</Html>

With all this functionality, we can move to the Void scene.

Enter the Void

So in this scene I wanted the user to be in zero-gravity environment, and see these little blobs of memories floating around. Then when you click one, you would zoom in to see the name of the memory and date. In this view you would also have access to the generated images.

For the zooming part, I needed to create a separate camera controller that would allow me to animate things smoothly, and after googling a while how to do this, I found a nice library called camera-controls. You'd hook it to React Three Fiber's useFrame hook, and update the camera position and where it's looking at based on some given coordinates. You can see the implementation in the Controls component.

It's hooked to a context, which stores the current camera position and look-at values from the blob click event, and then it does this nice transition when it needs to update from the previous position. The component also contains keyboard event handling to have that zero-gravity feel when moving the camera.

You'd update the camera by sending new three-dimensional vector values to the context in some component like in MemoryGroups for example.

setCamPos(new Vector3(position[0], position[1], position[2] + 20));
setLookAt(new Vector3(position[0], position[1], position[2]));

Positioning the groups of memories would be next, and it turned out to be one of the coolest things about this project in my opinion. If you've tried the app, you notice that the group spheres place out quite evenly in the space, and they don't really overlap each other. This is thanks to the pre-installed PostGIS plugin on the database.

Even before realizing that I can use PostGIS for the locations, I wanted to make the groups appear in random locations, and they shouldn't overlap each other. My initial idea was that I'd store XYZ coordinates in their own database columns, and I would just check in my code if there are any overlaps with any of the database rows within a given range. Doable? Sure. Reasonable? Maybe not. And here where I realized that since I'm actually working with coordinates, I can use PostGIS directly to handle these spatial coordinates.

PostGIS comes with a built-in function to check if some coordinates overlap each other, which made this whole thing a lot simpler: I could just let the database handle everything! Only thing I needed to do was to send in the given text for the memory, the player ID, and the group would automatically get assigned to some random place in the 3D environment. Of course, since this was a hackathon, and it was my first time using PostGIS, I actually asked the AI Assistant to generate a database function for me! It didn't work straight out of the box, however it was 99.9% there. Very cool and impressive, so kudos to the Supabase team for this feature.

  const { data: memoryGroupId } = await supabaseClient.rpc(
    "insert_memory_group",
    { memory: input, player_id: playerId }
  );

CREATE OR REPLACE FUNCTION public.insert_memory_group(memory text, player_id uuid)
 RETURNS bigint
 LANGUAGE plpgsql
AS $function$
DECLARE
  random_coordinates geometry;
  id bigint;
BEGIN

  LOOP
    -- Generate random coordinates
    random_coordinates := ST_MakePoint(
      random() * 180 - 90,
      random() * 180 - 90,
      random() * 180 - 90
    );

    -- Check for intersecting geometries
    IF NOT EXISTS (
      SELECT 1
      FROM memory_groups
      WHERE ST_Intersects(position, random_coordinates)
    ) THEN
      -- Insert and return data
      INSERT INTO memory_groups (memory, position, player_id)
      VALUES (memory, random_coordinates, player_id)
      RETURNING memory_groups.id into id;

      RETURN id;
      EXIT;
    END IF;
  END LOOP;

END;
$function$
;

Tip: Use Supabase's AI Assistant, it's amazing, and will only get better the more you use it.

However, while these coordinates (or geometries) are now stored properly, you cannot really use them as is in the code. This is because the stored format isn't a regular float for each axis: it's mix of numbers and letters, for example 01010000A0E6100000404C76755AFF35C03C48167A39544940DCF3805DD9663240. So in order to use these Points in our app, we'll need to convert these to floats. Here I used another database function to do the conversion:

CREATE OR REPLACE FUNCTION public.memory_groups_with_position()
 RETURNS TABLE(id integer, memory text, created_at date, x double precision, y double precision, z double precision)
 LANGUAGE sql
AS $function$
   select id, memory, created_at, st_x(position::geometry) as x, st_y(position::geometry) as y, st_z(position::geometry) as z from public.memory_groups;
$function$
;

In the code, when the scene loads, I'd just fetch the groups with an RPC call via Supabase. I also hooked it to the realtime feature, so the scene automatically updates with the latest added memory group if you happen to be there and someone else gives mirror some memory.

  useEffect(() => {
    async function fetchMemoryGroups() {
      const { data } = await supabase
        .rpc("memory_groups_with_position")
        .limit(1000);

      if (data) {
        setMemoryGroups((prev) => prev.concat(data));
      }
    }

    fetchMemoryGroups();
  }, []);

  useEffect(() => {
    const channel = supabase
      .channel("memory_groups")
      .on(
        "postgres_changes",
        { event: "INSERT", schema: "public", table: "memory_groups" },
        async (payload) => {
          const { data } = await supabase
            .rpc("memory_groups_with_position")
            .eq("id", payload.new.id)
            .single();

          setMemoryGroups((prev) => prev.concat([data]));
        }
      )
      .subscribe();

    return () => {
      channel.unsubscribe();
    };
  }, []);

You'll notice that I'll do another fetch for the newly added memory_group after getting notified by the INSERT event in database. This is because of what I mentioned earlier: since the group position is a geometry, I cannot use them directly to position them in the 3D space. Instead, I just use the RPC call to fetch the newly added group, which works perfectly in this case.

After all this, with bunch of little tweaks here and there, adding "transitions" between the scenes, adjusting functionalities, generating music with an AI, being desperate at why the Edge Function is not working at 7am in the morning after working on the project for the whole night, the end result ended up being something like seen in this demo video. It shows a bit more optimal experience since it's running locally, however I'm really happy how it turned out. I had a vision, and managed to complete it to my liking, which is always amazing.

I've left out quite a lot of details since there is a lot of code and things going on, so make sure to check out the GitHub repo for all the missing parts.

If you got this far, thanks for reading! Feel free to add comments if you have anything in mind that you wanna say.