Forem: Jozef Maxted 👻

Skeletonise yourself with AI pose detection

Jozef Maxted 👻 — Wed, 27 Oct 2021 11:24:20 +0000

This post was originally written on the Pixelhop blog: Skeletonise yourself with pose detection

We recently built a fun halloween experiment where we use something called pose detection to skeletonise your body in realtime using your body.

You can try it out for yourself here: Trick or Treat

In this post we will show you how you can create something similar using TensorFlow.js and pose detection models. Take a look here to see what you will build:https://skeletoniser.netlify.app

If you prefer to learn this sort of thing in video format we've got you covered too:

What is pose detection?

What in the world is pose detection, and why might I want to use it? Well in simple terms, it's the process of taking an image or video, locating any bodies in the frame, and then figuring out where their body parts are.

What is pose detection good for? Well it will probably have loads of real world uses! You could use it to create a Yoga app that detects when you are in the correct pose. You could use it to analyse your running technique and provide feedback. You could use it to create interactive art installations. In our case, we decided to test it out on probably the most important use case of all... turning you into a skeleton of course!

Before we begin

You can find the finished code for this tutorial here: https://github.com/pixelhop/skeletoniser

We will be using Vite to bundle our project and run a dev server, so start a new project with:

npm init vite@latest skeletoniser -- --template vanilla-ts

Starting the webcam

So before there's any chance of skeletonising happening, we need something to skeletonise! The first thing we need to do is grab the feed from a webcam to see the ghosts your body.

Creating HTML to display the webcam feed

To see the web cam we first need to add a HTML element that is capable of displaying it. The <video> element is the perfect tool for the job. Update your index.html to contain a video element like below. (Delete the existing <div id="app"></div>)

<!-- index.html -->
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <link rel="icon" type="image/svg+xml" href="favicon.svg" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Vite App</title>
  </head>
  <body>
    <video
      playsinline
      autoplay="true"
      width="640"
      height="480"
    />
    <script type="module" src="/src/main.ts"></script>
  </body>
</html>

Let's quickly talk few the attributes we have added to the video element. Firstly we have added playsinline. This attribute tells mobile browsers that video should play inline on the page rather than their default behaviour of opening up in fullscreen.

The second attribute is autoplay="true". This one does what it says on the tin and means the video element will play automatically as soon as a video is available rather than being paused by default.

Now that we have a video element let us reference it to use it in our code later. Modify src/main.ts to look like the following:

// src/main.ts
import './style.css'

const videoEl = document.querySelector<HTMLVideoElement>('video')!;

You might notice the ! after the querySelector call. This tells Typescript that the video element will definitely exist, so we don't need to account for the chance it could be missing in the code we write later.

Requesting access to the webcam and attaching to the video element

Our video element is all set up, and we have a reference to it. The next step is getting some media to play on it. To do that, we need to request a MediaStream.

// src/main.ts

async function initCamera() {
  const stream = await navigator.mediaDevices.getUserMedia({
    video: {
      facingMode: 'user',
      width: { ideal: 640 },
      height: { ideal: 480 },
    },
    audio: false,
  });

  let onVideoReady: (ready: boolean) => void;
  const videoReadyPromise = new Promise((resolve) => onVideoReady = resolve);
  videoEl.onloadedmetadata = () => onVideoReady(true);

  videoEl.srcObject = stream;

  return videoReadyPromise;
}

async function start() {
    await initCamera();
}

start();

When we call the initCamera function, the user will be prompted to allow access to their webcam. We pass an options object to the getUserMedia function.

Specifying the facingMode property in the video object lets the browser know that it should choose the camera facing the user if there are multiple cameras. On mobile devices, this should mean the selfie camera is selected by default.

We also specify width and height parameters. This tells the browser the ideal dimensions of the video we want to receive. In this case, we have chosen to limit it to 640x480 if possible so that the pose detection will be efficient. The bigger the video, the more resource-intensive the pose detection will be, so 640x480 is a happy medium.

After getting the stream, we can then assign it to the video element we created earlier. Before we do that, we make a new promise that will allow us to wait for the video to completely load before resolving. After assigning a video a media stream, it takes time to initialise everything and actually load the stream. By adding an onloadmetadata event listener to the video element that resolves our promise, we can ensure we don't use the video before it is ready. Once all that's set up, it's time to add the stream to the video by assigning it to the video element's srcObject property.

Hopefully, by this point, you should be seeing your face appearing on the page!

Drawing on top of the video with canvas

Displaying the webcam feed is excellent, but we need to start thinking about manipulating and adding to it. Our goal is to superimpose a skeleton on top of the feed, so we need to draw images on top of it.

Canvas is the perfect tool for the job here. We can add a new canvas element and position it precisely over the top of the video.

Modify index.html to look like the following:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <link rel="icon" type="image/svg+xml" href="favicon.svg" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Vite App</title>
  </head>
  <body>
    <div class="container">
      <video
        playsinline
        autoplay="true"
        width="640"
        height="480"
      ></video>
      <canvas 
        width="640"
        height="480"
      />
    </div>
    <script type="module" src="/src/main.ts"></script>
  </body>
</html>

We have added a wrapper div around the video element. This is so we can position the canvas element absolutely on top of the video. To do that we need to add some css:

// style.css
.container {
  position: relative;
  width: 640px;
  height: 480px;
}

canvas {
  position: absolute;
  top: 0;
  left: 0;
  width: 100%;
  height: 100%;
}

Finally, let's get a reference to the canvas element and get a rendering context so that we can use it later.

// main.ts
const videoEl = document.querySelector<HTMLVideoElement>('video');
const canvasEl = document.querySelector<HTMLCanvasElement>('canvas');
const ctx = canvasEl?.getContext('2d');

async function initCamera() {
  const stream = await navigator.mediaDevices.getUserMedia({
    video: {
      facingMode: 'user',
      width: { ideal: 640 },
      height: { ideal: 480 },
    },
    audio: false,
  });

  let onVideoReady: (ready: boolean) => void;
  const videoReadyPromise = new Promise((resolve) => onVideoReady = resolve);
  videoEl.onloadedmetadata = () => onVideoReady(true);

  videoEl.srcObject = stream;

  return videoReadyPromise;
}

async function start() {
    await initCamera();
}

start();

Pose detection with TensorFlow.js and BlazePose

Ok, here comes the fun part! Now that our webcam is up and running and we have a way to draw over the top of the video feed, it's time to use AI to analyse the video feed and detect any poses skeletons within the video.

At this point your might be thinking: "Jozef, I am just a lowly frontend developer. Don't I need to be genius AI, machine learning guru to be able to do pose detection?" That's very humble of you but don't put yourself down like that! TensorFlow.js make all of this very clever AI stuff extremely accessible to any frontend developer!

TensorFlow.js is a library that enables machine learning within the browser. There are loads of pre-built models that can achieve all sorts of tasks from object detection to speech recognition, and thankfully for us, pose detection. You can take a look at the available models here: https://www.tensorflow.org/js/models

In addition to the pre-built models, you can train your own, but let's save that for another article!

The TensorFlow.js pose detections models can be found here: https://github.com/tensorflow/tfjs-models/tree/master/pose-detection

As you can see, they have 3 models to choose from, each of which has pros and cons. We decided to go with BlazePose because we found it to have good performance, and it provides additional tracking points that could be useful.

Let's install the TensorFlow.js dependencies so we can use it in our project.

npm i @tensorflow-models/pose-detection@0.0.6 @tensorflow/tfjs-backend-webgl@3.8.0 @tensorflow/tfjs-converter@3.8.0

Note we have locked the TensorFlow dependencies to 3.8.0 because there were issues with later versions.

Now that we have got TensforFlow.js installed let's update src/main.ts to make use of the new packages and start pose detecting!

// main.ts
import * as poseDetection from '@tensorflow-models/pose-detection';
import '@tensorflow/tfjs-backend-webgl';
import './style.css';

const videoEl = document.querySelector<HTMLVideoElement>('video');
const canvasEl = document.querySelector<HTMLCanvasElement>('canvas');
const ctx = canvasEl?.getContext('2d');

async function initCamera() {
  const stream = await navigator.mediaDevices.getUserMedia({
    video: {
      facingMode: 'user',
      width: { ideal: 640 },
      height: { ideal: 480 },
    },
    audio: false,
  });

  let onVideoReady: (ready: boolean) => void;
  const videoReadyPromise = new Promise((resolve) => onVideoReady = resolve);
  videoEl.onloadedmetadata = () => onVideoReady(true);

  videoEl.srcObject = stream;

  return videoReadyPromise;
}

async function initPoseDetection() {
  const model = poseDetection.SupportedModels.BlazePose;
  const detector = await poseDetection.createDetector(model, {
    runtime: 'tfjs',
    modelType: 'lite',
    maxPoses: 1,
  } as poseDetection.BlazePoseTfjsModelConfig);

  return detector;
}

async function start() {
  await initCamera();
  const detector = await initPoseDetection();

  async function render() {
    const poses = await detector.estimatePoses(videoEl!);
    console.log(poses);
    requestAnimationFrame(render);
  }
}

start();

Let's talk through the changes here. Firstly at the top of the file, you can see the new imports. We import the pose detection library and the WebGL backend for TensorFlow.js. This allows TensorFlow.js to run on the graphics processor resulting in much faster performance than running on the CPU.

The next addition is the initPoseDetection function. Here we chose BlazePose as a model of choice, and then we instantiated a new detector using the BlazePose model. When doing this, we pass a config object allowing us to customise how the model will run.

We set the runtime attribute to "tfjs". This tells the detector to use TensorFlow.js (you can also use something called MediaPipe, but the performance varies massively between browsers).

The model attribute is set to "lite". There are three types of BlazePose model to choose from, "lite", "full", and "heavy". These change the quality of the detection model. Lite has the least accurate detection but is the best performing, whilst "heavy" has the best accuracy but is very performance-intensive, and "full" is somewhere in the middle. We have chosen "lite" because we want this to work in real-time and on weaker devices if possible.

In case you hadn't guessed, maxPoses specifies how many poses the detector should search for. But, again, for the sake of performance, we are limiting it to one.

The start function has been updated to create our pose skeleton detector after the camera is initialised, and we have also added a new render function. We will detect the poses on each video frame and eventually draw them to the canvas. For the time being, we are just detecting the poses with the line:

const poses = await detector.estimatePoses(videoEl!);

How easy is that! You just pass the estimatePoses function your video element, a lot of AI magic happens in the background, and before you know it, you'll have a nice array of poses.

If you look in your console now, you should see s steady stream of pose arrays logging into your console. If you open one of them out and look inside, it should look something a bit like this:

[
    {
        "score": 0.9999910593032837,
        "keypoints": [
            {
                "x": 362.1851530148483,
                "y": 235.49307254371956,
                "z": -729053.3069949468,
                "score": 0.9998758783456222,
                "name": "nose"
            },
            {
                "x": 381.78655583117063,
                "y": 195.58328362686387,
                "z": -684314.8261948047,
                "score": 0.9996265026574452,
                "name": "left_eye_inner"
            },
            {
                "x": 396.3415603433863,
                "y": 198.3687794162295,
                "z": -684276.2656579157,
                "score": 0.9997904934464069,
                "name": "left_eye"
            },
            {
                "x": 408.91519818260184,
                "y": 201.09340840610764,
                "z": -683932.6845901452,
                "score": 0.999563694033746,
                "name": "left_eye_outer"
            },
            {
                "x": 336.5293852396535,
                "y": 189.06441746316608,
                "z": -694288.547797726,
                "score": 0.9996265177033095,
                "name": "right_eye_inner"
            },
            {
                "x": 316.81846770951233,
                "y": 188.4412867665395,
                "z": -693936.2406338064,
                "score": 0.9997704015296288,
                "name": "right_eye"
            },
        ]
    }
]

The top-level score property is the confidence that a pose is actually in the frame. After that, we have keypoints. These contain the coordinates of all the bits of your body. Let's just stop a second and think about how amazing it is that you can get to this point entirely in the browser!

Not so long ago, pose tracking like this would require one of those ridiculous bobble suits and a costly camera set up like you see when CGI scenes are being recorded at the movies. Now you can get pretty good results with your selfie cam and Google Chrome!

Drawing the detected pose

It's about time we start putting all this clever AI to use and drawing our skeleton. For a simple first step, we will attempt the hello world of "skeletonising" and mark the components of our head using simple dots and lines.

To do this, we can create a new "Skeleton" class that will handle the drawing of each body part. For the time being, we will only draw the head. The start of our Skeleton class can be seen below:

Drawing the detected pose

To do this, we can create a new "Skeleton" class that will handle the drawing of each body part. For the time being, we will only draw the head. The start of our Skeleton class can be seen below:

// src/skeleton.ts
import { Pose } from '@tensorflow-models/pose-detection';

export class Skeleton {
    constructor(private ctx: CanvasRenderingContext2D) {}

    private drawHead(pose: Pose) {
        const leftEye = pose.keypoints.find((keypoint) => keypoint.name === 'left_eye');
        const rightEye = pose.keypoints.find((keypoint) => keypoint.name == 'right_eye');
        const leftMouth = pose.keypoints.find((keypoint) => keypoint.name === 'mouth_left');
        const rightMouth = pose.keypoints.find((keypoint) => keypoint.name === 'mouth_right');
        const nose = pose.keypoints.find((keypoint) => keypoint.name === 'nose');

        this.ctx.fillStyle = 'red';
        this.ctx.strokeStyle = 'red';
        this.ctx.lineWidth = 5;
        if (leftEye) {
            this.ctx.beginPath();
            this.ctx.arc(leftEye.x - 10, leftEye.y - 10, 10, 0, 2 * Math.PI);
            this.ctx.fill();
        }
        if (rightEye) {
            this.ctx.beginPath();
            this.ctx.arc(rightEye.x - 10, rightEye.y - 10, 10, 0, 2 * Math.PI);
            this.ctx.fill();
        }
        if (nose) {
            this.ctx.beginPath();
            this.ctx.arc(nose.x - 10, nose.y - 10, 10, 0, 2 * Math.PI);
            this.ctx.fill();
        }

        if (leftMouth && rightMouth) {
            this.ctx.beginPath();
            this.ctx.moveTo(leftMouth.x, leftMouth.y);
            this.ctx.lineTo(rightMouth.x, rightMouth.y);
            this.ctx.stroke();
        }
    }

    public draw(pose: Pose) {
        this.drawHead(pose);
    }
}

So here you can see we have a class with two methods. When we instantiate a new Skeleton, we will pass it to the canvas rendering context via the constructor. This will allow the Skeleton to use it later so it can draw the skeleton parts.

Next, we have the drawHead method. As you have probably guessed, this will be responsible for rendering the head. Eventually, we will have a function for each body part, but the head is enough to get things started.

Each time the drawHead method is called, we will pass it the current pose object from BlazePose. At the beginning of the function, we find each key point that makes up the head.

Once we have the key points, we then go about drawing them. For the eyes and nose, we draw a simple circle at their coordinates. For the mouth, we draw a line between the left and right sides of the mouth.

We can now update main.ts to create a new Skeleton and call its draw function on render.

// main.ts
async function start() {
  await initCamera();
  const detector = await initPoseDetection();
  const skeleton = new Skeleton(ctx);

  async function render() {
    const poses = await detector.estimatePoses(videoEl!, {
      maxPoses: 1,
      flipHorizontal: false,
      scoreThreshold: 0.4,
    });

    ctx.clearRect(0, 0, 640, 480);

    if (poses[0]) {
      skeleton.draw(poses[0]);
    }

    requestAnimationFrame(render);
  }

  render();
}

You should now have something that looks a bit like this:

The pose detection is working! Of course, the code we have just written to draw the head will be replaced, but it's good to see visual proof that the pose detection is working as it should.

Mapping images to body parts

So as you can see, drawing dots and lines over the pose key points is very easy. Unfortunately, dots and lines make a pretty awful Halloween costume, so replacing them with illustrated skeleton pieces would be better.

We got a skeleton illustration and separated all the body parts and exported them, individual, from Figma. Here is our skeleton:

To map an image to the correct position, there are a few things we need to work out.

The position of the image. It's x and y coordinates.
The angle of the image. Do we need to rotate it to fit correctly?
The height and width. Do we need to scale it horizontally or vertically?

For the head, we can use the nose keypoint as the central coordinates.
To figure out the rotation, we can calculate the angle between the two eye key points.
Finally, to figure out the horizontal scale, we can calculate the distance between the eyes and then compare it to the actual distance between the eyes in our head image.

We can do the same for the horizontal scale by measuring the height between the mouth and the eyes.

All of the above can be worked out using Vector math. We have created a small helper file to do this:

// src/vectors.ts
export interface Vector {
    x: number;
    y: number;
}

/**
 * Find the middle point between two vectors
 */
export function getMiddle(a: Vector, b: Vector): Vector {
  return {
    x: a.x + (b.x - a.x) * 0.50,
    y: a.y + (b.y - a.y) * 0.50,
  };
}

/**
 * Find the distance between two vectors
 * @param a
 * @param b
 */
export function getDistance(a: Vector, b: Vector): number {
  const x = a.x - b.x;
  const y = a.y - b.y;
  return Math.sqrt(x * x + y * y);
}

/**
 * Get the angle in degrees between two vectors
 * @param a
 * @param b
 */
export function getAngle(a: Vector, b: Vector) {
  return (Math.atan2(b.y - a.y, b.x - a.x) * 180) / Math.PI;
}

Our vector helper file can now be used in the Skeleton draw functions to figure out the image positioning:

import { Pose } from '@tensorflow-models/pose-detection';
import { getAngle, getDistance, getMiddle } from './vectors';

// Skeleton parts
import Head from './assets/skeleton/Head.png';

// Load the skeleton parts
const head = new Image();
head.src = Head;

export class Skeleton {
    constructor(private ctx: CanvasRenderingContext2D) {}

    private drawHead(pose: Pose) {
        const leftEye = pose.keypoints.find((keypoint) => keypoint.name === 'left_eye');
        const rightEye = pose.keypoints.find((keypoint) => keypoint.name == 'right_eye');
        const leftMouth = pose.keypoints.find((keypoint) => keypoint.name === 'mouth_left');
        const rightMouth = pose.keypoints.find((keypoint) => keypoint.name === 'mouth_right');
        const nose = pose.keypoints.find((keypoint) => keypoint.name === 'nose');

        // The real dimensions of keypoints in our head image
        const eyeWidth = 205;
        const eyesToMouth = 220;

        // If we are missing any parts return early
        if (!leftEye || !rightEye || !leftMouth || !rightMouth || !nose) {
            return;
        }

        const eyeAngleDeg = getAngle(leftEye, rightEye);
        const distance = getDistance(leftEye, rightEye);
        const xScale = distance / eyeWidth;

        const middleEye = getMiddle(leftEye, rightEye);
        const middleMouth = getMiddle(leftMouth, rightMouth);
        const mouthToEyeDistance = getDistance(middleEye, middleMouth);
        const yScale = mouthToEyeDistance / eyesToMouth;

        this.drawImage({
            image: head,
            x: nose.x,
            y: nose.y,
            height: head.height * yScale,
            width: head.width * xScale,
            rotation: eyeAngleDeg,
            offsetX: 0.55,
            offsetY: 0.8,
        });
    }

    private drawImage(options: {
        image: HTMLImageElement,
        x: number,
        y: number,
        height: number,
        width: number,
        rotation: number,
        offsetX: number,
        offsetY: number,
      }): void {
        const {
          image,
          x,
          y,
          height,
          width,
          rotation,
          offsetX,
          offsetY,
        } = options;
        // save the unrotated context of the canvas so we can restore it later
        // the alternative is to untranslate & unrotate after drawing
        this.ctx.save();

        // move to the center of the canvas
        this.ctx.translate(x, y);

        // rotate the canvas to the specified degrees
        this.ctx.rotate(((180 + rotation) * Math.PI) / 180);

        // draw the image
        // since the ctx is rotated, the image will be rotated also
        this.ctx.drawImage(image, 0 - (width * offsetX), 0 - (height * offsetY), width, height);

        // we’re done with the rotating so restore the unrotated ctx
        this.ctx.restore();
      }

    public draw(pose: Pose) {
        this.drawHead(pose);
    }
}

There's quite a lot of changes here so let's go through them. Starting at the top of the file we import our shiny new vector helpers. We also import the image that will be used for our head, and create a new Image and set its src. This will allow us to draw it to the canvas with drawImage later.

import { getAngle, getDistance, getMiddle } from './vectors';

// Skeleton parts
import Head from './assets/skeleton/Head.png';

// Load the skeleton parts
const head = new Image();
head.src = Head;

Moving on to the drawHead function:

private drawHead(pose: Pose) {
        const leftEye = pose.keypoints.find((keypoint) => keypoint.name === 'left_eye');
        const rightEye = pose.keypoints.find((keypoint) => keypoint.name == 'right_eye');
        const leftMouth = pose.keypoints.find((keypoint) => keypoint.name === 'mouth_left');
        const rightMouth = pose.keypoints.find((keypoint) => keypoint.name === 'mouth_right');
        const nose = pose.keypoints.find((keypoint) => keypoint.name === 'nose');

        // The real dimensions of keypoints in our head image
        const eyeWidth = 205;
        const eyesToMouth = 220;

        // If we are missing any parts return early
        if (!leftEye || !rightEye || !leftMouth || !rightMouth || !nose) {
            return;
        }

        const eyeAngleDeg = getAngle(leftEye, rightEye);
        const distance = getDistance(leftEye, rightEye);
        const xScale = distance / eyeWidth;

        const middleEye = getMiddle(leftEye, rightEye);
        const middleMouth = getMiddle(leftMouth, rightMouth);
        const mouthToEyeDistance = getDistance(middleEye, middleMouth);
        const yScale = mouthToEyeDistance / eyesToMouth;

        this.drawImage({
            image: head,
            x: nose.x,
            y: nose.y,
            height: head.height * yScale,
            width: head.width * xScale,
            rotation: eyeAngleDeg,
            offsetX: 0.55,
            offsetY: 0.8,
        });
    }

Here you can see we have gotten rid of the quick dot/line drawing and replaced it with a lot of maths! We have two constants for the eye width and the eye to mouth height.

These are the actual pixel distance between the eyes and the eyes to mouth in Head.png, and we can use these to scale our image relative to the lengths in the detected pose.

We work out the rotation for the head image by getting the angle between the two eye keypoints:

const eyeAngleDeg = getAngle(leftEye, rightEye);

The x-axis scale is worked out by measuring the distance between the two eye key points and then dividing it by the real pixel width:

const distance = getDistance(leftEye, rightEye);
const xScale = distance / eyeWidth;

We can then do something similar to figure out the y-axis scale:

const middleEye = getMiddle(leftEye, rightEye);
 const middleMouth = getMiddle(leftMouth, rightMouth);
 const mouthToEyeDistance = getDistance(middleEye, middleMouth);
 const yScale = mouthToEyeDistance / eyesToMouth;

The only difference here is that we find the middle point between the eyes and the central point between both sides of the mouth and then calculate the height.

Now that all the maths is done, its time to draw the head image to the canvas.

this.drawImage({
    image: head,
    x: nose.x,
    y: nose.y,
    height: head.height * yScale,
    width: head.width * xScale,
    rotation: eyeAngleDeg,
    offsetX: 0.55,
    offsetY: 0.8,
});

We are using another new function on the Skeleton class here called drawImage, which we will talk about in a bit. Still, for the time being, you can see that we are using the variables we calculated above to position, scale and rotate the image correctly.

Now back to the Skeleton's drawImage function.

private drawImage(options: {
    image: HTMLImageElement,
    x: number,
    y: number,
    height: number,
    width: number,
    rotation: number,
    offsetX: number,
    offsetY: number,
  }): void {
    const {
      image,
      x,
      y,
      height,
      width,
      rotation,
      offsetX,
      offsetY,
    } = options;
    // save the unrotated context of the canvas so we can restore it later
    this.ctx.save();

    // move to the center of the canvas
    this.ctx.translate(x, y);

    // rotate the canvas to the specified angle
    this.ctx.rotate(((180 + rotation) * Math.PI) / 180);

    // draw the image because ctx is rotated, the image will be rotated also
    this.ctx.drawImage(image, 0 - (width * offsetX), 0 - (height * offsetY), width, height);

    // restore the unrotated ctx
    this.ctx.restore();
}

This function allows us to draw an image and rotate it around an anchor point of our choosing. By default, the canvas drawImage function will position an image from its top-left coordinate. Unfortunately, this makes it awkward for us to position and rotate the image to match the pose.

Using the canvas translate and rotate functions before drawing the image allows us to position it and rotate how we need to.

Hopefully, by this point, you are starting to look a bit sillier like I am below:

Where's my body?

We've got a lovely looking skeleton skull, but we are still missing a lot of bones. Luckily we now have everything in place to draw the rest of the body. We need to create draw functions for each of the other body parts. I won't copy all the draw functions here because there are loads of them, but if you want to see them all, check them out on the repo: https://github.com/pixelhop/skeletoniser/blob/main/src/skeleton.ts

Happy Halloween 🎃

We hope you enjoyed this post and have been able to adequately skeletonise yourself as a result! We love projects that are a bit of fun like this and can't wait to keep experimenting.

As always, if you enjoyed this post or have any questions feel free to reach out to us on Twitter: @pixelhopio or subscribe to our newsletter to be kept up to date with what ever crazy things we build next!

Generate dynamic social images with Nuxt, Cloud functions and Cloudinary

Jozef Maxted 👻 — Wed, 16 Jun 2021 10:20:51 +0000

More and more, we are noticing websites with really slick automatically generated social sharing images, like the ones below:

This can really make your content stand ou and boost engagement when sharing.

We have created a tutorial on how to dynamically generate these images using Nuxt, Firebase cloud functions and Cloudinary. Follow along in the video below, or if you prefer learning in text format, we have written it up on our blog too. Dynamic social images with Nuxt, Cloud functions and Cloudinary

Let us know what you think, and we look forward to seeing your awesome sharing images soon! If you have any questions feel free to reach out to us on Twitter: @pixelhopio

Get started with Webmentions in Gridsome

Jozef Maxted 👻 — Wed, 13 May 2020 13:02:32 +0000

This article was originally posted here: Pixelhop: Get Started With Webmentions

We ❤️ Gridsome the static site generator for Vue.js. Gridsome is a great tool for creating personal blogs, and when you are creating a blog it's nice to allow people to interact with your posts. It's rewarding to see people commenting and liking your hard work.

It's not immediately obvious how you can achieve this with a static site. Everything is static so how can I allow dynamic interactions like commenting and liking? You could add a plugin like Disqus, but then you would be dumping a load of third party JavaScript onto your lean mean blogging machine. Why eat into that precious static site performance when there is another option? 😀

🚨 Webmentions to the rescue

Luckily for us, the W3C has got us covered with their draft of the Webmention spec. In case you haven't heard of Webmentions before here is a quick summary:

Webmention is a web standard for mentions and conversations across the web, a powerful building block that is used for a growing federated network of comments, likes, reposts, and other rich interactions across the decentralized social web. - https://indieweb.org/Webmention

You can think of Webmentions as a tool to record interactions with your content across multiple platforms. If someone mentions your post on Twitter, responds to it in their own blog post, or likes something you posted on Instagram, all of these things and more can be recorded as Webmentions and displayed on your site.

Sounds cool right? Let's start by seeing how we can pull Twitter mentions into a Gridsome site.

Setting up Webmentions

Before you can start with Webmentions you need to add a link tag to the head of your site that links your site to your other social accounts. We are going to connect our Twitter. To do this in Gridsome you will need to add some code to the src/main.js file:

export default function (Vue, { router, head, isClient }) {
  head.link.push({
    rel: 'me',
    href: 'https://twitter.com/pixelhopio',
  });

  // Set default layout as a global component
  Vue.component('Layout', DefaultLayout);
}

Remember to replace our Twitter profile with your own 🤪This will produce the following link in the head of your site once it's rendered:

<link rel="me" href="https://twitter.com/pixelhopio">

Webmentions.io

Next, we need login to https://webmention.io/ webmention.io is a free service that allows us to easily use Webmentions on our static sites. It's worth pointing out that you could host a service like webmention.io yourself if you wanted full control, but for the purpose of this tutorial, it's much quicker to use their offering.

Go to their homepage and enter your site's URL in the "Web Sign-In" field. If your site has deployed correctly with the Twitter link in your head you should be redirected to Twitter to sign in.

After signing in successfully you will be redirected back to a (probably) empty Webmentions dashboard.

Adding the Webmention and pingback links

We now need to add some more header links our site. Luckily we have had some practice at this already. From the webmentions dashboard go to the "Settings" page.

There you will find a few bits of information, but for the time being, we are interested in the link tags.

Open up src/main.js again and add the following, swapping out the hrefs for your own links.

...
export default function (Vue, { router, head, isClient }) {
  head.link.push({
    rel: 'me',
    href: 'https://twitter.com/pixelhopio',
  });

  head.link.push({
    rel: 'webmention',
    href: 'https://webmention.io/pixelhop.io/webmention',
  });

  head.link.push({
    rel: 'pingback',
    href: 'https://webmention.io/pixelhop.io/xmlrpc',
  });

  // Set default layout as a global component
  Vue.component('Layout', DefaultLayout);
}

Once you have done this deploy your site again and once it is done it will be ready to receive Webmentions 🎉

Receiving Webmentions from Twitter

Now that we have done the initial Webmentions setup we are ready to configure receiving Webmentions from Twitter. For this, we can use another free service call brid.gy

Like webmentions.io, you could host your own version of brid.gy if you really wanted to.

After configuring brid.gy anytime someone interacts we a tweet containing your site's domain name, it will send a Webmention.

Before you can connect brid.gy you need to make sure you have your site's domain listed in your Twitter profile. Like below where I have my personal site in my bio:

After doing that, you are ready to log in with brid.gy. From their homepage click "Twitter" and log in using your Twitter account.

That's it, Twitter will now produce Webmentions for your site.

At this point, I usually like to create a test tweet so we can check everything is working. Compose a tweet containing a link to a page on your site. Like it an retweet it with a comment. You will look a bit silly talking to yourself but it will produce a Webmention we can test with later. You can delete the tweet once you are done testing.

Once you have done that, click the "poll now" button in the brid.gy dashboard and you should see your tweet appear in the responses list.

Load Webmentions in Gridsome

Ok, so we have done most of the initial setup and its not time to load them as a source in Gridsome. The easiest way to do this is to use the Webmention source plugin I created:

https://gridsome.org/plugins/@zefman/gridsome-source-webmention

Install the plugin with:

npm install @zefman/gridsome-source-webmention

Once the plugin is installed you need to configure it in gridsome.config.js

export default {
  plugins: [
    {
      use: '@zefman/gridsome-source-webmention',
      options: {
        domain: 'pixelhop.io', // Your webmention domain
        token: 'your-token', // Your secret webmention token
      },
    }
  ]
}

You will need to get your webmention.io API key from the settings page: https://webmention.io/settings

If you restart your Gridsome dev server your Webmentions will now be available using the GraphQL API 🚀

Below is an example of how you can load Webmentions for a specific page in our Gridsome site. We can filter the Webmentions by using our pages path.

<page-query>
  query($path: String!) {
    mentions: allWebMention(filter: { wmTarget: { regex: $path } }) {
      edges {
        node {
          wmId
          url
          wmProperty
          wmSource
          content {
            text
          }
          author {
            name
            photo
            url
          }
        }
      }
    }
  }
</page-query>

Now that you can get the Webmention data in your page you can create a Vue component to render them easily. Heres an example of what the Webmention section on https://pixelhop.io looks like:

Trigger Builds With Webhooks

We are nearly there I promise! The final step in our Webmention journey is to get new Webmentions to appear automatically on your site. Currently, new Webmentions will only appear when you manually rebuild and deploy your site.

That's a bit rubbish, but luckily we can trigger a new build automatically using webhooks. We use Netlify for our sites, but must modern static site hosts allow you to trigger builds via a webhook. Here's how you do it in Netlify.

From the Netlify dashboard find your site, then go to Settings > Build & Deploy and scroll down the to "Build hooks" section.

Click add build hook, and call it "webmentions":

Copy the URL that is generated after clicking save.

Finally, head back to webmention.io and navigate to the "Web Hooks" section. As of the time of writing, you can find it here: https://webmention.io/settings/webhooks

Paste in the URL from Netlify into the URL field and click save. That's it now a build will be triggered when you have new mentions and they will automatically show up on your site.

And......relax 🧘‍♀️

Well, we did it! Hopefully, if you made it this far your site is swimming in lovely Webmentions. There's a quite a number of steps to get Webmentions working but the benefits make it worth it.

0 Responses