Forem: Mike Talbot ⭐

Fascinating reading, very thought-provoking content...

Mike Talbot ⭐ — Mon, 22 Dec 2025 15:29:50 +0000

GrahamTheDev

Dec 21 '25

Mean Time to Understanding 🤔: The Irreducible Human Element in the Age of Infinite Code 🤖.

#ai #webdev #programming

Comments 24

16 min read

Interview Question: how do you ensure you only virus scan once?

Mike Talbot ⭐ — Sun, 08 Dec 2024 15:19:22 +0000

I see a lot of interview questions being proposed that are around DSA and lower level algorithms that I don't really expect people to be writing in 2024.

Over the years I've interviewed thousands of developers and run teams of hundreds, so my team and I have developed questions that aim to test people's understanding of challenges they will face frequently.

Example interview question

You are writing a module that requires files to be uploaded to a server for processing. All uploaded files must be virus scanned. It is quite likely that users will upload the same file throughout their usage lifetime; multiple different users may also upload the same file. How do we minimise the time impact and cost of virus scanning the same file multiple times? If we need to store the file, can your solution optimise storage? How does your method affect the UX of the solution? How fault-tolerant is your method?

It's questions like this that I want my team to be able to reason out good answers to. A good solution will require an understanding of the tools that could be used and their performance and memory characteristics; I'd expect good answers to touch on services or tools that could be expected to help.

A suitable answer would be:

I would create a name for the file based on a V5 GUID, those kind of GUIDs are derived from the contents of the file. I'd check whether a file already existed with the name on my storage, and if it did, I'd see whether it was flagged as infected and report this immediately. If the name file did not exist, I'd virus scan the file and upload it with an indication of whether it was clean or infected. I only virus scan files once, and I only upload files once; this minimises the time taken and the cost of server resources while retaining a good UX.
If we need to store the file multiple times? My solution naturally stores the file only once and then uses the name as a reference. This has the added advantage that it's easy to find all the places where the same file is used.
How does your method affect the UX of the solution? Users can just upload files; they don't have to worry about anything, it minimises any impact on them.
How fault-tolerant is your method? It is pretty fault tolerant; although new viruses emerge and new detections are implemented, you could either record each filename and the version that was used to scan it, minimising rescan time, or you could change the namespace of the V5 GUID so that future uploaded files would be scanned again at the expense of duplicating storage.
What tools would you use? I'd store the name in a database for the user; I might use Redis to check file names to improve performance.
Why did you choose this method? The lookup of the file is an O(1) operation after the name has been created, which is the optimal approach as the system scales. The creation of the name is an O(N) operation based on the size of the file being uploaded, but I have that file and need to process it anyway so reading it seems to be something that already needs to happen.

Guiding interviewees

Suppose an interviewee started down a track of suggesting that the user should be responsible for ensuring a file existed only once. In that case, the key is to guide them into considering the poor UX this would give users. Some candidates can't get away from questions like these or challenges to the requirements, those people aren't the ones I'm looking for and if after guidance, they haven't found a solution that minimises costs and maximises UX, then I've got my answer.

Conclusion

I think it's vital to see how people reason with real-world problems rather than parrot back algorithms that you can learn by rote.

There are, of course, many ways to solve this problem; it's by understanding the thought processes of the interviewee that you learn their style and their ways of reasoning. This is the vital thing for me.

MapPlus: a better "Map" makes your code easier to read...

Mike Talbot ⭐ — Fri, 06 Dec 2024 10:26:49 +0000

TL;DR

I introduce MapPlus, a simple class that makes your code easier to read and ensures Maps are easier to reason with.

What is Map

In JavaScript, the Map is a very useful built-in class that creates an O(1) lookup between a key and a value.

const myMap = new Map()

for(const file of files) {
    const [,extension] = file.name.split(".")
    if(!myMap.has(extension)) {
        myMap.set(extension, [])
    }
    myMap.get(extension).push(file)
}

You can use Maps for all sorts of things you are likely to do regularly:

Creating grouped lists of data, like the above example grouping by file extension
Aggregating data, like counting or summing values across a range of keys

const items = ['apple','apple','orange','banana','apple'];
const counts = new Map();
for (const item of items) {
  counts.set(item, (counts.get(item) || 0) + 1);
}

Creating rapid lookups to be used in subsequent steps

const users = [
  {id:1,name:'A',role:'admin'},
  {id:2,name:'B',role:'user'},
  {id:3,name:'C',role:'user'}
];
const userMap = new Map();
for (const u of users) {
  userMap.set(u.id, u);
}

Why use Map?

Map is preferred to using a simple object ({}) for a couple of reasons, so long as you don't have to store the result using a stringify:

It can take keys which are not strings
It's slightly faster than an Object even if you are using string keys

There can be a lot of boilerplate and mixed concerns though, if the object you are storing in the map needs construction, which is anything from a simple array to a complex object, this needs to be interspersed with the code that uses it.


const map = new Map()

for(const item of items) {
   if(!map.has(item.type)) {
       const newType = new Type(item.type, getInfoForType(item.type))
       map.set(item.type, newType)
   }
   map.get(item.type).doSomething(item)

}

This "can" be ok, but it becomes harder to keep DRY if you need to update or initialise the value in multiple places.

For this reason I use a MapPlus class, which is an extension to Map that provides a missing key initialiser function that can be supplied to the constructor or as a second parameter to the get if the initialiser does need in context information beyond just the key.

The MapPlus Class

class MapPlus extends Map {
    constructor(missingFunction) {
        super()
        this.missingFunction = missingFunction
    }

    get(key, createIfMissing = this.missingFunction) {
        let result = super.get(key)
        if (!result && createIfMissing) {
            result = createIfMissing(key)
            if (result && result.then) {
                const promise = result.then((value) => {
                    super.set(key, value)
                    return value
                })
                super.set(key, promise)
            } else {
                super.set(key, result)
            }
        }
        return result
    }
}

With this you can just do things like:

const map = new MapPlus(()=>[])

for(const item of items) {
    map.get(item.type).push(item)
}

If the key is missing it will just make an empty array, but the loop itself is kept simple and clean, concerned only with making the list.

I often need two levels of this so I'll have maps defined like this:

const map = new MapPlus(()=>new MapPlus(()=>[]))
for(const item of items) {
   map.get(item.type).get(item.subType).push(item)
}

The constructor function does get the key being used so we can also do:

const map = new MapPlus((type)=>new Type(type, getInfoForType(type))

for(const item of items) {
    map.get(item.type).doSomething(item)
}

A debate to be had.

Mike Talbot ⭐ — Sat, 30 Nov 2024 10:34:49 +0000

Why AI Can't Replace Developers: The Real Limits of AI in Coding

Michael Amachree ・ Nov 26

#ai #coding #development #tools

One Byte Explainer: Deadly Embrace

Mike Talbot ⭐ — Fri, 14 Jun 2024 11:57:08 +0000

This is a submission for DEV Computer Science Challenge v24.06.12: One Byte Explainer.

Deadly embrace stops the world dead when one function holds a "lock" that is also required by a function it calls directly or indirectly. Neither can proceed so it's game over. It can be tough to find and debug.

Coding solutions with AI

Mike Talbot ⭐ — Sun, 24 Mar 2024 15:36:01 +0000

We talk a lot on Dev about how we are worried that AI will take our jobs, that it will write code instead of us. I guess we all have a position on that and this article isn't about this subject at all.

What I want to write about is the new multi-tool in my toolbox that I seem to reach for more and more frequently as it continually succeeds in solving my customers' problems. You may have guess that the tool is GPT4-Turbo.

The problem I list below was solved on a train ride of about 90 minutes, including all coding, debugging and running.

The world changed

I've found myself selecting problems to solve that would previously have been left low on the list because of complexity, or that would have been a mostly human task where the resources committed would have been way too costly for the benefit delivered.

Solving problems with AI

If we are going to use AI to solve a problem we have to break down the challenge using the thought processes we use on a daily basis as programmers. We need to decompose problems into multiple smaller units and then solve those.

We can choose to select AI as the solution for a sub problem when that problem is not a clear formula or data manipulation, the most obvious place for this is when there is natural language involved. Natural language is a complex problem, the same meaning can be represented in many different ways - there are lots of other similar classes of problem with scientific measurements, financial data etc, but language is the one I personally run into the most.

A simple example

I've already solved 5 major problems for my solution using the processes I'll talk about here. I'd bore you to tears if I explained some of the complicated ones and I'd need to give you way too much data to explain the issue.

My business

I'm the CTO of a business that aims to ensure buildings are safe for their residents or occupants. For more than 30 years SFG20 has created the industry standard for maintenance regimes that seek to ensure our clients do the activities that prevent harm to individuals and ensure equipment has the longest possible life. Many of these tasks are legal requirements and in recent years the law has changed dramatically after a residential tower block fire in London claimed 72 lives.

SFG20 is used by the UK government, hospitals, care homes, residential property developers and hundreds of private businesses to ensure that they meet legal requirements and maximise the health and safety of building occupants.

SFG20 content is typically used in Computer Aided Facilities Management solutions that actually instruct people to action the tasks at the appropriate time and then ensure that their completion is tracked.

The problem

I have thousands of maintenance schedules that contain tens of thousands of actions to take. These actions are authored by engineers and are ratified to ensure compliance with the law and best practice. The actions are written in free text and comprise a title and a list of steps to perform.

Many CAFM vendors want to allow their users to modify the frequency of some actions or to disable them because they are not relevant to a specific piece of equipment.

Recently a few schedules were rewritten for clarity which worked fine on paper, but meant task steps were grouped into time periods - this messed up the ability to identify specific sub tasks.

I need to make a definitive list of schedules where task titles are now solely time periods and find if there is a previous version where they were not organised that way.

The solution

First lets break down my task into steps.

For each maintenance schedule collect the task titles.
Identify if one or more of the task titles relate only to a time period: "6 monthly checks" is a problem, but "engine oil three monthly check" is not.
If there was a problem, loop over previous versions running steps 1 and 2 until we find a version without a problem.
Output a list of schedules where there is a problem and indicate if there is a previous version without an issue so that the Technical Authors can address the challenge.

Using AI

If we were to try to analyse the schedules by hand this would take forever.

If we were to look for tasks that NLP said had a time period: the list is still huge and would take weeks of work for the authors.

AI is good at this kind of task, acting like a person to read the title and identify if the task is solely a time period.

Building and debugging the solution

My approach to the prompt engineering for this task is to iterate.

First I ask ChatGPT to tell me what it would make as a prompt for my problem. Meta I know. This is the first debugging stage, if the prompt GPT comes up with seems to miss the point then I fix it here.
Next I construct the framework code and use some helper functions I've made to get the output I want, which for most of my cases is some kind of JSON.
The final iterative step is to run a bunch of test cases through the AI part and examine the result. If it doesn't do what I want then I give these as examples to GPT and ask it what prompt I should use to avoid these - back to step 1.

The solution

Here's the JS for the code that check a specific schedule. The resulting prompt here was based on two goes around my iteration cycle.

Running this resulted in 64 schedules to be looked at further, 6 of which had been changed in the previous 3 months - which are the ones we were receiving complaints about.

export async function checkScheduleTasks(schedule) {
    if (!schedule.tasks.filter((f) => !f.isHeader && !!f.title).length) return false
    const result = await chatJSON(
        [
            system(
                `
       You will be provided with a list of task names from a maintenance schedule. Your task is to examine the provided task names and determine if any of them represent only a time period, without being part of a more specific task description.

  Criteria for true:
  - A task name that consists solely of a time period description (e.g., "6 monthly", "Annual", "Bi-weekly").

  Criteria for false:
  - A task name that includes a time period as part of a broader task description (e.g., "Electrical tests - 6 monthly", "Annual safety inspection").

  Respond with {"response": true} if any task name meets the criteria for true. Otherwise, respond with {"response": false}.

    `
            ),
            jsonOutput(),
            `
       Here are the task names:

${schedule.tasks
    .filter((f) => !f.isHeader)
    .map((c) => c.title)
    .join("\n")}

    `,
        ],
        0.05
    )
    return result.response
}

Utilities

You can see from the above code I have utility functions to make system instructions and a default instruction for JSON. I'll list them here in case they are of some help.

import OpenAI from "openai"

export let openai
try {
    openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY })
} catch (e) {
    console.warn(`!!!!!! ERROR in OpenAI`)
    console.error(e)
}

export async function chat(messages, temperature = 0.15, model = "gpt-4-turbo-preview") {
    messages = messages.filter(Boolean).map((m) => (typeof m === "string" ? { role: "user", content: m } : m))
    let retry = false
    let response
    do {
        retry = false
        try {
            response = await openai.chat.completions.create({
                messages,
                model,
                temperature,
            })
        } catch (e) {
            if (e.status === 429) {
                await new Promise((resolve) => setTimeout(resolve, 250))
                retry = true
            }
        }
    } while (retry)
    return response.choices[0].message.content?.value ?? response.choices[0].message.content
}

export async function chatJSON(...params) {
    const text = await chat(...params)
    try {
        return JSON.parse(text)
    } catch (e) {
        console.log(text)
        console.error(e)
    }
}

export async function chatStream(messages, temperature = 0.15, model = "gpt-4-turbo-preview") {
    messages = messages.filter(Boolean).map((m) => (typeof m === "string" ? { role: "user", content: m } : m))
    return openai.chat.completions.create({
        messages,
        model,
        temperature,
        stream: true,
    })
}

export function cleanLines(message) {
    if (Array.isArray(message)) return message.map(cleanLines)
    return message
        .split("\n")
        .map((s) => s.trim())
        .join("\n")
}

export function system(message) {
    return {
        role: "system",
        content: cleanLines(message),
    }
}

export function jsonOutput() {
    return system(`
            Your output will be JSON, not wrapped in \`\`\`json.  Validate the JSON parses correctly before returning, adjust as necessary
        `)
}

An espresso shot of "Smooth": requestIdleCallback in <256 characters

Mike Talbot ⭐ — Fri, 22 Mar 2024 12:20:02 +0000

This is a submission for DEV Challenge v24.03.20, One Byte Explainer: Browser API or Feature.

Explainer

Keep your UI silky smooth by running code in the spare time at the end of each animation frame. requestIdleCallback allows you to provide a function to run in the frame gaps. Your function can query the amount of time left for the next step of your logic.

Avoid the Promise.all pitfall! Rate limit async function calls

Mike Talbot ⭐ — Thu, 12 Oct 2023 14:31:25 +0000

Have you fallen into the Promise.all pitfall? You know the one; you grab a list of things from somewhere and run a parallel function against all of them:



     const list = await getMeSomeList()
     const results = await Promise.all(list.map(someAsyncFunction))

Works a treat when the list has a few things in it, but lets say there are suddenly 10,000 records returned - this could really get messy.

You are really trying to spin too many plates, and memory or resources are going to become tight...

The Solution

Well you could just install the async package which has lots of useful functions like mapLimit which will reduce the burden and only run a number in parallel.

If that's overkill - then you can achieve a similar result using a simple rate limiter:



class Semaphore {
    constructor(maxConcurrency) {
        this.maxConcurrency = maxConcurrency
        this.currentConcurrency = 0
        this.queue = []
    }

    async acquire() {
        return new Promise((resolve) => {
            if (this.currentConcurrency < this.maxConcurrency) {
                this.currentConcurrency++
                resolve()
            } else {
                this.queue.push(resolve)
            }
        })
    }

    release() {
        if (this.queue.length > 0) {
            const resolve = this.queue.shift()
            resolve()
        } else {
            this.currentConcurrency--
        }
    }
}

export function rateLimit(asyncFunction, rate) {
    const semaphore = new Semaphore(rate)

    return async function process(...args) {
        await semaphore.acquire()
        try {
            return await asyncFunction(...args)
        } finally {
            semaphore.release()
        }
    }
}

With that in hand your code would change to be just this:



     const list = await getMeSomeList()
     const results = await Promise.all(list.map(rateLimit(someAsyncFunction, 20))

This would mean that it would keep 20 running at a time until the list way finished. Every time one of the someAsyncFunctions returns another one is started until the list is exhausted. Easy right :)

Delight your users with this instant scroll restoration custom hook

Mike Talbot ⭐ — Wed, 04 Oct 2023 12:59:00 +0000

TL;DR

In a multipage application, pressing the back button should restore the scrolling position of the previous page so that the user doesn't lose context.
The scroll position shouldn't glitch at all, it should be as if they never left.
The custom hook at the end of this article allows this in just a few lines of code.
The rest of this article describes how it works.
Thanks to some of the feedback this now works in browsers with privacy turned on

The Problem

There is a UX problem in some multipage React apps. A user scrolls down a long page and finds a link to click which opens a new page; then they press the back button on the new page and find themselves back at the top of the list, struggling to find their previous position manually.

The Solution

The solution is conceptually simple, store where the container was scrolled and restore that position when the previous page is reloaded.

The devil is in the details though. When is it time to restore the position? Loading the old page may not immediately fill the container with all of the content - perhaps it takes a few cycles for it to be built or perhaps a server call is required.

useScrollRestoration hook

To solve this problem I've built a custom hook that is easy to attach to a scrollable container and can even be used to remember multiple scroll positions for multiple lists.

The Concept

The concept of the useScrollRestoration hook is to provide a function that can be attached to the ref of the scrollable container and use this function to:

Listen for scroll events and store the scrollTop and scrollLeft positions
Attach a ResizeObserver and immediately restore the scroll positions when the content is large enough to enable the values to be restored.

This second point is the most important, unlike setTimeout or useLayoutEffect approaches, there is no glitch if the content isn't ready.

Building the Hook

Here's a walkthrough of how the hook works.

Storage

Firstly we need a place to store the scroll positions - these need to be persisted in sessionStorage in case we aren't using client-side routing and need to restore it on a page refresh. We also want to use a search parameter on the URL if the sessionStorage is not available due to privacy settings.

const KEY = "useScrollRestoration-store";

let scrolls;
let storageAvailable = true;
try {
  scrolls = JSON.parse(sessionStorage.getItem(KEY) ?? "{}");
} catch (e) {
  const params = new URLSearchParams(
    new URL(window.location.href).searchParams
  );
  storageAvailable = false;
  scrolls = JSON.parse(params.get("__scrollInfo") ?? "{}");
}

The `ref` function

Next we need to create the actual restoration function. I'm going to give it a key parameter for which scroll to store and restore - this will default to the current url (cleaned up to remove any scroll information) - and an optional timeout to abort restoring the scroll position if it's taking too long, which defaults to 1/2 a second.

export function useScrollRestoration(
  key = window.location.href,
  timeout = 500
) {
   // ... implementation
}

The next thing to do is to return something we can put as the ref of the scrolling container. In this case I'm going to supply a function that will be called by React when the element is mounted. We also have a useEffect, this needs to return a function to disconnect handlers on unmount, however, it also needs to reapply the connection to the scrollable element in debug and strict mode as React will call it twice and it will only attach the ref once.

export function useScrollRestoration(
  key = window.location.href,
  timeout = 500
) {
  // ... other variables
  const connectRef = useCallback(connect, [key, timeout]);
  const tracked = useRef();
  useEffect(() => {
    if (tracked.current) {
      connectRef(tracked.current);
    }
    return disconnect;
  }, [connectRef]);
  return connectRef;

   // ... functions
}

Connecting to a scrollable element

Ok so now we are into the meat of it. The connect function is responsible for adding the scroll event listeners, storing the scroll position AND restoring the scroll position when the content is big enough to accommodate it.

function connect(ref) {
    tracked.current = ref;
    disconnect();
    if (ref) {
      ref.addEventListener("scroll", store);
      handler.current = () => ref.removeEventListener("scroll", store);
      const scrollInfo = scrolls[key];
      if (scrollInfo) {
        ref.scrollTop = scrollInfo.top;
        ref.scrollLeft = scrollInfo.left;

        const resizeObserver = new ResizeObserver(() => {
          if (
            ref.scrollHeight > scrollInfo.top ||
            ref.scrollWidth > scrollInfo.left
          ) {
            ref.scrollTop = scrollInfo.top;
            ref.scrollLeft = scrollInfo.left;
            cleanUp.current();
          }
        });
        setTimeout(() => cleanUp.current(), timeout);

        resizeObserver.observe(ref);
        cleanUp.current = () => {
          resizeObserver.unobserve(ref);
          cleanUp.current = noop;
        };
      }
    }

    function store() {
      scrolls[key] = {
        top: ref.scrollTop,
        left: ref.scrollLeft
      };
      clearTimeout(updateTimer.current);
      updateTimer.current = setTimeout(() => {
        if (storageAvailable) {
          sessionStorage.setItem(KEY, JSON.stringify(scrolls));
        } else {
          const url = new URL(window.location.href);
          const params = new URLSearchParams(url.searchParams);
          params.set("__scrollInfo", JSON.stringify(scrolls));
          url.search = params.toString();
          window.history.replaceState(null, null, url.toString());
        }
      }, 50);
    }
  }

Firstly we disconnect any formerly connected events etc, then we check if this is React setting the reference (ref is truthy), and if it is we:

Attach a scroll listener
Set a detach function for the scroll listener into a ref called handler
Get scroll info for the current key if it exists

If we have scroll info then we are going to try to restore the scroll position. We do this by immediately setting it, in case the element is already full, and by waiting for the container to become the right size, using a ResizeObserver - it's this which really means that it scrolls without glitching.

We also add a cleanUp function that detaches the observer and we call this when we successfully get the right size, or after the timeout period.

Finally, in the scroll handler, we store the scroll position as the user scrolls the container element and then we write it to sessionStorage or add it to the URL after a 50ms debounce.

Disconnecting the handlers

Disconnection is easy, we call our ref stored clean-up functions.

  function disconnect() {
    handler.current();
    cleanUp.current();
  }

The Whole Thing

import { useCallback, useEffect, useRef } from "react";

const KEY = "useScrollRestoration-store";

let scrolls;
let storageAvailable = true;
try {
  scrolls = JSON.parse(sessionStorage.getItem(KEY) ?? "{}");
} catch (e) {
  const params = new URLSearchParams(
    new URL(window.location.href).searchParams
  );
  storageAvailable = false;
  scrolls = JSON.parse(params.get("__scrollInfo") ?? "{}");
}

/**
 * Custom hook for restoring scroll position based on a unique key. The hook returns a callback function
 * that should be set on the JSX element's ref attribute to manage scroll restoration.
 *
 * @param {string} [key=window.location.href] - A unique key to identify the scroll position, defaults to current URL.
 * @param {number} [timeout=500] - A timeout after which the scroll will not be restored, defaults to 1/2 a second.
 * @returns {Function} A callback function to set as the `ref` on a scrollable JSX element.
 *
 * @example
 * const scrollRef = useScrollRestoration();
 * return <div ref={scrollRef}>Your Content Here</div>;
 */

export function useScrollRestoration(
  key = window.location.href,
  timeout = 1500
) {
  key = removeScrollParameter(key);

  const updateTimer = useRef(0);
  const handler = useRef(noop);
  const cleanUp = useRef(noop);
  const connectRef = useCallback(connect, [key, timeout]);
  const tracked = useRef();
  useEffect(() => {
    if (tracked.current) {
      connectRef(tracked.current);
    }
    return disconnect;
  }, [connectRef]);
  return connectRef;

  function connect(ref) {
    disconnect();
    tracked.current = ref;
    if (ref) {
      ref.addEventListener("scroll", store);
      handler.current = () => {
        ref.removeEventListener("scroll", store);
      };

      const scrollInfo = scrolls[key];
      if (scrollInfo) {
        ref.scrollTop = scrollInfo.top;
        ref.scrollLeft = scrollInfo.left;
        const resizeObserver = new ResizeObserver(() => {
          if (
            ref.scrollHeight > scrollInfo.top ||
            ref.scrollWidth > scrollInfo.left
          ) {
            ref.scrollTop = scrollInfo.top;
            ref.scrollLeft = scrollInfo.left;
            cleanUp.current();
          }
        });
        setTimeout(() => cleanUp.current(), timeout);

        resizeObserver.observe(ref);
        cleanUp.current = () => {
          resizeObserver.unobserve(ref);
          cleanUp.current = noop;
        };
      }
    }

    function store() {
      scrolls[key] = {
        top: ref.scrollTop,
        left: ref.scrollLeft
      };
      clearTimeout(updateTimer.current);
      updateTimer.current = setTimeout(() => {
        if (storageAvailable) {
          sessionStorage.setItem(KEY, JSON.stringify(scrolls));
        } else {
          const url = new URL(window.location.href);
          const params = new URLSearchParams(url.searchParams);
          params.set("__scrollInfo", JSON.stringify(scrolls));
          url.search = params.toString();
          window.history.replaceState(null, null, url.toString());
        }
      }, 50);
    }
  }

  function disconnect() {
    handler.current();
    cleanUp.current();
  }
}

/**
 * Do nothing
 */
function noop() {}

/**
 * Remove the scroll info from the URL
 */
function removeScrollParameter(href) {
  href = href.replace(/__scrollInfo=[^&]+/gi, "");
  if (href.endsWith("/?")) return href.slice(0, -2);
  if (href.endsWith("/")) return href.slice(0, -1);
  return href;
}

Demo

Conclusion

I hope you've found this an interesting exploration into how we can use the features of refs as functions to create dynamic patterns for DOM element events and interactions. Feel free to use the hook wherever you like.

Game In A Month: Serverless Leaderboards

Mike Talbot ⭐ — Sat, 20 Nov 2021 09:21:52 +0000

I found myself out of work for the last month and decided to use my time with my partner to build a game. We've participated in Game Jams before but never really got anything to a production state. This time we wanted it to be different.

We decided to build the game in Unity and used some really nice Synty assets for the world and Malbers animations for our key rabbit characters along side some custom assets and a whole lot of level design :)

We needed three key things that fall outside of the Unity stack:

A website to host a free preview version of the game (https://wabbitsworld.com)
A service on that site that can share photos to Facebook that are uploaded from the game, even if from a mobile app etc
A highscore table that ran in seasons and could return the top 100 scores and the position in the total leaderboard of the current player.

Leaderboards

Leaderboards are a non-trivial problem - even if you have a server with a database you are having to do sorts on large numbers of records - albeit that indexes can help a lot with this, it's still quite a load. To find the relative position of a player in a million scores you need to traverse the sorted list. If you decide, like we did, that you don't want to go to the cost of running a server and opt for serverless (in our case Firebase) then your problem intensifies. It would be very expensive indeed to use one of the Firebase databases to try to run a leaderboard due to the pricing model and you can't benefit from in memory caching in Serverless architectures.

The ideal way to run a leaderboard is to use ZSets in Redis. Redis is fantastic at these kinds of operations and so I decided on implementing the following stack:

Run the website as a Cloud Function in Firebase - this way I can implement an Express app to record scores and download the current top scores. I use Pug to create sharing pages for a user's images with the correct Open Graph tags so Facebook posts link through properly and show the image.
Use Upstash as a serverless Redis implementation - it has a generous free tier and the price won't get out of hand even if the game is very successful
Use my cloud based Express app to query Redis for scores and to record new ones.
Create a React app for the site and host that in the same Express Cloud function

I also decided that I would do 14 day seasons so the leaderboard is for currently active players - not those who played months ago. This is easy with Redis - I just add the current date / 14 * 1000 * 60 * 60 * 24 rounded to an int to the key used for the highscores.

The Code

I'm going to start by showing you the entire code for the website (excluding the pug view). I'm doing this because I can't quite believe how tiny it is!

const functions = require("firebase-functions");
const express = require("express");
const path = require("path");
const bodyParser = require('body-parser');
const app = express();
app.use(require('compression')());
app.use(bodyParser.urlencoded({extended: true}));
app.use(bodyParser.json());
app.set('views', path.join(__dirname, 'views'));
app.set('view engine', 'pug');

// Facebook share page
app.get("/shared", (req,res)=>{
    res.render("shared", {image: req.query.image, token: req.query.token});
});

const season = Math.floor(Date.now()/ (1000 * 60 * 60 * 24 * 14) );
const HIGHSCORES = `highscores ${season}`;

const REDIS_PASSWORD="REDIS_PASSWORD_HERE";
const REDIS_HEADER= "Bearer MY BEARER TOKEN=";
const REDIS_BASEURL= "https://MY_SERVER.upstash.io/";
const Redis = require("ioredis");


function createRedisConnection(){
    return new Redis("redis://UPSTASH ADDRESS AND PASSWORD");
}

// Heartbeat api
app.get('/info', (req,res)=>{
    res.render("info");
});

//API to record a score
app.post("/addscorerecord", async ({body: {id, name, score}}, response)=>{
    const redis = createRedisConnection();
    await redis.zadd(HIGHSCORES, -score, id);
    await redis.set(id, JSON.stringify({id, name, score}));
    await redis.set(`${id}_name`, name);
    let rank = await redis.zrank(HIGHSCORES, id);
    if(rank === undefined || rank === null) rank = -1;
    redis.disconnect();
    response.send({rank, time: Date.now()});
});

function groupResults(results)
{
    const output = []
    for(let i = 0; i < results.length; i+=2)
    {
        output.push([results[i], results[i+1]]);
    }
    return output;
}

// API to get the Highscore table
app.post("/gethighscoretable", async ({body: {id}}, response) =>{
    const redis = createRedisConnection();
    let rank = await redis.zrank(HIGHSCORES, id);
    if(rank === null || rank === undefined) rank = -1;
    const topScores = await redis.zrange(HIGHSCORES, 0, 99, "withscores");
    const scores = []
    if(topScores && topScores.length) {
        const pipe = redis.pipeline();
        let groupedResults = groupResults(topScores)
        for (const [id, score] of groupedResults) {
            pipe.get(`${id}_name`);
        }
        const names = await pipe.exec();
        for (let i = 0; i < groupedResults.length; i++) {
            const [, score] = groupedResults[i];
            scores.push({score: -score, name: names[i][1]});
        }
    }
    redis.disconnect();
    response.send({rank, scores, time: Date.now()});
});

// API to get the server time
app.get("/time", (req,res)=>{
    res.send({time: Date.now()})
});

// This serves the Unity game
app.use(express.static(path.join(__dirname, "public")));

// Return all other paths to the index.html for React routing
app.use((req,res)=>{
   res.sendFile(path.join(__dirname, "public", "index.html"), err=>{
       res.status(500).send(err);
   });
});

exports.app = functions.https.onRequest(app);

Recording a score

The process of recording a score is pretty simple. The game provides a score, an id for the player and the name that they want displayed for their score.

The id and the score are placed in a ZSet with the score negated so that higher scores come first.

app.post("/addscorerecord", async ({body: {id, name, score}}, response)=>{
    const redis = createRedisConnection();
    await redis.zadd(HIGHSCORES, -score, id);

Next I record the name for the ID so we can look it up quickly and a whole record of the current score and name for the player - this latter is unnecessary in the current code, but I have a plan for it later.

    await redis.set(id, JSON.stringify({id, name, score}));
    await redis.set(`${id}_name`, name);

Finally we use Redis magic to quickly work out the current rank of the player.

    let rank = await redis.zrank(HIGHSCORES, id);
    if(rank === undefined || rank === null) rank = -1;

We finally package up the response and send it to Unity as a JSON packet.

    redis.disconnect();
    response.send({rank, time: Date.now()});
});

Getting the highscore table

It's not much harder to retrieve the highscore table - we get the top 100 scores and repeat the current player ranking operation. For this to work we just need the id of the player.

app.post("/gethighscoretable", async ({body: {id}}, response) =>{
    const redis = createRedisConnection();
    let rank = await redis.zrank(HIGHSCORES, id);
    if(rank === null || rank === undefined) rank = -1;

Next we request the top 100 scores including both the score and the id:

    const topScores = await redis.zrange(HIGHSCORES, 0, 99, "withscores");

The we need to turn ids into names.

    const scores = []
    if(topScores && topScores.length) {
        const pipe = redis.pipeline();
        let groupedResults = groupResults(topScores)
        for (const [id, score] of groupedResults) {
            pipe.get(`${id}_name`);
        }
        const names = await pipe.exec();
        for (let i = 0; i < groupedResults.length; i++) {
            const [, score] = groupedResults[i];
            scores.push({score: -score, name: names[i][1]});
        }
    }

You can see I use a pipeline operation in Redis to make the call for 100 things all at once for performance reasons.

Next we just need to return the data:

    redis.disconnect();
    response.send({rank, scores, time: Date.now()});
});

Calling From Unity

Unity makes it pretty easy to call these functions and use the results. I implemented an HTTP helper first, this allows HTTP requests as Unity coroutines:

namespace Wabbit
{
    public static class HttpHelp
    {
        public static IEnumerator GetJson<T>(string url, Action<T> response) where T: new()
        {
            var request = new UnityWebRequest(url, "GET");
            yield return request.SendWebRequest();
            while (!request.isDone)
            {
                yield return null;
            }

            if (request.result == UnityWebRequest.Result.Success)
            {
                var o = new T();
                var item = JsonUtility.FromJson<T>(request.downloadHandler.text);
                response(item);
            }
        }

        public static IEnumerator PostJson(string url, object data, Action<string> response = null)
        {
            var request = new UnityWebRequest(url, "POST");
            var body = Encoding.UTF8.GetBytes(JsonUtility.ToJson(data));
            request.uploadHandler = new UploadHandlerRaw(body);
            request.downloadHandler = new DownloadHandlerBuffer();
            request.SetRequestHeader("Content-Type", "application/json");
            yield return request.SendWebRequest();
            while (!request.isDone)
            {
                yield return null;
            }

            if (response != null && request.result == UnityWebRequest.Result.Success)
            {
                response(request.downloadHandler.text);
            }
        }
    }
}

Recording a score and retrieving scores use this helper function, but we have to define classes that will be translated to and from JSON, so they come first:

        [Serializable]
        public class ScoreRecord
        {
            public string id;
            public string name;
            public int score;
        }

        [Serializable]
        public class Ranking
        {
            public int rank;
        }

        [Serializable]
        public class ScoreEntry
        {
            public string name;
            public int score;
        }

        [Serializable]
        public class HighScoreTable
        {
            public int time;
            public int rank = -2;
            public ScoreEntry[] scores;
        }

Now recording a score is just a matter of using the helper with the correct class as a parameter:

        private static IEnumerator SendScore()
        {
            yield return HttpHelp.PostJson("https://wabbitsworld.com/addscorerecord", new ScoreRecord
            {
                id = Controls.PlayerInfo.id, name = Controls.PlayerInfo.userName, score = Controls.PlayerInfo.highScore
            }, result =>
            {
                var ranking = JsonUtility.FromJson<Ranking>(result);
                currentRank = ranking.rank;
                Events.Raise("GotRank");
            });
        }

Conclusion

I found it was pretty easy to setup a free tiered serverless environment that combines Firebase with Upstash to allow a simple leaderboard system to be developed. While this example doesn't cover some of the extensions you would add to avoid cheating, it shows a cheap and performant way to make simple highscore functionality.

You can download the iOS and Mac versions of Wabbits from the App Store. The Droid version is awaiting approval.

An extensible React "Plug & Play" Widget

Mike Talbot ⭐ — Sun, 26 Sep 2021 16:35:35 +0000

TLDR;

I'm building a blogging widget that allows authors to further engage their audience by creating interactive and gamified experiences right within their post. This article is part of a series that looks at how this is done.

In this article I'll look at how the Widget allows extension functionality to be created by authors so that they can add their own interactive, configurable extensions and we can build a library of useful tools anyone can use! The extension functionality works without having to access the core project and can be easily developed and deployed using any framework that can output Javascript and interact with the DOM.

Motivation

I'm building the interactive widget below, vote on what you'd like to interact with or embed in your own post:

Requirements

The key principle here is to create an API that can be used to easily add an extension to the widget so that an author can create powerful new functionality to "plug in". I don't want to enforce a technology stack choice on the developer so they should be able to write in anything from vanilla Javascript to a fully fledged framework.

The developer needs to build two things, an editor component that will allow a post author to configure the extension widget and a runtime that will be rendered inside the post and perform whatever actions are required.

The key features need to be:

Create and expose an API that allows a developer to register an extension for both editor and runtime concerns
Expose an API that allows a plugin developer to record information relevant to the reader, the article and the widget (for example a vote in a poll)
Provide a way of notifying the plugin developer about existing responses related to the article and changes to the data
Provide an API to allow the plugin developer to award points and badges to the reader
Provide a way for the plugin developer to have their extension code loaded when the plugin is to be used

Configuration Interface

I've built a configuration interface for the main Widget that allows the injection of the custom editor instances and saves all of the necessary data. To configure a widget the user works with a number of screens:

The homepage gives an author access to their profile, their articles and their comments. Each article or comment has a configuration for the widget.

The author makes an entry for each post and can use the summary view to see how many times the content has been viewed (including unique user views) and the number of times it has been interacted with.

The author can configure the main and footer widgets for their embed. They choose an available widget from a drop down list and its editor is displayed in line (here the example is for the simple HTML plugin).

If the widget is a custom built one then they can specify the files it should load on the "Advanced" tab. The entries here are for all of the Javascript files to load - while developing these could be hosted on a local development server or it could be hosted on GitHub or anywhere else, so long as the files are served as Javascript and not text. Many build systems output more than one file for inclusion in the core package (for instance a vendors file and a main source bundle) and they can all be listed here or the urls included in a .bundle file that is then used here.

Runtime Script Loading

Ok so to start with the system needs to load the extension code specified in the "Advanced" tab. It does this by splitting the list of files on \n and then checking if the file is one of three types (+ 1 filter):

A .editor.* file - which will only be loaded if the widget is in the configuration system
A .js file - in which case a <script/> tag is created and the src set to be the file. This means the file must be served with the correct mime type (which GitHub raw files are not, unless you use a CDN extension which will cache the file, making it unwise during development).
A .jsx or a .babel.js file - in which case browser babel is loaded and then an additional <script/> tag with a type of text/babel is created with the src attribute set to the file and an environment of env and react added to it. This allows lightweight React plugins as React is used to build the outer layer. It's a big fancy, and I won't go into too much more detail here apart from to say, that if one .jsx file imports another then it also needs to be specified here. Note that GitHub raw files are fine in this case.
A .bundle file - in which case the file is downloaded and the same process is applied to the contents of the file.

It is expected that plugins will be developed as bundled projects if using a framework and the output Javascript included. I've tested it with Webpack and Rollup, you just need to be sure to include all of the files that would have been included in the index.html.

Implementation


export async function loadPlugins(plugins) {
    let hadBabel = false
    for (let url of plugins) {
        let type = "text/javascript"
        if (url.endsWith(".bundle")) {
            const response = await fetch(url)
            if (!response.ok) {
                console.warn("Could not load bundle", url)
                continue
            }
            const usedBabel = await loadPlugins(
                (
                    await response.text()
                )
                    .split("\n")
                    .map((c) => c.trim())
                    .filter((c) => !!c)
            )
            hadBabel = hadBabel || usedBabel
            continue
        }
        if (document.body.querySelector(`script[src~="${url}"]`)) continue
        const script = document.createElement("script")
        if (url.includes(".babel") || url.includes(".jsx")) {
            hadBabel = true
            type = "text/babel"
            script.setAttribute("data-presets", "env,react")
            script.setAttribute("data-plugins", "transform-modules-umd")
            await loadBabel()
        }
        script.type = type
        script.src = `${url}`
        document.body.appendChild(script)
    }
    return hadBabel
}


function loadBabel() {
    return new Promise((resolve) => {
        const babelUrl = "https://unpkg.com/@babel/standalone/babel.min.js"
        if (document.body.querySelector(`script[src='${babelUrl}']`)) {
            return resolve()
        }
        const script = document.createElement("script")
        script.src = babelUrl
        script.onload = () => {
            resolve()
        }
        document.body.appendChild(script)
    })
}

I also wrote a custom hook to load the plugins and ensure the babel is transpiled:

import { useEffect } from "react"

export function usePlugins(definition, deps = []) {
    useEffect(() => {
        if (!definition) return
        setTimeout(async () => {
            const plugins = definition
                .split("\n")
                .map((c) => c.trim())
                .filter((c) => !!c)
            let hadBabel = false
            for (let url of plugins) {
                let type
                if (url.includes(".editor")) continue
                if (document.body.querySelector(`script[src~="${url}"]`))
                    continue
                if (url.includes(".babel") || url.includes(".jsx")) {
                    hadBabel = true
                    type = "text/babel"
                    await loadBabel()
                }
                const script = document.createElement("script")
                script.type = type
                script.src = `${url}?${Date.now()}`
                script.setAttribute("data-presets", "env,react")
                document.body.appendChild(script)
            }
            if (hadBabel) {
                window.dispatchEvent(new Event("DOMContentLoaded"))
            }
        })
        //eslint-disable-next-line react-hooks/exhaustive-deps
    }, [deps])
}

Registering New Plugins

Loading the code is one thing, but once loaded it needs to be able to interact with the outer widget. To accomplish this the outer widget exposes an API on window in a variable called Framework4C. This API provides all of the core functions required by a plugin.


window.Framework4C = {
    Accessibility: {
        reduceMotion  //User prefers reduced motion
    },
    Material,   // The whole of Material UI core
    showNotification,  // A function to show a toast
    theme,  // A material UI theme
    React,  // React 17
    ReactDOM, // ReactDOM 17
    Plugins: { 
        register, 
        PluginTypes, 
    }, // Function to register plugins
    Interaction: {
        awardPoints,
        respond,
        respondUnique,
        addAchievement,
    } // Response functions
}

To get involved in the process the only thing that the newly loaded code needs to do is to call register passing a valid PluginTypes value and a function that will render the editor or the runtime within a specified parent DOM element.

Registering A Plugin

Each plugin comprises an editor and a runtime.

The Editor

An editor is provided with a place to store configuration data and a function to call to say that the data has been changed. It is the job of the editor to set up any parameters that the runtime will need - these are all entirely at the discretion of the developer.

const {
   Plugins: { PluginTypes, register },
} = window.Framework4C

register(PluginTypes.MAIN, "Remote", editor, null /* Ignore Runtime */)

function editor({ parent, settings, onChange }) {
    /* Render the editor underneath parent */
}

If you were going to use React to render the editor you'd use ReactDOM.render passing the parent element. If you were using Vue you'd createApp and mount it inside the parent:

import { createApp } from "vue"
import App from "./App.vue"
import Render from "./Render.vue"

const {
    Plugins: { register, PluginTypes }
} = window.Framework4C || { Plugins: {} }


register(PluginTypes.MAIN, "Vue Example", editor)

function editor({ parent, settings, onChange }) {
    createApp({
        ...App,
        data() {
            // Initialize props for reactivity
            settings.message = settings.message || ""
            return settings
        },
        updated() {
            onChange()
        }
    }).mount(parent)
}

To register an editor we simply call the register function, specifying the type of plugin and pass a callback for when it is time to render the plugin's editor.

Poll Editor UI

Here's an example of the UI for the editor that made the poll on the article.

Poll Editor Code

import {
    Box,
    Button,
    ButtonGroup,
    CardContent,
    CssBaseline,
    DialogActions,
    DialogContent,
    DialogTitle,
    IconButton,
    TextField,
    ThemeProvider,
    Typography
} from "@material-ui/core"
import { nanoid } from "nanoid"
import randomColor from "randomcolor"
import React, { useState } from "react"
import reactDom from "react-dom"
import { FaEllipsisV } from "react-icons/fa"
import { MdDelete } from "react-icons/md"
import { Bound, useBoundContext } from "../lib/Bound"
import { BoundTextField } from "../lib/bound-components"
import { BoundColorField } from "../lib/ColorField"
import { downloadObject } from "../lib/downloadObject"
import { ListItemBox } from "../lib/ListItemBox"
import { Odometer } from "../lib/odometer"
import { PluginTypes, register } from "../lib/plugins"
import { setFromEvent } from "../lib/setFromEvent"
import { Sortable, SortableItem } from "../lib/Sortable"
import { theme } from "../lib/theme"
import { UploadButton } from "../lib/uploadButton"
import { useDialog } from "../lib/useDialog"
import { useEvent } from "../lib/useEvent"
import { useRefresh } from "../lib/useRefresh"

register(PluginTypes.MAIN, "Poll", editor)

function editor({ parent, ...props }) {
    reactDom.render(<Editor {...props} />, parent)
}

function Editor({ settings, onChange, response }) {
    const refresh = useRefresh(onChange)
    return (
        <ThemeProvider theme={theme}>
            <CssBaseline />
            <Bound
                refresh={refresh}
                target={settings}
                onChange={onChange}
                response={response}
            >
                <Box mt={2}>
                    <PollConfig />
                </Box>
            </Bound>
        </ThemeProvider>
    )
}

function PollConfig() {
    const { target, refresh } = useBoundContext()
    const answers = (target.answers = target.answers || [])
    const getName = useDialog(DownloadName)
    return (
        <>
            <ListItemBox>
                <Box flex={1} />
                <ButtonGroup size="small">
                    <UploadButton
                        accept="*.poll.json"
                        variant="outlined"
                        color="primary"
                        onFile={load}
                    >
                        Load
                    </UploadButton>
                    <Button onClick={save} variant="outlined" color="secondary">
                        Save
                    </Button>
                </ButtonGroup>
            </ListItemBox>
            <CardContent>
                <BoundTextField field="question" />
            </CardContent>
            <CardContent>
                <BoundTextField field="description" />
            </CardContent>
            <CardContent>
                <BoundColorField field="questionColor" default="white" />
            </CardContent>
            <CardContent>
                <Typography variant="overline" component="h3" gutterBottom>
                    Answers
                </Typography>
                <Sortable items={answers} onDragEnd={refresh}>
                    {answers.map((answer) => (
                        <Answer
                            answers={answers}
                            key={answer.id}
                            answer={answer}
                        />
                    ))}
                </Sortable>
            </CardContent>
            <Button color="primary" onClick={addAnswer}>
                + Answer
            </Button>
        </>
    )
    async function save() {
        const name = await getName()
        if (name) {
            downloadObject(target, `${name}.poll.json`)
        }
    }

    function load(data) {
        if (data) {
            Object.assign(target, data)
            refresh()
        }
    }

    function addAnswer() {
        answers.push({ id: nanoid(), answer: "", color: randomColor() })
        refresh()
    }
}

export function DownloadName({ ok, cancel }) {
    const [name, setName] = useState("")
    return (
        <>
            <DialogTitle>Name</DialogTitle>
            <DialogContent>
                <TextField
                    autoFocus
                    value={name}
                    onChange={setFromEvent(setName)}
                    fullWidth
                />
            </DialogContent>
            <DialogActions>
                <Button onClick={cancel}>Cancel</Button>
                <Button
                    onClick={() => ok(name)}
                    color="secondary"
                    variant="contained"
                >
                    Create
                </Button>
            </DialogActions>
        </>
    )
}

export function Answer({ answers, answer }) {
    const { refresh, response } = useBoundContext()
    const [dragProps, setDragProps] = useState({})
    useEvent("response", useRefresh())

    const votes = Object.values(response?.responses?.Poll || {}).reduce(
        (c, a) => (a === answer.id ? c + 1 : c),
        0
    )
    return (
        <SortableItem
            borderRadius={4}
            bgcolor="#fff8"
            setDragProps={setDragProps}
            m={1}
            display="flex"
            alignItems="center"
            id={answer.id}
        >
            <Bound target={answer} refresh={refresh}>
                <Box
                    aria-label="Drag handle"
                    mr={1}
                    color="#444"
                    fontSize={16}
                    {...dragProps}
                >
                    <FaEllipsisV />
                </Box>
                <Box flex={0.6} mr={1}>
                    <BoundTextField
                        field="answer"
                        InputProps={{
                            endAdornment: (
                                <Box
                                    ml={1}
                                    textAlign="right"
                                    color="#666"
                                    whiteSpace="nowrap"
                                >
                                    <small>
                                        <Odometer>{votes}</Odometer> vote
                                        <span
                                            style={{
                                                opacity: votes === 1 ? 0 : 1
                                            }}
                                        >
                                            s
                                        </span>
                                    </small>
                                </Box>
                            )
                        }}
                    />
                </Box>
                <Box flex={0.4} mr={1}>
                    <BoundTextField field="legend" />
                </Box>
                <Box flex={0.5} mr={1}>
                    <BoundColorField field="color" default="#999999" />
                </Box>
                <IconButton
                    aria-label="Delete"
                    onClick={remove}
                    color="secondary"
                >
                    <MdDelete />
                </IconButton>
            </Bound>
        </SortableItem>
    )

    function remove() {
        const idx = answers.indexOf(answer)
        if (idx !== -1) {
            answers.splice(idx, 1)
            refresh()
        }
    }
}

The Runtime

The runtime is used to render the component when it is viewed by a reader, presumably it takes the configuration information provided by the author and uses that to create the desired user interface.

The runtime is also supplied a parent DOM element, but it is also supplied with the settings made in the editor, the article that is being viewed, the current user and a response object that contains all of the responses. This response object may be updated after the initial render and a window event of response is raised passing the updated data.

Implementation

As far as the framework is concerned, the register function just records the callback for the editor and the runtime in a data structure and raises a change event. These entries are looked up for rendering.

import { raise } from "./raise"

export const PluginTypes = {
    MAIN: "main",
    FOOTER: "footer",
    NOTIFICATION: "notification"
}

export const Plugins = {
    [PluginTypes.MAIN]: {},
    [PluginTypes.FOOTER]: {},
    [PluginTypes.NOTIFICATION]: {}
}

export function register(type, name, editor, runtime) {
    const existing = Plugins[type][name] || {}
    Plugins[type][name] = {
        name,
        editor: editor || existing.editor,
        type,
        runtime: runtime || existing.runtime
    }
    raise("plugins-updated")
}

Runtime Responses

The plugin system gives you the ability to capture responses from the user and store them. All of the responses for the current article are provided to you, so for instance you can show the results of a poll or a quiz. Using these methods you can record information and display it to the reader in the way you want.

The system also raises events on window when the response changes so you can show real-time updates as data changes due to any current readers.

The most common way to capture a users response is to use the API call respondUnique(articleId, type, response). This API call will record a response object unique to the current user. The type parameter is an arbitrary string you use to differentiate your plugins response from others. The response passed is an object or value that will be recorded for the user and then made available to all plugin instances for the current article.

A response object populated due to a call passing “MyResponseType” as the type might look like this.

{
   MyReponseType: {
       UserId1: 1 /* something you recorded */,
       UserId2: { answer: 2 } /* something you recorded for user 2 */     
   }
}

So to display summaries or totals for a poll or a quiz you would calculate them by iterating over the unique user responses and calculating the answer.

If you call respondUnique multiple times, only the last value will be recorded for the current user, this is normally what you want for a poll or a quiz.

await respondUnique(article.uid, "Poll", answer.id)

You may also call respond with the same parameters. In this case, the response structure will contain an array of all of the responses for each user.

{
   MyReponseType: {
       UserId1: [{ /* something you recorded */ }, {/* another thing */}],
       UserId2: [{ /* something you recorded for user 2 */ }]     
   }
}

Runtime Rendering

The runtime rendering of the whole widget relies on calling the registered functions. The Widget builds a container DOM structure and then calls a function called renderPlugin passing in the settings. I'll put the whole code for this in a foldaway so you can examine it if you like, we'll concentrate on renderPlugin.


function renderPlugin(
    parent,
    type,
    pluginName,
    settings = {},
    article,
    user,
    response,
    previewMode
) {
    if (!settings || !pluginName || !type || !parent || !article || !user)
        return
    const plugin = Plugins[type][pluginName]
    if (!plugin || !plugin.runtime) return
    plugin.runtime({
        parent,
        article,
        settings,
        type,
        pluginName,
        user,
        response,
        previewMode
    })
}

Rendering the plugin is simply a matter of looking up the plugin required in the registered list and then calling its runtime function. The outer holder handles monitoring Firestore for changes to the response information and raising the custom event should it happen.

renderWidget

import { addAchievement, db, view } from "../lib/firebase"
import logo from "../assets/4C_logo.jpg"
import { Plugins, PluginTypes } from "../lib/plugins"
import { raise } from "../lib/raise"
import { merge } from "../lib/merge"

let response = { notLoaded: true }
let lastMain
export async function renderWidget(
    parent,
    id,
    user = { isAnonymous: true },
    useArticle = null
) {
    const definitionRef = db.collection("articles").doc(id)

    const definitionDoc = (parent._definitionDoc =
        parent._definitionDoc || (await definitionRef.get()))
    if (!definitionDoc.exists && !useArticle) {
        // Do some fallback
        return null
    }

    if (parent._uid !== user.uid) {
        if (!useArticle) {
            view(id).catch(console.error)
        }
    }
    // Get the actual data of the document
    const article = useArticle || definitionDoc.data()
    if (lastMain !== article[PluginTypes.MAIN]) {
        article.overrideBottomBackground = null
        article.overrideGradientFrom = null
        article.overrideGradientTo = null
    }
    lastMain = article[PluginTypes.MAIN]
    const removeListener = (parent._removeListener =
        parent._removeListener ||
        db
            .collection("responses")
            .doc(id)
            .onSnapshot((update) => {
                response.notLoaded = false
                const updatedData = update.data()
                Object.assign(response, updatedData)
                setTimeout(() => {
                    response.notLoaded = false
                    raise(`response-${id}`, response)
                    raise(`response`, response)
                })
            }))

    parent._uid = user.uid
    const author = await (
        await db.collection("userprofiles").doc(article.author).get()
    ).data()
    const holder = makeContainer(parent, article, user)
    holder.logoWidget.style.backgroundImage = `url(${logo})`
    if (author?.photoURL) {
        holder.avatarWidget.style.backgroundImage = `url(${author.photoURL})`
    }
    if (author.profileURL) {
        holder.avatarWidget.role = "button"
        holder.avatarWidget.style.cursor = "pointer"
        holder.avatarWidget["aria-label"] = "Link to authors profile page"
        holder.avatarWidget.onclick = () => {
            if (author.displayName) {
                addAchievement(
                    15,
                    `Visited profile of ${author.displayName}`
                ).catch(console.error)
            }
            window.open(author.profileURL, "_blank", "noreferrer noopener")
        }
    }
    article.pluginSettings = article.pluginSettings || {}
    renderPlugin(
        holder.mainWidget,
        PluginTypes.MAIN,
        article[PluginTypes.MAIN],
        article.pluginSettings[article[PluginTypes.MAIN]] || {},
        article,
        user,
        response,
        !!useArticle
    )
    renderPlugin(
        holder.footerWidget,
        PluginTypes.FOOTER,
        article[PluginTypes.FOOTER],
        article.pluginSettings[article[PluginTypes.FOOTER]] || {},
        article,
        user,
        response,
        !!useArticle
    )
    renderPlugin(
        holder.notificationWidget,
        PluginTypes.NOTIFICATION,
        article[PluginTypes.NOTIFICATION] || "defaultNotification",
        article.pluginSettings[article[PluginTypes.NOTIFICATION]] || {},
        article,
        user,
        response,
        !!useArticle
    )
    return () => {
        parent._removeListener = null
        removeListener()
    }
}

function renderPlugin(
    parent,
    type,
    pluginName,
    settings = {},
    article,
    user,
    response,
    previewMode
) {
    if (!settings || !pluginName || !type || !parent || !article || !user)
        return
    const plugin = Plugins[type][pluginName]
    if (!plugin || !plugin.runtime) return
    plugin.runtime({
        parent,
        article,
        settings,
        type,
        pluginName,
        user,
        response,
        previewMode
    })
}

function makeContainer(parent, article) {
    const isNarrow = window.innerWidth < 500
    parent = parent || document.body
    parent.style.background = `linear-gradient(45deg, ${
        article?.overrideGradientFrom ?? article?.gradientFrom ?? "#fe6b8b"
    } 30%, ${
        article?.overrideGradientTo ?? article?.gradientTo ?? "#ff8e53"
    } 90%)`
    if (parent._madeContainer) {
        parent._madeContainer.bottom.style.background =
            article.overrideBottomBackground ||
            article.bottomBackground ||
            "#333"
        parent._madeContainer.bottom.style.color =
            article.overrideBottomColor || article.bottomColor || "#fff"

        parent._madeContainer.bottom.style.display = isNarrow ? "none" : "flex"
        parent._madeContainer.notificationWidget.style.display = isNarrow
            ? "none"
            : "flex"

        return parent._madeContainer
    }
    window.addEventListener("resize", () => makeContainer(parent, article))
    const main = document.createElement("main")
    Object.assign(main.style, {
        display: "flex",
        flexDirection: "column",
        width: "100%",
        height: "100%",
        overflow: "hidden"
    })
    const top = document.createElement("div")
    Object.assign(top.style, {
        flex: 1,
        width: "100%",
        display: "flex",
        justifyContent: "stretch",
        overflow: "hidden"
    })
    main.appendChild(top)
    const mainWidget = document.createElement("section")
    Object.assign(mainWidget.style, {
        width: "66%",
        flex: 1,
        overflowY: "auto",
        display: "flex",
        flexDirection: "column",
        alignItems: "stretch",
        justifyContent: "stretch",
        position: "relative"
    })
    top.appendChild(mainWidget)
    const notificationWidget = document.createElement("section")
    Object.assign(notificationWidget.style, {
        width: "34%",
        display: isNarrow ? "none" : "block",
        maxWidth: "250px",
        overflowY: "hidden",
        overflowX: "visible"
    })
    top.appendChild(notificationWidget)
    const middle = document.createElement("div")
    Object.assign(middle.style, {
        height: "0px"
    })
    main.appendChild(middle)
    const bottom = document.createElement("div")
    Object.assign(bottom.style, {
        height: "76px",
        background:
            article.overrideBottomBackground ||
            article.bottomBackground ||
            "#333",
        color: article.overrideBottomColor || article.bottomColor || "#fff",
        marginLeft: "-4px",
        marginRight: "-4px",
        marginBottom: "-4px",
        boxShadow: "0 0 8px 0px #000A",
        padding: "8px",
        paddingTop: "4px",
        display: isNarrow ? "none" : "flex",
        paddingRight: window.padRightToolbar ? "142px" : undefined,
        flexGrow: 0,
        flexShrink: 0,
        alignItems: "center",
        width: "calc(100% + 8px)",
        overflow: "hidden",
        position: "relative"
    })
    main.appendChild(bottom)
    const avatarWidget = document.createElement("div")
    merge(avatarWidget.style, {
        borderRadius: "100%",
        width: "64px",
        height: "64px",
        backgroundRepeat: "no-repeat",
        backgroundSize: "cover"
    })
    avatarWidget["aria-label"] = "Author avatar"
    bottom.appendChild(avatarWidget)
    const footerWidget = document.createElement("section")
    Object.assign(footerWidget.style, {
        flex: 1
    })
    bottom.appendChild(footerWidget)
    const logoWidget = document.createElement("a")
    merge(logoWidget, {
        href: "https://4c.rocks",
        onclick: () => addAchievement(25, "Visited 4C Rocks"),
        target: "_blank",
        "aria-label": "Link to 4C Rocks site"
    })
    merge(logoWidget.style, {
        display: "block",
        width: "64px",
        height: "64px",
        borderRadius: "8px",
        backgroundSize: "contain"
    })
    bottom.appendChild(logoWidget)
    parent.appendChild(main)

    return (parent._madeContainer = {
        main,
        bottom,
        mainWidget,
        footerWidget,
        logoWidget,
        avatarWidget,
        notificationWidget
    })
}

Examples

If you've previously voted then you'll see the results, otherwise please vote to see what others think:

Conclusion

In this instalment we've seen how to load custom code into a widget, irrespective of the framework used and then how to use this code to make a pluggable UI.

miketalbot / cccc-widget

4C Blogging Widget

Open source widget for https://4c.rocks

View on GitHub

Making A Simple Serverless Gamification API With Firebase Functions

Mike Talbot ⭐ — Sun, 19 Sep 2021 11:18:56 +0000

TLDR;

I'm building a widget to help content authors grow their audience by asking them questions! Using quizzes and polls (for now) that are embedded in a post an author can reinforce learning through exercises and quizzes or sample opinion in a poll.

To make the widget more fun it has the basics of a gamification system with achievements and points so that the reader feels a reward for their interactions. This article discusses the API calls and method for doing this.

The Widget

You can try out the interactive widget below, you should get some badges and points for taking the quiz. It's just a bit of trivia fun this time - can you get 5/5?

Enter below!

Gamification

The idea of gamification is to reward people for performing actions that you would like them to do. Rewards could be anything from virtual points and badges to real world prizes (though you need some pretty good security for that - I know, I've done it!)

In our simple widget we are just going to give people points and award badges for various actions that they complete. Some badges can be awarded more than once (for example: "Read New Article") while others can only be earned once (for example: "Voted in 'XYZ' Poll" or "Got a quiz question correct").

Gamification is a way of saying thanks for interacting, and it can be very powerful user engagement tool as a part of a network of interactions, or just a little fun like here.

Both readers and content authors receive points and badges to keep everyone in the mix.

Requirements

Award points for actions
Award achievements for actions when awarding points
Award unique achievements and points
Acknowledge that we've told the user about their new achievements

This last is so we can display a "Snackbar" to show the achievement, and only show it once.

We also need to deal with the API for storing unique user responses:

Store a unique response per user, if they respond again (like retaking a quiz, replace it)

Data Model

Here's a reminder of the data model from earlier in the series.

Implementation

I've chosen to implement the API as a Google Firebase Function API. I am going to be reimplementing it as a Cloud Run version as I'm told that this will cost less, that'll come at the end of the series.

The Code

(internal) awardPoints

Let's start with a generic function to award points. This function has to try to stop someone cheating and writing a bot to keep submitting scores. It's not trying very hard to be honest! If you really wanted secure scores, you'd require at least a user login rather than an anonymous user, but for this use case I don't think many people would bother just for a bit of fun, so we will have to provide some kind of rudimentary cheat detection.

We will call awardPoints from other functions and provide an API for it in a moment.

First the signature has us pass a user id, a number of points, an optional achievement and a function that can award bonus achievements and points (for example if this is the first time something happened)

async function awardPoints(
    userUid,
    points = 1,
    achievement,
    bonus = () => [0]
) {
    if (!userUid) return

Next we make sure you can't take points away, then we get a reference to the user's scores

    points = Math.max(0, points)
    const scoreRef = db.collection("scores").doc(userUid)
    const snap = await scoreRef.get()
    const data = snap.exists ? snap.data() : {}

To prevent cheating we are going to be able to set a cool off date, if this is set and it's after now then we don't do any more:

    if ((data.coolOff || Date.now()) > Date.now()) return

Next to help with cheat prevention, we keep a record of the times that scoring events occurred, and we use an average of the last events to decide if we will allow this score to proceed:

    const times = (data.eventTimes = data.eventTimes || [])
    times.push(Date.now())
    if (times.length > 10) {
        let total = 0
        for (let i = 1; i < times.length; i++) {
            total += times[i] - times[i - 1]
        }
        const average = total / times.length

Having calculated the average time of the last few calls we first make a decision about them happening frequently, and if they do, we increment an error count and use it to decide on a cool off period:

        if (average < 5000) {
            data.errorCount = (data.errorCount || 0) + 1
            if (data.errorCount > 20) {
                data.coolOff = Date.now() + 1000 * 60 * 60
            }
        } else {
            // Reduce errors if no problem
            data.errorCount = Math.max(0, (data.errorCount || 0) - 1)
        }

Next if we are going really fast - we set a five minute cool off.

        if (average < 500) {
            data.coolOff = Math.max(data.coolOff, Date.now() + 1000 * 60 * 5)
        }

If we have an average over the last 10-20 scores of less than a second, we don't allow this score

        if (average < 1000) {
            return
        }

Then we keep just the last 20 events

        data.eventTimes = times.slice(-20)
    }

Next up we configure for achievements and increment the score, calling our bonus function to see if there is an additional score or achievement:

    data.achievements = data.achievements || {}
    const [extra = 0, extraAchievement] = bonus(data, points, achievement) || []
    data.score = (data.score || 0) + points + extra

Now if we have an achievement, we store the date on which it was earned

    if (achievement) {
        data.achievements[achievement] = Date.now()
        await incrementTag(`__event_${achievement}`, "count")
    }
    if (extraAchievement) {
        data.achievements[extraAchievement] = Date.now()
    }

Finally we end the transaction and store the updated score.

    await scoreRef.set(data)
}

awardPoints

The api version of award points is provided for plugin writers so that they can award what extra scores.

It follows the rules of AppCheck and also ensures you can't award more than 20 points:


exports.awardPoints = functions.https.onCall(
    async ({ points = 1, achievement, articleId }, context) => {
        points = Math.max(0, Math.min(points, 20))
        if (context.app === undefined) {
            throw new functions.https.HttpsError(
                "failed-precondition",
                "The function must be called from an App Check verified app."
            )
        }
        if (!context.auth.uid) return

        await awardPoints(context.auth.uid, points, achievement)

        return null
    }
)

addAchievement

This API function allows a plugin developer to add an achievement for the current user. It takes the article id for reporting purposes, some points to award if the achievement is new, and the name of the achievement.

exports.addAchievement = functions.https.onCall(
    async ({ points = 10, achievement, articleId }, context) => {
        if (context.app === undefined) {
            throw new functions.https.HttpsError(
                "failed-precondition",
                "The function must be called from an App Check verified app."
            )
        }

Firstly it ensures you can't award more than 50 points:

        points = Math.min(points, 50)

The it gets the score reference and checks if the achievement already exists, if it doesn't it adds it.

        if (!achievement) return
        const userUid = context.auth.uid
        const scoreRef = db.collection("scores").doc(userUid)
        const snap = await scoreRef.get()
        const data = snap.exists ? snap.data() : {}
        data.achievements = data.achievements || {}
        if (!data.achievements[achievement]) {
            await awardPoints(userUid, points)
            data.achievements[achievement] = Date.now()
            await scoreRef.set(data)
        }
    }
)

respondUnique

Finally we want to be able to decorate the response table with the unique information for the current user, this is how we store poll and quiz results. Each user has their own section of the "response" that contains the data they supplied.

The function takes an articleId, a type supplied by the developer and an object or value to store called response.


exports.respondUnique = functions.https.onCall(
    async ({ articleId, type = "general", response }, context) => {
        if (context.app === undefined) {
            throw new functions.https.HttpsError(
                "failed-precondition",
                "The function must be called from an App Check verified app."
            )
        }
        if (!context.auth.uid) return null

If there is a response then we award points to both the article author and the user.

        const article =
            (await db.collection("articles").doc(articleId).get()).data() || {}
        if (response) {
            await awardPoints(context.auth.uid, 100, "Interacted With Article")
            await awardPoints(article.author, 20, "Gained an interaction")
        }

Next we get a record from the "responses" collection and record the type of response we are making in it.

        const responseRef = db.collection("responses").doc(articleId)
        const doc = await responseRef.get()
        const data = doc.exists ? doc.data() : {}
        data.types = data.types || []
        if (!data.types.includes(type)) {
            data.types.push(type)
        }

Next we get the block of data that represents this type of response (for instance the Quiz uses "Quiz" for a type). This block will then contain one response for each unique user.

        const responseCollections = (data.responses = data.responses || {})
        const responses = (responseCollections[type] =
            responseCollections[type] || {})
        responses[context.auth.uid] = response

Next we increment some reporting counters

        data.responseCount = (data.responseCount || 0) + 1
        await db
            .collection("counts")
            .doc(articleId)
            .set({ responseCount: data.responseCount }, { merge: true })

And finally we complete the transaction and store the data back in the response.

        await responseRef.set(data)
        return null
    }
)

acknowledge

We need to record the last date on which we told a user about their achievements, so we don't keep repeating ourselves:


exports.acknowledge = functions.https.onCall(async ({ time }, context) => {
    if (context.app === undefined) {
        throw new functions.https.HttpsError(
            "failed-precondition",
            "The function must be called from an App Check verified app."
        )
    }
    const scoreRef = db.collection("scores").doc(context.auth.uid)
    scoreRef.set({ acknowledged: time }, { merge: true })
})

We just update a field in the scores with the time we showed the user their achievements.

Conclusion

In this article we've looked at an API for gamification and response tracking using serverless functions. In future instalments we'll see how to use these functions to make the front end of the Widget and the plugins like Quiz and Poll.

Thanks for reading!

miketalbot / cccc-widget

4C Blogging Widget

Open source widget for https://4c.rocks

View on GitHub

Forem: Mike Talbot ⭐

Fascinating reading, very thought-provoking content...

Mean Time to Understanding 🤔: The Irreducible Human Element in the Age of Infinite Code 🤖.

Interview Question: how do you ensure you only virus scan once?

Example interview question

Guiding interviewees

Conclusion

MapPlus: a better "Map" makes your code easier to read...

A debate to be had.

Why AI Can't Replace Developers: The Real Limits of AI in Coding

Michael Amachree ・ Nov 26

One Byte Explainer: Deadly Embrace

Coding solutions with AI

The world changed

Solving problems with AI

A simple example

My business

The problem

The solution

Using AI

Building and debugging the solution

The solution

Utilities

An espresso shot of "Smooth": requestIdleCallback in <256 characters

Explainer

Avoid the Promise.all pitfall! Rate limit async function calls

The Solution

Delight your users with this instant scroll restoration custom hook

TL;DR

The Problem

The Solution

useScrollRestoration hook

The Concept

Building the Hook

Storage

The ref function

Connecting to a scrollable element

Disconnecting the handlers

The Whole Thing

Demo

Conclusion

Game In A Month: Serverless Leaderboards

Leaderboards

The Code

Recording a score

Getting the highscore table

Calling From Unity

Conclusion

An extensible React "Plug & Play" Widget

TLDR;

Motivation

Requirements

Configuration Interface

Runtime Script Loading

Implementation

Registering New Plugins

Registering A Plugin

The Editor

Poll Editor UI

The Runtime

Implementation

Runtime Responses

Runtime Rendering

Examples

Conclusion

miketalbot / cccc-widget

4C Blogging Widget

Making A Simple Serverless Gamification API With Firebase Functions

TLDR;

The Widget

Enter below!

Gamification

Requirements

Data Model

Implementation

The Code

(internal) awardPoints

awardPoints

addAchievement

respondUnique

The `ref` function