Forem: Alan North

Handling Thousands Cards on an Infinite Canvas

Alan North — Tue, 03 Feb 2026 14:42:01 +0000

Click here to read on my site with proper video embeds

I'm working on an app, Pyramid Notes, which will have an infinite canvas.

It will have two modes mainly:

freeform mode - users can drop and place cards everywhere.
"structured" mode - with a preset layout. In this initial version, the layout will is a visualization of note tags (they function more like folders).

Initial DOM Based Attempt

I had initially planned to create an HTML based infinite canvas as it's just easier to get started with, but the DOM approach has a lot of lag, especially when zooming, which was a deal breaker. I needed really fast zoom in/out performance.

The html version was a bit naively coded, but the same approach with pixi.js (webgl canvas lib) was 10x more performant. Properly optimized HTML starts lagging at around 1000 notes, my initial quick canvas implementation started lagging at around 10,000.

Here is a link to 1000 notes on tldraw (and the blank version) which seems well optimized so you can see it's not me with a bad implementation.

I have a video of this, but it's not working on dev.to.

There's just like a very sluggish feeling, especially zooming. I suspect the culprit might be shadows because they seem to get turned on/off at the point zoom feels sluggish, but I don't think there's any way around that. The blank version is clearly faster so some of it is the text itself rendering when very zoomed out, but I tried different lod levels in DOM, and the zoom range where they change feels slow. Sometimes canvas also skips a frame at that point but it doesn't feel nearly as bad.

Switching to Pixi.js

Once I switched to pixi.js, I was way too tempted to properly test the limits and see how many notes I could get to render.

I setup some basic culling + lod states, and changed the layout structure (from flat to a tree) and got to around 50,000 notes before serious lagging, but I wanted to find the real limit. In part out of curiosity, in part to not attempt to load so many (users can limit what they load with a search filter).

Mind you, the 50,000 at that point was a bit of a lie because the notes weren't unique. I also made this mistake when I initially tweeted about reaching 100,000. See below for more details.

Actual Usage Expectations

From what I've researched and from the number of notes I have myself, I expect the average user to probably have ~1000-10,000 notes. As a maximum I can see maybe some power users will have 20-50,000.

I don't expect >50,000 notes, but it is possible, depending on how they organize them, that they might need to render more since notes can appear more than once in the structured tree.

I do not expect freeform mode to get nearly as big as notes must be added by hand.

Limits Reached

I ended up reaching about 60,000 unique cards before there started to be some minor lag (still snappier than the DOM approach), and 90,000 unique cards before it started to feel too sluggish. I reached 100,000 a few times, but past 95,000 the page would occasionally crash as it finished generating all the text.

One can sacrifice a bit of LOD to reach smoother 100,000, but I didn't like it. I think it's best to just warn the user to use a filter past 60,000. If they want to risk it they can (on chrome the crashes sometimes required I restart the entire browser 💀).

Here's what 100,000 feels like, you can see it's starting to struggle:

Here's 50,000 which feels super snappy (this was the thumb). 60,000 starts to occasionally skip a frame on zoom but still feels fast:

I'm actually sure we could go further (see #pooling) but I don't think the increase in code complexity is worth it.

Avoiding Accidental Optimizations

It's easy to accidentally get performance speedups from how one sets up the test scenario.

In this case initially I only had 1000 unique notes and initially they were just repeated to get 50,000/100,000. I started like this because generating 50,000 real random documents is currently still super slow, I can only do about 300 docs max at a time. This is because generating a note requires generating a random prosemirror document and this was tricky to even get working.

Now Pixi.js does caching internally to speed up scenerios with duplicate cards like this so I had to make adjustments to avoid all unreasonably performance optimizations and find the real number of unique notes I could render.

I say unreasonable because some repitition is expected in every day use. To control the uniqueness and take more reasonable advantage of pixi's internal caches (it has per word (temporary), and per text), I generated 170,000 unique "words" (they're just random characters/symbols) and replaced the card text with random slices from those words.

Why 170,000? This is about how many words are actively in use in the english language. They are distributed randomly here, unlike a realistic distribution where some words appear way more than others, but that just means real usage will be slightly faster.

Here's the code I used:

Click to Expand

import { faker } from "@faker-js/faker"

export const perfUtilsConfig = {
    minWordLength: 3,
    maxWordLength: 10,
    uniqueWords: 170000,
    uniqueNoteCount: 90000
}
const seenWords = new Set<string>()
export function createWords(count: number) {
    const res = Array.from(
        { length: count },
        () => {
            // i know faker has word generation, but I want random symbols included
            let word = faker.string.sample({ min: perfUtilsConfig.minWordLength, max: perfUtilsConfig.maxWordLength })
            while (seenWords.has(word)) {
                word = faker.string.sample({ min: perfUtilsConfig.minWordLength, max: perfUtilsConfig.maxWordLength })
            }
            return word
        }
    ).join(" ")
    return res
}
// estimated number of english words in active use
const additionalText = createWords(perfUtilsConfig.uniqueWords)
seenWords.clear()

const noteSlicePoints = Array.from({ length: perfUtilsConfig.uniqueNoteCount }).map(i => Math.floor(Math.random() * additionalText.length))
// slice random bits from the text
export function generateText(max: number) {
    const cut = noteSlicePoints[Math.floor(Math.random() * noteSlicePoints.length)]
    const res = additionalText.slice(cut, cut + max)
    return res
}

You will see in the top right of videos, in the debug details it sayse Uniqueness: [count], this is the number of unique cards generated. Card Count is the number of cards in the full tree. In View will show how many cards are actually in view.

Optimizations

Decoupled Reactive Lifecycle

The rendering loop is decoupled from vue’s reactivity. Instead of immediately reacting to every state change, we set dirty flags and do the appropriate action on the next tick.

I used a class (I did not have a good experience trying to figure out pixi-vue unfortunately) so watchers created here must be properly cleaned up (vue is not automatically aware of them like it is in SFCs).

Click to Expand

class PixiCanvas {
    toDestroy = []
    onMounted() {
        let layoutDirty = true
        this.toDestroy.push(watch(this.layout, () => {
            layoutDirty = true
        }).stop)
        app.ticker.add(() => {
            if (layoutDirty) {
                // ...
            }
        })
    }

    destroy() {
        if (!this.initialized) throw new Error("Pixi not initialized, cannot destroy.")
        for (const item of this.toDestroy) {
            item()
        }
        // textures should also be destroyed here
        this.app.destroy()
    }
}

Culling

Naive culling is not enough, checking thousands of notes is too expensive.

Take Advantage of Existing Structure

If your data is structured in any form of tree, take advantage of that even if it makes it harder to traverse/update the tree.

If a parent container is not visible, all children will be skipped making things way faster.

For the freeform mode, I plan to place them inside invisible containers/buckets to simulate this. The freeform layout will be saveable and we can cache the notes each bucket contains and we would only need to update this if a user moves a note.

Use a Custom Algorithm If Needed

I used a custom algorithm for two reasons:

I already know how big the viewport will be before pixi loads. I can do the culling beforehand.

I know the constraints of my layout. Nested areas are always contained within their parent, and notes are always contained by their areas. We can make optimizations based on this. For example, if the viewport is inside an area, only that area needs to be checked. All other areas can be immediately skipped.

Pixi does not do this internally because children can exist outside of their parent bounds.

Manual Texture Baking

This example demonstrates how to manually instruct the renderer to bake any display object (like a Container or Graphics object) into a reusable texture. This is highly efficient as the complex object is drawn only once.

Click to Expand

// example for the full LOD card background

// this is called once, then only again if the theme ref changes
createTextures() {
    // careful to give enough space if using shadows
    const shadowPadding = 4 * 4
    const container = new Container()

    const shadowGraphic = new Graphics()
        .roundRect(shadowPadding, shadowPadding, this.config.cardWidth, this.config.cardHeight, this.config.cardRounding)
        .fill({
            color: this.theme.value.note.shadow,
            alpha: this.theme.value.note.shadowAlpha
        })
    shadowGraphic.filters = [new BlurFilter({ strength: 4, quality: 3, resolution: window.devicePixelRatio })]
    container.addChild(shadowGraphic)

    const cardGraphic = new Graphics()
        .roundRect(shadowPadding, shadowPadding - 2, this.config.cardWidth, this.config.cardHeight, this.config.cardRounding)
        .fill({ color: this.theme.value.note.bg })
    container.addChild(cardGraphic)

    this.cardFullTexture = this.app.renderer.generateTexture({
        target: container,
        frame: new Rectangle(0, 0, this.config.cardWidth + shadowPadding * 2, this.config.cardHeight + shadowPadding * 2)
    })

    // save the texture for later use
    this.cardFullTexture = this.app.renderer.generateTexture({
        target: container,
        frame: new Rectangle(0, 0, this.config.cardWidth + shadowPadding * 2, this.config.cardHeight + shadowPadding * 2)
    })
}

// later during the actual card creation:
const cardSprite = new Sprite(this.cardFullTexture)

Do NOT attempt to cache the full card, only the background, see #avoid.

This is a bit more complicated for images. I have not added attachments yet, so that will be a problem for later.

Dynamic Level of Detail (LOD)

At full zoom cards show their titles and a short summary. Both require text wrapping and truncation which is very expensive.

Past a certain zoom point cards become unreadable so there’s no point trying to render the text which is very expensive. We can just a simple skeleton, no text. And past that we have two options:

If the app has some other large features we can visualize (in this case cards exist bounding areas in "structured" mode), we can render those instead.
We can render a very small single color rectangle. Again, we can cache this. This is slower than rendering some larger feature, so avoid at all costs.

Using this strategy we can fit roughly 10,000 skeleton cards in view or 1000 full-view cards.

This with the full card zoom cut off point being past the point that I can’t read it on a 4k screen (with the canvas resolution set to window.devicePixelRatio) which is the worst case scenario.

So the cutoff could be increased. It wouldn't look as nice though because even when we can't read the text, we can still get a sense of which note is which. Another option is to only hide the summary initially and only later the text.

In all cases, I will be adding a show-on-hover feature so users can quickly read any card they're hovering over.

Using Bitmap Text

As Pixi themselves suggest, use BitmapText instead of Text. This is absolutely vital past around 50,000 cards. It now supports word wrapping and it's easy to generate from a font, so it should suffice for most needs.

The only drawback is it's more expensive for languages with a lot of symbols (like chinese). I did a quick tests though and did not see much of a difference. Canvas still felt smooth enough and memory did not jump excessively. So good signs.

Loading and On-Demand Text Generation

If we tried to naively load and naively truncate (see below) the text of 100,000 cards on load it would take nearly 30 seconds with all optimizations.

Instead, on load, the card layout is created without any text. When the user zooms we generate the text on demand only when they need the full text LOD.

The generation is limited per so we don’t attempt to create the text for 1000 cards all at once.

We check how much time we have left and hand that ms budge over to generateCardText which will handle not exceeding it.

Batching like this also helps give the user a visual indication that cards are getting loaded.

Full example combined with idle generation below.

Background Idle Generation

The canvas loads zoomed out. This gives us some time, between load, and the user zooming/panning to where they want. We can use requestIdleCallback to start generating the card text regardless of visibility. This means for most users (less than < 10000 cards which take 3 seconds to load) they probably won’t even see the text get generated, it’ll just be there.

Here's the code combining the on demand generation with the idle generation.

Click to Expand

onMounted() {
    // ... initial setup

    // visibilityDirty and docsDirty are set by vue watchers

    app.ticker.add(() => {

        this.updateLodState() // calculates the lod state
        const canGenerateText = !firstTick && this.lodState.value === "full" && this.pendingTextGeneration.size > 0
        const start =  canGenerateText ? performance.now() : undefined

        if (visibilityDirty) {
            visibilityDirty = false
            this.updateQueuesAndCulling()
        }
        if (docsDirty) {
            docsDirty = false
            this.updateTree()
            this.initBackgroundQueue()
            this.scheduleTextGeneration()
        }

        if (canGenerateText) {
            const now = performance.now()
            // budget is a little lower than the real 16ms just in case
            const budget = 13 - (now - start!)
            this.generateCardText(this.pendingTextGeneration, "visible", { startTime: now, budget })
        }
    })
}


// we combine these actions because they both happen together and need to both iterate through the same data
updateQueuesAndCulling() {
    const { visibleNotes, visibleAreas } = this.visible.value

    this.updateVisibleAreas(visibleAreas)

    const showFullText = this.lodState.value === "full"
    const cardLodVisible = this.lodState.value !== "noCards"

    for (const [path, card] of this.noteSprites) {
        const isVisible = visibleNotes.has(path)
        const titleText = card.getChildByLabel("title-text") as Text
        const textAlreadyGenerated = titleText.text !== ""

        card.visible = isVisible && cardLodVisible

        if (!textAlreadyGenerated) {
            if (isVisible && showFullText) {
                this.backgroundPendingTextGeneration.delete(card)
                this.pendingTextGeneration.add(card)
            } else {
                this.pendingTextGeneration.delete(card)
                this.backgroundPendingTextGeneration.add(card)
            }
        } else {
            this.pendingTextGeneration.delete(card)
            this.backgroundPendingTextGeneration.delete(card)
        }
    }

    const cards = visibleNotes.keys()
        .map(path => this.noteSprites.get(path)!).filter(_ => !!_)
    this.updateCardsLodState(cards)
}


// initially, queue up all cards that don't have text yet
initBackgroundQueue() {
    for (const card of allNoteSprites) {
        if (card.text === "") {
            backgroundPendingTextGeneration.add(card);
        }
    }
}

// create the function that actually schedules the work
scheduleTextGeneration() {
    if (this.backgroundPendingTextGeneration.size > 0 && this.idleCallbackId === null) {
        this.idleCallbackId = requestIdleCallback((deadline) => {
            this.idleCallbackId = null
            this.generateCardText(this.backgroundPendingTextGeneration, "background", { deadline })
            this.scheduleTextGeneration()
        })
    } else if (this.backgroundPendingTextGeneration.size === 0) {
        // generation finished
    }
}

/** Generate text for cards from a pending queue with a deadline or a time budget. */
generateCardText(
    pending: Set<Container>,
    type: "visible" | "background",
    {
        deadline,
        budget,
        startTime
    }: {
        deadline?: IdleDeadline
        startTime?: number
        budget?: number
    } = {}
) {

    if (pending.size === 0) return

    // the 1 is the aproximate time the truncation takes
    if (budget !== undefined && budget <= 1) return

    const isVisible = type === "visible"

    const iterator = pending.values()
    let generated = 0

    while (pending.size > 0) {
        if (budget !== undefined && isVisible && (performance.now() - startTime!) >= budget - 1) {
            break
        }
        if (deadline && deadline.timeRemaining() <= 1) {
            break
        }
        const next = iterator.next()
        if (next.done) break

        const card = next.value as Container

        pending.delete(card)

        if (type === "visible") {
            this.backgroundPendingTextGeneration.delete(card)
            // skip if card became invisible or LOD changed
            if (!card.visible || this.lodState.value !== "full") {
                continue
            }
        } else if (type === "background" && this.pendingTextGeneration.has(card)) {
            continue
        }

        const titleText = card.getChildByLabel("title-text") as Text
        // const summaryText = ...
        const doc = this.getDoc(card.label)
        // doc might have been deleted by the time we get here
        if (!doc) continue

        generated++

        titleText.text = PixiCanvas.truncatePixiText( doc.title, /* ... */)
        // summaryText.text = PixiCanvas.truncatePixiText( ... )

        if (card.visible && this.lodState.value === "full") {
            // set visibility of elements
        }
    }
}

Faster Text Truncation

Initially I just looked up how to do text truncation and found this snippet on the repo which you'll probably come across if searching for how to do this. Here’s the modified version I initially used:

Click to Expand

/** Truncates text to a specific number of lines using PixiJS TextMetrics */
static truncatePixiText(
    /** The raw string to truncate */
    text: string,
    /** PixiJS TextStyle object or configuration */
    style: TextStyle,
    /** Maximum number of lines allowed before truncation */
    maxLines: number
): string {
    if (maxLines === Infinity) {
        return text
    }

    const wordWrapWidth = style.wordWrapWidth
    const { lines } = CanvasTextMetrics.measureText(text, style)

    if (lines.length <= maxLines) {
        return text
    }

    const truncatedLines = lines.slice(0, maxLines)
    const lastLine = truncatedLines[truncatedLines.length - 1]

    const chars = Array.from(lastLine)

    // measure the ellipsis and each character individually
    const charMetrics = CanvasTextMetrics.measureText(`...\n${chars.join("\n")}`, style)
    const [dotsLength, ...charLengths] = charMetrics.lineWidths

    let newLastLine = "";
    let currentLength = dotsLength;

    for (let i = 0; i < charLengths.length; i++) {
        const charLength = charLengths[i];

        if (currentLength + charLength >= wordWrapWidth) {
            break; 
        }

        newLastLine += chars[i];
        currentLength += charLength;
    }

    truncatedLines[truncatedLines.length - 1] = `${newLastLine}...`
    return truncatedLines.join("\n")
}

As you can see this text truncation function requires two measurements. First, based on the text style, it measures how many lines the wrapped text produces. Then, it takes the line and measure character by character to see where to cut it (the original measured by word, I wanted by character, both are slow).

Pixi does tons of caching internally, but it's just a slow thing to calculate. It's also specially slow with regular Text (which I was using at the beginning).

So how to speed this up (apart from using BitmapText)? See if you can spot it before reading on. The biggest performance gain feels a bit obvious once you see it.

Some low hanging fruit first, if you have long note summaries that aren't already truncated, it's measuring all that text which won't be visible.

Optimization 1 - Limiting Text Length

I measured the smallest character (usually a space) and calculated how many we could fit in a card.

We can then use this to limit how much text we even try to truncate. This should also done as soon as we can after we receive the data to avoid keeping the full summary in memory.

Optimization 2 - Avoiding the Second Measurement

We want to avoid the second measurement of the characters at all costs.

I tried caching the measurements of individual characters to calculate the aproximate width of the last line myself. Pixi is already technically doing this but it doesn't keep the cache around between calls. That was faster, but still slow.

I considered using a monospace font even though I didn't like the idea of being restricted to monospace fonts.

I asked the LLMs. They added a third measurement call just to rage bait me, and I went back to thinking.

And then I saw it.

When pixi does the first measurement, it also chops up the line, and we know what text is on which line and how long it is already! 🤦

Now, . is usually the shortest character in a font (and if not, we should pick a font where it is).

So when a line needs an ellipsis all the last characters we need to chop off are wider than a period. When we replace them, the line is always guaranteed to be shorter.

For example, say we want max 2 lines:

         cutoff ─┐
                 🭭 
Some long text we want to truncate

// pixi will wrap it to:
                 🭭 
Some long text
we want to truncate

// and we want:
                 🭭 
Some long text
we want to trun...

With a non-monospace font, here's how they compare:

we want to truncate

we want to trun...

The truncated line is always shorter.

If we want truncation at the word level, we can just find the first space that's at least 3 characters in.

Here's the final code which takes about 20% less time.

This version supports both Text and BitmapText. As mentioned above bitmap support is crucial as otherwise regular text is still too slow even with this speedup.

Click to Expand

/**
* Truncates text to a specific number of lines using PixiJS TextMetrics.
*
* Uses lots of optimizations and caching to make this as fast as possible.
*
* IMPORTANT: Be sure to pre-slice the text if it can get very long.
*/
static truncatePixiText(
    /** The raw string to truncate */
    text: string,
    /** PixiJS TextStyle object or configuration */
    style: TextStyle,
    /** Whether to use the bitmap text measuring method or the canvas text measuring method. Words will NOT wrap right if you use the wrong one. */
    isBitmapFont: boolean = true,
    /** Maximum number of lines allowed before truncation */
    maxLines: number,
    /** Since we recommend slicing the text, we can't be sure if there are more lines or not and whether to add an ellipsis to the last line.
    * @default false
    */
    containsMoreLines: boolean = false,
    /** Pass a cached width. */
    ellipsisWidth?: number
): string {
    if (maxLines === Infinity) {
        return text
    }

    const wordWrapWidth = style.wordWrapWidth

    let lineWidths = []
    let lines = []
    if (isBitmapFont) {
        const measure = BitmapFontManager.getLayout(text, style, true)

        for (let i = 0; i < measure.lines.length && i < maxLines; i++) {
                const line = measure.lines[i]
                const chars = line.chars.join("")
                if (chars.length === 0 && i === measure.lines.length - 1) {
                    // last line is always empty, idk why
                    break
                }
                lineWidths.push(line.width * measure.scale)
                lines.push(line.chars.join(""))
        }
    } else {
        const measure = CanvasTextMetrics.measureText(text, style)
        lineWidths = measure.lineWidths
        lines = measure.lines.slice(0, maxLines)
    }
    if (lines.length < maxLines) {
        return text
    }
    const lastLine = lines[lines.length - 1]

    if (ellipsisWidth === undefined) {
        if (isBitmapFont) {
                const measure = BitmapFontManager.getLayout("...", style, true)
                ellipsisWidth = measure.width * measure.scale
        } else {
                ellipsisWidth = CanvasTextMetrics.measureText("...", style).width
        }
    }
    const ellipsisDontFit = ellipsisWidth > wordWrapWidth
    if (ellipsisDontFit) {
        lines[lines.length - 1] = ""
        return lines.join("")
    }

    // this is a stylistic choice
    const shouldAddEllipsis = (containsMoreLines || lines.length > lines.length) && !lastLine.endsWith(".")

    if (lineWidths[lineWidths.length - 1] + ellipsisWidth < wordWrapWidth && !shouldAddEllipsis) {
        return lines.join("")
    }

    if (style.breakWords) {
        // dots happen to be the shortest width character in most fonts
        // so we can just cut off the last 3 characters and add the ellipsis
        lines[lines.length - 1] = `${lastLine.slice(0, lastLine.length - 3)}...`
    } else {
        // for words, we just have to find the last space that is at least 3 characters in
        const lastIndex = lastLine.lastIndexOf(" ", lastLine.length - 3)
        if (lastIndex === -1) lines[lines.length - 1] = `...`
        else lines[lines.length - 1] = `${lastLine.slice(0, lastIndex)}...`
    }
    return lines.join("")
}

Further Potential Optimizations

Pooling

This is probably the next biggest optimization I could make and is something Pixi.js recommends for handling a lot of objects.

If we cache the text wrapping and truncation, we could re-use the same ~1000-2000 text objects and just overlay them over the visible cards. This, I think will give the most additional performance boost, but complicates things a lot. Even just keeping the existing structure and adding/removing the child text objects gets complicated. I have a rough draft and it is hard to read.

Others

As mentioned at the start, all those optimizations that I avoided can be allowed to happen for a speedup.
I structured the app so we can potentially hook into document changes. That way instead of searching all notes for changes on updates, we can do surgically precise updates.
The wrapping indexes + truncation index could be saved with the document and/or view. Once saved, we can just pass a pre-split line to Pixi.js and avoid text wrapping entirely.
Due to how the app works, users can open multiple panes with the infinite canvas view. I will have to reduce the lod for unfocused instances past a certain note count to allow the focused / last focused canvas to stay performant.

Usability Improvements + Future Features

For < 1000 we can probably just disable the different lod states.
To help with navigation, as mentioned, hovering over any note will show it in the corner - we can use a single object and just change it's text.
Selection of notes - this can be done via sprite tinting.
Movement - initially I had used a flat list because that would have made it easier to move notes but because we are now using a tree, this will require: hiding the note, creating a temporary copy that's on top of everything, then moving that.

AVOID

Do NOT use cacheAsTexture for the card with text. You're creating dozens of tiny textures which is pointless and worsens performance. If your notes are contained within some parant container of known size that is not too big (less than 4096x4096 pixels) it might help.

How to Recover Locally Deleted Files From Github

Alan North — Wed, 31 Jan 2024 19:39:02 +0000

So let's just say, hypothetically, you rewrote your git history or created a fixup commit that deleted a file, force pushed to github, deleted the repo locally, then realized you accidentally deleted one file too many.

I know, is it possible to commit so many sins at once?

This did not happen with code for me, it was just some dotfiles stuff. I was moving things around and removed a sensitive file from the commit history, then accidentally deleted the repo, only to realize I hadn't backed up the sensitive file I still needed.

First Step, Don't Panic

If you did not delete the repo, you're good. You can search how to use git reflog to recover it.

Otherwise github should keep the commits for 90 days.

Now I found the partial answer to how to do this here.

The gist of it is we can use the github api to find the commit and extract a patch. Problem is with a private repo this answer won't work. Additionally if we committed multiple files we don't want to have to extract each patch as suggested by the answer. We just want the full patch.

Get the Github CLI

If you don't have it already you will need the github cli. Authenticate with gh auth, and now we can use the gh api function.

Find Commit to Recover

First, if we need to find which commit we need to recover we can search through the events:

gh api repos/{OWNER}/{REPO}/events

We can stick the result in a temp file for easier searching.

gh api repos/{OWNER}/{REPO}/events > temp

And we can then make it easier to find the commit by filtering the response with jq.

cat temp | jq ".[].payload.commits.[] | {url: .url, author: .author.name, message: .message}"

Once we find our commit we can query the url (just the end bit see here):

gh api repos/{OWNER}/{REPO}/commits/{SHA}

This will give us a json response which is not what we want but we can use it if we want to further find if it contains the correct changes we're looking for.

gh api repos/{OWNER}/{REPO}/commits/{SHA} | jq ".files[] | {status:.status, filename:.filename, patch: .patch}"

Get the Patch

To actually get the patch we will need to change a header.

# save the patch somewhere not in the repo
gh api -H "Accept: application/vnd.github.patch" repos/{OWNER}/{REPO}/commits/{SHA} > patch

Apply the patch.

We can then apply it (be sure to have committed any changes beforehand). How to apply it can depend, but the following should work even if the repo doesn't have the files anymore, you can then recover what you need:

git apply /path/to/patch --3way --ignore-space-change --ignore-whitespace

Further Notes

Note that if the patch contains binary file changes, applying them might bork your terminal. You can try sending reset one or two times, or try with a different terminal (Konsole worked for me).

How to use Dynamic Variables with Tailwind

Alan North — Mon, 09 Oct 2023 13:50:39 +0000

Have you ever wanted to do p-[${someDynamicValue}] in tailwind? Or maybe the more complicated after:content-[${someDynamicValue}]? Here's a better solution than resorting to a style tag.

Tailwind cannot have dynamic values in it's classes because the stylesheets are compiled at build time, neither the class name nor the value can be made up of dynamic variables for this to work.

This can be a bit of a pain in certain situations. One obvious solution is to move back to a style tag, but this means now the styles aren't all in one place, the main selling point of tailwind.

For certain situations, like p-[${someDynamicValue}] we can fallback to the style property:

<!-- I'm using vue here, but all frameworks have a way to do this. -->
<div
    :style="`padding:${someDynamicValue}`"
></div>

But this is not always the case. Take after:content[${someDynamicValue}]. Style tags cannot target psuedo-elements, we have to resort to a style tag, or do we?

Style tags can have inline css variables. So we can actually do this:

<div
    class="after:p-[var(--customPadding)]"
    :style="`--customPadding:${someDynamicValue}`"
></div>

Tailwind can now happily compile the class, it's name is no longer dynamic, it's .after\:p-\[var$--customPadding$\] and it's value is no longer dynamic, it's var(--customPadding). The part of the outputted css that we care about looks like this, here's a link to a demo on the playground.

.after\:p-\[var\(--customPadding\)\]::after {
    padding: var(--customPadding)
    //...
}

And we can change the css variable to whatever we need. It's also very easy to find where the custom value was declared.

I cave, Tailwind rocks!

Alan North — Thu, 14 Sep 2023 23:16:55 +0000

My first impression of tailwind was 🤮, css in html, not this again. What about separation of concerns! When I looked at examples they just looked like some sort of class soup. The first example has to be scrolled because the lines are so long! Utility functions are a nice idea I'd been toying with adopting, but surely this went to far?

I had looked into other css frameworks but it was easier to just roll my own than customize a utility framework so I didn't. I'd been toying with a theme management system / variable generator for apps, so using someone else's variables did not make sense. For a while I just used css with the variables my package produced to keep a consistent design.

Tailwind just seemed like another utility based css library that wouldn't benefit me very much.

But after seeing it praised again and again, I had to try it. The docs are not wrong about this.

I tried it on a complicated component library to really battle test it and I finally understood what all the praise was about.

It's not just the points you hear all the time. There were also some things that aren't obvious on first glance that I really liked:

It's very easy to customize and make custom plugins.
You can quickly use custom values when you need them (e.g. border-[calc(...)]).
Many utility functions do more than one thing and make it easy to do common things that tend to bloat css.
Predictable application and overriding of styles (assuming the use of something like tailwind-merge).

It's hard to believe, but these all add up to make the class soup actually far exceed my previous scss in simplicity, readability, and maintainability. The link above shows this to some extent but I did not even see this until I actually decided to try tailwind. It really should be one of the first things you see.

Also you don't have to put it all in one line! Why is it like that in so many examples, 😭?

You can still write very readable styles if you want. And once you get a hang for the basic utility classes (this does not take as long as you'd think), the intent of the styles is far far clearer.

Here's my own example. Where before in a my component library might have css that looked something like this (and yes, I know the css could be simplified, but in exchange for verbosity elsewhere¹):

.component {
    padding: 0 var(--padding-s);
    background: var(--color-bg-el) ;
    &.border {
        border: var(--border-width) solid var(--color-border);
        &:focus { border-color: var(--color-border-focus) ;}
    }
    &:focus { color: var(--color-text-focus) ; }
    &.disabled { background: var(--color-bg-disabled); }
    &.disabled:focus { color: var(--color-text-disabled);}
    &.border.disabled:focus { border-color: var(--color-border-focus) ;}
}
.wraper {
    .component.disabled.border & {
        text-decoration-line: line-through;
    }
}

with html (vue) that looked like this:

<div
    :class="classes"
>
    <div class="wrapper"/>
</div>

Where classes was a computed red in setup, often abstracted away to a function for common states:

const classes = computed(() => ({
    disabled: props.disabled,
    border: props.border
    ...
}))

Separation of concerns, right...

Now with tailwind, after a bit of experimenting and the help of tailwind merge I can have html (vue) that looks like this. Same number of css rules.

<div
    :class="twMerge(`
        p-3
        bg-neutral-50
        focus:text-blue-500
        `,
        border && `
            border
            border-neutral-900
            focus:border-blue-500
        `,
        disabled && `
            bg-neutral-300
            focus:text-neutral-500
        `,
        border && disabled && `
            border-neutral-500
        `)"
>
    <div :class="disabled && `line-through`"><div>
</div>

And that's it.

There is no more jumping to the js or the css to look things up.

All styles will apply predictably. It's clear what's happening, which classes are being applied when. With the help of tailwind merge we can rely on order and use variables like in the example to avoid specificity issues.*.

Just a breath of fresh air.

Even with one style per line for readability I have managed to shrink the file size of ALL the components I converted.

I only now have two sections to worry about now, the logic, and the presentation. I've come to understand that presentation is invariably linked to state, and separating the css actually always makes things more painful. The creator of tailwind actually mentions this in the blog post where he made the arguments for tailwind on it's release, and I had read this before, but it never sunk in how painful this was until I wasn't experiencing the pain anymore.

I was actually surprised how much faster I could style things (~3x) even when I had to have the tailwind docs pinned to lookup stuff and my tailwind autocomplete was not working well².

I've come to appreciate how easy it can do things that used to take multiple lines.

Take for example truncate which does:

overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;

Or gradients! If there's one thing I hate, it's writing css gradients. But in tailwind it's super simple, even with complex stops and everything:

<div class="bg-gradient-to-r from-white to-black"/>
<div class="
    bg-gradient-to-r
    from-white from-10%
    via-blue-500 via-20%
    to-black to-50%
"/>

* Tailwind Merge

But, tailwind merge is a MUST for complex applications and component libraries to avoid specificity issues. The reason why is that while tailwind cannot escape the css selector specificity problem completely, in fact we loose a bit of control.

See in regular css, if I have two selectors with equal specificity, the last one will win (note that this helps avoid the mess of having many variants and states):

.input {
    &.border:focus {...}
    &.disabled:focus {...}
}

In tailwind, you DO NOT have control over the order. The order of the classes is irrelevant, BUT the order tailwind declares them in the stylesheets is and this cannot be controlled (since a single rule might be used in many components).

If you do this, depending on what colors you picked, the style might or might not apply on disabled:

<div
    :class="
        border && `
        border
        border-red-500` +
        disabled && `
        border-neutral-500
    `"
>
</div>

Another thing I've seen done is using data attributes which looks cleaner and will seem to work at first, but this is because the data attributes are adding specificity to the selector, +1 for each data attribute.

Here's a specificity calculator you can use to see this.

Tailwind data attribute variants create the following selectors: {ESCAPED_NAME}:[data-border="true"]:[...] you can input .class:[data]:[data] to quickly test, it would be the same thing.

<div
    :data-border="border"
    :data-disabled="disabled"
    class="
        data-[border=true]:border
        data-[border=true]:border-red-500
        data-[border=true]:data-[disabled]:border-neutral-500
    ">
</div>

You can easily run into the following problem, where these have equal specificity (2):

<button
    :data-primary="primary"
    class="
        data-[primary=true]:text-white
        focus:text=blue-200
    ">
</button>

Or worse, here the text will never change on focus when the button is primary and has a border because that has a specificity of 3.

<button
    :data-primary="primary"
    :data-border="border"
    class="
        data-[border=true]:data-[primary=true]:text-white
        focus:text-blue-200
    ">
</button>

At this point you must pick a solution. You can:

Add an important focus:!text-blue-200 which can make it harder for the end user of components to override the rule.
Add data-[border=true]:data-[primary=true]:focus:text-blue-200, but apart from creating hard to read css, this will create a lot of unneeded class soup in the html since the button would contain all styles for all variants even if you only end up using the one of them.
Use tailwind merge to bypass the problem.

If you pick tailwind-merge though, it is still a good idea to leave the data attributes, at least on the root of the components if you care about users being able to theme the app with css.

<button
    :class="twMerge(`
        p-3
        bg-neutral-50
        focus:text-blue-500
        `,
        border && `
            border
            border-neutral-900
            focus:border-blue-500
        `,
        disabled && `
            bg-neutral-300
            focus:text-neutral-500
        `,
        border && disabled && `
            border-neutral-500
        `,
        $attrs.class
    )"
    v-bind="{...$attrs, class: undefined }"
>
    <div :class="disabled && `line-through`"><div>
</button>

The Downsides

There are, of course, always some downsides.

Inconsistent Naming

After some use, this is the one thing that really bugs me. You have py-* for top+bottom padding but gap-y-* for gaps? This happens in a few other places with units as well. You have spacing that goes down to -0.5 but opacity which in css is 0-1 goes from 0-100 ???

I can understand not always using the same scales (e.g. sm , md, xl vs 0-*) but for some of the odder names, because the auto complete does not work based on the css property name, it can be very hard to find the utility you want. I have to keep the docs opened and pinned for actual searching. But I guess that could be considered more the autocomplete's fault, and not tailwind.

Missing Utilities

Some css properties that are now pretty usable, do not exist yet. For example, text-shadow does not exist because which default styles to add has not been decided yet.

Which fair, but I really wish they would at least they were provided as experimental properties or something instead of having to rely on other's plugins.

Also some which should really exist, don't, such as grid with auto-fill and auto-fit.

Hard to use Existing Tailwind Variables in Custom Utilities

I've written a few custom utilities for gradients, but couldn't for example, re-use the to/from-* gradient utilities and the css variables they create
because I can't seperate the part I need (the color). For example, from-black from-0% will create:

-tw-gradient-from-position: 0%;
--tw-gradient-from: #000 var(--tw-gradient-from-position);

// which are then used like this:
--tw-gradient-stops: var(--tw-gradient-from);

I can use the position, but not the color, there is no --tw-gradient-from-color, so I have to create my own which can get messy.

Dynamic values

The stylesheet is compiled at build time (so that unused styles can be removed), so this means you can't use dynamic styles. This will not work. You have to use an inline style tag or a script tag.

<div :class="`p-[${customValue}]`">
</div>

Conclusion

I've been using it now for a few months and I can never go back to the old way.

Have you given it a try? Did you like it or still hate it?

I know, I could apply all the classes to the wrapper as well or use some BEM naming scheme and simplify the css, but then in complex elements, they need to be attached/created for every element which might only use a single state class, which probably isn't an issue performance wise if the classes aren't changing every second, but just irks me. And conditionally giving them the needed classes is beyond painful so I often stuck with this. ↩
I had trouble getting it to work with vscode + vue. Also you must type the name exactly and this is hard given the inconsistent naming. ↩

How to Publish v0.x.x Packages with Semantic Release

Alan North — Sat, 29 Jul 2023 13:03:10 +0000

I love semantic-release, but it unfortunately doesn't support and is probably never going to support 0 major versioned packages. The issue for it doesn't really contain a good workaround and has unfortunately been locked due to people being rude.

But I'd really like to v0 version some packages.

Why?

I really regret having had to publish some highly experimental packages starting with v1. In their highly experimental phase it's usually just me using them and I pin them when using them so breaking changes aren't an issue. If I start at v1 and properly commit breaking changes, by the time the package is stable it's at v3+ or something. And the entire time the package says it's a WIP but the version is v1+, and I hate it. Often I won't event properly committed breaking changes unless they're major, half defeating the point of semantic versioning.

I've also tried avoiding publishing them in the first place and just using them directly from the repo, but this often requires more effort to get working (since the prepare script needs to be defined to properly build the package after downloading). Still good to publish packages that just work like this, but I always end up running into obscure problems doing things this way.

Ideally I want to just start at v0, write commits that create proper release notes listing the breaking changes, but not have them major bump until the package is more stable and I publish v1.

I got really annoyed at this situation recently so started looking at some of the alternatives mentioned in the issue, but most do not do everything semantic-release does. release-please was the closest, but it's missing an easier way to manager branches and the workflow seems more error prone to me.

Workaround

This is the simplest workaround I've found. Instead of the regular feat/fix/... commits, I create v0feat/v0fix/.... You could do this for all the commit types. I chose to just create three, v0feat, v0fix, v0breaking, and technically v0fix is not needed since it's the same as fix, but I liked the consistent naming. How you set this up, depends on what you'd like to bump what. Technically the only one you truly need is v0breaking so release notes show breaking changes without a major bump.

Now I keep my own semantic-release-config so this was easier for me to do. You will need to either configure certain plugins manually each time or I highly suggest creating your own config and abstracting everything away (no more need to install all the deps/plugins needed, your config can just do it for you).

How to do so or the various places you can place your config is out of scope for this post.

I'll just be showing you how to configure the plugins.

First, commit-analyzer. This will tell semantic-release how much to bump.

[
    "@semantic-release/commit-analyzer",
    {
        // you can use a preset here if you like
        "preset": "conventionalcommits",
        "releaseRules": [
            { "type": "v0feat", "release": "patch" },
            { "type": "v0fix", "release": "patch" },
            { "type": "v0breaking",  "release": "minor" }
        ]
    }
],

Then, release-notes-generator. This will tell semantic-release how to format the commit message headers. The docs for presetConfig are a bit hidden. You can find the full spec for them here

[
    "@semantic-release/release-notes-generator",
    {
        // note that if you use a preset, you will inherit it's options regarding what text counts as breaking changes
        "preset": "conventionalcommits",
        "presetConfig": {
            "types": [
                { "type":"v0feat", "section": ":star: New Features" },
                { "type": "v0fix", "section": ":bug: Fixes" },
                // Simulate a breaking change
                { "type": "v0breaking", "section": ":warning: BREAKING CHANGES" }
            ]
        }

    }
]

You can see the configs are pretty similar. In my config I create one shared object to pass as necessary to the plugins.

Usage

First you will need to create your first commit. Usually I will just create an empty commit.

git commit --allow-empty "v0feat: empty"

Now if you tried to run semantic release now, you would still get version v1.0.0.

You will need to tag the initial commit as v0.0.0 to trick semantic-release into working how we want.

git tag v0.0.0
# don't forget!
git push --tags

Then add your code and your real commit.

git add .
git commit "v0feat: initial"

Now you can do a dry run. Note you will need to push to your repo before doing so. So disable your release workflow if you have one. I personally have an env variable ENABLE_RELEASE I use to enable/disable, which has not been enabled yet at this point.

npx semantic-release --dry-run --no-ci #--debug

It should now print v0.0.1!

Now so long as you do not commit any commit types that trigger a major bump or use any text that triggers a breaking change in your commit messages (that's what v0breaking is for), you should stay in v0.

How to map Mod+Number keys in Vim/Neovim (using Wezterm)

Alan North — Wed, 26 Jul 2023 13:39:10 +0000

I've been recently giving neovim a serious try (this is my fourth/fifth attempt). And I think this time it's going to work out, but I really miss Modifier+Number shortcuts.

Usually you cannot map to modifies + number keys in in neovim (e.g. Ctrl+1). You can try to map the raw key sequences but this is a pain and does not work for all numbers.

The best way to get around this is to map to an obscure keybinding then make the terminal remap Mod+Number to those odd keybindings. I never got around to actually doing this usually because most terminal's configs are static it's also often a pain to remap keys (e.g. kitty, I like you, but...)

Also the ideal way to do this is to map them to a key chain like {ObscureCombo} {Mods}+{Remap} so as to not take up useful keys. But that's even more painful.

Recently though I tried and switched to wezterm for my terminal emulator, and it's configured through lua which makes this super easy to do for all sequences.

For example, I picked Ctrl+Alt+Shift+F12 for the obscure key combo and letters for the remap part. So Ctrl+1 would map to Ctrl+Alt+Shift+F12 Ctrl+q.

The letter mapping is to avoid breaking anything. If we mapped them to, for example, the F keys again, we might accidentally shoot ourselves in the foot when pressing Alt+4 .

Note that we also can't map the numbers without the modifiers, since wezterm has no way to know if we're in normal mode or not, and we would be left unable to type numbers if we did so.

Here's my wezterm function for remapping with all mods.

-- Allows using Mods+Number in the receiving program if it does not support it.
-- In the receiving program, the keymap needs to be `<C-A-S-F12> <Mods-F{q-p}>`
local remapModNumber = function(keyConfig, modsToRemap)
    -- it's not <Mods-F{Number}> for obvious reasons (i.e. A-F4)
    local numMap = {
        ["1"] = "q",
        ["2"] = "w",
        ["3"] = "e",
        ["4"] = "r",
        ["5"] = "t",
        ["6"] = "y",
        ["7"] = "u",
        ["8"] = "i",
        ["9"] = "o",
        ["0"] = "p",
    }

    local createModNumberRemap = function(key, mods)
        local res = {
            key = key,
            mods = mods,
            action = act.Multiple({
                --obscure key to use as base
                act.SendKey({ key = "F12", mods = "CTRL|SHIFT|ALT" }),
                act.SendKey({ key = numMap[key], mods = mods }),
            }),
        }
        return res
    end
    print(keyConfig)
    for i = 0, 9 do
        for _, mod in ipairs(modsToRemap) do
            -- cant insert normally???
            keyConfig[#keyConfig + 1] = createModNumberRemap(tostring(i), mod)
        end
    end
end
config.keys = {
    -- ...your other keybindings
}

remapModNumber(config.keys, { "CTRL", "SHIFT", "ALT", "CTRL|ALT", "ALT|SHIFT", "CTRL|SHIFT", "ALT|CTRL", "CTRL|SHIFT|ALT" })

Then in vim, if we wanted to use <C-1> we would map to <C-S-A-F12><C-q>.

I've also added a small helper function in my local utils module to help with this. I might extract this later into a plugin.

local M = {}
local config = {
    baseSequence = "<C-S-A-F12>",
}
M.setup = function(opts)
    opts = opts or {}
    vim.tbl_extend("force", config, opts)
end
local replacements = {
    ["1"] = "q",
    ["2"] = "w",
    ["3"] = "e",
    ["4"] = "r",
    ["5"] = "t",
    ["6"] = "y",
    ["7"] = "u",
    ["8"] = "i",
    ["9"] = "o",
    ["0"] = "p",
}
M.mapNum = function(keysString)
    local str = ""
    for i = 1, #keysString do
        local prevPrevChar = i > 2 and keysString:sub(i - 2, i - 2) or nil

        local prevChar = i > 1 and keysString:sub(i - 1, i - 1) or nil
        local char = keysString:sub(i, i)
        local isNumber = tonumber(char) ~= nil
        local prevIsNumber = prevChar ~= nil and tonumber(prevChar) ~= nil
        local prevIsAllowed = prevChar == nil or prevChar ~= "F" or prevIsNumber
        local prevPrevIsAllowed = prevPrevChar == nil or prevPrevChar ~= "F"
        if isNumber and prevIsAllowed and prevPrevIsAllowed then
            str = str .. replacements[char]
        else
            str = str .. char
        end
    end
    return config.baseSequence .. str
end

return M

It can then be used like so:

local mn = require("utils").mapNum

vim.keymap.set("n", mn("<C-1>"), ...)
-- will not remap F1
vim.keymap.set("n", mn("<C-F1>"), ...)

How to get a local module to actually load using lazy.nvim is out of scope for this post, but I might write one on that later because it was not very straightforward.