Forem: insidewhy

Using ast-grep with a vue project

insidewhy — Wed, 01 Jan 2025 04:10:58 +0000

Motivation

I've create a node package that supplies the necessary config to support vue/scss for ast-grep (with binaries built for multiple systems and architectures) via pusang-ina. The following article explains how to configure this manually.

The amazing ast-grep tool does not support vue and scss by default but can be configured to do so.

ast-grep is like a much more powerful version of ripgrep or grep when it comes to searching for patterns within code as it understands the structure of the code (via the library tree-sitter).

For example the following command:

ast-grep -p '<h2>$A</h2>'

Will show all <h2> tags in your html (and also vue files after applying the config from this post). Then the following could change all <h2> tags to <h3> tags interactively (each change will be shown and then y can be used to accept it or n to reject it):

ast-grep -p '<h2>$A</h2>' -r '<h3>$A</h3>' -i

-U can be used instead of -i to apply the changes without asking or neither can be used to view all the changes as a patch.

ast-grep also has some overlap with eslint, rules can be stored within the project along with fixes and these can be tested/fixed using ast-grep scan and ast-grep scan -U.

Simple Configuration

The steps in the next section can be used to setup ast-grep for vue manually or the following can be run:

pnpm add -D pusang-ina
ln -sf node_modules/pusang-ina/sgconfig.yml .

Manual Configuration

All of the following commands should be run from the root directory of your vue project.

First create a "rules" directory and add it to git. This directory must exist for the config to be accepted so for now just create an empty directory. Rules may also use utilities so that common config can be shared across rules, so we're going to create both directories with this config:

mkdir ast/{rules,utils}
touch ast/{rules,utils}/.keep-dir
git add ast/{rules,utils}/.keep-dir

Then create a file sgconfig.yml:

ruleDirs:
  - ast/rules
utilDirs:
  - ast/utils

customLanguages:
  vue:
    libraryPath: .tree-sitter/vue.so
    extensions: [vue]
    expandoChar: $
  scss:
    libraryPath: .tree-sitter/scss.so
    extensions: [scss]
    expandoChar: $

languageInjections:
  - hostLanguage: vue
    rule:
      pattern: <template>$CONTENT</template>
    injected: html
  - hostLanguage: vue
    rule:
      pattern: <script $$$ lang="ts" $$$>$CONTENT</script>
    injected: typescript
  - hostLanguage: vue
    rule:
      pattern: <style $$$ lang="scss" $$$>$CONTENT</style>
    injected: scss

Now a script is needed to create the libraries that this ast-grep configuration needs to understand vue and scss files. This could be created at scripts/init-ast-grep-config.sh:

#!/usr/bin/env bash

outdir=$PWD/.tree-sitter/
mkdir $outdir

cd /tmp
git clone https://github.com/tree-sitter-grammars/tree-sitter-vue
cd tree-sitter-vue
pnpm dlx tree-sitter-cli build
cp vue.so $outdir/
cd ..
rm -rf tree-sitter-vue

git clone https://github.com/serenadeai/tree-sitter-scss
cd tree-sitter-scss
pnpm dlx tree-sitter-cli build
cp scss.so $outdir/
cd ..
rm -rf tree-sitter-scss

Then ensure this script is executable:

chmod a+x scripts/init-ast-grep-config.sh

This script stores files in the directory .tree-sitter within the project, these libraries are binaries so they are unique to the platform on which they were compiled/installed so it's a good idea to add the .tree-sitter directory to .gitignore.

Next add a reference to this script to the scripts section of package.json:

{
  "scripts": {
    "init-ast-grep-config": "./scripts/init-ast-grep-config.sh"
  }
}

Now document this in your readme.md, commit everything to git, and developers on the project can begin using ast-grep on your vue project after running pnpm run init-ast-grep-config.

Add some rules

Now that ast-grep is configured I'll show how easy it is to create a rule to enforce code practices. vue allows defineEmits to be written in a type safe way:

defineEmits<{
  'update:value': [string]
}>

Or an unsafe way:

defineEmits(['update:value'])

If using typescript then there's not much reason to use the type unsafe version and this can be enforced with a five line rule in the project (to do the same with eslint would involve a lot more work). Create the following yaml file at ast/rules/type-unsafe-define-emits.yaml:

id: type-unsafe-define-emits
language: typescript
rule:
  pattern: defineEmits([$$$])
message: Use type safe version of defineEmits

Then run ast-grep scan and it will raise an error is the type unsafe version of defineEmits has been used.

Suggested Jira setup for software engineering projects

insidewhy — Thu, 21 Nov 2024 06:30:29 +0000

I've been refining the way I deal with software development workflows for more than 20 years now. I've worked in different positions of seniority, with team/organisation sizes ranging from minuscule to huge, and at various stages of the software development life cycle. This article represents the current state of evolution of the Jira configuration I use.

Team-managed vs company-managed projects

I advise choosing company-managed, team-managed projects offer much less customisability, in particular the scrum/kanban board layout is extremely limited and cannot be customised to the degree shown in the configuration suggested by this article.

Custom fields

The following fields are not part of Jira by default but can be useful for tracking important information. These fields don't create any additional maintenance burden as they are updated automatically by the workflow suggestion shown later in the article.

Developer: The Jira user associated with the person who developed the work item.
Tester: The Jira user associated with the person who tested the work item.
Start Time: The date and time the work item started being actively worked on (post refinement).
Development Complete Time: The date and time the work item was transitioned from In Review to Ready for QA or Done.
End Time: The date and time the work item was closed. There is already the resolution field which includes the resolutiondate component, but for some reason Jira does not show the resolution date anywhere in the UI and it can be very useful to see.

Workflow

Statuses

Draft

The initial state of any work item to show it hasn't been refined. After a refinement session:

Discard updates Status to Done and:
- Updates Resolution to Won't Do
- Updates Story Points to 0
Mark as duplicate updates Status to Done and:
- Updates Resolution to Duplicate
- Updates Story Points to 0
Finish refinement updates Status to To do
Finish refinement but blocked updates Status to Blocked

Blocked

A work item that has been refined but is blocked due to other pending work items.

Unblock updates Status to To do
Redraft updates Status to Draft

To do

The work item is ready to work on unless there is are work items related to it with an is blocked by association that are not yet Done. Generally issues with a status of To do won't have an Assignee but a person can set this if they want to claim the work item to show that they would like to be the one to develop it in the future.

Redraft reverts Status to Draft in case something was missed or life changed
Start updates Status to In Progress and:
- Updates Assignee to the user who performed the transition
- Copies Assignee over Developer
- Updates Start time to the current date and time

In Progress

Pause updates Status to To do
Request review updates Status to In Review and:
- Updates Ready For Review Time to the current date and time
Mark invalid updates Status to Done and:
- Updates Resolution to Won't Do
- Clears Ready for Review Time
- Clears Start Time
- Clears Development Complete Time
- Clears End Time
- Updates Story Points to 0

In Review

Merge code and skip testing can be useful for updating Status to Done when it doesn't require any testing. This is usually not a good idea but an escape hatch when life doesn't make sense. This transition also:
- Updates Resolution to Done
- Copies Developer over Tester to indicate that the developer tested their own work item (which is generally bad practice but can be useful in limited circumstances)
- Updates End Time to the current date and time
- Updates Development Complete Time to the current date and time
Merge code updates Status to Ready for review and:
- Clears Assignee. The Developer custom field will still be there for use later
- Updates Development Complete Time to the current date and time

Ready for QA

The work item is ready for QA and won't have an Assignee, but a person may set the Assignee to claim the work item to show that they would like to be the one to test the work item in the future.

Start testing updates Status to In testing and:
- Updates Assignee to the user who performed the transition.
- Copies Assignee over Tester
Request change updates Status to To do in case something was noticed before testing started, when this happens a comment should be left in the work item indicating what should change. This also:
- Copies Developer over Assignee

In testing

The work item is actively being tested.

Request change updates Status to To do, if this is done the tester should leave a comment in the work item to say what bugs were found or what changes should be made. It also:
- Copies Developer over Assignee
Pass testing updates Status to Done and:
- Updates Resolution to Done
- Copies Developer over Assignee. It's useful to quickly associated a closed work item with the person who developed it and the Tester custom field is still there for deep diving.
- Updates End time to the current date and time.

Done

The work item is most likely done with forever but may be reopened in case something comes up at a later date.

Reopen updates Status to To Do and:
- Clears Assignee
- Clears End time
- Clears Resolution which also has the effect of clearing the resolutiondate

Automations

Jira automations can be used to prevent naughty things:

Do not assign QA issues to developer

A developer shouldn't test their own code, any attempts to assign a work item in Ready for QA or In testing statuses to the user in the Developer custom field will revert the assignation.

Prevent Developer transitioning issue to "In Test"

A person shouldn't transition an issue to In testing when they were the one who developed it, any attempts to do this will be reverted. Here the Edit issue fields advanced option has the following content:

{
  "fields": {
    "Tester": null
  }
}

The board

The following filter creates a nice order for the kanban/scrum board:

project = YOURPROJECT
and (
  status != Done
  or resolution not in (Duplicate, "Cannot Reproduce", "Won't Do")
)
order by End Time desc, fixVersion asc, priority desc

Don't show issues which were discarded.
Order the work items in the Done column by the resolution date in descending order (i.e. most recently resolved work items at the top). End Time won't be set for work items without a status of Done (as enforced by the workflow) so this order by clause won't have any effect for work items in other columns.
For columns other than Done work items will be shown in order of their fixVersion (i.e. work items for the upcoming release will be shown at the top of the board), and within each release, work items will be ordered by priority.

Releases

Assigning work items to releases is a great way to keep track of progress of a particular release and for ordering the scrum/kanban board. Note that the releases list should be in descending date order (i.e. the most recent release should be at the top of the list and the oldest release should be at the bottom of the list). Releases can be dragged and dropped to reorder the list.

Filters

The following filters can be useful:

Open issues sorted by release then priority

project = MYPROJECT
and type != Epic and status != Done
order by fixVersion ASC, priority desc

Closed issues ordered by resolution date

project = MYPROJECT
and resolution not in (Duplicate, "Cannot Reproduce", "Won't Do")
order by resolutiondate

Issues in progress

project = MYPROJCT
and type != Epic
and status in ("In progress",  "In review", "In testing")
order by fixVersion ASC, priority desc

Issues for next release

project = MYPROJECT
and fixVersion = earliestUnreleasedVersion()
and (
  status != Done
  or resolution not in (Duplicate, "Cannot Reproduce", "Won't Do")
)
order by resolutiondate desc, fixVersion asc, priority desc

Issues to be tested

project = MYPROJECT
and type != Epic and status = "Ready for QA"
order by fixVersion ASC, priority desc

Github

If the project "short code" in Jira is PJ then issue identifiers will look like PJ-1, PJ-42 etc. When creating a it's useful to name the branch after the Jira work items with an optional prefix to show the type of work e.g.

fix/PJ-1: A fix relating to the issue PJ-1.
chore/PJ-2: A task relating to the issue PJ-2.
feature/PJ-3: A feature/story relating to the issue PJ-3.
PJ-4: If you don't care to mark the task type in the branch name.

Assuming this convention has been used (or that the PR title begins with PJ-42:) a github action can be used to:

Set the pull request title to the Jira work item title.
Add a text block to the pull request description with a link to the Jira work item (this block can be updated when the Jira content changes as updating text blocks is a supported feature of insidewhy/actions-body-fields).

To do this the following file can be created within the github repository at a file location such as .github/workflows/jira-linker.yml:

name: jira-linker

on:
  pull_request:
    types:
      - opened
      - reopened

concurrency:
  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
  cancel-in-progress: true

jobs:
  action-jira-linker:
    runs-on: ubuntu-latest

    permissions:
      pull-requests: write
      issues: write

    steps:
      - name: determine associated issue id
        id: get-ticket-id
        run: |
          ticket_match='grep -Eqx [A-Z]+-[0-9]+'

          # first try to get the ticket id from the title
          title="${{github.event.pull_request.title}}"
          ticket_id=${title%%:*}
          if ! echo $ticket_id | $ticket_match; then
            # otherwise try to get the ticket id from the branch
            branch=${{ github.head_ref || github.ref_name }}
            ticket_id=${branch#*/}
            ticket_id=${ticket_id%%_*}
            if ! echo $ticket_id | $ticket_match; then
              exit 0
            fi
          fi
          echo "ticket-id=$ticket_id" >> $GITHUB_OUTPUT

      - uses: insidewhy/action-get-jira-issue@v1
        id: jira
        if: steps.get-ticket-id.outputs.ticket-id
        with:
          user: ${{ secrets.JIRA_USER }}
          token: ${{ secrets.JIRA_TOKEN }}
          base-url: ${{ secrets.JIRA_BASE_URL }}
          ticket-id: ${{ steps.get-ticket-id.outputs.ticket-id }}

      - uses: insidewhy/action-body-fields@v1
        if: steps.get-ticket-id.outputs.ticket-id
        with:
          prepend: true
          title: '${{ steps.get-ticket-id.outputs.ticket-id }}: ${{ steps.jira.outputs.summary }}'
          header: '## Status'
          fields: |
            Jira: [${{ steps.get-ticket-id.outputs.ticket-id }}: ${{ steps.jira.outputs.summary }}](${{ steps.jira.outputs.link }})

For this to work a Jira token must be created, then information about this token must be set via github secrets:

JIRA_USER: The user that created the Jira token.
JIRA_TOKEN: The token itself.
JIRA_BASE_URL: The URL of the Jira instance e.g. https://my-kitten-project.atlassian.net

Installing kde 6 (plasma 6) from source on ubuntu 24.04

insidewhy — Sat, 20 Apr 2024 20:06:32 +0000

Install qt 6

Install the latest qt 6.6 releases using qt online installer. Just select qt 6.6, no other packages are needed. I chose to install it to /opt/qt and created this directory with sudo mkdir /opt/qt && chown $USER /opt/qt. Then put something like /opt/qt/<qt-version>/gcc_64/bin to the head of the PATH environment variable via ~/.zshrc or ~/.bashrc (for me <qt-version> was 6.6.3, the latest 6.6 release at the time I wrote this).

Setup `pkg-config`

Ensure pkg-config can find the qt libraries installed above by adding /opt/qt/<qt-version>/gcc_64/lib/pkgconfig to PKG_CONFIG_PATH via your shell startup file. If you use brew then it's probably installed its own pkg-config which is now at the head of your $PATH. In this case ensure it can find the system pkgconfig files by also adding /usr/lib/x86_64-linux-gnu/pkgconfig:/usr/share/pkgconfig to PKG_CONFIG_PATH.

Something like this should be okay:

if [ -d /home/linuxbrew ]; then
  export PKG_CONFIG_PATH=/home/linuxbrew/.linuxbrew/lib/pkgconfig:/usr/lib/x86_64-linux-gnu/pkgconfig:/usr/share/pkgconfig:$PKG_CONFIG_PATH
fi

QT6DIR=/opt/qt/6.6.3/gcc_64
if [ -d $QT6DIR/bin ]; then
  # for zsh:
  path=($QT6DIR/bin $path)
  # for bash:
  # export PATH="$QT6DIR/bin:$PATH"
  export PKG_CONFIG_PATH=$QT6DIR/lib/pkgconfig:$PKG_CONFIG_PATH
fi

Install kde-builder

Follow the initial instructions here at the Generic Installation section only.

The kde-builder --install-distro-packages doesn't seem much use as:

The qt6 package versions it installed are out of date.
It doesn't install most of the required dependencies.

I did run it and then manually uninstalled the qt6 dependencies since qt6 was already installed and configured using the steps above.

Then run

kde-builder --generate-config

After this I edited ~/.config/kdesrc-buildrc to install to /opt/plasma6 by changing the line starting install-dir:

install-dir /opt/plasma6

And created this directory with:

sudo mkdir /opt/plasma6
sudo chown $USER /opt/plasma6

Add some unavailable python dependencies

Create a directory to store the dependencies:

sudo mkdir /opt/python
sudo chown $USER /opt/python

Add this path to PYTHONPATH in your shell config:

if [ -d /opt/python ]; then
  export PYTHONPATH=/opt/python/local/lib/python$(python3 --version | grep -o '3.[0-9]*')/dist-packages
fi

Then install them:

pip install --prefix /opt/python chai

Be careful that brew has not installed python otherwise you'll have something like python3 pointing to your system python and python3.12 pointing to brew's version, then the build scripts will detect brew's version and not see your system packages. I just uninstalled brew's version.

Start installing

kde-builder workspace

This will inevitably fail many times and give the location of a log that indicates the error, which is usually a missing package. When installing that package use one with a -dev suffix.

After installing the package run:

kde-builder --resume-from <name of package that failed> workspace

To resume building from where it left off.

I needed the following additional packages:

sudo apt install libxkbcommon-dev libpolkit-gobject-1-dev libpolkit-agent-1-dev libxcb-damage0-dev libyaml-dev libsystemd-dev libstemmer-dev itstool libpcap-dev libsensors-dev libxcb-dpms0-dev libpipewire-0.3-dev libxcb-xtest0-dev libpam0g-dev libeis-dev libxcvt-dev liblcms2-dev libopenjp2-7 libappimage-dev libattr1-dev libnm-dev libqrencode-dev libdmtx-dev libzxing-dev libqalculate-dev libxcursor-dev libxtst-dev libpulse-dev modemmanager-dev python3-cairo libgtk-3-dev libxxf86vm-dev wl-clipboard libaccounts-glib-dev libsdl2-dev libxkbregistry-dev xserver-xorg-input-libinput-dev xserver-xorg-input-evdev-dev xorg-dev

I also had to install a perl cpan module:

cpan install URI::Escape

This worked for me because I had perl installed via brew (I haven't used perl for over 20 years, I guess it was a dependency of a package I did use), otherwise sudo may be necessary at the risk of polluting your /usr directory with files not managed by apt

kaccounts seems to be missing from the dependency list of plasma-desktop so at some point I had to install this manually via:

kde-builder kaccounts

Before resuming:

kde-builder --resume-from plasma-desktop workspace

Create a startup script

Create the following file at /opt/plasma6/bin/start-kde6:

#!/usr/bin/env bash

# replace $USER with the location of the user's home directory
# who built KDE6
source $USER/.config/kde-env-master.sh

if [[ -z "$DBUS_SESSION_BUS_ADDRESS" ]]; then
  exec dbus-run-session startplasma-wayland
else
  exec startplasma-wayland
fi

Ensure it is executable:

chmod a+x /opt/plasma6/bin/start-kde6

Add desktop entry to `sddm`

Create the following file:

sudo touch /usr/share/wayland-sessions/plasmawayland6.desktop
sudo chown $USER /usr/share/wayland-sessions/plasmawayland6.desktop

Then add the following content:

[Desktop Entry]
Exec=/opt/plasma6/bin/start-kde6
DesktopNames=KDE6
Name=Plasma 6 (Wayland)
Comment=Plasma 6 by KDE
X-KDE-PluginInfo-Version=6.0.6

Start kde6

Logout of your window manager and select the plasma 6 entry from sddm and see if it starts.

For me it didn't.

I got a black screen, the mouse cursor could be moved. Using my krunner keybinding I noticed some glitches at the top of the screen where krunner would usually appear. I could switch to another virtual console via ctrl+alt+f3 to restart sddm, after this trying to start plasma6 again froze my computer and I had to reboot it. When I switched back to kde5, most of my keybindings were gone. Luckily I backed them up several kde config files before switching over and could restore them.

When I was in the virtual console and plasma6 was running, I noticed it had started /usr/bin/plasmashell instead of /opt/plasma6/bin/plasmashell. Disappointing. I verified that PATH and XDG_CONFIG_DIRS were set properly by looking at the environment of the startplasma-wayland process in the /proc filesystem, so why it's still choosing /usr/bin/plasmashell I don't know.

Some of my code is in KDE, for the notification system and scripting system. I'd love to be able to contribute again. The matrix chat for KDE is not open for registration so I'm not sure how I can get help to solve this problem.

I'm using the nvidia-drivers currently at version 550. Wayland on plasma5 is okay, but copy and paste is not reliable. Especially in slack. Sometimes it works for a while and then breaks forever. To paste into my terminal I usually need to hit paste several times for the very first paste into that terminal, after which point it works.

A sequential work queue for typescript in 11 lines of code

insidewhy — Mon, 20 Feb 2023 08:06:07 +0000

class WorkQueue<T> {
  private lastJob: Promise<T> | undefined

  async queueWork(work: () => Promise<T>): Promise<T> {
    const nextJob = this.lastJob = this.lastJob?.then(work, _ => {}) ?? work()
    try {
      return await nextJob
    } finally {
      // ensure previous results can be garbage collected
      if (nextJob === this.lastJob) { this.lastJob = undefined }
    }
  }
}

Okay maybe you should use a few more lines to be clearer but I like how neat this is.

A nice work queue for TypeScript/JavaScript

insidewhy — Wed, 11 Jan 2023 11:35:58 +0000

This posts is about an open source work queue library I wrote for processing data at my current job: horse-sparkle.

Allows adding work to a queue to be processed one at a time and in order.
Puts work to the back of the queue when it fails.
Resumes work from where it failed rather than having to restart it from scratch (via the power of async iterators).
Allows limiting the size of the queue to avoid unbounded memory usage:
- Context is associated with each piece of work, after failure the memory associated with the work is reclaimed but the context can be used to recreate it from scratch.
- Overflows are consumed from an async generator, the user may recreate the work from the associated context when the queue is empty, throw it away, or perform any other action they want.
Can customise the behaviour which occurs when a failure is detected:
- The queue awaits the error handler which can be used to add delays on failure.
- The consecutive failure count is passed to the error handler which may be used for exponential backoff.

horse-sparkle is written in TypeScript and the project includes extensive unit testing.

Documentation

Basic usage

It can be useful to create a queue of work, and on error, it may not be great to restart the entire piece of work from the beginning. horse-sparkle enables this pattern:

import { yieldErrors, WorkQueue, WorkIterator } from 'horse-sparkle'

async function* doWork(dbId: string): WorkIterator {
  const largeObject = yield* yieldErrors(() => grabLargeObjectFromDatabase(dbId))
  const bigResult = yield* yieldErrors(() => doExpensiveProcessingOnObject(largeObject))
  yield* yieldErrors(() => doFinalProcessingOfBigResult(bigResult))
}

// Here the <string> generic should be the context parameter, we will see how
// this is used later
const queue = new WorkQueue<string>()
queue.start()

function doWorkInTurn(dbId: string): void {
  // The dbId relates to the <string> generic used above
  queue.queueWork(dbId, doWork(dbId))
}

The function yieldErrors will continuously run the function passed to it until it does not throw an error. Any thrown error will be yielded by the generator, this causes the work queue to suspend the task. The work queue will then push the piece of work to the back of the queue and once the work is back at the head of the queue it will resume the task from where it left off e.g. If largeObject was retrieved and bigResult was calculated, but then doFinalProcessingOfBigResult threw an error, when the work is tried the next time it will not have to calculate bigResult again, instead the work will resume with another attempt to call doFinalProcessingOfBigResult.

Error behaviour

In the previous example the work queue will continuously process information without delay. In many instances this would not be good, for example a database may be down and it would not be desirable to continuously attempt to access it. This can be solved using the onError parameter to WorkQueue:

const queue = new WorkQueue({
  onError: (error: Error | undefined, consecutiveErrorCount: number): Promise<void> => {
    console.error("Work queue error", error)
    delay(Math.min(2 ** (consecutiveErrorCount - 1) * 100, 400)),
  }
})

The return value of the onError handler is awaited before processing the next queue item, with the function used above the first delay would be 100ms, the second would be 200ms and subsequent delays would be 400ms. The consecutiveErrorCount is reset to 1 after a piece of work successfully completes, so an error received after a successful completion would cause a 100ms delay.

Limiting the queue length

In the first example the object bigResult may take up significant amount of memory. If the size of the work queue was allowed to grow indefinitely and many failures were detected this could result in memory running out. The WorkQueue accepts a maxQueueLength parameter for dealing with this.

const queue = new WorkQueue({ maxQueueLength: 100 })

However it would probably not be useful to completely throw the work away, instead it may be good to reclaim the memory associated with the async iterator and create the work again from scratch later. The context parameter can then be used to do this, it will be more expensive to repeat the processing but at least the memory usage will have a maximum limit. To facilitate this WorkQueue provides a method overflows that returns an async iterator which yields the context related with each piece of work that was thrown out of the queue due to an overflow. The waitForSpaceInWorkQueue method can be used to asynchronously wait for the queue to have space in order to requeue the work if desired.
Repeating the example above:

import { yieldErrors, WorkQueue, WorkIterator } from 'horse-sparkle'

async function* doWork(dbId: string): WorkIterator {
  let largeObject: Massive | undefined = yield* yieldErrors(() =>
    grabLargeObjectFromDatabase(dbId)
  )
  const bigResult = yield* yieldErrors(() =>
    doExpensiveProcessingOnObject(largeObject)
  )
  // reclaim memory from largeObject now it is no longer needed
  largeObject = undefined
  yield* yieldErrors(() =>
    doFinalProcessingOfBigResult(bigResult)
  )
}

const queue = new WorkQueue<string>()

function doWorkInTurn(dbId: string): void {
  queue.queueWork(dbId, doWork(dbId))
}

async function* overflowHandler(): Promise<void> => {
  for await (const dbId of queue.overflows()) {
    // this is one suggestion enabled by the API in which the work is eventually
    // pushed to the back of the queue to be resumed from scratch
    await queue.waitForSpaceInWorkQueue()
    queue.queueWork(dbId, doWork(dbId))
  }
}

function startQueue(): Promise<void> {
  return Promise.all([
    queue.start(),
    overflowHandler(),
  ])
}

Stopping the queue

The stop method may be called on the queue to stop processing. This return a promise which resolves after the current piece of asynchronous work performed by the queue succeeds or fails. The promise returned by the start method also resolves at this point, and any async iterator created by the overflows method will be completed.

Advent of code 2022 - each solution in a different language

insidewhy — Thu, 29 Dec 2022 07:16:41 +0000

I added my solutions to a github repo. I've done days 1-11 so far. I'll blog about the rest in a follow up post once I get around to them.

ruby

The first problem was very easy, in each case two lines of code. There's a lot I like about Ruby but I don't like the implementation of the typesystem that was added in 2020. In order to add types all classes/functions etc. must be repeated in a second file where types are allowed.

I don't like having to switch back and forth between my code and the type definition file to lookup types, types are such a useful part of the documentation and the code itself that I feel like they should be colocated.
"Repeat yourself twice" instead of DRY is kind of sad for a language that is usually expressive.

crystal

Crystal is a compiled version of Ruby where the types are colocated, it's not 100% compatible but close enough.

I'd love to be able to use this language more but it doesn't seem to be getting much attention.
I wish I could use const variables and mark data as immutable.
The compiler is a little slow.

TypeScript

I use TypeScript more than any other language in the last few years so it was easy to write this without having to reference any documentation. The problem wasn't very hard but there were two small areas where you could use modular arithmetic in a fun way. I checked some other solutions online and many people just used a lookup table where looked up each line from the file against a score table and this works for both parts since there are only six combinations of hands.

c++

C++ gets a lot of hate for being too complicated and for baggage it inherited from C. On the other hand a lot of people dislike the more recent versions of C++ and only want to use classic C++, the version where it's hard to avoid that baggage. I wonder if the most pragmatic solution to keep things simple is to take the opposite approach to both these camps and disregard earlier features rather than modern features. The solution here is quite neat and performs very well.

dart

Dart is a very conservative language so solutions tend to be quite verbose. This conservatism extends to the typesystem so it can be harder to write typesafe programs without resorting to repetition that can be avoided in other languages. TypeScript is a lot more expressive in this respect. I find it a little joyless to use dart, both for this solution and in my day job where I occasionally write flutter apps.

go

Not much to say about go that's different from dart, I don't like writing go so much as it feels very soulless and a little patronising. It feels like a language that was written in which to achieve day jobs, the fun you get is from solving business problems and not from the joy of writing code itself. I'm not really sure if the verbosity of the code needed to achieve simple things really is a boon for maintainable software. Surely there's a balance between having to write a lot of boilerplate and writing understandable code? Error handling in go is a PAIN. Rust doesn't have exceptions either but they added some nice syntax to the language for dealing with errors.

haskell

Haskell is possibly the opposite of dart/go, the expressiveness of the language is off the charts. I'm not sure if it's a pragmatic choice for most of the world, Haskell code can be impenetrable even to the people that wrote it. There's a fine balance between simplicity, verbosity and how fun a language is to use and that depends a lot on your own experience and personality. For me it's somewhere closer to haskell than go but still very far away from either.

scala

I had so much fun with this one. Maybe scala strikes the expressiveness/readability balance for me. There were a lot of instances where I was really impressed at how readable and tersely I could express things.

php

I was kind of interested to see how far php had come and ended up disappointed. The typechecking happens at runtime rather than via static analysis!?!?!? As usual, the designers of php seem to want things from a programming language that are completely foreign to me. The way constants are defined is still annoying. The amount of $ symbols in php code is still annoying. They took a lot of nice things from JavaScript and TypeScript that make it more expressive than it used to be but I still strongly dislike it.

rust

Rust is pretty fun, I've used it a little in the past also. The borrow checker is cool but it might really annoy you until you understand it well. The macro system is very fun and the shortcuts for error handling are very nice. It forced me to use integers to index vectors instead of iterators which made some of the code less readable than it could be, but maybe it's worth it to have code that is guaranteed to be safe.

raku

raku is the successor to perl 5 which I learned when I was 16 (after Metacomco basic on the Atari ST and bash). I haven't used perl in about 15 years and I think it was helpful having some distance because, while raku is reminiscent of perl 5, it's unmistakeably a very different language. I thought it would be fun to use it for solution 11 as it requires a lot of parsing and raku has the ability to define grammars built into the language (it operates much like a very advanced version of regexes). There were things I loved about raku and things that confounded me. The typesystem is a little limiting (no variants, no parametric types). The difference between constructors and the BUILD method was also a little frustrating, you can only assign to private attributes in the latter but this involves duplicating the names of attributes you want to assign directly from arguments. The amount of sigils in the code also makes it a little difficult to type; they call them twigils in the documentation. A lot of stuff in raku works (or is named, or is expressed) very differently than you might be used to. It was fun to learn and use but definitely the language where I hit the most unexpected results and that required the most learning and research. It's doubtful I'll get to use it again in the future due to its niche popularity. One thing I really liked about part 2 of this solution is that it relies on your knowledge of modular arithmetic in order to produce a solution that runs in a reasonable length of time. Without applying this then you'll really start hitting the performance limits of big integers and the code will take several days to provide a solution.

An excellent way to deal with (lerna) monorepos in CircleCI: circletron

insidewhy — Fri, 25 Jun 2021 11:25:23 +0000

We started with a monorepo at my current company and have been using circle almost since the beginning. It was tough. It required a lot of boilerplate in our config and the necessity to get every developer to generate a circle API key and add it to git. It never felt that great but at least it worked. In April CircleCI released the dynamic configuration API and this allowed us to refactor our monorepo support into something we think is pretty great. We've gone from tolerating Circle to enjoying it, and now we've released the code as an open source project and provided the functionality as an orb so that any project can benefit.

The project is circletron, the code is hosted on github and the orb is here. It currently supports lerna monorepos but we plan to support other monorepos in the future.

Motivating example

Typically a circleci configuration exists in a single file within a repository at .circleci/config.yml. For a monorepo a very minimal example may look something like this:

version: 2.1

jobs:
  validate-everything:
    steps:
      - checkout
      - run: npm run general-validation

  test-subpackage-a:
    steps:
      - checkout
      - run: cd packages/package-a && npm run test

  test-subpackage-b:
    steps:
      - checkout
      - run: cd packages/package-b && npm run test

  test-subpackage-c:
    steps:
      - checkout
      - run: cd packages/package-c && npm run test

  publish-subpackage-c:
    steps:
      - checkout
      - run: cd packages/package-c && npm run publish


workflows:
  validate-everything:
    jobs:
      - validate-everything

  subpackage-a:
    jobs:
      - test-subpackage-a

  subpackage-b:
    jobs:
      - test-subpackage-b

  subpackage-c:
    jobs:
      - test-subpackage-c
      - publish-subpackage-c

Obviously there are many problems with this, for a start all the CI is defined in one file at the root of the project. Worse, each of the jobs may take some time to complete and on every commit to the project, circleci will run every single job no matter whether there are any changes to the respective subpackages or not. It may not look so bad in this tiny example but the bigger the configuration gets, the worse it is to deal with.

Using circletron to solve this

With circletron the .circle/config.yml is always the same:

version: 2.1
setup: true
orbs:
  circletron: circletron/circletron@3.0.1

workflows:
  trigger-jobs:
    jobs:
      - circletron/trigger-jobs

The trigger job step will take many individual circle.yml distributed within the project and combine them into a single configuration which will be issued via the continuation API. It will modify the configuration to ensure that jobs that are not necessary are no longer run, in a way that is friendly to CI branch protection rules.

The single configuration file can now be split up across the monorepo, with one optional circle.yml in the root of the project and where the CI for each subpackage lives in the directory for that subpackage.

In this instance the circle.yml in the root of the project will host configuration not specific to any subpackage:

version: 2.1

jobs:
  validate-everything:
    steps:
      - checkout
      - run: npm run general-validation

workflows:
  validate-everything:
    jobs:
      - validate-everything

The jobs here are run on every commit and it's also a good place to set version. It's also a good place to provide commands that can be used in subpackage specific circle.yml files.

Now lets look at packages/package-a/circle.yml:

jobs:
  test-subpackage-a:
    steps:
      - checkout
      - run: cd packages/package-a && npm run test

workflows:
  subpackage-a:
    jobs:
      - test-subpackage-a

Everything related to this package all in one place. Even better, when the PR contains no changes within packages/package-a the jobs for this subpackage will be skipped. You will probably want to use branch protection rules to ensure that when the test-subpackage-a job fails the PR will be blocked so omitting this job entirely would not be ideal. Omitted jobs remain in pending state permanently, blocking the PR. circletron replaces unneeded jobs with a simple job using the busybox:stable docker image that echoes "Job not required" and issues a success error code.

Dependencies and unconditional jobs

What if the code in packages/subpackage-c uses code from packages/subpackage-a? In this case the jobs within this package should run when the code in subpackage-a changes, even if the code in the subpackage itself doesn't change.

There may also be some instances where a job should run on every push.

The configuration at packages/subpackage-c/circle.yml shows how to add dependencies and create jobs which run unconditionally:

dependencies:
  - subpackage-a

jobs:
  test-subpackage-c:
    steps:
      - checkout
      - run: cd packages/package-b && npm run test

  publish-subpackage-c:
    conditional: false
    steps:
      - checkout
      - run: cd packages/package-c && npm run publish

workflows:
  subpackage-c:
    jobs:
      - test-subpackage-c
      - publish-subpackage-a

In this instance test-subpackage-c will only be run when changes to package-a or package-c are detected and publish-subpackage-c will run on every push.

Target branches

CircleCI does not pass the target branch to workflows or jobs so it becomes necessary to help circletron out. By default circletron considers the branches main, master, develop and any branch starting with release/ as a target branch. The latest commit from the branch commit history which belongs to one of these branches is considered to be the branch-point. For pushes to one of the branches above, all of the jobs are run. This can be changed via the configuration file at .circleci/circletron.yml:

targetBranches: ^(release/|main$|master$|develop$)

Any branch which matches this regex is considered a target branch, the above configuration shows the default.

circletron is available to use now.

How to deal with the first few months of a new job writing software

insidewhy — Tue, 01 Jun 2021 18:06:28 +0000

Read git log and see how people write their commits. Are they all in imperative mood? Then write your commits in imperative mood. Do they start with a capital letter? Do the same. Include a cat emoji after every word? Now you're stuck with that too. But what if every commit message follows a completely different style? Then you're lucky, you have a chance to advocate for a standard, to be recognised as someone who's trying to bring some order to chaos.

The next thing you could avoid if you were a master of Zen, but you are not, so you're going to panic for at least two months. There is going to be horrible crazy code and horrible design and you're going to question what crazy incompetent programmers could commit such crimes against code. But recognise that there are probably good reasons for most of these horrible things you see, and realise that panic for at least the first few months in a new job is a completely normal and understandable reaction. Instead of criticising, ask questions. When you get the answers you'll often find that the crazy design idea was actually a reasonable and pragmatic work-around. Or that the other developers around you have recognised the same code debt and also have a plan to fix it. If you get defensive and criticise the code you stand the chance of making enemies but if you ask good questions there's a good chance people will recognise and appreciate your ideas and feedback.

You're going to get code reviews, the second time you get a change request for the same type of issue you've already received a change request for in the past, it's probably going to stick in your head. Write it down somewhere so you don't do the same thing a third time. Being caught repetitively making the same "mistake" might be seen as a lack of attention to detail.

Make sure you respond to all change requests in your PR reviews. If you're using GitHub, then see the next tip, if not then maybe you'll have to keep track of this yourself and make sure you go through all your pull requests each morning manually. Respond to every single change request you get, even if it's just to add an "eyes" emoji or a thumbs up. If you disagree with a change request, respond with a comment and keep following up until you reach consensus. Everyone has worked a developer who says "I have nothing to do" despite having multiple change requests open against their pull requests. It's very frustrating. Every change request is "something to do", sometimes you might have to be proactive in following up your change requests to reach a resolution but that's one of the most important parts of a development job.

If your company is using GitHub then before you do anything else, learn how GitHub notifications work. With an adequate understanding of this you'll never miss another piece of information relating to your code again, then all you need to worry about is how to keep up with your project management tool.

Update your issue statuses in your project management tool at least once every morning. Leaving code that's been merged in In Progress state can be tolerate for up to a day, but any more than this looks lazy. 90% of your communication might be with other developers but advocating for, and facilitating transparency amongst the organisation will be recognised beyond your team.

Filtering the types of a tuple at compile time in typescript

insidewhy — Sat, 18 Apr 2020 17:26:56 +0000

I needed to remove all of the boolean types from a tuple for a parser generator I'm writing. The code turned out to be way crazier than I would have liked:

type NumberMap = {
  0: 1
  1: 2
  2: 3
  3: 4
  4: 5
  5: 6
  6: 7
  7: 8
  8: 9
  9: 10
  10: 11
  11: 12
  // up to twelve supported
  12: 12
}

// prepending is relatively easy
type Prepend<H, T extends any[]> = ((h: H, ...t: T) => void) extends (
  ...l: infer L
) => void
  ? L
  : never

// appending is possible but expensive and hard, must
// build lists in reverse and reverse the result when done
type Reverse<L extends any[], R extends any[] = []> = {
  0: R
  1: ((...l: L) => void) extends (h: infer H, ...t: infer T) => void
    ? Reverse<T, Prepend<H, R>>
    : never
}[L extends [any, ...any[]] ? 1 : 0]

type Equals<I extends number, L extends number> = I extends L ? 1 : 0

type FilterBoolean<T, R extends any[]> = T extends boolean ? R : Prepend<T, R>

type FilterBooleansNext<
  I extends keyof NumberMap,
  T extends any[],
  R extends any[]
> = T extends []
  ? R
  : {
      0: FilterBooleansNext<NumberMap[I], T, FilterBoolean<T[I], R>>
      1: R
    }[Equals<I, T['length']>]

// append is hard/expensive, so keep prepending and reverse the result
export type FilterBooleans<T extends any[]> = Reverse<
  FilterBooleansNext<0, T, []>
>

Oh boy. Going through it bit by bit:

type NumberMap = {
  0: 1
  1: 2
  2: 3
  3: 4
  4: 5
  5: 6
  6: 7
  7: 8
  8: 9
  9: 10
  10: 11
  11: 12
  // up to twelve supported
  12: 12
}

Can't do I + 1 in a typescript type. Nope. So instead there is this map. The last map item has to point to itself or the code won't compile, because even when no tuples with 12 items are used, the code will still instantiate up the limit and then complain about item 13 not existing.

type Prepend<H, T extends any[]> = ((h: H, ...t: T) => void) extends (
  ...l: infer L
) => void
  ? L
  : never

This prepends a type to the head of a list. It's a bit disappointing that it's necessary to use a function just to infer the prepended type but that's how typescript is. This is used to build the "result type". You're probably thinking, isn't it better to append types gradually when building the result? Well yeah, but writing an append type turns out to be far more complicated and much more expensive, it involves reversing the list twice. No way, this metaprogram already uses up a ridiculous amount of CPU.

type Equals<I extends number, L extends number> = I extends L ? 1 : 0

type FilterBoolean<T, R extends any[]> = T extends boolean ? R : Prepend<T, R>

type FilterBooleansNext<
  I extends keyof NumberMap,
  T extends any[],
  R extends any[]
> = T extends []
  ? R
  : {
      0: FilterBooleansNext<NumberMap[I], T, FilterBoolean<T[I], R>>
      1: R
    }[Equals<I, T['length']>]

export type FilterBooleans<T extends any[]> = Reverse<
  FilterBooleansNext<0, T, []>
>

Oh dear, why the object and why the Equals? Why not just I extends T['length'] ? ... : .... Well that introduces a recursive reference from FilterBooleansNext back to itself that typescript just won't tolerate. But it will tolerate this hack. Now the final bit:

export type FilterBooleans<T extends any[]> = Reverse<
  FilterBooleansNext<0, T, []>
>

Remember I said appending was too hard, so we keep pretending types to the result and reverse it at the end.

The result of all this? While I'm editing my project with this type one of my CPU cores is constantly at 100%. The compile time of my program doubled from 3 seconds to 6. I threw it away and my API suffers for it because now the user has to manually specify types when they shouldn't have to.