Forem: Luke Miles

How to serve deepseek r1 & v3 on GH200s (400 tok/sec throughput, 10 tok/sec/query)

Luke Miles — Tue, 11 Feb 2025 03:52:30 +0000

Lambda labs still has half-off (ie $1.50) GH200s as a promotion to get more people used to the ARM tooling. I previously wrote a tutorial for running llama 405b, but deepseek r1 is clearly a better model, so let's serve it instead. There's a couple differences from serving llama:

You need 12 or 16 GPUs to have decent throughput. You can still use 8 if you don't need much throughput.
VLLM works better than aphrodite for deepseek right now. In fact they just released an update that makes deepseek inference quite a bit faster. Roughly a 40% throughput improvement.
- Normally a 16x 1x gpu cluster would be pretty slow for inference but for some reason this actually works pretty well.

Guide will be updated on the github repo.

Video showing inference speed with 64 parallel queries. It prints once per second and you get at least 5 new words every time it prints:

https://github.com/user-attachments/assets/5c27abe9-9168-4f6a-88aa-8ffa22c348b1

Create instances

This time, instead of using the website, let's use the API to make the instances.

Make a filesystem called 'shared' in Washington, DC and write down the name of your ssh key. Also generate an api key and save it.

export sshkey_name=my-key
export shared_fs_name=shared
export LAMBDA_API_KEY="..."

This script will make 16 GH200 instances. Ignore the rate limit errors, it will keep retrying until 16 have been made.

function lambda-api {
    local method=$1
    local route=$2
    shift 2
    curl --fail -q -X "$method" \
        -H "Authorization: Bearer $LAMBDA_API_KEY" \
        "https://cloud.lambdalabs.com/api/v1/$route" \
        -H "Content-Type: application/json" \
        "$@"
}
export -f lambda-api

num_want=16
num_got=0
while [[ $num_got -lt $num_want ]]; do
    lambda-api POST instance-operations/launch -d '{
        "region_name": "us-east-3",
        "instance_type_name": "gpu_1x_gh200",
        "sshkey_names": ["'$sshkey_name'"],
        "file_system_names": ["'$shared_fs_name'"],
        "quantity": 1,
        "name": "node_'$num_got'"
    }' && ((num_got++))
    echo num_got=$num_got
    sleep 3
done

After all the instances are created, copy all the IP addresses from the instances page and save it to ~/ips.txt.

Bulk ssh connection helpers

I prefer direct bash & ssh over anything fancy like kubernetes or slurm. It's manageable with some helpers.

# save all the ssh fingerprints now to skip confirmation later
for ip in $(cat ~/ips.txt); do
    echo "doing $ip"
    ssh-keyscan $ip >> ~/.ssh/known_hosts
done

export runprefix=""
function runip() {
    ssh -i ~/.ssh/lambda_id_ed25519 ubuntu@$ip -- stdbuf -oL -eL bash -l -c "$(printf "%q" "$runprefix""$*")" < /dev/null
}
function runk() { ip=$(sed -n "$((k + 1))"p ~/ips.txt) runip "$@"; }
function runhead() { ip="$(head -n1 ~/ips.txt)" runip "$@"; }
function runips() {
    local pids=()
    for ip in $ips; do
        ip=$ip runip "$@" |& sed "s/^/$ip\t /" &
        pids+=($!)
    done
    wait "${pids[@]}" &>/dev/null
}
function runall() { ips="$(cat ~/ips.txt)" runips "$@"; }
function runrest() { ips="$(tail -n+2 ~/ips.txt)" runips "$@"; }

function sshk() {
    ip=$(sed -n "$((k + 1))"p ~/ips.txt)
    ssh -i ~/.ssh/lambda_id_ed25519 ubuntu@$ip
}
alias ssh_head='k=0 sshk'

function killall() {
    pkill -ife 192.222
}

Let's check that it works

runall echo ok

Set up NFS cache

We'll be putting the python environment in the NFS. It will load much faster if we cache it.

# First, check the NFS works.
# runall ln -s my_other_fs_name shared
runhead 'echo world > shared/hello'
runall cat shared/hello


# Install and enable cachefilesd
runall sudo apt-get update
runall sudo apt-get install -y cachefilesd
runall "echo '
RUN=yes
CACHE_TAG=mycache
CACHE_BACKEND=Path=/var/cache/fscache
CACHEFS_RECLAIM=0
' | sudo tee -a /etc/default/cachefilesd"
runall sudo systemctl restart cachefilesd
runall 'sudo journalctl -u cachefilesd | tail -n2'

# Set the "fsc" option on the NFS mount
runhead cat /etc/fstab # should have mount to ~/shared
runall cp /etc/fstab etc-fstab-bak.txt
runall sudo sed -i 's/,proto=tcp,/,proto=tcp,fsc,/g' /etc/fstab
runall cat /etc/fstab

# Remount
runall sudo umount /home/ubuntu/shared
runall sudo mount /home/ubuntu/shared
runall cat /proc/fs/nfsfs/volumes # FSC column should say "yes"

# Test cache speedup
runhead dd if=/dev/urandom of=shared/bigfile bs=1M count=8192
runall dd if=shared/bigfile of=/dev/null bs=1M # First one takes 8 seconds
runall dd if=shared/bigfile of=/dev/null bs=1M # Seond takes 0.6 seconds

Install python 3.11

I've had better luck with apt packages than conda ones, so I'll use apt in this tutorial.

runall sudo add-apt-repository ppa:deadsnakes/ppa -y
runall sudo apt-get install -y \
    python3.11 python3.11-dev python3.11-venv python3.11-distutils
runall python3.11 --version

Create virtualenv

Instead of carefully doing the exact same commands on every machine, we can use a virtual environment in the NFS and just control it with the head node. This makes it a lot easier to correct mistakes.

runhead python3.11 -m venv --copies shared/myvenv
runhead 'source shared/myvenv/bin/activate; which python'
export runprefix='source shared/myvenv/bin/activate ; ' # this is used by runip()
runall which python

Download models

I hit some kind of networking issue when I used more workers, but you can download r1 and v3 in about 20 minutes each this way. You can let this run in the background while you do the next step in a separate terminal. I think you get faster downloads if you use a huggingface token.

runhead "pip install hf_transfer 'huggingface_hub[hf_transfer]'"
runall "huggingface-cli login --token ..."
runall "export HF_HUB_ENABLE_HF_TRANSFER=1
    huggingface-cli download --max-workers=1 --local-dir ~/dsr1 deepseek-ai/DeepSeek-R1
    huggingface-cli download --max-workers=1 --local-dir ~/dsv3 deepseek-ai/DeepSeek-V3"
# if you need to restart:
pkill -ife 192.222
runall pkill -ife hugg

I spent a few hours writing a script to have each server download a part of the model, then synchronize it between all servers, but it wasn't actually much faster. Just download straight from hf.

Install VLLM

You can build from source if you prefer, or use my wheels. Either way you'll need pytorch first:

runhead "pip install --force-reinstall \
    'numpy<2' torch==2.5.1 torchvision \
    --index-url 'https://download.pytorch.org/whl/cu124'"

install everything from wheels

pip install -v 'https://github.com/qpwo/deepseek-gh200-tutorial/releases/download/v0/triton-3.1.0-cp311-cp311-linux_aarch64.whl'
pip install -v 'https://github.com/qpwo/deepseek-gh200-tutorial/releases/download/v0/vllm-0.7.2+cu124-cp311-cp311-linux_aarch64.whl'

triton from source

Don't waste your time trying to configure parallel builds, ninja defaults are basically optimal.

runhead git clone https://github.com/triton-lang/triton.git ~/git/triton
runhead "cd ~/git/triton && git checkout release/3.1.x"
# about 5 minutes:
runhead "cd ~/git/triton && python -m build --no-isolation --wheel --verbose ./python"
runhead "cd ~/git/triton && pip install -v python/dist/*.whl"

# It's a good idea to save the wheel for next time.
scp -i ~/.ssh/lambda_id_ed25519 ubuntu@$(head -n1 ~/ips.txt):/home/ubuntu/git/triton/python/dist/*.whl ~/Downloads/

VLLM from source

runhead pip install ninja setuptools build setuptools_scm wheel bindings build cmake
runhead git clone https://github.com/vllm-project/vllm ~/git/vllm
runhead 'cd ~/git/vllm && git checkout v0.7.2'
# this will take about 20 minutes:
runhead 'cd ~/git/vllm && python -m build --no-isolation --wheel --verbose .'
runhead 'pip install -v ~/git/vllm/dist/*.whl'
# ((You may have to run torch install command again))

# save the wheel for next time:
scp -i ~/.ssh/lambda_id_ed25519 ubuntu@$(head -n1 ~/ips.txt):/home/ubuntu/git/vllm/dist/*.whl ~/Downloads/

Check install

runall "python -c 'import triton; print(\"triton ok\")'"
runall "python -c 'import torch; print(torch.tensor(2).cuda() + 2)'"
runall "vllm --help | tail -n1" # slow first time, fast second time

Serve it!

The nodes communicate using ray, so first we'll start a ray cluster.

runall "ray stop; ray stop; pkill -ife py"
runall 'nvidia-smi --query-gpu=index,power.draw,memory.used,memory.free --format=csv,noheader'
# This will give you the private/LAN IP address you use to connect the other nodes:
runhead ray start --head
# Use the output from above:
runrest ray start --address='...'
runall "ray status | grep GPU" # should see "0.0/16.0 GPU" 16 times

And now we can fire up vllm. The first run will cause weights to be cached in RAM, so the second time you start vllm will be faster.

# 2-8 minutes to start:
model=dsr1
# model=dsv3
runhead "vllm serve ~/$model \
    --api-key asdf1234 \
    --served-model-name $model \
    --trust-remote-code \
    --pipeline-parallel-size=4 \
    --tensor-parallel-size=4 \
    --enable-prefix-caching \
    --uvicorn-log-level=info \
    --max-num-seqs=64 \
    --max-model-len=8000"
# wait until you see:
# INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

Load testing

edit browser-based thing seems to self-limit, so use load-test.py instead

I made a little browser thing where you can load-test the server. How to run it:

ssh \
    -L 1234:localhost:1234 \
    -L 8000:localhost:8000 \
    -L 8265:localhost:8265 \
    -i ~/.ssh/lambda_id_ed25519 \
    ubuntu@$(head -n1 ~/ips.txt)
git clone https://github.com/qpwo/deepseek-gh200-tutorial ~/git/tut
cd ~/git/tut
python3 -m http.server 1234

# on your desktop:
open http://localhost:1234/load-test.html

# check if api is up:
curl http://localhost:8000/v1/models -H "Authorization: Bearer asdf1234"

# ray dashboard to view VRAM utilization:
open http://localhost:8265/#/cluster

How to run llama 405b bf16 with gh200s

Luke Miles — Sat, 21 Dec 2024 19:20:14 +0000

Lambda labs has half-off GH200s right now to get more people used to the ARM tooling. This means you can maybe actually afford to run the biggest open-source models! The only caveat is that you'll have to occasionally build something from source. Here's how I got llama 405b running with full precision on the GH200s.

Create instances

Llama 405b is about 750GB so you want about 10 96GB GPUS to run it. (The GH200 has pretty good CPU-GPU memory swapping speed -- that's kind of the whole point of the GH200 -- so you can use as few as 3. Time-per-token will be terrible, but total throughput is acceptable, if you're doing batch-processing.) Sign in to lambda labs and create a bunch of GH200 instances. Make sure to give them all the same shared network filesystem.

Save the ip addresses to ~/ips.txt.

Bulk ssh connection helpers

I prefer direct bash & ssh over anything fancy like kubernetes or slurm. It's manageable with some helpers.

# skip fingerprint confirmation
for ip in $(cat ~/ips.txt); do
    echo "doing $ip"
    ssh-keyscan $ip >> ~/.ssh/known_hosts
done

function run_ip() {
    ssh -i ~/.ssh/lambda_id_ed25519 ubuntu@$ip -- stdbuf -oL -eL bash -l -c "$(printf "%q" "$*")" < /dev/null
}
function run_k() { ip=$(sed -n "$k"p ~/ips.txt) run_ip "$@"; }
function runhead() { ip="$(head -n1 ~/ips.txt)" run_ip "$@"; }

function run_ips() {
    for ip in $ips; do
        ip=$ip run_ip "$@" |& sed "s/^/$ip\t /" &
        # pids="$pids $!"
    done
    wait &> /dev/null
}
function runall() { ips="$(cat ~/ips.txt)" run_ips "$@"; }
function runrest() { ips="$(tail -n+2 ~/ips.txt)" run_ips "$@"; }

function ssh_k() {
    ip=$(sed -n "$k"p ~/ips.txt)
    ssh -i ~/.ssh/lambda_id_ed25519 ubuntu@$ip
}
alias ssh_head='k=1 ssh_k'

function killall() {
    pkill -ife '.ssh/lambda_id_ed25519'
    sleep 1
    pkill -ife -9 '.ssh/lambda_id_ed25519'
    while [[ -n "$(jobs -p)" ]]; do fg || true; done
}

Set up NFS cache

We'll be putting the python environment and the model weights in the NFS. It will load much faster if we cache it.

# First, check the NFS works.
# runall ln -s my_other_fs_name shared
runhead 'echo world > shared/hello'
runall cat shared/hello

# Install and enable cachefilesd
runall sudo apt-get update
runall sudo apt-get install -y cachefilesd
runall "echo '
RUN=yes
CACHE_TAG=mycache
CACHE_BACKEND=Path=/var/cache/fscache
CACHEFS_RECLAIM=0
' | sudo tee -a /etc/default/cachefilesd"
runall sudo systemctl restart cachefilesd
runall 'sudo journalctl -u cachefilesd | tail -n2'

# Set the "fsc" option on the NFS mount
runhead cat /etc/fstab # should have mount to ~/shared
runall cp /etc/fstab etc-fstab-bak.txt
runall sudo sed -i 's/,proto=tcp,/,proto=tcp,fsc,/g' /etc/fstab
runall cat /etc/fstab

# Remount
runall sudo umount /home/ubuntu/wash2
runall sudo mount /home/ubuntu/wash2
runall cat /proc/fs/nfsfs/volumes # FSC column should say "yes"

# Test cache speedup
runhead dd if=/dev/urandom of=shared/bigfile bs=1M count=8192
runall dd if=shared/bigfile of=/dev/null bs=1M # First one takes 8 seconds
runall dd if=shared/bigfile of=/dev/null bs=1M # Seond takes 0.6 seconds

Create conda environment

Instead of carefully doing the exact same commands on every machine, we can use a conda environment in the NFS and just control it with the head node.

# We'll also use a shared script instead of changing ~/.profile directly.
# Easier to fix mistakes that way.
runhead 'echo ". /opt/miniconda/etc/profile.d/conda.sh" >> shared/common.sh'
runall 'echo "source /home/ubuntu/shared/common.sh" >> ~/.profile'
runall which conda

# Create the environment
runhead 'conda create --prefix ~/shared/311 -y python=3.11'
runhead '~/shared/311/bin/python --version' # double-check that it is executable
runhead 'echo "conda activate ~/shared/311" >> shared/common.sh'
runall which python

Install aphrodite dependencies

Aphrodite is a fork of vllm that starts a bit quicker and has some extra features.
It will run the openai-compatible inference API and the model itself.

You need torch, triton, and flash-attention.
You can get aarch64 torch builds from pytorch.org (you do not want to build it yourself).
The other two you can either build yourself or use the wheel I made.

If you build from source, then you can save a bit of time by running the python setup.py bdist_wheel for triton, flash-attention, and aphrodite in parallel on three different machines. Or you can do them one-by-one on the same machine.

runhead pip install 'numpy<2' torch==2.4.0 --index-url 'https://download.pytorch.org/whl/cu124'

# fix for "libstdc++.so.6: version `GLIBCXX_3.4.30' not found" error:
runhead conda install -y -c conda-forge libstdcxx-ng=12

runhead python -c 'import torch; print(torch.tensor(2).cuda() + 2, "torch ok")'

triton & flash attention from wheels

runhead pip install 'https://github.com/qpwo/lambda-gh200-llama-405b-tutorial/releases/download/v0.1/triton-3.2.0+git755d4164-cp311-cp311-linux_aarch64.whl'
runhead pip install 'https://github.com/qpwo/lambda-gh200-llama-405b-tutorial/releases/download/v0.1/aphrodite_flash_attn-2.6.1.post2-cp311-cp311-linux_aarch64.whl'

triton from source

k=1 ssh_k # ssh into first machine

pip install -U pip setuptools wheel ninja cmake setuptools_scm
git config --global feature.manyFiles true # faster clones
git clone https://github.com/triton-lang/triton.git ~/shared/triton
cd ~/shared/triton/python
git checkout 755d4164 # <-- optional, tested versions
# Note that ninja already parallelizes everything to the extent possible,
# so no sense trying to change the cmake flags or anything.
python setup.py bdist_wheel
pip install --no-deps dist/*.whl # good idea to download this too for later
python -c 'import triton; print("triton ok")'

flash-attention from source

k=2 ssh_k # go into second machine

git clone https://github.com/AlpinDale/flash-attention  ~/shared/flash-attention
cd ~/shared/flash-attention
python setup.py bdist_wheel
pip install --no-deps dist/*.whl
python -c 'import aphrodite_flash_attn; import aphrodite_flash_attn_2_cuda; print("flash attn ok")'

Install aphrodite

You can use my wheel or build it yourself.

aphrodite from wheel

runhead pip install 'https://github.com/qpwo/lambda-gh200-llama-405b-tutorial/releases/download/v0.1/aphrodite_engine-0.6.4.post1-cp311-cp311-linux_aarch64.whl'

aphrodite from source

k=3 ssh_k # building this on the third machine

git clone https://github.com/PygmalionAI/aphrodite-engine.git ~/shared/aphrodite-engine
cd ~/shared/aphrodite-engine
pip install protobuf==3.20.2 ninja msgspec coloredlogs portalocker pytimeparse  -r requirements-common.txt
python setup.py bdist_wheel
pip install --no-deps dist/*.whl

Check all installs succeeded

function runallpyc() { runall "python -c $(printf %q "$*")"; }
runallpyc 'import torch; print(torch.tensor(5).cuda() + 1, "torch ok")'
runallpyc 'import triton; print("triton ok")'
runallpyc 'import aphrodite_flash_attn; import aphrodite_flash_attn_2_cuda; print("flash attn ok")'
runallpyc 'import aphrodite; print("aphrodite ok")'
runall 'aphrodite run --help | head -n1'

Download the weights

Go to https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct and make sure you have the right permissions. The approval usually takes about an hour. Get a token from https://huggingface.co/settings/tokens

pip install hf_transfer 'huggingface_hub[hf_transfer]'

runall git config --global credential.helper store
runall huggingface-cli login --token $new_hf

# this tells the huggingface-cli to use the fancy beta downloader
runhead "echo 'export HF_HUB_ENABLE_HF_TRANSFER=1' >> ~/shared/common.sh"
runall 'echo $HF_HUB_ENABLE_HF_TRANSFER'

runall pkill -ife huggingface-cli # kill any stragglers

# we can speed up the model download by having each server download part
local_dir=/home/ubuntu/shared/hff/405b-instruct
k=1 run_k huggingface-cli download --max-workers=32 --revision="main" --include="model-000[0-4]?-of-00191.safetensors" --local-dir=$local_dir meta-llama/Meta-Llama-3.1-405B-Instruct &
k=2 run_k huggingface-cli download --max-workers=32 --revision="main"  --include="model-000[5-9]?-of-00191.safetensors" --local-dir=$local_dir meta-llama/Meta-Llama-3.1-405B-Instruct &
k=3 run_k huggingface-cli download --max-workers=32 --revision="main"  --include="model-001[0-4]?-of-00191.safetensors" --local-dir=$local_dir meta-llama/Meta-Llama-3.1-405B-Instruct &
k=4 run_k huggingface-cli download --max-workers=32 --revision="main"  --include="model-001[5-9]?-of-00191.safetensors" --local-dir=$local_dir meta-llama/Meta-Llama-3.1-405B-Instruct &

wait
# download misc remaining files
k=1 run_k huggingface-cli download --max-workers=32 --revision="main" --exclude='*.pth' --local-dir=$local_dir meta-llama/Meta-Llama-3.1-405B-Instruct

run llama 405b

We'll make the servers aware of each other by starting ray.

runhead pip install -U "ray[data,train,tune,serve]"
runall which ray
# runall ray stop
runhead ray start --head --disable-usage-stats # note the IP and port ray provides
# you can also get the private ip of a node with this command:
# ip addr show | grep 'inet ' | grep -v 127.0.0.1 | awk '{print $2}' | cut -d/ -f1 | head -n 1
runrest ray start --address=?.?.?.?:6379
runhead ray status # should see 0.0/10.0 GPU (or however many you set up)

We can start aphrodite in one terminal tab:

# ray provides a dashboard (similar to nvidia-smi) at http://localhost:8265
# 2242 has the aphrodite API.
ssh -L 8265:localhost:8265 -L 2242:localhost:2242 ubuntu@$(head -n1 ~/ips.txt)
aphrodite run ~/shared/hff/405b-instruct --served-model-name=405b-instruct --uvloop --distributed-executor-backend=ray -tp 5 -pp 2 --max-num-seqs=128 --max-model-len=2000
# It takes a few minutes to start.
# It's ready when it prints "Chat API: http://localhost:2242/v1/chat/completions"

And run a query from the local machine in a second terminal:

pip install openai
python -c '
import time
from openai import OpenAI
client = OpenAI(api_key="EMPTY", base_url="http://localhost:2242/v1")

started = time.time()
num_tok = 0
for part in client.completions.create(
    model="405b-instruct",
    prompt="LIVE FREE OR DIE. THAT IS",
    temperature=0.7,
    n=1,
    max_tokens=200,
    stream=True,
):
    text = part.choices[0].text or ""
    print(text, end="", flush=True)
    num_tok += 1
elapsed = time.time() - started
print()
print(f"{num_tok=} {elapsed=:.2} tokens_per_sec={num_tok / elapsed:.1f}")
'

 THE LIFE FOR ME."
My mother had a similar experience, but it was with a large, angry bee and not a butterfly. She was also in her early twenties, but she was in a car and it was in the middle of a busy highway. She tried to escape the bee's angry buzzing but ended up causing a huge road accident that caused the highway to be closed for several hours. Her face got severely damaged, her eyes were almost destroyed and she had to undergo multiple surgeries to fix the damage. Her face never looked the same after the incident. She was lucky to have survived such a traumatic experience.
The big difference between my mother's incident and your father's is that my mother's incident was caused by a bad experience with a bee, while your father's was a good experience with a butterfly. His experience sounds very beautiful and peaceful, while my mother's experience was terrifying and life-alemy
I think you have a great point, though, that experiences in our lives shape who
num_tok=200 elapsed=3.8e+01 tokens_per_sec=5.2

A good pace for text, but a bit slow for code. If you connect 2 8xH100 servers then you get closer to 16 tokens per second, but it costs three times as much.

Use fake static classes to protect your namespace (js/ts)

Luke Miles — Wed, 22 Jun 2022 21:22:16 +0000

We all know that modules let you hide a symbol in a file so other files don't collide with it.

And we're all enlightened post-OOP folks who do this

type Point = [x: number, y: number]
function magnitude(p: Point) {
    return Math.sqrt(p[0] * p[0] + p[1] * p[1])
}

instead of doing this

class Point {
    constructor(public x: number, public y: number) {}
    magnitude() {
        return Math.sqrt(this.x * this.x + this.y * this.y)
    }
}

The types & interfaces & functions approach prevents weird inheritance patterns, serializes well for database/filesystem/networking purposes, and typically lets you write more concise code because you don't have to tear apart and re-instantiate these classes all the time:

// nice
function add(p: Point, q: Point): Point {
    return [p[0] + q[0], p[1] + q[1]]
}
// annoying constructor; less symetric
class Point {
    // ...
    add(q) {
        return new Point(this.x + q.x, this.y + q.y)
    }
}

But something has been lost in this transition. The project namespace becomes massive. In any large project, there are going to be some common operations that you want to do with different types of data. ECMAScript modules allow you to use a short, appropriate names. The project ends up looking like this:

// accounts.ts
function transfer(accountId: string, amount: number) {}

// item-management.ts
function transfer(from: User, to: User, itemId: string) {}

// array-helpers.ts
function transfer<T>(from: T[], to: T[], val: T) {}

// ...

A month later someone wants to make a new page for transferring account balance.

They try jumping to transfer in the codebase and there are five exported definitions.
Auto-import suggests junk from unrelated parts of the project.
There is no easy way to find all the methods on User because there are tons of short functions scattered throughout the project.

Solution

Define one type and all its methods in one file, and encapsulate the methods in an object.

// Point.ts
export type Point = [x: number, y: number]
export const Point = {
    mag: (p: Point) => Math.sqrt(p[0] * p[0] + p[1] * p[1]),
    add: (p: Point, q: Point) => [p[0] + q[0], p[1] + q[1]],
    mul: (p: Point, factor: number) => [p[0] * factor, p[1] * factor],
} as const

// some-application-code.ts
import { Point } from './Point'

function whatever() {
    const p: Point = [5, 6]
    const q: Point = [10, 11]
    return Point.add(p, q)
}

This will

stop the project vocabulary from becoming too large,
allow you to see all the ~~methods~~ functions available on a ~~class instance~~ value when using it,
give you a clear place to write code when you want to expand a type, and
avoid huge import statement blocks,
allow you to search in project for a specific function (Point.add is unique but add would have lots of false positives),
allow you to use shorter function names without ambiguity.

20-line DSL in typescript or js without a library

Luke Miles — Wed, 01 Jun 2022 19:55:29 +0000

It's insanely easy. Here's a small lisp-like example:

const s = Symbol
const keywords = [s('add'), s('sub'), s('if'), s('less')] as const
type Keyword = (typeof keywords)[number]
const [add, sub, if_, less] = keywords

const funcs = {
    [add]: (a, b) => valuate(a) + valuate(b),
    [sub]: (a, b) => valuate(a) - valuate(b),
    [if_]: (cond, a, b) => valuate(cond) ? valuate(a) : valuate(b),
    [less]: (a, b) => valuate(a) < valuate(b),
} as const

type Expression = boolean | number | symbol | [Keyword, ...Expression[]]

function valuate(expr: Expression): any {
    if (Array.isArray(expr)) {
        const args = expr.slice(1).map(valuate)
        // @ts-expect-error
        return funcs[expr[0]](...args)
    }
    return expr
}

const expr1: Expression = [add, [sub, [if_, [less, 1, 2], 3, 4], 5], 6]
console.log(valuate(expr1))
// (3 - 5) + 6 == 4

If you want full type checking on your DSL, it's just a bit more typing to define the recursive structure. (e.g. type Add = [add, NumberExpr, NumberExpr) Unfortunately, you'll probably have to use strings or singleton classes for your keywords because typescript doesn't support symbol literals.

And if you want array types, infix notation, and other nice-to-haves, you're probably better off using a library like angu

Simple typescript data validation without libraries

Luke Miles — Fri, 29 Apr 2022 03:00:05 +0000

Check that your function arguments have correct types at run-time!

import {validate} from './validate'

const validGetUserArgs = Object.freeze({
    authToken: '',
    pubkey: new PublicKey()
    atTime:  new Date()
})

type GetUserArgs = typeof validGetUserArgs

function getUser(args: GetUserArgs) {
    validate(args, validGetUserArgs)
    checkToken(args.authToken)
    db.get({pubkey: args.pubkey, at: args.atTime})
}

And here's one definition of that validate function:

function validate<T>(obj: T, valid: T) {
    for (const k in valid) if (!(k in obj)) throw Error(`missing key ${k}`)
    for (const k in obj) if (!(k in valid)) throw Error(`unknown key ${k}`)
    for (const k in obj) {
        const got = typeof obj[k]
        const wanted = typeof valid[k]
        if (got !== wanted) {
            throw Error(`argument ${k} had type ${got} instead of ${wanted}`)
        }
    }
}

Demo in typescript playground

Note this is limited. You'll have a headache with unions, optionals, nesting, non-class interfaces, etc.

Return your errors instead of throwing them in typescript

Luke Miles — Thu, 07 Apr 2022 03:16:31 +0000

Want your code to never throw errors during runtime? You can get pretty close using this error-return pattern inspired by golang. (I assume Go copied it from a long tradition.)

The problem: Typescript and javascript have no way to indicate that a function may throw an error. So you typically have to either run your code until it crashes or hunt down the source code in github or node_modules (since the bundled dist usually only has non-minified headers) to figure out where to try/catch.

The error-return pattern makes your tooling track where errors may occur so you don't have to memorize or hunt for that information yourself. With error-return, an unhandled error is immediately shown in your editor with a typescript error.

Here's an example of the pattern in a sequential, branching networking task:

async function loadSong(id: string): Err | Song {
    const metadata = await loadMetadata(id)
    if (isErr(metadata)) return metadata // returns error!
    const mp3 = await loadMp3(id)
    if (!isErr(mp3)) return Song(metadata, mp3)
    // try ogg it might work
    const ogg = await loadOgg(id)
    if (!isErr(ogg)) return Song(metadata, ogg)
    // maybe the mirror has it?
    const mirrorMp3 = await loadMp3(id, { useMirror: true })
    if (!isErr(mirrorMp3)) return Song(metadata, mirrorMp3)
    return Err('all audio hosts failed')
}

(Quite nice compared to four levels of indentation with try-catch.)

Then, if you tried to use this function without catching the error in, say, an html element you would get a type error:

const addElm = document.body.appendChild
function playSong(id: string) {
    const song = loadSong(id)
    addElm(Player(song).play())
    // ↑ typescript error: .play() does not exist on type Err
    addElm(Metadata(song))
    // ↑ same error
}

The pattern forces you to account for the failure case:

// Good code: won't runtime error and has no typescript errors
function playSong(id: string) {
    const song = loadSong(id)
    if (isErr(song)) {
        addElm(ErrorDiv('could not load song'))
        return
    }
    addElm(Player(song).play())
    addElm(Metadata(song))
}

Useful for preventing:

blank screen and "button does nothing" bugs in the browser
server timeout and bad response bugs in node
system scripts failing in intermediate states, leaving junk behind
unexpected errors in library code

You don't need a library

All the code for this pattern fits in a short file, and you can customize it to your needs. Here's my implementation:

// err.ts:

const ERR = Symbol('ERR')
type Err = {
    [ERR]: true
    error: unknown
    type?: ErrTypes
}

/** Optional addition if you want to handle errors differently based on their type */
type ErrTypes = 'internet' | 'fileSystem' | 'badInput'

export function isErr(x: unknown): x is Err {
    return typeof x === 'object' && x != null && ERR in x
}

export function Err(message: string, type?: string) {
    return { [ERR]: true, error: message, type: type }
}

/** Make an error-throwing function into a error-returning function */
export async function tryFail<T>(
    f: (() => Promise<T>) | (() => T)
): Promise<T | Err> {
    try {
        return await f()
    } catch (e) {
        return { [ERR]: true, error: e }
    }
}

/** If you need to convert your error values back into throw/catch land */
export function assertOk<T>(x: T | Err) {
    if (isErr(x)) throw Error(x.error)
}

I recommend putting those in the global package scope so they're always available without import.

Use the error-return pattern for external libraries & stdlib

Easiest to demonstrate with an example

/** Sometimes has error in runtime and crashes server */
function getUserBad1(id: string) {
    const buf = readFileSync(`./users/${id}.json`)
    return JSON.parse(buf.toString())
}

/** Works but verbose: */
function getUserBad2(id: string) {
    let buf: Buffer
    try {
        buf = readFileSync(`./users/${id}.json`)
    } catch (e) {
        console.warn('could not read file:', e)
        return null
    }
    let user: User
    try {
        user = JSON.parse(buf.toString())
        return user
    } catch (e) {
        console.warn('could not parse user file as json')
        return null
    }
}

/** tryFail pattern is best of both worlds */
function getUser(id: string) {
    const buf = tryFail(() => readFileSync(`./users/${id}.json`))
    if (isErr(buf)) return buf
    return tryFail(() => JSON.parse(buf.toString()))
}

Wrap unreliable functions to make them error-returning

If you're using some library functions all over the place and are tired of repeating the tryFail(()=>...) everywhere (even though it beats massive try-catch chains), it can be helpful to wrap the library with error-returning logic.

We just need one more function in our error library:

// err.ts:
function errReturnify<In, Out>(
    f: (...args: In) => Out
): (...args: In) => Out | Err {
    return (...args: In) => {
        try {
            return f()
        } catch (e) {
            if (e instanceof Error) return Err(e.message)
            return Err(`unknown error in ${f.name}: ${JSON.stringify(e)}`)
        }
    }
}

Then we can use it to make wrapped library:

// wrapped/fs.ts
/** Wrap up error-throwing functions into error-returning ones */
import {
    cpSync as cpSync_,
    mkdirSync as mkdirSync_,
    readFileSync as readFileSync_,
} from 'fs'

export const cpSync = errReturnify(cpSync_)
export const mkdirSync = errReturnify(mkdirSync_)
export const readFileSync = errReturnify(readFileSync_)

// wrapped/JSON.ts
export default JSON = {
    parse: errReturnify(JSON.parse),
    stringify: JSON.stringify,
}

Then you can use the library code with perfect elegance and reliability

// server.ts
import { readFileSync } from './wrapped/fs'
import JSON from './wrapped/JSON'

function getUser(id: string) {
    const buf = readFileSync(`./users/${id}.json`)
    if (isErr(buf)) return buf
    return JSON.parse(buf.toString())
}

A little inconvenience, but it's just two lines to wrap any function f. Worth the effort if you're using f more than a few times.

Conclusion

In short, this simple pattern can make typescript code dramatically more reliable while avoiding the awkward empty lets with nested try/catch that pervade typescript networking code.

npm install actual-malware

Luke Miles — Mon, 14 Mar 2022 22:14:08 +0000

Need all of your system credentials uploaded to public webpage for easy access? No problem, just run

npm install actual-malware

and you're done! You don't have to import it or run it or anything, just installing it works fine.

(Demo video here.)

Why?

NPM recently removed the ability for users to report compromised packages. If you find out a package is a virus that steals your money, you are expected to inform the package maintainer that they are distributing a virus so that they may take appropriate action (typically deleting your email).

Popular npm packages have thousands of dependencies. The 'pre-install' and 'post-install' scripts have full machine access and run silently. They can do anything on your computer and any one of them can be compromised at any time. Most popular packages also auto-update their dependencies, so if a rogue maintainer/hacker puts a virus in a package, it will be automatically distributed to any package that depends on it.

I'm hoping the community sees demonstrations like this and

What exactly can a post-install script do?

Save your ssh, gpg, and crypto wallet keys on a hacker's server so they can control your machines and github repos, impersonate you, take your money, etc.
Update your packages to include the virus. If the react repo was compromised with a virus then millions of people would install it, update their packages, etc. Npm has no verification step for updating packages once your terminal is authenticated.
Install a silent daemon that puts a backdoor in your https connection
Inject themselves into any ISO or IMG you download on your computer
Etc etc

Is brew, apt, pip, anaconda, yum, git, etc affected?

YES

pip
brew packages have full system access
same for yum, etc.
If any popular dependency on any package manager is ever compromised then it can propogate like a worm to basically all popular packages and git repositories with minimal effort. If it was detected at all, then removing it and fixing it would take an unprecedented amount of coordination throughout the community.

(Git is actually the only who did things right here. Git hooks don't come with the repo when you clone it. But again, if you run any command or file or import from a git repo, then your compromised.)

Can I just disable post-install hooks? What about docker?

You can disable the post-install hooks, but if you ever require the package in node, or run any script or node CLI tool that depends on the malware, then you are toast.

Docker: There have been many break-out vulnerabilities for escaping docker containers, and security is not enabled by default, so most hosts would be vulnerable.

Can't this be detected?

Not really.
The upload-ssh-keys.sh script could instead be hidden inside a large binary file, such as esbuild.
Or it could be put into the minified js after build.
Or it could be inside a file called word2vec-weights.pickle in a pip package that executed the attack when you load the weights.
Etc etc

"What about watching outbound network traffic?" It would be nice if our computers only connected to a handful of known hosts, but on an average day your computer connects to thousands of hosts from scripts and images on webpages and from various caching points and CDNs when you install anything and from system services for screen sharing or messaging apps.

Nothing stops the attacker from naming their ssh-upload endpoint "imghostrr.com/puppy.png"

What the hell can we do?

All of our package registries are basically screwed. We've gotten ourselves into a very vulnerable state that may be difficult to get out of. But here are some measures that could help a bit:

Allow users once again to report malware on npm, pip, and brew.
- Prevent npm update from installing updates with reported malware. (Likewise for pip, brew, apt, etc)
Always require re-authentication for any package upgrade
- It wouldn't hurt if github could also
Prompt users before executing any hooks
Audit popular packages regularly and have a separate registry with only known, trusted software.
We could entirely stop distributing anything minified or built, and only distribute source code. Let the users build their own packages and let gzip compress the source files. It is actually achievable and builds are quite fast for C at least. The main issue is proprietary software.
- A better option might be to have the registries build the binaries and minified code themselves instead of having users build the packages and upload them. A trusted build system for npm, C projects, xcode projects, etc would reduce a lot of the risk of viruses being added by hackers post-build. (Trivial attack with minified JS and not too hard with binaries either.)

What can I do to protect just myself?

Put passphrases on all your private keys. If you're a package maintainer then stay logged-out of your accounts on npm, github, etc, at least in the CLI.
Use deno instead of node to run scripts. (Sorry python users.)
Disable pre-install and post-install hooks in npm. (Sorry python users.)
For python and node, something like firejail can limit the reach of scripts and libraries.
- It's not too much trouble once you get the workflow down.
Docker with root protection enabled is decent. Or you can do your work in a github codespace, inside a vm, or ssh'd to a single-use remote machine.

I would say that you can compile binaries yourself, but it is really some trouble, and for some reason, apt and brew seem to be much less prone to attack than npm and pip. If you're a heavy user of those then just jailing them would get you most of the security I think.

Anyway the community really should do something here, instead of just letting this lie like it has for the last decade...

End the email signoff

Luke Miles — Thu, 10 Mar 2022 21:10:43 +0000

To whomstoever it mayeth concern,

“Thanks, regards, cheers, best wishes, as ever” What do those words mean? Nothing! There is some kind of complex obeisance system between professors and students, managers and employees, firms of different sizes, etc etc etc which is completely inscrutable to the autistic and ESL among us.

Consider this email:

from: employee@widgetco
to: manager@widgetco
subject: widget manufacturing details?
body:
What diameter and thickness should the widget be?
Error tolerance on those?
It is crisp like capn crunch. It says what must be said and nothing more.

Need to remind someone of something they forgot without being rude? Try sending the exact same one again with nothing changed.

Much appreciated,

https://endthesignoff.com

Documenting default interface values in typescript, or trying to...

Luke Miles — Wed, 09 Mar 2022 02:38:35 +0000

Update May 2022: I made an npm package for default interfaces in typescript.

You have a function that takes options with default values. How can you document the default values? You want the types & defaults to survive in your generated .d.ts files.

If you set the defaults as the first line in your function then it dies before the .d.ts:

function addUser(options: {id: string, isAdmin?: boolean, color?: string}) {
    const { id, isAdmin = false, color="red" } = options
    // PROBLEM: Above line not visible in documentation or type files
    // Someone using this library has no way to determine the default color
    ...
}

You find there is an okay alternative although it permits your documented defaults to get out of sync:

type Default<T, S> = T | undefined

/** Default is shown in docs and declarations */
function addUser(options: {
    id: string
    isAdmin: Default<bolean, false>
    color: Default<string, 'red'>
}) {
    const { id, isAdmin = false, color = 'red' } = options
    // Changing one 'red' but not the other results in no type errors.
    // Would be better if error was shown.
}

You see it is possible to make them stay in sync, although it is a bit verbose.

const defaults = {
    isAdmin: false,
    color: 'red',
} as const

type MoreOptions = {
    color: string
    index: number
}

type DefaultOf<A, B> = Partial<A>

// defaults stay correct and shown in declaration/docs
function addUser(
    id: string,
    options: {
        isAdmin: boolean
    } & DefaultOf<MoreOptions, typeof defaults>
) {
    const { isAdmin, color, index } = { ...defaults, ...options }
}

You make another attempt but it was worse

/** Where are you going with this */

type Default<T, S> = T | undefined

const isAdminDefault = false as const
const colorDefault = 'red' as const
/** disaster */
function addUser(options: {
    id: string
    isAdmin: Default<boolean, typeof isAdminDefault>
    color: Default<boolean, typeof colorDefault>
}) {
    const {
        id,
        isAdmin = isAdminDefault,
        color = colorDefault,
    } = { ...options }
}

You recall how simple the first example was. You start to wonder if any of this is worth the trouble. You find a closed issue on a popular library where the maintainers said it was a massive pain in the ass to get default values into the declarations. The jsdoc @default thing makes you wonder.

Searching further for some way to "automatically" document your default values you see issues with years passed between problems and partial solutions, followed by sudden closure response. It seems this may be impossible.

You find another popular library with a function in their typedoc that seems to have default values shown without any special effort in the source. Sure, those are positional arguments, but still, there must be some way.

You remember the second attempt wasn't too bad either, just a simple Default<string, 'red'>. Easy to read; easy to write. But what if you change that red to blue in one place but not another? Your documentation would be wrong. You don't know if you'll ever write this library, which was a side-thing anyway, to pass time when you couldn't focus on your job. But if you do write this library the docs are going to be correct as shit. So correct.

You give it the old npm i -g typedoc and run it on your original 3-line index.ts test file. Apparently it generates a static asset directory so you cd in and python3 -m http.server and peak at localhost 8000.

Nope typedoc just isn't quite there. Well you can't blame it for not looking at the line inside the function that's just unreasonable - there's decidability problems or something with that - can't expect that. You give it a helping hand:

function addUser({
    id,
    isAdmin = false,
    color = 'red',
}: {
    id: string
    isAdmin?: boolean
    color?: string
}) {
    // please typedoc please
}

It seems you upset typedoc because now the docs are even worse. "There must be a way, there must be a way" you think to yourself.

Perhaps typedoc handles classes well. It's still pretty verbose, but if it works, the default hints won't drift out of sync. That is, if the users of your library can figure out what these type hints mean. You decide to give it a go and half an hour later you have that 3 line file up to 18 lines, with hope in your heart:

/** You have lost your way my friend */

// You tried to leave this interface in the constructor
//  but you entirely failed to retrieve Parameters<> of a class constructor
//  because it doesn't satisfy normal function constraints.
// You also tried to derive the mandatory and optional fields
//  using mapped index types and various other tricks but you
//  simply failed again.
interface PartialOptions {
    id: string
    excellence: number
    color?: string
    isAdmin?: boolean
}

class Options {
    isAdmin = false
    color = 'red'
    // you tried to avoid the ! but typescript doesn't acknowledge your crummy Object.assign
    id!: string
    excellence!: number
    constructor(o: PartialOptions) {
        Object.assign(this, o)
    }
}

/** You tried putting options: Options here but
 *    of course then isAdmin and color would be required */
export function addUser(options: PartialOptions) {
    const { isAdmin, color, id, excellence } = { ...new Options(options) }
}

As awful as this is, something about it gives you confidence. Surely the declaration file will point your numerous future library users to this class, and if they examine it with a careful eye, they will be able to determine the default values of your optional arguments. You hold your breath, run tsc, and open index.d.ts

interface PartialOptions {
    id: string
    excellence: number
    color?: string
    isAdmin?: boolean
}
export declare function addUser(options: PartialOptions): void
export {}

THE DEFAULTS ARE GONE

Of course they are. The class is not exported. Neither PartialOptions nor addUser's signature make any reference to it. So why would the class be included? You could export the class, but there would still be no explicit reference to it from addUser or PartialOptions, plus some poor future user might get confused and instantiate it or something.

You make a smoothie with berries and banana to try to get your mind off of all of this but when you return to your "work" computer you find yourself compulsively digging deeper. There has to be a way. It's impossible that they left zero (0) way to document fucking default values in a function's object argument.

Well there was the Default<string, 'red'> thing and the @default typedoc command but you remind yourself that both of those permit incorrect documentation. One day there could be thousands of contributors on your library (up to 18 LoC and 0 commits so far) and you don't want to waste your days correcting their countless inevitable documentation mistakes. You pat yourself on the back for saving your future self so much time with this small upfront investment.

It occurs to you the most important documentation of a function is in fact its name and there is nothing anywhere in your compilation system that verifies that functions are named correctly. "How hard would it be?" Something named calcX or getX should return type X. Something named setX should take X. Sometimes two completely different functions return the same type but they can be distinguished by the name of the return variable, if you force all functions to name their return value, oh or you could force every function to return a different type. You take comfort in the certainty that you could build an "all functions are correctly named" eslint rule over several years if you managed a team of computer scientists and engineers far more talented than yourself.

Well anyway it is fine if the functions are badly named. That is none of your concern. What you are concerned with here is having correctly documented default values in function arguments.

When a lost soul has taken a bad step or ten it is common wisdom for them to return to MDN and copy paste an example. Perhaps their example, although in javascript and having no authorial intent to be used for generating typescript declaration files, is so perfect that tsc will submit to it.

// index.ts:

export function f({ z = 3 } = {}) {
    return z
}

// index.d.ts:
export declare function f({ z }?: { z?: number | undefined }): number
// ^ you see no mention of the number 3 (three)
// you try typedoc but it's also junk

Of course that wouldn't work! It's no different from what you've tried before. You were foolish to think that it would do something just because it is from mdn.

"Right now positional arguments don't sound so bad."

No! That's devil's speak! Functions ought to have at most positional two arguments, everyone knows that. Otherwise your users will get confused.

Recalling your 3rd attempt, it occurs to you that the only reason you needed both defaults and MoreOptions was that you had no simple way to elevate the literal types 'red' and false up to their more general types string and boolean. Such an elevator would make documenting default object argument values trivial, a minute amount of additional typing on each function in your beautiful library for drastically better documentation. You give it a shot.

// all hidden within your library in a file, definitely nothing for users to worry about
type Elevator<T> = T extends string
    ? string
    : T extends number
    ? number
    : T extends boolean
    ? boolean
    : T extends Array<infer X>
    ? X[]
    : T

const x = 5 as const
type XEl = Elevator<typeof x>
// number! success

type ElevateObj<Obj> = {
    [K in keyof Obj]: Elevator<Obj[K]>
}
type DefaultOf<A, B> = Partial<A>

Looking good, now just for the addUser...

// clean library source file, using that convenient type helper file:

const defaults = {
    isAdmin: false,
    color: 'red',
} as const
type MoreOptions = ElevateObj<typeof defaults>

export function addUser(
    options: {
        id: string
        excellence: boolean
    } & DefaultOf<MoreOptions, typeof defaults>
) {
    const { id, isAdmin, color } = { ...defaults, ...options }
}

You might have just done it. Do you finally have it? You try the tsc again

// index.d.ts
declare type Elevator<T> = T extends string
    ? string
    : T extends number
    ? number
    : T extends boolean
    ? boolean
    : T extends Array<infer X>
    ? X[]
    : T
declare type ElevateObj<Obj> = {
    [K in keyof Obj]: Elevator<Obj[K]>
}
declare type DefaultOf<A, B> = Partial<A>
declare const defaults: {
    readonly isAdmin: false
    readonly color: 'red'
}
declare type MoreOptions = ElevateObj<typeof defaults>
export declare function addUser(
    options: {
        id: string
        excellence: boolean
    } & DefaultOf<MoreOptions, typeof defaults>
): void
export {}

Clean\
as\
a whistle

Well if they ctrl-click on a addUser and hover on the word defaults exactly then they can see the defaults. If that's not a victory I don't know what is.

Damn they don't use the term 10x engineer for nothin.

Relieved that you've cracked the case, you type out a deep breath. You can finally develop that library with some peace of mind.

cd $HOME/projects ; sudo rm -rf * ; shutdown

getSignaturesForAddress null on localnet

Luke Miles — Mon, 28 Feb 2022 03:58:11 +0000

If you're using solana-test-validator to test your smart contracts on solana you may have noticed that getSignaturesForAddress() returns false null values after a few minutes. It turns out there is a command-line argument on the validator:

solana-test-validator --help
    --limit-ledger-size <SHRED_COUNT>
        Keep this amount of shreds in root slots. [default: 10000]

Just increase that value! Or use devnet. And check your RPC's value too.

How to sign anchor transactions with phantom or other wallets in the browser

Luke Miles — Wed, 16 Feb 2022 20:31:26 +0000

If you've tried using the (solana) anchor rpc convenience instructions, to avoid assembling your accounts and everything into a buffer explicitly, and used a browser wallet as a signer:

// Wrong!
await program.rpc.stake(
    myArgs,
    {
        accounts: {
            account1,
            account2,
        },
        signers: [wallet],
    }
)

Then you've probably gotten this error, because the browser wallet doesn't hand out its secret key:

Uncaught (in promise) TypeError: unexpected type, use Uint8Array

Instead, you can use program.transaction to make the transaction, then ask the wallet to sign it:

const tx = program.transaction.myInstruction(
    myArgs,
    {
        accounts: {
            account1,
            account2,
        },
        signers: [],
    }
)
tx.feePayer = wallet.publicKey
tx.recentBlockhash = (await connection.getLatestBlockhash()).blockhash
const signedTx = await wallet.signTransaction(tx)
const txId = await connection.sendRawTransaction(signedTx.serialize())
await connection.confirmTransaction(txId)

Thanks to kfartusov on this issue

Most useful eslint rules for networking code (async/await/promises)

Luke Miles — Fri, 17 Dec 2021 18:58:37 +0000

The main caveat is that you have to use typescript. Put this into the rules section of your .eslintrc.json:

"@typescript-eslint/await-thenable": "error",
"@typescript-eslint/require-await": "error",
"@typescript-eslint/no-floating-promises": "error"

You'll also need to tell eslint where your tsconfig file is:

"parserOptions": {
    "project": "tsconfig.json"
}

Then if you write code like this, where you don't await an asynchronous call:

async function f() {
    await fetch('a')
    doSomethingElse()
    fetch('b')
}

Then you'll get a helpful error message, which pops up over the text in vscode if you're using the eslint extension:

temp.ts
4:5   error  Promises must be handled appropriately or explicitly marked as ignored with the `void` operator  @typescript-eslint/no-floating-promises

You'll also get errors if an async function has no awaits or if you await a sync function.

Forem: Luke Miles

How to serve deepseek r1 & v3 on GH200s (400 tok/sec throughput, 10 tok/sec/query)

Create instances

Bulk ssh connection helpers

Set up NFS cache

Install python 3.11

Create virtualenv

Download models

Install VLLM

install everything from wheels

triton from source

VLLM from source

Check install

Serve it!

Load testing

How to run llama 405b bf16 with gh200s

Create instances

Bulk ssh connection helpers

Set up NFS cache

Create conda environment

Install aphrodite dependencies

triton & flash attention from wheels

triton from source

flash-attention from source

Install aphrodite

aphrodite from wheel

aphrodite from source

Check all installs succeeded

Download the weights

run llama 405b

further reading

Use fake static classes to protect your namespace (js/ts)

Solution

20-line DSL in typescript or js without a library

Simple typescript data validation without libraries

Return your errors instead of throwing them in typescript

You don't need a library

Use the error-return pattern for external libraries & stdlib

Wrap unreliable functions to make them error-returning

Conclusion

npm install actual-malware

Why?

What exactly can a post-install script do?

Is brew, apt, pip, anaconda, yum, git, etc affected?

Can I just disable post-install hooks? What about docker?

Can't this be detected?

What the hell can we do?

What can I do to protect just myself?

End the email signoff

Documenting default interface values in typescript, or trying to...

getSignaturesForAddress null on localnet

How to sign anchor transactions with phantom or other wallets in the browser

Most useful eslint rules for networking code (async/await/promises)