Forem: Jay

asdf, python and automatically enabling virtual envs

Jay — Tue, 13 Jun 2023 12:08:00 +0000

I've switched wholly to asdf for all my projects now, both work and personal, thanks to @emma who first put me on to it. In the day job, it's used for managing ruby and node versions with the asdf-ruby and asdf-nodejs plugins.

I've started to use the asdf-python plugin here on this blog, as I have a small python script I use to extract my music posts from Slack to archive to my Music I've liked page.

When I first switched over, everything stopped working, because I'd forgotten to first enable the python virtual env by calling source env/bin/activate. I've since forgotten to do this a couple of times, so it's time to automate!

autoenv will run commands in a .env file when you cd into a directory, and will also run any in .env.leave when you cd out, so it's perfect for setting and unsetting your python env.

I installed autoenv, and created the .env and .env.leave files:

echo -e "AUTOENV_ENABLE_LEAVE=1\nsource \"$AUTOENV_CUR_DIR/env/bin/activate\"" > .env
echo deactivate > .env.leave

Then when you next cd into your project folder, you'll be prompted:

autoenv:
autoenv: WARNING:
autoenv: This is the first time you are about to source /Users/jay/Library/CloudStorage/Dropbox/Work/gooby.org/2021/2021.gooby.org/.env:
autoenv:
autoenv:   --- (begin contents) ---------------------------------------
autoenv:     AUTOENV_ENABLE_LEAVE=1$
autoenv:     source "$AUTOENV_CUR_DIR/env/bin/activate"$
autoenv:
autoenv:   --- (end contents) -----------------------------------------
autoenv:
autoenv: Are you sure you want to allow this? (y/N)

to check you really want to run that .env in the future. Ditto the first time you cd out of the project¹:

autoenv:
autoenv: WARNING:
autoenv: This is the first time you are about to source /Users/jay/Work/gooby.org/2021/2021.gooby.org/.env.leave:
autoenv:
autoenv:   --- (begin contents) ---------------------------------------
autoenv:     deactivate$
autoenv:
autoenv:   --- (end contents) -----------------------------------------
autoenv:
autoenv: Are you sure you want to allow this? (y/N)

From then on, you won't see the prompts again. You can hide them completely if you add AUTOENV_ASSUME_YES=1 to the top of both .env and .env.leave.

Because we're passing the full path to activate in the .env file using source "$AUTOENV_CUR_DIR/env/bin/activate", if we cd within the project folder structure, we'll still keep our activated environment, because autoenv looks for a parent .env file. If we just had source env/bin/activate then we'd lose the environment activation, because the path to env/bin/activate is relative and would no longer be correct. ↩

Clink makes working with Windows’ `cmd.exe` so much nicer

Jay — Tue, 06 Jun 2023 15:16:00 +0000

Due to a project at work, I’m using Windows’ cmd.exe for something like the first time in a decade. I can’t use WSL on the project, and Clink with its syntax colouring and familiar readline history has already made it a 100 times more pleasant!

You get a searchable history and tab completion. What wasn’t immediately obvious to me, was how to choose the suggested completion - if you press tab, Clink will start cycling through commands and paths; you need to press End or use the right arrow key. Loads more detail on completetion here.

The repo lives at github.com/chrisant996/clink - give it a star!

Automating Jekyll card generation with ruby's Ferrum gem

Jay — Thu, 12 May 2022 07:04:44 +0000

I've now got a simple, ruby-only, automated way of making Twitter summary cards and OpenGraph image previews for the posts on this blog. Now I just need to iterate on the currently horrible version-zero look of them!

Historically you would have done this using javascript calling Puppeteer driving a headless Chrome instance to generate the screenshot. Mikkel Hartmann has a great write up of doing it this way.

Ruby now has the awesome Ferrum gem which uses the Chrome DevTools Protocol to control Chrome, so you can drop the need for JS.

How I did it

In my Jekyll install I've got a _cards symbolic link to my _posts directory, and in my _config.yml I've got a defaults section that looks like this:

defaults:
  -
    scope:
      path: "_posts"
    values:
      layout: post  
  -
    scope:
      path: "_cards"
    values:
      layout: card

You need to do this so your posts don't specify layout: post in their individual frontmatter. If they did, this would override the layout: card that we need the cards collection to use, and we'd just end up with a screenshot of the entire page.

So there's now a cards collection that mirrors my posts, but that render with their own _layouts/cards.html layout. I use this to generate the post image from.

Because of the symbolic link, every _post entry like _posts/2022-05-11-automating-jekyll-card-generation-with-ruby-ferrum.md will have a corresponding /cards/2022-05-11-automating-jekyll-card-generation-with-ruby-ferrum.html page when you run jekyll serve or jekyll build. It's these pages we'll point Chrome at to generate the images.

After adding the Ferrum gem to my Gemfile, I can use this ruby to make images for my posts:

require "Rubygems"
require "Ferrum"

def generate_card(browser, card, png, options={})
  browser.go_to("http://localhost:4000/cards/#{card}")
  # see all the options here https://github.com/rubycdp/ferrum#screenshots
  browser.screenshot(path: "./images/cards/#{png}",
                     full: true,
                     # final image size is window_size x scale
                     scale: 2)
end

browser = Ferrum::Browser.new(window_size: [800, 418])

# Check what cards we need to make
Dir.glob("_posts/*").each do |post|

  post = File.basename(post, ".md")
  png  = post + ".png"
  card = post + ".html"

  generate_card(browser, card, png) unless File.exists?("./images/cards/#{png}")
end

This will use your local version of the jekyll site to create the screenshots from. To ensure it's running, and for a bunch of other tasks, I use a Makefile. Here's a simplified version:

default: deploy

clean-site:
# Do this step to ensure we don't accidentally publish drafts etc
 rm -rf _site

# JEKYLL_ENV=production to avoid localhost urls in your jekyll-seo-tag links
serve-site:
     JEKYLL_ENV=production jekyll serve & echo $$! > /tmp/jekyll.pid && sleep 15

build-cards: serve-site
    ruby ./scripts/generate-cards.rb
    kill -9 $(shell cat /tmp/jekyll.pid)
    rm -f /tmp/jekyll.pid

deploy: clean-site build-cards
    rsync --progress --delete -a -e ssh  _site/ gooby.org:~/jay.gooby.org/public

The deploy recipe cleans the site, serves it, generates the images using the ruby script, then "deploys" the site by copying it to my remote host with rsync. Because it's the default recipe, I can just call make in the root of my project to get the site live.

Use a basic Gmail account to “Send mail as” with a domain that uses Cloudflare email routing

Jay — Thu, 05 May 2022 23:00:00 +0000

If you’ve got a basic Gmail account like your.name@gmail.com (ie not a fullGoogle Workspace account) and a custom domain that you want to send email from, using Gmail’s “Send mail as” functionality, and you want to use this domain with Cloudflare’s email routing then this guide is for you…

ytho?

First some background. Why would you want to do this? Typically, if you have a custom domain that you want to use for email, then you’d have to administer or have access to an email server for it. This email server will need to have DNS MX records set up for it, and will also need a good sender reputation if your emails aren’t going to end up in spam.

Normally, to “Send mail as” in Gmail, you’d enter the custom domain’s email server SMTP details, and your email username and password. Then when you send emails from Gmail, Google contacts that mail server, and the emails go out from it, rather than from Gmail.

The advantage of this is that you can let the domain’s email server sign the emails with a DKIM signature, as well as having whatever email addresses you want on that custom domain. The downside, is that you need an email server, and inboxes for those custom addresses.

However, now that Cloudflare have made email routing available for domains where they are the authoritative nameserver (host your domain’s DNS records), you can use Gmail to send emails using your custom domain, and Cloudflare to route them, doing away with the need for a custom mailserver entirely.

How to use Gmail’s “Send mail as” with a custom domain and Cloudflare email routing

1. Configure Cloudflare

First you’ll need a domain configured to use Cloudflare, if not you’ll need to change the domain’s authoritative nameservers first. Then you can setup Cloudflare email routing.

You’ll be given some DNS records to set up by default, but you’ll want to alter these slightly.

1.1 Edit your SPF record

Your spf TXT record will need to look like this:

v=spf1 a mx include:_spf.google.com include:_spf.mx.cloudflare.net ~all

– we’ve added a mx include:_spf.google.com to indicate that google can send on our behalf, along with Cloudflare.

1.2 Edit your DMARC record

Change it so it looks like this:

v=DMARC1; p=none; rua=mailto:you@example.com; aspf=r;

– where example.com is your custom domain. The email address in the rua field can be anything at your custom domain; it’s where email providers will periodically send you aggregated reports about your domain’s email.

We’ve set the domain policy p to none (other options are quarantine and reject if sending mail fails to pass DMARC checks). The SPF alignment policy aspf is set to relaxed r.

Setting the above is critical to not getting your custom domain’s email bounced or rejected, especially as it won’t be DKIM signed by Gmail.

1.3 Create an Email Route

In your Cloudflare dashboard, click the Email option, then add a destination address - use your regular gmail address your.name@gmail.com etc. You’ll need to click the email that’s sent to you to confirm this.

Once you’ve done that, you can add a custom email address e.g. special@example.com and route it the gmail address you just confirmed.

2. Configure Gmail

Next we’ll do the Gmail configuration.

2.1 Create an app password

First you’ll need to create an email app password in your Google account. Go to https://myaccount.google.com/apppasswords and choose Email from the “Select app” dropdown and Other for the device.

Copy the password that’s generated for you.

2.2 Add the email address to Gmail’s “Send mail as” section

There are detailed instructions on adding a new email address, but it’s relatively easy. Go to your Gmail account settings and in the Send mail as: section, click the Add another email address option.

In the pop-up, enter your custom domain’s email address, untick the Treat as an alias option, click the Specify a different “reply-to” address link and add the same custom email address in there. Then click Next Step.

Overwrite the value for SMTP Server. Use smtp.gmail.com, leave the Port option as is.

Username should be the name part of your regular gmail address, so if you’re your.name@gmail.com then you’d enter your.name. The Password is the email app password that you generated above. Click the Add Account button. You’ll be sent an email to the custom email address you entered. Click the link in this, and you’re good to go.

All done

You’ll be able to compose emails in Gmail and set the sender to your custom domain email address, and when people reply, Cloudflare will route these back to your regular gmail account, all without you needing a separate custom mail server.

Recursively searching and downloading objects from AWS S3 with the `aws` CLI

Jay — Mon, 25 Apr 2022 23:00:00 +0000

Yesterday I was asked if we had some (very) old files from a project. They were originally generated by one of our freelancers, who at the end of the project, had given us a backup of his external drive for archiving.

My go-to for this kind of long-term storage is AWS S3 or Glacier. I’d backed up the drive to a bucket and had forgotten about it until now.

The aws CLI is powerful but inscrutable, given that it can do anything the web console can, for any AWS service.

The files were from Sibelius (.sib) and for Clarinet, Flute and Sax. Cue some googling about the aws s3 list-objects command.

You can combine the search terms to the --query parameter to limit the results of the list-objects call. Searches are case sensitive, and I wasn’t sure how the files would have been named originally, so I went for:

--query "Contents[?(contains(Key,'larinet') || contains(Key,'lute') || contains(Key,'ax')) && contains(Key,'.sib')]

To give me the best chance of finding Clarinet or clarinet, etc (no good of they’re named CLARINET, but what kind of monster would do that?).

Because I only need the path to the S3 object for the get-object call, I’ll filter on just those:

aws s3api list-objects --bucket "my-bucket-name" --prefix 'path/to/files/' --query "Contents[?(contains(Key,'larinet') || contains(Key,'lute') || contains(Key,'ax')) && contains(Key,'.sib')]"

This gives me an array of json objects like:

[
    {
        "Key": "path/to/files/Project backup/Clarinets/example.sib",
        "LastModified": "2017-09-18T21:01:19+00:00",
        "ETag": "\"e9abc61fab413ec5de1126e7ab48ac38\"",
        "Size": 226180,
        "StorageClass": "STANDARD",
        "Owner": {
            "DisplayName": "owner-name",
            "ID": "d9fc4a52c54711ec9d640242ac120002f54ce640c54711ec9d640242ac120002"
        }
    },
    ...,
    ...,
]

We can then loop through these, downloading the files. Instead of just dumping them all in one directory, let’s replicate the path structure they’ve been saved to:

# Get the list of objects to download - note that the Key search is case sensitive
# hence using larinet instead of clarinet or Clarinet, etc
aws s3api list-objects --bucket "my-bucket-name" --prefix 'path/to/files/' --query "Contents[?(contains(Key,'larinet') || contains(Key,'lute') || contains(Key,'ax')) && contains(Key,'.sib')]" > files.json

# loop through the list, grabbing each object, putting it in the relevant folder
cat files.json | jq -r '.[] | .Key'| while read key
  do dir="$(dirname "$key")"
     file="$(basename "$key")"     
     mkdir -p "$dir"
     aws s3api get-object --bucket "my-bucket-name" --key "$key" "${dir}"/"${file}"
  done

Set up a freshly installed macos Monterey machine as a gitlab-runner host

Jay — Wed, 13 Apr 2022 16:09:00 +0000

You've got a newly installed macos Monterey machine you want to use as a gitlab-runner host to test and build your ios and macos projects on. What do you need?

macos command line tools

Get the macos Command Line Tools. In a terminal, run:

xcode-select --install
softwareupdate -l

# Then pick the most recent version of Command Line tools, e.g.
softwareupdate --install "Command Line Tools for Xcode-13.3"
sudo xcodebuild -license accept

homebrew

Then install homebrew:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

and this Brewfile (save it to ~/Brewfile)

brew "asdf"
brew "bash"
brew "bash-completion"
brew "ack"
brew "autoconf"
brew "automake"
brew "coreutils"
brew "curl"
brew "gettext"
brew "git"
brew "gnu-sed"
brew "gnu-tar"
brew "gnu-indent"
brew "gnu-which"
brew "pkg-config"
brew "robotsandpencils/made/xcodes"

Then run:

brew bundle

in the same folder as the Brewfile (e.g. cd ~ && brew bundle).

Xcode

And get and install the latest Xcode so you have the ios simulator, etc:

XCODES_USERNAME="your_apple_id" XCODES_PASSWORD="your_apple_id_password" xcodes install --latest

`gitlab-runner`

Now install and configure the gitlab-runner:

# Install gitlab-runner
# uses `arch` to fetch the correct version
[ "$(arch)" = "i386" ] && sudo curl --output /usr/local/bin/gitlab-runner "https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-darwin-amd64" || sudo curl --output /usr/local/bin/gitlab-runner "https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-darwin-arm64"

sudo chmod +x /usr/local/bin/gitlab-runner

# Register the runner
gitlab-runner register

# Install gitlab-runner as a user service.
# PLEASE NOTE you must enable autologin to this user
# on the macos machine or your runner will be showing
# as `stuck` the next time your macos machine reboots
# and you don't login.
su - $(whoami)
cd ~
gitlab-runner install
gitlab-runner start

Disable macos sleep for the logged in user

Ensure the mac doesn't sleep, if you don't set your mac not to sleep, it will, and then your runner won't be able to pick up jobs, and will be showing as stuck:

sudo pmset -a sleep 0; sudo pmset -a hibernatemode 0; sudo pmset -a disablesleep 1

Optional ruby install with `asdf` when using Fastlane

I also need ruby, because I'm using Fastlane to manage the ios and macos builds, and I'm using asdf to manage ruby versions:

echo "ruby 2.7.6" > ~/.tool-versions
asdf plugin-add ruby
asdf install ruby 2.7.6
gem install fastlane

Positional arguments in git aliases

Jay — Mon, 12 Jul 2021 23:00:00 +0000

I wanted to make a git add-and-commit alias, but getting the positional arguments to handle filenames with spaces in was a lot more complicated than I’d expected.

My naive first try was:

[alias]
  add-and-commit = "!git add $@ && git commit"

which works as long as you don’t have filenames that you don’t need to quote, and that you’re happy using git’s default method for capturing your commit message. You can call it like:

git add-and-commit readme.md example.txt

and it will add the two files readme.md and example.txt and then ask for a commit message from you.

But if one of your files is e.g. called file with a space.txt it barfs:

git add-and-commit readme.md "file with a space.txt"
fatal: pathspec 'file' did not match any files

Also, what if you want to pass your commit message at the same time? Now we need to consider positional parameters, how we ensure they’re escaped properly and how we can grab some of them for the git add portion of the alias, and the others for the git commit portion. The aim is so that we can call:

git add-and-commit readme.md "file with a space.txt" -m "A nice long commit message"

Capture the filenames

Breaking this down, we want to separately capture all the filename arguments (in this case, readme.md and file with space.txt) before the last two, which are the -m and the commit message itself.

bash parameter expansion is a our friend¹ here. This Ask Ubuntu answer has some nice examples that helped me out.

We want args $1 to the total number of args ($#) minus 2, for the filenames (-2, because of the -m and then the message itself).

We need each argument quoted in case it has spaces in, so we use the quoted "{$@}" format instead of "{$*}" and we use bash arithetic to subtract 2 from the argument length $(($#-2)).

So to capture all the filenames, correctly quoted:

# arguments, quoted, starting from 1 and going to -2 the total number of arguments
"${@:1:$(($#-2))}"

Capture the commit message arguments

And then we want to capture the last two arguments to pass to git commit. The space before the -2 is important, and don’t forget to use the quoted "${@}" style, because our commit message is bound to have spaces in too.

# a negative value needs either a preceding space or parentheses
"${@: -2}"

Wrapping this all up, I ended up with this, which is slightly more complicated by the need to escape the quoting of the arguments to the git alias. First we git add all the filenames, then we git commit using our -m message.

[alias]
  add-and-commit = "!git add \"${@:1:$(($#-2))}\" && git commit \"${@: -2}\" #"

LOL, no ↩

Tricks with less

Jay — Tue, 02 Mar 2021 16:41:00 +0000

less is one of those utilities I use day in and day out, and it's easy to forget that it has some really powerful features.

Case-insenstive searching

Nine times out of ten, I less a file and then start finding lines using the / command. You can make this case-insentive if you enter -I and hit return - you'll see a prompt saying Ignore case in searches and in patterns (press RETURN)

Then when you search using / or ? you can be confident that you won't miss something due to case sensitivity.

Filtering

Even more useful is filtering; you can strip out all the lines you don't care about by typing & - you'll get a &/ prompt where you can enter your filter text. Say you have an nginx access log:

107.174.242.123 - - [02/Mar/2021:05:15:07 -0800] "GET /2021/01/04/hello-2021 HTTP/1.1" 200 7073 "https://jay.gooby.org/" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.18 Safari/537.36"
104.199.125.198 - - [02/Mar/2021:05:15:33 -0800] "HEAD / HTTP/2.0" 200 390 "https://t.co/Pp644lxwd6" "-"
64.225.34.92 - - [02/Mar/2021:05:16:16 -0800] "GET /feed.xml HTTP/1.1" 200 13370 "-" "Feedbin feed-id:2056147 - 2 subscribers"
40.77.167.97 - - [02/Mar/2021:05:24:34 -0800] "GET /robots.txt HTTP/1.1" 200 4785 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
64.225.34.92 - - [02/Mar/2021:05:26:56 -0800] "GET /feed.xml HTTP/1.1" 200 13370 "-" "Feedbin feed-id:2056147 - 2 subscribers"
13.66.139.48 - - [02/Mar/2021:05:29:23 -0800] "GET / HTTP/1.1" 200 7706 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

You can hide anything that isn't a HEAD request by typing & and then at the &/ prompt enter HEAD

104.199.125.198 - - [02/Mar/2021:05:15:33 -0800] "HEAD / HTTP/2.0" 200 390 "https://t.co/Pp644lxwd6" "-"

Use `less +F` as a replacement for `tail -f`

How many times have you been tailing a file with tail -f and then wanted to search for something, so you end up quitting the tail and then lessing it instead? I used to do this all the time, until I discovered that less has a tail mode. Use it like this:

less +F /some/file

You can ctrl-c to exit the tailing mode and then use all the above tricks to filter and search, and then when you're done, press shift-f to go back to tail mode.

uknown-host

Jay — Wed, 10 Feb 2021 10:03:00 +0000

I updated a server cluster last night and was getting loads of

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

warnings. You get told about the offending line number:

Offending RSA key in ~/.ssh/known_hosts:531

And usually I’d just edit it and be done, but after doing it a couple of times I wrote unknown-host

#!/bin/bash
#
# Use this to remove the offending line from your
# ~/.ssh/known-hosts file when you know the host
# has actually changed and you get the
#
# @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
# @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
# @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
#
# Offending RSA key in ~/.ssh/known_hosts:531
#
# warning telling you what line needs removing.
#
# usage: unknown-host <line number>
#
# So calling:
#
# unknown-host 531
#
# will remove line 531 and then you can ssh again with the nag.
#
# jay@gooby.org
# @jaygooby

line=$1

# number detection via https://stackoverflow.com/a/3951175/391826
case $line in
    ''|*[!0-9]*) line="" ;;
esac

if [ -z "$line" ]; then
  echo -e "Usage: $(basename $0) <line number>" >&2
  exit 1
else
  sed -i"" -e "${line}d" ~/.ssh/known_hosts
fi

so I wouldn’t have to do that again, in the great spirit of xkcd’s “Automation” webcomic