Forem: Harsh Shandilya

Tips and tricks for using Renovate

Harsh Shandilya — Tue, 17 Jan 2023 19:32:18 +0000

Mend Renovate is a free to use dependency update management service powered by the open-source renovate, and is a compelling alternative to GitHub’s blessed solution for this problem space: Dependabot. Renovate offers a significantly larger suite of supported language ecosystems compared to Dependabot as well as fine-grained control over where it finds dependencies, how it chooses updated versions, and a lot more. TL;DR: Renovate is a massive upgrade over Dependabot and you should evaluate it if any aspect of Dependabot has caused you grief, there’s a good chance Renovate does it better.

I’m collecting some tips here about “fancy” things I’ve done using Renovate that may be helpful to other folks. You’ll be able to find more details about all of these in their very high quality docs at docs.renovatebot.com.

Disabling updates for individual packages

There are times where you’re sticking with an older version of a package (temporarily or otherwise) and you just don’t want to see PRs bumping it, wasting CI resources for an upgrade that will probably fail and is definitely not going to be merged. Renovate offers a convenient way to do this:

{
  "packageRules": [
    {
      "managers": ["gradle"],
      "packagePatterns": ["^com.squareup.okhttp3"],
      "enabled": false,
    },
  ],
}

Grouping updates together

Renovate already includes preset configurations for monorepos that publish multiple packages with identical versions, but you can also easily add more of your own. As an example, here’s how you can combine updates of the serde crate and its derive macro.

{
  "packageRules": [
    {
      "managers": [
        "cargo"
      ],
      "matchPackagePatterns": [
        "serde",
        "serde_derive"
      ],
      "groupName": "serde"
    }
  ]
}

Set a semver range for upgrades

Sometimes there are cases where you may need to set an upper bound on a package dependency to avoid breaking changes or regressions. Renovate offers intuitive support for the same.

{
  "packageRules": [
    {
      "matchPackageNames": ["com.android.tools.build:gradle"],
      "allowedVersions": "<=7.4.0"
    }
  ]
}

Supporting non-standard dependency declarations

Dependency versions are sometimes specified without their package names, for example in config files. These cannot be automatically detected by Renovate, but you can use a regular expression to teach it how to identify these dependencies.

For example, you can specify the version of Hugo to build your Netlify site with in the netlify.toml file in your repository.

[build.environment]
  HUGO_VERSION = "0.109.0"

This is how the relevant configuration might look like with Renovate

{
  "regexManagers": [
    {
      "description": "Update Hugo version in Netlify config",
      "fileMatch": [".toml$"],
      "matchStrings": [
        "HUGO_VERSION = \"(?<currentValue>.*?)\""
      ],
      "depNameTemplate": "gohugoio/hugo",
      "datasourceTemplate": "github-releases"
    }
  ]
}

You can read more about Regex Managers here.

Making your GitHub Actions usage more secure

According to GitHub’s official recommendations, you should be using exact commit SHAs instead of tags for third-party actions. However, this is a pain to do manually. Instead, allow Renovate to manage it for you!

{
  "extends": [
    "config:base",
    "helpers:pinGitHubActionDigests",
  ]
}

Automatically merging compatible updates

Every person with a JavaScript project has definitely loved getting 20 PRs from Dependabot about arbitrary transitive dependencies that they didn’t even realise they had. With Renovate, that pain can also be automated away if you have a robust enough test suite to permit automatic merging of minor updates.

{
  "automergeType": "branch",
  "packageRules": [
    {
      "description": "Automerge non-major updates",
      "matchUpdateTypes": ["minor", "patch", "digest", "lockFileMaintenance"],
      "automerge": true
    },
  ]
}

With this configuration, Renovate will push compatible updates to renovate/$depName branches and merge it automatically to your main branch if CI runs on the branch and passes. To make that happen, you will also need to update your GitHub Actions workflows.

 name: Run tests
 on:
   pull_request:
     branches:
       - main
+ push:
+ branches:
+ - renovate/**

Closing notes

This list currently consists exclusively of things I’ve used in my own projects. There is way more you can achieve with Renovate, and I recommend going through the docs at docs.renovatebot.com to find any useful knobs for the language ecosystem you wish to use it with. If you come across something interesting not covered here, let me know either below or on Mastodon at @msfjarvis@androiddev.social!

Writing your own Nix Flake checks

Harsh Shandilya — Sun, 18 Dec 2022 00:00:00 +0000

Preface

Ever since discovering nix(3) flake check from crane (wonderful tool btw, highly recommend it if you’re building Rust things), I’ve wanted to be able to quickly write my own flake checks. Unfortunately, as with everything Nix, dummy-friendly documentation was hard to come by so I started trying out a bunch of things until I ended up with something that worked, which I’ll share below.

The premise

I had been using a basic shell script with a nix-shell shebang for a while to run formatters on my scripts repo and while it worked, nix-shell startup is fairly slow and it just wasn’t cutting it for me. So I decided to try porting it to nix flake check which would benefit from evaluation caching and be faster while removing the overhead of nix-shell from the utility script.

The thing you’re here for

Like everything in Nix, the checks needed to be derivations that Nix will build and run the respective checkPhase of. So naively, I put together this to run the alejandra Nix formatter, shfmt to format shell scripts and shellcheck to lint them:

outputs = {
  self,
  nixpkgs,
  flake-utils,
}:
  flake-utils.lib.eachDefaultSystem (system: let
    pkgs = import nixpkgs {inherit system;};
    files = pkgs.lib.concatStringsSep " " [
      # Individual shell scripts from the repository
    ];
    fmt-check = pkgs.stdenv.mkDerivation {
      name = "fmt-check";
      src = ./.;
      nativeBuildInputs = with pkgs; [alejandra shellcheck shfmt];
      checkPhase = ''
        shfmt -d -s -i 2 -ci ${files}
        alejandra -c .
        shellcheck -x ${files}
      '';
    };
  in {
    checks = {inherit fmt-check;};
  });

I needed a space separated list of my shell scripts to pass to shfmt and shellcheck, so I used a library function from nixpkgs called concatStringsSep that takes a list, and concatenates it together with the given separator. That’s the files binding declared in the snippet above.

Here I ran into my first problem: Nix expects every derivation to generate an output which meant this doesn’t actually build.

➜ nix flake check
error: flake attribute 'checks.fmt-check.outPath' is not a derivation

There’s been some discussion about this but the TL;DR is that mkDerivation must produce an output. So I tried to cheat around this requirement by faking an output.

diff --git flake.nix flake.nix
index b7fef3b99110..a531a30ad88e 100644
--- flake.nix
+++ flake.nix
@@ -18,6 +18,7 @@
       ];
       fmt-check = pkgs.stdenv.mkDerivation {
         name = "fmt-check";
+ dontBuild = true;
         src = ./.;
         nativeBuildInputs = with pkgs; [alejandra shellcheck shfmt];
         checkPhase = ''
@@ -25,6 +26,11 @@
           alejandra -c .
           shellcheck -x ${files}
         '';
+ installPhase = ''
+ mkdir "$out"
+ '';
       };
     in {
       checks = {inherit fmt-check;};

dontBuild does exactly what you’d think and makes Nix not execute the buildPhase of the derivation, and the mkdir $out in the installPhase generates the output directory Nix was looking for which is still valid even if completely empty.

You can make this slightly faster by using a smaller stdenv that won’t pull in a compiler toolchain or be rebuilt when said toolchain is updated:

diff --git flake.nix flake.nix
index 7ce7a2ba80f8..b69db13fbc6d 100644
--- flake.nix
+++ flake.nix
@@ -16,7 +16,7 @@
       files = pkgs.lib.concatStringsSep " " [
         # bunch of shell scripts since I didn't have an extension I could glob against
       ];
- fmt-check = pkgs.stdenv.mkDerivation {
+ fmt-check = pkgs.stdenvNoCC.mkDerivation {
         name = "fmt-check";
         dontBuild = true;
         src = ./.;

The final result

This is what the flake looked like for me after all this

{
  description = "A very basic flake";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable";
    flake-utils.url = "github:numtide/flake-utils/master";
  };

  outputs = {
    self,
    nixpkgs,
    flake-utils,
  }:
    flake-utils.lib.eachDefaultSystem (system: let
      pkgs = import nixpkgs {inherit system;};
      files = pkgs.lib.concatStringsSep " " [
        # Individual shell scripts from the repository
      ];
      fmt-check = pkgs.stdenvNoCC.mkDerivation {
        name = "fmt-check";
        dontBuild = true;
        src = ./.;
        nativeBuildInputs = with pkgs; [alejandra shellcheck shfmt];
        checkPhase = ''
          shfmt -d -s -i 2 -ci ${files}
          alejandra -c .
          shellcheck -x ${files}
        '';
        installPhase = ''
          mkdir "$out"
        '';
      };
    in {
      checks = {inherit fmt-check;};
    });
}

It’s probably not idiomatic Nix (for some definition of idiomatic) but the entire thing has been a trial and error anyway so 🤷

I’m very much a noob when it comes to Nix so any feedback is very welcome and appreciated!

Mastodon on your own domain without hosting a server, Netlify edition

Harsh Shandilya — Wed, 16 Nov 2022 00:00:00 +0000

Preface

I recently came across a blog post from Maarten Balliauw that explained how they had managed to create an ActivityPub compatible identity for themselves, without hosting Mastodon or any other ActivityPub server.

I recommend going to their blog and reading the whole thing, but here’s a TL;DR

ActivityPub has the notion of an “actor” that sends messages
This “actor” must be discoverable via a protocol called WebFinger
WebFinger is ridiculously easy to implement

For all practical purposes, WebFinger is essentially a JSON document that is served at /.well-known/webfinger from a domain and is used to identify “actors” across the Fediverse.

Maarten’s approach to implementing this was to simply place the JSON document at /.well-known/webfinger on their domain balliauw.be, which allowed @maarten@balliauw.be to become a WebFinger-compatible identity that can be searched for on Mastodon and will return their actual @maartenballiauw.be@mastodon.online profile.

Maarten did however note that since they’re relying on static hosting, they’re unable to restrict what identities they can enforce as valid, and thus a search for @anything@balliauw.be will also return their mastodon.online identity.

The implementation

I wanted to also set up something like this, but without the limitation Maarten had run into. Since my website runs on Netlify, I decided to try out using an Edge Function to build this up.

Similar to Maarten, I first obtained my current Fediverse identity from the Mastodon server I am on: androiddev.social (incredible props to Mikhail for making it a reality).

➜ curl -s https://androiddev.social/.well-known/webfinger?resource=acct:msfjarvis@androiddev.social | jq .
{
  "subject": "acct:msfjarvis@androiddev.social",
  "aliases": [
    "https://androiddev.social/@msfjarvis",
    "https://androiddev.social/users/msfjarvis"
  ],
  "links": [
    {
      "rel": "http://webfinger.net/rel/profile-page",
      "type": "text/html",
      "href": "https://androiddev.social/@msfjarvis"
    },
    {
      "rel": "self",
      "type": "application/activity+json",
      "href": "https://androiddev.social/users/msfjarvis"
    },
    {
      "rel": "http://ostatus.org/schema/1.0/subscribe",
      "template": "https://androiddev.social/authorize_interaction?uri={uri}"
    }
  ]
}

With this in hand, now we can get started on wiring this up into our website.

First, create an Edge Function using the Netlify CLI. Here’s the options I chose.

➜ yarn exec ntl functions:create --name webfinger
? Select the type of function you'd like to create: Edge function (Deno)
? Select the language of your function: TypeScript
? Pick a template: typescript-json
? Name your function: webfinger
◈ Creating function webfinger
◈ Created netlify/edge-functions/webfinger/webfinger.ts
? What route do you want your edge function to be invoked on?: /.well-known/webfinger
◈ Function 'webfinger' registered for route `/.well-known/webfinger`. To change, edit your `netlify.toml` file.

Next, add the following code to the TypeScript file just created for you. I’ve added comments inline to explain what each part of the code does so you can customize it according to your needs.

// Netlify Edge Functions run on Deno (https://deno.land), so imports use URLs rather than package names.
import { Status } from "https://deno.land/std@0.136.0/http/http_status.ts";
import type { Context } from "https://edge.netlify.com";

export default async (request: Request, context: Context) => {
  // We obtain the value of the 'resource' query parameter so that we
  // can ensure a response is only sent for the identity we want.
  const url = new URL(request.url);
  const resourceParam = url.searchParams.get("resource");
  if (resourceParam === null) {
    return context.json(
      {
        error: "No 'resource' query parameter was provided",
      },
      {
        status: Status.BadRequest,
      }
    );
    // I want to be searchable as `@harsh@msfjarvis.dev`, so I only
    // allow requests that set the resource query param to this value.
  } else if (resourceParam !== "acct:harsh@msfjarvis.dev") {
    return context.json(
      {
        error: "An invalid identity was requested",
      },
      {
        status: Status.BadRequest,
      }
    );
  } else {
    // Here's the JSON object we got earlier
    return context.json({
      subject: "acct:msfjarvis@androiddev.social",
      aliases: [
        "https://androiddev.social/@msfjarvis",
        "https://androiddev.social/users/msfjarvis",
      ],
      links: [
        {
          rel: "http://webfinger.net/rel/profile-page",
          type: "text/html",
          href: "https://androiddev.social/@msfjarvis",
        },
        {
          rel: "self",
          type: "application/activity+json",
          href: "https://androiddev.social/users/msfjarvis",
        },
        {
          rel: "http://ostatus.org/schema/1.0/subscribe",
          template: "https://androiddev.social/authorize_interaction?uri={uri}",
        },
      ],
    });
  }
};

And that’s it! You can test it out as below to verify things work as expected.

➜ curl -s https://msfjarvis.dev/.well-known/webfinger | jq .
{
  "error": "No 'resource' query parameter was provided"
}

➜ curl -s https://msfjarvis.dev/.well-known/webfinger?resource=acct:anything@msfjarvis.dev | jq .
{
  "error": "An invalid identity was requested"
}

➜ curl -s https://msfjarvis.dev/.well-known/webfinger?resource=acct:harsh@msfjarvis.dev | jq .
{
  "subject": "acct:msfjarvis@androiddev.social",
  "aliases": [
    "https://androiddev.social/@msfjarvis",
    "https://androiddev.social/users/msfjarvis"
  ],
  "links": [
    {
      "rel": "http://webfinger.net/rel/profile-page",
      "type": "text/html",
      "href": "https://androiddev.social/@msfjarvis"
    },
    {
      "rel": "self",
      "type": "application/activity+json",
      "href": "https://androiddev.social/users/msfjarvis"
    },
    {
      "rel": "http://ostatus.org/schema/1.0/subscribe",
      "template": "https://androiddev.social/authorize_interaction?uri={uri}"
    }
  ]
}

Thanks again to Maarten for doing the initial research for this and writing about it!

Writing Paparazzi tests for your Kotlin Multiplatform projects

Harsh Shandilya — Sun, 26 Jun 2022 00:00:00 +0000

Introduction

Paparazzi is a Gradle plugin and library that enables writing UI tests for Android screens that run entirely on the JVM, without needing a physical device or emulator. This is massive, since it significantly increases the speed of UI tests as well as allows them to run on any CI system, not just ones using macOS or Linux with KVM enabled.

Unfortunately, Paparazzi does not directly work with Kotlin Multiplatform projects so you cannot apply it to a KMP + Android module and start putting your tests in the androidTest source set (not to be confused with androidAndroidTest. Yes, I know). Why would you want to do this in the first place? Like everything cool and new in Android land, Compose! Specifically, compose-jb, JetBrains’ redistribution of Jetpack Compose optimised for Kotlin Multiplatform.

I’ve sent a PR to Paparazzi that will resolve this issue, and in the mean time we can workaround this limitation.

Setting things up

To begin, we’ll need a new Gradle module for our Paparazzi tests. Since Paparazzi doesn’t understand Kotlin Multiplatform yet, we’re gonna hide that aspect of our project and present it a pure Android library project. Set up the module like so:

// paparazzi-tests/build.gradle.kts
plugins {
  id("com.android.library")
  id("app.cash.paparazzi")
}

android {
  buildFeatures { compose = true }
}

Now, add dependencies in this module to the modules that contain the composables you’d like to test. As you might have guessed, this approach currently limits you to only being able to test public composables. However, if you’re trying to test the UI exposed by a “common” module like I am, that might not be such a big deal.

// paparazzi-tests/build.gradle.kts
dependencies {
  testImplementation(projects.common)
}

And that’s pretty much it! You can now be off to the races and start writing your tests:

// paparazzi-tests/src/test/kotlin/UserProfileTest.kt
class UserProfileTest {
  @get:Rule val paparazzi = Paparazzi()

  @Test
  fun light_mode() {
    paparazzi.snapshot {
      MaterialTheme(colorScheme = LightThemeColors) { UserProfile() }
    }
  }

  @Test
  fun dark_mode() {
    paparazzi.snapshot {
      MaterialTheme(colorScheme = DarkThemeColors) { UserProfile() }
    }
  }
}

Consult the Paparazzi documentation for the Gradle tasks reference and customization options.

Recipes

Disable release build type for test module

If you use ./gradlew check in your CI, our new module will be tested in both release and debug build types. This is fairly redundant, so you can disable the release build type altogether:

// paparazzi-tests/build.gradle.kts
androidComponents {
  beforeVariants { variant ->
    variant.enable = variant.buildType == "debug"
  }
}

Running with JDK 12+

You will run into this issue if you use JDK 12 or above to run Paparazzi-backed tests. I’ve started working on a fix for it upstream, in the mean time it can be worked around by forcing the test tasks to run with JDK 11.

// paparazzi-tests/build.gradle.kts
tasks.withType<Test>().configureEach {
  javaLauncher.set(javaToolchains.launcherFor {
    languageVersion.set(JavaLanguageVersion.of(11))
  })
}

Testing with multiple themes easily

Using an enum and Google’s TestParameterInjector you can write a single test and have it run against all your themes.

// paparazzi-tests/src/test/kotlin/Theme.kt
import androidx.compose.material3.ColorScheme

enum class Theme(val colors: ColorScheme) {
  Light(LightThemeColors),
  Dark(DarkThemeColors),
}


// paparazzi-tests/src/test/kotlin/UserProfileTest.kt
@RunWith(TestParameterInjector::class)
class UserProfileTest {
  @get:Rule val paparazzi = Paparazzi()

  @Test
  fun verify(@TestParameter theme: Theme) {
    paparazzi.snapshot(name = theme.name) {
      MaterialTheme(colorScheme = theme.colors) { UserProfile() }
    }
  }
}

Converting Gradle convention plugins to binary plugins

Harsh Shandilya — Sun, 17 Apr 2022 00:00:00 +0000

Introduction

Gradle’s convention plugins are a powerful feature that allow creating simple, reusable Gradle plugins that can be used across your multi-module projects to ensure all modules of a certain type are configured the same way. As an example, if you want to enforce that none of your Android library projects contain a BuildConfig class then the convention plugin for it could look something like this:

com.example.android-library.gradle.kts

plugins {
  id("com.android.library")
}

android {
  buildFeatures {
    buildConfig = false
  }
}

Then in your modules, you can use this plugin like so:

library-module/build.gradle.kts
plugins {
  id ("com.example.android-library")
}

Setting up convention plugins in your project

Gradle’s official sample linked above mentions buildSrc as the location for your convention plugins. I’m inclined to disagree, buildSrc has historically had issues with IDE support and it’s special status within Gradle’s project handling means any change within buildSrc invalidates caches for your entire project resulting in incredible amounts of time lost during incremental builds.

The solution to all of these problems is composite builds, and Josef Raska has a fantastic article that thoroughly explains the shortcomings of buildSrc and how composite builds solve them.

A full explainer on the topic is slightly out of scope for this post, but I can wholeheartedly endorse Jendrik Johannes’ idiomatic-gradle repository as an example of setting up the Gradle build of a real-world project while leveraging features introduced in recent versions of Gradle. I highly recommend also checking out their ‘Understanding Gradle’ video series.

Why would you want to make binary plugins out of convention plugins

First, let’s answer this: what is a binary plugin?

A Gradle plugin that is resolved as a dependency rather than compiled from source is a binary plugin. Binary plugins are cool because the next best thing after a cached compilation task is one that doesn’t exist in the first place.

For most use cases, convention plugins will need to be updated very infrequently. This means that having each developer execute the plugin build as part of their development process is needlessly wasteful, and we can instead just distribute them as maven dependencies.

This also makes it significantly easier to share convention plugins between projects without resorting to historically painful solutions like Git submodules or just straight up copy-pasting.

Publishing your convention plugins

To their credit, Gradle supports this ability very well and you can actually publish all plugins within a build/project with minimal configuration. The changes required to publish Android Password Store’s convention plugins for Android are:

build-logic/android-plugins/build.gradle.kts

-plugins { `kotlin-dsl` }
+plugins {
+   `kotlin-dsl`
+   id("maven-publish")
+}
+
+group = "com.github.android-password-store"
+
+version = "1.0.0"

After that you can run gradle -p build-logic publishToMavenLocal and it will Just Work™️. You can configure additional publishing repositories in similar fashion to how you’d do it for a library project.

If like me you need to publish these to Maven Central, you’ll need slightly more setup since it enforces multiple security and publishing related best practices. Here’s how I use gradle-maven-publish-plugin to configure the same (gradle.properties changes omitted for brevity, the GitHub repository explains what you need):

build-logic/settings.gradle.kts

+pluginManagement {
+   repositories {
+     mavenCentral()
+     gradlePluginPortal()
+   }
+   plugins {
+     id("com.vanniktech.maven.publish.base") version "0.19.0"
+   }
+}
+

build-logic/android-plugins/build.gradle.kts

+import com.vanniktech.maven.publish.JavadocJar
+import com.vanniktech.maven.publish.JavaLibrary
+import com.vanniktech.maven.publish.MavenPublishBaseExtension
+import com.vanniktech.maven.publish.SonatypeHost
+import org.gradle.kotlin.dsl.provideDelegate
+
-plugins { `kotlin-dsl` }
+plugins {
+   `kotlin-dsl`
+   id("com.vanniktech.maven.publish.base")
+   id("signing")
+}
+
+configure<MavenPublishBaseExtension> {
+   group = requireNotNull(project.findProperty("GROUP"))
+   version = requireNotNull(project.findProperty("VERSION_NAME"))
+   publishToMavenCentral(SonatypeHost.DEFAULT)
+   signAllPublications()
+   configure(JavaLibrary(JavadocJar.Empty()))
+   pomFromGradleProperties()
+}
+
+ afterEvaluate {
+   signing {
+     val signingKey: String? by project
+     val signingPassword: String? by project
+     useInMemoryPgpKeys(signingKey, signingPassword)
+   }
+ }

This will populate your POM files with the properties required by Maven Central and sign all artifacts with PGP.

Consuming your new binary plugins

With your convention plugins converted to shiny new binary plugins, you might be inclined to start using them like so:

autofill-parser/build.gradle.kts

plugins {
- id("com.github.android-password-store.published-android-library")
- id("com.github.android-password-store.kotlin-android")
- id("com.github.android-password-store.kotlin-library")
- id("com.github.android-password-store.psl-plugin")
+ id("com.github.android-password-store.published-android-library") version "1.0.0"
+ id("com.github.android-password-store.kotlin-android") version "1.0.0"
+ id("com.github.android-password-store.kotlin-library") version "1.0.0"
+ id("com.github.android-password-store.psl-plugin") version "1.0.0"
}

However, this fails because kotlin-android and kotlin-library plugins resolve to the same binary JAR that encompasses all plugins from the build-logic/kotlin-plugins module and results in a classpath conflict. To better understand how this resolution works, check out the docs on plugin markers.

The way to resolve this problem is to define the plugin versions in your settings.gradle.kts file, where these classpath conflicts will be resolved automatically by Gradle:

settings.gradle.kts

@@ -14,6 +14,25 @@ pluginManagement {
  mavenCentral()
  gradlePluginPortal()
}
+ plugins {
+   id("com.github.android-password-store.kotlin-android") version "1.0.0"
+   id("com.github.android-password-store.kotlin-library") version "1.0.0"
+   id("com.github.android-password-store.psl-plugin") version "1.0.0"
+   id("com.github.android-password-store.published-android-library") version "1.0.0"
+ }
}

And you’re off to the races!

Closing notes

This post was motivated by my goal of sharing a common set of Gradle configurations across my projects such as Android Password Store and Claw, which maintain a nearly identical set of convention plugins shared between the projects that I manually copy-paste back and forth. I’ve extracted the build-logic subproject of APS to a separate aps-build-logic repository, set it up for standalone development and configured publishing support. My goal is to supplement this with a continuous deployment workflow where an automatic version bump + release happens after each commit to the main, after which I can migrate my projects to it.

Backing up your content from Google Photos

Harsh Shandilya — Mon, 04 Apr 2022 00:00:00 +0000

Google Photos has established itself as one of the most popular photo storage services, and like a typical Google service, it makes it impressively difficult to get your data back out of it :D

There are many good reasons why you’d want to archive your pictures outside of Google Photos, having an extra backup never hurts, maybe you want an offline copy for reasons, or you just want to get your stuff out so you can switch away from Google Photos entirely.

How to archive your images from Google Photos

You can use Takeout, except it always strips the EXIF metadata of your images, and often generates incomplete archives. Losing EXIF metadata is a deal-breaker, because you can no longer organize images automatically based on properties like date, location, and camera type.
You can download directly from photos.google.com which preserves metadata, but is embarassingly manual and basically impossible to use if you’re trying archive a few years of history.

So, what’s the solution?

gphotos-cdp

gphotos-cdp is a tool that uses the nearly-perfect method number 2 and makes it automated. It does so by using the Chrome DevTools Protocol to drive an instance of the Google Chrome browser, and emulates all the manual actions you’d take as a human to ensure you get copies of your pictures with all the EXIF metadata retained.

Setting up gphotos-cdp

Disclaimer: I’ve only tested this on Linux. This should be doable on other platforms, but it’s not relevant to my needs so I will not be investigating that.

Ideally you’d want to run this tool on a schedule on a NAS or a server to keep archiving images automatically as they get added to your Google Photos. I personally run this inside a hosted VM on a daily schedule.

For gphotos-cdp to run in a non-interactive manner, it requires your browser data directory with your Google login cookies. You can easily create this with the following command:

google-chrome \
  --user-data-dir=gphotos-cdp \
  --no-first-run \
  --password-store=basic \
  --use-mock-keychain \
  https://photos.google.com/

This will launch Google Chrome with a brand new profile. Login to Photos, and then close the browser. Optionally, re-run the command to ensure that you do not need to login again.

The flags passed to google-chrome are extracted from the default set of parameters used by gphotos-cdp. I wish I could explain why each flag is necessary, but all I know is that it does the trick. I got them from this GitHub comment on the issue tracker for gphotos-cdp.

Once done, you’ll have a gphotos-cdp directory that you’ll need to move to the /tmp directory of whichever machine you wish to run gphotos-cdp on.

gphotos-cdp is written in Golang so you’ll need to install it first. Once done, run the following command to install the latest version of gphotos-cdp

go install github.com/perkeep/gphotos-cdp@latest

Then you can go ahead and start using gphotos-cdp, as given below

~/go/bin/gphotos-cdp \ # go install puts things in ~/go/bin by default
  -v \ # Enable verbose logging
  -dev \ # Enable dev mode which always uses /tmp/gphotos-cdp as the profile directory
  -headless \ # Run Chrome in headless mode so it works on servers and such
  -dldir ~/photos # Download everything to ~/photos

The automation techniques used are not completely reliable and can often fail. You’ll want to implement some kind of retry-on-failure logic to ensure this is run a few times every day.

Monitoring

With anything built on such a brittle foundation, it’s useful to be able to constantly monitor that things are working as they should.

Using healthchecks.io you can easily set up alerts that notify you of failures running the tool or unintentional gaps in the schedule you run the gphotos-cdp on. I use my healthchecks-monitor CLI in a cron job to run gphotos-cdp every day, and healthchecks.io notifies me via Telegram when it fails. The script running in cron looks like this

#!/usr/bin/env bash

HEALTHCHECKS_CHECK_ID=<UUID for check as given on healthchecks> \
HEALTHCHECKS_USERAGENT=crontab \
~/bin/healthchecks-monitor --retries 3 \ # Try running the command thrice before giving up
  --timer \ # Start off a server-side timer on healthchecks
  --logs \ # Record execution logs on healthchecks in case of failure, to help with debugging
  --exec "~/go/bin/gphotos-cdp -v -dev -headless -dldir ~/photos"

Conclusion

As evident, it’s not an easy task to automatically archive your pictures from Google Photos. The setup is tedious and prone to breakage when any authentication related change happens, such as you accidentally logging out the “device” being used by gphotos-cdp or changing your password, in which case you will need to create the gphotos-cdp directory with Chrome again.

Also, the technique in this post could easily stop working at any time if Google chooses to break it. That being said, gphotos-cdp was last updated in 2020 and still continues to function as-is so there is some degree of hope that it can be used for quite a bit more.

Hopefully this setup causes you minimal grief and allows you to back up your precious memories without relying only on Google :)

Building static Rust binaries for Linux

Harsh Shandilya — Sun, 17 Oct 2021 16:31:45 +0000

Rust has supported producing statically linked binaries since RFC #1721 which proposed the target-feature=+crt-static flag to statically link the platform C library into the final binary. This was initially only supported for Windows MSVC and the MUSL C library. While MUSL works for most people, it
has many problems by virtue of being a work-in-progress such as unpredictable performance and many unimplemented features which programs tend to assume are present due to glibc being ubiquitous. In lieu of these concerns, support was added to Rust in 2019 to be able to statically link against glibc.

Unfortunately, if you try to directly use it with RUSTFLAGS='-C target-feature=+crt-static' cargo build there is a good chance you'll run into an error similar to this:

cannot produce proc-macro for `async-trait v0.1.51` as the target `x86_64-unknown-linux-gnu` does not support these crate types

This is a bit of a head scratcher, because the target (your host machine) definitely supports proc-macro crates. Turns out, even Rust contributors were confused by this. The "fix" for this is apparently to pass in the --target explicitly. The reason behind this seems to be a bug with cargo, where the RUSTFLAGS are applied to the target platform only when --target is explicitly provided. Without it, RUSTFLAGS values are set for the host only which results in the errors we see. More details are available Rust issue #78210

Therefore, the correct way to build a statically linked glibc executable for an x86_64 machine is this:

RUSTFLAGS='-C target-feature=+crt-static' cargo build --release --target x86_64-unknown-linux-gnu

Tips and Tricks for GitHub Actions

Harsh Shandilya — Mon, 04 Jan 2021 05:36:23 +0000

GitHub Actions has grown at a rapid pace, and has become the CI platform of choice for most open source projects. The recent changes to Travis CI's pricing for open source is certainly bound to accelerate this even more.

Due to it being a first-party addition to GitHub, Actions has nearly infinite potential to run jobs in reaction to changes on GitHub. You can automatically set labels to newly opened pull requests, greet first time contributors, and more.

Let's go over some things that you can do with Actions, and we'll end it with some safety related tips to ensure that your workflows are secure from both rogue action authors as well as rogue pull requests.

Running workflows based on a cron trigger

GitHub Actions can trigger the execution of a workflow in response to a large list of events as given here, one of them being a cron schedule. Let's see how we can use the schedule feature to automate repetitive tasks.

For Android Password Store, we maintain a list of known public suffixes to be able efficiently detect the 'base' domain of the website we're autofilling into. This list changes frequently, and we typically sync our repository with the latest copy on a weekly basis. Actions enables us to do this automatically:

name: Update Publix Suffix List data
on:
  schedule:
    - cron: '0 0 * * 6'

jobs:
  update-publicsuffix-data:
  # The actual workflow doing the update job

Putting the cron expression into crontab guru, you can see that it executes at 12AM on every Saturday. Going through the merged pull requests in APS, you will also notice that the publicsuffixlist pull requests indeed happen no sooner than 7 days apart.

Mine is a very naive example of how you can use cron triggers to automate parts of your workflow. The Rust project uses these same triggers to implement a significantly more important aspect of their daily workings. Rust maintains a repository called glacier which contains a list of internal compiler errors (ICEs) and code fragments to reproduce each of them. Using a similar cron trigger, this repository checks each new nightly release of Rust to see if any of these compiler crashes were resolved silently by a refactor. When it comes across a ICE that was fixed (compiles correctly or fails with errors rather than crashing the compiler), it files a pull request moving the reproduction file to the fixed pile.

Running jobs based on commit message

Continuous delivery is great, but sometimes you want slightly more control. Rather than run a deployment task on each push to your repository, what if you want it to only run when a specific keyword is in the commit message? Actions has support for this natively, and the deployment pipeline of this very site relies on this feature:

name: Deploy to Cloudflare Workers Sites

on:
  push:
    branches:
      - main

jobs:
  deploy-main:
    if: "contains(github.event.head_commit.message, '[deploy]')"
    # Set up wrangler and push to the production environment

  deploy-staging:
    if: "contains(github.event.head_commit.message, '[staging]')"
    # Set up wrangler and push to the staging environment

This snippet defines a job that is only executed when the top commit of the push contains the text [deploy] in its message, and another that only runs when the commit message contains [staging]. Together, these let me control if I want a change to not be immediately deployed, deployed to either the main or staging site, or to both at the same time. So now I can update a draft post without a full re-deployment of the main site, or make a quick edit to a published post that doesn't need to be reflected in the staging environment.

The core logic of this operation is composed of three parts. The github context, the if conditional and the contains method. The linked documentation for each does a great job at explaining them, and has further references to allow you to fulfill even more advanced use cases.

Testing across multiple configurations in parallel

Jobs in a workflow run in parallel by default, and GitHub comes with an amazing matrix functionality that can automatically generate multiple jobs for you from a single definition. Take this specific example:

	Windows	MacOS	Ubuntu
Stable	Windows + Stable	MacOS + Stable	Ubuntu + Stable
Beta	Windows + Beta	MacOS + Beta	Ubuntu + Beta
Nightly	Windows + Nightly	MacOS + Nightly	Ubuntu + Nightly

This particular matrix is for Rust, to test a codebase across Windows, Ubuntu, and macOS using the Rust stable, beta, and nightly toolchains.

In GitHub Actions, we can simply provide the platforms (Windows, MacOS and, Ubuntu) and the Rust channels (Stable, Beta, and Nightly) inside a single job and let it figure out how to make the permutations and create separate jobs for them. To configure such a matrix, we write something like this:

jobs:
  check-rust-code:
    strategy:
      # Defines a matrix strategy
      matrix:
        # Sets the OSes we want to run jobs on
        os: [ubuntu-latest, windows-latest, macOS-latest]
        # Sets the Rust channels we want to test against
        rust: [stable, beta, nightly]
    # Make the job run on the OS picked by the matrix
    runs-on: ${{ matrix.os }}

    steps:
    - uses: actions-rs/toolchain@v1
      with:
        profile: minimal
        components: rustfmt, clippy
        # Installs the Rust toolchain for the channel picked by the matrix
        toolchain: ${{ matrix.rust }}

This will automatically generate 9 (3 platforms * 3 Rust channels) parallel jobs to test this entire configuration, without requiring us to manually define each of them. DRY at its finest :)

Make a job run after another

By default, jobs defined in a workflow file run in parallel. However, we might need a more sequential order of execution for some cases, and GHA does include support for this case. Let's try another real world example!

LeakCanary has a checks job that runs on each push to the main branch and on each pull request. They wanted to add support for snapshot deployment, in order to finally retire Travis CI. To make this happen, I simply added a new job to the same workflow, having it run only on push events and have a dependency on the checks job. This ensures that there won't be a snapshot deployment until all tests are passing on the main branch. The relevant parts of the workflow configuration are here:

on:
  pull_request:
  push:
    branches:
      - main

jobs:
  checks:
  # Runs automated unit and instrumentation tests

  snapshot-deployment:
  # Only run if the push event triggered this workflow run
  if: "github.event_name == 'push'"
  # Run after the 'checks' job has passed
  needs: [checks]

Mitigating security concerns with Actions

GitHub Actions benefits from a vibrant ecosystem of user-authored actions, which opens it up to equal opportunities for abuse. It is relatively easy to work around the common ones, and I'm going to outline them here. I'm no authority on security, and these recommendations are based on a combination of my reading and understanding. These should be helpful, but this list is not exhaustive, and you should exercise all the caution you can.

Use exact commit hashes rather than tags

Tags are moving qualifiers, and can be force pushed at any moment. If the repository for an Action you use in your workflows is compromised, the tag you use could be force pushed with a malicious version that can send your repository secrets to a third-party server. Auditing the source of a repository at a given tag, then using the SHA1 commit hash it currently points to as the version addresses that concern due to it being nearly impossible to fake a new commit with the exact hash.

To get the commit hash for a specific tag, head to the Releases page of the repository, then click the short SHA1 hash below the tag name and copy the full hash from the URL.

Here, the commit hash is feb985e. Ideally, you want to click that link and copy the full hash from the URL

job:
  checks:
-   - uses: burrunan/gradle-cache-actions@v1.6
 +  - uses: burrunan/gradle-cache-actions@feb985ecf49f57f54f31920821a50d0394faf122

Alternate solution

A more extreme fix for this problem is to vendor each third-party action you use into your own repository, and then use the local copy as the source. This puts you in charge of manually syncing the source to each version, but allows you to restrict the allowed Actions to ones in your repository thereby greatly increasing security. However, having to manually sync can get tedious if your workflows involve a lot of third-party actions. However, the same manual sync also gives you slightly better visibility into the changes between versions since they'd be available in a single PR diff.

To use an Action from a local directory, replace the uses: line with the relative path to the local copy in the repository.

job:
  checks:
    - name: Checkout repository
    # Assuming the copy of actions/checkout is at .github/actions/checkout
-   - uses: actions/checkout@v2
+   - uses: ./.github/actions/checkout

Replace `pull_request_target` with `pull_request`

pull_request_target grants a PR access to a github token that can write to your repository, exposing your code to modification by a malicious third-party who simply needs to open a PR against your repository. Most people will already be using the safe pull_request event, but if you are not, audit your requirements for pull_request_target and make the switch.

-on: [push, pull_request_target]
+on: [push, pull_request]

I'm still learning about Actions, and there is a lot that I did not cover here. I highly encourage readers to refer the GitHub docs for Workflow syntax and Context and expressions syntax to gain more knowledge of the workflow configuration capabilities. Let me know if you find something cool that I did not cover here!

Improvements to inline classes in Kotlin 1.4.30

Harsh Shandilya — Tue, 22 Dec 2020 00:00:00 +0000

What are inline classes?

Inline classes are a Kotlin language feature introduced in Kotlin 1.3 that allow creating no-cost 'wrapper' classes. This is best understood with an example.

Imagine that your project requires an EmailAddress type.

class EmailAddress(val value: String)

This is fine, but there are hidden performance costs. Classes are initialized on the heap, and fields are much cheaper, fields of primitive types even more so. We only really need to be able to call some Strings an EmailAddress, so we can make this class inline.

inline class EmailAddress(val value: String)

This will result in the EmailAddress wrapper being erased at runtime into direct access of the underlying value field. The rules around this are not as simple as "inline means it will always be inlined", and I recommend reading the representation of inline classes documentation to get a better idea of what particular situations the compiler can optimize.

Upcoming improvements for inline classes

With that small refresher on inline classes, let's talk about the restrictions that are being lifted in 1.4.30 and how they are helpful for us as developers.

`init {}` blocks are now allowed

This is pretty straightforward: you can now have initialization logic in inline classes. Let's re-use our email example but now try to perform some validation.

inline class EmailAddress(val value: String) {
  // 🎉 New in Kotlin 1.4.30
  init {
    require(value.isNotEmpty()) { "empty email address does not make sense" }
  }
}

On Kotlin 1.4.20, this will fail to compile with Inline class cannot have an initializer block. This change is valuable because ultimately most of us will be wrapping primitive types into these classes and will frequently have restrictions on what is valid, so being able to validate is extremely helpful to catch invalid values early.

Primary constructor can be made non-public

The primary constructor of an inline class can now be made internal or private, which was previously impossible.

// Sample courtesy https://youtrack.jetbrains.com/issue/KT-28056#focus=Comments-27-4357288.0-0
inline class EmailAddress @PublishedApi
/* 🎉 New in Kotlin 1.4.30 */
internal constructor(private val value: String) {
  override fun toString(): String = value
  companion object {

    fun parse(string: String): EmailAddress {
      check(string.contains('@')) { "invalid email address." }
      return EmailAddress(string)
    }
  }
}

Weeeeeeeeeeeeeeeeeeeeeeeeeee

Yeah. What's happening here is that we've made the constructor of EmailAddress internal to restrict its use to the current module, but annotated it with @PublishedApi which makes it legal to use in public inline functions. Read the docs for the @PublishedApi annotation to get a better idea about what this means for you, since it's out of scope here.

Doing this allows us to have an EmailAddress type that can be used to verify that a String is a valid email and then have that information be carried downstream as the value's type. Why is having the type important? Allow me to refer you to "Aiming for correctness with types", provided you have a free hour or so. It's worth it.

Why upgrade Android?

Harsh Shandilya — Thu, 23 Jul 2020 00:00:00 +0000

A couple days ago I tweeted this out, partly in response to a security conscious user who was quick to point out why a particular feature had to be added to APS, but failed to realise the fact that the problem wouldn't even exist if they were running the latest version of Android (we'll talk about the behavior change that fixed it later here).

Harsh Shandilya

@msf_jarvis

Something that I shouldn't have to be saying: Being concerned about your security and then using a 4 year old version of Android can't really go hand in hand.

19:41 PM - 21 Jul 2020

I completely stand by what I said, and for good reason. Android upgrades bring massive changes to the platform, improving security against both known and unknown threats. You sign off that benefit when you buy into an incompetent OEM's cheap phones, and it has become a bit too 'normal' than anybody would prefer.

That's not what we're going to talk about, though. This post is going to be purely about privacy, and how it has changed, nay, improved, over the years of Android. My apps support a minimum of Android 6, so I will begin with the next version, Android 7, and go through Google's release notes, singling out privacy related changes.

Android 7

Android 7 had a very passing focus on privacy and thus did not have a lot of obvious or concrete changes around it. Background execution limits introduced in Android 6 were improved in Android 7 to apply even more restrictions after devices became stationary, which can be loosely interpreted as 'bad' for data exfiltration SDKs that apps ship but in reality didn't do much.

Android 8

Locking down background location access

In Android 8, access to background location was severely throttled. Apps received less frequent updates for location and thus couldn't track you in real time.

Introduction of Autofill

The Android Autofill framework was debuted, along with support for Web form Autofill. This paved the way for password managers to fill fields for you without relying on hacked up accessibility services or the Android clipboard. This was a major win!

Better HTTPS defaults

Android 8.0's implementation of HttpsURLConnection did not perform insecure TLS/SSL protocol version fallback, which means connections that failed to negotiate a requested TLS version would now abort rather than fall back to an older version of TLS.

ANDROID_ID changes

Access to the ANDROID_ID field was changed significantly. It is generated per-app and per-signature as opposed to the entire system making it harder to fingerprint users who have multiple apps installed with the same advertising-related SDKs.

Android 9

Limited access to sensors

Beginning Android 9, background access to device sensors was greatly reduced. Access to microphone and camera was completely denied, and so was the gyroscope, accelerometer and other sensors of that class.

Granular call log access

For apps that need to access the user's call logs for any reason, a new permission group was introduced. Now, you don't require granting access to all phone-related permissions to let an app back up your call logs.

Restricted access to phone numbers

There are multiple ways to monitor phone calls on Android, and with the introduction of the CALL_LOG permission group, these were locked down to only expose phone numbers to apps that were allowed explicit access to call logs.

Making Wi-Fi and cellular networks less privacy invasive

A combination of changes to what permissions apps require to know about your WiFi and how much personally identifiable data is provided by these APIs further improves your privacy against rogue apps. Disabling device location now disables the ability to get information on cell towers your phone is connected to.

No more serials

Requesting access to the device serial number now requires phone state permissions making it more explicit when apps are trying to fingerprint you.

Android 10

Scoped Storage

Probably the most controversial change in 10, Scoped Storage segregated the device storage into scopes and gave apps access to them without needing extra permissions.

Explicit background permission access

Android 10 introduces the ACCESS_BACKGROUND_LOCATION permission and completely disables background access for apps targeting SDK 29 that don't declare it. For older apps, the framework treats granting location access as effectively background location access. When the app upgrades to target SDK 29, the background permission is revoked and must be explicitly requested again.

Removal of contacts affinity

Beginning Android 10, the system no longer keeps track of what contacts you interact with most and thus search results are not weighted anymore.

MAC randomization enabled by default

Connecting to a Wi-FI network now uses a randomized MAC address to prevent fingerprinting.

Removal of access to non-resettable identifiers

Access to identifiers such as IMEI and serial was restricted to priviledged apps which means apps served by the Play Store can no longer see them.

Restriction on clipboard access

This is the problem we first talked about. Apps before Android 10 could monitor clipboard events and potentially exfil confidential data like passwords. In Android 10 this was completely disabled for apps that were not in foreground or not your active input method. This change was made with no compatibility changes, which means even older apps would not be able to access clipboard data out of turn.

More WiFi and location improvements

Apps can no longer toggle WiFi or read a list of configured networks, and getting access to methods that expose device location requires the ACCESS_FINE_LOCATION permission to make it obvious that an app is doing it. The last change also affects telephony related APIs, a full list is available here.

Permissions controls

Apps no longer have silent access to screen contents, and the platform now prompts users to disallow permissions for legacy apps that target Android 5.1 or below that would earlier be granted at install time. Physical activity recognition is now given its own permission and common libraries for the purpose like Google's Play Services APIs will now send empty data when an app requests activity without the permissions.

Android 11 (tentative)

Storage changes

Apps targeting Android 11 are no longer allowed to opt out of scoped storage.
All encompassing access to a large set of directories and files is completely disabled, including the root of the internal storage, the Download folder, and the data and obb subdirectories of the Android folder.

Permission changes

Location, microphone and camera related permissions can now be granted on a one-off basis, meaning they'll automatically get revoked when the app process exits.
Apps that are not used for a few months will have their permissions automatically revoked.
A new READ_PHONE_NUMBERS permission has been added to call certain APIs that expose phone numbers.

Location changes

One time access is now an option for location, allowing users to not grant persistent access when they don't wish to.
Background location needs to be requested separately now and asking for it together with foreground location will throw an exception.

Data access auditing

To allow apps to audit their own usage of user data, a new callback is provided. Apps can implement it and then log all accesses to see if there's any unexpected data use that needs to be resolved.

Redacted MAC addresses

Unpriviledged apps targetting SDK 30 will no longer be able to get the device's real MAC address.

Closing notes

As you can tell, improving user privacy is a constant journey and Android is doing a better job of it with every new release. This makes it crucial that you stay up-to-date, either by buying phones from an OEM that delivers timely updates for a sufficiently long support period, or by using a trusted custom ROM like GrapheneOS or LineageOS.

Oh JavaScript...

Harsh Shandilya — Sat, 11 Jul 2020 07:19:26 +0000

Harsh Shandilya

@msf_jarvis

Them: "You don't need a strongly-typed language"
Also them:

06:47 AM - 11 Jul 2020

Switching my email to Purelymail

Harsh Shandilya — Mon, 13 Apr 2020 11:42:27 +0000

Email is a very crucial part of my workflow, and I enjoy using it (and also why I'm beyond excited for what Basecamp has in store with hey.com). I have switched emails a couple times over the many years I have had an internet presence, finally settling on me@msfjarvis.dev when I bought my domain. There began the problem.

I attempt to self-host things when reasonable, to retain some control and not have a single point of failure outside my control that would lock me out. With email, that part is a constant, uphill battle against spam filters to ensure your domain doesn't land in a big filter list that will then start trashing all your email and make life hard. Due to this, I never self-hosted email, instead choosing to forward it through Google Domains (the registrar for this domain) to my existing Google account. While this is a very reliable approach, it still involves depending heavily on my Google account. This has proven to be a problem in many ways, including being locked out after opting into Advanced Protection and people's accounts being banned for a number of reasons completely unrelated to email. If something like this were to happen to me, I would lose both my Google as well as my domain email instantly. A very scary position to be in for anybody.

A couple days ago, Jake Wharton retweeted a blog post from Roland Szabo titled 'Moving away from GMail'. I read through it, looked at PurelyMail, and was convinced that it was really the solution for my little problem. I am a big believer in paying in dollaroos rather than data so I really loved the transparency behind pricing, data use, infrastructure and just about everything else. Signed up!

Migration

Like any other email provider, all you need to configure for PurelyMail to work is DNS. I use Cloudflare for my sites, so there was nothing to do on the Google Domains side of things. I left the forwarding setup as-is to allow any lagging DNS resolvers to still be able to get email to me, even if its to my Google account. I hope to get rid of that setting in the near future since I believe the change will have propagated by then. I maintain my DNS settings under a git repository, using StackExchange's excellent dnscontrol tool. DNSControl operates on a JS-like syntax that is parsed, evaluated and then used to publish to the DNS provider of choice. Neat stuff! The changes required looked something like this:

diff --git dnsconfig.js dnsconfig.js
index 29b8d1a927ab..01ea2af1d448 100644
--- dnsconfig.js
+++ dnsconfig.js
@@ -6,7 +6,7 @@

 var REG_NONE = NewRegistrar('none', 'NONE');
 var DNS_CF = NewDnsProvider('cloudflare', 'CLOUDFLAREAPI');
+var CF_PROXY_OFF = {'cloudflare_proxy': 'off'};     // Proxy disabled.

@@ -18,25 +18,17 @@ var RECORDS = [
+    CNAME('_dmarc', '_dmarcroot.purelymail.com.', CF_PROXY_OFF),
+    CNAME('purelymail1._domainkey', 'key1._dkimroot.purelymail.com.', CF_PROXY_OFF),
+    CNAME('purelymail2._domainkey', 'key2._dkimroot.purelymail.com.', CF_PROXY_OFF),
+    CNAME('purelymail3._domainkey', 'key3._dkimroot.purelymail.com.', CF_PROXY_OFF),
-    MX('@', 10, 'alt1.gmr-smtp-in.l.google.com.', TTL('1d')),
-    MX('@', 20, 'alt2.gmr-smtp-in.l.google.com.', TTL('1d')),
-    MX('@', 30, 'alt3.gmr-smtp-in.l.google.com.', TTL('1d')),
-    MX('@', 40, 'alt4.gmr-smtp-in.l.google.com.', TTL('1d')),
-    MX('@', 5, 'gmr-smtp-in.l.google.com.', TTL('1d')),
-    TXT('@', 'v=spf1 include:_spf.google.com ~all', TTL('3600s')),
+    MX('@', 50, 'mailserver.purelymail.com.'),
+    TXT('@', 'v=spf1 include:_spf.purelymail.com ~all'),
+    TXT('@', 'purelymail_ownership_proof=**redacated**'),
 ];

The only 'unexpected' change I had to make was to disable Cloudflare's proxy feature for the CNAME records. Once that was done, PurelyMail was instantly able to verify all DNS records and I was in business.

Pros and Cons of the switch

I've been on PurelyMail for about a day now and poked around enough to have a comprehensive idea of what's different from my usual GMail flow, so let's get into that.

Pros

You pay for it

By now everybody must have realized a simple fact: If you're not paying, you're the product. I do not wish to be a product. Your hard-earned money is more likely to keep companies from being shady than your emails. PurelyMail is a one man operation, which makes it more trustworthy to me than Google's massive scale. Google does not care for a single user, PurelyMail will.

Transparency

PurelyMail tells you upfront about what they charge and how they arrive at that number. There is no contract period, and if you wish to have fine grained control over what you pay, you can use their advanced pricing section to calculate your costs based on your exact needs. The website is straightforward and to the point, there is no glossy advertising to obscure flaws, and their security practices are all documented on their site, front and center.

Failsafe

My GMail is tied to my Google account, which means anything that flags my Google account will bring down my ability to have email. This is a scary position to be in. Having my email separate from my Google account frees me from that looming danger.

Easy export

PurelyMail has a tool called mailPort that lets you move email between PurelyMail and other providers. You can bring your entire mailbox to PurelyMail when switching to it, or back to wherever you go next should it not feel sufficient for your needs. No questions asked, and no bullshit. It just works.

No client lock-in

Because PurelyMail has no bells and whistles, you won't be penalized on the feature side of things if you use one client compared to another. Things stay consistent.

Cons

You pay for it

I am in a fortunate position where I can pay for things solely based on principle, without having to worry too much. Not everybody is similarly blessed, or you may simply have technical issues with being able to pay online for internet things. Stripe and PayPal are not available globally, and fees are often insane. I completely understand.

Roundcube is great, but it ain't no GMail

PurelyMail uses the Roundcube frontend for its webmail offering, with a couple extra themes. It's not the prettiest, and does not have a lot of bells and whistles that you might get accustomed to from GMail. The change is a bit rough honestly, but the pros certainly outweigh the cons. On the bright side, its easier to influence product direction at PurelyMail, so get on the issue tracker and request or vote for features!

No dedicated client

Not having a specialized client unfortunately also means that you'll have to shop around for what works. I still use the GMail mobile app, but K-9 Mail is also pretty decent.

Conclusion

I have begun moving my various accounts to my domain mail as and when they remind me of their existence (55 left still, if my pass repository is to be believed), and hope to eventually be able to get by without the pinned GMail tab in my browser :)

PurelyMail has proven to be an excellent platform so far. Support has been swift and helpful, and I haven't had any bad surprises. I hope to be a content user for as long as I possibly can :)

Forem: Harsh Shandilya

Tips and tricks for using Renovate

Disabling updates for individual packages

Grouping updates together

Set a semver range for upgrades

Supporting non-standard dependency declarations

Making your GitHub Actions usage more secure

Automatically merging compatible updates

Closing notes

Writing your own Nix Flake checks

Preface

The premise

The thing you’re here for

The final result

Mastodon on your own domain without hosting a server, Netlify edition

Preface

The implementation

Writing Paparazzi tests for your Kotlin Multiplatform projects

Introduction

Setting things up

Recipes

Disable release build type for test module

Running with JDK 12+

Testing with multiple themes easily

Converting Gradle convention plugins to binary plugins

Introduction

Setting up convention plugins in your project

Why would you want to make binary plugins out of convention plugins

Publishing your convention plugins

Consuming your new binary plugins

Closing notes

Backing up your content from Google Photos

How to archive your images from Google Photos

gphotos-cdp

Setting up gphotos-cdp

Monitoring

Conclusion

Building static Rust binaries for Linux

Other potential problems

Tips and Tricks for GitHub Actions

Running workflows based on a cron trigger

Running jobs based on commit message

Testing across multiple configurations in parallel

Make a job run after another

Mitigating security concerns with Actions

Use exact commit hashes rather than tags

Alternate solution

Replace pull_request_target with pull_request

Improvements to inline classes in Kotlin 1.4.30

What are inline classes?

Upcoming improvements for inline classes

init {} blocks are now allowed

Primary constructor can be made non-public

Why upgrade Android?

Android 7

Android 8

Locking down background location access

Introduction of Autofill

Better HTTPS defaults

ANDROID_ID changes

Android 9

Limited access to sensors

Granular call log access

Restricted access to phone numbers

Making Wi-Fi and cellular networks less privacy invasive

No more serials

Android 10

Scoped Storage

Explicit background permission access

Removal of contacts affinity

MAC randomization enabled by default

Removal of access to non-resettable identifiers

Restriction on clipboard access

More WiFi and location improvements

Permissions controls

Android 11 (tentative)

Storage changes

Permission changes

Location changes

Data access auditing

Replace `pull_request_target` with `pull_request`

`init {}` blocks are now allowed