Forem: Tony Robalik

Enshittification: Why I’m leaving dev.to

Tony Robalik — Sun, 30 Nov 2025 20:47:58 +0000

With thanks to The Oatmeal for the cover art.

Please consider reading this post on autonomousapps.com instead.

This will be my last post on the site dev.to. I’ve been posting my blog to that site since April 2018. My very first post was written when I was the Android team lead at Chess.com, titled Rewriting Chess.com’s Android app. Over the years, I’ve been more or less active, and have published 42 posts in all (counting this one).¹

I’ve had concerns about the long-term viability of my presence on that site for a few years, as I’ve observed layers and layers of cruft accumulate around my actual content. I always had higher priorities though, and justified the cruft by recognizing that the site is providing a free service, and it frankly makes posting a tech blog to the internet fairly easy.

The last straw came when, after my most recent post, two separate people contacted me about their experience trying to read it (one of them twice). First I was told that there’s a kind of soft login wall when visiting the post. I wasn’t aware of this because, as a writer on the site, I’m always logged in. The wall opens when interacting with one of those annoying popups so common on the Web these days:

Note there’s only one option. Clicking it leads to this:

Which fortunately can be dismissed. This is a dark pattern.

Then later, someone sent me a screenshot of what it looks like after the login cruft has been brushed away:

I’ve helpfully highlighted the portions of this screencap that are actual content. 21% of the screen real estate! And more than half the remaining is an ad for some bullshit “AI” product.

Let me state this very clearly: fuck “AI.” It’s pure marking speak for a set of technologies founded upon harm, whose primary use-cases are harm. I want nothing to do with it.

An auspicious number. ↩

Is the Java ecosystem cursed? A dependency analysis perspective

Tony Robalik — Mon, 24 Nov 2025 20:15:12 +0000

I am the author of the moderately popular (⭐ 2k) Dependency Analysis Gradle Plugin, a static analysis tool that helps Gradle build authors maintain a healthy dependency graph. I also maintain some of the largest Gradle repos on the planet: a Kotlin backend repo with over 2500 subprojects, and an Android repo with more than 7200 subprojects (both proprietary). I have… seen some shit.

Note: I refer to both the cases above as being part of the "Java ecosystem," though both use Kotlin as the preferred language, and one runs on the JVM while the other runs on ART (the Android runtime) on mobile devices.

I come to you with a simple proposition: I believe the Java ecosystem is cursed. Hear me out.

We are cursed with…

Lying metadata, overuse of "fat" jars with underuse of package relocation, split packages, undocumented usage of reflection to access upstream dependencies, usage of terms like "upstream" that have different meanings in different contexts, misuse of protobuffers, different compilers with different notions of their obligations vis-a-vis the Java class file format…

Lying metadata

This was already covered in-depth in This is why we can't have nice things: When POM files lie, but the summary is: sometimes dependencies have hand-written metadata, which is certainly A Choice given that build tools exist. I suppose it's harder to teach a build tool to lie.

It's just a list, man

Despite the bewildering complexity of dependency resolution engines in tools like Gradle and Maven, at the end of the day a classpath is just a list of class files (and jars that package class files). When your running program "sees" a class or interface for the first time, it has to load it. It does this with a ClassLoader. The classloader searches the classpath (just a list of class files!)¹ and picks the first class file that matches the class it just encountered. Importantly, your classpath may have more than one class file for that class. Even well-behaved builds may have this problem, for a variety of reasons, some of which are noted below.

As I was writing this post, I saw yet another reason to fear the classpath, in the November Gradle newsletter: Maven-Hijack: Software Supply Chain Attack: Exploiting Packaging Order. Bad actors can make use of this fundamental property of the JVM to insert malicious code into your applications. Or, as we'll see below, you can just do it to yourself!

Fat jars without package relocation

Shadow is a powerful tool for creating "uber" or "fat" jars, which are jars that contain all their external dependencies rather than relying on a classpath. This can simplify deployments of applications since deployers only need to worry about a single jar instead of dozens, hundreds, or thousands of jars. This is fine. It becomes cursed when libraries make use of this tool, resulting in broken classpaths that contain duplicate class files such that runtime behavior is dependent on the classpath's order. I would like to point the maintainers of these libraries at Shadow's powerful relocation abilities, which enable it to change the package of bundled classes such that there can be no duplicate class problem.

As I've said before, the extent to which Java's packages exist in a global namespace is not well-appreciated.

Split packages

As will be discussed tangentially below, the existence of split packages complicates dependency analysis because it makes it harder to connect class names with the modules that provide them, since there is now a 1-to-many relationship between packages and modules.

I mostly work in Kotlin repos, both backend and Android, neither of which use JPMS (the Java Platform Module System). I can't say from direct experience how widely used is JPMS in the pure Java world, but as the maintainer of an increasingly complicated static analysis tool, I can say I wish more projects used it.

(Kotlin users would say they get the benefits of JPMS thanks to the internal visibility modifier, but they're wrong.)

Funhouse mirrors (aka reflection)

The com.amazonaws:aws-java-sdk-core has a method, getProfileCredentialService(), which uses reflection to access a class from the com.amazonaws:aws-java-sdk-sts library. This is a compound curse, composed of these properties:

com.amazonaws:aws-java-sdk-sts depends on com.amazonaws:aws-java-sdk-core, not the other way around.
Triggering the code path for getProfileCredentialService() will throw an exception if com.amazonaws:aws-java-sdk-sts is not on the classpath, raising the question of why the dependencies are structured this way.
The Java ecosystem has first-class functionality, Service Loaders, for dynamically instantiating something that might or might not be on the classpath.

The Dependency Analysis Gradle Plugin has had support for Service Loaders since the beginning of its existence. First-class features such as service loading are great for static analysis tools like DAGP, as they give it well-known places to search during analysis. Ad hoc approaches like reflection are trickier, and require substantially more complex approaches to handle. DAGP added support for Class.forName("...") in v3.3.0. Pre-3.3.0, DAGP would suggest removing the sts dependency as unused if it couldn't detect any direct reference to any of the classes it provides in the bytecode, leading to runtime failures, either in CI (ok) or post-deployment (bad).

Protocol Buffers

Protocol buffers, aka protobufs, are an amazing tool for making a build engineer's days a living nightmare. First we must note that there are at least two competing protobuf compilers in the JVM world: Google's protoc and Square's Wire. I happen to work at a company that uses both. I don't think I hate myself, but maybe God does. These compilers generate code (Java or Kotlin) from the protobuf format that are mutually incompatible without adapters,² meaning that once you have both in your codebase, you will probably always have both—congrats.

I have also seen several modules with both plugins in use simultaneously. Well.

I work with Gradle. Each of the competing compilers comes with a Gradle plugin. I may be slightly biased, but I think the Wire Gradle Plugin is better. Nevertheless, the relative ease with which either can be configured leads to Fun Situations such as: two modules can each depend on the same proto files, possibly at different versions, leading to generated code with the same exact class name but different definitions. And now if you have a third module that depends on these two modules, you're in a situation where your module may fail to compile if you just so happen to change the order of your dependency declarations, or worse, it may compile in both cases but fail at runtime for a similar reason. This is because, as discussed above, a classpath is just a collection of jars and class files, and whichever class file gets loaded first wins forever.

I have now worked in two separate extremely large codebases that have significant usage of protos, and it is no exaggeration to say that dealing with them is almost the worst part of my job ("AI" has recently taken that crown).

Yolo compilers

It turns out that different compilers have different ideas of what the resultant class files should look like. Chapter 4 of the JVM specification discusses the Constant Pool. class files contain a table, known as the constant_pool, which contains a reference to every constant present in the source code of a Java file. This is useful for static analysis because the various JVM compilers all³ inline constants for runtime efficiency. This means that a constant like public static final String CONSTANT = "magic" gets turned into simply "magic" at the use-site, and similarly for Kotlin's const val. Therefore simply analyzing the bytecode directly with a tool like asm won't enable static analysis tools to connect the user of a constant to the maybe-separate module that provides the constant. Thanks to the constant pool, however, we can see the full reference to the provider and make the connection.

This only works for class files compiled with javac, however. For both kotlinc and ec4j (the Eclipse compiler for Java, yes this does exist and Real Teams in the world rely on it), keeping these full references to inlined constants in the constant pool is considered unnecessary.

The Dependency Analysis Gradle Plugin has a class, ConstantPoolParser, which parses the constant pool of a class file and extracts the set of class file references for all the class's inlined constants. When passed a reference to a class file compiled with something other than javac, the returned set is empty. This leads to "unused dependency" false positives, when a dependency is only used for the constants it contains, a surprisingly common situation.

As a consequence, DAGP utilizes some heuristics to try to workaround this situation—with imperfect results. I won't go into details here, but they involve parsing source code for import statements,⁴ looking at the ldc bytecode instruction,⁵ etc. Source parsing falls over in the presence of split packages, and the ldc bytecode only provides the constant value, not its name. Together, the heuristics get most of the way there, and it's unlikely the tool will get more accurate here without much more sophisticated source code analysis. Happy to work on that if you want to fund me!

Special thanks

Special thanks to Luis Cortés once again for the thorough review!

Who stalks us in the darkness?

The above list of ~~grievances~~ curses should not be taken as comprehensive. It is merely the list of things that have most recently destroyed my will to live.⁶

…wait, who is that behind me, in the dark wood…?

Baba Yaga (no not that one) This is what I think of when I imagine the Kotlin compiler given human form.

I'm eliding some complexity around the classloader hierarchy, and the possibility you may have a custom classloader that doesn't follow standard behavior. See A crash course in classpaths for more information. ↩
To be clear, this is not about Java/Kotlin interop, but the fact that each compiler (protoc and wire) simply emit different code from the same protobuf schema. ↩
I think they all do, but I haven't checked exhaustively. At least java, ec4j, and kotlinc do. ↩
See here for where DAGP parses source code in a very simplified way. ↩
This post is already too long for me to explain in depth what I mean here. You can see where DAGP visits the LDC instruction here. ↩
Kidding! Once again the thing that's killing my will to live is just "AI." ↩

This is why we can't have nice things: When POM files lie

Tony Robalik — Fri, 07 Feb 2025 23:54:47 +0000

Photo by Marco Bicca on Unsplash. I really hope there's a light at the end of this sewer.

Sorry, my country is dissolving into a Nazi sewer of world-historical proportions and so I'm dealing with my otherwise fruitless rage by yelling about JVM things.

The fact that Java classes exist in a global namespace is not well-appreciated, even by vendors of major parts of the ecosystem (apparently).

Caused by: java.lang.ClassCastException: class com.google.common.graph.ImmutableGraph cannot be cast to class com.google.common.graph.SuccessorsFunction (com.google.common.graph.ImmutableGraph and com.google.common.graph.SuccessorsFunction are in unnamed module of loader org.gradle.internal.classloader.VisitableURLClassLoader$InstrumentingVisitableURLClassLoader @4617120)

A confusing error at the best of times, but to a build engineer (i.e. me) trying to help out during a SEV, a despair-inducing example of how Dependency Management on the JVM is Completely Broken, What Are We Even Doing Here.

A ClassCastException always means Someone Somewhere Hates You, or at least values their own KPIs more than not polluting the entire goddamn ecosystem.

Let's debug! My first step is to navigate to the Graph class and check its hierarchy. I can confirm that, yes, in Guava 33.3.1-jre at least, that class does indeed implement the SuccessorsFunction interface. It should not be throwing a ClassCastException!

Next step. In Intellij (the vendor which is ironically the source of this amongst most of the rest of my woes), I set a breakpoint at the exception and evaluate the following expression.¹

// `this` is an instance of the `com.google.common.graph.Graph` class
this.javaClass.superclass.superclass.superclass.interfaces.first().protectionDomain.codeSource.location

That expression resolves to this:

file:/Users/<<ME!>>/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-compiler/2.0.21/88f09afc2536e38d528e78eb8349504de10ac436/kotlin-compiler-2.0.21.jar

What the fuck! That is not the right jar!! Let's double-check:

jar tf path/to/kotlin-compiler-2.0.21.jar
…
com/google/common/graph/Graph.class

😭

Maybe it's a bug! Let's look at the latest version of the jar at time of writing, 2.2.10 (nope, same problem). Not only has Jetbrains, the inventor of the Kotlin language, released a library that is bundling Guava (among many other things!!) in a fatjar, they're not even bothering to relocate the damn packages. Ok, I'm curious, what does the pom.xml have to say for itself?

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <modelVersion>4.0.0</modelVersion>
  <groupId>org.jetbrains.kotlin</groupId>
  <artifactId>kotlin-compiler</artifactId>
  <version>2.1.10</version>
  <name>Kotlin Compiler</name>
  <description>Kotlin Compiler</description>
  <url>https://kotlinlang.org/</url>
  <licenses>
    <license>
      <name>The Apache License, Version 2.0</name>
      <url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
    </license>
  </licenses>
  <developers>
    <developer>
      <name>Kotlin Team</name>
      <organization>JetBrains</organization>
      <organizationUrl>https://www.jetbrains.com</organizationUrl>
    </developer>
  </developers>
  <scm>
    <connection>scm:git:https://github.com/JetBrains/kotlin.git</connection>
    <developerConnection>scm:git:https://github.com/JetBrains/kotlin.git</developerConnection>
    <url>https://github.com/JetBrains/kotlin</url>
  </scm>
  <dependencies>
    <dependency>
      <groupId>org.jetbrains.kotlin</groupId>
      <artifactId>kotlin-stdlib-jdk8</artifactId>
      <version>2.1.10</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>org.jetbrains.kotlin</groupId>
      <artifactId>kotlin-script-runtime</artifactId>
      <version>2.1.10</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>org.jetbrains.kotlin</groupId>
      <artifactId>kotlin-reflect</artifactId>
      <version>1.6.10</version>
      <scope>compile</scope>
      <exclusions>
        <exclusion>
          <groupId>*</groupId>
          <artifactId>*</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.jetbrains.intellij.deps</groupId>
      <artifactId>trove4j</artifactId>
      <version>1.0.20200330</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>org.jetbrains.kotlinx</groupId>
      <artifactId>kotlinx-coroutines-core-jvm</artifactId>
      <version>1.6.4</version>
      <scope>compile</scope>
    </dependency>
  </dependencies>
</project>

There is no indication that this is a fatjar in the metadata, not to mention the hilarious fact that they are declaring a dependency on kotlin-reflect 1.6.10. But someone at some point noticed that this was Causing A Problem, so they excluded all of its dependencies. Are you kidding me. Guys, you publish a BOM, just use it!

It is entirely unclear why some dependencies are being declared and others are being silently bundled into the final artifact. Maybe I can get an answer at this issue I raised.

Why am I even using this dependency?

Well, I had been using kotlin-compiler-embeddable for source-code analysis,² which at least relocates its bundled classes. However, using that in the same project that has a dependency on the Kotlin Gradle Plugin leads to this build warning:

w: The artifact `org.jetbrains.kotlin:kotlin-compiler-embeddable` is present in the build classpath along Kotlin Gradle plugin. 
This may lead to unpredictable and inconsistent behavior.   
For more details, see: https://kotl.in/gradle/internal-compiler-symbols

A search online for a suitable replacement for kotlin-compiler-embeddable turned up this on discuss.kotlinlang.org:

kotlin-compiler-embeddable should be used in scenarios when it’s necessary to have the compiler packaged as a single jar with no external dependencies. In all other cases, use the regular kotlin-compiler.

That answer implies that kotlin-compiler does have external dependencies (which is true), while also further implying that it abides by the standard JVM library contract by Actually Declaring all its dependencies (which is false).³

Coping

This raises the question, how can we cope with this? Well, assuming your use-case for this is in a Gradle task and your task is relatively well-written, you can migrate the task action to use the Worker API with classloader isolation. The link provided in the warning does a fair job of explaining how to do that, assuming you're ok with drawing the owl.

Ah, fuck it, let's draw the owl

Drawing the build script

// plugin/build.gradle.kts
…etc…

dependencies {
  implementation(project(":shared-lib")) {
    exclude(group = "org.jetbrains.kotlin", module = "kotlin-compiler")
  }
}

Jetbrains' docs suggest using compileOnly("org.jetbrains.kotlin:kotlin-compiler-embeddable:2.1.10"), which isn't wrong, but also won't work if that lib is actually coming from a transitive dependency.

Drawing the task

abstract class OwlTask @Inject constructor(
  private val workerExecutor: WorkerExecutor
) : DefaultTask() {

  @get:Classpath 
  abstract val kotlinCompiler: ConfigurableFileCollection

  @get:Input
  abstract val otherInput: Property<Boolean>

  @TaskAction fun action() {
    workerExecutor
      .classLoaderIsolation { spec ->
        // please note that from() adds to the
        // classpath, while setFrom() would
        // completely override it.
        spec.classpath.from(kotlinCompiler)
      }
      .submit(Action::class.java) { params ->
        params.otherInput.set(otherInput)
      }
  }

  interface Parameters : WorkParameters {
    val otherInput: Property<Boolean>
  }

  abstract class Action : WorkAction<Parameters> {
    override fun execute() {
      val otherInput = parameters.otherInput.get()
      …
    }
  }
}

Drawing the plugin

abstract class OwlPlugin : Plugin<Project> {
  override fun apply(target: Project) {
    // Use the project's version of Kotlin, if present. Else default to what we use.
    val kotlinVersion = …some method for getting the version of Kotlin you want…
    val dependencyScope = project.configurations.dependencyScope("owl").get()
    val resolvable = project.configurations.resolvable("owlClasspath") {
      extendsFrom(dependencyScope)
    }

    project.dependencies.add(dependencyScope.name, "org.jetbrains.kotlin:kotlin-compiler:$kotlinVersion")

    project.tasks.register("owl", OwlTask::class.java) { t ->
      t.kotlinCompiler.setFrom(resolvable)
      t.otherInput(true) // demonstration purposes only
    }
  }
}

Hello, owl!

I can confirm this works, it eliminated the ClassCastException. Now, a day and incredible amounts of frustration later, I guess I can go back to my original problem?

I'm just some asshole

Look, I'm just some asshole who always tries to do the right thing. But what the fuck am I supposed to do when larger and better-resourced teams just abdicate responsibility? Fuck! I guess I'll write a blog post!

Special thanks

I am much obliged Luis Cortes, who convinced me that some of my salty language was in fact acidic. All the remaining salt and/or acid is entirely of my own devising.

Endnotes

¹ Guava's Graph class has a complicated hierarchy! up

² This dependency exhibits many of the same metadata problems. up

³ To be fair, this is a very widespread problem. up

Gradle extensions part 2: Now with shenanigans

Tony Robalik — Wed, 11 Dec 2024 23:05:12 +0000

Photo by Karsten Würth on Unsplash

Welcome to the spiritual successor to Gradle plugins and extensions: A primer for the bemused (one of my most popular posts, such that it competes for space with actual Gradle documentation at the top of a Google search).

As part of my long-running quest to Destroy buildSrc With Fire, I have recently had occasion to learn how to add extensions to other kinds of types, such as tasks. We have code like this duplicated across many many repos that are under our care:

// buildSrc/src/main/kotlin/magic/Magic.kt
package magic

object Magic {
  fun shouldPracticeTheDarkArts(): Boolean {
    return System.getenv("DO_ANCIENT_MAGIC")?.toBoolean()
      ?: System.getenv("DO_SLIGHTLY_MORE_MODERN_MAGIC")?.toBoolean()
      ?: false
  }
}

This code is used in build scripts like this:

// build.gradle.kts
import magic.Magic

tasks.withType<Test>().configureEach {
  if (Magic.shouldPracticeTheDarkArts()) {
    logger.quiet("👻")
  }
}

There are several things about this that I'd like to improve:

I don't want this code in buildSrc. I want a version of it in our build-logic that is under test and which is shared widely (instead of duplicated in a dozen different repos).
I don't like the import. It is Unclean. (Build scripts should be simple, declarative, easy for tools to parse.)
I'm not a big fan of calling System.getenv() in a Gradle context. I prefer to use the ProviderFactory.

Extending `Test` tasks

Many Gradle types, including all Tasks (and of course the Project type), are ExtensionAware. This means they all have an ExtensionContainer available on which new extensions can be created and added.

// build-logic/src/main/kotlin/magic/TestMagicExtension.kt
package magic

abstract class TestMagicExtension @Inject constructor(
  private val providers: ProviderFactory
) {

  internal companion object {
    const val NAME = "magic"

    fun create(
      testTask: Test,
      providers: ProviderFactory,
    ) {
      testTask.extensions.create(
        NAME, 
        TestMagicExtension::class.java, 
        providers,
      )
    }
  }

  fun shouldPracticeTheDarkArts(): Boolean {
    return providers
      .environmentVariable("DO_ANCIENT_MAGIC")
      .orElse(providers.environmentVariable("DO_SLIGHTLY_MORE_MODERN_MAGIC"))
      .map { it.isNotEmpty() }
      .getOrElse(false)
  }

And in our plugin, we can add this to all our Test tasks:

project.tasks.withType<Test>().configureEach { t ->
  TestMagicExtension.create(t, project.providers)
}

And now we can update our build scripts:

// build.gradle.kts
import magic.TestMagicExtension

tasks.withType<Test>().configureEach {
  // the "extensions" call is on the Test instance,
  // not the project instance
  val magic = extensions.getByType(TestMagicExtension::class.java)
  if (magic.shouldPracticeTheDarkArts()) {
    logger.quiet("👻")
  }
}

...that's not better at all!

Groovy: An interlude

First of all, let's take a step back and remind ourselves that "we love Kotlin, type safety is great, I don't care that performance is worse..." We can say that over and over again a few times while rocking in a fetal position on the floor till we feel better. Now, here's the same build script but in Groovy:

// build.gradle
tasks.withType(Test).configureEach {
  if (magic.shouldPracticeTheDarkArts()) {
    // I'm being cheeky by also omitting the 
    // "redundant" parentheses
    logger.quiet "👻"
  }
}

Groovy isn't supposed to be better! Damnit!

Sprinkle on some shenanigans

How the heck does Gradle Kotlin DSL do it? Why isn't it generating "typesafe accessors" for my test task extension? Well, that second one is a good question and I have no answer. But for the first... let's just "generate" (that is, write) our own typesafe accessors!

We add some code in a new (to us) package:

package org.gradle.kotlin.dsl

import magic.TestMagicExtension

public val Test.magic: TestMagicExtension
  get() = extensions.getByType(TestMagicExtension::class.java)

public fun Test.magic(configure: TestMagicExtension.() -> Unit) {
  configure(TestMagicExtension.NAME, configure)
}

And now we can update our Kotlin DSL build script:

// build.gradle.kts
tasks.withType<Test>().configureEach {
  if (magic.shouldPracticeTheDarkArts()) {
    logger.quiet("👻")
  }
}

Here we're (ab)using the fact that Gradle automatically imports everything in the org.gradle.kotlin.dsl package into build scripts, so all those functions are Just There (in a global namespace, so be careful!).

This is a common enough pattern that Gradle itself uses it in its test-retry-gradle-plugin. There's also an open issue (from, er, 2018) on Gradle's issue tracker with a feature request to permit custom plugins to add their own default imports without resorting to using split packages like this.

Now go forth and be merry, for it is that time of year.

One click dependencies fix

Tony Robalik — Tue, 08 Oct 2024 18:48:51 +0000

Photo by Maxim Hopman on Unsplash

If you maintain a JVM¹ or Android project, chances are you've heard of the Dependency Analysis Gradle Plugin (DAGP). With over 1800 stars, it's used by some of largest Gradle projects in the world, as well as by Gradle itself. It fills what would otherwise be a substantial hole in the Gradle ecosystem: without it, I know of no other way to eliminate unused dependencies and to correctly declare all your actually-used dependencies. In other words, when you use this plugin, your dependency declarations are exactly what you need to build your project: nothing more, nothing less.

That might sound like a small thing, but for industrial-scale projects, a healthy dependency graph is a superpower that prevents bugs, eases debugging (at build and runtime), keeps builds faster, and keeps artifacts smaller. If developer productivity work is the public health of the software engineering world, then a healthy dependency graph is a working sewer system. You don't know how much you rely on it till it stops working and you've got shit everywhere.

The problem is that, if your tool only tells you all the problems you have but doesn't also fix them, you might have a massive(ly annoying) problem on your hands. I mentioned this as an important consideration in my recent rant against code style formatters. This is why, since v1.11.0, DAGP has had a fixDependencies task, which takes the problem report and rewrites build scripts in-place. Even before that, in v0.46.0, the plugin had first-class support for registering a "post-processing task" to enable advanced users to consume the "build health" report in any manner of their choosing. Foundry (née The Slack Gradle Plugin), for example, has a feature called the "dependency rake", which predates and inspired fixDependencies.

fixDependencies hasn't always worked well, though. For one thing, there might be a bug in the analysis such that, if you "fix" all the issues, your build might break. (DAGP is under very active development, so if this ever happens to you, please file an issue!) In this case, it can take an expert to understand what broke and how to fix it, or you can fall back to manual changes and iteration.

For another thing, the build script rewriter has relied on a simplified grammar for parsing and rewriting Gradle Groovy and Kotlin DSL build scripts. That grammar can fail if your scripts are complex.² This problem will soon be solved with the introduction of a Gradle Kotlin DSL parser built on the KotlinEditor grammar, which has full support for the Kotlin language. (Gradle Groovy DSL scripts will continue to use the old simplified grammar, for now.)

There have also been many recent bugfixes to (1) improve the correctness of the analysis and (2) make the rewriting process more robust in the face of various common idioms. DAGP now has much better support for version catalog accessors, for example (no support yet for experimental project accessors).

With these improvements (real and planned), it's become feasible to imagine automating large-scale dependency fixes across hundreds of repos containing millions of lines of code and have it all just work. Here's the situation:

Over 500 repositories.
Each with its own version catalog.
Most of the entries in the version catalogs use the same names, but there's some incidental skew in the namespace (multiple keys pointing to the same dependency coordinates).
Over 2000 Gradle modules.
Close to 15 million lines of Kotlin and Java code spread out over more than 100 thousand files, along with over 150 thousand lines of "Gradle" code in more than 3 thousand build scripts. This last point isn't as relevant as the first four, but helps to demonstrate what I mean when I say "industrial scale."³

Additionally, the build code we want to write to manage all this should follow Gradle best practices: it should be cacheable to the extent possible, should work with the configuration cache, and for bonus points should not violate the isolated projects contract either (which is also good for maximal performance). The ultimate goal is for developers and build maintainers to be able to run a single task and have it (1) fix all dependency declarations, which might mean adding new declarations to build scripts; (2) all build script declarations should have a version catalog entry wherever possible; (3) and all version catalog entries should come from the same global namespace so that the entire set of 500+ repositories are fully consistent with each other. This last part is an important requirement because we're migrating these repos into a single mono/mega repo for other reasons.

Here's the task they can now run, for the record:

gradle :fixAllDependencies

(nb: we use gradle and not ./gradlew because we manage gradle per-repo with hermit.)

So, how do we do it?

Pre-processing

The first step was creating the global version catalog namespace. We did not attempt to actually create a single published global version catalog because, until we finish our megarepo migration, an important contract is that each repo maintains its own dependencies (and their versions). So instead, we collected the full map of version catalog names to dependency identifiers (the dependency coordinates less the version string). We eliminated all the duplication using pre-existing large-scale change tools we have, and then populated the final global set (now with 1:1 mappings) into our convention plugin that is already applied everywhere.

Conceptual framework

The Gradle framework, in general, takes the Project as the most important point of reference.⁴ A Project instance is what backs all your build.gradle[.kts] scripts, for example, and most plugins implement the Plugin<Project> interface. Safe, performant, high-quality build code respects this conceptual boundary and treats each project (AKA "module") as an atomic unit.

If Tasks have well-defined inputs and outputs (literally annotated @Input<X> and @Output<X>), then it might help to also think of projects as having inputs and outputs. In general, a project's inputs are its source code (which by convention is in the src/ directory at the project root), and its dependencies. A project's outputs are the artifacts it produces. For Java projects, the primary artifacts are jars (for external consumption), or class files (for consumption by other projects in a multi-project build).⁵

With that in mind, we can decide that if two projects need to talk to each other, they should do so via their well-defined inputs and outputs. We define relationships between projects via dependencies (A -> B means A depends on B, so B is an input to A), and we can flavor that connection such that we tell Gradle which of B's outputs A cares about. The default is the primary artifact (usually class files for classpath purposes), but it can also be anything (that can be written to disk). It can, for example, be some metadata about B. It can also be both! (You can declare multiple dependencies between the same two projects, with each edge having a different "flavor," that is, representing a different variant.) This may make more sense in a bit when we get to a concrete example.

Implementation: `:fixAllDependencies`

The rest of this post will focus on implementation, but at a relatively high level of detail. Some of the code will essentially be pseudocode. My goal is to demonstrate the full flow at a conceptual level, such that a (highly) motivated reader could implement something similar in their own workflow or, more likely, simply learn about how to do something Cool™️ with Gradle.

Here's a sketch of the simplified task graph with Excalidraw:

Note how each project is independent of the other. Well-defined Gradle builds maximize concurrency by respecting project boundaries.

Step 1: The global namespace

As mentioned in the pre-processing section, we need a global namespace. We want all dependency declarations to refer to version catalog entries, i.e., libs.amazingMagic, rather than "com.amazing:magic:1.0". Since DAGP already supports version catalog references in its analysis, this will Just Work if your version catalog already has an entry for amazingMagic = "com.amazing:magic:1.0". However, if you don't, DAGP defaults to the "raw string" declaration. If we want, we can tell DAGP about other mappings that it can't detect by default:

// root build script
dependencyAnalysis {
  structure {
    map.putAll(
      "com.amazing:magic" to "libs.amazingMagic",
      // more entries
    )
  }
}

where dependencyAnalysis.structure.map is a MapProperty<String, String>, which you can modify directly in your build scripts or via a plugin. Note the "raw string" version of the declaration doesn't include version information; this is important because the version you declare may not match the version that Gradle resolves.

Step 2: Update the version catalog, part 1

With Step 1, DAGP will rewrite build scripts via the built-in fixDependencies task to match your desired schema, but your next build will fail because you'll have dependencies referencing things like libs.amazingMagic which aren't actually present in your version catalog. So now we have to update the version catalog to ensure it has all of these new entries. This will be a multi-step process.

First, we have to calculate the possibly-missing entries. We write a new task, ComputeNewVersionCatalogEntriesTask, and have it extend AbstractPostProcessingTask, which comes from DAGP itself. This exposes a function, projectAdvice(), which gives subclasses access to the "project advice" that DAGP emits to the console, but in a form amenable to computer processing. We'll take that output, filter it for "add advice", and then write those values out to disk via our task's output. We only care about the add advice because that's the only type that might represent a dependency not in a version catalog.

// in a custom task action
val newEntries = projectAdvice()
  .dependencyAdvice
  .filter { it.isAnyAdd() }
  .filter { it.coordinates is ModuleCoordinates }
  .map { it.coordinates.gav() }
  .toSortedSet()

outputFile.writeText(newEntries.joinToString(separator = "\n")

Note that with Gradle task outputs, it's best practice to always sort outputs for stability and to enable use of the remote build cache.

Next we tell DAGP about this post-processing task (which is how it can access projectAdvice():

// subproject's build script
computeNewVersionCatalogEntries = tasks.register(...)

dependencyAnalysis {
  registerPostProcessingTask(computeNewVersionCatalogEntries)
}

And finally we also have to register our new task's output as an artifact of this project!

val publisher = interProjectPublisher(
  project,
  MyArtifacts.Kind.VERSION_CATALOG_ENTRIES
)
publisher.publish(
  computeNewVersionCatalogEntries.flatMap { 
    it.newVersionCatalogEntries 
  }
)

where the interProjectPublisher and related code is heavily inspired by DAGP's artifacts package, because I wrote both. The tl;dr is that this is what teaches Gradle about a project's secondary artifacts. I wish Gradle had a first-class API for this, alas.

Step 3: Update the version catalog, part 2

Back in the root project, we need to declare our dependencies to each subproject, flavoring that declaration to say we want the VERSION_CATALOG_ENTRIES artifact:

// root project
val resolver = interProjectResolver(
  project,
  MyArtifacts.Kind.VERSION_CATALOG_ENTRIES
)

// Yes, this CAN BE OK, but you must only access 
// IMMUTABLE PROPERTIES of each project p.
// This sets up the dependencies from the root to 
// each "real" subproject, where "real" filters
// out intermediate directories that don't have
// any code
allprojects.forEach { p ->
  // implementation left to reader
  if (isRealProject(p)) {
    dependencies.add(
      resolver.declarable.name, 
      // p.path is an immutable property, so we're
      // good
      dependencies.project(mapOf("path" to p.path))
    )
  }
}

val fixVersionCatalog = tasks.register(
  "fixVersionCatalog", 
  UpdateVersionCatalogTask::class.java
) { t ->
    t.newEntries.setFrom(resolver.internal)
    t.globalNamespace.putAll(...)
    t.versionCatalog.set(layout.projectDirectory.file("gradle/libs.versions.toml"))
  }

The root project is the correct place to register this task, because the version catalog will typically live in the root at gradle/libs.versions.toml.

With this setup, a user could now run gradle :fixVersionCatalog, and it would essentially run <every module>*:projectHealth, followed by <every module>*:computeNewVersionCatalogEntries, followed finally by :fixVersionCatalog, because those are the necessary steps as we've declared and wired them.

This updates the version catalog to contain every necessary reference to resolve all the potential libs.<foo> dependency declaration throughout the build.

Step 4: Fix all the dependency declarations

This step leverages DAGP's fixDependencies task, and is really just about wrapping everything up in a neat package.

We want a single task registered on the root. Let's call it :fixAllDependencies. This will be a lifecycle task, and invoking it will trigger :fixVersionCatalog as well as all the <every module>*:fixDependencies tasks.

// root project
val fixDependencies = mutableListOf<String>()

allprojects.forEach { p ->
  if (isRealProject(p)) {
    // ...as before...

    // do not use something like `p.tasks.findByName()`,
    // that violates Isolated Projects as well as
    // lazy task configuration.
    fixDependencies.add("${p.path}:fixDependencies")    
  }
}

tasks.register("fixAllDependencies") { t ->
  t.dependsOn(fixVersionCatalog)
  t.dependsOn(fixDependencies)
}

And we're done.⁶

(Optional) Step 5: Sort dependency blocks

If you do all the preceding, you should have a successful build with a minimal dependency graph. 🎉 But your dependency blocks will be horribly out-of-order, which can make them hard to visually scan. DAGP makes no effort to keep the declarations sorted because that is an orthogonal concern and different teams might have different ordering preferences. This is why I've also authored and published the Gradle Dependencies Sorter CLI and plugin, which applies what I consider to be a reasonable default. If you apply this to your builds (which we do to all of our builds via our convention plugins), you can follow-up :fixAllDependencies with

gradle sortDependencies

and this will usually Just Work. This plugin is in fact already using the enhanced Kotlin grammar from KotlinEditor, so Gradle Kotlin DSL build scripts shouldn't pose a problem for it.

And now we're really done.

Endnotes

¹ Currently supported languages: Groovy, Java, Kotlin, and Scala. up

² This is one reason why I think it's important to keep scripts simple and declarative. up

³ Measured with the cloc tool. up

⁴ Gradle's biggest footgun, in my opinion, is that the API doesn't enforce this conceptual boundary. up

⁵ This paragraph is an oversimplification for discussion purposes. up

⁶ Well, except for automated testing and blog-post writing. up

ACAB: Fire the (code style) cop in your head

Tony Robalik — Sun, 29 Sep 2024 18:47:32 +0000

Photo by Karsten Winegeart on Unsplash

Let's get something out of the way right at the start: I hate enforced automated code style formatters. This post started because I had an axe to grind, and I made the ill-advised decision to haul my grindstone out into the public square. Multiple times. Over weeks.

Disclaimer! I primarily write Kotlin code that I build with Gradle, in Intellij IDEA (in order of importance). Since many of my complaints are about specific implementation-level issues with code style formatting tools, if you don't also build Kotlin with Gradle, don't worry about it! I'm just an internet crank and you are relieved from the duty of telling me how wrong I am.

Second disclaimer! Enforced code style is not the same thing as linting for usage issues. I like linters! Please tell me I forgot to close a stream, or am calling a Java API with ambiguous nullability.

Ok, let's gooooo.

Arguments in favor of enforcement of consistent code style

Consistent code style is better than inconsistent code style

Sure. But what do we mean by better? The general claims are:

More readable
More maintainable
Aids with achieving flow
Contains fewer errors

I think we've already run into an issue, which is we need to clarify what "style" means. Are we just talking about where we break lines and how long those lines are? Or do we mean something deeper? If we are talking about line breaks, I think the argument is pretty weak. Sure, all else being equal, if two pieces of source code follow the same general style, I will have an easier time with pattern-matching and "understanding" them. But what if my pattern-matching misleads me? What if I gloss over the fact that two structurally-identical blocks contain a single semantic difference that entirely changes the behavior? Maybe that single difference is easier to spot if the whitespace is identical, or maybe it's easier to miss because my brain too-readily dismisses the second block as identical to the first. This is actually what comprehensive tests are for. If this is the kind of error you want to catch, you actually just want tests.

What if we mean "style" in the deeper sense? I had a colleague once who routinely pumped out massive amounts of code, all formatted the same as older code near it, but whose style was so different that loading it into memory (that is, reading and understanding it) was a huge effort.¹ The naming conventions were different; there was very heavy use of Kotlin scoping functions like run and let; and lazy callbacks, generic extension functions, lambdas and lambdas-with-receivers were simply everywhere. (None of that is to say there was anything "wrong" with it. It was just different from all the other code nearby.) Automated style enforcement failed to achieve any of the above-listed goals regarding this code, although I suppose one could argue they prevented comprehensibility from being somehow even worse.

Ok, so, automated style formatters can't possibly help with "code style" in this second sense. We can dismiss it from further consideration but we also must acknowledge that our goals with automated style enforcement may be entirely defeated by orthogonal, essentially social, concerns.

Another argument I've seen in favor of consistent code style is that it can help spot code smells, or usages of known-problematic APIs, such as Java's ObjectOutputStream.² I would say that if this is something you care about, then what you really want is a linter. As I said above, linters and code style formatters are different things. We can forbid ObjectOutputStreams without going anywhere near the question of consistent code style enforcement.

To conclude this section, I will say that, all else being equal, consistent code style is Good™️. Its goals, however, are easily defeated by orthogonal issues, or actually achieved with other kinds of tools (tests, linters) that are themselves completely orthogonal to code style per se.³

Arguments against enforcement of consistent code style

To reiterate part of the conclusion above, in some cases I think we are simply making a category error. We don't want consistent code style, we want more and better tests, and stricter linting for issues that can't be caught by the compiler.

Therefore, my primary argument against enforcing code style is that its proponents haven't made a strong enough case. In a world where managers are telling me to do "less with less," and there have been mass layoffs and productivity concerns and increasingly strict requirements that all work be explicitly tied to identified business needs, why am I spending any time at all on whitespace-normalization tools?

I want peer-reviewed, high-quality research showing that enforcing consistent code style helps with business outcomes, or I would like to never talk about this again. There is a massive opportunity cost here that is simply not being acknowledged.

Most of the rest of my arguments are around how painful it has proven to use the available tools. That is, they're implementation-level issues that could, in principle, be resolved. But I will simply point to the above paragraph and ask why I should spend any time on this.

For a code formatter to even be considered usable, it has to be able to autoremediate all issues it can report. If it can report issues—as errors even—then it must also be able to fix them. Yet, not all code formatting tools can do this. What's worse, they may change their rules in minor or patch releases, breaking automated dependency updates because they'll require a human to manually add line breaks to dozens or hundreds of source files. This might be a bug or it might be a design limitation. Either way, we can simply dismiss these immediately from consideration. (Bearing in mind that we won't be fixing them, because who has time for that?)

Now, what remains are the class of tools that can autoremediate all issues. This is great! Maybe we can just skip this argument about enforcing code style and just slap this tool onto our project and away we go! But... when should we run these tools? Do we run them only in CI to avoid adding friction to the inner dev loop? This would certainly lead to an explosion of commits like "appeasing code style cop" and longer time-to-merge. Can we run them locally instead? Ok, when? Here are some options:

Pre-compile. Using Gradle, simply add a configuration block like this:⁴
```
tasks.named("kotlinCompile") { t ->
  t.dependsOn("fixStyle")
}
```
But that's not great, because if there's a compilation error, then the fixStyle task will fail with its own opaque parsing error, instead of the more expected compilation failure. Ok, what about...
Post-compile. Using Gradle, simply add a configuration block like this:
```
tasks.named("kotlinCompile") { t ->
  t.finalizedBy("fixStyle")
}
```
Turns out that's not great either. The fixStyle whitespace-normalizer will rewrite your source code after it's been compiled, meaning there is now a mismatch between the compiled class files and the source files. Debugging breakpoints can move around, and if you are indeed in a debugging session and commenting/uncommenting blocks of code, your style formatter will treat the block comment differently from a source block, such that when you later uncomment it, it may not be syntactically correct, requiring an additional step in this innermost part of your dev loop. And since you're debugging, that suggests your focus is on fixing bugs, and now you have to contend with the annoying friction of this tool you already don't like breaking your code.

Both of these approachs have another issue, which is they make your builds slower. A well-configured tool will at least only run on changed files, but nevertheless it is "wasting time" when you just want to run your code.
Pre-commit. Add a git pre-commit hook that runs the tool before you can commit. This approach is better, but you have to make a choice:

a. Pre-commit with Gradle task. This is going to make your git commit much slower, because Gradle is slow.⁵ That's annoying.

b. Pre-commit with CLI tool. This can be much faster, since you won't have to deal with the Gradle configuration overhead, but now you have to be very careful to ensure that the CLI tool and the Gradle task do the same thing, else you might end up with mysterious CI failures that frustrate your developers. So be careful.

In both these cases, you might run into issues if you use git add -p to make fine-grained commits. There are ways to resolve this, but it makes the tooling more complex to maintain. And of course, git is usually very fast, so making it slower can be frustrating. But, in some tests my team ran, the cost was on the order of ~2 seconds. Not great, but tolerable I guess, especially if it lets us move on from this topic forever.
Pre-push. Add a git pre-push hook that runs the tool before you can push.

We have similar concerns to 3a and 3b above, so I won't repeat them. The biggest difference here is it resolves some issues with pre-commit, but also will require developers to add a new commit, orthogonal to their main concern, that is basically git commit -am "Appease code formatter." Not the worst thing in the world.

Grinding my axe

As I said at top, this post is really about grinding an axe in public. So now I want to share some of my least-favorite formatting decisions. I truly do not understand how any of these can be considered optimal for readability.

before

@get:PathSensitive(PathSensitivity.RELATIVE)
@get:InputFiles
public abstract val buildFiles: SetProperty<File>

after

@get:PathSensitive(PathSensitivity.RELATIVE) @get:InputFiles public abstract val buildFiles: SetProperty<File>

Wtf? The code formatter, rather than treating the max line length as a maximum, appears to be treating it as an ideal. If the property plus all of its annotations can fit on one line, it does that! Otherwise, it leaves the declaration untouched.

before

testCases().flatMap { testCase ->
  gradleVersions().map { gradleVersion ->
    arrayOf(testCase, gradleVersion)
  }
}

after

testCases().flatMap { testCase -> gradleVersions().map { gradleVersion -> arrayOf(testCase, gradleVersion) } }

Same problem with treating the max line length like an ideal to be achieved, with the additional wrinkle that now it's collapsing multiple scopes onto a single line, making it harder to visually match curly brace pairs.

before

if (
  veryLongFunctionThatReturnsABooleanYeeeeeaahhhhhhhhhhhhhhhh()
  || veryLongFunctionThatReturnsABooleanYeeeeeaahhhhhhhhhhhhhhhh()
) {
  // do something
}

after

if (
  veryLongFunctionThatReturnsABooleanYeeeeeaahhhhhhhhhhhhhhhh() ||
    veryLongFunctionThatReturnsABooleanYeeeeeaahhhhhhhhhhhhhhhh()
) {
  // do something
}

This one really annoys me. In the before-case, while I'm iterating on my branching logic, I can easily comment-out individual lines and the code remains syntactically correct. In the after-case, I simply cannot. It takes more keypresses or even mouse movement where before it was a single keypress. I like to minimize mouse movement to avoid repetitive strain injuries.

Axe-grinding in the community

In the Mastodon thread linked at top, some math-centric devs posted examples of auto-formatting that made their code less readable in their view, when they have data that is best represented as a table. I typed their examples directly into my editor and ran my team's tool on it—it produced an even worse result than the screenshot!

before

val v = (data[pos++].toLong() and 0xffL shl 56
  or (data[pos++].toLong() and 0xffL shl 48)
  or (data[pos++].toLong() and 0xffL shl 40)
  or (data[pos++].toLong() and 0xffL shl 32)
  or (data[pos++].toLong() and 0xffL shl 24)
  or (data[pos++].toLong() and 0xffL shl 16)
  or (data[pos++].toLong() and 0xffL shl 8)
  or (data[pos++].toLong() and 0xffL))

after

val v =
  (data[pos++].toLong() and
    0xffL shl
    56 or
    (data[pos++].toLong() and 0xffL shl 48) or
    (data[pos++].toLong() and 0xffL shl 40) or
    (data[pos++].toLong() and 0xffL shl 32) or
    (data[pos++].toLong() and 0xffL shl 24) or
    (data[pos++].toLong() and 0xffL shl 16) or
    (data[pos++].toLong() and 0xffL shl 8) or
    (data[pos++].toLong() and 0xffL))

Wat. Among other weird problems, it exhibits the same behavior as my complex boolean above: it moves the or from a prefix-position to a postfix-position.

Someone else suggested adding 0 or at the front... and yes, that plus an additional pair of parenthese did help (note, I am not personally very good at bit operations so I'm not sure if I didn't just break this code!).

before

val v =
  (0L
    or (data[pos++].toLong() and 0xffL shl 56)
    or (data[pos++].toLong() and 0xffL shl 48)
    or (data[pos++].toLong() and 0xffL shl 40)
    or (data[pos++].toLong() and 0xffL shl 32)
    or (data[pos++].toLong() and 0xffL shl 24)
    or (data[pos++].toLong() and 0xffL shl 16)
    or (data[pos++].toLong() and 0xffL shl 8)
    or (data[pos++].toLong() and 0xffL))

after

val v =
  (0L or
    (data[pos++].toLong() and 0xffL shl 56) or
    (data[pos++].toLong() and 0xffL shl 48) or
    (data[pos++].toLong() and 0xffL shl 40) or
    (data[pos++].toLong() and 0xffL shl 32) or
    (data[pos++].toLong() and 0xffL shl 24) or
    (data[pos++].toLong() and 0xffL shl 16) or
    (data[pos++].toLong() and 0xffL shl 8) or
    (data[pos++].toLong() and 0xffL))

This is, admittedly, better. However, it's still moving the or over to the end, which I just do not understand. And maybe more to the point, I've just had to adjust my behavior to appease a tool I don't like in the first place. It's producing uglier code that's harder to read and maintain, and now it's also making me dance! Badly! Not a fan!

What about selectively switching formatting off and on? Maybe we just wrap our tabular data in //stylecop:off and //stylecop:on. Boom, problem solved. Well, the fact that this is an option means that we cannot just slap this tool onto our codebase and never talk about whitespace again. Now we have to talk about it in every damn PR where these options appear, or could appear in the eye of the reviewer.

Summing up

I put off writing this post for a long time because, as I said, I find this whole subject a waste of time in the face of competing priorities. My hope is that, having written it, I will never have to write about it again—I'll just link back to this post. Thanks for reading! Looking forward to not reading your contrary arguments ❤️

Endnotes

¹ He used to work at Google where, the rumors say, developers are rewarded for writing complex code. up

² Described as "the JVM equivalent of contracting a minor illness". up

⁴ There is no task (that I'm aware of) named fixStyle. I have invented it for this post to avoid getting into fights with specific tool makers, who are working on truly difficult problems (that I simply don't care about). up

⁵ I am obligated to say that it's Gradle's single-threaded configuration phase that is slow and annoying. Task execution will be basically identical to running the CLI tool. Configuration caching and the promised land of isolated projects may yet save us. up

Announcing Dependency Analysis Gradle Plugin 2.0.0!

Tony Robalik — Tue, 27 Aug 2024 21:24:18 +0000

Cover photo by Sandie Clarke on Unsplash

The Dependency Analysis Plugin v2.0.0 has just been released!

Why 2.0.0?

Breaking changes, of course. The ABI has been updated, which is relevant for anyone accessing the analysis results programmatically. There has also been an important behavioral change, which will be relevant to all users.

ABI changes

The PluginAdvice class has been moved from the com.autonomousapps.advice package to the com.autonomousapps.model package, where all the other public model classes live.

The FindInlineMembersTask has been renamed to FindKotlinMagicTask, to better reflect what it actually does. This is unlikely to impact any users, but since it has public visibility, it might.

Finally, the deprecated ignoreKtx() option has been removed from the issues block. It can now only be configured from the structure block.

Behavioral changes

The plugin no longer automatically applies itself to every subproject in your build. Therefore the dependency.analysis.autoapply flag is now a no-op, and using it will either emit a warning (when the value is set to false) or fail your build (if the value is set to true, which was the old default). Users now have two choices if they want to analyze subprojects in their build:

First, you can apply the plugin directly to every module you want to analyze. This has long been the best practice for how to use this plugin.

Second, you can use the new com.autonomousapps.build-health plugin! Apply it to your settings.gradle[.kts] like so:

// settings.gradle[.kts]
plugins {
  id("com.autonomousapps.build-health") version "2.0.0"
}

// And you can configure it from the settings
// script, too!
dependencyAnalysis {
  ...
}

This new plugin is only available from Gradle 8.8. See the wiki for more information, especially if you also use Android or Kotlin.

In order to assist users who might update to 2.0.0 without reading the release notes, the plugin has been updated to attempt to detect the case where the plugin has been applied only to the root project but not to any subproject; this would almost certainly be a user error.

Happy building!

Thanks for being a user, and please keep filing issues if you run into any bugs.

Gradle's leaky abstractions: Declarative(ish) shell, imperative core: Implementing a safe(ish) global configuration DSL

Tony Robalik — Sat, 23 Mar 2024 19:20:30 +0000

Gradle is very aware they have a complexity problem. Fundamentally, the problem is that Gradle build scripts use an Actual Programming Language (either Groovy or Kotlin), and therefore provide users access to the complete Java/Groovy/Kotlin ecosystem—the JDK, the standard libraries, and all the other libraries too.

Gradle wants you to write your scripts like this:

// build.gradle.kts
// so declarative!
plugins {
  id("foo")
}

dependencies {
  implementation(libs.coolThing)
}

myExtension { ... }

but what that boils down to is just this:

// er, imperative?
project.pluginManager.apply("foo")

project.dependencies.add("implementation", libs.coolThing)

project.extensions.getByType(MyExtension::class.java).run { ... }

which in turn is this:

// yeah, definitely imperative
FooPlugin().apply(project) {
  ... all the code in FooPlugin.apply() ...
}

... etc ...

Where one of the most important takeaways here is that build scripts are evaluated top-down, one "declarative block" after the other. Although, even that is not necessarily true. Kotlin DSL confuses this. Here's a valid settings script:

// settings.gradle.kts
rootProject.name = "..."
pluginManagement { ... }
buildscript { ... }
dependencyResolutionManagement { ... }
plugins { ... }

Gradle will not complain if you do this, but actually some of these blocks are Special and get evaluated not in top-down order. buildscript is the Primordial Special block; it is always evaluated first (which makes sense, as it contributes dependencies to the script itself). pluginManagement is evaluated next, followed by plugins, followed by everything else. Sometimes, when I'm working on a build, I will reorder blocks to match actual evaluation order because I want users to remember this (implicitly), but mostly I've given up. Note that, in a Groovy DSL script, Gradle will actually complain and refuse to compile a script that is written out-of-order. ¯\_(ツ)_/¯

What about `afterEvaluate` and various other lazy callbacks?

This isn't a master's thesis.

Sigh

Anyway.

Can I create a custom extension that is configured in the root project and which configures all subprojects, and is safe for isolated projects?

Yes, but not easily. Here's what we want to do, from a "UI" perspective:

// root build.gradle.kts
plugins {
  id("my-root-plugin")
}

myExtension { ... }

// subproject/build.gradle.kts
plugins {
  id("my-sub-plugin")
}

// values set in rootProject.myExtension are
// available here, safely

Where "safely" means these projects aren't, uh, unsafely coupled, and in particular don't access one another's mutable state. In very particular, they should follow the Isolated Projects contract (which is currently a WIP). I'll elaborate on "unsafely coupled" in a moment.

Naively, what we'd like to do is something like this:

class MySubPlugin : Plugin<Project> {
  fun apply(target: Project) {
    val myExtension = target.rootProject
      .extensions
      .getByType(MyExtension::class.java)

    // do things with values from rootProject's
    // myExtension extension.
  }
}

That, however, is not safe. A project's ExtensionContainer is mutable and accessing another project's mutable state is not safe (maybe the API should forbid this 🤔).

So instead of getting access to that state directly, we will instead use one of Gradle's few blessed ways to manage global mutable state for access at configuration time: the Shared Build Service.

The data flow will be as follows:

Users configure the extension in the root project.
Method calls on the extension immediately push data into mutable objects held by the build service.
Subprojects query the data in the service during configuration.
With a little bit of cleverness, we could also have extensions in subprojects configure data just for their subprojects. (This is left as an exercise for the reader.)

Important! Note that subprojects are coupled to parent projects! (The root project, in this case.) But so long as we don't access the parent projects' mutable state directly, it's ok! I promise! Gradle promises to always configure parent projects before child projects, so there's always this temporal coupling. (The main caveat to this is if you use an API like evaluationDependsOn()).

Finally, evaluate each Project by executing its "build.gradle" file, if present, against the project. The projects are evaluated in breadth-wise order, such that a project is evaluated before its child projects. This order can be overridden by calling evaluationDependsOnChildren() or by adding an explicit evaluation dependency using evaluationDependsOn(String).

So, uh, don't do that or everything goes sideways.

Implementing the root extension

The following demonstrates a custom DSL that's configured at the root level. The existence of an inner DSL is strictly unnecessary, but included as a demonstration because it's often useful. From the user perspective, it would be configured like so:

// root build.gradle.kts
myExtension {
  handler {
    foo(/* true or false */)
    bar(/* a String */)
  }
}

It's important that the inner handler expose methods, not properties. Methods let us do whatever we want most easily. (A Kotlin property with custom setter would also work, but I think that's unnecessarily complicated.)

abstract class MyExtension @Inject constructor(project: Project) {

  private val objects: ObjectFactory = project.objects

  private val service = MyService.of(project)
  private val myHandler = objects.newInstance(MyHandler::class.java, project, dslService)

  fun handler(action: Action<MyHandler>) {
    action.execute(myHandler)
  }

  internal companion object {
    fun register(project: Project): MyExtension {
      return project.extensions.create(
        "myExtension",
        MyExtension::class.java,
        project,
      )
    }
  }
}

abstract class MyHandler @Inject constructor(
  private val project: Project,
  private val service: Provider<MyService>,
) {

  private val objects = project.objects

  private val foo = objects.property(Boolean::class.java)
  private val bar = objects.property(String::class.java)

  fun foo(value: Boolean) {
    foo.set(value)
    foo.disallowChanges()

    service.get().configureHandlerFor(project) { config ->
      config.foo = value
    }
  }

  fun bar(value: String) {
    bar.set(value)
    bar.disallowChanges()

    service.get().configureHandlerFor(project) { config ->
      config.bar = value
    }
  }
}

We could actually skip the Property creation entirely, but I like it because of the disallowChanges() API, which ensures the method can only ever be called once, and I want to ensure a single source of truth for most cases.

Implementing the build service

A ("shared") build service is kind of like a singleton, in that when you register one in any project, it's available in all projects as a single instance. (This unfortunately turns out not to be true, in some cases, when using composite builds, but can be worked around.) An actual singleton (global static instance) doesn't work at all, for the record—try it if you want to lose some sanity. Anyway, use a build service whenever you need global mutable state in your build.

Please note that the examples in this post demonstrate a project extension having a reference to the build service. Do not attempt to do this in the other direction. You don't want a globally-available object (the build service) having access to a mutable object "owned by" any individual project.

abstract class MyService : BuildService<BuildServiceParameters.None> {

  private val handlerConfigurations = createConfigMap<HandlerConfiguration>()

  internal fun configureHandlerFor(
    project: Project, 
    action: Action<HandlerConfiguration>
  ) {
    handlerConfigurations.merge(
      project.path,
      HandlerConfiguration().apply(action::execute)
    ) { acc, _ ->
      acc.apply(action::execute)
    }
  }

  internal fun findHandlerConfig(project: Project): HandlerConfiguration? {
    return handlerConfigurations.findConfig(project)
  }

  private fun <T> createConfigMap(): MutableMap<String, T> = mutableMapOf()

  private fun <T> Map<String, T>.findConfig(project: Project): T? {
    return get(project.rootProject.path)
  }

  internal companion object {
    fun of(project: Project): Provider<MyService> {
      return project
        .gradle
        .sharedServices
        .registerIfAbsent(
          "myService",
          MyService::class.java
        ) {}
    }
  }
}

project.rootProject.path is okay because project.path is immutable state and safe to access from another project.

HandlerConfig is just a mutable value object.

internal class HandlerConfig(
  var foo: Boolean = false,
  var bar: String? = null
)

Consuming the values in a subproject

In your convention plugin that you apply to your subprojects (you're doing that, right?), you can get a reference to the shared build service, and then query the configuration object and do whatever you like with it.

class MySubPlugin : Plugin<Project> {
  fun apply(target: Project) {
    val service = MyService.of(target).get()
    service.findHandlerConfig(target)?.let { config ->
      // do something with config.foo
      // do something with config.bar
    }
  }
}

And there we have it. If you want the ability to configure a big project from the root project, with complex data, in a way that keeps your projects mostly decoupled (sigh) from each other, I have a variant of the above running in a large production system.

Is there another way?

Sure. You could rely entirely on Gradle properties and organize your configuration knobs as a set of key-value pairs. This is much easier to implement but much harder to use because Gradle properties don't support code completion, don't have easily accessible Javadoc, don't have any IDE support really, and it's shockingly easy to scatter your properties across your entire codebase, making them practically undiscoverable by feature engineers.

Configuration roles and the blogging-industrial complex

Tony Robalik — Tue, 17 Oct 2023 21:52:27 +0000

On the perils of getting what you ask for
Or, building cool stuff and making everyone think you're a wizard

Gradle 8.4 has released an exciting new feature, an API for easily creating role-focused Configurations. For those who have been around the block a few times, it is well-known that the Configuration interface is amongst the most confusing in Gradle lore, not to mention having a very overloaded name. It also has many responsibilities and a massive memory footprint.

The new API is meant as a step towards disambiguation. The new methods on ConfigurationContainer (aka configurations) are designed to simplify the creation of immutable, purpose-built Configuration instances. They do that. But they also expose complexity that was previously obscured by the prior legacy situation. On the plus side, this additional complexity is a boon to bloggers and build engineers eager to justify their (my) existence.

All of the code used in this post is available on Github.

First a warning

Most Gradle projects won’t need this kind of setup, and most Gradle plugins won’t require this elaboration of responsibilities in the Configurations they create. The primary use-case is for ecosystem plugins (aka “core” plugins), but what we’ll do here is also fairly common. Even if you don’t need this yourself, understanding Configurations more deeply can be very valuable.

What to expect if you keep reading

A brief exploration of the three kinds of Configuration types, an explanation of how to use them, and a dip of the toes in the deep waters of safe cross-project publishing and aggregation.

To guide our exploration, we will consider a concrete use-case. We want to be able to aggregate data from all subprojects at the root project (a natural location), and we want to do it in a safe way that doesn’t violate project boundaries. Here’s how we want our build scripts to look:

// aggregator/build.gradle (or root build.gradle)
plugins {
  id 'org.jetbrains.kotlin.jvm'
  id 'mutual.aid.configuration-roles'
}

dependencies {
  sourceFiles project(':feature-1')
  sourceFiles project(':feature-2')
}

and

// both *feature-1/build.gradle* and *feature-2/build.gradle*
plugins {
  id 'org.jetbrains.kotlin.jvm'
  id 'mutual.aid.configuration-roles'
}

We will apply our plugin to each of our projects.¹ It will set up publication of interesting data from the subprojects, as well as aggregation of that data in our aggregator, or root, project. The data we’re collecting for this example is information about the source files in our subprojects, hence the name of our imaginary new Configuration, sourceFiles.

What are `Configuration`s, anyway? The three kinds

Configurations are used for…
…declaring dependencies
…resolving dependencies within a project
…sharing artifacts between projects

To create these three kinds of instances in the past (before Gradle 8.4), we had to do the following:

configurations {
  // you declare dependencies here
  create("implementation") { 
    isCanBeResolved = false 
    isCanBeConsumed = false
  }

  // plugins will resolve dependencies from this one
  create("runtimeClasspath") {
    isCanBeResolved = true  // Defaults to true
    isCanBeConsumed = false
    extendsFrom(configurations["implementation"]) 
  }

  // plugins expose artifacts to other projects on this one
  create("runtimeElements") { 
   isCanBeResolved = false
   isCanBeConsumed = true  // Defaults to true
   extendsFrom(configurations["implementation"]) 
 }
}

Note that we first create the instances, and then we mutate them in configuration blocks (yes, the word “configuration” is being used in two separate senses here).

In practice, most community plugins were not so scrupulous as to create three separate configurations for these three separate use-cases—and nor were they required to, since Gradle’s API is generally very permissive and legacy-friendly. The new APIs (see below) expose this complexity and enforce rigor… without, unfortunately, much (if anything) in the way of documentation or explanation.

Here are the new methods:

configurations { 
  dependencyScope("implementation")
  resolvable("runtimeClasspath") {
    extendsFrom(configurations["implementation"]) 
  }
  consumable("runtimeElements") { 
    extendsFrom(configurations["implementation"]) 
  }
}

We can all agree that this is fewer lines of code, but this obscures the additional rigor that is suddenly required to actually interact with these configurations. Because in point of fact, in the past, plugin authors would probably just do this:

configurations {
  create(“implementation”)
}

This creates a single configuration that Gradle will happily let you use to…
…declare dependencies
…resolve dependencies within a project
…share artifacts between projects

For reasons that are outside of the scope of this post, this is Bad™. Everyone™® agrees™®© that the Configuration API should be destroyed in a fire and/or thrown into the seam between universes, never to be seen again. The new API is meant as a step towards this glorious future.

How to do something useful and interesting with this stuff

With that out of the way, how do we actually use this API, keep up with the times, and make our plugins (slightly more) future-proof?

Let’s create a plugin! (Yes, this is my answer to everything.) The purpose of our new plugin is to aggregate information across all projects in our build in a way that is safe, that doesn’t violate project boundaries. It will demonstrate the usage of all three kinds of configuration, as well as very lightly touching on variant attributes. This is how Gradle models things like compile and runtime classpath separation, among other things, and are a very powerful (although deeply verbose and annoying) concept.

Setting up our configurations

First let’s create our three configurations:

class ConfigurationRolesPlugin : Plugin<Project> {
  override fun apply(project: Project): Unit = project.run {
    // Following the naming pattern established by the Java Library
    // plugin. More on this in a moment.
    val declarableName = "sourceFiles"
    val internalName = "${declarableName}Classpath"
    val externalName = "${declarableName}Elements"

    // Dependencies are declared on this configuration
    val declarable = configurations.dependencyScope(declarableName)
      // The new APIs return the new configuration wrapped in a lazy
      // Provider, for consistency with other Gradle APIs. However,
      // there is no value in having a lazy Configuration, since we
      // use  it immediately anyway. So, call get() to realize it, and
      // call it a day.
      .get()

    // The plugin will resolve dependencies against this internal
    // configuration, which extends from  the declared dependencies
    val internal = configurations.resolvable(internalName) { c ->
      c.extendsFrom(declarable)
      // Same as below
      c.attributes { a ->
        a.attribute(
          Category.CATEGORY_ATTRIBUTE,     
          objects.named(Category::class.java, Category.DOCUMENTATION)
        )
      }
    }

    // The plugin will expose dependencies on this configuration, 
    // which extends from the declared dependencies
    val external = configurations.consumable(externalName) { c ->
      c.extendsFrom(declarable)
      // Same as above
      c.attributes { a ->
        a.attribute(
          Category.CATEGORY_ATTRIBUTE,
          objects.named(Category::class.java, Category.DOCUMENTATION)
        )
      }
    }
  }
}

As indicated by the comments, we are following the naming convention established by the Java Library plugin.

Our declarable configuration is named “sourceFiles”, and corresponds to one of the green configurations in the diagram (let’s say “implementation” for a concrete example). It is on this configuration that we declare our dependencies. We extend this configuration with a resolvable (internal) configuration (colored blue), named “sourceFilesClasspath”,² and this is used within the project by tasks which resolve these dependencies. Finally, we also extend the original configuration with a consumable (external) configuration (colored pink), upon which we will publish artifacts for use by dependent projects.

We use the attributes to tell Gradle what kind of artifact we’ll be attaching to our configurations. In this case, we say we are publishing “documentation.” Note that this is, in a manner of speaking, arbitrary, though it is important to be consistent. If we own a Very Important ecosystem plugin such as the java-library or Android Gradle plugin, then it’s also important that our attributes be meaningful or our users will hate us. For this post, I just picked something easy and readily available without giving it a lot of thought.

Publishing a custom (non-jar) artifact

override fun apply(project: Project): Unit = project.run {
  val kotlin = extensions
    .getByType(KotlinJvmProjectExtension::class.java)

  // src/main
  val mainSource = kotlin.sourceSets.named("main")
  // src/main/{kotlin,java} as a FileTree
  val mainKotlinSource = mainSource.map { it.kotlin }

  // Register a task to produce a custom output for consumption by 
  // other projects
  val producer = tasks
    .register("collectSources", ProducerTask::class.java) { t ->
      t.source.from(mainKotlinSource)
      t.output.set(
        layout
          .buildDirectory
          .file("reports/configuration-roles/sources.txt")
      )
    }

...

configurations.consumable(externalName) { c ->
  ... as before ...

  // Teach Gradle which task produces the artifact associated with
  // this external/consumable configuration
  c.outgoing.artifact(producer.flatMap { it.output })
}

It’s a few lines of code, but what we’ve basically done is simply:

Registered a task that takes as input all of the source files of this project.
Register the output of that task as an outgoing artifact on our consumable (external) configuration.

The task that produces the artifact we’re sharing is not very interesting for our purposes here, but if you want to take a look, here it is.

Consuming the artifacts from dependency projects

Here we define and register a task that will consume the published artifacts of other projects. Recall this works because our aggregator project has this in its build script:

// aggregator/build.gradle (or root build.gradle)
dependencies {
  sourceFiles project(':feature-1')
  sourceFiles project(':feature-2')
}

This is what links our aggregator to the other projects we’re collecting data from. Now let’s register our new task:

// Register a task to consume the custom outputs produced by other 
// projects
tasks
  .register("printDependencySources", ConsumerTask::class.java) { t ->
    t.reports.setFrom(internal)
  }

and

abstract class ConsumerTask : DefaultTask() {

  @get:PathSensitive(PathSensitivity.RELATIVE)
  @get:InputFiles
  abstract val reports: ConfigurableFileCollection

  @TaskAction fun action() {
    val globalSources = reports.joinToString(separator = "\n") { 
      it.readText()
    }
    logger.quiet(globalSources)
  }
}

There’s nothing special here. The artifacts that our other projects are publishing are ergonomically available (as simple text files in this case) on our new consumable configurations, which we can feed directly and easily into our custom task, which reads those files into memory and then prints them to console for reading by your users.

Happy Gradling!

Cover image: a developer trying to find something in Gradle's documentation

Endnotes

¹ I can imagine we might want two separate plugins for this, one for publishing and one for consuming or aggregating. But this is not strictly necessary. up
² Yes, even though this won’t be a classpath in any normal sense. It’s just a convenient name. up

Become a Gradle expert in 3 easy steps!

Tony Robalik — Thu, 15 Jun 2023 19:47:18 +0000

Step 1. Your build fails "for no reason" and your small team has no dedicated build support. Everyone is complaining in Slack and you can't bear to look at another poor quality Geordi meme.

You copy a portion of the stacktrace and paste it into Google. Turns out jcenter is experiencing yet another outage. You look at the suggested workarounds and do a find-and-replace: s/jcenter/mavenCentral. Your build works again! You don't really understand why, but you also don't care. Back to writing beautiful UI!

Step 2. Your build fails "for no reason" and your small team has no dedicated build support. Everyone remembers that you fixed it last time and ask you to look into it.

Step 3. Now you're the expert. Congrats!

Nihilism and the anti-corruption layer

Tony Robalik — Fri, 24 Jun 2022 21:29:36 +0000

Every now and then someone asks me if I intend a more technical follow-up to Herding Elephants, which was a fairly high-level overview of the strategy I initiated to modernize that large Android repo I help maintain. And the answer is—not really. It is hard to appreciate how many moving parts exist in a repo at that scale without seeing it for yourself. Many of the lessons learned are irrelevant for smaller projects, which is most of them. The solutions to these medium-large problems¹ are expensive to build and expensive to maintain, and the cost-benefit analysis is only in their favor at a certain, well, scale. Dozens of developers and hundreds or thousands of modules. Millions of lines of code.

Of all the components that form part of our current build system, however, one stands out as particularly cool and also as a metaphor for the human condition, which is what I really want to talk about.

What to expect if you keep reading

I will elaborate on the anti-corruption layer I built to protect our build from upstream changes coming from the rapidly-evolving Android Gradle Plugin, and discuss how this has helped our build maintain stability and migrate to newer releases reliably and with increasing rapidity. I will wave my hands at some of the guard rails I built. I will discourage you from attempting this yourself. I will then pivot to how this relates to nihilism and the human condition.

The anti-corruption layer

As any Android developer with more than a year or two of experience knows, the Android Gradle Plugin (AGP) is constantly evolving. For the very simplest projects, these changes are largely transparent, but I would guess that most Android projects have had frustrations with these changes over the years.

One of the pain points of AGP is that its API is constantly changing. (Google even has a documented "migration timeline" for this API; it spans years.) Google also takes the idiosyncratic perspective that many of its public classes aren't really public API, but internal implementation detail, and because of this, it often pushes breaking changes without any corresponding signal via semantic versioning.

There are two extreme solutions to this problem. At one extreme, which I highly recommend you follow, is being a Conformist. This is a design pattern whereby you essentially "give up" and admit that you will never really be able to craft a model specific to your project. Your model is AGP's model. This is fine.

At the other extreme we have the Anti-corruption layer. You probably shouldn't do this.²

An anti-corruption layer is used when a downstream project has a dependency on an upstream that it (1) has no control over; (2) can't avoid; and (3) doesn't trust. This precisely describes our relationship with AGP. The concept is often summarized as "wrapping" dependencies, but it is more than that. Let's consider the implementation of this pattern that we use in the "convention plugins" I wrote about in Herding elephants.

I want to be clear here that I'm speaking in a highly technical sense. I am not making normative judgments about AGP's structure nor the direction of its evolution. In point of fact, I often have fruitful discussions with members of the team that maintains AGP, whom I like and respect. Nevertheless, the only power I hold over that upstream dependency is the power of persuasion, which doesn't cut it when my code is used tens of thousands of times per day to build software that powers a multi-billion dollar business.

We start with the facade, or wrapper interface. A facade is the interface we wish we had, and which takes care to redirect to the actual interface provided by the dependency (AGP in this case). Here's a simplified example:

On the right is AGP, which has complicated methods like majorArcanum(...) and minorArcanum(...), which each take many arguments. On the left we have our desired interface, which is just a single method that takes no arguments. We can do this because, while AGP is a general purpose build tool that has to solve problems for Every Android Developer Everywhere, we have a different, smaller problem: how to build our apps for our developers.

We wrap our facade in an "adapter" that knows how to talk to different versions of AGP. This is how we solve the problem of AGP's rapidly changing interface. Our adapter can provide a different implementation for each version of AGP we support, which is the current version plus the next two release candidates in the AGP pipeline. Through this mechanism, we can update to each new release with ease—no more months-long migrations.

To help you visualize how this works, consider this simplified implementation:

class AndroidGradlePluginFactory(
  private val project: Project,
  private val agpVersion: AgpVersion,
) {

  companion object {
    fun getAdapter(project: Project): AndroidGradlePlugin {
      return AndroidGradlePluginFactory(
        project = project,
        agpVersion = AgpVersion.current(project)
      ).newAdapter()
    }
  }

  fun newAdapter(): AndroidGradlePlugin = when {
    agpVersion >= AgpVersion.version("7.3") -> AndroidGradlePlugin7_3(project)
    agpVersion >= AgpVersion.version("7.2") -> AndroidGradlePlugin7_2(project)
    agpVersion >= AgpVersion.version("7.1") -> AndroidGradlePlugin7_1(project)
    else -> throw UnknownAgpException("No adapter for AGP $agpVersion")
  }
}

Please be warned that AgpVersion.current(project) encapsulates a non-trivial implementation for retrieving the actual version of AGP that is present at runtime. Also note that each implementation is encapsulated in its own project (module) to isolate its classpath.

To use this factory, your plugins should look like this:

class MyPlugin : Plugin<Project> {
  override fun apply(target: Project) {
    val agp = AndroidGradlePluginFactory.getAdapter(target)
    // all accesses of AGP functionality _must_ go through `agp`
  }
}

You can also inject an agp instance into your custom extension and then require your developers to use that extension (rather than the backing android extension) for customizing their projects.

abstract class CircleExtension(agp: AndroidGradlePlugin) { 
  fun buildMyAppPlease(how: String) { … }
}

// app/build.gradle
plugins {
  id 'com.circle.android.app'
}

circle {
  buildMyAppPlease 'with style'
}

Guard rails

This is already pretty complicated, yet it's also not enough. This is why you shouldn't do it. But if you insist, below are some of the guards we've had to institute to keep from going off the rails.

Please note that, in each case, I'm dramatically simplifying our build setup so that it fits into a blog post shorter than a master's thesis. This isn't easy and you should think long and hard before deciding to do this.

Comprehensive test suite

All of our plugins are thoroughly tested against all supported versions of AGP. This is very hard to get right, but utterly necessary.

NoAgpAtRuntime

We have a task, noAgpAtRuntime, that is registered on all of our build-logic modules. This task will fail if it discovers a class from AGP on the build classpath. We expect the build that applies our plugins (the "main build") to be responsible for providing AGP.

Strict dependencies

We set a strict dependency constraint on our root build classpath to ensure it's using the version of AGP we specify for every build.

// root build.gradle of main build
buildscript {
  dependencies {
    constraints {
      classpath('com.android.tools.build:gradle') {
        version { strictly agpVersion /* a string */ }
      }
    }
  }
}

DependencyGuard

We use DependencyGuard to fail our build if the build classpath changes unexpectedly.

// root build.gradle
plugins {
  id 'com.dropbox.dependency-guard'
}
dependencyGuard {
  configuration('classpath')
}

Nihilism gets a bad rap

What has any of this to do with nihilism?³ As a concept, it's something I've been flirting with my entire life, and it came to a head recently when I happened to catch an episode of Radiolab from 2014, about a book called "In the Dust of This Planet"—a book I subsequently bought and devoured.

I think most people have weird ideas about what nihilism means; my first encounter with it was probably when I saw The Big Lebowski as a teenager.

This is not the kind of nihilism I'm talking about. I mean it as more of a renunciation of received truth. In the context of software development, we have lots of received truths, and simply accepting them as such leads to the conformist approach (or "pattern") discussed above. Rejecting those truths and going your own way contains, for me, an element of nihilism that I find myself embracing.

Nothing matters, so do what you want

I was already working on this post when I finally saw Everything Everywhere All at Once the other day. That movie is profoundly nihilistic and I <3 it so much. The primary antagonist in the film, Jobu Tupaki, has come to realize that nothing matters and so has created the everything-bagel-of-nothing, a thing which annihilates all who gaze upon it; this self-destruction is what she herself seeks. Her mother, Evelyn, comes to the same realization, but lands on a different conclusion, which is that, because nothing matters, she can do what she wants, and what she wants is to spend her one life with her loved ones.

One of the ways I know that nothing matters is that we've radically destabilized the climate such that it is a foregone conclusion that billions will die and civilizations will fall, and the present mass extinction will continue. We're far too late to stop it, and this against a background of the majority of people wanting to stop it, but our public institutions being so corrupt that we can only watch in horror as they go in the opposite direction. Truly, "nothing matters."

I guess you could say I'm an eco-nihilist.

On a day like today, I must also point out that the US Supreme Court, in an act of destructive nihilism, has further proven my point by going against the democratic majority to expand gun rights and constrict women's rights, and is almost certainly setting up to further curtail personal liberty with attacks on LGBTQ rights in the coming years.

The anti-corruption layer, take 2

If you've read this far, you have the patience of a saint, and so I'm going to cut to the chase. My anti-corruption layer, that lets me continue to persist despite knowing, to my core, that nothing matters, is a deep well of incandescent rage. Conformity would be so much easier—it's what practically everyone around me wants me to do. But I find it literally impossible. We live on a garden world in an endlessly empty universe; it could be a paradise yet we've built a death machine instead. I am so angry. That anger is my shield, but increasingly my nihilism is my shield, too. Nothing matters, so I'm going to do what I want.

Endnotes

¹ We're not truly large (cf Google). up
² The world of software development is thick with metaphor. This post uses numerous metaphors from the world of design patterns, with a focus on two in particular: Conformist and Anti-corruption layer. I learned the names of these patterns from the book Domain-driven design by Eric Evans, but discovered them independently through an intent focus on solving the kinds of problems I often write about on this blog. See also Christopher Alexander and A Pattern Language. up
³ I am not a philosopher (IANAP). up

Tony's rules for Gradle plugin authors

Tony Robalik — Mon, 25 Apr 2022 01:30:01 +0000

The Gradle API surface is huge. It is also littered with unspoken rules whose enforcement mechanism is inscrutable runtime failures.

I want to say "we can do better," but really, we can't. The best we can do at present is mitigate by internalizing the following rules.

A rule by any other name

I call these "rules," but in many cases they can be only guidelines. Sometimes we have to break a rule because there really is no other way to achieve our goals. Nevertheless, the following rules were all learned the hard way, and should only be violated consciously.

The rules

An important bit of context for the following is that a Gradle build is divided into two¹ primary phases: configuration and execution. Most of the rules are about what it is permissible to do in one phase or the other. Each phase carries with it different restrictions.

Don't do expensive computations in the configuration phase

It slows down the build. Such computations should be encapsulated in a task action.

Avoid the create method on Gradle's container types

Use register instead.

Avoid the all callback on Gradle's container types

Use configureEach instead.

Don't assume your plugin is applied after another

Instead, use pluginManager.withPlugin().

Avoid making any ordering assumptions of any kind

Lazy configuration, callbacks, and provider chains are the name of the game.

Don't access a Project instance inside a task action

It breaks the configuration cache, and will eventually be deprecated.

Don't access another project's Project instance

This is called cross-project configuration and is extremely fragile. It creates implicit, nearly un-modelable dependencies between projects and can only lead to grief. Instead, share artifacts across projects by declaring dependencies.

It also breaks the experimental project isolation feature, but that won't be truly relevant for a while.

Avoid afterEvaluate

It introduces subtle ordering issues which can be very challenging to debug.

What you're looking for is probably a Provider or Property (see also lazy configuration).

Don't call get() on a Provider outside a task action

The whole point of using a provider is to evaluate it as late as possible. Calling get()—evaluating it—will lead to painful ordering issues if done too early.

Instead, use map or flatMap.

Don't use internal APIs

Gradle considers internal APIs fair game for making breaking changes in even minor releases. Therefore, using such an API is inherently fragile and will lead to major, completely avoidable, headaches.

Don't use Kotlin lambdas in your public API

I know, it's tempting. They're right there. Use Action<T> instead. Gradle enhances the bytecode at runtime to provide a nicer DSL experience for users of your plugin

Don't create objects yourself

Use the ObjectFactory instead. (This is a configuration time concern.)

Don't use lists in your custom extensions

Use domain object containers instead. Once again, Gradle is able to provide enhanced DSL support this way.

Don't skip the documentation

I know, it's a lot.

Do test your plugins

Especially if you publish them. See this two part series for help.

Special thanks

Special thanks to Zac Sweers for offering feedback on this post.

Endnotes

¹ There are in fact three phases, but the Initialization phase is rarely of interest to the typical build maintainer. up

Forem: Tony Robalik

Enshittification: Why I’m leaving dev.to

Is the Java ecosystem cursed? A dependency analysis perspective

We are cursed with…

Lying metadata

It's just a list, man

Fat jars without package relocation

Split packages

Funhouse mirrors (aka reflection)

Protocol Buffers

Yolo compilers

Special thanks

Who stalks us in the darkness?

This is why we can't have nice things: When POM files lie

Why am I even using this dependency?

Coping

Ah, fuck it, let's draw the owl

Drawing the build script

Drawing the task

Drawing the plugin

Hello, owl!

I'm just some asshole

Special thanks

Endnotes

Gradle extensions part 2: Now with shenanigans

Extending Test tasks

Groovy: An interlude

Sprinkle on some shenanigans

One click dependencies fix

Pre-processing

Conceptual framework

Implementation: :fixAllDependencies

Step 1: The global namespace

Step 2: Update the version catalog, part 1

Step 3: Update the version catalog, part 2

Step 4: Fix all the dependency declarations

(Optional) Step 5: Sort dependency blocks

Endnotes

ACAB: Fire the (code style) cop in your head

Arguments in favor of enforcement of consistent code style

Arguments against enforcement of consistent code style

Grinding my axe

Axe-grinding in the community

Summing up

Endnotes

Announcing Dependency Analysis Gradle Plugin 2.0.0!

Why 2.0.0?

ABI changes

Behavioral changes

Happy building!

Gradle's leaky abstractions: Declarative(ish) shell, imperative core: Implementing a safe(ish) global configuration DSL

What about afterEvaluate and various other lazy callbacks?

Sigh

Can I create a custom extension that is configured in the root project and which configures all subprojects, and is safe for isolated projects?

Implementing the root extension

Implementing the build service

Consuming the values in a subproject

Is there another way?

Configuration roles and the blogging-industrial complex

First a warning

What to expect if you keep reading

What are Configurations, anyway? The three kinds

How to do something useful and interesting with this stuff

Setting up our configurations

Publishing a custom (non-jar) artifact

Consuming the artifacts from dependency projects

Endnotes

Become a Gradle expert in 3 easy steps!

Nihilism and the anti-corruption layer

What to expect if you keep reading

The anti-corruption layer

Guard rails

Comprehensive test suite

NoAgpAtRuntime

Strict dependencies

DependencyGuard

Nihilism gets a bad rap

Nothing matters, so do what you want

The anti-corruption layer, take 2

Endnotes

Extending `Test` tasks

Implementation: `:fixAllDependencies`

What about `afterEvaluate` and various other lazy callbacks?

What are `Configuration`s, anyway? The three kinds