Forem: Kaoru

Dependency Hell

Kaoru — Mon, 02 Dec 2019 04:08:01 +0000

Resolving dependency conflicts is not fun. I personally have not had to pleasure of dealing with conflicts until more recently. Bundler and the entire Ruby community does an amazing job making sure I didn't have to. In the JVM world, things are less clear. For example, with SBT, the latest version of a dependency always wins, but if the latest version is higher by a major version, things will most likely break. SBT won't tell you when it replaces dependencies, you have to ask! I covered more details of the strange behavior in a previous post. What I didn't go over is how we can get into this situation in the first place.

Setting the stage

The best way I learn is by doing, tweaking things, and breaking it, then fixing it...maybe. Let's come up with a contrived example (follow along by cloning this repo). Below we have 3 files describing classes depending on each other Life > Person > Cat.

// com/kaoruk/Life.java =============================
package com.kaoruk;

import com.kaoruk.Person;

class Life {
  public static void main(String args[]) {
    Person person = new Person("Kaoru");
    if (args.length > 0) {
        person.adoptCat();
    }
    System.out.println(person.sayHello());
  }
}


// com/kaoruk/Person.java =============================
package com.kaoruk;

import com.kaoruk.Cat;
import java.util.Objects;

class Person {
  private final String name;
  private Cat cat;

  public Person(String name) {
    this.name = name;
  }

  public void adoptCat() {
    this.cat = new Cat();
  }

  public String sayHello() {
    if (Objects.isNull(cat)) {
      return this.name + ": Life has no meaning without a cat";
    } else {
      return this.name + ": Oh hai, I haz kitty! " + cat.sayHello();
    }
  }
}


// com/kaoruk/Cat.java =============================
package com.kaoruk;

public class Cat {
  public String sayHello() {
    return "meow!";
  }
}

Now let's compile each file and place them in a jar:

#!/bin/bash

set -e

mkdir jars || true

echo "Compling Cat..."
javac com/kaoruk/Cat.java
jar -cvf jars/Cat.1.0.0.jar com/kaoruk/Cat.class
rm com/kaoruk/Cat.class

echo "Compling Person..."
javac -cp jars/Cat.1.0.0.jar com/kaoruk/Person.java
jar -cvf jars/Person.1.0.0.jar com/kaoruk/Person.class
rm com/kaoruk/Person.class

echo "Compling Life..."
javac -cp jars/Person.1.0.0.jar com/kaoruk/Life.java
jar -cvf jars/Life.1.0.0.jar com/kaoruk/Life.class
rm com/kaoruk/Life.class

If all goes well we should be able to run Life:

$ java -cp jars/Life.1.0.0.jar:jars/Person.1.0.0.jar:jars/Cat.1.0.0.jar com.kaoruk.Life

Kaoru: Life has no meaning without a cat

$ java -cp jars/Life.1.0.0.jar:jars/Person.1.0.0.jar:jars/Cat.1.0.0.jar com.kaoruk.Life Foo

Kaoru: Oh hai, I haz kitty! meow!

Great! Okay so let's say we want to teach our Cat how to say "Can I haz job?" but also we decide that the method name "sayHello" doesn't really accurately portray the message. We change Cat.java look like:

// com/kaoruk/Cat.java =============================
package com.kaoruk;

public class Cat {
  public String beg() {
    return "Can I haz job?";
  }
}

Great, let's compile it, replace Cat.1.0.0.jar and run it:

$ javac com/kaoruk/Cat.java
$ jar -cvf jars/Cat.2.0.0.jar com/kaoruk/Cat.class

$ java -cp jars/Life.1.0.0.jar:jars/Person.1.0.0.jar:jars/Cat.2.0.0.jar com.kaoruk.Life

Kaoru: Life has no meaning without a cat


$ java -cp jars/Life.1.0.0.jar:jars/Person.1.0.0.jar:jars/Cat.2.0.0.jar com.kaoruk.Life foo

Exception in thread "main" java.lang.NoSuchMethodError: com.kaoruk.Cat.sayHello()Ljava/lang/String;
    at com.kaoruk.Person.sayHello(Person.java:22)
    at com.kaoruk.Life.main(Life.java:11)

Oh hai, dependency hell!

Detection is difficult

You might be thinking to yourself, well duh Life is broken because you updated a method in Cat and replaced it with a new JAR. This is exactly what build tools do! To detect the problem above, before running the code, we would need to recompile Person.1.0.0.jar. But more importantly, we would need to recompile it using Cat.2.0.0.jar, ignoring its explicit reliance on Cat.1.0.0.jar. As a build tool author, not that I am one, I would imagine my users would not expect the tool to recompile the entire world, because it would take too long.

The logic to recompile the entire world is no easy task. First, you'd have to list all dependencies and transitive dependencies, override dependencies if there is a higher version, create a graph, find the leaves, and the compile JARs all the way back. I'm not even sure this would work, I'm just spitballing!

Tests are not thou Savior

In our example, if our tests only cover running the main method without an argument, we wouldn't catch this bug, one of our poor users will. So that means we have to have tests for every single branch of our code? Yes, regardless if it doesn't work in this scenario! Having a test for every branch might not work because you'd also have to test every branch of your dependencies and transitive dependencies! In our example, if we depended on Life, our tests probably would not have hit the bug, because why would we test every branch of Life?

It ain't so bad...right?

Yes, if we follow a standard. In this case, the standard is semantic versioning, or semver. If a library's major version has changed, we know that something will break, we don't have to go digging into the source code of the library or every dependee of the library. Unfortunately, not all library maintainers follow semver. For example, Scala Akka, does not follow this pattern. Their minor versions introduce breaking changes that are incompatible with older minor versions. The reality is we have to keep track of which library follows semver and which follows a different pattern.

Okay so what can we do about it?

Bundler solves this problem by allowing engineers to specify an acceptable range. For example, gem 'thin', '~> 1.1' means that Bundler is allowed to install newer versions of thin < 2.0. If I bring in a gem which requires thin 2.0, Bundler throws an exception forcing me to deal with the problem. Maven also has this functionality, so Coursier shouldn't be too far behind? But it still doesn't solve the problem if library maintainers ignore semver. So I'm not entirely sure there's a solid solution to this problem yet. I think Bazel, rules_jvm_external to be specific, at least, makes a step towards the right direction informing us when there is a conflict.

Surprising behaviors of SBT

Kaoru — Sun, 17 Nov 2019 23:49:32 +0000

I recently learned about an interesting behavior of SBT while working on migrating my company's monorepo from SBT to Bazel. Why we're migrating deserves a blog post of its own (coming soon). The fundamental problem is: what does your build tool do when it encounters two versions of the same library? The problem may seem trivial at face value, but consider the fact that libraries do not exist in isolation. Resolving conflicting library versions becomes exponentially worse when taking into consideration transitive dependencies.

Helpful definitions

direct dependency: a dependency your project has on an external library.
transitive dependency: a dependency which your direct dependency depends on. For example, I depend on my cats for unlimited cuddle time, my cats depend on cat food. Thus, I have a transitive dependency on cat food.
compile-time: an environment which a program, the compiler (i.e. javac or scalac), run to convert your human-readable code into java byte code.
runtime: the environment which your java-byte code runs in.

Latest and greatest

The default behavior of SBT, version 1.3.0, is to defer to Coursier, the library which manages dependencies for your project. To be honest, SBT's documentation leaves you alone in a desert of the internet to understand the actual behavior. I found this Scala blog post which sheds light on how to control the behavior, but again does not really go into detail on what the actual behavior is. Eugene Yokota, a maintainer of SBT, thankfully, explained in detail what the behavior is in this blog post. The TLDR: Coursier will choose the latest version. Good enough right?

Devil in the detail

In this Github repo we have a core library is using CheckedFuture, an interface in com.google.guava::guava27.1-jre. A project, hello is using core and pulling in com.google.guava:guava:28.0-jre. You might be thinking, "well, obviously, the compiler will catch this and throw an error because core is using a class which was removed in com.google.guava:guava:28.0.0", and you'd be wrong.

Why does this happen? I assume when SBT runs scalac to compile core, it will include all of the necessary jars in the classpath to compile core correctly. Once core is compiled correctly, SBT will then move to hello. When compiling hello SBT won't recompile core, thus avoiding the compile-time guard to make sure CheckedFuture is available in the classpath. This example is similar to how Bazel behaves, so there's no difference there. I can only speculate the reason this happens is to avoid having to compile the entire world. The result, your code will break during runtime, aka Production. Hopefully you'll catch the error when running tests, otherwise, you'll find NoClassDefFoundError or NoSuchMethodError in your logs.

Problems with Abstraction

SBT and Coursier basically deal and hide version conflicts from you, the SBT user. While, yes you can control the rules which it determines how to choose dependency versions, the tool itself does not allow you to detect conflicts by default nor does it give you the proper mechanisms to resolve them. What's the result? Your system crashes or throws a 5xx caused by a NoSuchMethodError. NoSuchMethodError inherits from Error which according to the Java documentation:

...indicates serious problems that a reasonable application should not try to catch.

Bazel, the clearly opinionated

Bazel aims to work with monorepo giving it the freedom to have opinions on certain things. One opinion is to make sure there is only one version of an external library. Bazel forces..err asks you to list all your dependencies in one logical location, which Bazel calls the repository. From the repository, you will then tell Bazel which dependencies (without version) your project will use. For example:

java_library(
  name = "core",
  dependencies = [
    "@maven//com_google_guava_guava" # Note the omission of the version.
  ]
)

When Bazel detects conflicts due to either direct or transitive dependencies it will, by default, choose the highest version. Here's the difference though: Bazel will then place the highest version of the jar when it compiles all projects. Thus avoiding the issue we saw with SBT, as we won't be able to compile the core project. Problem solved, let's go home.

Well actually...

Unfortunately, this does not solve all the issues you might see. Specifically, Bazel does not solve the problem when two transitive dependencies are using two different versions of the same library. Let's break this down. Say you have a project which uses io.grpc:grpc-protobuf:1.25.0 and com.fasterxml.jackson.datatype:jackson-datatype-guava:2.10.1. Jackson-datatype-guava depends on com.google.guava:guava:20.0 and gRPC-Protobufs uses com.google.guava:guava:28.1-android, Bazel will compile and package your project's JAR with com.google.guava:guava:28.1-android, but it won't recompile Jackson-datatype-guava (cause it doesn't have to). If a user hits a code path that uses a method of Jackson-datatype-guava which relies on a class or method present in guava:20.0 but removed in guava:28.1-android, you just a lost a user, or maybe much worse!

Fortunately, if you catch this during testing it can be easily solved by Bazel or SBT, but that's a BIG if. I feel Bazel, or more specifically, rules_jvm_external, has the ability to address this by explicitly calling out these conflicts. I have opened an issue on rules_jvm_external to hopefully add a feature which makes it more clear when transitive conflicts occur. Until then, we're left to manually looking out for these changes, write comprehensive tests (you should be doing this anyway), and hope our users don't do weird things.