[Nix] Version of maven dependency com.google.code.gson is not locked

[juliankuners]

Recently, the mvnHash in the k nix derivation had to be changed even though nothing changed in the repository. This caused CI to break on pull requests, see here and here.

The hash changing is apparently caused by a non-locked version of com.google.code.gson. In particular, I compared upstream with version k-7.1.211 that was still stored in my nix store. The last change of mvnHash, except for the recent hash change due to same kind of issue, happened in 7.1.210.

Comparing the respective nix derivation outputs showed the following differences:

Previously

<?xml version="1.0" encoding="UTF-8"?>
<metadata modelVersion="1.1.0">
  <groupId>com.google.code.gson</groupId>
  <artifactId>gson</artifactId>
  <versioning>
    <latest>2.12.1</latest>
    <release>2.12.1</release>
    <versions>
      <version>1.1</version>
      <version>1.4</version>
      <version>1.5</version>
      <version>1.6</version>
      <version>1.7</version>
      <version>1.7.1</version>
      <version>1.7.2</version>
      <version>2.0</version>
      <version>2.1</version>
      <version>2.2</version>
      <version>2.2.1</version>
      <version>2.2.2</version>
      <version>2.2.3</version>
      <version>2.2.4</version>
      <version>2.3</version>
      <version>2.3.1</version>
      <version>2.4</version>
      <version>2.5</version>
      <version>2.6</version>
      <version>2.6.1</version>
      <version>2.6.2</version>
      <version>2.7</version>
      <version>2.8.0</version>
      <version>2.8.1</version>
      <version>2.8.2</version>
      <version>2.8.3</version>
      <version>2.8.4</version>
      <version>2.8.5</version>
      <version>2.8.6</version>
      <version>2.8.7</version>
      <version>2.8.8</version>
      <version>2.8.9</version>
      <version>2.9.0</version>
      <version>2.9.1</version>
      <version>2.10</version>
      <version>2.10.1</version>
      <version>2.11.0</version>
      <version>2.12.0</version>
      <version>2.12.1</version>
    </versions>
    <lastUpdated>20250130225640</lastUpdated>
  </versioning>
</metadata>

Now

<?xml version="1.0" encoding="UTF-8"?>
<metadata modelVersion="1.1.0">
  <groupId>com.google.code.gson</groupId>
  <artifactId>gson</artifactId>
  <versioning>
    <latest>2.13.1</latest>
    <release>2.13.1</release>
    <versions>
      <version>1.1</version>
      <version>1.4</version>
      <version>1.5</version>
      <version>1.6</version>
      <version>1.7</version>
      <version>1.7.1</version>
      <version>1.7.2</version>
      <version>2.0</version>
      <version>2.1</version>
      <version>2.2</version>
      <version>2.2.1</version>
      <version>2.2.2</version>
      <version>2.2.3</version>
      <version>2.2.4</version>
      <version>2.3</version>
      <version>2.3.1</version>
      <version>2.4</version>
      <version>2.5</version>
      <version>2.6</version>
      <version>2.6.1</version>
      <version>2.6.2</version>
      <version>2.7</version>
      <version>2.8.0</version>
      <version>2.8.1</version>
      <version>2.8.2</version>
      <version>2.8.3</version>
      <version>2.8.4</version>
      <version>2.8.5</version>
      <version>2.8.6</version>
      <version>2.8.7</version>
      <version>2.8.8</version>
      <version>2.8.9</version>
      <version>2.9.0</version>
      <version>2.9.1</version>
      <version>2.10</version>
      <version>2.10.1</version>
      <version>2.11.0</version>
      <version>2.12.0</version>
      <version>2.12.1</version>
      <version>2.13.0</version>
      <version>2.13.1</version>
    </versions>
    <lastUpdated>20250424011321</lastUpdated>
  </versioning>
</metadata>

This shows that the version of GSON must be locked, otherwise mvnHash must continously be updated every time a new version of GSON is released. In addition, it should be validated whether other maven dependencies are not locked as well and could cause the same kind of issue.

Non-locked versions also have the major disadvantage that old revisions stop to build successfully, requiring an updated hash. In turn, updating the hash makes the builds not reproducible anymore due to old revisions requiring the use of new versions of dependencies.

[juliankuners]

Maven apparently has no complete lock file support for dependencies and specifically transitive dependencies, but dependencies can be manually restricted in the pom.xml file, as long as the list is complete. Though this does not cover Maven plugins that are used in the command line that also fetch dependencies into the Maven repository, therefore having an impact on the nix derivation that is being built. I guess the command line maven plugins can be locked in the settings.xml file, though I haven't tried this.

There is dependency-lock-maven-plugin, which can be used in the main pom.xml and the submodules. This plugin creates a json file that captures all dependencies and transitive dependencies being used to built artifacts. The check goal of this plugin then reads this lock file and fails the build if the actually used dependencies do not match the lock file. While this allows to iteratively extend the dependency version restrictions to enforce the locked versions, it will still make a previously successfully building nix derivation fail, akin to the mvnHash failing. Also, this plugin does not cover the Maven plugins that are used on the command line in the nix derivation for k.

Another approach would be to create a Maven repository mirror containing all dependencies that are required for the nix derivation and enforcing the use of that mirror, while disallowing the use of the central Maven repository. This would indefinitely lock all versions, but restrict the flexibility of updating or changing dependencies.

Also, other build tools such as Gradle seem to support lock files, though the effort seems quite substantial, considering that the Java tooling is not under frequent development as of right now. Additionally, changing build tools without proper consideration of both build tools capabilities and differences is also very risky, as unforeseen challenges might arise.

Conclusion

Right now either the dependency locking in pom.xml and settings.xml with semi-automated scripts or using a Maven repository mirror seem to be the most viable solutions. Not addressing this issue will cause old nix derivations fail to build due to changed upstream dependencies and requiring updating mvnHash more or less frequently, but apparently rather less often.

What are your thoughts on this? @tothtamas28 @jberthold @palinatolmach @ehildenb
We should also ping people from Pi Squared. Do you have an idea, who to ping on this topic?

[juliankuners]

@dwightguth and @Robertorosmaninho, do you guys have a particular opinion on this issue and the potential solutions?

[jberthold]

I am in favour of the maven mirror suggestion to address this problem. It will stabilise the versions we have, and I don't see why the dependencies of the K frontend would need upgrades since we don't actively develop new features.
However, I am not familiar with the details of how to restrict maven to only use this mirror, nor do I know where we would host it.

The dependency-lock-maven-plugin does not prevent builds from failing but it can still be useful to add to the build because we get more visibility into the reasons for a broken build.

A complete change of the build away from maven makes a lot of sense in the long-term but it carries high risk and does not help for versions before the build system switch.

[juliankuners]

The pull request #4820 mostly fixes the replicability problem. It introduces an informational lock file that can be used to identify changed transitive dependencies quickly and fixes the replicability issue with the GSON library. Additionally, it was discovered that the GSON issue was caused instead by an updated xml file in the offline Maven repository that contains up-to-date information about all GSON libraries. This file is critical for building the project. Therefore, the pull request added logic to provide this file in a replicable manner instead of fetching the upstream file, which can cause the mvnHash to change, when new versions get released.

Should the mvnHash change again, then the respective transitive dependency or respective maven xml file can be fixed in the nix derivation. Therefore, this issue can be closed in my opinion, even though a migration to a build tool that has proper lock file support would allow for higher replicability. Though such a migration is considered out of scope for this issue.