Use Gradle module available-at tag instead of files for redirecting jre vs android consumers

[liutikas]

Description

com.google.guava:guava has recently started publishing Gradle metadata. There are funky games with supporting JVM vs Android consumers. In Gradle metadata now there are new variants so when asked for com.google.guava:guava:32.1.3-android, if you are a java project, you get 32.1.3-jre/32.1.3-jre.jar artifact instead.

this happens in here:
https://repo1.maven.org/maven2/com/google/guava/guava/32.1.3-android/guava-32.1.3-android.module

specifically

      "files": [
        {
          "name": "guava-32.1.3-jre.jar",
          "url": "../32.1.3-jre/guava-32.1.3-jre.jar"
        }
      ],

The same is done for -jre version to get android artifact.

This is all good at the compilation level, but when you enable signature verification, gradle gets very confused as it thinks these ../32.1.3-jre/guava-32.1.3-jre.jar are unsigned.

Example

Repro project https://github.com/liutikas/guava-signature-repro

Expected Behavior

Signatures are valid and everything is validated without allowlisting

Actual Behavior

Failure seen:

* What went wrong:
Execution failed for task ':app:checkDebugDuplicateClasses'.
> Dependency verification failed for configuration ':app:debugRuntimeClasspath'
  One artifact failed verification: guava-32.1.3-android.jar (com.google.guava:guava:32.1.3-jre) from repository MavenRepo
  If the artifacts are trustworthy, you will need to update the gradle/verification-metadata.xml file. For more on how to do this, please refer to https://docs.gradle.org/8.7/userguide/dependency_verification.html#sec:troubleshooting-verification in the Gradle documentation.

Packages

No response

Platforms

Android

Checklist

• I agree to follow the code of conduct.

[liutikas]

What would work better in this case is to use available-at instead of files

If guava-32.1.3-android.module instead has this:

{
      "name": "jreApiElements",
      "attributes": {
        "org.gradle.category": "library",
        "org.gradle.dependency.bundling": "external",
        "org.gradle.jvm.version": "8",
        "org.gradle.jvm.environment": "standard-jvm",
        "org.gradle.libraryelements": "jar",
        "org.gradle.usage": "java-api"
      },
      "available-at": {
        "url": "../32.1.3-jre/guava-32.1.3-jre.module",
        "group": "com.google.guava",
        "module": "guava",
        "version": "32.1.3-jre"
      }
    },
    {
      "name": "jreRuntimeElements",
      "attributes": {
        "org.gradle.category": "library",
        "org.gradle.dependency.bundling": "external",
        "org.gradle.jvm.version": "8",
        "org.gradle.jvm.environment": "standard-jvm",
        "org.gradle.libraryelements": "jar",
        "org.gradle.usage": "java-runtime"
      },
      "available-at": {
        "url": "../32.1.3-jre/guava-32.1.3-jre.module",
        "group": "com.google.guava",
        "module": "guava",
        "version": "32.1.3-jre"
      }
    }

signature verification succeeds.

it also reduces redundancy of declared dependencies.

[liutikas]

@jjohannes since you added the Gradle module metadata support in the first place

[liutikas]

Note that available-at matches how Kotlin Multiplatform artifacts handle disambiguation in gradle module metadata.

See https://repo1.maven.org/maven2/com/squareup/okio/okio/3.9.0/okio-3.9.0.module as an example

[jjohannes]

Thank you for investigating and sharing @liutikas.

I am afraid that available-at cannot be used in this case.

The difference between how Guava does variants compared to how it is done elsewhere (e.g. Kotlin Multiplatform) is that Guava uses different versions of the same component to represent the variants.

(1) In Guava you have

com/google/guava                      // group
└── guava                             // name
    ├── 33.1.0-android                // version (!)
    │   └── guava-33.1.0-android.jar
    └── 33.1.0-jre                    // version (!)
        └── guava-33.1.0-jre.jar

(2) If Guava would represent the android variant as classified Jar

Ideally, you would use just multiple artifacts with classifiers. That what I would do for a Java library today if I would publish it with two variants for "jre" and "android". (And the metadata we have now is very close to it, just that it hast to point into the other version folder – ../32.1.0-jre/ – to find the other variant Jar file.)

com/google/guava                      // group
└── guava                             // name
    └── 33.1.0                        // version
        └── guava-33.1.0.jar          // standard/fallback without classifier (jre)
        └── guava-33.1.0-android.jar

(3) If Guava would represent its variants as Kotlin Multiplatform does

If there is a specific reason to have each variant in a separate component (like in Kotlin Multiplatform) you can do that and use available-at to "connect" these components as "variants of one main component":

com/google/guava                      // group
├── guava                             // name (main component)
│   └── 33.1.0                        // version
│       └── guava-33.1.0.module       // only metadata (no Jar) with 'availbale-at'
│
├── guava-android                     // name (separate component that is also used as variant)
│   └── 33.1.0                        // version
│      └── guava-android-33.1.0.jar
└── guava-jre                         // name (separate component that is also used as variant)
    └── 33.1.0                        // version
        └── guava-jre-33.1.0.jar

---

I can still emphasize with why solution (1) was chosen years ago. It is "misusing" the version conflict resolution of the build tools to make sure that never both variants are selected together. Today, Gradle is "variant aware" and solution (2) would be better and the metadata we have tries to get to that as close as possible. (But even if it would be reinvented today, what about Maven which does not have such variant awareness? yet?)

In the current setup (1), we cannot use available-at, because it would point at another version of itself. Which is weird, because not both versions can be selected together. But in the available-at is treated as an edge between two nodes in the dependency graph. But these two nodes can not both exist. I haven't tried, but you might get an "invalid metadata" error directly, because the spec explicitly states that this is not possible here: https://github.com/gradle/gradle/blob/master/platforms/documentation/docs/src/docs/design/gradle-module-metadata-latest-specification.md#available-at-value

Note that the group:module cannot be the same as the group:module of the root component element.

[liutikas]

If I add the following diff to https://github.com/liutikas/guava-signature-repro repro project

diff --git a/gradle/verification-metadata.xml b/gradle/verification-metadata.xml
index 901027d..c611b39 100644
--- a/gradle/verification-metadata.xml
+++ b/gradle/verification-metadata.xml
@@ -500,5 +500,21 @@
             <sha256 value="b51f8867c92b6a722499557fc3a1fdea77bdf9ef574722fe90ce436a29559454"/>
          </artifact>
       </component>
+      <component group="com.google.guava" name="guava" version="32.1.3-jre">
+         <artifact name="guava-32.1.3-jre.module">
+            <ignored-keys>
+               <ignored-key id="BDB5FA4FE719D787FB3D3197F6D4A1D411E9D1AE" />
+            </ignored-keys>
+            <sha256 value="e43870007081e972b73a57fe1ad840ed469290c47bb5d51dac5939756258064d"/>
+         </artifact>
+      </component>
+      <component group="com.google.guava" name="guava" version="32.1.3-android">
+         <artifact name="guava-32.1.3-android.module">
+            <ignored-keys>
+               <ignored-key id="BDB5FA4FE719D787FB3D3197F6D4A1D411E9D1AE" />
+            </ignored-keys>
+            <sha256 value="79b98e8056a24d271609ab9e9b8dc73da2dd3d980aadd9a85021ce3def21beee"/>
+         </artifact>
+      </component>
    </components>
 </verification-metadata>
diff --git a/lib/build.gradle.kts b/lib/build.gradle.kts
new file mode 100644
index 0000000..d261c86
--- /dev/null
+++ b/lib/build.gradle.kts
@@ -0,0 +1,7 @@
+plugins {
+    id("java-library")
+}
+
+dependencies {
+    implementation("com.google.guava:guava:32.1.3-android")
+}
\ No newline at end of file
diff --git a/lib/src/main/java/com/example/lib/MyClass.java b/lib/src/main/java/com/example/lib/MyClass.java
new file mode 100644
index 0000000..58f7288
--- /dev/null
+++ b/lib/src/main/java/com/example/lib/MyClass.java
@@ -0,0 +1,4 @@
+package com.example.lib;
+
+public class MyClass {
+}
\ No newline at end of file
diff --git a/settings.gradle.kts b/settings.gradle.kts
index 7e67774..a997ab2 100644
--- a/settings.gradle.kts
+++ b/settings.gradle.kts
@@ -14,6 +14,9 @@ pluginManagement {
 dependencyResolutionManagement {
     repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS)
     repositories {
+        maven {
+            url = uri("/usr/local/google/home/aurimas/Code/androidx-main/prebuilts/androidx/external")
+        }
         google()
         mavenCentral()
     }
@@ -21,4 +24,4 @@ dependencyResolutionManagement {
 
 rootProject.name = "My Application"
 include(":app")
- 
\ No newline at end of file
+include(":lib")

and the local copy maven has the following diff

diff --git a/com/google/guava/guava/32.1.3-android/guava-32.1.3-android.module b/com/google/guava/guava/32.1.3-android/guava-32.1.3-android.module
index efc9b8ca6..6bb088cc4 100644
--- a/com/google/guava/guava/32.1.3-android/guava-32.1.3-android.module
+++ b/com/google/guava/guava/32.1.3-android/guava-32.1.3-android.module
@@ -164,68 +164,12 @@
         "org.gradle.libraryelements": "jar",
         "org.gradle.usage": "java-api"
       },
-      "dependencies": [
-        {
-          "group": "com.google.guava",
-          "module": "failureaccess",
-          "version": {
-            "requires": "1.0.1"
-          }
-        },
-        {
-          "group": "com.google.guava",
-          "module": "listenablefuture",
-          "version": {
-            "requires": "9999.0-empty-to-avoid-conflict-with-guava"
-          }
-        },
-        {
-          "group": "com.google.code.findbugs",
-          "module": "jsr305",
-          "version": {
-            "requires": "3.0.2"
-          }
-        },
-        {
-          "group": "org.checkerframework",
-          "module": "checker-qual",
-          "version": {
-            "requires": "3.37.0"
-          }
-        },
-        {
-          "group": "com.google.errorprone",
-          "module": "error_prone_annotations",
-          "version": {
-            "requires": "2.21.1"
-          }
-        },
-        {
-          "group": "com.google.j2objc",
-          "module": "j2objc-annotations",
-          "version": {
-            "requires": "2.8"
-          }
-        }
-      ],
-      "files": [
-        {
-          "name": "guava-32.1.3-jre.jar",
-          "url": "../32.1.3-jre/guava-32.1.3-jre.jar"
-        }
-      ],
-      "capabilities": [
-        {
-          "group": "com.google.guava",
-          "name": "guava",
-          "version": "32.1.3-android"
-        },
-        {
-          "group": "com.google.collections",
-          "name": "google-collections",
-          "version": "32.1.3-android"
-        }
-      ]
+      "available-at": {
+        "url": "../32.1.3-jre/guava-32.1.3-jre.module",
+        "group": "com.google.guava",
+        "module": "guava",
+        "version": "32.1.3-jre"
+      }
     },
     {
       "name": "jreRuntimeElements",
@@ -237,61 +181,12 @@
         "org.gradle.libraryelements": "jar",
         "org.gradle.usage": "java-runtime"
       },
-      "dependencies": [
-        {
-          "group": "com.google.guava",
-          "module": "failureaccess",
-          "version": {
-            "requires": "1.0.1"
-          }
-        },
-        {
-          "group": "com.google.guava",
-          "module": "listenablefuture",
-          "version": {
-            "requires": "9999.0-empty-to-avoid-conflict-with-guava"
-          }
-        },
-        {
-          "group": "com.google.code.findbugs",
-          "module": "jsr305",
-          "version": {
-            "requires": "3.0.2"
-          }
-        },
-        {
-          "group": "org.checkerframework",
-          "module": "checker-qual",
-          "version": {
-            "requires": "3.37.0"
-          }
-        },
-        {
-          "group": "com.google.errorprone",
-          "module": "error_prone_annotations",
-          "version": {
-            "requires": "2.21.1"
-          }
-        }
-      ],
-      "files": [
-        {
-          "name": "guava-32.1.3-jre.jar",
-          "url": "../32.1.3-jre/guava-32.1.3-jre.jar"
-        }
-      ],
-      "capabilities": [
-        {
-          "group": "com.google.guava",
-          "name": "guava",
-          "version": "32.1.3-android"
-        },
-        {
-          "group": "com.google.collections",
-          "name": "google-collections",
-          "version": "32.1.3-android"
-        }
-      ]
+      "available-at": {
+        "url": "../32.1.3-jre/guava-32.1.3-jre.module",
+        "group": "com.google.guava",
+        "module": "guava",
+        "version": "32.1.3-jre"
+      }
     }
   ]
 }
diff --git a/com/google/guava/guava/32.1.3-jre/guava-32.1.3-jre.module b/com/google/guava/guava/32.1.3-jre/guava-32.1.3-jre.module
index 98d7896af..705815cc9 100644
--- a/com/google/guava/guava/32.1.3-jre/guava-32.1.3-jre.module
+++ b/com/google/guava/guava/32.1.3-jre/guava-32.1.3-jre.module
@@ -164,68 +164,12 @@
         "org.gradle.libraryelements": "jar",
         "org.gradle.usage": "java-api"
       },
-      "dependencies": [
-        {
-          "group": "com.google.guava",
-          "module": "failureaccess",
-          "version": {
-            "requires": "1.0.1"
-          }
-        },
-        {
-          "group": "com.google.guava",
-          "module": "listenablefuture",
-          "version": {
-            "requires": "9999.0-empty-to-avoid-conflict-with-guava"
-          }
-        },
-        {
-          "group": "com.google.code.findbugs",
-          "module": "jsr305",
-          "version": {
-            "requires": "3.0.2"
-          }
-        },
-        {
-          "group": "org.checkerframework",
-          "module": "checker-qual",
-          "version": {
-            "requires": "3.37.0"
-          }
-        },
-        {
-          "group": "com.google.errorprone",
-          "module": "error_prone_annotations",
-          "version": {
-            "requires": "2.21.1"
-          }
-        },
-        {
-          "group": "com.google.j2objc",
-          "module": "j2objc-annotations",
-          "version": {
-            "requires": "2.8"
-          }
-        }
-      ],
-      "files": [
-        {
-          "name": "guava-32.1.3-android.jar",
-          "url": "../32.1.3-android/guava-32.1.3-android.jar"
-        }
-      ],
-      "capabilities": [
-        {
-          "group": "com.google.guava",
-          "name": "guava",
-          "version": "32.1.3-jre"
-        },
-        {
-          "group": "com.google.collections",
-          "name": "google-collections",
-          "version": "32.1.3-jre"
-        }
-      ]
+      "available-at": {
+        "url": "../32.1.3-android/guava-32.1.3-android.module",
+        "group": "com.google.guava",
+        "module": "guava",
+        "version": "32.1.3-android"
+      }
     },
     {
       "name": "androidRuntimeElements",
@@ -237,61 +181,12 @@
         "org.gradle.libraryelements": "jar",
         "org.gradle.usage": "java-runtime"
       },
-      "dependencies": [
-        {
-          "group": "com.google.guava",
-          "module": "failureaccess",
-          "version": {
-            "requires": "1.0.1"
-          }
-        },
-        {
-          "group": "com.google.guava",
-          "module": "listenablefuture",
-          "version": {
-            "requires": "9999.0-empty-to-avoid-conflict-with-guava"
-          }
-        },
-        {
-          "group": "com.google.code.findbugs",
-          "module": "jsr305",
-          "version": {
-            "requires": "3.0.2"
-          }
-        },
-        {
-          "group": "org.checkerframework",
-          "module": "checker-qual",
-          "version": {
-            "requires": "3.37.0"
-          }
-        },
-        {
-          "group": "com.google.errorprone",
-          "module": "error_prone_annotations",
-          "version": {
-            "requires": "2.21.1"
-          }
-        }
-      ],
-      "files": [
-        {
-          "name": "guava-32.1.3-android.jar",
-          "url": "../32.1.3-android/guava-32.1.3-android.jar"
-        }
-      ],
-      "capabilities": [
-        {
-          "group": "com.google.guava",
-          "name": "guava",
-          "version": "32.1.3-jre"
-        },
-        {
-          "group": "com.google.collections",
-          "name": "google-collections",
-          "version": "32.1.3-jre"
-        }
-      ]
+      "available-at": {
+        "url": "../32.1.3-android/guava-32.1.3-android.module",
+        "group": "com.google.guava",
+        "module": "guava",
+        "version": "32.1.3-android"
+      }
     }
   ]
 }

both ./gradlew app:assembleDebug and ./gradlew lib:jar succeeds.

However, I do see that the spec does seem to forbid this :/

[dsvensson]

@liutikas And the version specific entry in verification-metadata.xml makes all dependabot PRs in need of manual fixups compared to just verifying the signature of an already trusted key.