License report & automatic `LICENSE` content check #54

snazy · 2024-08-01T14:31:02Z

Adds a Gradle plugin to generate license reports, which is then used for two purposes:

Generate a license report
Verify that dependencies that have to be mentioned in LICENSE are mentioned there

The Gradle check task depends on the "dependencies to mention in LICENSE" - so CI will fail if a dependency isn't mentioned there. Those are dependencies that are not licensed as Apache and are either MIT, BSD, Go, ISC or "Universal Permissive".

The licence report is generated as HTML and packed as a ZIP file for each Gradle project that is considered a distribution (currently only :polaris-service).

A new project aggregated-license-report is there to collect the NOTICE + LICENSE files and all individual projects' license reports and add those to a single ZIP file.

This PR is a port of the same functionality that Nessie used for quite a while now.

Depends on #10 and #53

snazy · 2024-08-01T14:32:49Z

Only the last two commits actually belong to this PR:

one that adds the build code
one that updates the LICENSE file

snazy · 2024-08-07T13:14:56Z

/cc @jbonofre

build-logic/src/main/kotlin/LicenseFileValidation.kt

RussellSpitzer · 2024-08-26T01:18:18Z

build-logic/src/main/kotlin/LicenseFileValidation.kt

+  fun needsNoMention(license: String?): Boolean = license != null && (license.contains("Apache"))
+
+  fun needsMention(license: String?): Boolean =
+    license != null &&


Could do Set: NeedsMentionLicenses.contains(license) instead of || chain

RussellSpitzer · 2024-08-26T01:20:10Z

build-logic/src/main/kotlin/polaris-license-report.gradle.kts

+plugins { id("com.github.jk1.dependency-license-report") }
+
+afterEvaluate {
+  // Need to configure after evaluation, because the spark-extensions project use a custom


Comment from another project I think, shouldn't apply here

yup - copy-paste left-over

RussellSpitzer · 2024-08-26T01:20:53Z

build-logic/src/main/kotlin/polaris-license-report.gradle.kts

+    renderers =
+      arrayOf(InventoryHtmlReportRenderer("index.html"), JsonReportRenderer(), XmlReportRenderer())
+    excludeBoms = true
+    excludes =


I'm not sure these excludes are relevant to Polaris (yet)

RussellSpitzer · 2024-08-26T01:21:06Z

build-logic/src/main/kotlin/polaris-license-report.gradle.kts

+        "io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-alpha"
+      )
+    outputDir = "${project.layout.buildDirectory.get()}/reports/dependency-license"
+    excludeGroups = arrayOf("org.projectnessie.nessie", "org.projectnessie.nessie-integrations")


Not relevant to this project as well

RussellSpitzer

I have a few questions on this related to why we aren't including some elements in our license file.

There are also a few comments/ exclusions that I think are particular to Nessie where this was I think copied from that need to be cleaned up.

RussellSpitzer · 2024-08-28T03:21:03Z

aggregated-license-report/build.gradle.kts

+
+import org.gradle.kotlin.dsl.support.unzipTo
+
+val licenseReports by configurations.creating { description = "Used to reference license reports" }


"Used to generate reference license reports"

RussellSpitzer · 2024-08-28T03:23:34Z

build-logic/src/main/kotlin/LicenseFileValidation.kt

@@ -0,0 +1,117 @@
+/*
+ * Copyright (c) 2024 Snowflake Computing Inc.


Shouldn't this be the Apache license here?

Yes correct

RussellSpitzer · 2024-08-28T03:24:15Z

build-logic/src/main/kotlin/LicenseFileValidation.kt

+import org.gradle.api.GradleException
+
+/**
+ * Validates that all dependencies with MIT/BSD/Go/UPL/ISC licenses, which do not have an Apache


Comment is no longer accurate

RussellSpitzer · 2024-08-28T03:27:26Z

gradle/libs.versions.toml

@@ -28,6 +28,10 @@ swagger = "1.6.14"


 [libraries]
+#
+# If a dependency is removed, check whether the LICENSE and/or NOTICE files need to be adopted


Couldn't we just automatically check whether or not our license file matches our dependencies ? We could always do this in a future PR

RussellSpitzer · 2024-08-28T03:29:10Z

gradle/license/allowed-licenses.json

@@ -0,0 +1,96 @@
+{


This is probably a feature for the plugin and not our build, but it would be great if this json could include the rules for license reproduction rather than us coding it ourselves. (IE. Apache needs x type license, MIT needs full repro, etc ...)

RussellSpitzer

I think we just need to fix the Copyright notice on LicenseFileValidation.kt and the comment in the same file and we are good to merge here.

jbonofre · 2024-08-28T05:55:20Z

aggregated-license-report/build.gradle.kts

@@ -0,0 +1,59 @@
+/*
+ * Copyright (c) 2024 Snowflake Computing Inc.


We can update the header here.

jbonofre · 2024-08-28T05:56:16Z

build-logic/src/main/kotlin/LicenseFileValidation.kt

@@ -0,0 +1,117 @@
+/*
+ * Copyright (c) 2024 Snowflake Computing Inc.


Yes correct

RussellSpitzer · 2024-08-29T03:34:11Z

@snazy There are still some Snowflake headers in there, could you please tidy them up?

snazy · 2024-09-04T14:28:50Z

Ping @RussellSpitzer

RussellSpitzer · 2024-09-04T15:35:36Z

Thanks @snazy!

snazy force-pushed the license-report-notice-check branch from 3cfc906 to e70f1a5 Compare August 1, 2024 14:58

snazy changed the title ~~License report & automatic NOTICE content check~~ License report & automatic LICENSE content check Aug 1, 2024

snazy force-pushed the license-report-notice-check branch 3 times, most recently from 16605b2 to b622b0b Compare August 7, 2024 13:13

snazy marked this pull request as ready for review August 7, 2024 13:14

snazy requested a review from a team as a code owner August 7, 2024 13:14

snazy force-pushed the license-report-notice-check branch from b622b0b to befed2c Compare August 8, 2024 22:56

snazy force-pushed the license-report-notice-check branch from befed2c to 346f06e Compare August 22, 2024 12:02

snazy requested review from jbonofre, ashvina, RussellSpitzer, vvcephei and jackye1995 as code owners August 22, 2024 12:02

snazy force-pushed the license-report-notice-check branch 4 times, most recently from ed95168 to 2214ad4 Compare August 25, 2024 06:38

RussellSpitzer reviewed Aug 26, 2024

View reviewed changes

build-logic/src/main/kotlin/LicenseFileValidation.kt Outdated Show resolved Hide resolved

RussellSpitzer reviewed Aug 26, 2024

View reviewed changes

snazy force-pushed the license-report-notice-check branch from 2214ad4 to bc0dd26 Compare August 26, 2024 10:26

snazy requested a review from RussellSpitzer August 26, 2024 16:18

RussellSpitzer reviewed Aug 28, 2024

View reviewed changes

RussellSpitzer previously approved these changes Aug 28, 2024

View reviewed changes

jbonofre reviewed Aug 28, 2024

View reviewed changes

snazy dismissed RussellSpitzer’s stale review via 8797220 August 28, 2024 06:28

snazy force-pushed the license-report-notice-check branch from bc0dd26 to 8797220 Compare August 28, 2024 06:28

snazy requested a review from takidau as a code owner August 28, 2024 06:28

snazy force-pushed the license-report-notice-check branch from 2cd772e to 267030d Compare August 29, 2024 11:46

snazy added 7 commits August 30, 2024 14:03

Add license report + NOTICE content check

9860a72

Update LICENSE

997f5dd

Must mention Apache licensed artifacts, address review comments

c4c4063

update LICENSE

ae4fd03

fix license header

d28bb6b

typos et al

10e65d9

another left-over licesne

6d709a8

snazy force-pushed the license-report-notice-check branch from 267030d to 6d709a8 Compare August 30, 2024 12:03

RussellSpitzer approved these changes Sep 4, 2024

View reviewed changes

RussellSpitzer merged commit fe4d16c into apache:main Sep 4, 2024
3 checks passed

snazy deleted the license-report-notice-check branch September 4, 2024 15:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

License report & automatic `LICENSE` content check #54

License report & automatic `LICENSE` content check #54

snazy commented Aug 1, 2024 •

edited

Loading

snazy commented Aug 1, 2024 •

edited

Loading

snazy commented Aug 7, 2024

RussellSpitzer Aug 26, 2024

RussellSpitzer Aug 26, 2024

snazy Aug 26, 2024

RussellSpitzer Aug 26, 2024

snazy Aug 26, 2024

RussellSpitzer Aug 26, 2024

snazy Aug 26, 2024

RussellSpitzer left a comment

RussellSpitzer Aug 28, 2024

RussellSpitzer Aug 28, 2024

jbonofre Aug 28, 2024

RussellSpitzer Aug 28, 2024

RussellSpitzer Aug 28, 2024

RussellSpitzer Aug 28, 2024

RussellSpitzer left a comment

jbonofre Aug 28, 2024

jbonofre Aug 28, 2024

RussellSpitzer commented Aug 29, 2024

snazy commented Sep 4, 2024

RussellSpitzer commented Sep 4, 2024


		import org.gradle.kotlin.dsl.support.unzipTo

		val licenseReports by configurations.creating { description = "Used to reference license reports" }

		@@ -0,0 +1,117 @@
		/*
		* Copyright (c) 2024 Snowflake Computing Inc.

		@@ -0,0 +1,59 @@
		/*
		* Copyright (c) 2024 Snowflake Computing Inc.

License report & automatic LICENSE content check #54

License report & automatic LICENSE content check #54

Conversation

snazy commented Aug 1, 2024 • edited Loading

snazy commented Aug 1, 2024 • edited Loading

snazy commented Aug 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RussellSpitzer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RussellSpitzer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RussellSpitzer commented Aug 29, 2024

snazy commented Sep 4, 2024

RussellSpitzer commented Sep 4, 2024

License report & automatic `LICENSE` content check #54

License report & automatic `LICENSE` content check #54

snazy commented Aug 1, 2024 •

edited

Loading

snazy commented Aug 1, 2024 •

edited

Loading