GH-39214: [Java] Support reproducible build #39215

jbonofre · 2023-12-13T14:23:11Z

Rationale for this change

This PR add reproducible builds support.

Are these changes tested?

I tested the builds are reproducible on arrow-vector, arrow-memory, ... using mvn clean verify artifact:compare.

Are there any user-facing changes?

No

Closes: [Java] Provide reproducible builds arrow-java#146
GitHub Issue: ARROW-293: [C++] Implement Arrow IO interfaces for operating system files #146

github-actions · 2023-12-13T14:23:39Z

⚠️ GitHub issue apache/arrow-java#146 has been automatically assigned in GitHub to PR creator.

lidavidm

Thanks!

jbonofre · 2023-12-13T20:09:07Z

@lidavidm let me check why GH Action is not happy 😃

lidavidm · 2023-12-13T20:12:49Z

@github-actions crossbow submit java

github-actions · 2023-12-13T20:15:19Z

Revision: 502eb17

Submitted crossbow builds: ursacomputing/crossbow @ actions-b2f5994f24

Task	Status
java-jars
verify-rc-source-java-linux-almalinux-8-amd64
verify-rc-source-java-linux-conda-latest-amd64
verify-rc-source-java-linux-ubuntu-20.04-amd64
verify-rc-source-java-linux-ubuntu-22.04-amd64
verify-rc-source-java-macos-amd64

java/pom.xml

davisusanibar · 2023-12-13T21:54:44Z

java/format/pom.xml

+  <properties>
+    <project.build.outputTimestamp>2023-12-13T00:00:00Z</project.build.outputTimestamp>
+  </properties>


Can you provide some commands or steps for validating a reproducible build, such as in the Arrow Format module?

When I tried these commands, it finished ok with or without any changes:

$ mvn clean install $ mvn clean verify artifact:compare ... [INFO] Saved info on build to /Users/dsusanibar/fork/arrow/java/format/target/arrow-format-15.0.0-SNAPSHOT.buildinfo [INFO] Checking against reference build from central... [INFO] Reference buildinfo file not found: it will be generated from downloaded reference artifacts [INFO] Reference build java.version: 11 (from MANIFEST.MF Build-Jdk-Spec) [INFO] Reference build os.name: Unix (from pom.properties newline) [INFO] Minimal buildinfo generated from downloaded artifacts: /Users/dsusanibar/fork/arrow/java/format/target/reference/arrow-format-15.0.0-SNAPSHOT.buildinfo [INFO] Reproducible Build output summary: 5 files ok [INFO] Reproducible Build output comparison saved to /Users/dsusanibar/fork/arrow/java/format/target/arrow-format-15.0.0-SNAPSHOT.buildcompare ...

You are right for the check: mvn clean install and then mvn clean verify artifact:compare do a check.
This PR ensures we have reproducible build, not meaning that some module are not ok already (AFAIR, flight and algorithm were not reproducible).

You can also check with artifact:check-buildplan.

What we might want is to add this to CI somewhere (or possibly the release verification process) so we can make sure it stays working.

@lidavidm good idea. Let me check to add a workflow in GH actions.

jbonofre · 2023-12-18T09:36:30Z

FYI, I'm resuming work on this PR:

checking the build failure on CI
checking to add a GH Action workflow to verify

lidavidm · 2023-12-18T13:07:22Z

checking to add a GH Action workflow to verify

It might be best to add it to an existing workflow, or again, adding it to the release verification script even (what do you think @assignUser?)

jbonofre · 2023-12-18T13:21:21Z

checking to add a GH Action workflow to verify

It might be best to add it to an existing workflow, or again, adding it to the release verification script even (what do you think @assignUser?)

@lidavidm it makes sense, I do like this. Thanks !

assignUser · 2023-12-19T04:47:23Z

How exactly does this work, it compares the current build against a reference build? How does that work for dev versions?

We do have binary verification scripts so it would make sense to add it there, that would automatically also run it. dev/release/verify-release-candidate.sh with `TEST_JARS=1' is run by the jar binary verification script (this downloads previously build nightly jars iirc) cc @raulcd who is a bit more familiar with the release scripts

lidavidm · 2023-12-19T14:11:10Z

https://maven.apache.org/plugins/maven-artifact-plugin/compare-mojo.html

Compare current build output (from package) against reference either previously install-ed or downloaded from a remote repository: comparison results go to .buildcompare file.

So maybe we finally have a useful thing to do in the binary JAR verification step.

jbonofre · 2023-12-19T14:15:20Z

@lidavidm yup, it's what I plan to do (and I used while working on this PR 😄 ).

First I will fix the failing checks (AMD64 Conda Integration Test, etc).

jbonofre · 2023-12-19T14:16:22Z

Error:  Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.4.1:enforce (enforce-maven-version) on project arrow-bom: 
Error:  Rule 0: org.apache.maven.enforcer.rules.version.RequireMavenVersion failed with message:
Error:  Detected Maven Version: 3.5.0 is not in the allowed range [3.6.3,).
Error:  -> [Help 1]

That's due to the Apache POM update. I will propose fix as part of this PR.

lidavidm · 2023-12-19T14:20:55Z

See #39279

lidavidm · 2024-01-19T14:04:36Z

@github-actions crossbow submit java

github-actions · 2024-01-19T14:05:12Z

Only contributors can submit requests to this bot. Please ask someone from the community for help with getting the first commit in.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/7584811545

lidavidm · 2024-01-19T15:05:44Z

Hmm well Archery doesn't like me but it seems CI still has a few issues even after a re-run.

jbonofre · 2024-01-19T15:27:42Z

@lidavidm yup, to be honest, I don't know why it's failing 😄 I will probably try on main branch to compare.
I will do a new pass again, maybe I missed something 😄

kou · 2024-01-19T22:11:08Z

@assignUser Could you take a look at #39215 (comment) and #39215 (comment) ?

#39610 is related.

assignUser · 2024-01-20T03:52:54Z

@kou hm weird asf membership is public for David so the bot should pick it up... if this keeps happening we may need to add a non gh api based method to identify committers (e.g. committer list, IIRC we have a yaml one around somehwere right?)

lidavidm · 2024-01-22T14:46:46Z

I fixed my membership after that. You may want to update the bot's error message to tell people what to do, I only knew because I happened to see someone else ask the same question in a private channel.

jbonofre · 2024-01-25T13:48:42Z

I'm rebasing and doing new tests on this one.

Co-authored-by: Sutou Kouhei <[email protected]>

…aven-assemly-plugin

lidavidm · 2024-03-31T15:32:01Z

@github-actions crossbow submit java

github-actions · 2024-03-31T15:34:24Z

Revision: e6e5d16

Submitted crossbow builds: ursacomputing/crossbow @ actions-02510129a8

Task	Status
java-jars
verify-rc-source-java-linux-almalinux-8-amd64
verify-rc-source-java-linux-conda-latest-amd64
verify-rc-source-java-linux-ubuntu-20.04-amd64
verify-rc-source-java-linux-ubuntu-22.04-amd64
verify-rc-source-java-macos-amd64

jbonofre · 2024-03-31T16:53:48Z

@lidavidm as I'm back on Arrow, I'm resuming several PRs/works I have in progress 😄 Thanks !

vibhatha · 2024-05-31T23:57:33Z

@jbonofre need any help here?

jbonofre · 2024-06-01T09:16:58Z

I can resume on this one yeah :)

jbonofre · 2024-09-03T07:41:43Z

I'm back on Arrow. I will rebase and retest this PR.

github-actions · 2024-11-26T19:16:06Z

⚠️ GitHub issue #39214 has no components, please add labels for components.

kou · 2024-11-27T00:55:07Z

We're moving java/ to https://github.com/apache/arrow-java .
Could you open a new PR in apache/arrow-java?

jbonofre requested a review from lidavidm as a code owner December 13, 2023 14:23

github-actions bot added Component: Java awaiting review Awaiting review labels Dec 13, 2023

lidavidm approved these changes Dec 13, 2023

View reviewed changes

github-actions bot added awaiting merge Awaiting merge and removed awaiting review Awaiting review labels Dec 13, 2023

davisusanibar reviewed Dec 13, 2023

View reviewed changes

java/pom.xml Outdated Show resolved Hide resolved

davisusanibar reviewed Dec 13, 2023

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting merge Awaiting merge labels Dec 14, 2023

jbonofre force-pushed the GH-39214 branch from 502eb17 to 7dfca83 Compare December 15, 2023 09:34

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Dec 15, 2023

jbonofre force-pushed the GH-39214 branch from 7dfca83 to ca6c208 Compare December 19, 2023 14:22

jbonofre mentioned this pull request Dec 19, 2023

MINOR: [Java] Bump org.apache:apache from 18 to 31 in /java #39279

Closed

jbonofre force-pushed the GH-39214 branch 3 times, most recently from a65906a to fe22e18 Compare December 19, 2023 14:57

jbonofre force-pushed the GH-39214 branch from 8414e04 to a812077 Compare January 17, 2024 19:25

jbonofre force-pushed the GH-39214 branch 3 times, most recently from 35de715 to 4f4343f Compare January 30, 2024 16:27

jbonofre and others added 6 commits March 30, 2024 06:51

GH-39214: [Java] Support reproducible build

c8b30cb

Update dev/release/test-helper.rb

f0feef8

Co-authored-by: Sutou Kouhei <[email protected]>

Update dev/release/test-helper.rb

51ce4f7

Co-authored-by: Sutou Kouhei <[email protected]>

Use 1970-01-01 as project.build.outputTimestamp

fe05eb1

Use 1980-01-01 as project.build.outputTimestamp to avoid issue with m…

2f36124

…aven-assemly-plugin

Update outputTimestamp in test-helper

e6e5d16

jbonofre force-pushed the GH-39214 branch from 4f4343f to e6e5d16 Compare March 30, 2024 05:53

assignUser mentioned this pull request Mar 31, 2024

[CI] Update crossbow message about private org visibility #40916

Open

jbonofre mentioned this pull request May 2, 2024

[Java] java-jars fails with "We have duplicated artifacts attached" apache/arrow-java#92

Closed

kou closed this Nov 27, 2024

GH-39214: [Java] Support reproducible build #39215

GH-39214: [Java] Support reproducible build #39215

Conversation

jbonofre commented Dec 13, 2023 • edited by github-actions bot Loading

Rationale for this change

Are these changes tested?

Are there any user-facing changes?

github-actions bot commented Dec 13, 2023

lidavidm left a comment

Choose a reason for hiding this comment

jbonofre commented Dec 13, 2023

lidavidm commented Dec 13, 2023

github-actions bot commented Dec 13, 2023

davisusanibar Dec 13, 2023

Choose a reason for hiding this comment

davisusanibar Dec 13, 2023

Choose a reason for hiding this comment

jbonofre Dec 14, 2023

Choose a reason for hiding this comment

jbonofre Dec 14, 2023

Choose a reason for hiding this comment

lidavidm Dec 14, 2023

Choose a reason for hiding this comment

jbonofre Dec 15, 2023

Choose a reason for hiding this comment

jbonofre commented Dec 18, 2023

lidavidm commented Dec 18, 2023

jbonofre commented Dec 18, 2023

assignUser commented Dec 19, 2023

lidavidm commented Dec 19, 2023

jbonofre commented Dec 19, 2023

jbonofre commented Dec 19, 2023

lidavidm commented Dec 19, 2023

lidavidm commented Jan 19, 2024

github-actions bot commented Jan 19, 2024

lidavidm commented Jan 19, 2024

jbonofre commented Jan 19, 2024

kou commented Jan 19, 2024

assignUser commented Jan 20, 2024

lidavidm commented Jan 22, 2024

jbonofre commented Jan 25, 2024

lidavidm commented Mar 31, 2024

github-actions bot commented Mar 31, 2024

jbonofre commented Mar 31, 2024

vibhatha commented May 31, 2024

jbonofre commented Jun 1, 2024

jbonofre commented Sep 3, 2024

github-actions bot commented Nov 26, 2024

kou commented Nov 27, 2024

jbonofre commented Dec 13, 2023 •

edited by github-actions bot

Loading