[Release][Docs] Version of versions.json in uploaded docs not updated

[amoeba]

Describe the bug, including details regarding any error messages, version, and platform.

In apache/arrow-site#553 we noticed that the copy of versions.json contained in the uploaded docs artifact for 18.0.0 was out of date, see this comment. We need to figure out what happened and fix it.

Component(s)

Documentation, Release

[kou]

The document was generated by the test-debian-12-docs job https://github.com/ursacomputing/crossbow/actions/runs/11364841192/job/31611707243 in #44440 (comment) .

[kou]

https://github.com/ursacomputing/crossbow/actions/runs/11364841192/job/31611707243#step:2:123

/usr/bin/git log -1 --format=%H
d0e7d07dfdcbf5382932292e512d7b9013b29cb6

d0e7d07
https://github.com/apache/arrow/commits/main/?after=2df7b23af717f505c16fedbef6e4223ac163ac5a+34

https://github.com/apache/arrow/commits/apache-arrow-18.0.0-rc0/

It seems that I used wrong commit to generate documents and binaries...
Oh...

[amoeba]

Hrm. It looks like there's a pretty decent difference between d0e7d07 and the 18.0.0 RC tag. What needs to happen next?

[kou]

dev/release/03-binary-submit.sh must specify --arrow-site:

diff --git a/dev/release/03-binary-submit.sh b/dev/release/03-binary-submit.sh
index 0dc63aa708..9281597df5 100755
--- a/dev/release/03-binary-submit.sh
+++ b/dev/release/03-binary-submit.sh
@@ -32,7 +32,8 @@ release_tag="apache-arrow-${version}-rc${rc}"
 rc_branch="release-${version_with_rc}"
 
 : ${ARROW_REPOSITORY:="apache/arrow"}
-: ${ARROW_BRANCH:=$release_tag}
+: ${ARROW_BRANCH:=${release_tag}}
+: ${ARROW_SHA:=${release_tag}}
 
 # archery will submit a job with id: "${crossbow_job_prefix}-0" unless there
 # are jobs submitted with the same prefix (the integer at the end is auto
@@ -43,6 +44,7 @@ archery crossbow submit \
     --arrow-version ${version_with_rc} \
     --arrow-remote "https://github.com/${ARROW_REPOSITORY}" \
     --arrow-branch ${ARROW_BRANCH} \
+    --arrow-sha ${ARROW_SHA} \
     --group packaging
 
 # archery will add a comment to the automatically generated PR to track

Without this change on main:

https://github.com/ursacomputing/crossbow/actions/runs/11585712918/job/32255095398#step:2:123

/usr/bin/git log -1 --format=%H
2df7b23af717f505c16fedbef6e4223ac163ac5a

The current HEAD is used.

With this change on main:

https://github.com/ursacomputing/crossbow/actions/runs/11585744340/job/32255177733#step:2:126

/usr/bin/git log -1 --format=%H
9105a4109a80a1c01eabb24ee4b9f7c94ee942cb

The apache-arrow-18.0.0-rc0 was used.

[kou]

Implementation related changes between d0e7d07 and https://github.com/apache/arrow/tree/apache-arrow-18.0.0-rc0 :

• d661607 C++
• 25a2440 C++
• 253cab0 C#
• 1dcd145 C#
• da5a295 C#
• 974a46a Java
• a153dcb Java
• 3ddfb60 Java
• 7df4748 Java
• 34ce119 Ruby
• 8973278 C++

And there are version related changes: 18.0.0-SNAPSHOT -> 18.0.0

Hmm. We should release 18.0.1... Sorry...

[kou]

@raulcd What do you think about this?

[kou]

Ah, wait.
.deb/.rpm packages used the RC0 tarball even when we use different commit for crossbow job.

https://github.com/ursacomputing/crossbow/actions/runs/11364842048/job/31611711388#step:8:80

Downloading... https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-18.0.0-rc0/apache-arrow-18.0.0.tar.gz

[kou]

MATLAB used wrong commit:

https://github.com/ursacomputing/crossbow/actions/runs/11364842086/job/31611715971#step:5:99

-- Arrow version: 18.0.0 (full: '18.0.0-SNAPSHOT')

[kou]

C# used wrong commit:

https://github.com/ursacomputing/crossbow/actions/runs/11364842118/job/31611712501#step:2:122

 /usr/bin/git log -1 --format=%H
d0e7d07dfdcbf5382932292e512d7b9013b29cb6

[kou]

R used wrong commit:

https://github.com/ursacomputing/crossbow/actions/runs/11364841882/job/31611732011#step:7:1247

-- Arrow version: 18.0.0 (full: '18.0.0-SNAPSHOT')

[kou]

Wheel used wrong commit:

https://github.com/ursacomputing/crossbow/actions/runs/11364842277/job/31611713791#step:12:193

-- Arrow version: 18.0.0 (full: '18.0.0-SNAPSHOT')

Oh...

[raulcd]

Were the binaries released from main instead of RC0, then?
I am unsure at this point what was included or not, from what I understand is the list with 11 commits above.
I am ok doing a new patch release but I'll be able to start with it in ~10 days.
I suppose that, apart from the identified commits, we should also add the ones identified with backport-candidate:
https://github.com/apache/arrow/issues?q=is%3Aissue+label%3Abackport-candidate+is%3Aclosed

[raulcd]

Issue resolved by pull request 44577
#44577

[jorisvandenbossche]

Note that the actual reported issue here (versions.json was not updated) is not yet fixed (although the assumption is that it will be fixed by doing a 18.0.1? And otherwise it is something for in the arrow-site repo to fix manually for now)

[kou]

It was already fixed by manual fix in the arrow-site repo.
See apache/arrow-site@a95c937 in apache/arrow-site#553 .

[kou]

Were the binaries released from main instead of RC0, then?

Yes.

I am ok doing a new patch release but I'll be able to start with it in ~10 days.

Or I can be a release manager.
Or we can recruit a volunteer.

I suppose that, apart from the identified commits, we should also add the ones identified with backport-candidate: https://github.com/apache/arrow/issues?q=is%3Aissue+label%3Abackport-candidate+is%3Aclosed

I think that we don't need it.

We've published wheel/sdist but haven't published other affected packages (jar/MATLAB/R/C#) yet.
So we don't need to backport them.

Ah, wait. C# packages are published. We need to cherry-pick only C# related changes.

[raulcd]

Or I can be a release manager. Or we can recruit a volunteer.

Thanks @kou ! Can you manage this one, please? I am going to be mostly out next week.

[amoeba]

I'm also around to help and can probably do more tasks than I did this release (if needed).

[kou]

Sure!
Let's discuss this further in the mailing list: https://lists.apache.org/thread/t8k7l2hsbgdt7cszj7hrpjdfpn91n5zb