Bump Spark 3.5.4 RC2

[pan3793]

VOTE 3.5.4 RC1 https://lists.apache.org/thread/btdsos7b92962lh10hzswkfy3oj1v0fb
VOTE 3.5.4 RC2 https://lists.apache.org/thread/lttktocm6g6pvhg8zmp37jbhpf51wbyv

[pan3793]

the upgrading causes a core dump in iceberg-spark-3.5's tests

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f8529a34563, pid=83775, tid=83793
#
# JRE version: OpenJDK Runtime Environment Zulu17.54+21-CRaC-CA (17.0.13+11) (build 17.0.13+11-LTS)
# Java VM: OpenJDK 64-Bit Server VM Zulu17.54+21-CRaC-CA (17.0.13+11-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0xc34563]  ProtectionDomainEntry::object_no_keepalive()+0x3
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /home/runner/work/iceberg/iceberg/spark/v3.5/spark/core.83775)
#
# An error report file with more information is saved as:
# /home/runner/work/iceberg/iceberg/spark/v3.5/spark/hs_err_pid83775.log
#
# Compiler replay data is saved as:
# /home/runner/work/iceberg/iceberg/spark/v3.5/spark/replay_pid83775.log
#
# If you would like to submit a bug report, please visit:
#   http://www.azul.com/support/
#
> Task :iceberg-spark:iceberg-spark-3.5_2.12:test FAILED

[pan3793]

the failure seems related to arrow

minimal reproduce command

./gradlew -DsparkVersions=3.5 -DscalaVersion=2.12 -DhiveVersions= -DflinkVersions= -DkafkaVersions= \
    :iceberg-spark:iceberg-spark-3.5_2.12:test --tests=TestRewriteDataFilesAction \
    -Pquick=true -x javadoc

[pan3793]

update: the failure is related to SPARK-50235, and the test passed on Spark 3.5.4 RC1 with reverting that patch

[viirya]

Please check for some discussion there apache/spark#49131 (comment)

[LuciferYang]

Please check for some discussion there apache/spark#49131 (comment)

Yes, I tested it locally and found that after adding a no-op closeIfFreeable method to the IcebergArrowColumnVector.java in version 3.5, all the test cases in TestRewriteDataFilesAction were able to pass. We can further test it after the release of 3.5.4 RC2.

[pan3793]

@viirya @LuciferYang thank you for the guidance, let me try

[pan3793]

will rename to closeIfFreeable after RC2 available

[singhpk234]

If a column vector is writable or constant, it should override this method and do nothing.

we have ConstantColumnVector as well, should we do it vector classes which absolutely requires it ?

[pan3793]

I checked the codebase, other vectors overwrite void close() {}, so they are not affected

[singhpk234]

Nice catch @pan3793 !

[singhpk234]

If a column vector is writable or constant, it should override this method and do nothing.

we have ConstantColumnVector as well, should we do it vector classes which absolutely requires it ?

[pan3793]

FYI @LuciferYang @viirya @dongjoon-hyun @wForget

CI is green now, and I think the current Spark branch-3.5 is in good shape for Iceberg.

[dongjoon-hyun]

Thank you for checking Apache Iceberg side, @pan3793 .