Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Arrow] Another try at building Arrow #5425

Merged
merged 22 commits into from
Nov 6, 2022
Merged
Show file tree
Hide file tree
Changes from 18 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 98 additions & 0 deletions A/Arrow/build_tarballs.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# Note that this script can accept some limited command-line arguments, run
# `julia build_tarballs.jl --help` to see a usage message.
using BinaryBuilder, Pkg

name = "Arrow"
version = v"10.0.0"

# Collection of sources required to complete build
sources = [
ArchiveSource("https://github.com/apache/arrow/archive/refs/tags/apache-arrow-$(version).zip", "42995abe620c41c42b8fbc486f1c63a1e5b1da534718ac66dbc790a88efeaa37")
evetion marked this conversation as resolved.
Show resolved Hide resolved
DirectorySource("./bundled")
]

# Bash recipe for building across all platforms
script = raw"""

cd $WORKSPACE/srcdir/arrow-apache-arrow-*

# Set toolchain for building external deps
for f in ${WORKSPACE}/srcdir/patches/*.patch; do
atomic_patch -p1 ${f}
done

cd cpp && mkdir build_dir && cd build_dir

# Ignore check for availibility on older macOS versions
if [[ "${target}" == x86_64-apple-darwin* ]]; then
CXXFLAGS="${CXXFLAGS} -D_LIBCPP_DISABLE_AVAILABILITY"
fi

CMAKE_FLAGS=(-DCMAKE_INSTALL_PREFIX=$prefix
-DCMAKE_TOOLCHAIN_FILE=${CMAKE_TARGET_TOOLCHAIN}
-DARROW_CXXFLAGS="${CXXFLAGS}"
-DCMAKE_BUILD_TYPE=Release
-DARROW_BUILD_UTILITIES=OFF
-DARROW_WITH_UTF8PROC=OFF
-DARROW_DEPENDENCY_SOURCE=SYSTEM
-DARROW_VERBOSE_THIRDPARTY_BUILD=ON
-DARROW_BUILD_STATIC=OFF
-DARROW_DATASET=ON
-DARROW_COMPUTE=OFF
-DARROW_WITH_RE2=OFF
-DARROW_WITH_BZ2=ON
-DARROW_IPC=OFF
-DARROW_WITH_LZ4=ON
-DARROW_WITH_ZSTD=OFF
-DARROW_WITH_ZLIB=ON
-DARROW_WITH_SNAPPY=ON
-DARROW_THRIFT_USE_SHARED=ON
-DARROW_PARQUET=ON
-DPARQUET_BUILD_EXECUTABLES=OFF
-DARROW_SIMD_LEVEL=NONE
-DARROW_USE_XSIMD=OFF
-DARROW_JEMALLOC=OFF
-Dxsimd_SOURCE=AUTO)

# CMake is doubling the suffixes...
if [[ "${target}" == *-mingw32 ]]; then
ln -s ${prefix}/lib/libthrift.dll.a ${prefix}/lib/libthrift.a.dll.a
ln -s ${prefix}/lib/libutf8proc.a ${prefix}/lib/libutf8proc.dll.a.a
fi
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds like something to fix in CMake, rather than creating files with wrong extensions which go into the tarball (at very least you should delete them at the end)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still of the opinion this should be fixed in CMake, not worked around in this way. If that's of any help, you want to look at to CMAKE_FIND_LIBRARY_SUFFIXES.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, I will look into it fixing/upstreaming (also the other bugs).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is now removed, turned out to be unnecessary in Arrow v10.


cmake .. "${CMAKE_FLAGS[@]}"

make -j${nproc}

# Remove double suffixes
if [[ "${target}" == *-mingw32 ]]; then
rm ${prefix}/lib/libthrift.a.dll.a
rm ${prefix}/lib/libutf8proc.dll.a.a
fi

make install
"""

# These are the platforms we will build for by default, unless further
# platforms are passed in on the command line
platforms = expand_cxxstring_abis(supported_platforms())

# The products that we will ensure are always built
products = [
LibraryProduct("libparquet", :libparquet),
LibraryProduct("libarrow", :libarrow)
]

# Dependencies that must be installed before this package can be built
dependencies = [
Dependency("boost_jll", compat="=1.76.0")
Dependency("Zlib_jll")
Dependency("Bzip2_jll", compat="1.0.7")
evetion marked this conversation as resolved.
Show resolved Hide resolved
Dependency("Lz4_jll")
Dependency("Thrift_jll")
Dependency("snappy_jll")
Dependency("CompilerSupportLibraries_jll")
giordano marked this conversation as resolved.
Show resolved Hide resolved
]
evetion marked this conversation as resolved.
Show resolved Hide resolved

# Build the tarballs, and possibly a `build.jl` as well.
build_tarballs(ARGS, name, version, sources, script, platforms, products, dependencies; julia_compat="1.6", preferred_gcc_version=v"8")
evetion marked this conversation as resolved.
Show resolved Hide resolved
15 changes: 15 additions & 0 deletions A/Arrow/bundled/patches/cxxflags.patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
diff --git a/cpp/cmake_modules/SetupCxxFlags.cmake b/cpp/cmake_modules/SetupCxxFlags.cmake
index cef4eb0b1..97f866938 100644
--- a/cpp/cmake_modules/SetupCxxFlags.cmake
+++ b/cpp/cmake_modules/SetupCxxFlags.cmake
@@ -29,7 +29,9 @@ if(NOT DEFINED ARROW_CPU_FLAG)
elseif(CMAKE_SYSTEM_PROCESSOR MATCHES "aarch64|ARM64|arm64")
set(ARROW_CPU_FLAG "armv8")
elseif(CMAKE_SYSTEM_PROCESSOR MATCHES "armv7")
- set(ARROW_CPU_FLAG "armv7")
+ set(ARROW_CPU_FLAG "armv7")
+ elseif(CMAKE_SYSTEM_PROCESSOR MATCHES "armv6")
+ set(ARROW_CPU_FLAG "armv7") # Prevents another patch, will add -latomic to linker
elseif(CMAKE_SYSTEM_PROCESSOR MATCHES "powerpc|ppc")
set(ARROW_CPU_FLAG "ppc")
elseif(CMAKE_SYSTEM_PROCESSOR MATCHES "s390x")
12 changes: 12 additions & 0 deletions A/Arrow/bundled/patches/thrift.patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
diff --git a/cpp/cmake_modules/FindThrift.cmake b/cpp/cmake_modules/FindThrift.cmake
index 2f20a5cb5..2d1e728aa 100644
--- a/cpp/cmake_modules/FindThrift.cmake
+++ b/cpp/cmake_modules/FindThrift.cmake
@@ -146,6 +146,7 @@ if(Thrift_FOUND)
endif()
set_target_properties(thrift::thrift
PROPERTIES IMPORTED_LOCATION "${THRIFT_LIB}"
+ IMPORTED_IMPLIB "${THRIFT_LIB}"
evetion marked this conversation as resolved.
Show resolved Hide resolved
INTERFACE_INCLUDE_DIRECTORIES "${THRIFT_INCLUDE_DIR}")
if(WIN32 AND NOT MSVC_TOOLCHAIN)
# We don't need this for Visual C++ because Thrift uses
14 changes: 14 additions & 0 deletions A/Arrow/bundled/patches/toolchain.patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
diff --git a/cpp/cmake_modules/ThirdpartyToolchain.cmake b/cpp/cmake_modules/ThirdpartyToolchain.cmake
index b7cd31f3d..676f9b93b 100644
--- a/cpp/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cpp/cmake_modules/ThirdpartyToolchain.cmake
@@ -753,8 +753,7 @@ endif()
# directory. This leads to issues if the variables are exported in a subshell
# and the invocation of make/ninja is in distinct subshell without the same
# environment (CC/CXX).
-set(EP_COMMON_TOOLCHAIN -DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}
- -DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER})
+set(EP_COMMON_TOOLCHAIN -DCMAKE_TOOLCHAIN_FILE=$ENV{CMAKE_TARGET_TOOLCHAIN})

if(CMAKE_AR)
set(EP_COMMON_TOOLCHAIN ${EP_COMMON_TOOLCHAIN} -DCMAKE_AR=${CMAKE_AR})