Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[C++] Allow building against system opentelemetry #44982

Closed
h-vetinari opened this issue Dec 9, 2024 · 26 comments
Closed

[C++] Allow building against system opentelemetry #44982

h-vetinari opened this issue Dec 9, 2024 · 26 comments

Comments

@h-vetinari
Copy link
Contributor

Describe the enhancement requested

I've been wanting to add opentelemetry support in conda-forge for a long time, but given the amount of mutation that happens in packaging there, I wasn't comfortable in doing that before landing conda-forge/arrow-cpp-feedstock#1058.

Now that this is in, I've rebased the PR, but am running into problems pretty quickly. For one, arrow hardcodes

set(opentelemetry-cpp_SOURCE "AUTO")

even if the config explicitly asks for -Dopentelemetry-cpp_SOURCE="SYSTEM". With the following patch

diff --git a/cpp/cmake_modules/ThirdpartyToolchain.cmake b/cpp/cmake_modules/ThirdpartyToolchain.cmake
index 0b215b5b2..b9cc05953 100644
--- a/cpp/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cpp/cmake_modules/ThirdpartyToolchain.cmake
@@ -4977,7 +4977,6 @@ if(ARROW_WITH_OPENTELEMETRY)
   # cURL is required whether we build from source or use an existing installation
   # (OTel's cmake files do not call find_curl for you)
   find_curl()
-  set(opentelemetry-cpp_SOURCE "AUTO")
   resolve_dependency(opentelemetry-cpp)
   set(ARROW_OPENTELEMETRY_LIBS
       opentelemetry-cpp::trace

I then run into further problems in ThirdpartyToolchain.cmake

CMake Error at cmake_modules/ThirdpartyToolchain.cmake:313 (find_package):
  By not providing "Findopentelemetry-cpp.cmake" in CMAKE_MODULE_PATH this
  project has asked CMake to find a package configuration file provided by
  "opentelemetry-cpp", but CMake did not find one.

  Could not find a package configuration file provided by "opentelemetry-cpp"
  with any of the following names:

    opentelemetry-cppConfig.cmake
    opentelemetry-cpp-config.cmake

  Add the installation prefix of "opentelemetry-cpp" to CMAKE_PREFIX_PATH or
  set "opentelemetry-cpp_DIR" to a directory containing one of the above
  files.  If "opentelemetry-cpp" provides a separate development package or
  SDK, be sure it has been installed.
Call Stack (most recent call first):
  cmake_modules/ThirdpartyToolchain.cmake:4980 (resolve_dependency)
  CMakeLists.txt:546 (include)

despite the fact that our otel packages clearly contain the CMake metadata.

CC @lidavidm @kou

Component(s)

C++, Packaging

@assignUser assignUser changed the title ENH: Allow building against system opentelemetry [C Allow building against system opentelemetry Dec 9, 2024
@assignUser assignUser changed the title [C Allow building against system opentelemetry [C++] Allow building against system opentelemetry Dec 9, 2024
@assignUser
Copy link
Member

Hm you explicitly set SYSTEM vs CONDA. Can you try setting adding export CMAKE_PREFIX_PATH="$CONDA_PREFIX" in build.sh?

@assignUser
Copy link
Member

Looking at the code that area of the cmake could probably be substantially improved when we move to newer cmake as discussed in #44950

@kou
Copy link
Member

kou commented Dec 9, 2024

It seems that we can remove the set(opentelemetry-cpp_SOURCE "AUTO").

Could you share full error log URL?

@h-vetinari
Copy link
Contributor Author

Could you share full error log URL?

I restarted a new build, here are the logs

Hm you explicitly set SYSTEM vs CONDA. Can you try setting adding export CMAKE_PREFIX_PATH="$CONDA_PREFIX" in build.sh?

Done, logs here (no change)

@assignUser
Copy link
Member

I don't think we modify the cmake prefix path in anyway that should prevent otel from being found. I can confirm that a normal find_package(opentelemetry-cpp) does indeed find it within a conda env without any special additions.
Could you run a build with:

diff --git a/cpp/cmake_modules/ThirdpartyToolchain.cmake b/cpp/cmake_modules/ThirdpartyToolchain.cmake
index 35ad4089e..c12d6c7af 100644
--- a/cpp/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cpp/cmake_modules/ThirdpartyToolchain.cmake
@@ -4966,8 +4966,9 @@ if(ARROW_WITH_OPENTELEMETRY)
   # cURL is required whether we build from source or use an existing installation
   # (OTel's cmake files do not call find_curl for you)
   find_curl()
-  set(opentelemetry-cpp_SOURCE "AUTO")
+  set(CMAKE_FIND_DEBUG_MODE TRUE)
   resolve_dependency(opentelemetry-cpp)
+  set(CMAKE_FIND_DEBUG_MODE FLASE)
   set(ARROW_OPENTELEMETRY_LIBS
       opentelemetry-cpp::trace
       opentelemetry-cpp::logs

@h-vetinari
Copy link
Contributor Author

@assignUser, here are the logs for your suggestion, lightly cleaned up by resubstituting common variables.

I really don't know ThirdpartyToolchain.cmake, but ISTM that the problem begins by looking for Findopentelemetry-cpp.cmake, rather than just going directly to $PREFIX/lib/cmake/opentelemetry-cpp/opentelemetry-cpp-config.cmake (which exists as I pointed out in the OP; it appears in a kind of no-man's-land in the log below, no idea what to make of that).

CMake Debug Log at cmake_modules/ThirdpartyToolchain.cmake:313 (find_package):
  find_package considered the following paths for
  Findopentelemetry-cpp.cmake:

    $SRC_DIR/cpp/cmake_modules/Findopentelemetry-cpp.cmake
    $PREFIX/lib/cmake/glog/../../../share/glog/cmake/Findopentelemetry-cpp.cmake
    $PREFIX/lib/cmake/grpc/modules/Findopentelemetry-cpp.cmake
    $BUILD_PREFIX/share/cmake-3.31/Modules/Findopentelemetry-cpp.cmake

  The file was not found.

    $PREFIX/lib/cmake/opentelemetry-cpp/opentelemetry-cppConfig.cmake
    $PREFIX/lib/cmake/opentelemetry-cpp/opentelemetry-cpp-config.cmake              # <- see here

  The internally managed CMAKE_FIND_PACKAGE_REDIRECTS_DIR.

    $SRC_DIR/cpp/build/CMakeFiles/pkgRedirects

  <PackageName>_ROOT CMake variable [CMAKE_FIND_USE_PACKAGE_ROOT_PATH].

    $PREFIX

  CMAKE_PREFIX_PATH variable [CMAKE_FIND_USE_CMAKE_PATH].

    none

  CMAKE_FRAMEWORK_PATH and CMAKE_APPBUNDLE_PATH variables
  [CMAKE_FIND_USE_CMAKE_PATH].

    none

  Env variable opentelemetry-cpp_DIR [CMAKE_FIND_USE_CMAKE_ENVIRONMENT_PATH].

    none

  CMAKE_PREFIX_PATH env variable [CMAKE_FIND_USE_CMAKE_ENVIRONMENT_PATH].

    $BUILD_PREFIX/x86_64-conda-linux-gnu/sysroot/usr

  CMAKE_FRAMEWORK_PATH and CMAKE_APPBUNDLE_PATH env variables
  [CMAKE_FIND_USE_CMAKE_ENVIRONMENT_PATH].

  Paths specified by the find_package HINTS option.

    none

  Standard system environment variables
  [CMAKE_FIND_USE_SYSTEM_ENVIRONMENT_PATH].

    $BUILD_PREFIX
    /opt/conda/condabin
    /opt/conda
    /usr/local
    /usr
    /
    /home/conda

  CMake User Package Registry [CMAKE_FIND_USE_PACKAGE_REGISTRY].

    none

  CMake variables defined in the Platform file
  [CMAKE_FIND_USE_CMAKE_SYSTEM_PATH].

    /usr/X11R6
    /usr/pkg
    /opt

  CMake System Package Registry
  [CMAKE_FIND_PACKAGE_NO_SYSTEM_PACKAGE_REGISTRY].

    none

  Paths specified by the find_package PATHS option.

    none

  Prepending the following roots to each prefix:

  CMAKE_FIND_ROOT_PATH

    $PREFIX
    $BUILD_PREFIX/x86_64-conda-linux-gnu/sysroot

  CMAKE_SYSROOT_COMPILE

    none

  CMAKE_SYSROOT_LINK

    none

  CMAKE_SYSROOT

    none

  find_package considered the following locations for opentelemetry-cpp's
  Config module:

    $PREFIX/opentelemetry-cppConfig.cmake
    $PREFIX/opentelemetry-cpp-config.cmake
    $PREFIX/opentelemetry-cppConfig.cmake
    $PREFIX/opentelemetry-cpp-config.cmake
    $BUILD_PREFIX/x86_64-conda-linux-gnu/sysroot/usr/opentelemetry-cppConfig.cmake
    $BUILD_PREFIX/x86_64-conda-linux-gnu/sysroot/usr/opentelemetry-cpp-config.cmake
    $BUILD_PREFIX/x86_64-conda-linux-gnu/sysroot/usr/opentelemetry-cppConfig.cmake
    $BUILD_PREFIX/x86_64-conda-linux-gnu/sysroot/usr/opentelemetry-cpp-config.cmake
    $BUILD_PREFIX/x86_64-conda-linux-gnu/sysroot/opentelemetry-cppConfig.cmake
    $BUILD_PREFIX/x86_64-conda-linux-gnu/sysroot/opentelemetry-cpp-config.cmake
    $SRC_DIR/cpp/build/CMakeFiles/pkgRedirects/opentelemetry-cppConfig.cmake
    $SRC_DIR/cpp/build/CMakeFiles/pkgRedirects/opentelemetry-cpp-config.cmake
    $PREFIX/opentelemetry-cppConfig.cmake
    $PREFIX/opentelemetry-cpp-config.cmake
    $BUILD_PREFIX/x86_64-conda-linux-gnu/sysroot/usr/opentelemetry-cppConfig.cmake
    $BUILD_PREFIX/x86_64-conda-linux-gnu/sysroot/usr/opentelemetry-cpp-config.cmake
    $BUILD_PREFIX/opentelemetry-cppConfig.cmake
    $BUILD_PREFIX/opentelemetry-cpp-config.cmake
    /opt/conda/condabin/opentelemetry-cppConfig.cmake
    /opt/conda/condabin/opentelemetry-cpp-config.cmake
    /opt/conda/opentelemetry-cppConfig.cmake
    /opt/conda/opentelemetry-cpp-config.cmake
    /usr/local/opentelemetry-cppConfig.cmake
    /usr/local/opentelemetry-cpp-config.cmake
    /usr/opentelemetry-cppConfig.cmake
    /usr/opentelemetry-cpp-config.cmake
    /opentelemetry-cppConfig.cmake
    /opentelemetry-cpp-config.cmake
    /home/conda/opentelemetry-cppConfig.cmake
    /home/conda/opentelemetry-cpp-config.cmake
    /opt/opentelemetry-cppConfig.cmake
    /opt/opentelemetry-cpp-config.cmake

  The file was not found.

Call Stack (most recent call first):
  cmake_modules/ThirdpartyToolchain.cmake:4970 (resolve_dependency)
  CMakeLists.txt:546 (include)

@kou
Copy link
Member

kou commented Dec 10, 2024

Findopentelemetry-cpp.cmake must not be a problem.
If Findopentelemetry-cpp.cmake doesn't exist, opentelemetry-cpp-config.cmake is searched automatically.

Does /home/conda/feedstock_root/build_artifacts/apache-arrow_1733797696439/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/cmake/opentelemetry-cpp/opentelemetry-cpp-config.cmake really exist?

@h-vetinari
Copy link
Contributor Author

Yes, see the package contents I posted above, in particular (2nd line under "Files")

lib/cmake/opentelemetry-cpp/opentelemetry-cpp-config.cmake

(which is relative to $PREFIX, which will have a different value per build invocation)

@kou
Copy link
Member

kou commented Dec 10, 2024

Can we check just to be sure? For example, can we log installed locations as verbose log messages?

@kou
Copy link
Member

kou commented Dec 10, 2024

/home/conda/feedstock_root/build_artifacts/apache-arrow_1733797696439/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/opentelemetry-cpp-config.cmake was searched but it doesn't have lib/cmake/opentelemetry-cpp/.

@h-vetinari
Copy link
Contributor Author

Can we check just to be sure?

It's 100% certain to be there.

This is the environment location that everything gets installed in

## Package Plan ##

  environment location: /home/conda/feedstock_root/build_artifacts/apache-arrow_1733797696439/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol

and the package content I keep referencing gets unpacked there.

@h-vetinari
Copy link
Contributor Author

h-vetinari commented Dec 10, 2024

/home/conda/feedstock_root/build_artifacts/apache-arrow_1733797696439/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/opentelemetry-cpp-config.cmake was searched

Let me rewrite this

$PREFIX/opentelemetry-cpp-config.cmake            # the above, reformulated
$PREFIX/lib/cmake/opentelemetry-cpp-config.cmake  # where the file actually is
        ^^^^^^^^^

@kou
Copy link
Member

kou commented Dec 10, 2024

-Dopentelemetry-cpp_DIR=/home/conda/feedstock_root/build_artifacts/apache-arrow_1733797696439/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/cmake/opentelemetry-cpp may be strange.

@kou
Copy link
Member

kou commented Dec 10, 2024

@h-vetinari
Copy link
Contributor Author

-Dopentelemetry-cpp_DIR=$PREFIX/lib/cmake/opentelemetry-cpp may be strange.

This is in reaction to the admonition by CMake mentioned in the OP:

  Add the installation prefix of "opentelemetry-cpp" to CMAKE_PREFIX_PATH or
  set "opentelemetry-cpp_DIR" to a directory containing one of the above
  files.

Do we really need [...]?

If you're referring to -Dopentelemetry-cpp_DIR=%LIBRARY_LIB%\cmake\opentelemetry-cpp, it doesn't work without it either (but causes a warning to add it).

@kou
Copy link
Member

kou commented Dec 10, 2024

Is libopentelemetry-cpp really installed?

The following log doesn't include libopentelemetry-cpp:

2024-12-10T02:32:55.6097843Z ## Package Plan ##
2024-12-10T02:32:55.6098031Z 
2024-12-10T02:32:55.6098463Z   environment location: /home/conda/feedstock_root/build_artifacts/apache-arrow_1733797696439/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol
2024-12-10T02:32:55.6098985Z 
2024-12-10T02:32:55.6115218Z 
2024-12-10T02:32:55.6115716Z The following NEW packages will be INSTALLED:
2024-12-10T02:32:55.6115884Z 
2024-12-10T02:32:55.6116090Z     _libgcc_mutex:                    0.1-conda_forge                conda-forge
2024-12-10T02:32:55.6116354Z     _openmp_mutex:                    4.5-2_gnu                      conda-forge
2024-12-10T02:32:55.6116611Z     aws-c-auth:                       0.8.0-h8cdb5fa_13              conda-forge
2024-12-10T02:32:55.6123436Z     aws-c-cal:                        0.8.1-h3cd0186_1               conda-forge
2024-12-10T02:32:55.6123862Z     aws-c-common:                     0.10.4-hb9d3cd8_0              conda-forge
2024-12-10T02:32:55.6125074Z     aws-c-compression:                0.3.0-ha1387bf_3               conda-forge
2024-12-10T02:32:55.6125463Z     aws-c-event-stream:               0.5.0-hd0e25c6_9               conda-forge
2024-12-10T02:32:55.6125802Z     aws-c-http:                       0.9.2-h6018f57_2               conda-forge
2024-12-10T02:32:55.6126138Z     aws-c-io:                         0.15.3-hde364ed_2              conda-forge
2024-12-10T02:32:55.6126466Z     aws-c-mqtt:                       0.11.0-h5a6ad62_10             conda-forge
2024-12-10T02:32:55.6126790Z     aws-c-s3:                         0.7.5-he8ac2e1_3               conda-forge
2024-12-10T02:32:55.6127118Z     aws-c-sdkutils:                   0.2.1-ha1387bf_2               conda-forge
2024-12-10T02:32:55.6127455Z     aws-checksums:                    0.2.2-ha1387bf_2               conda-forge
2024-12-10T02:32:55.6127783Z     aws-crt-cpp:                      0.29.7-ha1550dd_4              conda-forge
2024-12-10T02:32:55.6128303Z     aws-sdk-cpp:                      1.11.458-hfca1285_2            conda-forge
2024-12-10T02:32:55.6128637Z     azure-core-cpp:                   1.14.0-h5cfcd09_0              conda-forge
2024-12-10T02:32:55.6128976Z     azure-identity-cpp:               1.10.0-h113e628_0              conda-forge
2024-12-10T02:32:55.6129328Z     azure-storage-blobs-cpp:          12.13.0-h3cf044e_1             conda-forge
2024-12-10T02:32:55.6129683Z     azure-storage-common-cpp:         12.8.0-h736e048_1              conda-forge
2024-12-10T02:32:55.6130039Z     azure-storage-files-datalake-cpp: 12.12.0-ha633028_1             conda-forge
2024-12-10T02:32:55.6130423Z     binutils_impl_linux-64:           2.43-h4bf12b8_2                conda-forge
2024-12-10T02:32:55.6130764Z     brotli:                           1.1.0-hb9d3cd8_2               conda-forge
2024-12-10T02:32:55.6131072Z     brotli-bin:                       1.1.0-hb9d3cd8_2               conda-forge
2024-12-10T02:32:55.6131397Z     bzip2:                            1.0.8-h4bc722e_7               conda-forge
2024-12-10T02:32:55.6131717Z     c-ares:                           1.34.3-hb9d3cd8_1              conda-forge
2024-12-10T02:32:55.6132048Z     ca-certificates:                  2024.8.30-hbcca054_0           conda-forge
2024-12-10T02:32:55.6132592Z     clang:                            18.1.8-default_h9e3a008_5      conda-forge
2024-12-10T02:32:55.6132932Z     clang-18:                         18.1.8-default_hf981a13_5      conda-forge
2024-12-10T02:32:55.6133274Z     clang-format:                     18.1.8-default_hf981a13_5      conda-forge
2024-12-10T02:32:55.6133618Z     clang-format-18:                  18.1.8-default_hf981a13_5      conda-forge
2024-12-10T02:32:55.6133964Z     clang-tools:                      18.1.8-default_hf981a13_5      conda-forge
2024-12-10T02:32:55.6134309Z     clangdev:                         18.1.8-default_hf981a13_5      conda-forge
2024-12-10T02:32:55.6134644Z     clangxx:                          18.1.8-default_h3d5eb1d_5      conda-forge
2024-12-10T02:32:55.6134977Z     cudatoolkit:                      11.8.0-h4ba93d1_13             conda-forge
2024-12-10T02:32:55.6135309Z     gflags:                           2.2.2-h5888daf_1005            conda-forge
2024-12-10T02:32:55.6135632Z     glog:                             0.7.1-hbabe93e_0               conda-forge
2024-12-10T02:32:55.6135955Z     gtest:                            1.15.2-h434a139_0              conda-forge
2024-12-10T02:32:55.6136273Z     icu:                              75.1-he02047a_0                conda-forge
2024-12-10T02:32:55.6136606Z     kernel-headers_linux-64:          3.10.0-he073ed8_18             conda-forge
2024-12-10T02:32:55.6136948Z     keyutils:                         1.6.1-h166bdaf_0               conda-forge
2024-12-10T02:32:55.6137902Z     krb5:                             1.21.3-h659f571_0              conda-forge
2024-12-10T02:32:55.6138261Z     ld_impl_linux-64:                 2.43-h712a8e2_2                conda-forge
2024-12-10T02:32:55.6138607Z     libabseil:                        20240722.0-cxx17_h5888daf_1    conda-forge
2024-12-10T02:32:55.6138938Z     libboost:                         1.86.0-h6c02f8c_3              conda-forge
2024-12-10T02:32:55.6139249Z     libboost-devel:                   1.86.0-h1a2810e_3              conda-forge
2024-12-10T02:32:55.6139595Z     libboost-headers:                 1.86.0-ha770c72_3              conda-forge
2024-12-10T02:32:55.6139929Z     libbrotlicommon:                  1.1.0-hb9d3cd8_2               conda-forge
2024-12-10T02:32:55.6140258Z     libbrotlidec:                     1.1.0-hb9d3cd8_2               conda-forge
2024-12-10T02:32:55.6140588Z     libbrotlienc:                     1.1.0-hb9d3cd8_2               conda-forge
2024-12-10T02:32:55.6141065Z     libclang:                         18.1.8-default_hf981a13_5      conda-forge
2024-12-10T02:32:55.6141438Z     libclang-cpp:                     18.1.8-default_hf981a13_5      conda-forge
2024-12-10T02:32:55.6141786Z     libclang-cpp18.1:                 18.1.8-default_hf981a13_5      conda-forge
2024-12-10T02:32:55.6142131Z     libclang13:                       18.1.8-default_h9def88c_5      conda-forge
2024-12-10T02:32:55.6142462Z     libcrc32c:                        1.1.2-h9c3ff4c_0               conda-forge
2024-12-10T02:32:55.6142785Z     libcurl:                          8.10.1-hbbe4b11_0              conda-forge
2024-12-10T02:32:55.6143109Z     libedit:                          3.1.20191231-he28a2e2_2        conda-forge
2024-12-10T02:32:55.6143431Z     libev:                            4.33-hd590300_2                conda-forge
2024-12-10T02:32:55.6143750Z     libevent:                         2.1.12-hf998b51_1              conda-forge
2024-12-10T02:32:55.6144075Z     libgcc:                           14.2.0-h77fa898_1              conda-forge
2024-12-10T02:32:55.6144408Z     libgcc-devel_linux-64:            14.2.0-h41c2201_101            conda-forge
2024-12-10T02:32:55.6144743Z     libgcc-ng:                        14.2.0-h69a702a_1              conda-forge
2024-12-10T02:32:55.6145089Z     libgomp:                          14.2.0-h77fa898_1              conda-forge
2024-12-10T02:32:55.6145741Z     libgoogle-cloud:                  2.31.0-h804f50b_0              conda-forge
2024-12-10T02:32:55.6146117Z     libgoogle-cloud-devel:            2.31.0-hb763f0a_0              conda-forge
2024-12-10T02:32:55.6146710Z     libgoogle-cloud-storage:          2.31.0-h0121fbd_0              conda-forge
2024-12-10T02:32:55.6147052Z     libgoogle-cloud-storage-devel:    2.31.0-h4b7be92_0              conda-forge
2024-12-10T02:32:55.6147388Z     libgrpc:                          1.67.1-hc2c308b_0              conda-forge
2024-12-10T02:32:55.6147715Z     libiconv:                         1.17-hd590300_2                conda-forge
2024-12-10T02:32:55.6148040Z     libllvm18:                        18.1.8-h8b73ec9_2              conda-forge
2024-12-10T02:32:55.6148361Z     liblzma:                          5.6.3-hb9d3cd8_1               conda-forge
2024-12-10T02:32:55.6148682Z     libnghttp2:                       1.64.0-h161d5f1_0              conda-forge
2024-12-10T02:32:55.6149018Z     libprotobuf:                      5.28.2-h5b01275_0              conda-forge
2024-12-10T02:32:55.6149359Z     libre2-11:                        2024.07.02-hbbce691_1          conda-forge
2024-12-10T02:32:55.6149691Z     libsqlite:                        3.47.2-hee588c1_0              conda-forge
2024-12-10T02:32:55.6150015Z     libssh2:                          1.11.1-hf672d98_0              conda-forge
2024-12-10T02:32:55.6150334Z     libstdcxx:                        14.2.0-hc0a3c3a_1              conda-forge
2024-12-10T02:32:55.6150673Z     libstdcxx-devel_linux-64:         14.2.0-h41c2201_101            conda-forge
2024-12-10T02:32:55.6151115Z     libstdcxx-ng:                     14.2.0-h4852527_1              conda-forge
2024-12-10T02:32:55.6151445Z     libthrift:                        0.21.0-h0e7cc3e_0              conda-forge
2024-12-10T02:32:55.6151770Z     libutf8proc:                      2.9.0-hb9d3cd8_1               conda-forge
2024-12-10T02:32:55.6152098Z     libuv:                            1.49.2-hb9d3cd8_0              conda-forge
2024-12-10T02:32:55.6152418Z     libxml2:                          2.13.5-h8d12d68_1              conda-forge
2024-12-10T02:32:55.6152774Z     libzlib:                          1.3.1-hb9d3cd8_2               conda-forge
2024-12-10T02:32:55.6153102Z     llvm-tools:                       18.1.8-h8b73ec9_2              conda-forge
2024-12-10T02:32:55.6153436Z     llvm-tools-18:                    18.1.8-h8b73ec9_2              conda-forge
2024-12-10T02:32:55.6153762Z     llvmdev:                          18.1.8-h8b73ec9_2              conda-forge
2024-12-10T02:32:55.6154087Z     lz4-c:                            1.10.0-h5888daf_1              conda-forge
2024-12-10T02:32:55.6154407Z     minio-server:                     2024.03.10.02.53.48-hbcca054_0 conda-forge
2024-12-10T02:32:55.6154746Z     ncurses:                          6.5-he02047a_1                 conda-forge
2024-12-10T02:32:55.6155070Z     nlohmann_json:                    3.11.3-he02047a_1              conda-forge
2024-12-10T02:32:55.6155396Z     nodejs:                           22.11.0-hf235a45_0             conda-forge
2024-12-10T02:32:55.6155716Z     openssl:                          3.4.0-hb9d3cd8_0               conda-forge
2024-12-10T02:32:55.6156041Z     orc:                              2.0.3-h97ab989_1               conda-forge
2024-12-10T02:32:55.6156368Z     rapidjson:                        1.1.0.post20240409-hac33072_1  conda-forge
2024-12-10T02:32:55.6156704Z     re2:                              2024.07.02-h77b4e00_1          conda-forge
2024-12-10T02:32:55.6157032Z     readline:                         8.2-h8228510_1                 conda-forge
2024-12-10T02:32:55.6157352Z     s2n:                              1.5.9-h0fd0ee4_0               conda-forge
2024-12-10T02:32:55.6157673Z     snappy:                           1.2.1-h8bd8927_1               conda-forge
2024-12-10T02:32:55.6157992Z     sqlite:                           3.47.2-h9eae976_0              conda-forge
2024-12-10T02:32:55.6158316Z     sysroot_linux-64:                 2.17-h4a8ded7_18               conda-forge
2024-12-10T02:32:55.6158646Z     thrift-compiler:                  0.21.0-h5888daf_0              conda-forge
2024-12-10T02:32:55.6159069Z     thrift-cpp:                       0.21.0-ha770c72_0              conda-forge
2024-12-10T02:32:55.6159424Z     tzdata:                           2024b-hc8b5060_0               conda-forge
2024-12-10T02:32:55.6159742Z     xsimd:                            13.0.0-h297d8ca_0              conda-forge
2024-12-10T02:32:55.6160062Z     zlib:                             1.3.1-hb9d3cd8_2               conda-forge
2024-12-10T02:32:55.6160379Z     zstd:                             1.5.6-ha6fb4c9_0               conda-forge

@assignUser
Copy link
Member

Yeah it's not there, I added ls -R $PREFIX/lib/cmake to the build.sh:

$PREFIX/lib/cmake:
absl
AwsCFlags.cmake
AwsCheckHeaders.cmake
aws-cpp-sdk-cognito-identity
aws-cpp-sdk-config
aws-cpp-sdk-core
aws-cpp-sdk-identity-management
aws-cpp-sdk-logs
aws-cpp-sdk-monitoring
aws-cpp-sdk-s3
aws-cpp-sdk-sns
aws-cpp-sdk-sqs
aws-cpp-sdk-sts
aws-cpp-sdk-transfer
AwsCRuntime.cmake
AwsFeatureTests.cmake
AwsFindPackage.cmake
AwsLibFuzzer.cmake
AwsSanitizers.cmake
AWSSDK
AwsSharedLibSetup.cmake
AwsSIMD.cmake
AwsTestHarness.cmake
Boost-1.86.0
boost_atomic-1.86.0
boost_charconv-1.86.0
boost_chrono-1.86.0
boost_cobalt-1.86.0
boost_container-1.86.0
boost_context-1.86.0
boost_contract-1.86.0
boost_coroutine-1.86.0
boost_date_time-1.86.0
BoostDetectToolset-1.86.0.cmake
boost_exception-1.86.0
boost_fiber-1.86.0
boost_filesystem-1.86.0
boost_graph-1.86.0
boost_graph_parallel-1.86.0
boost_headers-1.86.0
boost_iostreams-1.86.0
boost_json-1.86.0
boost_locale-1.86.0
boost_log-1.86.0
boost_log_setup-1.86.0
boost_math_c99-1.86.0
boost_math_c99f-1.86.0
boost_math_c99l-1.86.0
boost_math_tr1-1.86.0
boost_math_tr1f-1.86.0
boost_math_tr1l-1.86.0
boost_mpi-1.86.0
boost_nowide-1.86.0
boost_prg_exec_monitor-1.86.0
boost_process-1.86.0
boost_program_options-1.86.0
boost_random-1.86.0
boost_regex-1.86.0
boost_serialization-1.86.0
boost_stacktrace_addr2line-1.86.0
boost_stacktrace_basic-1.86.0
boost_stacktrace_from_exception-1.86.0
boost_stacktrace_noop-1.86.0
boost_system-1.86.0
boost_test_exec_monitor-1.86.0
boost_thread-1.86.0
boost_timer-1.86.0
boost_type_erasure-1.86.0
boost_unit_test_framework-1.86.0
boost_url-1.86.0
boost_wave-1.86.0
boost_wserialization-1.86.0
c-ares
clang
Crc32c
gflags
glog
google_cloud_cpp_common
google_cloud_cpp_googleapis
google_cloud_cpp_grpc_utils
google_cloud_cpp_rest_internal
google_cloud_cpp_rest_protobuf_internal
google_cloud_cpp_storage
grpc
GTest
libevent
libssh2
libxml2
llvm
OpenSSL
protobuf
RapidJSON
re2
Snappy
thrift
utf8_range
xsimd
zstd

So this is not on our end.

@h-vetinari
Copy link
Contributor Author

So this is not on our end.

I apologise, the recipe had changed enough since I initially opened the PR for otel that I must have missed a spot while rebasing. I had added it as a dependency to libarrow, but not the global build environment (that's used for running build.sh).

I've pushed a new run with more minimal changes (e.g. -Dopentelemetry-cpp_DIR=).

@h-vetinari
Copy link
Contributor Author

Good news: the builds have passed the config stage on all platforms!

I left the patch removing set(opentelemetry-cpp_SOURCE "AUTO") for now, that likely still worth doing.

I'll also need to double-check if a patch I had come with 1.5 years ago for this (that still applies) is still relevant

diff --git a/cpp/src/arrow/util/tracing_internal.h b/cpp/src/arrow/util/tracing_internal.h
index 6ed731599..79907cf0c 100644
--- a/cpp/src/arrow/util/tracing_internal.h
+++ b/cpp/src/arrow/util/tracing_internal.h
@@ -122,13 +122,13 @@ struct Scope {
   opentelemetry::trace::Scope scope_impl;
 };

-opentelemetry::nostd::shared_ptr<opentelemetry::trace::Span>& UnwrapSpan(
+ARROW_EXPORT opentelemetry::nostd::shared_ptr<opentelemetry::trace::Span>& UnwrapSpan(
     ::arrow::util::tracing::SpanDetails* span);

-const opentelemetry::nostd::shared_ptr<opentelemetry::trace::Span>& UnwrapSpan(
+ARROW_EXPORT const opentelemetry::nostd::shared_ptr<opentelemetry::trace::Span>& UnwrapSpan(
     const ::arrow::util::tracing::SpanDetails* span);

-opentelemetry::nostd::shared_ptr<opentelemetry::trace::Span>& RewrapSpan(
+ARROW_EXPORT opentelemetry::nostd::shared_ptr<opentelemetry::trace::Span>& RewrapSpan(
     ::arrow::util::tracing::SpanDetails* span,
     opentelemetry::nostd::shared_ptr<opentelemetry::trace::Span> ot_span);

It likely is though, as shared builds on windows tend to fail if required symbols aren't exported.

@h-vetinari
Copy link
Contributor Author

Hm. Windows fails with something that sounds related to NOMINMAX. Surprising that this triggers based on whether otel is present/enabled or not...

[115/935] Building CXX object src\arrow\CMakeFiles\arrow_util_shared.dir\util\thread_pool.cc.obj
FAILED: src/arrow/CMakeFiles/arrow_util_shared.dir/util/thread_pool.cc.obj 
C:\PROGRA~1\MICROS~2\2022\ENTERP~1\VC\Tools\MSVC\1442~1.344\bin\Hostx64\x64\cl.exe  /nologo /TP -DABSL_CONSUME_DLL -DARROW_EXPORTING -DARROW_HAVE_RUNTIME_AVX2 -DARROW_HAVE_RUNTIME_AVX512 -DARROW_HAVE_RUNTIME_BMI2 -DARROW_HAVE_RUNTIME_SSE4_2 -DBOOST_ALL_DYN_LINK -DBOOST_ALL_NO_LIB -DHAVE_ABSEIL -DHAVE_MSGPACK -DOPENTELEMETRY_ABI_VERSION_NO=1 -DPROTOBUF_USE_DLLS -DURI_STATIC_BUILD -DUSE_IMPORT_EXPORT -D_CRT_SECURE_NO_WARNINGS -D_ENABLE_EXTENDED_ALIGNED_STORAGE -I%SRC_DIR%\cpp\build\src -I%SRC_DIR%\cpp\src -I%SRC_DIR%\cpp\src\generated -external:I%PREFIX%\Library\include -external:W0 /DWIN32 /D_WINDOWS /GR /EHsc /D_SILENCE_TR1_NAMESPACE_DEPRECATION_WARNING  /EHsc /wd5105 /bigobj /utf-8 /W3 /wd4800 /wd4996 /wd4065  /O2 /Ob2 /DNDEBUG -std:c++17 -MD /showIncludes /Fosrc\arrow\CMakeFiles\arrow_util_shared.dir\util\thread_pool.cc.obj /Fdsrc\arrow\CMakeFiles\arrow_util_shared.dir\ /FS -c %SRC_DIR%\cpp\src\arrow\util\thread_pool.cc
%SRC_DIR%\cpp\src\arrow/util/async_generator.h(1636): warning C4003: not enough arguments for function-like macro invocation 'max'

kou added a commit that referenced this issue Dec 10, 2024
### Rationale for this change

We don't need to disallow `-Dopentelemetry-cpp_SOURCE=SYSTEM`.

### What changes are included in this PR?

Remove forcing `opentelemetry-cpp_SOURCE=AUTO`.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.
* GitHub Issue: #44982

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
@kou kou added this to the 19.0.0 milestone Dec 10, 2024
@kou
Copy link
Member

kou commented Dec 10, 2024

Issue resolved by pull request 44983
#44983

@kou kou closed this as completed Dec 10, 2024
@kou
Copy link
Member

kou commented Dec 10, 2024

The Windows related problems are another problems. Let's work on them in another issue.

@h-vetinari
Copy link
Contributor Author

Issue resolved by pull request 44983

Thanks a lot! 🙏

Sorry again for the false start.

The Windows related problems are another problems.

How so? They appeared only once I enabled otel support...

Let's work on them in another issue.

Fine by me!

@kou
Copy link
Member

kou commented Dec 11, 2024

The set(opentelemetry-cpp_SOURCE "AUTO") problem is that -DARROW_DEPENDENCY_SOURCE=SYSTEM is ignored. Bundled opentelemetry-cpp may be used with -DARROW_DEPENDENCY_SOURCE=SYSTEM.

The ARROW_EXPORT/max problems are that our OpenTelemetry integration doesn't work on Windows. It doesn't depend on bundled opentelemetry-cpp nor system opentelemetry-cpp.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants