Unable to build on Windows

[jrobcary]

I am trying to build windows static. I configure with

'/winsame/cary/dev3/contrib-llvm17_vs2022/cmake-3.26.4-ser/bin/cmake.exe'
-DCMAKE_INSTALL_PREFIX:PATH=D:/winsame/cary/dev3/contrib-llvm17_vs2022/ginkgo-1.8.0-ser
-DCMAKE_BUILD_TYPE:STRING=Release
-DCMAKE_COLOR_MAKEFILE:BOOL=FALSE
-DCMAKE_VERBOSE_MAKEFILE:BOOL=TRUE
-DCMAKE_C_COMPILER:FILEPATH='C:/Program Files/LLVM17/bin/clang-cl.exe'
-DCMAKE_CXX_COMPILER:FILEPATH='C:/Program Files/LLVM17/bin/clang-cl.exe'
-DCMAKE_C_FLAGS:STRING='/W3 -Wno-implicit-function-declaration'
-DCMAKE_CXX_FLAGS:STRING='/W3 /GR /EHsc /Zc:__cplusplus /D_HAS_AUTO_PTR_ETC /bigobj -fms-compatibility-version=19 -Wno-register -Wno-implicit-function-declaration -Wno-microsoft-include'
-DBUILD_SHARED_LIBS=OFF
-DCMAKE_CUDA_COMPILER:FILEPATH='C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.8/bin/nvcc.exe'
-DCMAKE_CUDA_ARCHITECTURES:STRING='61;70;80'
-DGINKGO_BUILD_HIP:BOOL=OFF
-DGINKGO_BUILD_OMP:BOOL=OFF
-DGINKGO_BUILD_MPI:BOOL=OFF
-G 'NMake Makefiles JOM' D:/winsame/cary/xsd3/builds/ginkgo-1.8.0

But when I build (with jom), it dies at

LINK: command "C:\PROGRA~1\LLVM17\bin\lld-link.exe @CMakeFiles\mixed-spmv.dir\objects1.rsp /out:mixed-spmv.exe /implib:mixed-spmv.lib /pdb:D:\winsame\cary\xsd3\builds\ginkgo-1.8.0\ser\examples\mixed-spmv\mixed-spmv.pdb /version:0.0 /machine:x64 /INCREMENTAL:NO /subsystem:console -LIBPATH:C:\PROGRA~1\NVIDIA~2\CUDA\v11.8\lib\x64 ..\..\lib\ginkgo.lib ..\..\lib\ginkgo_omp.lib ..\..\lib\ginkgo_cuda.lib C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64\cudart.lib C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64\cublas.lib C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64\cublasLt.lib C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64\cusparse.lib C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64\curand.lib C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64\cufft.lib ..\..\lib\ginkgo_reference.lib ..\..\lib\ginkgo_hip.lib ..\..\lib\ginkgo_dpcpp.lib ..\..\lib\ginkgo_device.lib cudadevrt.lib cudart_static.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST /MANIFESTFILE:mixed-spmv.exe.manifest" failed (exit code 1) with the following output:
lld-link: error: undefined symbol: void __cdecl gko::kernels::reference::dense::add_scaled_identity<float, float>(class std::shared_ptr<class gko::ReferenceExecutor const>, class gko::matrix::Dense<float> const *, class gko::matrix::Dense<float> const *, class gko::matrix::Dense<float> *)^M
>>> referenced by ginkgo.lib(dense.cpp.obj):(public: virtual void __cdecl gko::detail::RegisteredOperation<class `<auto> __cdecl gko::matrix::dense::`anonymous namespace'::make_add_scaled_identity<class gko::matrix::Dense<float> const *&, class gko::matrix::Dense<float> const *&, class gko::matrix::Dense<float> *&>(class matrix::dense::Dense<float> const *&, class matrix::dense::Dense<float> const *&, class matrix::dense::<auto> *&)'::`1'::<lambda_1>>::run(class std::shared_ptr<class gko::ReferenceExecutor const>) const)^M
...

[upsj]

Just making sure, can you reproduce this when using MSBuild (default) or Ninja (-GNinja) as the CMake generator? We don't yet test compilation with clang-cl on Windows, so this might just be an issue that slipped through the cracks

[jrobcary]

I can build with Ninja and visual studio 2022. However, hoping to use clang-cl to have a consistent build chain.

[jrobcary]

Also fails with LLVM19:

[10/560] Linking CXX executable reference\test\base\batch_multi_vector_kernels.exe
FAILED: reference/test/base/batch_multi_vector_kernels.exe
cmd.exe /C "cd . && D:\winsame\cary\dev2\contrib-llvm19_vs2022\cmake-3.26.4-ser\bin\cmake.exe -E vs_link_exe --intdir=reference\test\base\CMakeFiles\reference_test_base_batch_multi_vector_kernels.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100190~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100190~1.0\x64\mt.exe --manifests  -- C:\PROGRA~1\LLVM19\bin\lld-link.exe  reference\test\base\CMakeFiles\reference_test_base_batch_multi_vector_kernels.dir\batch_multi_vector_kernels.cpp.obj  /out:reference\test\base\batch_multi_vector_kernels.exe /implib:reference\test\base\batch_multi_vector_kernels.lib /pdb:reference\test\base\batch_multi_vector_kernels.pdb /version:0.0 /machine:x64 /INCREMENTAL:NO /subsystem:console -LIBPATH:C:\PROGRA~1\NVIDIA~2\CUDA\v11.8\lib\x64 lib\ginkgo_gtest_main.lib  lib\ginkgo.lib  lib\gtest.lib  lib\ginkgo_omp.lib  lib\ginkgo_cuda.lib  "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64\cudart.lib"  "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64\cublas.lib"  "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64\cublasLt.lib"  "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64\cusparse.lib"  "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64\curand.lib"  "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64\cufft.lib"  lib\ginkgo_reference.lib  lib\ginkgo_hip.lib  lib\ginkgo_dpcpp.lib  lib\ginkgo_device.lib  cudadevrt.lib  cudart_static.lib  kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib && cd ."
LINK: command "C:\PROGRA~1\LLVM19\bin\lld-link.exe reference\test\base\CMakeFiles\reference_test_base_batch_multi_vector_kernels.dir\batch_multi_vector_kernels.cpp.obj /out:reference\test\base\batch_multi_vector_kernels.exe /implib:reference\test\base\batch_multi_vector_kernels.lib /pdb:reference\test\base\batch_multi_vector_kernels.pdb /version:0.0 /machine:x64 /INCREMENTAL:NO /subsystem:console -LIBPATH:C:\PROGRA~1\NVIDIA~2\CUDA\v11.8\lib\x64 lib\ginkgo_gtest_main.lib lib\ginkgo.lib lib\gtest.lib lib\ginkgo_omp.lib lib\ginkgo_cuda.lib C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64\cudart.lib C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64\cublas.lib C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64\cublasLt.lib C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64\cusparse.lib C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64\curand.lib C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64\cufft.lib lib\ginkgo_reference.lib lib\ginkgo_hip.lib lib\ginkgo_dpcpp.lib lib\ginkgo_device.lib cudadevrt.lib cudart_static.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST /MANIFESTFILE:reference\test\base\batch_multi_vector_kernels.exe.manifest" failed (exit code 1) with the following output:
lld-link: error: undefined symbol: void __cdecl gko::kernels::reference::dense::add_scaled_identity<float, float>(class std::shared_ptr<class gko::ReferenceExecutor const>, class gko::matrix::Dense<float> const *, class gko::matrix::Dense<float> const *, class gko::matrix::Dense<float> *)
>>> referenced by ginkgo.lib(dense.cpp.obj):(public: virtual void __cdecl gko::detail::RegisteredOperation<class `<auto> __cdecl gko::matrix::dense::`anonymous namespace'::make_add_scaled_identity<class gko::matrix::Dense<float> const *&, class gko::matrix::Dense<float> const *&, class gko::matrix::Dense<float> *&>(class matrix::dense::Dense<float> const *&, class matrix::dense::Dense<float> const *&, class matrix::dense::<auto> *&)'::`1'::<lambda_1>>::run(class std::shared_ptr<class gko::ReferenceExecutor const>) const)

lld-link: error: undefined symbol: void __cdecl gko::kernels::reference::dense::add_scaled_identity<double, double>(class std::shared_ptr<class gko::ReferenceExecutor const>, class gko::matrix::Dense<double> const *, class gko::matrix::Dense<double> const *, class gko::matrix::Dense<double> *)
>>> referenced by ginkgo.lib(dense.cpp.obj):(public: virtual void __cdecl gko::detail::RegisteredOperation<class `<auto> __cdecl gko::matrix::dense::`anonymous namespace'::make_add_scaled_identity<class gko::matrix::Dense<double> const *&, class gko::matrix::Dense<double> const *&, class gko::matrix::Dense<double> *&>(class matrix::dense::Dense<double> const *&, class matrix::dense::Dense<double> const *&, class matrix::dense::<auto> *&)'::`1'::<lambda_1>>::run(class std::shared_ptr<class gko::ReferenceExecutor const>) const)

lld-link: error: undefined symbol: void __cdecl gko::kernels::reference::dense::add_scaled_identity<class std::complex<float>, class std::complex<float>>(class std::shared_ptr<class gko::ReferenceExecutor const>, class gko::matrix::Dense<class std::complex<float>> const *, class gko::matrix::Dense<class std::complex<float>> const *, class gko::matrix::Dense<class std::complex<float>> *)
>>> referenced by ginkgo.lib(dense.cpp.obj):(public: virtual void __cdecl gko::detail::RegisteredOperation<class `<auto> __cdecl gko::matrix::dense::`anonymous namespace'::make_add_scaled_identity<class gko::matrix::Dense<class std::complex<float>> const *&, class gko::matrix::Dense<class std::complex<float>> const *&, class gko::matrix::Dense<class std::complex<float>> *&>(class matrix::dense::Dense<class std::complex<float>> const *&, class matrix::dense::Dense<class std::complex<float>> const *&, class matrix::dense::<auto> *&)'::`1'::<lambda_1>>::run(class std::shared_ptr<class gko::ReferenceExecutor const>) const)

lld-link: error: undefined symbol: void __cdecl gko::kernels::reference::dense::add_scaled_identity<class std::complex<double>, class std::complex<double>>(class std::shared_ptr<class gko::ReferenceExecutor const>, class gko::matrix::Dense<class std::complex<double>> const *, class gko::matrix::Dense<class std::complex<double>> const *, class gko::matrix::Dense<class std::complex<double>> *)
>>> referenced by ginkgo.lib(dense.cpp.obj):(public: virtual void __cdecl gko::detail::RegisteredOperation<class `<auto> __cdecl gko::matrix::dense::`anonymous namespace'::make_add_scaled_identity<class gko::matrix::Dense<class std::complex<double>> const *&, class gko::matrix::Dense<class std::complex<double>> const *&, class gko::matrix::Dense<class std::complex<double>> *&>(class matrix::dense::Dense<class std::complex<double>> const *&, class matrix::dense::Dense<class std::complex<double>> const *&, class matrix::dense::<auto> *&)'::`1'::<lambda_1>>::run(class std::shared_ptr<class gko::ReferenceExecutor const>) const)
ninja: build stopped: subcommand failed.

Configured as

'/winsame/cary/dev2/contrib-llvm19_vs2022/cmake-3.26.4-ser/bin/cmake.exe' \
  -DCMAKE_INSTALL_PREFIX:PATH=D:/winsame/cary/dev2/contrib-llvm19_vs2022/ginkgo-1.8.0-ser \
  -DCMAKE_BUILD_TYPE:STRING=Release \
  -DCMAKE_COLOR_MAKEFILE:BOOL=FALSE \
  -DCMAKE_VERBOSE_MAKEFILE:BOOL=TRUE \
  -DCMAKE_C_COMPILER:FILEPATH='C:/Program Files/LLVM19/bin/clang-cl.exe' \
  -DCMAKE_CXX_COMPILER:FILEPATH='C:/Program Files/LLVM19/bin/clang-cl.exe' \
  -DCMAKE_C_FLAGS:STRING='/W3 -Wno-implicit-function-declaration' \
  -DCMAKE_CXX_FLAGS:STRING='/W3 /GR /EHsc /Zc:__cplusplus /D_HAS_AUTO_PTR_ETC /bigobj -fms-compatibility-version=19 -Wno-register -Wno-implicit-function-declaration -Wno-microsoft-include' \
  -DBUILD_SHARED_LIBS=OFF \
  -DCMAKE_CUDA_COMPILER:FILEPATH='C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.8/bin/nvcc.exe' \
  -DCMAKE_CUDA_ARCHITECTURES:STRING='61;70;80' \
  -DGINKGO_BUILD_HIP:BOOL=OFF \
  -DGINKGO_BUILD_OMP:BOOL=OFF \
  -DGINKGO_BUILD_MPI:BOOL=OFF \
  -G 'Ninja' D:/winsame/cary/xsd2/builds/ginkgo-1.8.0

[jrobcary]

@upsj , any ideas?

[upsj]

I can reproduce the issue with clang-cl, I'll take a closer look.

[upsj]

Seems like a more complex issue, but I get the feeling this would be solved with Ginkgo 2.0 and another attempt at #715

[jrobcary]

How do I get 2.0?

[upsj]

Sorry for the misleading wording - Ginkgo 2.0 is a future release where we need to make a few interface breaks to fix architectural issues that we can't address with a minor release.

[yhmtsai]

Hi @jrobcary

We figure out the failed reason behind this.
It is available in #1725 , which also contains more details about the issue.
Could you check whether it works for you?

[jrobcary]

Working with

cary@grating/.../ginkgo$ git remote -v
origin  https://github.com/ginkgo-project/ginkgo.git (fetch)
origin  https://github.com/ginkgo-project/ginkgo.git (push)
cary@grating/.../ginkgo$ git branch -v
* develop dc8cfeb0a Merge Fix complex solver benchmark

Which results the link error (I think same)

LINK: command "C:\PROGRA~1\LLVM17\bin\lld-link.exe reference\test\base\CMakeFiles\reference_test_base_batch_multi_vector_kernels.dir\batch_multi_vector_kernels.cpp.obj /out:reference\test\base\batch_multi_vector_kernels.exe /implib:reference\test\base\batch_multi_vector_kernels.lib /pdb:reference\test\base\batch_multi_vector_kernels.pdb /version:0.0 /machine:x64 /INCREMENTAL:NO /subsystem:console -LIBPATH:C:\PROGRA~1\NVIDIA~2\CUDA\v11.8\lib\x64 lib\ginkgo_gtest_main.lib lib\ginkgo.lib lib\gtest.lib lib\ginkgo_omp.lib lib\ginkgo_cuda.lib C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64\cudart.lib C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64\cublas.lib C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64\cublasLt.lib C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64\cusparse.lib C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64\curand.lib C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64\cufft.lib lib\ginkgo_reference.lib lib\ginkgo_hip.lib lib\ginkgo_dpcpp.lib lib\ginkgo_device.lib cudadevrt.lib cudart_static.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST /MANIFESTFILE:reference\test\base\batch_multi_vector_kernels.exe.manifest" failed (exit code 1) with the following output:
lld-link: error: undefined symbol: void __cdecl gko::kernels::reference::dense::add_scaled_identity<float, float>(class std::shared_ptr<class gko::ReferenceExecutor const>, class gko::matrix::Dense<float> const *, class gko::matrix::Dense<float> const *, class gko::matrix::Dense<float> *)
>>> referenced by ginkgo.lib(dense.cpp.obj):(public: virtual void __cdecl gko::detail::RegisteredOperation<class `<auto> __cdecl gko::matrix::dense::`anonymous namespace'::make_add_scaled_identity<class gko::matrix::Dense<float> const *&, class gko::matrix::Dense<float> const *&, class gko::matrix::Dense<float> *&>(class matrix::dense::Dense<float> const *&, class matrix::dense::Dense<float> const *&, class matrix::dense::<auto> *&)'::`1'::<lambda_1>>::run(class std::shared_ptr<class gko::ReferenceExecutor const>) const)

Do I have the correct branch?

[jrobcary]

Changed to clang_cl branch and it builds!

[yhmtsai]

Thanks for confirmation!