CMake: Support CMake build within an isolated Conda environment for Linux #5322

HuidaeCho · 2025-03-06T22:22:18Z

This PR fixes two issues when building with Conda libraries for Linux:

Avoid using the built-in iconv because Conda dependency libraries use libiconv
- Without this PR, libgrass_gis.so is compiled with the built-in iconv (function names iconv_open/close), but building db/drivers/pg, which depends on libgrass_gis.so, fails because its another dependency PostgreSQL requires libiconv (function names libiconv_open/close).
- For this, use cmake .. -DIconv_IS_BUILT_IN=FALSE
Make DEPENDS public
- Again, without this PR, db/driver/pg fails because it cannot find the dependency libraries of libgrass_gis.so.

…inux

echoix · 2025-03-06T22:31:14Z

Does this take into account the recent PR #5305?

HuidaeCho · 2025-03-06T22:44:30Z

Does this take into account the recent PR #5305?

Yes, I used the latest main branch and I saw that merged code.

wenzeslaus · 2025-03-06T23:11:52Z

The CI is not using conda, so I assume this works well without conda. (?)

HuidaeCho · 2025-03-07T01:08:05Z

The CI is not using conda, so I assume this works well without conda. (?)

Yes, tested without Conda as well.

nilason · 2025-03-07T19:38:19Z

Regarding Iconv: I suggest to configure with -DIconv_IS_BUILT_IN=OFF|ON, it shouldn't be our responsibility to enforce this either way for any specific build environment.

I very much doubt linking every dependency to our libraries as "PUBLIC" is the best approach. I believe it rather conceal the real problem, which we rather should try to find out what it is.
Are you building in a "pure" conda environment, with a recipe and so forth, or are you just using conda packages? And could you please post the error message.

wenzeslaus · 2025-03-07T20:12:10Z

I very much doubt linking every dependency to our libraries as "PUBLIC" is the best approach. I believe it rather conceal the real problem, which we rather should try to find out what it is.
Are you building in a "pure" conda environment, with a recipe and so forth, or are you just using conda packages? And could you please post the error message.

If I depend on A, I assume it will bring all its dependencies, i.e., the transitive dependencies should be implicitly carried over. Is that what we are talking about here?

HuidaeCho · 2025-03-07T20:25:17Z

Regarding Iconv: I suggest to configure with -DIconv_IS_BUILT_IN=OFF|ON, it shouldn't be our responsibility to enforce this either way for any specific build environment.

Makes sense. I'm doing the same for Linux anyway. Ah.. never thought about that!

HuidaeCho · 2025-03-07T20:27:40Z

I very much doubt linking every dependency to our libraries as "PUBLIC" is the best approach. I believe it rather conceal the real problem, which we rather should try to find out what it is.
Are you building in a "pure" conda environment, with a recipe and so forth, or are you just using conda packages? And could you please post the error message.

If I depend on A, I assume it will bring all its dependencies, i.e., the transitive dependencies should be implicitly carried over. Is that what we are talking about here?

Yes, you're right. A explicitly depends on B, which depends on C. A doesn't directly depends on C. We still need to carry over both B and C for A, unless we want to accumulate all these dependencies manually in CMakeLists.txt.

HuidaeCho · 2025-03-07T20:40:55Z

I believe it rather conceal the real problem, which we rather should try to find out what it is.

It might be a LD_LIBRARY_PATH=${CONDA_PREFIX}/lib thing... Let me try that. FYI, this is the error message:

[ 12%] Linking C executable ../../output/lib64/grass85/driver/db/pg
/home/hcho/usr/local/opt/miniconda/envs/grass/bin/../lib/gcc/x86_64-conda-linux-gnu/14.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: warning: libz.so.1, needed by ../../output/lib64/grass85/lib/libgrass_gis.so.8.5.0dev, not found (try using -rpath or -rpath-link)
/home/hcho/usr/local/opt/miniconda/envs/grass/bin/../lib/gcc/x86_64-conda-linux-gnu/14.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: warning: libgomp.so.1, needed by ../../output/lib64/grass85/lib/libgrass_gis.so.8.5.0dev, not found (try using -rpath or -rpath-link)
/home/hcho/usr/local/opt/miniconda/envs/grass/bin/../lib/gcc/x86_64-conda-linux-gnu/14.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: warning: libzstd.so.1, needed by ../../output/lib64/grass85/lib/libgrass_gis.so.8.5.0dev, not found (try using -rpath or -rpath-link)
/home/hcho/usr/local/opt/miniconda/envs/grass/bin/../lib/gcc/x86_64-conda-linux-gnu/14.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: warning: libiconv.so.2, needed by ../../output/lib64/grass85/lib/libgrass_gis.so.8.5.0dev, not found (try using -rpath or -rpath-link)
/home/hcho/usr/local/opt/miniconda/envs/grass/bin/../lib/gcc/x86_64-conda-linux-gnu/14.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: ../../output/lib64/grass85/lib/libgrass_gis.so.8.5.0dev: undefined reference to `uncompress'
/home/hcho/usr/local/opt/miniconda/envs/grass/bin/../lib/gcc/x86_64-conda-linux-gnu/14.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: ../../output/lib64/grass85/lib/libgrass_gis.so.8.5.0dev: undefined reference to `libiconv'
/home/hcho/usr/local/opt/miniconda/envs/grass/bin/../lib/gcc/x86_64-conda-linux-gnu/14.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: ../../output/lib64/grass85/lib/libgrass_gis.so.8.5.0dev: undefined reference to `omp_set_num_threads@OMP_1.0'
/home/hcho/usr/local/opt/miniconda/envs/grass/bin/../lib/gcc/x86_64-conda-linux-gnu/14.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: ../../output/lib64/grass85/lib/libgrass_gis.so.8.5.0dev: undefined reference to `compress2'
/home/hcho/usr/local/opt/miniconda/envs/grass/bin/../lib/gcc/x86_64-conda-linux-gnu/14.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: ../../output/lib64/grass85/lib/libgrass_gis.so.8.5.0dev: undefined reference to `compressBound@ZLIB_1.2.0'
/home/hcho/usr/local/opt/miniconda/envs/grass/bin/../lib/gcc/x86_64-conda-linux-gnu/14.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: ../../output/lib64/grass85/lib/libgrass_gis.so.8.5.0dev: undefined reference to `libiconv_close'
/home/hcho/usr/local/opt/miniconda/envs/grass/bin/../lib/gcc/x86_64-conda-linux-gnu/14.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: ../../output/lib64/grass85/lib/libgrass_gis.so.8.5.0dev: undefined reference to `ZSTD_decompress'
/home/hcho/usr/local/opt/miniconda/envs/grass/bin/../lib/gcc/x86_64-conda-linux-gnu/14.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: ../../output/lib64/grass85/lib/libgrass_gis.so.8.5.0dev: undefined reference to `zError'
/home/hcho/usr/local/opt/miniconda/envs/grass/bin/../lib/gcc/x86_64-conda-linux-gnu/14.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: ../../output/lib64/grass85/lib/libgrass_gis.so.8.5.0dev: undefined reference to `ZSTD_isError'
/home/hcho/usr/local/opt/miniconda/envs/grass/bin/../lib/gcc/x86_64-conda-linux-gnu/14.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: ../../output/lib64/grass85/lib/libgrass_gis.so.8.5.0dev: undefined reference to `libiconv_open'
/home/hcho/usr/local/opt/miniconda/envs/grass/bin/../lib/gcc/x86_64-conda-linux-gnu/14.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: ../../output/lib64/grass85/lib/libgrass_gis.so.8.5.0dev: undefined reference to `ZSTD_compressBound'
/home/hcho/usr/local/opt/miniconda/envs/grass/bin/../lib/gcc/x86_64-conda-linux-gnu/14.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: ../../output/lib64/grass85/lib/libgrass_gis.so.8.5.0dev: undefined reference to `ZSTD_compress'
/home/hcho/usr/local/opt/miniconda/envs/grass/bin/../lib/gcc/x86_64-conda-linux-gnu/14.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: ../../output/lib64/grass85/lib/libgrass_gis.so.8.5.0dev: undefined reference to `ZSTD_getErrorName'
/home/hcho/usr/local/opt/miniconda/envs/grass/bin/../lib/gcc/x86_64-conda-linux-gnu/14.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: ../../output/lib64/grass85/lib/libgrass_gis.so.8.5.0dev: undefined reference to `omp_get_num_procs@OMP_1.0'
collect2: error: ld returned 1 exit status
make[2]: *** [db/drivers/CMakeFiles/pg.dir/build.make:347: output/lib64/grass85/driver/db/pg] Error 1
make[1]: *** [CMakeFiles/Makefile2:8412: db/drivers/CMakeFiles/pg.dir/all] Error 2 
make: *** [Makefile:146: all] Error 2

HuidaeCho · 2025-03-07T21:07:04Z

Yay! That was it! LD_LIBRARY_PATH=$CONDA_PREFIX/lib make Closing this PR.

nilason · 2025-03-08T08:23:36Z

Great that we could sort that out!

What PUBLIC linking is doing is passing any dependency of a (in this case) library to the target, leading to overlinking. Let us take an example d.grid, which only depends on internal GRASS libraries (gis, symb, gproj and display):

# using PRIVATE linking:
otool -L /tmp/lib/grass85/bin/d.grid  
/tmp/lib/grass85/bin/d.grid:
	@rpath/libgrass_symb.8.dylib (compatibility version 8.0.0, current version 8.5.0)
	@rpath/libgrass_display.8.dylib (compatibility version 8.0.0, current version 8.5.0)
	@rpath/libgrass_gproj.8.dylib (compatibility version 8.0.0, current version 8.5.0)
	@rpath/libgrass_gis.8.dylib (compatibility version 8.0.0, current version 8.5.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1351.0.0)


# using PUBLIC linking:
/tmp/lib/grass85/bin/d.grid:
	@rpath/libgrass_symb.8.dylib (compatibility version 8.0.0, current version 8.5.0)
	@rpath/libgrass_display.8.dylib (compatibility version 8.0.0, current version 8.5.0)
	@rpath/libgrass_raster.8.dylib (compatibility version 8.0.0, current version 8.5.0)
	@rpath/libgrass_gproj.8.dylib (compatibility version 8.0.0, current version 8.5.0)
	/opt/local/lib/libgdal.36.dylib (compatibility version 36.0.0, current version 36.3.10)
	/opt/local/lib/proj9/lib/libproj.25.dylib (compatibility version 25.0.0, current version 25.9.5)
	@rpath/libgrass_parson.8.dylib (compatibility version 8.0.0, current version 8.5.0)
	@rpath/libgrass_htmldriver.8.dylib (compatibility version 8.0.0, current version 8.5.0)
	@rpath/libgrass_pngdriver.8.dylib (compatibility version 8.0.0, current version 8.5.0)
	/opt/local/lib/libpng16.16.dylib (compatibility version 62.0.0, current version 62.0.0)
	@rpath/libgrass_psdriver.8.dylib (compatibility version 8.0.0, current version 8.5.0)
	@rpath/libgrass_cairodriver.8.dylib (compatibility version 8.0.0, current version 8.5.0)
	@rpath/libgrass_driver.8.dylib (compatibility version 8.0.0, current version 8.5.0)
	@rpath/libgrass_gis.8.dylib (compatibility version 8.0.0, current version 8.5.0)
	@rpath/libgrass_datetime.8.dylib (compatibility version 8.0.0, current version 8.5.0)
	/opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.3.1)
	/opt/local/lib/libzstd.1.dylib (compatibility version 1.0.0, current version 1.5.6)
	/opt/local/lib/postgresql17/libpq.5.dylib (compatibility version 5.0.0, current version 5.17.0)
	/opt/local/lib/libiconv.2.dylib (compatibility version 9.0.0, current version 9.1.0)
	/opt/local/lib/libfreetype.6.dylib (compatibility version 27.0.0, current version 27.2.0)
	/opt/local/lib/libcairo.2.dylib (compatibility version 11709.0.0, current version 11709.0.0)
	/opt/local/lib/libfontconfig.1.dylib (compatibility version 14.0.0, current version 14.1.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1351.0.0)

Looking at the above makes it clear, that quite obviously, the PRIVATE linking is the preferred way in this/our case.

CMake: Support CMake build within an isolated Conda environment for L…

b8eb247

…inux

HuidaeCho self-assigned this Mar 6, 2025

github-actions bot added the CMake label Mar 6, 2025

Move Iconv_IS_BUILT_IN=FALSE to command line

83d77ba

HuidaeCho closed this Mar 7, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CMake: Support CMake build within an isolated Conda environment for Linux #5322

CMake: Support CMake build within an isolated Conda environment for Linux #5322

HuidaeCho commented Mar 6, 2025 •

edited

Loading

echoix commented Mar 6, 2025

HuidaeCho commented Mar 6, 2025 •

edited

Loading

wenzeslaus commented Mar 6, 2025

HuidaeCho commented Mar 7, 2025

nilason commented Mar 7, 2025

wenzeslaus commented Mar 7, 2025

HuidaeCho commented Mar 7, 2025 •

edited

Loading

HuidaeCho commented Mar 7, 2025

HuidaeCho commented Mar 7, 2025

HuidaeCho commented Mar 7, 2025

nilason commented Mar 8, 2025

CMake: Support CMake build within an isolated Conda environment for Linux #5322

CMake: Support CMake build within an isolated Conda environment for Linux #5322

Conversation

HuidaeCho commented Mar 6, 2025 • edited Loading

echoix commented Mar 6, 2025

HuidaeCho commented Mar 6, 2025 • edited Loading

wenzeslaus commented Mar 6, 2025

HuidaeCho commented Mar 7, 2025

nilason commented Mar 7, 2025

wenzeslaus commented Mar 7, 2025

HuidaeCho commented Mar 7, 2025 • edited Loading

HuidaeCho commented Mar 7, 2025

HuidaeCho commented Mar 7, 2025

HuidaeCho commented Mar 7, 2025

nilason commented Mar 8, 2025

HuidaeCho commented Mar 6, 2025 •

edited

Loading

HuidaeCho commented Mar 6, 2025 •

edited

Loading

HuidaeCho commented Mar 7, 2025 •

edited

Loading