Add cmake build system

[trws]

This is an experiment to see if we can use a fake configure script to get the benefits of both cmake, and all the CI tooling flux projects rely on.

[trws]

Hey @grondo, I'm playing with this a bit as a compromise alternative. There are still some rough edges on it, but what would you think of an approach like this that can still be used as though it had an autoconf-generated configure script? Non-trivial upside is it's already almost fully compatible with the CI scripts without modifying them.

[grondo]

I was just going to try this out and I can't figure out how to get it to configure against a side installed flux-core. The autotools build configures flux-sched using the flux-core corresponding to the first flux found in PATH. Therefore you can do

$ /path/to/flux-core/bin/flux ./configure ...

To configure against a flux-core with prefix /path/to-flux-core/. How can I do the same with cmake?

I tried PKG_CONFIG_PATH=/tmp/grondo/lib/pkgconfig/ cmake ., but get the following error that I can't make heads nor talis of:

$ PKG_CONFIG_PATH=/tmp/grondo/lib/pkgconfig/ cmake .
-- Checking for module 'flux-taskmap'
--   Found flux-taskmap, version 0.52.0-1-ga35e212a0
CMake Error at cmake/FindFluxCore.cmake:19 (add_library):
  add_library cannot create ALIAS target "flux::core" because target
  "PkgConfig::FLUX_CORE" is IMPORTED.
Call Stack (most recent call first):
  CMakeLists.txt:48 (find_package)


CMake Error at cmake/FindFluxCore.cmake:20 (add_library):
  add_library cannot create ALIAS target "flux::hostlist" because target
  "PkgConfig::FLUX_HOSTLIST" is IMPORTED.
Call Stack (most recent call first):
  CMakeLists.txt:48 (find_package)

[grondo]

note that if we go this route another TODO will be to update the RPM spec file in the TOSS build farm.

Also, not sure we can drop the bionic CI builds since we haven't dropped support for that distro yet. Was there something that doesn't work there?

[trws]

That error is because your cmake is too old, I bumped it down to 3.10 in hopes it would work with the default cmake on focal and bionic but they are just too old, it needs 3.12. The env var actually worked, this just means we can’t name the target afterwards.

…

--- Sent from Workspace ONE Boxer<https://whatisworkspaceone.com/boxer> On July 13, 2023 at 8:05:40 AM PDT, Mark Grondona ***@***.***> wrote: note that if we go this route another TODO will be to update the RPM spec file in the TOSS build farm. Also, not sure we can drop the bionic CI builds since we haven't dropped support for that distro yet. Was there something that doesn't work there? — Reply to this email directly, view it on GitHub<https://urldefense.us/v3/__https://github.com/flux-framework/flux-sched/pull/1049*issuecomment-1634411220__;Iw!!G2kpM7uM-TzIFchu!3GqohwFiYCIHNLVeRAx-7J8ALc6iOdGAN86vwQQOKbI439mdijc5cfTvqcuMrF4PyCztlWB_ml2hT5ILUIZSlKrimqg$>, or unsubscribe<https://urldefense.us/v3/__https://github.com/notifications/unsubscribe-auth/AAFBFNJTWEMPFJ3M3DLSHGLXQAFD7ANCNFSM6AAAAAA2H5DSH4__;!!G2kpM7uM-TzIFchu!3GqohwFiYCIHNLVeRAx-7J8ALc6iOdGAN86vwQQOKbI439mdijc5cfTvqcuMrF4PyCztlWB_ml2hT5ILUIZS6xNcLS8$>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[trws]

It’s the default cmake, I have a blurb from other containers to pull a more recent binary cmake directly that we can use, but I wanted to make sure I could get all the features working. The other things still hanging are to add a coverage target and test the ctest result reporter. If that works it should integrate directly with the GitHub test panel so that might be somewhat nice.

…

--- Sent from Workspace ONE Boxer<https://whatisworkspaceone.com/boxer> On July 13, 2023 at 8:05:40 AM PDT, Mark Grondona ***@***.***> wrote: note that if we go this route another TODO will be to update the RPM spec file in the TOSS build farm. Also, not sure we can drop the bionic CI builds since we haven't dropped support for that distro yet. Was there something that doesn't work there? — Reply to this email directly, view it on GitHub<https://urldefense.us/v3/__https://github.com/flux-framework/flux-sched/pull/1049*issuecomment-1634411220__;Iw!!G2kpM7uM-TzIFchu!3GqohwFiYCIHNLVeRAx-7J8ALc6iOdGAN86vwQQOKbI439mdijc5cfTvqcuMrF4PyCztlWB_ml2hT5ILUIZSlKrimqg$>, or unsubscribe<https://urldefense.us/v3/__https://github.com/notifications/unsubscribe-auth/AAFBFNJTWEMPFJ3M3DLSHGLXQAFD7ANCNFSM6AAAAAA2H5DSH4__;!!G2kpM7uM-TzIFchu!3GqohwFiYCIHNLVeRAx-7J8ALc6iOdGAN86vwQQOKbI439mdijc5cfTvqcuMrF4PyCztlWB_ml2hT5ILUIZS6xNcLS8$>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[trws]

I was just going to try this out and I can't figure out how to get it to configure against a side installed flux-core. The autotools build configures flux-sched using the flux-core corresponding to the first flux found in PATH. Therefore you can do

$ /path/to/flux-core/bin/flux ./configure ...

Is this supposed to work? I just tried it, and sched's autoconf doesn't configure under those conditions, it seems to look for flux-core only in PKG_CONFIG_PATH and the selected PREFIX. I just pushed an update that matches the ax_flux_core macro in the cmake search, it also now checks for the correct cmake version. Given anything 3.12 or higher basic cases like that or with compiler or prefix flags should work. Coverage, sanitizers, and "--with" options would still be TODO.

[grondo]

Yes, it works for me (I tend to use it most of the time to configure against a side-installed flux-core). E.g. for flux-core installed with prefix=/tmp/grondo:

$ /tmp/grondo/bin/flux ./configure
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
[snip]
checking for FLUX_CORE... yes
checking for flux... /tmp/grondo/bin/flux
checking for FLUX_IDSET... yes
checking for FLUX_SCHEDUTIL... yes
checking for FLUX_OPTPARSE... yes
checking for FLUX_HOSTLIST... yes
[snip]
  flux-sched version 0.28.0
  Prefix...........: /tmp/grondo
  Debug Build......: 
  C Compiler.......: gcc
  C++ Compiler.....: g++
  CFLAGS...........: -g -O2
  CPPFLAGS.......... 
  CXXFLAGS.......... -g -O2
  FLUX.............: /tmp/grondo/bin/flux
  FLUX_VERSION.....: 0.52.0-1-ga35e212a0
  FLUX_CORE_CFLAGS.: -I/tmp/grondo/include
  FLUX_CORE_LIBS...: -L/tmp/grondo/lib -lflux-core
  LIBFLUX_VERSION..: 0.52.0-1-ga35e212a0
  FLUX_PREFIX......: /tmp/grondo
  LDFLAGS..........: 
  LIBS.............: 
  Linker...........: /usr/bin/ld -m elf_x86_64

[grondo]

On TOSS4 systems, PYTHON=/usr/bin/python3 ./configure is required to successfully configure flux-core and flux-sched (the TCE python doesn't have the correct python deps). How do we do this with cmake? It seems to ignore PYTHON when run as cmake . or ./configure.

[trws]

I'll add support for the env var, the cmake way is to do one of these:

-D Python_EXECUTABLE=<path to exe>
-D Python_ROOT_DIR=<path to install prefix>

[grondo]

Thanks, that worked!

[trws]

Clearly working, but when I try exactly the same it doesn't work... somehow I'm missing something. Looking through ax_flux_core I only see it using pkg-config to look it up, for the "checking for FLUX_CORE" then using the result to look for the rest, but that can't be all if that works for you. Do you happen to remember where the autoconf code for that is? It would help me debug my environment.

…

On 14 Jul 2023, at 13:31, Mark Grondona wrote: Yes, it works for me (I tend to use it most of the time to configure against a side-installed flux-core). E.g. for flux-core installed with `prefix=/tmp/grondo`: ``` $ /tmp/grondo/bin/flux ./configure checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /bin/mkdir -p [snip] checking for FLUX_CORE... yes checking for flux... /tmp/grondo/bin/flux checking for FLUX_IDSET... yes checking for FLUX_SCHEDUTIL... yes checking for FLUX_OPTPARSE... yes checking for FLUX_HOSTLIST... yes [snip] flux-sched version 0.28.0 Prefix...........: /tmp/grondo Debug Build......: C Compiler.......: gcc C++ Compiler.....: g++ CFLAGS...........: -g -O2 CPPFLAGS.......... CXXFLAGS.......... -g -O2 FLUX.............: /tmp/grondo/bin/flux FLUX_VERSION.....: 0.52.0-1-ga35e212a0 FLUX_CORE_CFLAGS.: -I/tmp/grondo/include FLUX_CORE_LIBS...: -L/tmp/grondo/lib -lflux-core LIBFLUX_VERSION..: 0.52.0-1-ga35e212a0 FLUX_PREFIX......: /tmp/grondo LDFLAGS..........: LIBS.............: Linker...........: /usr/bin/ld -m elf_x86_64 ``` -- Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/flux-framework/flux-sched/pull/1049*issuecomment-1636403907__;Iw!!G2kpM7uM-TzIFchu!1JmiMEin0c0-8VTeVU-qpY3KxTsmdiX3SlWUy9_It75IXiB35R2Ngsi_FKpvEluq1deYdoOpo7u3daIrRrJGNgKrrxg$ You are receiving this because you authored the thread. Message ID: ***@***.***>

[grondo]

Ok, I had forgotten how this works. The main support was introduced in

b4bd5f0

which sets the default --prefix to the first flux(1) found in PATH :

commit b4bd5f09a2743600bd96d44f763d01e4d8ced501
Author: Mark A. Grondona <[email protected]>
Date:   Fri Feb 28 10:07:40 2020 -0800

    build: set --prefix to same prefix as flux(1) by default
    
    Use AC_PREFIX_PROGRAM([flux]) to have the default prefix of flux-sched
    set to the same prefix as the flux executable in PATH. For example,
    for a system with flux installed as /usr/bin/flux, prefix will default
    to /usr instead of /usr/local for flux-sched, unless
    
     ./configure --prefix=/usr/local
    
    is used.
    
    This also allows flux-sched to be configured against side-installed
    flux with simply:
    
     /path/to/side/install/bin/flux env ./configure

Then the stuff in ax_flux_core.m4 which prepends to PKG_CONFIG_PATH based on prefix takes over.

So, if you are using --prefix=/somewhere/else then that overrides the default prefix and the convenience hack to configure against side-installed flux-core is overridden.

[trws]

Thanks @grondo, that helps a lot! I think I was using a wrapper script that set prefix and expecting it to still work because of the flux in path. I can replicate that in cmake pretty easily. Also tempted to add something to the cmake to find flux in path, then add $(dirname $(dirname $FLUX))/lib/pkgconfig to the PKG_CONFIG_PATH, that way even if they're in a separate prefix we get the "use the flux-core in path" behavior.

[grondo]

Sounds great to me!

[trws]

Ok, finding flux-core by finding flux and matching prefix if it isn't explicitly set implemented. This allows flux-core to be found by path even when a different prefix is selected if the prefix for flux-core isn't explicitly specified. The order of priority is: FLUX_CORE_PREFIX > PKG_CONFIG_PATH > flux in path > default locations

[trws]

Ok, this is now all there. I want to add a better C++ test system (Catch2 probably unless there are objections), include-what-you-use for CI, and a few other things, but this is a coherent build system and set of fixes. Given where it is, and the issues with boost I keep running into on some platforms, I would like to remove the autotools setup at some point, but not now. If we have a successful release using the cmake wrappers and everything, that's when I'd like to do it if that works for everyone.

[milroy]

I'm in the process of reviewing the PR, but I ran into a small issue in my bookworm Docker build:

milroy1@docker-desktop:/usr/src$ git config --global --add safe.directory /usr/src
milroy1@docker-desktop:/usr/src$ ./autogen.sh 
milroy1@docker-desktop:/usr/src$ ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var
-- VER 37a272b7
CMake Error at CMakeLists.txt:11 (project):
  VERSION "37a272b7" format invalid.

It looks like this is caused by the fork not containing the upstream tags. To get the project to build and test successfully, I needed to set the version to something like 0.28.0 in a new file (flux-sched.ver).

You might want to allow a user-specified environment variable that overrides the version returned by git describe --always ....

[milroy]

This is a huge improvement to the build system in terms of speed, and I like the maintainability improvements, too. Thanks again for taking the time to walk me through the contributions last week!

I identified what I think is a minor problem with builds on Fluxion forks without tags, as well as a few small issues and nits related to commit messages. In general, the commit messages need to be wrapped, and in the standard Flux Framework Problem: ... format.

From a high-level, it's somewhat difficult to follow the PR because files are updated and lines changes several times along the course of the commits. This PR should be restructured so that the refactor commits are fixups that can be squashed in a rebase. I think that will condense the changes and help make them more understandable.

[milroy]

Nit: wrap the commit body at 72 or 80 characters (can't remember which is preferred for git).

[trws]

These comments are ending up on lines of code rather than on commits, not sure if there's anything to do about that but it would be good to find a way to make the link to the commit more obvious.

[grondo]

It has always been annoying that there's no way to comment on a commit message itself. However, going through the commits in the "Commits" tab will only show the comments pertaining to that commit (though it still appears attached to a line of code)

[trws]

I keep hoping we can do something better for that. Tools like stacked git, jujutsu, and even facebook's sapling with their experimental example stacked PR UI for github look promising, but nothing first party. 😢

Short of going as far as reviewing with a third-party UI or switching to gerritt it feels like we're a bit stuck.

[trws]

Here's the UI I was referring to from facebook: https://reviewstack.dev/ it also works with ghstack, if only

[milroy]

High-level comment on this commit: it's not clear why the bionic, el7, and fedora33 dockerfiles are deleted in this commit.

[trws]

That's an artifact of this not having been rebased on top of the build-system update PR yet. This should evaporate after the rebase I just did. Will post after some cleanup.

[grondo]

One random other suggestion. We typically autogenerate release notes from PR titles (since they are used by default as the merge commit message), so before a merge the PR should possibly be titled something other than "Cmake revert checks" (which admittedly I'm not sure I understand what that means). Maybe just "switch build to cmake"

[trws]

I’ll fix that. The current title is because this started as a fork of the previous cmake PR where I reverted all the changes to the *checks.sh scripts to try the configure script approach.

…

--- Sent from Workspace ONE Boxer<https://whatisworkspaceone.com/boxer> On September 6, 2023 at 2:47:42 PM PDT, Mark Grondona ***@***.***> wrote: One random other suggestion. We typically autogenerate release notes from PR titles (since they are used by default as the merge commit message), so before a merge the PR should possibly be titled something other than "Cmake revert checks" (which admittedly I'm not sure I understand what that means). Maybe just "switch build to cmake" — Reply to this email directly, view it on GitHub<https://urldefense.us/v3/__https://github.com/flux-framework/flux-sched/pull/1049*issuecomment-1709173256__;Iw!!G2kpM7uM-TzIFchu!3LCh7SzNM3cu4cXQ3KzGIGGzVJcjdu-_jYnzwzu75HeCg4soXfcSkG4iPXEdjhIoqaxgZEcp5h9NarQ1FuZWJYTsJ_w$>, or unsubscribe<https://urldefense.us/v3/__https://github.com/notifications/unsubscribe-auth/AAFBFNJF6SBQO7UKDLT73QDXZDVPTANCNFSM6AAAAAA2H5DSH4__;!!G2kpM7uM-TzIFchu!3LCh7SzNM3cu4cXQ3KzGIGGzVJcjdu-_jYnzwzu75HeCg4soXfcSkG4iPXEdjhIoqaxgZEcp5h9NarQ1FuZWYM8jUy0$>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[trws]

Ok, that was one heck of a rebase... This may be the first time I've actually saved a backup of the origin branch just in case something got munged up in all the rewriting, but there are a lot less commits now and should be less files actually changed in this PR.

[milroy]

I started taking a look at the updates. The monster rebase definitely cleaned up the commits!

I ran into a problem with the update to CMakeLists.txt:

$ ./configure
-- VER dc8f6c88
CMake Error at CMakeLists.txt:14 (string):
  string sub-command REGEX, mode REPLACE needs at least 6 arguments total to
  command.
CMake Warning at CMakeLists.txt:15 (project):
  VERSION keyword not followed by a value or was followed by a value that
  expanded to nothing.
-- Building flux-sched version dc8f6c88
-- Boost version: 1.74.0
-- Configuring incomplete, errors occurred!
See also "/usr/src/CMakeFiles/CMakeOutput.log".

Which you don't hit unless all three are true: undefined FLUX_SCHED_VER environment variable, no VER_FILE, and no tags present in the fork. Quoting the final REGEX argument fixes the issue:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 07b32932..6c299b01 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -11,7 +11,7 @@ else()
   execute_process(COMMAND sh -c [=[git describe --always | awk '/.*/ {sub(/^v/, ""); printf "%s",$1; exit}']=] OUTPUT_VARIABLE FLUX_SCHED_VER)
 endif()
 message(STATUS "VER ${FLUX_SCHED_VER}")
-string(REGEX REPLACE "-.*$" "" FLUX_SCHED_VER_NOGIT ${FLUX_SCHED_VER})
+string(REGEX REPLACE "-.*$" "" FLUX_SCHED_VER_NOGIT "${FLUX_SCHED_VER}")
 project(flux-sched VERSION ${FLUX_SCHED_VER_NOGIT} LANGUAGES CXX C)
 message(STATUS "Building flux-sched version ${FLUX_SCHED_VER}")

[trws]

Thanks for that @milroy, patched up and push incoming. I'll set MWP after that. If you get through a review and you're happy, approving should send it through.

[milroy]

The first commit "ensure superceded PR workflows are cancelled" needs to wrap at 72 characters. Beyond that this PR is great and ready to merge. I dropped the MWP label and am approving this PR.

@trws, you can re-add MWP when you fix first commit message.

[trws]

wrapped and pushed, resetting MWP

[codecov]

Codecov Report

Merging #1049 (f9efce9) into master (c619fbe) will decrease coverage by 1.9%.
The diff coverage is n/a.

❗ Current head f9efce9 differs from pull request most recent head 5ecbd64. Consider uploading reports for the commit 5ecbd64 to get more accurate results

@@           Coverage Diff            @@
##           master   #1049     +/-   ##
========================================
- Coverage    73.4%   71.6%   -1.9%     
========================================
  Files          98      89      -9     
  Lines       11889   11569    -320     
========================================
- Hits         8737    8289    -448     
- Misses       3152    3280    +128     

see 19 files with indirect coverage changes