Skip to content

Conference call notes 20241023

Jump to bottom
Bart Oldeman edited this page Oct 23, 2024 · 10 revisions

(back to Conference calls)

Notes on the 257th EasyBuild conference call, Wednesday 23 Oct 2024 (15:00 UTC)

Attendees

List of attendees (14):

  • Alex Domingo (Vrije Universiteit Brussel, Belgium)
  • Em Dragowsky (Case Western Reserve University, Ohio, US)
  • Davide Grassano (CECAM, Switzerland)
  • Jasper Grimm (University of York, UK)
  • Georgios Kafanas (University of Luxembourg)
  • Kurt Lust (UAntwerpen, Belgium + LUMI User Support Team)
  • Mikael Öhman (Chalmers University of Technology, Sweden)
  • Bart Oldeman (McGill University, Digital Research Alliance of Canada)
  • Jure Pečar (EMBL)
  • Jan Reuter (JSC, Germany)
  • Jörg Saßmannshausen (Imperial College London, UK)
  • Åke Sandgren (Umeå University, Sweden)
  • Davide Vanzo (Azure)
  • Cintia Willemyns (Vrije Universiteit Brussel, Belgium)

Agenda

  • overview of recent developments
  • outlook to EasyBuild 5.0 release
  • 2024b update of common toolchains
  • Q&A

Recent developments

  • latest EasyBuild release: 4.9.4 (22 Sept 2024)
  • next EasyBuild release:
    • very likely EasyBuild v5.0.0 🔥
    • additional EasyBuild 4.9.x versions could still be done via 4.9.x branches, but none are planned currently
  • EasyBuild v5.0.0
    • project board: https://github.com/orgs/easybuilders/projects/18/views/2
    • detailed notes on latest developments in https://hackmd.io/tZtz6vMmRfGL6N9Rf__OEA
    • to test development version of EasyBuild 5.0: 
      # set up Python virtual environment, and jump into it
python3 -m venv eb5
source eb5/bin/activate

# install EasyBuild 5.0 development version into it
pip install https://github.com/easybuilders/easybuild-framework/archive/5.0.x.tar.gz
pip install https://github.com/easybuilders/easybuild-easyblocks/archive/5.0.x.tar.gz
pip install https://github.com/easybuilders/easybuild-easyconfigs/archive/5.0.x.tar.gz

# go!
eb --version
  • collapsing of 5.0.x branches in develop post-poned
    • impact on people using EasyBuild 4.x considered too big, so we want to limit the time window in which problems will arise as much as possible
    • impact:
      • --from-pr will be broken with EasyBuild 4.x due to major version mismatch
        • --from-commit can be used as a workaround
      • open easyconfig PRs should be synced with develop branch before they get merged
        • due to changes in easyconfigs test suite implemented in 5.0.x
        • for example use_pip = True should no longer be used
        • same goes for framework & easyblocks

Merged PRs

  • docs (merged PRs)

    • ...

  • framework (merged PRs)

    • bug fixes
      • [5.0.x] Use more robust mechanism for unsetting environment variables (PR #4685)
    • enhancements
      • [5.0.x] disable strict RPATH sanity check by default, allow re-enabling it via --strict-rpath-sanity-check configuration option (PR #4475)
      • [5.0.x] include path to cmd.sh script in output generated by run_shell_cmd when a command fails + use colors: red for ERROR line, yellow for path to output files + cmd.sh script (PR #4666)
      • [develop] add support for %(rpath_enabled)s template value (PR #4670)
      • [5.0.x] don't raise error when required extensions are not found when installing extensions in parallel (PR #4671)
    • changes
      • ...
    • code cleanup
      • ...
    • tests
      • ...

  • easyblocks (merged PRs)

    • bug fixes
      • [develop] update Kokkos CPU mapping for LAMMPS >= 2Aug2023 to use icx on Intel Icelake systems (PR #3321)
      • [5.0.x] Make MesonNinja respect the toolchainopts with buildtype as well as --debug and --optimization flags (PR #3454)
      • [5.0.x] take into account that 'pic' toolchain option may not be defined in CMakeMake easyblock (PR #3470)
      • [5.0.x] add missing return for customised install_extension_async methods in Rserve and Rmpi easyblocks (PR #3474)
      • [develop] disable CPU-specific optimizations for generic builds (by using a custom processor_arch) in LAMMPS easyblock (PR #3484)
      • [develop] also add path to libtorch.so & co to $LIBRARY_PATH in generated module file for PyTorch (PR #3488)
      • [5.0.x] use *DESCRIPTION rather than */DESCRIPTION in RPackage.requires_deps (PR #3490)
      • [develop] use DYNAMIC_ARCH=1 when building OpenBLAS with --optarch=GENERIC (PR #3492)
    • enhancements
      • [5.0.x] enhance CMakeMake easyblock to check whether correct Python installation was picked up by CMake (PR #3399)
      • [develop] let CMakeMake easyblock also set Python_EXECUTABLE option, as well as Python3_EXECUTABLE and Python2_EXECUTABLE derivatives (when appropriate) (PR #3463)
      • [develop] enhance cuDNN easyblock to verify that EULA is accepted before installing it ((PR #3473)
      • [develop] Enhance AOCC EasyBlock to correctly pass GCC toolchain and compiler driver (PR #3480)
      • [develop] enhance Cargo easyblock to printmessage for generating Cargo.lock if it's missing (PR #3491)
    • updates
      • ...
    • changes
      • ...
    • new easyblocks
      • ...
    • code cleanup
      • [5.0.x] refactor search for python_cmd (PR #3475)

  • easyconfigs (merged PRs)

    • ~65 easyconfig PRs were merged since last conf call
    • bug fixes/reports
      • [develop] fix installation of NextPolish v1.4.1 when RPATH linking is used (+ move to GCC toolchain) (PR #21588)
      • [develop] stop using --buildtype=release in easyconfigs using MesonNinja easyblock (PR #21619)
      • [develop] demote hatchling to build-only dependency in easyconfigs for anndata 0.10.5.post1, Cassiopeia 2.0.0, phonemizer 3.2.1, scanpy 1.9.8, wandb 0.16.1 (PR #21657)
      • [5.0.x] fix failing RPATH sanity check for recent dorado easyconfigs using foss/2023a toolchain (PR #21673)
      • [develop] Fix alignment bug in patchelf 0.18.0 by adding patch (PR #21674)
      • [develop] Remove redundant sanity check from jedi easyconfigs (PR #21686)
      • [develop] fix the zero division bug of networkx-3.1 (PR #21702)
    • enhancements
      • [develop] Add trimmomatic wrapper script for recent Trimmomatic 0.39 easyconfigs (PR #21704)
    • (noteworthy) new software
      • ...
    • noteworthy software updates
      • ...
    • cleanup
      • ...
    • changes
      • [develop] remove explicit use of ld.gold in recent PLUMED easyconfigs (PR #21613)
      • [develop] move jedi package from IPython 8.14.0 to own easyconfig (PR #21650)
    • EasyBuild 5.0
      • ...

Open (active) PRs

  • docs (open PRs + issues)

    • ...

  • framework (open PRs + issues)

    • bug fixes

      • Problem using $CPATH in modulefiles overwriting system paths (issue #3331)
        • [5.0.x] add --search-path-cpp-headers configuration option to control how EasyBuild sets paths to headers at build time (PR #4645)
        • [5.0.x] refactor generation of required environment variables in module files (PR #4653)
          • test was added to verify that making ModuleLoadEnvironment a singleton causes trouble
          • needs more love, additional test is broken
          • easyblocks will need to be updated accordingly to not use make_module_req_guess anymore
        • [5.0.x] add module-search-path-headers configuration option to control how modules set search paths to header files (PR #4655)
      • [develop] add support for injecting checksums for cargo crates (PR #4661) - [5.0.x] Reverse order for parsing files in $XDG_CONFIG_DIRS (PR #4630)
      • [5.0.x] improve portability of reproducible tarballs by replacing external tar command with tarfile module (PR #4660)
        • to fix issue #4657: reproducible tarballs have unexpected checksums on mac
        • tarfile module in Python allows creating tarballs in a reproducible way across systems (Linux + macOS)
          • but there's a breaking change in Python 3.9, so we get different tarballs across different Python versions :man-facepalming:
          • two options to deal with this:
            • only support creating reproducible tarballs with Python 3.9+
              • people using Python < 3.9 would have to use --ignore-checksums
              • EasyBuild 5.0 could print a warning when using Python < 3.9 that checksums can not be verified?
              • this seems like the best way forward?
              • we should probably consider deprecating support for Python < 3.9 in EasyBuild 5.0
            • always include two checksums for source tarballs created via git_config: one for Python < 3.9, one for Python >= 3.9
          • binary patching generated tarballs with older Python versions would be very difficult
      • get_os_name should be updated to include Rocky and Alma (PR #4663)
      • [develop] fix dry-run output when using multi_deps (PR #4678)
      • [develop] Make fetching source tarball by git commit more robust (PR #4680)

    • enhancements

      • [develop] Copy build log and artifacts to a permanent location after failures (WIP PR #4601)
      • [develop] allow extensions to add text to the module file (PR #4652)
      • [develop] Add resolve_template function to easyconfig instances (PR #4677)
      • [develop] Allow templates in custom_paths & custom_commands sanity-check arguments (PR #4679)
      • [develop] Add function to modify path-like environment variable in a context (PR #4681)
      • [5.0.x] Update findPythonDeps (PR #4682)
      • [5.0.x] Allow use of custom delimiter for paths in module generator (PR #4687)
      • [5.0.x] Add global option to set default toolchain option of "debug" (PR #4688)

    • changes

      • [5.0.x] Deprecate use of parallel easyconfig parameter and fix updating the template value (PR #4580)
      • [5.0.x] Rename SOURCE_STEP to EXTRACT_STEP (PR #4629)
      • [5.0.x] Deprecate post_install_step, rename it to post_processing_step (issue #4656)
      • [5.0.x] proposed changest to add a timeout to run_shell_cmd (PR #4665)
      • [develop] Added possibility to call amend/try-amend multiple times from easystack (PR #4667)
      • [5.0.x] Let jobs retweak easyconfigs themselves (draft PR #4669)
      • [5.0.x] mark support for installing extensions in parallel as being mature, since it's no longer experimental (PR #4672)
      • [5.0.x] mark easystack support as being mature, since it's no longer experimental (PR #4673)
      • [5.0.x] simplify code for determining the PYTHONPATH module entries (PR #4686)
      • [5.0.x] simplify loop in collect_exts_file_info (PR #4689)

  • easyblocks (open PRs + issues)

    • bug fixes
      • [5.0.x] run pip check only once for PythonBundle (PR #3432)
      • [develop] Avoid failure in Python sanity check when PIP_REQUIRE_VIRTUALENV is set (PR #3460)
      • [develop] take into account that f90wrap_version may be None in MetalWalls easyblock (PR #3468)
      • [develop] fix setting of CMAKE_OPTIONS for SuiteSparse versions between 5.1.2 and 6.0.0 (PR #3471)
    • enhancements
      • [develop] Set CMake hints for path to Python installation if it's direct or indirect dependency (when using CMake >= 3.12) (PR #3282)
      • [develop -> 5.0.x] enhance LLVM easyblock for compilation of clang/flang + other llvm-projects (PR #3373)
      • [develop] ignore user python packages by using -s option when running Python code via "python -c" (PR #3440)
      • [develop] enhance generic Bundle EasyBlock to transfer module requirements of components (PR #3472)
      • [develop] Enhance Cargo easyblock to better support crates from git repositories (PR #3476)
      • [develop] fix enhance_sanity_check for PythonPackage (PR #3477)
      • [5.0.x] Fix handling of maximum python version in pick_python_cmd (PR #3478)
      • [develop] avoid name clash in cargo crates downloaded from git repositories (PR #3481)
      • [develop] Handle cargo workspaces in git repositories (PR #3482)
      • [develop] enhance cargo easyblock for sources from git repositories (PR #3483)
      • [develop] generate version from the source files when using software-commit (PR #3485)
      • [develop] flesh out logic to determine location to binutils commands into a standalone function det_binutils_bin_path in TensorFlow easyblock, and leverage it from jaxlib easyblock (PR #3486)
      • [develop] Modularized getting of CMake options (PR #3487)
    • updates
      • [develop] update custom easyblock for CP2K >= v2024 (PR #3433)
    • changes
      • ...
    • new
      • [develop] custom easyblock for DeepSpeed (PR #3450)

  • easyconfigs (open PRs + issues)

    • bug fixes/reports
      • [develop -> 5.0.x] switch to CargoPythonBundle easyblock for orjson 3.9.15 (PR #21392)
        • requires being able to produce reproducible source tarballs from Git repository (WIP in 5.0.x branch of EasyBuild framework)
      • [5.0.x] promote pybind11 to runtime dependency for CPPE (to fix pip check failure) (PR #21564)
      • [develop] add patch to SciPy-bundle 2024.05 that fixes test failure on aarch64 (PR #21559)
      • [develop] AlphaFold v2.3.2 fix: move run_alphafold_test.py from sanity_check step to testing step ([PR #21678] (https://github.com/easybuilders/easybuild-easyconfigs/pull/21678))
      • [develop] add patch for SciPy-bundle 2023.07 + 2023.11 to fix vectorization bug in scipy 1.11.1 (PR #21693)
      • [develop] Switch to using new spaces as delimiters for TCLLIBPATH (PR #21724)
    • enhancements
      • [develop] ignore user-installed packages when running python in sanitycheck (PR #21362)
      • [develop] also install w90spn2spn.x + w90chk2chk.x for Wannier90 3.1.0 (PR #21383)
    • (noteworthy) new software
      • ...
    • software updates
    • changes
      • [5.0.x] Replace parallel by maxparallel (PR #19375)
      • [develop] GTDB-Tk v2.4.0 should use skani and not FastANI (PR #21518)
        • (Åke) make sure that CPU boost is working on your systems if people are using this...

2024b common toolchains

  • GCC 14.2 as a base (see easyconfigs PR #21114)
  • we should define candidate toolchains using latest version of all components
    • are newer versions of Intel components compared to intel/2024a already available? No, not really, just a minor 2024.2.1.
  • should we consider replacing intel with Clang-based toolchain as 2nd common toolchain next to foss?
    • Fortran frontend in Clang is becoming mature enough?
    • Kurt: it's a good as many others, e.g. Cray.
    • Maybe not there yet, but almost, with the rename of flang-new to flang in LLVM 20 (2025).

Q&A / others

  • Jure: compute capability < 5.3 not supported in CUDA for CUTLASS (half precision), fails when 5.0 is provided, will open issue.
  • Jure: CUTLASS/SCALENE downloading things via git, will open issue. Either use git as builddependency or figure out how to build without git.
  • Jörg: easyconfigs issue #21715, why does fi_info only have issues with 2023b only? Should strace it.
  • Jörg: NVHPC: CUDA compute capability (only one). It's a default, can be overridden in nvcc command line etc.
Clone this wiki locally