Skip to content

Conference call notes 20210428

Jump to bottom
Kenneth Hoste edited this page May 11, 2021 · 7 revisions

(back to Conference calls)

Notes on the 171st EasyBuild conference call, Wednesday April 28th 2021 (15:00 UTC)

Attendees

Alphabetical list of attendees (15):

  • Kenneth Hoste (HPC-UGent, Belgium)
  • Åke Sandgren (Umeå University, Sweden)
  • Mikael Öhman (Chalmers University of Technology, Sweden)
  • Sam Moors (Vrije Universiteit Brussel, Belgium)
  • Fotis Georgatos (SDSC, Switzerland)
  • Kurt Lust (Univ. of Antwerp, Belgium + LUMI User Support Team)
  • Adam Huffman (Big Data Institute, Oxford, UK)
  • Alexander Grund (TU Dresden, Germany)
  • Bart Oldeman (McGill University, Compute Canada)
  • Francis Dang (Texas A&M Univ, US)
  • Davide Vanzo (Microsoft)
  • Jörg Saßmannshausen (NIHR Biomedical Research Centre, UK)
  • Robert Mijakovic (LuxProvide)
  • Sebastian Achilles (Jülich Supercomputing Centre, Germany)
  • Simon Branford (University of Birmingham, UK)

Agenda

  • update on recent developments
  • 2021a update of common toolchains
    • progress update
    • BLAS toolchain for foss/2021a: stick to OpenBLAS, switch to FlexiBLAS or BLIS?
    • approach to collapse of foss and fosscuda
  • Q&A

Recent developments

  • last release: EasyBuild v4.3.4 (April 9th)
  • next release: ETA mid May'21
  • recent changes
    • framework
      • bug fixes
        • (nothing major)
      • enhancements
        • update HierarchicalMNS for Intel OneAPI compilers (PR #3653)
        • add support for running commands asynchronously via run_cmd + complete_cmd functions (PR #3656)
          • required for supporting installation of extensions in parallel
        • replace log_error parameter of which() by on_error (PR #3661)
          • see also corresponding easyblocks PR #2406
      • changes
        • (nothing major)
    • easyblocks
      • bug fixes
        • (nothing major)
      • enhancements
        • add support for post install commands for Python extensions (PR #2381)
          • introduces new custom ext_postinstallcmds easyconfig parameter...
          • would be better to reuse existing postinstallcmds instead, see also framework PR #3663 + easyblocks PR #2404
      • changes
        • add option to disable pip connecting to PyPi (enable use of --no-index) (PR #2390)
          • avoid periodic pip version check + avoid that pip checks PyPI for (updated) dependencies
    • easyconfigs
      • over 150 (!) merged easyconfig PRs since last conf call
      • bug fixes
        • fix potential memory leak in OpenBLAS 0.3.12 (PR #12649)
        • avoid 0.0.0 install version for various Python apps (PR #12522)
        • fix numpy tests for recent SciPy-bundle easyconfig on POWER (PR #12481)
        • override host compiler check in CUDAcore (PR #12701)
        • fix source URLs for recent ELPA versions (PR #12700)
      • enhancements
        • (nothing major?)
      • new software
        • (nothing major?)
      • noteworthy software updates
        • GCC 11.1.0 (PR #12769)
        • OpenMPI 4.1.1
        • CUDAcore 11.2.2 + 11.3.0
        • lots of easyconfigs for GCCcore/10.3.0 (2021a common toolchains), incl. Autotools, CMake, Perl, Tcl, ...
      • noteworthy changes
        • remove PYPI_SOURCE source URL from easyconfigs using GCC(core) 9.3.0 or 10.2.0 as toolchain (PR #12453)
  • to merge/fix/tackle soon
    • framework
      • bug fixes
        • performance improvements for easyconfig parsing (PR #3555)
        • re-enable write permissions when installing with --read-only-installdir (PR #3649)
        • allow module-only on existing installations (without write permissions) (PR #3659)
      • enhancements
        • support additional features in easystack files
          • support for filtering via labels (PR #3620)
        • allow for overriding RPATH with higher priority paths (PR #3650)
        • avoid using a priority in prepend_module_path (Lmod) to avoid costly module calls (PR #3636)
        • allow amending easyconfig parameters which are not the default (PR #3651)
        • add support for postinstallcmds for extensions (PR #3663)
        • add support for --sanity-check-only (PR #3655)
          • needs more work to ensure that sanity checks for extensions are also done when using eb --sanity-check-only
        • add support for using oneAPI versions of 'intel' toolchain components (PR #3665)
        • add support for installing extensions in parallel (WIP) (PR #3667)
        • add toolchain definition for gofbf (foss with FlexiBLAS rather than OpenBLAS) (PR #3666)
      • changes
        • deprecate adding a non-existing path to $MODULEPATH (PR #3637)
        • don't skip sanity check for --module-only --rebuild (PR #3645)
    • easyblocks
      • bug fixes
        • treat files/directories of unpacked sources equally in PackedBinary (PR #2306)
        • fix installopts before installing the extension in GROMACS easyblock (PR #2391)
        • use explicit build toolset and compiler path in Boost easyblock (PR #2402)
        • --module-only doesn't always work as expected
          • see issues #2397 (Bundle), #2399 (GROMACS), PR #2401 (Tkinter)
          • we need a better way of catching this in tests
          • problem is that you typically need an actual installation to catch these problems, so can't be done in easyconfigs or easyblocks test suite run in CI
          • test installations done on generoso via boegelbot could be enhanced to catch problems with --module-only?
      • enhancements
        • enhance test and install step of CMakePythonPackage easyblock (PR #2318)
        • allow for Perl modules being part of other, already installed Perl modules (PR #2386)
        • enhance Python easyblock: install pip2 when available, tweak defaults, create unversioned pip symlink (PR #2388)
        • make it possible to force disabling kernel features in Qt easyblock (PR #2403)
        • update imkl easyblock to support oneAPI versions (>= 2021.x) (PR #2407)
      • changes
        • don't unpack whl files by default in generic PythonPackage easyblock (PR #2366)
      • new software
        • new easyblock for NCCL (built from source) (PR #2337)
        • custom easyblock for FlexiBLAS (PR #2369)
    • easyconfigs
      • bug fixes
        • add patches for PyTorch 1.7.1 avoiding failures on POWER and A100 (PR #12753)
        • improve check for multi-variant dependencies per generation of easyconfigs (PR #12687)
      • enhancements
        • (nothing major)
      • new software
      • software updates

2021a update of common toolchains

  • outlook to component versions
    • foss/2021a
    • intel/2021a
      • GCC 10.3 + binutils 2.36.1 as a base
      • intel-compilers 2021.2.0 (done, see PR #12765)
      • impi 2021.2.0 (done, see PR #12766)
      • imkl 2021.2.0
        • WIP by Kenneth
        • see toolchain fixes in framework (WIP, PR #3665) + imkl easyblock update (PR #2407)
  • BLAS/LAPACK component for foss; options:
    • stick to OpenBLAS
    • switch to BLIS
    • switch to FlexiBLAS
      • use OpenBLAS as only backend
      • easy testing with other BLAS/LAPACK backends (like BLIS)
      • easily collecting statistics on which BLAS/LAPACK functions are most relevant (focused continued evaluation)
    • for now (in develop):
      • define a foss/2021.x toolchain that's a candidate for foss/2021a, using FlexiBLAS with OpenBLAS backend
      • evaluate progress during next conf call, decide then?
  • collapsing foss and fosscuda toolchains
    • see https://github.com/easybuilders/easybuild-easyconfigs/issues/12484
    • we have a good way of handling this via an OpenMPI "plugin"
      • to be installed via gompi, on top of UCX+CUDA
    • what about intelcuda?
      • do the same thing just for symmetry, unclear whether Intel MPI actually supports this functionality to leverage CUDA
    • we won't really have a fosscuda toolchain anymore
      • the OpenMPI CUDA plugin will just be a different dependency
      • MPI+CUDA aware installations could be indicated via a versionsuffix
      • or another hierarchy level (cfr. ComputeCanada)

Q&A

  • Mikael: code-server "extensions", which are just to be unpacked + copied
    • doesn't really need proper extension support, just add stuff to unpacked to sources + copy via postinstallcmds?
    • could consider making Tarball usable to install extensions...
Clone this wiki locally