Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge spack develop as of 2024/05/09 into spack-stack-dev #432

Conversation

climbfuji
Copy link
Collaborator

@climbfuji climbfuji commented May 10, 2024

Description

Yet another round of updates from spack develop as of 2024/05/09 (**Spack:** 0.22.0.dev0 (4154b2d83f87ae1408ceb604d6b9d5fee444f866))

Please note the merge conflicts that I had to fix and additional commits to fix installation errors on my macOS for the unified environment:

Merge branch 'develop' of https://github.com/spack/spack into feature/merge_spack_develop_into_spack_stack_dev_20240509

# Conflicts:
#       lib/spack/spack/modules/common.py
#       var/spack/repos/builtin/packages/awscli-v2/package.py
#       var/spack/repos/builtin/packages/fms/package.py
#       var/spack/repos/builtin/packages/fontconfig/package.py
#       var/spack/repos/builtin/packages/g2c/package.py
#       var/spack/repos/builtin/packages/g2tmpl/package.py
#       var/spack/repos/builtin/packages/gdal/package.py
#       var/spack/repos/builtin/packages/glib/package.py
#       var/spack/repos/builtin/packages/grads/package.py
#       var/spack/repos/builtin/packages/grib-util/package.py
#       var/spack/repos/builtin/packages/hdf-eos2/package.py
#       var/spack/repos/builtin/packages/libjpeg-turbo/package.py
#       var/spack/repos/builtin/packages/nco/package.py
#       var/spack/repos/builtin/packages/netcdf-fortran/package.py
#       var/spack/repos/builtin/packages/pixman/package.py
#       var/spack/repos/builtin/packages/prod-util/package.py
#       var/spack/repos/builtin/packages/py-torch/package.py
#       var/spack/repos/builtin/packages/python/package.py
#       var/spack/repos/builtin/packages/scotch/package.py
#       var/spack/repos/builtin/packages/w3nco/package.py
#       var/spack/repos/builtin/packages/zlib-ng/package.py
#       var/spack/repos/builtin/packages/zstd/package.py
#

AND

commit 37dce57a0fa0e0d6dcee917d3f37bff3db6a7f7e (HEAD -> feature/merge_spack_develop_into_spack_stack_dev_20240509)
Author: Dom Heinzeller <[email protected]>
Date:   Fri May 10 11:09:03 2024 -0600

    Bug fix in var/spack/repos/builtin/packages/ecflow/package.py: 5.11.4 does not build with boost 1.85:

AND

commit e3a929305c1febd70c48c081fd51d225a9aabcf2 (HEAD -> feature/merge_spack_develop_into_spack_stack_dev_20240509, dom/feature/merge_spack_develop_into_spack_stack_dev_20240509)
Author: Dom Heinzeller <[email protected]>
Date:   Thu May 16 14:10:19 2024 -0600

    Revert netcdf-cxx4 back to autotools version

AND

commit c30b4fc075ccccc130beb5873c26529d758ed007 (HEAD -> feature/merge_spack_develop_into_spack_stack_dev_20240509)
Author: Dom Heinzeller <[email protected]>
Date:   Fri May 10 13:31:54 2024 -0600

    Fix new syntax in lib/spack/spack/modules/common.py to have case-preserving {name}_ROOT env variables in auto-generated module files

AND

commit 101cce38ae799d83e457077edafb35a64a74c88a (HEAD -> feature/merge_spack_develop_into_spack_stack_dev_20240509, dom/feature/merge_spack_develop_into_spack_stack_dev_20240509)
Author: Dom Heinzeller <[email protected]>
Date:   Fri May 10 14:57:31 2024 -0600

    Fix package audit errors in var/spack/repos/builtin/packages/ufs-weather-model/package.py

We should consider cherry-picking the first two commits and sending them upstream to address spack#44122 and spack#44116.

Update. I think it's unlikely they will accept the reversal of the builder for netcdf-cxx4, but I already sent the changes for ecflow in spack#44181.

Issue(s) addressed

Resolves #429 !

Dependencies

n/a

Impact

Updates to spack-stack required, potentially more bug fixes when building the unified environment on tier 1 platforms and configurable platforms.

Checklist

  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation
  • I have run the unit tests before creating the PR

haampie and others added 30 commits April 23, 2024 07:44
When looking at where we spend our time in solver setup, I noticed a fair bit of time is spent
in `Spec.format()`, and `Spec.format()` is a pretty old, slow, convoluted method.

This PR does a number of things:
- [x] Consolidate most of what was being done manually with a character loop and several
      regexes into a single regex.
- [x] Precompile regexes where we keep them 
- [x] Remove the `transform=` argument to `Spec.format()` which was only used in one 
      place in the code (modules) to uppercase env var names, but added a lot of complexity
- [x] Avoid escaping and colorizing specs unless necessary
- [x] Refactor a lot of the colorization logic to avoid unnecessary object construction
- [x] Add type hints and remove some spots in the code where we were using nonexistent
      arguments to `format()`.
- [x] Add trivial cases to `__str__` in `VariantMap` and `VersionList` to avoid sorting
- [x] Avoid calling `isinstance()` in the main loop of `Spec.format()`
- [x] Don't bother constructing a `string` representation for the result of `_prev_version`
      as it is only used for comparisons.

In my timings (on all the specs formatted in a solve of `hdf5`), this is over 2.67x faster than the 
original `format()`, and it seems to reduce setup time by around a second (for `hdf5`).
The default runner changed on GitHub for macOS, and that
revealed a bug in a package when running audits
If there's no compiler we currently don't have any external libc for the solver.

This commit adds a fallback on libc from the current Python process, which works if it is dynamically linked.

Co-authored-by: Massimiliano Culpo <[email protected]>
Add `%oneapi`, remove compilers that have been discontinued upstream
test dependencies constrains build / link type deps, so avoid that
Some packages can't be redistributed in source or binary form. We need an explicit way to say that in a package.

This adds a `redistribute()` directive so that package authors can write, e.g.:

```python
    redistribute(source=False, binary=False)
```

You can also do this conditionally with `when=`, as with other directives, e.g.:

```python
    # 12.0 and higher are proprietary
    redistribute(source=False, binary=False, when="@12.0:")

    # can't redistribute when we depend on some proprietary dependency
    redistribute(source=False, binary=False, when="^proprietary-dependency")
```


To prevent Spack from adding either their sources or binaries to public mirrors and build caches. You can still unconditionally add things *if* you run either:
* `spack mirror create --private`
* `spack buildcache push --private`

But the default behavior for build caches is not to include non-redistributable packages in either mirrors or build caches.  We have previously done this manually for our public buildcache, but with this we can start maintaining redistributability directly in packages.

Caveats: currently the default for `redistribute()` is `True` for both `source` and `binary`, and you can only set either of them to `False` via this directive.

- [x] add `redistribute()` directive
- [x] add `redistribute_source` and `redistribute_binary` class methods to `PackageBase`
- [x] add `--private` option to `spack mirror`
- [x] add `--private` option to `spack buildcache push`
- [x] test exclusion of packages from source mirror (both as a root and as a dependency)
- [x] test exclusion of packages from binary mirror (both as a root and as a dependency)
* fix so trilinos@master uses correct kokkos (@4.3.00)

* Update var/spack/repos/builtin/packages/trilinos/package.py
If a flang-new exists, which is rather unlikely, it probably means the
user wants it as a fortran compiler.
* Updates for rocm

Updated for rocm@6
Added conflict between rocprofiler and roctracer.
Request either +rocprofiler or +roctracer when +rocm. In this case, it automatically builds for one, instead of displaying the message. 
Request +rocm when using either +rocprofiler or +roctracer. In this case, it automatically builds with +rocm, instead of displaying the message.

Disabled the tests. Will update them with the new test method.

* [@spackbot] updating style on behalf of jordialcaraz

---------

Co-authored-by: jordialcaraz <[email protected]>
* Add intel-oneapi-mpi package

* Fix style
We run `extend spack_flags_list SPACK_LDFLAGS` for `$mode in ld|ccld`.

That's problematic, cause `ccld` needs `-Wl,--flag` whereas `ld` needs
`--flag` directly. Only `-L` and `-l` are common to compiler & linker.

In all build systems `LDFLAGS` is for the compiler not the linker, cause
any linker flag `-x` can be passed as a compiler flag `-Wl,-x`, and there
are many compiler flags that affect the linker invocation, like `-fopenmp`,
`-fuse-ld=`, `-fsanitize=` etc.

So don't pass `LDFLAGS` to the linker directly.

This way users can set `ldflags: -Wl,--allow-shlib-undefined` in compilers.yaml
to work around an issue where the linker tries to resolve the `libcuda.so.1`
stub lib which cannot be located by design in `cuda`.
…/merge_spack_develop_into_spack_stack_dev_20240509
… r3evert to using netcdf-c with autotools builder as dependency, and fix different library names of netcdf-cxx4 with cmake builder
@climbfuji climbfuji self-assigned this May 10, 2024
@climbfuji climbfuji added INFRA JEDI Infrastructure NOAA-EMC OAR-EPIC NOAA Oceanic and Atmospheric Research and Earth Prediction Innovation Center NAVY United States Naval Research Lab labels May 10, 2024
Copy link
Collaborator

@srherbener srherbener left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for doing this immense amount of work!

@climbfuji climbfuji merged commit a11d824 into JCSDA:spack-stack-dev May 21, 2024
16 of 17 checks passed
climbfuji added a commit to JCSDA/spack-stack that referenced this pull request May 21, 2024
…nto_spack_stack_dev_20240509_BUGFIX_PTR

Bug fix after merging spack-stack PR 1109 before spack submodule PR JCSDA/spack#432
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
INFRA JEDI Infrastructure NAVY United States Naval Research Lab NOAA-EMC OAR-EPIC NOAA Oceanic and Atmospheric Research and Earth Prediction Innovation Center
Projects
No open projects
Development

Successfully merging this pull request may close these issues.

spack unit tests broken since Apr 24?