forked from spack/spack
-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merge spack develop as of 2024/05/09 into spack-stack-dev #432
Merged
climbfuji
merged 1,409 commits into
JCSDA:spack-stack-dev
from
climbfuji:feature/merge_spack_develop_into_spack_stack_dev_20240509
May 21, 2024
Merged
Merge spack develop as of 2024/05/09 into spack-stack-dev #432
climbfuji
merged 1,409 commits into
JCSDA:spack-stack-dev
from
climbfuji:feature/merge_spack_develop_into_spack_stack_dev_20240509
May 21, 2024
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When looking at where we spend our time in solver setup, I noticed a fair bit of time is spent in `Spec.format()`, and `Spec.format()` is a pretty old, slow, convoluted method. This PR does a number of things: - [x] Consolidate most of what was being done manually with a character loop and several regexes into a single regex. - [x] Precompile regexes where we keep them - [x] Remove the `transform=` argument to `Spec.format()` which was only used in one place in the code (modules) to uppercase env var names, but added a lot of complexity - [x] Avoid escaping and colorizing specs unless necessary - [x] Refactor a lot of the colorization logic to avoid unnecessary object construction - [x] Add type hints and remove some spots in the code where we were using nonexistent arguments to `format()`. - [x] Add trivial cases to `__str__` in `VariantMap` and `VersionList` to avoid sorting - [x] Avoid calling `isinstance()` in the main loop of `Spec.format()` - [x] Don't bother constructing a `string` representation for the result of `_prev_version` as it is only used for comparisons. In my timings (on all the specs formatted in a solve of `hdf5`), this is over 2.67x faster than the original `format()`, and it seems to reduce setup time by around a second (for `hdf5`).
The default runner changed on GitHub for macOS, and that revealed a bug in a package when running audits
If there's no compiler we currently don't have any external libc for the solver. This commit adds a fallback on libc from the current Python process, which works if it is dynamically linked. Co-authored-by: Massimiliano Culpo <[email protected]>
Add `%oneapi`, remove compilers that have been discontinued upstream
test dependencies constrains build / link type deps, so avoid that
…k#43459)" (spack#43819) This reverts commit 9e2558b.
Some packages can't be redistributed in source or binary form. We need an explicit way to say that in a package. This adds a `redistribute()` directive so that package authors can write, e.g.: ```python redistribute(source=False, binary=False) ``` You can also do this conditionally with `when=`, as with other directives, e.g.: ```python # 12.0 and higher are proprietary redistribute(source=False, binary=False, when="@12.0:") # can't redistribute when we depend on some proprietary dependency redistribute(source=False, binary=False, when="^proprietary-dependency") ``` To prevent Spack from adding either their sources or binaries to public mirrors and build caches. You can still unconditionally add things *if* you run either: * `spack mirror create --private` * `spack buildcache push --private` But the default behavior for build caches is not to include non-redistributable packages in either mirrors or build caches. We have previously done this manually for our public buildcache, but with this we can start maintaining redistributability directly in packages. Caveats: currently the default for `redistribute()` is `True` for both `source` and `binary`, and you can only set either of them to `False` via this directive. - [x] add `redistribute()` directive - [x] add `redistribute_source` and `redistribute_binary` class methods to `PackageBase` - [x] add `--private` option to `spack mirror` - [x] add `--private` option to `spack buildcache push` - [x] test exclusion of packages from source mirror (both as a root and as a dependency) - [x] test exclusion of packages from binary mirror (both as a root and as a dependency)
* fix so trilinos@master uses correct kokkos (@4.3.00) * Update var/spack/repos/builtin/packages/trilinos/package.py
If a flang-new exists, which is rather unlikely, it probably means the user wants it as a fortran compiler.
* Updates for rocm Updated for rocm@6 Added conflict between rocprofiler and roctracer. Request either +rocprofiler or +roctracer when +rocm. In this case, it automatically builds for one, instead of displaying the message. Request +rocm when using either +rocprofiler or +roctracer. In this case, it automatically builds with +rocm, instead of displaying the message. Disabled the tests. Will update them with the new test method. * [@spackbot] updating style on behalf of jordialcaraz --------- Co-authored-by: jordialcaraz <[email protected]>
* Add intel-oneapi-mpi package * Fix style
We run `extend spack_flags_list SPACK_LDFLAGS` for `$mode in ld|ccld`. That's problematic, cause `ccld` needs `-Wl,--flag` whereas `ld` needs `--flag` directly. Only `-L` and `-l` are common to compiler & linker. In all build systems `LDFLAGS` is for the compiler not the linker, cause any linker flag `-x` can be passed as a compiler flag `-Wl,-x`, and there are many compiler flags that affect the linker invocation, like `-fopenmp`, `-fuse-ld=`, `-fsanitize=` etc. So don't pass `LDFLAGS` to the linker directly. This way users can set `ldflags: -Wl,--allow-shlib-undefined` in compilers.yaml to work around an issue where the linker tries to resolve the `libcuda.so.1` stub lib which cannot be located by design in `cuda`.
…/merge_spack_develop_into_spack_stack_dev_20240509
… does not build with boost 1.85:
… r3evert to using netcdf-c with autotools builder as dependency, and fix different library names of netcdf-cxx4 with cmake builder
14 tasks
…erving {name}_ROOT env variables in auto-generated module files
…her-model/package.py
…ckages/py-shapely/shapely-1.8.0-geos.py.patch
…introspection/package.py
climbfuji
requested review from
srherbener,
RatkoVasic-NOAA,
natalie-perlin and
AlexanderRichert-NOAA
May 15, 2024 02:45
srherbener
approved these changes
May 15, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for doing this immense amount of work!
climbfuji
added a commit
to JCSDA/spack-stack
that referenced
this pull request
May 21, 2024
…nto_spack_stack_dev_20240509_BUGFIX_PTR Bug fix after merging spack-stack PR 1109 before spack submodule PR JCSDA/spack#432
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Yet another round of updates from spack develop as of 2024/05/09 (
**Spack:** 0.22.0.dev0 (4154b2d83f87ae1408ceb604d6b9d5fee444f866)
)Please note the merge conflicts that I had to fix and additional commits to fix installation errors on my macOS for the unified environment:
AND
AND
AND
AND
We should consider cherry-picking the first two commits and sending them upstream to address spack#44122 and spack#44116.
Update. I think it's unlikely they will accept the reversal of the builder for
netcdf-cxx4
, but I already sent the changes forecflow
in spack#44181.Issue(s) addressed
Resolves #429 !
Dependencies
n/a
Impact
Updates to spack-stack required, potentially more bug fixes when building the unified environment on tier 1 platforms and configurable platforms.
Checklist
I have made corresponding changes to the documentation