Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update project_charter/documentation, update util/check_package_config.py, remove upp from unified-dev #1022

Merged
merged 11 commits into from
Mar 11, 2024
Merged

Update project_charter/documentation, update util/check_package_config.py, remove upp from unified-dev #1022

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
0e41a34
Fix several bugs in util/check_package_config.py b/util/check_package…
climbfuji Mar 7, 2024
0e6fc0c
Remove upp from spack-ext/repos/spack-stack/packages/ufs-srw-app-env/…
climbfuji Mar 7, 2024
5ceaf1d
Clean up configs/templates/unified-dev/spack.yaml
climbfuji Mar 7, 2024
1c06828
Update README.md, project_charter.md and documentation with new maint…
climbfuji Mar 8, 2024
815cd37
Update util/ldd_check.py and documentation - this tool is available f…
climbfuji Mar 8, 2024
abb3a4e
Update util/util_tests.sh
climbfuji Mar 8, 2024
83b3cbd
Update submodule pointer for spack
climbfuji Mar 8, 2024
db9ae43
Merge branch 'develop' of https://github.com/jcsda/spack-stack into f…
climbfuji Mar 8, 2024
2ae50c5
Fix util tests
climbfuji Mar 8, 2024
05a8245
Merge branch 'develop' of https://github.com/jcsda/spack-stack into f…
climbfuji Mar 8, 2024
e17ac1e
Merge branch 'develop' into feature/update_check_package_config_and_r…
climbfuji Mar 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/util-tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,11 @@ jobs:
- name: checkout
uses: actions/checkout@v2
with:
submodules: false
submodules: true

- name: run-util-tests
run: |
export SPACK_STACK_DIR=$PWD
source ./setup.sh
cd util/
./util_tests.sh
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ Full documentation with table of contents can be found at https://spack-stack.re
Spack-stack is a collaborative effort between:
* [NOAA Environmental Modeling Center (EMC)](https://www.emc.ncep.noaa.gov/emc_new.php): [Alex Richert](https://www.github.com/AlexanderRichert-NOAA), [Hang Lei](https://www.github.com/Hang-Lei-NOAA), [Ed Hartnett](https://www.github.com/edwardhartnett)
* [UCAR Joint Center for Satellite Data Assimilation (JCSDA)](https://www.jcsda.org/): [Dom Heinzeller](https://www.github.com/climbfuji), [Steve Herbener](https://github.com/srherbener)
* [Earth Prediction Innovation Center (EPIC)](https://epic.noaa.gov/): [Cam Book](https://github.com/ulmononian), [Natalie Perlin](https://github.com/natalie-perlin)
* [Earth Prediction Innovation Center (EPIC)](https://epic.noaa.gov/): [Cam Book](https://github.com/ulmononian), [Natalie Perlin](https://github.com/natalie-perlin), [Ratko Vasic](https://github.com/ratkovasic-noaa)

For more information about the organization of the spack-stack
project, see the [Project Charter](project_charter.md).
4 changes: 4 additions & 0 deletions configs/templates/skylab-dev/spack.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ spack:
- compilers: ['%aocc', '%apple-clang', '%gcc', '%intel']
- packages:
- ewok-env +ecflow +cylc
- ai-env
- geos-gcm-env
- jedi-fv3-env
- jedi-geos-env
Expand All @@ -30,3 +31,6 @@ spack:
- matrix:
- [$packages]
- [$compilers]
exclude:
# py-torch in ai-env doesn't build with Intel
- ai-env%intel
4 changes: 0 additions & 4 deletions configs/templates/unified-dev/spack.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,9 @@ spack:
- jedi-tools-env
- jedi-ufs-env
- jedi-um-env
#- nceplibs-env
- soca-env
- ufs-srw-app-env
#- ufs-utils-env
- ufs-weather-model-env
#- upp-env
#- ww3-env

# Various fms tags (list all to avoid duplicate packages)
- fms@release-jcsda
Expand Down
4 changes: 2 additions & 2 deletions doc/source/Overview.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,11 @@ spack-stack is mainly a collection of Spack configuration files, but provides a

spack-stack is maintained by:

- Alex Richert (@AlexanderRichert-NOAA) and Ed Hartnett (@edwardhartnett), NOAA-EMC
- Alex Richert (@AlexanderRichert-NOAA), Hang Lei (@Hang-Lei-NOAA) and Ed Hartnett (@edwardhartnett), NOAA-EMC

- Dom Heinzeller (@climbfuji) and Steve Herbener (@srherbener), JCSDA

- Cameron Book (@ulmononian) and Mark Potts (@mark-a-potts), EPIC
- Cameron Book (@ulmononian), Natalie Perlin (@natalie-perlin) and Ratko Vasic (@ratkovasic-noaa), EPIC

===============
Getting started
Expand Down
40 changes: 20 additions & 20 deletions doc/source/PreConfiguredSites.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,49 +16,49 @@ Ready-to-use spack-stack 1.6.0 installations are available on the following, ful
On selected systems, developmental versions / release candidates are installed that are newer than spack-stack 1.6.0 (see following table). For information on the spack-stack 1.6.0 releases on this platforms, please revert to version 1.6.0 of the documentation (https://spack-stack.readthedocs.io/en/1.6.0/PreConfiguredSites.html#pre-configured-sites-tier-1).

+---------------------+----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| Organization | System | Compilers | Location | Maintainers |
| Organization | System | Compilers | Location | Maintainers (principal/backup)|
+=====================+==================================+=================+=========================================================================================================+===============================+
| **HPC platforms** |
+---------------------+----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| MSU | Hercules GCC+OpenMPI recommended | GCC | ``/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/ue-gcc12-openmpi416`` | Dom Heinzeller |
| MSU | Hercules GCC+OpenMPI recommended | GCC | ``/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/ue-gcc12-openmpi416`` | EPIC / JCSDA |
+---------------------+----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| | Hercules | (GCC), Intel | ``/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/unified-env`` | Cam Book / Dom Heinzeller |
| | Hercules | (GCC), Intel | ``/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/unified-env`` | EPIC / JCSDA |
| MSU +----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| | Orion | GCC, Intel | ``/work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/envs/unified-env`` | Cam Book / Dom Heinzeller |
| | Orion | GCC, Intel | ``/work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/envs/unified-env`` | EPIC / JCSDA |
+---------------------+----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| | Discover SCU16 | GCC, Intel | ``/gpfsm/dswdev/jcsda/spack-stack/scu16/spack-stack-20240228/envs/unified-env-*`` | Dom Heinzeller / ??? |
| | Discover SCU16 | GCC, Intel | ``/gpfsm/dswdev/jcsda/spack-stack/scu16/spack-stack-20240228/envs/unified-env-*`` | JCSDA |
| NASA +----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| | Discover SCU17 | GCC, Intel | ``/gpfsm/dswdev/jcsda/spack-stack/scu17/spack-stack-20240228/envs/unified-env-*`` | Dom Heinzeller / ??? |
| | Discover SCU17 | GCC, Intel | ``/gpfsm/dswdev/jcsda/spack-stack/scu17/spack-stack-20240228/envs/unified-env-*`` | JCSDA |
+---------------------+----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| | Casper | GCC | ``/glade/work/epicufsrt/contrib/spack-stack/casper/spack-stack-1.6.0/envs/unified-env`` | Dom Heinzeller / ??? |
| | Casper | GCC | ``/glade/work/epicufsrt/contrib/spack-stack/casper/spack-stack-1.6.0/envs/unified-env`` | JCSDA / EPIC |
| NCAR-Wyoming +----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| | Derecho | GCC, Intel | ``/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.6.0/envs/unified-env`` | Dom Heinzeller / Cam Book |
| | Derecho | GCC, Intel | ``/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.6.0/envs/unified-env`` | EPIC / JCSDA |
+---------------------+----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| NOAA (NCEP) | Acorn | Intel | ``/lfs/h1/emc/nceplibs/noscrub/spack-stack/spack-stack-1.6.0/envs/unified-env-intel{19,2022}`` | Hang Lei / Alex Richert |
| NOAA (NCEP) | Acorn | Intel | ``/lfs/h1/emc/nceplibs/noscrub/spack-stack/spack-stack-1.6.0/envs/unified-env-intel{19,2022}`` | NOAA-EMC |
+---------------------+----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| | Gaea C5 | Intel | ``/ncrc/proj/epic/spack-stack/spack-stack-1.6.0/envs/unified-env`` | Cam Book / Dom Heinzeller |
| | Gaea C5 | Intel | ``/ncrc/proj/epic/spack-stack/spack-stack-1.6.0/envs/unified-env`` | EPIC / NOAA-EMC |
| +----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| NOAA (RDHPCS) | Hera | GCC, Intel | ``/scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/unified-env`` | Mark Potts / Dom Heinzeller |
| NOAA (RDHPCS) | Hera | GCC, Intel | ``/scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/unified-env`` | EPIC / NOAA-EMC |
| +----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| | Jet | GCC, Intel | ``/mnt/lfs4/HFIP/hfv3gfs/role.epic/spack-stack/spack-stack-1.6.0/envs/unified-env`` | Cam Book / Dom Heinzeller |
| | Jet | GCC, Intel | ``/mnt/lfs4/HFIP/hfv3gfs/role.epic/spack-stack/spack-stack-1.6.0/envs/unified-env`` | EPIC / NOAA-EMC |
+---------------------+----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| | Narwhal | GCC, Intel | ``/p/app/projects/NEPTUNE/spack-stack/spack-stack-1.6.0/envs/unified-env-*`` | Dom Heinzeller / Sarah King |
| | Narwhal | GCC, Intel | ``/p/app/projects/NEPTUNE/spack-stack/spack-stack-1.6.0/envs/unified-env-*`` | JCSDA / NRL |
| +----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| U.S. Navy (HPCMP) | Nautilus | Intel | ``/p/app/projects/NEPTUNE/spack-stack/spack-stack-1.6.0/envs/unified-env`` | Dom Heinzeller / Sarah King |
| U.S. Navy (HPCMP) | Nautilus | Intel | ``/p/app/projects/NEPTUNE/spack-stack/spack-stack-1.6.0/envs/unified-env`` | JCSDA / NRL |
| +----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| | Nautilus | AOCC | *currently not supported* | Dom Heinzeller / Sarah King |
| | Nautilus | AOCC | *currently not supported* | JCSDA / NRL |
+---------------------+----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| | S4 | Intel | ``/data/prod/jedi/spack-stack/spack-stack-1.6.0/envs/unified-env`` | Dom Heinzeller / Mark Potts |
| | S4 | Intel | ``/data/prod/jedi/spack-stack/spack-stack-1.6.0/envs/unified-env`` | JCSDA |
| Univ. of Wisconsin +----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| | S4 | GCC | *currently not supported* | Dom Heinzeller / Mark Potts |
| | S4 | GCC | *currently not supported* | JCSDA |
+---------------------+----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| **Cloud platforms** |
+---------------------+----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| | AMI Red Hat 8 | GCC | ``/home/ec2-user/spack-stack/spack-stack-1.6.0/envs/unified-env`` | Dom Heinzeller / ??? |
| | AMI Red Hat 8 | GCC | ``/home/ec2-user/spack-stack/spack-stack-1.6.0/envs/unified-env`` | JCSDA |
+ Amazon Web Services +----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| | Parallelcluster JCSDA R&D | Intel | ``/mnt/experiments-efs/skylab-v8/spack-stack-20240207/envs/unified-env-*`` | Dom Heinzeller / ??? |
| | Parallelcluster JCSDA R&D | Intel | ``/mnt/experiments-efs/skylab-v8/spack-stack-20240207/envs/unified-env-*`` | JCSDA |
+---------------------+----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| NOAA (RDHPCS) | RDHPCS Cloud (Parallel Works) | Intel | ``/contrib/spack-stack/spack-stack-1.6.0/envs/unified-env`` | Mark Potts / Cam Book / Dom H |
| NOAA (RDHPCS) | RDHPCS Cloud (Parallel Works) | Intel | ``/contrib/spack-stack/spack-stack-1.6.0/envs/unified-env`` | EPIC / JCSDA |
+---------------------+----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+

For more information about a specific platform, please see the individual sections below.
Expand Down
4 changes: 2 additions & 2 deletions doc/source/Utilities.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,10 +46,10 @@ The utility located at util/check_permissions.sh can be run inside any spack-sta
.. _LDD_Checker:

------------------------------
ldd_check.py
ldd_check.py (Linux only)
------------------------------

The util/ldd_check.py utility should be run for new installations to ensure that no shared library or executable that uses shared libraries is missing a shared library dependency. If the script returns a warning for a given file, this may indicate that Spack's RPATH substitution has not been properly applied. In some instances, missing library dependencies may not indicate a problem, such as a library that is intended to be found through $LD_LIBRARY_PATH after, say, a compiler or MPI environment module is loaded. Though these paths should probably also be RPATH-ified, such instances of harmless missing dependencies may be ignored with ldd_check.py's ``--ignore`` option by specifying a Python regular expression to be excluded from consideration (see example below), or can be permanently whitelisted by modifying the ``whitelist`` variable at the top of the ldd_check.py script itself (in which case please submit a PR). The script searches the 'install/' subdirectory of a given path and runs ``ldd`` on all shared objects. The base path to be search can be specified as a lone positional argument, and by default is the current directory. In practice, this should be ``$SPACK_ENV`` for the environment in question.
The util/ldd_check.py utility should be run for new installations to ensure that no shared library or executable that uses shared libraries is missing a shared library dependency. If the script returns a warning for a given file, this may indicate that Spack's RPATH substitution has not been properly applied. In some instances, missing library dependencies may not indicate a problem, such as a library that is intended to be found through $LD_LIBRARY_PATH after, say, a compiler or MPI environment module is loaded. Though these paths should probably also be RPATH-ified, such instances of harmless missing dependencies may be ignored with ldd_check.py's ``--ignore`` option by specifying a Python regular expression to be excluded from consideration (see example below), or can be permanently whitelisted by modifying the ``whitelist`` variable at the top of the ldd_check.py script itself (in which case please submit a PR). The script searches the 'install/' subdirectory of a given path and runs ``ldd`` on all shared objects. The base path to be search can be specified as a lone positional argument, and by default is the current directory. In practice, this should be ``$SPACK_ENV`` for the environment in question. This utility is available for Linux only.

.. code-block:: console

Expand Down
Loading
Loading