From 463a9dba1c2cedaebe6df90ceb0a6d84ee4eebae Mon Sep 17 00:00:00 2001 From: Brad Chamberlain Date: Wed, 20 Mar 2024 10:55:16 -0700 Subject: [PATCH 1/3] Look for HPE/EX-related showstoppers in cray.html While reviewing the website for the 2.0 release today, we realized that HPE Cray EX isn't highlighted as a platform on our download page alongside XC, which is increasingly being put to bed. Updating the website is easy, but most of the material is here in the platform docs, which was still very XC-centric in terms of ordering of material. Here, I've taken a pass to try and bring HPE EX and Apollo more to the forefront relative to Cray XC and CS, while also touching up some other things that caught my eye. I think there's more we could/should do here to bring HPE systems to the forefront and give them equal time and attention to the historical Cray systems, but am currently imagining that to be post-2.0 work. --- Signed-off-by: Brad Chamberlain --- doc/rst/platforms/cray.rst | 152 +++++++++++++++++++------------------ 1 file changed, 80 insertions(+), 72 deletions(-) diff --git a/doc/rst/platforms/cray.rst b/doc/rst/platforms/cray.rst index a6923227ddb7..f0ef9f940af0 100644 --- a/doc/rst/platforms/cray.rst +++ b/doc/rst/platforms/cray.rst @@ -1,26 +1,25 @@ .. _readme-cray: -============================ -Using Chapel on Cray Systems -============================ +================================ +Using Chapel on HPE Cray Systems +================================ -The following information is assembled to help Chapel users get up and running -on HPE Cray\ |reg| systems including the Cray XC\ |trade| and -HPE Cray EX series systems. +The following information is assembled to help Chapel users get up and +running on HPE Cray systems including the HPE Cray EX and Cray XC +series systems. .. contents:: -------------------------------------------------------------- -Getting Started with Chapel on Cray XC or HPE Cray EX Systems -------------------------------------------------------------- +-------------------------------------------------------- +Getting Started with Chapel on HPE Cray EX or XC Systems +-------------------------------------------------------- -Chapel is available as a module for Cray XC and HPE Cray EX systems. -When it is installed on your system, you do not need to build Chapel -from the source release (though you can). With either the traditional -Tcl-based module system or the newer Lua-based Lmod module system, to -use Chapel with the default settings and confirm it is correctly -installed, do the following: +Chapel is available as a module for HPE Cray EX and XC systems. When +it is installed on your system, you do not need to build Chapel from +the source release (though you can). Using the module systems on such +platforms, you can use Chapel with the default settings and confirm it +is correctly installed, as follows: 1) Ensure this required module is loaded. Normally it will be loaded for you, but under some circumstances you may need to load or @@ -28,7 +27,7 @@ installed, do the following: .. code-block:: sh - PrgEnv-cray or PrgEnv-gnu + PrgEnv-gnu or PrgEnv-cray 2) Load the Chapel module:: @@ -46,10 +45,10 @@ installed, do the following: ./hello6-taskpar-dist -nl 4 -This should be all that is necessary to use Chapel on a Cray XC or HPE -Cray EX system. If the installation setup by your system administrator +This should be all that is necessary to use Chapel on an HPE Cray EX or XC +system. If the installation setup by your system administrator deviates from the default settings, or you are interested in other -configuration options, see `Using Chapel on a Cray System`_ below. If +configuration options, see `Using Chapel on an HPE Cray System`_ below. If instead you wish to build Chapel from source, continue on to `Building Chapel for a Cray System from Source`_ just below. @@ -61,48 +60,59 @@ For information on obtaining and installing the Chapel module please contact your system administrator. ----------------------------------------------- -Getting Started with Chapel on Cray CS Systems ----------------------------------------------- +------------------------------------------------------------- +Getting Started with Chapel on HPE Apollo and Cray CS Systems +------------------------------------------------------------- -On Cray CS systems, Chapel is not currently available as a module due -to the wide diversity of software packages that Cray CS customers may -choose to install on their system. For this reason, Chapel must be -built from source on Cray CS systems using the -`Building Chapel for a Cray System from Source`_ instructions just below. +On HPE Apollo and Cray CS systems, Chapel is not currently available +as a module due to the wide diversity of configurations that these +systems support. For this reason, Chapel must be built from source on +Cray CS systems using the `Building Chapel for a Cray System from +Source`_ instructions just below. --------------------------------------------- Building Chapel for a Cray System from Source --------------------------------------------- -1) If using an XC or EX system, continue to step 2. If using a CS series - system, set ``CHPL_HOST_PLATFORM`` to ``cray-cs``. +1) Set ``CHPL_HOST_PLATFORM`` to the string representing your system + type. For example: .. code-block:: sh - export CHPL_HOST_PLATFORM=cray-cs + export CHPL_HOST_PLATFORM=hpe-cray-ex - These are the supported systems and strings. Note that these values - are used by default when building on the given systems. They can - also be set manually. + The following table lists the supported systems and strings. Note + that on HPE Cray EX and XC systems, these values should typically + be inferred automatically and not need to be set manually. That said, + there is also no downside to setting them manually. As with most + `CHPL_*` environment variables, the current set and inferred values + can be determined by running ``$CHPL_HOME/util/printchplenv``. - ========= ================== - System CHPL_HOST_PLATFORM - ========= ================== - CS series cray-cs - XC series cray-xc - EX series hpe-cray-ex - ========= ================== + ============= ================== + System CHPL_HOST_PLATFORM + ============= ================== + EX series hpe-cray-ex + Apollo series hpe-apollo + XC series cray-xc + CS series cray-cs + ============= ================== 2) Optionally, set the ``CHPL_LAUNCHER`` environment variable to indicate how Chapel should launch jobs on your system: ======================================== ========================= - On a Cray CS system, to... set CHPL_LAUNCHER to... + On an HPE Cray EX system, ... set CHPL_LAUNCHER to... + ======================================== ========================= + ...queue jobs using SLURM (sbatch) slurm-srun + ...run jobs interactively using PALS pals + ======================================== ========================= + + ======================================== ========================= + On a HPE Apollo or Cray CS system, to... set CHPL_LAUNCHER to... ======================================== ========================= ...run jobs interactively on your system gasnetrun_ibv ...queue jobs using SLURM (sbatch) slurm-gasnetrun_ibv @@ -118,24 +128,20 @@ Building Chapel for a Cray System from Source ...queue jobs using SLURM (sbatch) slurm-srun ======================================== ========================= - ======================================== ========================= - On an HPE Cray EX system, ... set CHPL_LAUNCHER to... - ======================================== ========================= - ...queue jobs using SLURM (sbatch) slurm-srun - ...run jobs interactively using PALS pals - ======================================== ========================= - - You can also set CHPL_LAUNCHER to ``none`` if you prefer to manually - manage all queuing and job launch commands yourself. + On HPE Cray EX systems, ``CHPL_LAUNCHER`` defaults to ``slurm-srun`` + if ``srun`` is in your path and ``none`` otherwise. - On Cray CS systems, ``CHPL_LAUNCHER`` defaults to ``gasnetrun_ibv``. + On HPE Apollo and Cray CS systems, ``CHPL_LAUNCHER`` defaults to + ``gasnetrun_ibv``. On Cray XC systems, ``CHPL_LAUNCHER`` defaults to ``aprun`` if ``aprun`` is in your path, ``slurm-srun`` if ``srun`` is in your path and ``none`` otherwise. - On HPE Cray EX systems, ``CHPL_LAUNCHER`` defaults to ``slurm-srun`` - if ``srun`` is in your path and ``none`` otherwise. + You can also set CHPL_LAUNCHER to ``none`` if you prefer to manually + manage all queuing and job launch commands yourself, though the + advantage of using a launcher is to make sure that Chapel maps + processes and threads to the appropriate target hardware units. For more information on Chapel's launcher capabilities and options, refer to :ref:`readme-launcher`. @@ -144,20 +150,22 @@ Building Chapel for a Cray System from Source 3) Select the target compiler that Chapel should use when compiling code for the compute node: - On a Cray CS series system, set the ``CHPL_TARGET_COMPILER`` environment - variable to indicate which compiler to use (and make sure that the compiler - is in your path). + On an HPE Apollo or Cray CS series system, set the + ``CHPL_TARGET_COMPILER`` environment variable to indicate which + compiler to use (and make sure that the compiler is in your path). =========================== ============================== To request... set CHPL_TARGET_COMPILER to... =========================== ============================== - ...the GNU compiler (gcc) gnu (default) + ...the LLVM/clang backend llvm (default) + ...the GNU compiler (gcc) gnu + ...the Clang compiler clang ...the Intel compiler (icc) intel =========================== ============================== - On a Cray XC or HPE Cray EX system, ensure that you have one of the - following Programming Environment modules loaded to specify your - target compiler:: + On an HPE Cray EX or Cray XC system, when using the C back-end, + ensure that you have one of the following Programming Environment + modules loaded to specify your target compiler:: PrgEnv-allinea (ARM only) PrgEnv-cray @@ -179,9 +187,9 @@ Building Chapel for a Cray System from Source between any of these configurations without rebuilding. ------------------------------ -Using Chapel on a Cray System ------------------------------ +---------------------------------- +Using Chapel on an HPE Cray System +---------------------------------- 1) If you are working from a Chapel module: @@ -219,10 +227,10 @@ Using Chapel on a Cray System is not large enough. Increase the number of KVS entries by setting env variable PMI_MAX_KVS_ENTRIES to a higher value. - Having the Cray PMI modules loaded when the program is compiled will - prevent this problem. We expect that eventually these modules will - be loaded by default on EX systems, but so far this has not been the - case. + Having the Cray PMI modules loaded when the program is compiled + will prevent this problem. We expect that eventually these modules + will be loaded by default on EX systems, but so far this has not + consistentlybeen the case. 3) Compile your Chapel program. For example: @@ -282,10 +290,10 @@ Using Chapel on a Cray System Requests the program to be executed using two locales. -6) If your Cray system has compute nodes with varying numbers of +6) If your HPE Cray system has compute nodes with varying numbers of cores, you can request nodes with at least a certain number of cores using the variable ``CHPL_LAUNCHER_CORES_PER_LOCALE``. For - example, on a Cray system in which some compute nodes have 24 or + example, on a system in which some compute nodes have 24 or more cores per compute node, you could request nodes with at least 24 cores using: @@ -311,10 +319,10 @@ Using Chapel on a Cray System between Chapel launchers and workload managers. -7) If your Cray system has compute nodes with varying numbers of CPUs +7) If your HPE Cray system has compute nodes with varying numbers of CPUs per compute unit, you can request nodes with a certain number of CPUs per compute unit using the variable ``CHPL_LAUNCHER_CPUS_PER_CU``. - For example, on a Cray XC series system with some nodes having at + For example, on a Cray XC system with some nodes having at least 2 CPUs per compute unit, to request running on those nodes you would use: @@ -655,7 +663,7 @@ Known Constraints and Bugs * Redirecting stdin when executing a Chapel program under PBS/qsub may not work due to limitations of qsub. -* For XC and EX systems, there is a known issue with the Cray MPI +* For EX and XC systems, there is a known issue with the Cray MPI release that causes some programs to assert and then hang during exit. A workaround is to set the environment variable, ``MPICH_GNI_DYNAMIC_CONN`` to ``disabled``. Setting this environment From 1ffd6f06253d89ec185f022ebace302d01d169d1 Mon Sep 17 00:00:00 2001 From: Brad Chamberlain Date: Wed, 20 Mar 2024 13:09:00 -0700 Subject: [PATCH 2/3] Incorporate Jade's comments --- Signed-off-by: Brad Chamberlain --- doc/rst/platforms/cray.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/doc/rst/platforms/cray.rst b/doc/rst/platforms/cray.rst index f0ef9f940af0..2bbaa0d8353f 100644 --- a/doc/rst/platforms/cray.rst +++ b/doc/rst/platforms/cray.rst @@ -50,7 +50,7 @@ system. If the installation setup by your system administrator deviates from the default settings, or you are interested in other configuration options, see `Using Chapel on an HPE Cray System`_ below. If instead you wish to build Chapel from source, continue on to `Building -Chapel for a Cray System from Source`_ just below. +Chapel for an HPE Cray System from Source`_ just below. Note that the Chapel module for HPE Cray EX systems does not yet have the gasnet communication layer built into it. For multilocale execution @@ -67,13 +67,13 @@ Getting Started with Chapel on HPE Apollo and Cray CS Systems On HPE Apollo and Cray CS systems, Chapel is not currently available as a module due to the wide diversity of configurations that these systems support. For this reason, Chapel must be built from source on -Cray CS systems using the `Building Chapel for a Cray System from +these systems using the `Building Chapel for an HPE Cray System from Source`_ instructions just below. ---------------------------------------------- -Building Chapel for a Cray System from Source ---------------------------------------------- +-------------------------------------------------- +Building Chapel for an HPE Cray System from Source +-------------------------------------------------- 1) Set ``CHPL_HOST_PLATFORM`` to the string representing your system type. @@ -230,7 +230,7 @@ Using Chapel on an HPE Cray System Having the Cray PMI modules loaded when the program is compiled will prevent this problem. We expect that eventually these modules will be loaded by default on EX systems, but so far this has not - consistentlybeen the case. + consistently been the case. 3) Compile your Chapel program. For example: From 51a968c6d14eb1ca5cb6a72c3d9ad33402926ab6 Mon Sep 17 00:00:00 2001 From: Brad Chamberlain Date: Wed, 20 Mar 2024 13:56:50 -0700 Subject: [PATCH 3/3] Minor wording tweak based on my own thoughts around a John reaction --- Signed-off-by: Brad Chamberlain --- doc/rst/platforms/cray.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/rst/platforms/cray.rst b/doc/rst/platforms/cray.rst index 2bbaa0d8353f..b55929387f07 100644 --- a/doc/rst/platforms/cray.rst +++ b/doc/rst/platforms/cray.rst @@ -52,8 +52,8 @@ configuration options, see `Using Chapel on an HPE Cray System`_ below. If instead you wish to build Chapel from source, continue on to `Building Chapel for an HPE Cray System from Source`_ just below. -Note that the Chapel module for HPE Cray EX systems does not yet have -the gasnet communication layer built into it. For multilocale execution +Note that the Chapel module for HPE Cray EX systems does not currently have +the GASNet communication layer built into it. For multilocale execution on EX systems please use the ofi communication layer instead. For information on obtaining and installing the Chapel module please