From 6bd4f62188a9cc709121ee91667f7995609d2617 Mon Sep 17 00:00:00 2001 From: "John H. Hartman" Date: Wed, 21 Feb 2024 15:31:49 -0700 Subject: [PATCH 1/4] Documented suffix to -nl argument Added documentation for the suffix to the -nl argument, and cleaned up the co-locale description in general. Signed-off-by: John H. Hartman --- doc/rst/usingchapel/multilocale.rst | 48 +++++++++++++++++++++++------ 1 file changed, 38 insertions(+), 10 deletions(-) diff --git a/doc/rst/usingchapel/multilocale.rst b/doc/rst/usingchapel/multilocale.rst index d8642c888983..18988daa3299 100644 --- a/doc/rst/usingchapel/multilocale.rst +++ b/doc/rst/usingchapel/multilocale.rst @@ -198,16 +198,44 @@ ofi slurm-srun ========= ============= There are two ways to create co-locales. The first is to set the -``CHPL_RT_LOCALES_PER_NODE`` environment variable. If its value is greater -than one and less than or equal to the number of sockets on a node, then the -Chapel runtime will run each locale in its own socket. The second way to -create co-locales is to use the command-line argument ``-nl NxL``, where ``N`` -is the number of nodes and ``L`` is the number of locales per node. The ``L`` -is optional; if it isn't specified then Chapel will run the "ideal" number of -locales based on the node architecture. Currently this is limited to the -value of ``CHPL_RT_LOCALES_PER_NODE``; in future releases we plan to include -more sophisticated heuristics such as automatically running one locale per -socket on nodes with multiple sockets. +``CHPL_RT_LOCALES_PER_NODE`` environment variable. If set, Chapel will run +the specified number of locales per node. The second way to create co-locales +is to use the command-line argument ``-nl NxLt``, where ``N`` is the number +of nodes, ``L`` is the number of locales per node, and ``t`` is an optional +suffix indicating the architectural feature to which the co-locales should be +bound. The ``L`` is optional; if it isn't specified then Chapel will run +the "ideal" number of locales based on the node architecture. Currently this +is limited to the value of ``CHPL_RT_LOCALES_PER_NODE``; in future releases +we plan to include more sophisticated heuristics such as automatically +running one locale per socket on nodes with multiple sockets. + +By default, Chapel will try to bind co-locales to an architectural feature. +For example, launching a Chapel program with the argument ``-nl 1x2`` on a +node with two sockets will bind each co-locale to its own socket. Chapel +looks at the number of sockets, NUMA domains, Level 3 caches, and cores on +the node, in that order, to determine if the co-locales can be bound to +an architectural feature. If the number of co-locales does not match the +number of a feature then Chapel simply assigns an equal number of cores +to each co-locale. Any remaining cores are unused. + +You can force Chapel to bind co-locales to an architectural feature using a +suffix to the ``-nl`` argument. The valid suffixes and their bindings are as +follows: + +========= ============= +Suffix Binding +========= ============= +s socket +n NUMA domain +L Level 3 cache +c core +========= ============= + +It is an error if the number of co-locales is greater than the number of +specified architectural features. For example, specifying ``-nl 1x2s`` on a +node with a single socket results in an error. Any remaining cores are +unused; for example, specifying ``-nl 1x1s`` on a node with two sockets +will leave the cores in one socket unused. Troubleshooting +++++++++++++++ From e96c1a5b03d5f0c6b91427792d6dd6db27cdeb3d Mon Sep 17 00:00:00 2001 From: "John H. Hartman" Date: Mon, 26 Feb 2024 15:36:38 -0600 Subject: [PATCH 2/4] Addressed reviewer's comments Signed-off-by: John H. Hartman --- doc/rst/usingchapel/multilocale.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/rst/usingchapel/multilocale.rst b/doc/rst/usingchapel/multilocale.rst index 18988daa3299..41583f48c7e4 100644 --- a/doc/rst/usingchapel/multilocale.rst +++ b/doc/rst/usingchapel/multilocale.rst @@ -213,9 +213,9 @@ By default, Chapel will try to bind co-locales to an architectural feature. For example, launching a Chapel program with the argument ``-nl 1x2`` on a node with two sockets will bind each co-locale to its own socket. Chapel looks at the number of sockets, NUMA domains, Level 3 caches, and cores on -the node, in that order, to determine if the co-locales can be bound to -an architectural feature. If the number of co-locales does not match the -number of a feature then Chapel simply assigns an equal number of cores +the node, in that order, to determine if the co-locales can be bound to an +architectural feature. If the number of co-locales requested does not match +the number of any feature then Chapel simply assigns an equal number of cores to each co-locale. Any remaining cores are unused. You can force Chapel to bind co-locales to an architectural feature using a @@ -231,8 +231,8 @@ L Level 3 cache c core ========= ============= -It is an error if the number of co-locales is greater than the number of -specified architectural features. For example, specifying ``-nl 1x2s`` on a +It is an error to specify a number of co-locales greater than the number of +the specified architectural feature. For example, specifying ``-nl 1x2s`` on a node with a single socket results in an error. Any remaining cores are unused; for example, specifying ``-nl 1x1s`` on a node with two sockets will leave the cores in one socket unused. From 9c69ee1a259e4b9830195f9223c81a777ac6b603 Mon Sep 17 00:00:00 2001 From: "John H. Hartman" Date: Thu, 7 Mar 2024 16:12:37 -0700 Subject: [PATCH 3/4] updated with long-form co-locale suffix syntax Signed-off-by: John H. Hartman --- doc/rst/usingchapel/multilocale.rst | 30 ++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/doc/rst/usingchapel/multilocale.rst b/doc/rst/usingchapel/multilocale.rst index 41583f48c7e4..8c2da59a7fc9 100644 --- a/doc/rst/usingchapel/multilocale.rst +++ b/doc/rst/usingchapel/multilocale.rst @@ -207,33 +207,33 @@ bound. The ``L`` is optional; if it isn't specified then Chapel will run the "ideal" number of locales based on the node architecture. Currently this is limited to the value of ``CHPL_RT_LOCALES_PER_NODE``; in future releases we plan to include more sophisticated heuristics such as automatically -running one locale per socket on nodes with multiple sockets. +running one locale per socket on nodes with multiple sockets. Note: the +``-nl NxLt`` syntax is considered unstable and may change in the future. By default, Chapel will try to bind co-locales to an architectural feature. For example, launching a Chapel program with the argument ``-nl 1x2`` on a node with two sockets will bind each co-locale to its own socket. Chapel -looks at the number of sockets, NUMA domains, Level 3 caches, and cores on -the node, in that order, to determine if the co-locales can be bound to an +looks at the number of sockets, NUMA domains, caches, and cores on the node, +in that order, to determine if the co-locales can be bound to an architectural feature. If the number of co-locales requested does not match the number of any feature then Chapel simply assigns an equal number of cores to each co-locale. Any remaining cores are unused. -You can force Chapel to bind co-locales to an architectural feature using a -suffix to the ``-nl`` argument. The valid suffixes and their bindings are as -follows: +You can force Chapel to bind co-locales to an architectural feature with a +suffix to the ``-nl`` argument. The valid suffixes and their bindings are: -========= ============= -Suffix Binding -========= ============= -s socket -n NUMA domain -L Level 3 cache -c core -========= ============= +=========== ============= +Suffix Binding +=========== ============= +s or socket socket +numa NUMA domain +llc last-level cache +c or core core +=========== ============= It is an error to specify a number of co-locales greater than the number of the specified architectural feature. For example, specifying ``-nl 1x2s`` on a -node with a single socket results in an error. Any remaining cores are +node with a single socket is an error. Any remaining cores are unused; for example, specifying ``-nl 1x1s`` on a node with two sockets will leave the cores in one socket unused. From ad7bd304e6b3629401cc5e32b1a9c4cbbd4a5256 Mon Sep 17 00:00:00 2001 From: "John H. Hartman" Date: Fri, 8 Mar 2024 13:16:56 -0700 Subject: [PATCH 4/4] Make a note about -nl syntax being unstable Addressed reviewer's comments and make the note a real rst note. Signed-off-by: John H. Hartman --- doc/rst/usingchapel/multilocale.rst | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/doc/rst/usingchapel/multilocale.rst b/doc/rst/usingchapel/multilocale.rst index 8c2da59a7fc9..af1e32bab6b4 100644 --- a/doc/rst/usingchapel/multilocale.rst +++ b/doc/rst/usingchapel/multilocale.rst @@ -207,8 +207,12 @@ bound. The ``L`` is optional; if it isn't specified then Chapel will run the "ideal" number of locales based on the node architecture. Currently this is limited to the value of ``CHPL_RT_LOCALES_PER_NODE``; in future releases we plan to include more sophisticated heuristics such as automatically -running one locale per socket on nodes with multiple sockets. Note: the -``-nl NxLt`` syntax is considered unstable and may change in the future. +running one locale per socket on nodes with multiple sockets. + +.. note:: + + The ``-nl NxLt`` syntax is considered unstable and may change in the + future. By default, Chapel will try to bind co-locales to an architectural feature. For example, launching a Chapel program with the argument ``-nl 1x2`` on a