Add Urania machine and validate flag to scheduler #6184

guilara · 2024-07-30T09:00:40Z

Proposed changes

Add submit script information about Urania. Add no-validate flag to scheduler to prevent it from attempting to run mpi on the head node ( #6183).

Upgrade instructions

Code review checklist

The code is documented and the documentation renders correctly. Run
make doc to generate the documentation locally into BUILD_DIR/docs/html.
Then open index.html.
The code follows the stylistic and code quality guidelines listed in the
code review guide.
The PR lists upgrade instructions and is labeled bugfix or
new feature if appropriate.

Further comments

nilsvu · 2024-07-30T13:09:41Z

support/Python/Schedule.py

@@ -854,6 +857,11 @@ def scheduler_options(f):
            "You may also want to use '--clean-output'."
        ),
    )
+    @click.option(
+        "--no-validate",


Make this --validate/--no-validate, and set default=True. Then the variable is positive, which is much easier to understand (you have to change the variable name no_validate to validate)

Changed the name.

nilsvu · 2024-07-30T13:11:37Z

support/SubmitScripts/Urania.sh

+{{ super() -}}
+#SBATCH -D ./
+#SBATCH --nodes {{ num_nodes | default(1) }}
+#SBATCH --ntasks-per-node=1


We have found that 2 tasks per node work well on other clusters of this size, maybe even 3. I see you currently do 1 task but 2 comm threads. Have you tried 2 tasks with 1 comm thread each? Not sure if there's a difference @knelli2 @nilsdeppe do you know?

Typically each task = an MPI rank = a comm core

I haven't really experimented with this. So I don't know.

nilsvu · 2024-07-30T13:12:21Z

support/SubmitScripts/Urania.sh

+#SBATCH --cpus-per-task=72
+#SBATCH --mem=240000
+#SBATCH -t {{ time_limit | default("1-00:00:00") }}
+#SBATCH -p {{ queue | default("p.debug") }}


The default in the yaml is p.urania. Make it the same here?

Changed. Had left it when testing.

nilsvu · 2024-07-30T13:14:27Z

support/SubmitScripts/Urania.sh

+
+{% block head %}
+{{ super() -}}
+#SBATCH -D ./


This isn't needed I think

support/SubmitScripts/Urania.sh

nilsvu · 2024-07-30T16:27:34Z

support/SubmitScripts/Urania.sh

+module load gsl/1.16
+module load cmake/3.26
+module load hdf5-serial/1.12.2
+module load anaconda/3/2021.11


For the other clusters we have a file like support/Environments/caltech_hpc_gcc.sh that loads the modules. That enables you to load the modules to configure a build directory. Then you don't need to load the modules in the submit script, or rather, you would load the modules before submitting a job. I also load the modules in my ~/.bashrc sometimes (that's personal preference).

I added an environment file. I tried to remove the module loads from the submit script but it printed there where no modules loaded even if I had them loaded on my terminal when submitting. So I put them back in

You really don't want to duplicate the list of modules in these two places. Replace it here with this:

source ${SPECTRE_HOME}/support/Environments/urania.sh spectre_load_modules

nilsvu

You can squash

nilsvu · 2024-07-31T17:02:56Z

support/Environments/urania.sh

+    source /u/guilara/repos/spack/share/spack/setup-env.sh
+    spack env activate env3_spectre_impi


Note that you can have Spack generate a load file for you so you don't have to source setup-env.sh (which can take a long time): https://spack.readthedocs.io/en/latest/environments.html#loading

nilsvu · 2024-07-31T17:04:24Z

support/Environments/urania.sh

+    source /u/guilara/repos/spack/share/spack/setup-env.sh
+    spack env activate env3_spectre_impi
+    # Load python environment
+    source $SPECTRE_HOME/env/bin/activate


I suggest you put this Python environment in some fixed location so others can use it, e.g. /u/guilara/envs/spectre_python. Otherwise everyone has to create a new Py env in every spectre checkout.

I created this

nilsvu · 2024-07-31T17:07:08Z

support/Environments/urania.sh

+spectre_setup_charm_paths() {
+    # Define Charm paths
+    export CHARM_ROOT=/u/guilara/charm_impi_2/mpi-linux-x86_64-smp
+    export PATH=$PATH:/u/guilara/charm_impi_2/mpi-linux-x86_64-smp/bin


I suggest you create a module file like /u/guilara/modules/charm/7.0.0-impi-2 that sets these paths. Then you can module load it below and don't have to call this extra spectre_setup_charm_paths function.

I don't think I want to tackle this right now. Installing charm was a bit tricky

Then better set these paths in spectre_load_modules, or else you always have to call this extra function?

nilsvu · 2024-07-31T17:08:23Z

support/Environments/urania.sh

+          -D MEMORY_ALLOCATOR=JEMALLOC \
+          -D BUILD_PYTHON_BINDINGS=ON \
+          -D MACHINE=Urania \
+          -D Python_EXECUTABLE=${SPECTRE_HOME}/env/bin/python \


You can delete this line since the Py env is sourced, so cmake should find the correct python.

nilsvu · 2024-07-31T17:11:29Z

support/Environments/urania.sh

+          -D Python_EXECUTABLE=${SPECTRE_HOME}/env/bin/python \
+          -D Catch2_DIR=/u/guilara/repos/Catch2/install_dir/lib64/cmake/Catch2 \
+          -D MPI_C_COMPILER=/mpcdf/soft/SLE_15/packages/skylake\
+/impi/gcc_11-11.2.0/2021.7.1/bin/mpigcc \


So cmake doesn't find these without help even though the impi/2021.7 module is loaded?

I just tried removing them. I must have added them because it couldn't find it.

-- Could NOT find MPI_C (missing: MPI_C_LIB_NAMES MPI_C_HEADER_DIR MPI_C_WORKS) -- Could NOT find MPI (missing: MPI_C_FOUND C)

Is MPI needed? If everything works without explicitly linking MPI then just remove these lines.

I think we do need it, no?

No we don't link to it explicitly, only charm is built with it.

nilsvu · 2024-07-31T17:12:30Z

support/Python/Schedule.py

-    validate_input_file(
-        input_file_path.resolve(), executable=executable, work_dir=run_dir
-    )
+    if validate or validate is None:


if validate

nilsvu · 2024-07-31T17:12:44Z

support/Python/Schedule.py

@@ -861,6 +866,11 @@ def scheduler_options(f):
            "You may also want to use '--clean-output'."
        ),
    )
+    @click.option(
+        "--validate/--no-validate",
+        default=None,


default=True

nilsvu · 2024-07-31T17:13:50Z

support/SubmitScripts/Urania.sh

+{% endblock %}
+
+{% block charm_ppn %}
+# One thread for communication


~~One~~ Two threads

nilsvu · 2024-07-31T17:20:14Z

support/SubmitScripts/Urania.sh

+#SBATCH --ntasks-per-node=1
+#SBATCH --ntasks-per-core=1
+#SBATCH --cpus-per-task=72
+#SBATCH --mem=240000


--mem=240G (is this needed?)

nilsvu · 2024-07-31T17:26:42Z

support/SubmitScripts/Urania.sh

+module load gsl/1.16
+module load cmake/3.26
+module load hdf5-serial/1.12.2
+module load anaconda/3/2021.11


You really don't want to duplicate the list of modules in these two places. Replace it here with this:

source ${SPECTRE_HOME}/support/Environments/urania.sh spectre_load_modules

Fix validate flag Fix

knelli2

I think this looks good for now. If you want to change anything later, you can.

Please squash this all into two commits (after @nilsvu has looked at this again). 1 for the new validate option in python, and 2 for adding the Urania env, machine, and submit files.

knelli2 · 2024-08-13T16:26:21Z

support/SubmitScripts/Urania.sh

+source ${SPECTRE_HOME}/support/Environments/urania.sh
+spectre_load_modules
+spectre_setup_charm_paths


[optional] Might also want to module list after this so you can see what's loaded

nilsvu · 2024-08-13T16:42:19Z

support/Environments/urania.sh

+spectre_setup_charm_paths() {
+    # Define Charm paths
+    export CHARM_ROOT=/u/guilara/charm_impi_2/mpi-linux-x86_64-smp
+    export PATH=$PATH:/u/guilara/charm_impi_2/mpi-linux-x86_64-smp/bin


Then better set these paths in spectre_load_modules, or else you always have to call this extra function?

nilsvu · 2024-08-13T16:42:43Z

support/Environments/urania.sh

+    module load anaconda/3/2021.11
+    module load paraview/5.10
+    # Load Spack environment
+    source /u/guilara/repos/spack/share/spack/setup-env.sh


You can delete this line

nilsvu · 2024-08-13T16:44:55Z

support/Environments/urania.sh

+          -D Python_EXECUTABLE=${SPECTRE_HOME}/env/bin/python \
+          -D Catch2_DIR=/u/guilara/repos/Catch2/install_dir/lib64/cmake/Catch2 \
+          -D MPI_C_COMPILER=/mpcdf/soft/SLE_15/packages/skylake\
+/impi/gcc_11-11.2.0/2021.7.1/bin/mpigcc \


No we don't link to it explicitly, only charm is built with it.

nikwit · 2024-08-07T14:28:52Z

support/Environments/urania.sh

+    source /urania/u/guilara/repos/spack/var/spack/environments\
+/env3_spectre_impi/loads
+    # Load python environment
+    source /u/guilara/envs/spectre_env


missing "bin/activate" ?

nilsvu · 2024-10-24T18:37:45Z

support/Environments/urania.sh

+          -D MEMORY_ALLOCATOR=JEMALLOC \
+          -D BUILD_PYTHON_BINDINGS=ON \
+          -D MACHINE=Urania \
+          -D Catch2_DIR=/u/guilara/repos/Catch2/install_dir/lib64/cmake/Catch2


Backslash at end of line missing

guilara force-pushed the PRs-Urania-SubmitScript branch from 2700b51 to 5a846f9 Compare July 30, 2024 09:02

guilara requested a review from nilsvu July 30, 2024 09:09

guilara marked this pull request as draft July 30, 2024 09:37

guilara force-pushed the PRs-Urania-SubmitScript branch from 5a846f9 to 9ab841b Compare July 30, 2024 09:44

guilara marked this pull request as ready for review July 30, 2024 09:45

guilara force-pushed the PRs-Urania-SubmitScript branch 3 times, most recently from 6f41978 to 455043b Compare July 30, 2024 12:13

nilsvu requested changes Jul 30, 2024

View reviewed changes

guilara force-pushed the PRs-Urania-SubmitScript branch 3 times, most recently from eddd75c to 3d4f862 Compare July 31, 2024 15:45

guilara requested a review from nilsvu July 31, 2024 15:48

guilara changed the title ~~Add Urania machine and no-validate flag to scheduler~~ Add Urania machine and validate flag to scheduler Jul 31, 2024

nilsvu requested changes Jul 31, 2024

View reviewed changes

guilara added 4 commits August 5, 2024 16:18

Add Urania machine

73ccfaa

Add no-validate flag to scheduler

fd60a33

Change name of option

a94ce0f

Add environment file

ba2c34a

guilara force-pushed the PRs-Urania-SubmitScript branch from 3d4f862 to 4121fc5 Compare August 5, 2024 16:18

Clean script

1438c07

Fix validate flag Fix

guilara force-pushed the PRs-Urania-SubmitScript branch from 2c0688e to 1438c07 Compare August 5, 2024 18:11

guilara requested a review from nilsvu August 5, 2024 18:14

knelli2 reviewed Aug 13, 2024

View reviewed changes

nilsvu requested changes Aug 13, 2024

View reviewed changes

nikwit reviewed Aug 14, 2024

View reviewed changes

knelli2 added cli/pybindings Command line interface & Python bindings clusters Super computer support (env, submit, machine files) labels Aug 27, 2024

nilsvu requested changes Oct 24, 2024

View reviewed changes

		source /u/guilara/repos/spack/share/spack/setup-env.sh
		spack env activate env3_spectre_impi

Add Urania machine and validate flag to scheduler #6184

Are you sure you want to change the base?

Add Urania machine and validate flag to scheduler #6184

Conversation

guilara commented Jul 30, 2024 • edited Loading

Proposed changes

Upgrade instructions

Code review checklist

Further comments

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nilsvu Jul 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nilsvu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nilsvu Jul 31, 2024 • edited Loading

Choose a reason for hiding this comment

knelli2 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guilara commented Jul 30, 2024 •

edited

Loading

nilsvu Jul 31, 2024 •

edited

Loading

nilsvu Jul 31, 2024 •

edited

Loading